Working in Notebooks

Use this skill to create, maintain, and choose between notebook environments (Jupyter, marimo, Colab) for data work. Covers tool selection, reproducibility patterns, and workflow best practices.

When to use this skill

• Setting up a notebook environment — choosing between Jupyter, marimo, VS Code, or Colab
• Converting between notebook formats — Jupyter to marimo, .ipynb to .py, or vice versa
• Making notebooks reproducible — pinning dependencies, managing random seeds, avoiding hardcoded paths
• Improving notebook structure — organizing cells, refactoring code, adding tests
• Publishing or sharing notebooks — nbconvert, Quarto, Voilà, or Git workflows
• Jupyter-specific features — magic commands, widgets, extensions, kernel management
• Marimo-specific workflows — reactive execution, UI components, version control patterns

When NOT to use this skill

Use a different skill for these related but distinct tasks:

Instead of...	Use this skill	Because...
Building a stakeholder-facing dashboard	building-data-apps	Apps are for external users; notebooks are for analysts/developers
Creating interactive data explorers for non-technical users	building-data-apps	Streamlit, Panel, Gradio are purpose-built for this
Exploratory data analysis patterns	analyzing-data	EDA patterns (profiling, statistical tests) belong there
Visualization library selection	analyzing-data	Chart types and library comparison is covered there
Production ML feature engineering	engineering-ml-features	Feature engineering logic is domain-specific
Model evaluation and cross-validation	evaluating-ml-models	Model comparison and metrics belong there

Quick boundary check

• Notebook = code + markdown + outputs in cells, run interactively, often .ipynb or .py format
• Data app = deployed web interface with widgets, for non-coders to interact with
• If the user asks for a "dashboard," "app," or mentions "users clicking buttons," use building-data-apps
• If the user asks for "notebook," "Jupyter," "marimo," or "explore data interactively," use this skill

Tool selection guide

Quick decision checklist

Question	If yes, consider
Need reactive execution (cells auto-update)?	marimo
Want pure Python files for version control?	marimo
Need specific Jupyter extensions ecosystem?	JupyterLab
Using Google Colab features (TPU, shared GPUs)?	Google Colab
Want IDE-native experience (IntelliSense, debugger)?	VS Code + Jupyter
Converting from existing .ipynb files?	Jupyter → marimo with marimo convert
Teaching beginners (familiarity matters)?	JupyterLab or Colab

Tool comparison

Tool	Best For	Key Feature	File Format
JupyterLab	Traditional data science, rich extensions	Full IDE experience, 1000+ extensions	.ipynb (JSON)
marimo	Reproducible notebooks, reactive execution	Python-native, version-control friendly	.py (pure Python)
VS Code + Jupyter	IDE-native notebook experience	IntelliSense, debugging, git integration	.ipynb
Google Colab	Cloud GPUs, easy sharing, collaboration	Free TPU/GPU, zero setup	.ipynb (cloud)

Core workflow: Creating a reproducible notebook

Step 1: Choose your tool

See decision checklist above. If starting fresh and reproducibility matters → marimo. If ecosystem/extensions matter → JupyterLab.

Step 2: Set up the environment

# Cell 1: Environment setup (run first)
# Set random seeds for reproducibility
import numpy as np
import random

np.random.seed(42)
random.seed(42)

# For torch users:
# import torch
# torch.manual_seed(42)

Step 3: Pin dependencies

Create requirements.txt or environment.yml:

# requirements.txt
pandas==2.1.0
numpy==1.24.0
matplotlib==3.7.0

Or use modern tools:

# With uv
uv pip freeze > requirements.txt

# With poetry
poetry export -f requirements.txt > requirements.txt

Step 4: Structure for readability

# Title: Clear project/question description

## Setup
Imports and configuration

## Data Loading
Load and validate data

## Analysis
- Subsection per question/hypothesis
- Clear markdown explanations
- Visualizations with interpretations

## Conclusions
Key findings and next steps

Step 5: Never hardcode secrets

# ✅ Use environment variables
import os

api_key = os.environ.get("OPENAI_API_KEY")

# ❌ Never do this
api_key = "sk-abc123..."

Step 6: Clean outputs before git (Jupyter)

# Install nbstripout
pip install nbstripout
nbstripout --install

# Or use pre-commit
pip install pre-commit
pre-commit install

Validation and feedback loop

Self-check questions

Before considering a notebook "done":

1. Can someone else run this from a fresh environment?
2. Are all random seeds set?
3. Are dependencies pinned (requirements.txt or similar)?
4. Are secrets loaded from environment variables?
5. Are cells organized logically (not execution-order dependent)?
6. Are helper functions extracted to .py files if >30 lines?
7. Are outputs stripped before committing (if using Jupyter)?

Testing notebook code

See ../analyzing-data/references/notebook-testing.md for:

• Unit tests for notebook code
• nbval for output validation
• Papermill for parameterized execution

Progressive disclosure

Core references

• references/jupyter-guide.md — Jupyter/JupyterLab deep dive: magic commands, widgets, extensions, kernel management
• references/marimo-guide.md — marimo deep dive: reactive execution, UI components, migration from Jupyter
• references/reproducibility-patterns.md — Environment management, dependency pinning, nbstripout, secrets handling

Related references (in other skills)

• ../analyzing-data/references/notebook-testing.md — Unit tests, nbval, Papermill for notebook validation
• ../analyzing-data/references/sharing-publishing.md — nbconvert, Quarto, Voilà for publishing notebooks

External resources

• Jupyter Documentation
• marimo Documentation
• nbstripout
• Quarto

Common anti-patterns

• ❌ Running cells out of order (Jupyter) → Use "Run All" to verify, or switch to marimo
• ❌ Giant cells with mixed concerns → One concept per cell, <50 lines
• ❌ Hardcoded file paths → Use relative paths or environment variables
• ❌ Hardcoded secrets → Load from environment
• ❌ Committing large output files → Use .gitignore, data/ folder, or strip outputs
• ❌ Inline data → Use data/ folder or external sources
• ❌ No markdown explanations → Every code block deserves context

Quick commands reference

Jupyter

# Start JupyterLab
jupyter lab

# Convert notebook
jupyter nbconvert notebook.ipynb --to html
jupyter nbconvert notebook.ipynb --to script

# List kernels
jupyter kernelspec list

# Install kernel for virtual environment
python -m ipykernel install --user --name=myenv

marimo

# Create/edit a notebook
marimo edit notebook.py

# Run as app (read-only)
marimo run notebook.py

# Convert from Jupyter
marimo convert notebook.ipynb -o notebook.py

# Export to HTML
marimo export html notebook.py -o notebook.html

Environment validation

# Check installed versions
import pandas as pd
import numpy as np

print(f"pandas: {pd.__version__}")
print(f"numpy: {np.__version__}")

Related skills

Skill	Relationship	When to use
analyzing-data	Complementary	EDA patterns, profiling, statistical tests—use with notebooks
building-data-apps	Distinct boundary	Building stakeholder-facing dashboards—not this skill
evaluating-ml-models	Complementary	Cross-validation, metrics, experiment tracking
engineering-ml-features	Complementary	Feature engineering patterns and transformations

Migration notes

This skill replaces data-science-notebooks with the following changes:

• Removed dependsOn from frontmatter (non-standard field)
• Added explicit when-to-use and when-not-to-use sections
• Split content into focused reference files
• Clear boundary documentation vs building-data-apps
• Progressive disclosure with direct file paths (no @skill hybrid syntax)