Managing Development Environments

Guidelines for working with Python virtual environments (venv) and conda environments. This skill ensures safe, organized package installations by always checking and confirming the active environment before proceeding.

When to Use This Skill

Activate this skill whenever:

• Installing Python packages or tools
• User requests to install dependencies
• Setting up a new Python project
• Debugging import or package issues
• Working with any Python development

Core Principles

1. Always check environment status before any installation
2. Always confirm with user which environment to use
3. Never install without environment confirmation
4. Warn if no environment is active
5. Help user choose appropriate environment type

---

Environment Detection Workflow

Step 1: Check Environment Status

Before ANY installation command, run these checks:

# Check for active venv
echo "Python executable: $(which python)"
echo "Virtual environment: $VIRTUAL_ENV"

# Check for conda environment
echo "Conda environment: $CONDA_DEFAULT_ENV"
conda info --envs 2>/dev/null || echo "Conda not available"

Step 2: Interpret Results

Scenario A: venv is active

Python executable: /path/to/project/.venv/bin/python
Virtual environment: /path/to/project/.venv
Conda environment:

→ Python venv is active

Scenario B: conda environment is active

Python executable: /path/to/miniconda3/envs/myenv/bin/python
Virtual environment:
Conda environment: myenv

→ Conda environment is active

Scenario C: No environment (system Python)

Python executable: /usr/bin/python
Virtual environment:
Conda environment:

→ ⚠️ No environment active! Warn user.

Scenario D: Both detected (rare)

Virtual environment: /path/to/.venv
Conda environment: base

→ Both active, prioritize what which python shows, but confirm with user

Step 3: Confirm with User

Always ask before proceeding:

I've detected the following environment:
- Environment type: [venv/conda/none]
- Location: [path]
- Python version: [version]

Is this the environment you want me to use for installing [package/tool]?

Wait for user confirmation before proceeding.

---

Installation Patterns by Environment Type

For Python venv

# Activate (if needed)
source .venv/bin/activate  # Linux/Mac
# or
.venv\Scripts\activate  # Windows

# Install packages
pip install package-name

# Install with specific version
pip install package-name==1.2.3

# Install from requirements
pip install -r requirements.txt

# Install development dependencies
pip install -e ".[dev]"

For Conda Environment

# Activate (if needed)
conda activate environment-name

# Install from conda-forge (preferred for scientific packages)
conda install -c conda-forge package-name

# Install with pip (if not available in conda)
pip install package-name

# Install from environment file
conda env update -f environment.yml

# Install specific version
conda install package-name=1.2.3

Critical: Channel Priority for Conda

For bioinformatics/scientific packages:

1. Try conda-forge first: conda install -c conda-forge package
2. Try bioconda for bio tools: conda install -c bioconda package
3. Fall back to pip only if not available: pip install package

Why: Conda manages binary dependencies better than pip for scientific packages.

Handling Conda TOS Acceptance Errors

If conda installation fails with:

CondaToSNonInteractiveError: Terms of Service have not been accepted

Solution: Use pip instead (installed packages work identically):

/path/to/envs/ENV_NAME/bin/pip install PACKAGE_NAME

Example - Installing JupyterLab:

# This fails with TOS error
conda install -n curation_paper -c conda-forge jupyterlab -y

# Use pip instead
/Users/delphine/miniconda3/envs/curation_paper/bin/pip install jupyterlab

Packages installed via pip integrate seamlessly with conda environments.

---

No Environment Active - Warning & Planning

If no environment is detected, DO NOT PROCEED with installation. Instead:

Step 1: Warn User

⚠️ WARNING: No Python environment detected!

You're currently using system Python:
- Location: [path to python]
- Version: [version]

Installing packages to system Python can:
- Cause conflicts with system packages
- Require sudo/admin privileges
- Make projects difficult to reproduce
- Break system tools that depend on specific versions

I recommend creating a virtual environment first.

Step 2: Help Choose Environment Type

Ask user about their project needs:

Decision Tree:

Question: What type of project are you working on?

A. Pure Python project (web dev, scripting, etc.)
   → Recommend: Python venv
   → Fast, lightweight, standard Python tool

B. Data science / Scientific computing
   → Ask: Do you need non-Python dependencies? (R, C libraries, etc.)

   B1. Yes (or using packages like numpy, scipy, pandas, etc.)
       → Recommend: Conda
       → Better binary dependency management

   B2. No, only Python packages
       → Recommend: Python venv
       → Simpler and faster

C. Bioinformatics / Genomics
   → Recommend: Conda (with bioconda channel)
   → Most tools available via bioconda
   → Manages complex dependencies well

D. Galaxy tool development
   → Recommend: Conda
   → Galaxy uses conda for tool dependencies
   → Direct compatibility

Step 3: Offer to Create Environment

For venv:

# Create venv
python -m venv .venv

# Activate
source .venv/bin/activate  # Linux/Mac

# Verify
which python

For conda:

# Create conda environment
conda create -n project-name python=3.11

# Activate
conda activate project-name

# Verify
conda info --envs
which python

---

Common Scenarios

Scenario 1: User Asks to Install Package

❌ Bad (without checking):

pip install pandas

✅ Good (with environment check):

1. Check environment status
2. Show user what environment is active
3. Ask: "Is this the environment you want to use?"
4. Wait for confirmation
5. Then: pip install pandas (or conda install as appropriate)

Scenario 2: Multiple Packages Needed

User: "Install pandas, numpy, and matplotlib"

Response:

Let me first check your environment...

[run environment detection]

I've detected: [environment details]

I need to install:
- pandas
- numpy
- matplotlib

Is this the correct environment? Should I use [pip/conda install -c conda-forge]?

Scenario 3: User in Wrong Environment

User: "Install scikit-learn" (but they're in a different project's venv)

Response:

⚠️ I've detected you're in a virtual environment at:
/path/to/other-project/.venv

But we're working in directory:
/path/to/current-project

This might be the wrong environment. Options:
1. Deactivate and create/activate the correct environment for this project
2. Continue with current environment (if this is intentional)
3. Cancel installation

Which would you prefer?

Scenario 4: Conda Base Environment

Detection: User is in conda base environment

Response:

⚠️ You're currently in the conda 'base' environment.

It's generally better practice to create a separate environment for each project
to avoid dependency conflicts.

Options:
1. Create a new conda environment for this project
2. Continue with base environment (not recommended)

Would you like me to create a project-specific environment?

---

Environment Selection Guidelines

Use Python venv When:

• ✅ Pure Python project
• ✅ Simple dependencies (all available on PyPI)
• ✅ Standard web development (Django, Flask, FastAPI)
• ✅ No compiled extensions or C libraries
• ✅ Want fastest environment creation
• ✅ Working with Python 3.3+

Advantages:

• Lightweight and fast
• Built into Python (no extra installation)
• Simple and standard
• Works on all platforms

Disadvantages:

• Harder to manage non-Python dependencies
• Binary packages may need system libraries
• Version conflicts harder to resolve

Use Conda Environment When:

• ✅ Data science / machine learning
• ✅ Scientific computing (numpy, scipy, pandas)
• ✅ Bioinformatics / genomics
• ✅ Need specific Python versions
• ✅ Cross-language dependencies (R, C++, etc.)
• ✅ Galaxy tool development
• ✅ Complex binary dependencies

Advantages:

• Manages binary dependencies
• Cross-language support
• Better for scientific packages
• Can manage Python version
• Excellent for reproducibility

Disadvantages:

• Slower than venv
• Larger disk space usage
• Requires conda installation
• More complex

---

Best Practices

1. One Environment Per Project

Good:

project-a/
├── .venv/          # project-a's environment
├── requirements.txt
└── src/

project-b/
├── .venv/          # project-b's environment
├── requirements.txt
└── src/

Bad:

# Using same environment for multiple projects
# → Version conflicts inevitable

2. Document Environment Setup

Always create environment specification files:

For venv:

# Create requirements.txt
pip freeze > requirements.txt

# Or for development
pip freeze > requirements-dev.txt

For conda:

# Create environment.yml
conda env export > environment.yml

# Or minimal version
conda env export --from-history > environment.yml

3. Add Environment to .gitignore

# Python venv
.venv/
venv/
env/

# Conda
.conda/

4. Include Python Version

For venv (in README):

Python 3.11+ required

For conda (in environment.yml):

name: myproject
dependencies:
  - python=3.11
  - pandas
  - numpy

---

Resumable Data Fetch Scripts

Pattern for Long-Running Data Retrieval

When building scripts that fetch data from external sources (APIs, S3, etc.), implement these patterns for robustness:

def fetch_with_caching(item_id, output_dir):
    """Fetch data with local caching to support resume."""
    output_file = output_dir / f"{item_id}_data.txt"

    # Skip if already downloaded
    if output_file.exists():
        with open(output_file, 'r') as f:
            content = f.read()
        return True, "cached", content

    # Fetch logic here...
    # Save to file on success

    return success, status, content

def main():
    # Load existing results to skip completed work
    existing_csv = base_dir / 'results.csv'
    if existing_csv.exists():
        df_existing = pd.read_csv(existing_csv)
        existing_ids = set(df_existing['id'].tolist())
        df_to_process = df_to_process[~df_to_process['id'].isin(existing_ids)]
        print(f"Skipping {len(existing_ids)} items with existing data")

    # Process remaining items
    for item in df_to_process:
        success, status, data = fetch_with_caching(item, output_dir)
        # Process and accumulate results...

    # Merge with existing results
    if existing_csv.exists():
        df_existing = pd.read_csv(existing_csv)
        df_combined = pd.concat([df_existing, df_new], ignore_index=True)
    else:
        df_combined = df_new

    # Save combined results
    df_combined.to_csv(output_csv, index=False)

Key Features

1. File-level caching: Save each fetch result immediately
2. Resume capability: Skip already-processed items automatically
3. Incremental results: Merge new results with existing data
4. Interruption-safe: Can be stopped and restarted without data loss

Progress Tracking

try:
    from tqdm import tqdm
    iterator = tqdm(items, total=len(items), desc="Processing")
except ImportError:
    print("(Install tqdm for progress bar: pip install tqdm)")
    iterator = items

Rate Limiting

import time

for item in items:
    result = fetch_data(item)
    if result.success:
        time.sleep(0.2)  # Be respectful to external services

This pattern makes scripts production-ready and user-friendly for long-running operations.

---

Troubleshooting

Issue 1: "pip: command not found" in venv

Problem: venv created but pip not available

Solution:

python -m ensurepip
python -m pip install --upgrade pip

Issue 2: Conda environment activation fails

Problem: conda activate doesn't work

Solution:

# Initialize conda for shell
conda init bash  # or zsh, fish, etc.

# Restart shell or source
source ~/.bashrc

Issue 3: Wrong Python version in environment

For venv:

# Specify Python version when creating
python3.11 -m venv .venv

For conda:

# Specify version explicitly
conda create -n myenv python=3.11

Issue 4: Package conflicts

For venv:

# Start fresh
rm -rf .venv
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For conda:

# Try updating resolver
conda update -n base conda
conda install -c conda-forge mamba  # Faster solver
mamba install package-name

---

Quick Reference

Environment Detection Commands

# Check what's active
which python
echo $VIRTUAL_ENV
echo $CONDA_DEFAULT_ENV

# Python version
python --version

# Installed packages
pip list          # for pip
conda list        # for conda

# Environment location
pip show package-name  # shows where package is installed

Activation Commands

# venv
source .venv/bin/activate                    # Linux/Mac
.venv\Scripts\activate                       # Windows

# conda
conda activate environment-name

# Deactivation
deactivate                                   # venv
conda deactivate                             # conda

---

Examples

Example 1: Safe Installation Flow

User: "Install requests"

Claude:

Let me check your environment first...

[Checks environment]

I see you're in a Python venv at: /home/user/project/.venv
Python version: 3.11.5

Is this the correct environment for installing requests?

User: "Yes"

Claude:

pip install requests

Example 2: No Environment Warning

User: "Install scikit-learn"

Claude:

⚠️ WARNING: No virtual environment detected!

You're using system Python at: /usr/bin/python3
Installing to system Python is not recommended.

This is a data science package. I suggest creating a conda environment:

conda create -n ml-project python=3.11
conda activate ml-project
conda install -c conda-forge scikit-learn

Would you like me to help set this up?

Example 3: Environment Type Suggestion

User: "I need to install biopython, numpy, and pandas"

Claude:

I notice you need scientific and bioinformatics packages.

Currently: No environment active

I recommend creating a conda environment because:
- Better binary dependency management for numpy/pandas
- Bioconda channel has biopython
- Manages versions more reliably

Suggested setup:
conda create -n bio-project python=3.11
conda activate bio-project
conda install -c conda-forge pandas numpy
conda install -c bioconda biopython

Would you like to proceed with this?

---

Summary

Always remember:

1. ✅ Check environment before any installation
2. ✅ Show user what environment is active
3. ✅ Confirm with user before proceeding
4. ⚠️ Warn if no environment is active
5. 💡 Help choose appropriate environment type
6. 📝 Document environment setup for reproducibility

Never:

• ❌ Install without checking environment
• ❌ Assume user wants to use system Python
• ❌ Install without user confirmation
• ❌ Skip warning about base conda environment

This skill ensures clean, reproducible, conflict-free Python environments across all projects.