Python Packaging

Create, structure, and distribute Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

When to Invoke

Creating Python libraries for distribution
Building CLI tools with entry points
Publishing packages to PyPI or private repositories
Setting up Python project structure (src layout vs flat)
Configuring pyproject.toml, setup.py, or setup.cfg
Building wheels and source distributions
Versioning and releasing Python packages
Creating namespace packages or multi-package projects

Core Concepts

Package Structure

Source layout: src/package_name/ (recommended)
Flat layout: package_name/ (simpler but less flexible)
Package metadata: pyproject.toml, setup.py, or setup.cfg
Distribution formats: wheel (.whl) and source distribution (.tar.gz)

Modern Packaging Standards

PEP 517/518: Build system requirements
PEP 621: Metadata in pyproject.toml
PEP 660: Editable installs
pyproject.toml: Single source of configuration

Build Backends

setuptools: Traditional, widely used
hatchling: Modern, opinionated
flit: Lightweight, for pure Python
poetry: Dependency management + packaging

Distribution

PyPI: Python Package Index (public)
TestPyPI: Testing before production
Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/
  pyproject.toml
  README.md
  LICENSE
  src/
    my_package/
      __init__.py
      module.py
  tests/
    test_module.py

Minimal pyproject.toml

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
version = "0.1.0"
description = "A short description"
authors = [{name = "Your Name", email = "you@example.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
    "requests>=2.28.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "black>=22.0",
]

Source Layout Configuration

[tool.setuptools.packages.find]
where = ["src"]

Package Structure Patterns

Source Layout (Recommended)

my-package/
  pyproject.toml
  README.md
  LICENSE
  .gitignore
  src/
    my_package/
      __init__.py
      core.py
      utils.py
      py.typed
  tests/
    __init__.py
    test_core.py
    test_utils.py
  docs/
    index.md

Advantages: prevents accidental imports from source, cleaner test imports, better isolation.

Flat Layout

my-package/
  pyproject.toml
  README.md
  my_package/
    __init__.py
    module.py
  tests/
    test_module.py

Simpler, but can import package without installing.

Multi-Package Project

project/
  pyproject.toml
  packages/
    package-a/
      src/
        package_a/
    package-b/
      src/
        package_b/
  tests/

Dynamic Versioning

[build-system]
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
dynamic = ["version"]

[tool.setuptools.dynamic]
version = {attr = "my_package.__version__"}

In init.py:

__version__ = "1.0.0"

# Or with setuptools-scm
from importlib.metadata import version
__version__ = version("my-package")

Building and Publishing

Build Package Locally

pip install build twine

# Build distribution
python -m build

# Check the distribution
twine check dist/*

Publishing to PyPI

# Test on TestPyPI first
twine upload --repository testpypi dist/*

# Install from TestPyPI to verify
pip install --index-url https://test.pypi.org/simple/ my-package

# Publish to PyPI
twine upload dist/*

Using API tokens (recommended):

# Create ~/.pypirc
[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username = __token__
password = pypi-...your-token...

[testpypi]
username = __token__
password = pypi-...your-test-token...

Testing Installation

Editable Install

pip install -e .
pip install -e ".[dev]"
pip install -e ".[dev,docs]"

Testing in Isolated Environment

python -m venv test-env
source test-env/bin/activate

pip install dist/my_package-1.0.0-py3-none-any.whl
python -c "import my_package; print(my_package.__version__)"
my-tool --help

deactivate
rm -rf test-env

Version Constraints

dependencies = [
    "requests>=2.28.0,<3.0.0",  # Compatible range
    "click~=8.1.0",              # Compatible release (>=8.1.0,<8.2.0)
    "pydantic>=2.0",             # Minimum version
    "numpy==1.24.3",             # Exact version (avoid if possible)
]

Best Practices

Use src/ layout for cleaner package structure
Use pyproject.toml for modern packaging
Pin build dependencies in build-system.requires
Version appropriately with semantic versioning
Include all metadata (classifiers, URLs, etc.)
Test installation in clean environments
Use TestPyPI before publishing to PyPI
Document thoroughly with README and docstrings
Include LICENSE file
Automate publishing with CI/CD

Checklist for Publishing

References

references/packaging-guide.md - full-featured pyproject.toml example with all tool configs, CLI patterns (Click and argparse), CI/CD publishing with GitHub Actions, multi-architecture wheels, data files, namespace packages, C extensions, git-based versioning, private package index, file templates (.gitignore, MANIFEST.in, README.md)

Resources

Python Packaging Guide: https://packaging.python.org/
PyPI: https://pypi.org/
TestPyPI: https://test.pypi.org/
setuptools documentation: https://setuptools.pypa.io/
build: https://pypa-build.readthedocs.io/
twine: https://twine.readthedocs.io/

python-packaging