Python - Dependency Management

development python reference

---

The Old Way: requirements.txt

Historically, Python projects listed dependencies in a plain text file:

# requirements.txt
Flask>=3.0.0
click>=8.0
pytest>=7.0

Install with:

pip install -r requirements.txt

Problems with requirements.txt

Problem	Description
No project metadata	Can’t define project name, version, description, or entry points
No dependency groups	Dev tools, test tools, and production deps lumped together (or you maintain multiple files like requirements-dev.txt)
Not a package	Your project isn’t installable — can’t run pip install . or share it
No build system	No standard way to define how to build or distribute your project
Manual management	You have to keep the file in sync manually; nothing enforces it
No extras	Can’t define optional feature sets (e.g., install web deps only if needed)

Many projects ended up with a mess of files:

requirements.txt
requirements-dev.txt
requirements-test.txt
requirements-prod.txt
setup.py
setup.cfg
MANIFEST.in

---

The Modern Way: pyproject.toml

PEP 621 (2021) standardized pyproject.toml as the single source of truth for Python project configuration. It replaces setup.py, setup.cfg, requirements.txt, and most config files.

Anatomy of pyproject.toml

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
 
[project]
name = "site-discovery-agent"
version = "0.2.0"
description = "Network discovery and audit tool"
requires-python = ">=3.10"
 
# Production dependencies — always installed
dependencies = [
    "click>=8.0",
    "Flask>=3.0.0",
    "pydantic>=2.0",
    "rich>=13.0",
]
 
# Optional dependency groups — installed on demand
[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-mock>=3.0",
]
web = [
    "Flask>=3.0.0",
]
lint = [
    "ruff>=0.1.0",
    "mypy>=1.0",
]
 
# CLI entry points
[project.scripts]
site-discovery = "discovery.cli:main"

What Each Section Does

[build-system] — Tells pip how to build the project. setuptools is the most common backend, but hatchling, flit, and poetry-core are also popular.

[project] — Core metadata: name, version, Python version requirement, and production dependencies.

dependencies — Packages that are always required. These get installed whenever someone runs pip install .

[project.optional-dependencies] — Groups of packages needed only in certain contexts. These are called “extras.”

[project.scripts] — Creates executable commands. After install, you can run site-discovery directly from the terminal instead of python -m discovery.

---

Understanding pip install -e ".[dev,web]"

pip install

Standard pip install command.

. (dot)

Means “install the project in the current directory.” Pip reads pyproject.toml from . and installs the package along with its dependencies.

-e (editable mode)

Stands for editable. Instead of copying your code into the Python site-packages directory, pip creates a symlink back to your source code.

Without -e:

pip install .
# Copies code to: .venv/lib/python3.12/site-packages/discovery/
# If you edit source files, changes are NOT reflected until you reinstall

With -e:

pip install -e .
# Links to: /home/you/project/discovery/
# Edit source files and changes take effect immediately
# Perfect for active development

[dev,web] (extras)

Installs the optional dependency groups named dev and web from [project.optional-dependencies].

pip install .              # Only production dependencies
pip install ".[dev]"       # Production + dev (pytest, etc.)
pip install ".[web]"       # Production + web (Flask)
pip install ".[dev,web]"   # Production + dev + web

The quotes around ".[dev,web]" are needed because square brackets have special meaning in most shells (bash, zsh).

Putting It All Together

pip install -e ".[dev,web]"

This single command:

1. Reads pyproject.toml in the current directory
2. Installs all production dependencies (click, pydantic, rich, etc.)
3. Installs the dev extras (pytest, pytest-mock)
4. Installs the web extras (Flask)
5. Links the project in editable mode so code changes are live
6. Registers any CLI entry points (site-discovery command)

---

Common Patterns

Development setup (first time)

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,web]"

CI/CD pipeline (no editable needed)

pip install ".[dev]"    # Install with test deps, run tests

Production deployment

pip install .           # Only production dependencies

Adding a new dependency

Edit pyproject.toml directly, then reinstall:

# Add "requests>=2.31" to the dependencies list in pyproject.toml, then:
pip install -e ".[dev,web]"

---

Version Specifiers

Specifier	Meaning	Example
>=	Minimum version	click>=8.0
==	Exact version	Flask==3.0.0
~=	Compatible release	pydantic~=2.0 (allows 2.x, not 3.0)
<	Maximum version	urllib3<2.0
Combined	Range	click>=8.0,<9.0

Best practice: Use >= for libraries (flexible) and == for applications where you want reproducibility. For most projects, >= with a major version floor is the sweet spot.

---

When You Might Still Want requirements.txt

Use Case	Why
Pinned/locked dependencies	pip freeze > requirements.lock captures exact versions for reproducible deploys
Legacy CI systems	Some CI tools only understand requirements.txt
Non-package scripts	A loose collection of scripts that isn’t really a “project”

Even for pinning, modern tools like pip-compile (from pip-tools) can generate a lockfile from your pyproject.toml:

pip install pip-tools
pip-compile -o requirements.lock pyproject.toml

---

Quick Reference

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate       # Linux/Mac
.venv\Scripts\activate          # Windows
 
# Install project (editable + all extras)
pip install -e ".[dev,web]"
 
# Install only production deps
pip install .
 
# See what's installed
pip list
 
# Generate lockfile for reproducible deploys
pip freeze > requirements.lock
 
# Upgrade a dependency
pip install --upgrade Flask

---

Summary

	requirements.txt	pyproject.toml
Standard	Convention only	PEP 621 (official)
Project metadata	No	Yes
Dependency groups	Multiple files	Built-in extras
Editable installs	No	Yes (-e .)
CLI entry points	No	Yes
Build/distribute	Needs setup.py	Self-contained
Tool config	Separate files	Unified (pytest, ruff, mypy, etc.)

pyproject.toml is the standard. Use it for any new Python project.

---

Related

• Python - Quick Reference
• Python - OOP
• Docker - Cheat Sheet