DevOps and Deployment

Task Queues

My recommended task queues:

Luigi

Why I recommend it:

• Pipeline workflow management
• Dependency resolution
• Task scheduling and monitoring
• Built-in visualization
• Originally by Spotify

Key Features:

• 📈 Dependency graphs
• 📅 Task scheduling
• 📊 Visualization UI
• 🔄 Failure handling
• 💾 Storage integration

Quick Example:

import luigi

class FetchData(luigi.Task):
    date = luigi.DateParameter()

    def output(self):
        return luigi.LocalTarget(f"data_{self.date}.csv")

    def run(self):
        # Fetch and save data
        with self.output().open('w') as f:
            f.write('data')

class ProcessData(luigi.Task):
    date = luigi.DateParameter()

    def requires(self):
        return FetchData(date=self.date)

    def output(self):
        return luigi.LocalTarget(f"processed_{self.date}.csv")

    def run(self):
        # Process the input data
        with self.input().open('r') as fin, \
             self.output().open('w') as fout:
            fout.write(fin.read().upper())

Environment Management

My recommended environment management tools:

UV - Recommended!

Why I recommend it:

• Ultra-fast Python environment and package manager
• Compatible with pip and requirements.txt
• Lockfile support for reproducible installs
• Simple CLI usage
• Cross-platform

Key Features:

• 🚀 Speed
• 🔒 Lockfile support
• 🐍 Virtualenv management
• 📦 Package installation
• 🔄 Pip compatibility

Quick Example:

# Install uv (one-liner)
pip install uv

# Create a new virtual environment
uv venv .venv

pyenv

Why I recommend it:

• Python version management
• Multiple Python versions
• Project-specific Python versions
• No sudo required

Key Features:

• 🔄 Version switching
• 💻 Local installation
• 🔧 Shell integration
• 📦 Plugin system

Quick Example:

# Install Python versions
pyenv install 3.12.0
pyenv install 3.11.0

# Set global version
pyenv global 3.12.0

# Set local version for project
cd myproject
pyenv local 3.11.0