The Ultimate Python Project Development Workflow

Overview

In this blog post, I’ll introduce the tools and philosophy I’ve used to successfully manage over 500 Python projects, including 150+ open source repositories, 200+ closed-source proof-of-concept initiatives, 100+ enterprise-grade applications, and 50+ production systems that consistently deliver value. I’ll demonstrate how I maintain exceptionally high quality across all projects through production-grade dependency management, robust CI/CD pipelines, and documentation-as-code practices. These techniques have enabled me to achieve what typically requires an entire team - all as a single developer.

Challenge and Motivation

The Enterprise Onboarding Maze

If you’ve worked in enterprise environments, you’re familiar with the onboarding process for new projects: asking about Git branching strategies, dependency management approaches, coding standards, testing methodologies, and CI/CD pipelines. Even when focusing solely on Python projects, there are countless different implementation approaches. Teams may have different folder structures, prefer different dependency management tools (Poetry, Pipenv, UV, PDM), and utilize various frameworks for testing, packaging, and deployment.

Beyond Single Team Boundaries

This diversity exists within a single team at one enterprise. Throughout my career, I’ve encountered a much wider variety of project types - enterprise applications, open-source libraries, internal reusable components, standalone applications, and personal projects. Each type requires different tooling choices and setup configurations. The sheer variety makes it impossible to memorize every approach, and adapting to different styles consumes valuable time and mental energy.

Common Patterns, Different Implementations

Taking a step back, I realized that despite these differences, all Python projects share fundamental needs: setting up virtual environments, managing dependencies, running tests, packaging code, and implementing CI/CD. The concepts remain consistent, but the specific CLI commands or scripts vary based on project style and type. This insight led me to ask: Could I create lightweight adapters that would allow me to memorize only key concepts—each represented by a simple, concise command—while a tool handles the implementation details behind the scenes?

The Setup Bottleneck

However, even these simplified commands require a properly structured repository with the right folder organization, tool configuration, and script placement. Setting this up for each new project is time-consuming, especially when you’re juggling dozens or hundreds of projects. I wondered: What if I could create template projects for each type? Then, whenever I need a new project, I could execute a single command, specify a name, and have everything configured automatically. This would allow me to focus on development rather than setup, using those same abstract commands to handle all the complex operations behind the scenes.

A 15-Year Evolution

This approach - templates combined with simplified workflow commands - is what I’ve built and refined over the past 15 years. In this document, I’ll explain my design philosophy and methodology in detail, showing how this system has enabled me to efficiently manage hundreds of projects of varying types and requirements while maintaining consistently high quality.

System Architecture: Template Based Project Automation

Two-Tier Framework

The architecture consists of two key components working together to streamline project development and management: template repositories and an automation library. This approach separates project structure from operational workflows, creating a scalable system for handling multiple project types.

Template Repositories

Template repositories provide the foundation for each project type - whether open source, enterprise, or proof-of-concept. Each template includes pre-configured folder structures, configuration files, and essential scripts. By simply entering basic information like project name and email, you can generate a fully structured project ready for development, eliminating repetitive setup tasks.

Automation Library Design

The automation library serves as a command abstraction layer, wrapping complex CLI operations in simple, memorable Python functions. Built using Python’s subprocess module, it transforms multi-parameter commands into streamlined operations with minimal arguments. The library incorporates intelligent conditional logic to assess the current project state and execute appropriate commands accordingly.

Evolution From Embedded to Standalone

Initially, I embedded automation code directly within each template repository for transparency. However, I discovered that automation logic rarely changes for a specific project type. To reduce code duplication across hundreds of projects, I extracted these components into standalone, installable libraries—dramatically reducing the template’s footprint while maintaining functionality.

Integration Mechanics

Together, these components create a powerful development ecosystem. Template repositories simplify project creation, while the automation library normalizes day-to-day operations across all projects. This standardization allows me to focus on development rather than remembering project-specific commands or procedures, dramatically increasing productivity when managing hundreds of projects simultaneously.

Creating Reusable Project Templates: From Concept to Implementation

To demonstrate this system, I’ll walk through developing a reusable open-source project template - the most versatile template type I’ve created and the foundation for hundreds of my GitHub repositories. This example showcases all key architectural elements while providing a practical implementation you can adapt for your own needs.

The Workflow Lifecycle

The open-source project workflow typically includes:

• environment creation

• dependency management

• testing

• CI integration with GitHub Actions

• coverage reporting via CodeCov

• documentation hosting on ReadTheDocs

• and publishing to both PyPI and GitHub Releases.

Implementing this workflow systematically requires thoughtful architecture and planning.

Three-Repository Architecture

My approach uses three distinct repositories that work together to create a maintainable template system:

1. Seed Repository - A fully functional yet business-logic-free open-source project that serves as a testing ground for workflow automation. All automation logic resides in the bin/ directory as a Python package.

2. Automation Library - A standalone Python package containing workflow actions extracted from the Seed Repository’s bin/ directory. This library can be installed globally, providing consistent command interfaces across all projects, similar to Poetry’s functionality.

3. Cookiecutter Template - Generated from the Seed Repository using the cookiecutter_maker tool, this creates a cookiecutter template that becomes the starting point for all new projects.

Development Workflow

This architecture creates a clear development path: I first implement and test all workflow actions in the Seed Repository. Once proven effective, I copy the implementation to the Automation Library for distribution. Finally, I transform the Seed Repository into a Cookiecutter Template.

Before publishing the template, I validate it by generating a new project and comparing it to the original Seed Repository using git diff. If there are no differences and all workflow actions execute successfully, the template is ready for release. Future modifications follow the same cycle through all three repositories.

Naming Conventions

For clarity and consistency, I follow specific naming patterns:

• Seed Repository: cookiecutter_pywf_open_source_demo-project (cookiecutter_pywf_${workflow_name}_demo-project)

• Automation Library: pywf_open_source-project (pywf_${workflow_name}).

• Cookiecutter Template: cookiecutter_pywf_open_source-project (cookiecutter_pywf_${workflow_name})

Common Workflow Action:

$ make
make
help                                     ** Show this help message
venv-create                              ** Create Virtual Environment
venv-remove                              ** Remove Virtual Environment
poetry-lock                              Resolve dependencies using poetry, update poetry.lock file
install-root                             Install Package itself without any dependencies
install                                  ** Install main dependencies and Package itself
install-dev                              Install Development Dependencies
install-test                             Install Test Dependencies
install-doc                              Install Document Dependencies
install-automation                       Install Dependencies for Automation Script
install-all                              Install All Dependencies
poetry-export                            Export dependencies to requirements.txt
test                                     ** Run test
test-only                                Run test without checking test dependencies
cov                                      ** Run code coverage test
cov-only                                 Run code coverage test without checking test dependencies
int                                      ** Run integration test
int-only                                 Run integration test without checking test dependencies
view-cov                                 View code coverage test report
build-doc                                Build documentation website locally
view-doc                                 View documentation website locally
build                                    Build Python library distribution package
publish                                  Publish Python library to Public PyPI
release                                  Create Github Release using current version
setup-codecov                            Setup Codecov Upload token in GitHub Action Secrets
setup-rtd                                Create ReadTheDocs Project