In [*]: F(mind)

Doctoral Researcher / Teacher Assistant @ uni.lu

Freelance / Data Scientist Mentor @ OpenClassrooms

I work on Android Security, Big Data and Machine Learning

Improve your Python Project: A Layered Approach

Cookiecutter is a wonderful tool to create Python projects from the command line.

Yet, I've become frustrated by the structure of cookiecutter templates I found in the wild.

On one hand, project templates should be extensible and allow the inclusion of new tools.

On the other hand, it should be possible to update existing tools without impacting the project.

In this article, I propose a layered approach to better create, maintain and organize Python project.

This approach is based on cookiecutter and inspired by Spacemacs layer configurations.

In the first section, I explain the structure of the main template and of the layer templates.

In the second section, I demonstrate how to create and maintain projects based on this approach.

Structure

The goal of a layered approach is to start small and extend the code base dynamically as requirements evolve.

The main template for my Python project can be found at this address: https://git.fmind.me/fmind/cookiecutter-python.

This template contains the bare minimum to start a new Python project:

  • {{cookiecutter.name}} will be transformed to an empty module by cookiecutter.
  • .gitignore ignores common files that should be kept outside source control.
  • .python-version sets Python version for the project thanks to pyenv).
  • requirements.txt specifies the project dependencies (excluding the ones required to create the development environment).
  • LICENSE.txt contains the name of the project license (which is enough to reference the license in my opinion).
  • README.md which is used as the long project description by setup.py file.
  • setup.py contains the core project meta data project recommended by PyPA.
    • note: meta data are stored in a dict so they can be read by external processes.
  • Makefile contains 3 extensible tasks described in the next paragraph.

The main template can be extended by other templates that I call layers, they are like any other cookiecutter templates.

But instead of creating a new project, layer templates are rendered as a sub directory inside the main project.

As an example, here is a layer template that includes black , the uncompromising Python code formatter: http://git.fmind.me/fmind/cookiecutter-python-black.

The main project contains two elements that makes the structure extensible:

  • setup.py registers every requirements.txt file contained in the layer directory as extra dependencies.
    • e.g. when the cookiecutter-python-black template is rendered to the formats directory, setup.py dynamically creates an entry in extras_requires called formats that contains black library.
  • Makefile includes every Makefile contained in the layer directories and make them available to the project Makefile
    • e.g. the formattask is defined by cookiecutter-python-black layer and is available from the project Makefile.

In addition, the project Makefile contains 3 generics tasks that can call layer tasks based on their prefix and the name of the layer:

export PYTHONPATH=${PWD}

MKFILES = $(wildcard */Makefile)

.venv:
    python -m venv .venv --clear

init: .venv
    @for MK in ${MKFILES}; do make --no-print-directory -f $$MK init-$$(dirname $$MK); done

clean:
    @for MK in ${MKFILES}; do make --no-print-directory -f $$MK clean-$$(dirname $$MK); done

commit: .venv
    @set -e; \
    for MK in ${MKFILES}; do make --no-print-directory -f $$MK commit-$$(dirname $$MK); done

include */Makefile

Thanks to this approach, I can now add and delete layers without impacting the main project or my other layers.

I can also recreate my layers to push update from my layer templates (e.g. tasks, dependencies, configuration ...).

Example

Let's create a simple Python project to demonstrate how to use the layered approach in practice.

This demo will follow the same steps I used to create onset, a utility script that performs set operations on files.

First, let's create a new project from the main Python template: https://git.fmind.me/fmind/cookiecutter-python.

I use the default templates value from my ~/.cookiecutterrc file to fill the blanks.

project create

Second, let's add a few layers to our main project.

In this example, I'm adding black, pylint and pytest layer to the project.

Note that I use the --no-input to set the default values from the layer template.

After this step, one folder was added to the project for each layer: lints, tests, formats.

porject layers

Third, let's initialize the project and its dependencies.

I use the init task of the main Makefile to call init tasks of each layer: init-lints, init-tests, init-formats.

project init

At any point, I can execute the project clean task to call the clean task of each layer: clean-lints, clean-tests, clean-formats.

project clean

Finally, let's execute the main commit task to check the project state.

If the githooks is installed, the commit task must return 0 to allow git to commit.

In the current setup, the commit task will run pytest, pylint and black on the code base.

project commit

Conclusion

In this article, I presented a layered approach to organise Python project.

I've used this approach in my recent Python project, and I had a great experience so far.

I appreciate the flexibility to add, remove and update layers with a limited impact on my main project.

This approach allowed me to reduce the maintenance time of both my Python project and my cookiecutter templates.

PS: I would like to thank all the developers of cookiecutter and the authors of cookiecutter templates for their contributions.