About Python project (directory) structures

I have previously asked some questions about Python Project (directory) Structures.

for example this question here, it is not important to read this I simply provide it for comparison to the suggestion presented here

I want to ask again about this as I have a bit more experience building larger projects purely with Python. I know that some are of the opinion that questions about Python Project Structures are opinion based.

I don’t think this is quite the right way to view things. While it is certainly true that there are several ways of creating a project directory structure which work well, I am converging towards the opinion that there is probably only one Pythonic way which uses Python modules and packages in the way which was intended.

In the linked example above, I previous discussed using a pyproject.toml in combination with a src directory, which uses pip and venv to make everything work together correctly.

It leverages the fact that pip can be used to install local packages in editable mode.

I have come to the conclusion that this is too complicated. It has too many moving parts, and I think there is a better way. However, I don’t yet have all the answers, and that is what this question is about. In particular, I don’t like that a src directory has to be manually specified in a pyproject.toml file and that this file has to be created as an additional dependency, I don’t like that a special pip3 install -e . command has to be run to “glue” everything together, and I don’t like that pip3 freeze > requirements.txt dumps out a spurious line (which in this particular case) contains a git ssh address.

A Simplified Approach

Let’s start with this fundamental “problem”, and turn the “problem” around so that it becomes a solution.

Firstly: Let us forget everything we know about other languages where we would usually build libraries seperatly to executables and we would usually put library code into their own directories. Rather than trying to put a square peg in a round hole by making our Python project structure look similar to what we are used to, let’s redesign assuming we know nothing about other languages.

Problem: The Python runtime can only find packages in the same directory as the file which is being executed.

Rather than thinking of this as a problem (I view this as a problem) let’s see if it can become a solution.

Here is a simple example. Here we have a single “executable” and a single library, which contains all the code which has essentially been “deferred” from the main.py executable. This might be because we expect it to be shared code, or simply because main.py has become very long and we want to put some of the code elsewhere.

My-Python-Project
    main.py
    lib/
        __init__.py
        my_lib.py

This works, but we appear to run into an issue when the number of executables grows.

We can test it with some very simple code.

# my_lib.py

def my_function():
    print(f'hello world')

# main.py

from lib.my_lib import my_function

my_function()

If you move main.py to a new sub-directory you will see the Python interpreter can no longer find lib.

Normally we would want to do two things:

Group related executables together into a single directory
Split libraries such that related code is all contained together within a single library. We would usually expect groups of related executables to use one or more libraries. (This is just a statement about what libraies are.)

With Python, I don’t think it is possible to do (1). It is possible to do (2). This can be seen in the following example.

My question is whether or not (1) can be achieved.

# Design 1
My-Python-Project
    main1.py
    main2.py
    ...
    mainN.py
    lib/
        __init__.py # we must have a single Python Package `lib`
        my_lib_1/
            file1.py
            file2.py
            ...
        my_lib_2/
            file1.py
            file2.py
            ...
        ...
        my_lib_N/ # not necessarily 1:1 with the number of `main_.py` files
                  # structure libraries as necessary

Points to note:

There is a single library package but multiple library modules
Libraries are contained within independent modules, but the same package
You could put each library into its own package but then the root directory becomes excessively cluttered. If you do this, you will revert back to the undesirable situation whereby multiple unrelated binaries and multiple unrelated libraries are just dumped in the same directory with no struture or obvious way to comprehend the design.
The below shows an example of this final point to make it clear

# Don't build this structure,
# because the root directory becomes cluttered
# and it is hard to understand the design
My-Python-Project
    main1.py
    main2.py
    ...
    mainN.py
    lib_1/
        __init__.py
        ...
    lib_2/
        ...
    ...
    lib_N/
        ...

The design presented in Design 1 is not totally perfect. The most significant flaw is that it is not possible to create sub-directories to group together related executables. Here is an example directory structure which it is not possible to build.

# Cannot build this structure!
My-Python-Project
    executable_group_1/ # these executables do not know what `lib` is
        main1.py
        main2.py
        local_library.py
    lib/
        __init__.py
        my_lib_1/
            ...
        my_lib_2/
            ...
        ...
        my_lib_N/
            ...

Using Design 1 it is not possible to create a “local” library which lives in the same sub-directory as an executable or group of related executables (as seen in the example directly above). This necessitates careful naming of the libraries. To explain this:

In general there are two kinds of library which we might want to build
One which is tightly related with a particular executable. We are essentially deferring code from the executable file into an associated library which no other executable uses.
One which is more general and is expected to be used by multiple executables.
To make it obvious that one module (library) is closely related to a particular executable we would have to be careful with the naming conventions.

Questions

How can we implement (1)? (Grouping together related executables.) Is it possible, or are we forced to use something like pyproject.toml + setuptools?
Are there any problems with this design beyond the fact that if there are many executables, the root directory will become quite cluttered. This might be an acceptable tradeoff for the simplicity.
More generally, I would like to hear some feedback in the comments about this project structure design

Filed under: Kiến thức lập trình - @ 18:21

Thẻ: python

Thiết kế website giá rẻ

Danh mục

About Python project (directory) structures

A Simplified Approach

Questions