I have a Python project that looks something like this.

project/
    tests/
        __init__.py
        tests.py
    main_module/
        __init__.py
        sub_modules/
        ...
    task1.py
    task2.py
    ...
    outputs/
    data/

For context, main_module implements a whole bunch of numerical computing code and simulations and the files task1.py, task2.py are scripts that glue together things from main_module and data then store output (figures, tables, etc.) in outputs. These are distinct from things in tests and could maybe be described as user code.

Each task file can take quite a long time to run, so it is quite convenient to develop them interactively and plot intermediate results and so on to check what is going on — this cannot realistically be simply placed into tests/.

If I put each task file into a subdirectory main_tasks/ or something, then I can’t import from main_module unless I make project/ itself a module. However, this seems to make interactivity very difficult since you get ImportErrors on any relative imports. So, what is the right way to organize these top level task scripts into a subfolder so that they aren’t polluting project/? Is it proper to just do something like:

sys.path.insert(0, "/path/to/main_module/")

? In a tasks subfolder? Should I install main_module in site-packages? Should I develop interactively in the top level, then move the tasks into a bona fide module after they’re working?

For context I’ve looked at https://docs.python-guide.org/writing/structure/ and some other project organization questions on SO but didn’t feel my issue was adequately addressed.