Organizing a Python project

[A.T.Hofkamp]

On 2008-05-19, ivarnelispam at gmail.com <ivarnelispam at gmail.com> wrote:
> Hello all,
>
> I'm starting work on what is going to become a fairly substantial
> Python project, and I'm trying to find the best way to organize
> everything. The project will consist of:
>
> - A few applications
> - Several small scripts and utilities
> - Unit tests and small interactive test programs
> - A number of custom libraries and modules that may be shared and
> referenced among all of the above
>
> I have the following general structure in mind:
>
> myproject/
>   app1/
>     main.py
>     file1.py
>     file2.py
>     tests/
>       test_abc.py
>       test_xyz.py
>   app2/
>     ...
>   scripts/
>     script1.py
>     script2.py
>   shared/
>     mylib1/
>       file1.py
>       file2.py
>       tests/
>         test_foo.py
>         test_bar.py
>     mylib2/
>       ...
>
>
> The files that you might want to execute directly are:
>   - Any of the "main.py" files under app*/
>   - Any of the files under shared/
>   - Any of the files under app*/tests or shared/mylib*/tests
>
> So, my questions:
> First of all, does this look like a reasonable overall structure, or
> are there better alternatives?

You could make a 'bin' directory next to 'myproject' with executable programs
which would usually do something like

#!/usr/bin/env python
from myproject.app1 import main
main.run()

to make a more clear seperation between code that can be executed and code that
is imported in an application.


Also, why do you make a distinction between shared and non-shared code?
You could simply eliminate 'shared' directory, and put its contents directly
under myproject.

> Second (and the thing I'm primarily interested in), what is the best
> way to deal with importing the shared modules in the applications,
> scripts, test programs, and possibly other shared modules? I think the
> most obvious solution is to add /path/to/myproject to PYTHONPATH.
> However, this seems like an annoying little dependency that you are
> likely to forget whenever you move your workspace to a new path, open
> up a branch of the project in a different directory, or download and
> work on the project using a different computer.

What I am missing here is how you plan to do the development.
If you want to do branch-based development, you may want to have a look at
Combinator (at divmod.org). It handles branch management, adds executable
programs from bin to your path (in your current branch), and extends PYTHONPATH
(with your current branch).

Even if you have just 1 branch (namely 'trunk') it may be useful.


> Is there a way to set this up that is a bit more self contained? For
> example, at first I was somewhat hopeful that Python could ascend
> parent directories until it reached a directory that did not include
> an __init__.py file, and it could use this as a root for referring to
> packages and modules from any file contained within. (e.g. in the
> example project above, any file could refer to myproject.shared.mylib1
> so long as 'myproject' and all subdirectories contained an
> __init__.py, and the parent of 'myproject' didn't contain such a
> file). Evidently this is not the case, but it seems like it could be a
> useful feature in these situations.

Work is being done on relative imports. Not sure of its state.

> Anyway, I'm sure this is not an unusual situation, so I'm curious to
> hear how other people have handled it.

Most people probably run scripts from the root, ie where 'myproject' is a
sub-directory. Since Python automatically adds '.' to its path, it will work.

Sincerely,
Albert