[Distutils] Third-party packages inside a project

[Giovanni Bajo]

Hello,

for our Windows-only applications, we use a directory setup (working 
copy) like this:

   /lib
     /a
     /b
     /c

   /src
     boot.py
     pkg1/
     pkg2/
     pkg3/


/lib/{a,b,c} are third-party standard distutils packages downloaded from 
Internet. We stuff them within the working copy because we don't want 
any external dependency besides Python itself (basically, if you install 
Python and do a checkout, you must have everything needed to run the 
application).

/src contains our own application source code. boot.py is basically the 
entry point, and the code is contained within several sub-packages.

Third-party packages are installed into /src. Basically, after 
compilation, you end up with something like:

   /src
     boot.py
     a/  [installed]
     b/  [installed]
     c/  [installed]
     pkg1/
     pkg2/
     pkg3/

If you run boot.py from /src, this works fine (and has worked fine for 
years).

In the past few months, we realized that this layout is probably going 
to break in the future (moving to newer python versions and making the 
application cross-platform) because:

   1) It doesnt't work for third-party packages which use setuptools. 
Setuptools always creates and installs a .pth file (to the best of my 
understanding), and .pth files are not process inside /src, because it's 
not part of PYTHONPATH. I can't find a viable fix for this:
     * Creating a sitecustomize.py inside /src (which does something 
like site.addsitedir(os.getcwd()) does not work anymore since Python 
2.5. It looks like it never worked under *NIX, and was "fixed" under 
Windows lately. sitecustomize.py is not imported from the current 
directory, probably for security reason (code injection).
     * Pointing PYTHONPATH to /src is a PITA, since developers always 
move among projects and different branches of the same project (so 
different working copy), and asking them to remember to change 
PYTHONPATH everytime is simply a recipe for disaster.
     * Using a "Custom Installation Locations" as suggested by 
EasyInstall docs is not a solution, because those are "tricks" to 
install python packages into an user-specific directory (instead of the 
system directory). I need a solution for a project-specific directory.

   2) In Python 2.4, we were able to directly run a script in a 
subpackages (eg: a test). Basically, from /src, you could run "python 
pkg1/test_foo.py", and that would work correctly (= absolute imports 
like "import pkg2" were working as expected). In Python 2.5, this does 
not work anymore.

I was wondering if someone could suggest a fix, especially for #1: I 
can't believe there's no way to ask distutils/setuptools to install a 
project-specific package (I am aware of the chapter "Custom Installation 
Locations" in EasyInstall docs, but that's more about installing 
packages at user-level instead of system-level; it doesn't cover the 
project-level which I need).

Sorry for the long mail (I reckon it's better to err on the side of 
verbosity to make sure everything is clear).
-- 
Giovanni Bajo