[Distutils] Fixing parallel installs with easy_install / setuptools

[David Cournapeau]

Garrett Cooper wrote:
> Hello distutils folks,
>     My group at Cisco uses easy_install / setuptools extensively for
> installing packages with our make system and there are 2 issues with
> using this with parallel make:
>     1. Duplicate dependencies cause corruption when installing the
> same package twice.
>     2. easy-install.pth is never updated according to the installed
> packages. Thus when we install 8+ packages and they all depend on
> python being installed
>     3. Cross-compilation is impossible (at least with what I've seen
> -- either that or the folks who hacked the sources before I got to it
> didn't understand that feeding in the correct variables would ensure
> that things could cross-compile). Then again many things Python don't
> cross-compile from what I've seen :(... the interpreter itself is a
> prime example =\.
>     My goal is to fix these issues and contribute back the fixes, but
> I want to ensure that I use the best technical solution for the
> problems I mentioned.
>     About problems 1 + 2:
>         I'm short on time so I'm going to implement a simple locking
> mechanism around easy-install.pth. I also realize that .pth files are
> chosen because they are flat files and are included simply from within
> python with setup.py. Would it make more sense to use a backend
> database like pysqlite to store the package data though? That would
> require some reworking with setup.py, but considering that sqlite3 is
> _already_ included with 2.5+ and it's a public domain licensed piece
> of OSS software, would it make more sense to store packaging data with
> a stable system like SQLite, especially when it would make removal a
> trivial task?

Why would using sqlite make remove a trivial task ? I don't see anything
that sqlite would solve compared to a file for uninstall. I think the
main problem of uninstall is that there are many possible installation
locations, and for one package there may be multiple installations. I am
afraid I don't see a reliable way to do this without python's help (one
'registry' / python installation).

Although note setuptools needs to support python < 2.5 (a lot of
platforms do not have python > 2.4, for example).

>     About problem 3:
>         Is the solution simple enough to solve using --build, --host,
> and/or --target, like configure, or does more gross work need to be
> done under the covers to make things all work?
>   

If you want to do it as quickly as possible, then hacking something in
distutils may be possible, but options handling is a bit of a pain in
distutils (each command is independent and has its own option set).

If you want something reliable, the only way is to bypass the build part
IMHO (the build_* commands). It is not meant as a general thing, but you
may find my project numscons helpful. It adds a scons distutils command,
so that you can build your extensions with scons instead of distutils -
you then have a sane build system. It is far from perfect, but it can
build non trivial codebase on many platforms (windows included), and you
can control flags, change compilers, etc... in scons.

http://github.com/cournape/numscons/tree/master

the code to plug into distutils is ugly and here:

http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/scons.py)

cheers,

David