[Distutils] Fixing parallel installs with easy_install / setuptools

[Garrett Cooper]

On Wed, Mar 11, 2009 at 10:36 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Garrett Cooper wrote:
>> Hello distutils folks,
>>     My group at Cisco uses easy_install / setuptools extensively for
>> installing packages with our make system and there are 2 issues with
>> using this with parallel make:
>>     1. Duplicate dependencies cause corruption when installing the
>> same package twice.
>>     2. easy-install.pth is never updated according to the installed
>> packages. Thus when we install 8+ packages and they all depend on
>> python being installed
>>     3. Cross-compilation is impossible (at least with what I've seen
>> -- either that or the folks who hacked the sources before I got to it
>> didn't understand that feeding in the correct variables would ensure
>> that things could cross-compile). Then again many things Python don't
>> cross-compile from what I've seen :(... the interpreter itself is a
>> prime example =\.
>>     My goal is to fix these issues and contribute back the fixes, but
>> I want to ensure that I use the best technical solution for the
>> problems I mentioned.
>>     About problems 1 + 2:
>>         I'm short on time so I'm going to implement a simple locking
>> mechanism around easy-install.pth. I also realize that .pth files are
>> chosen because they are flat files and are included simply from within
>> python with setup.py. Would it make more sense to use a backend
>> database like pysqlite to store the package data though? That would
>> require some reworking with setup.py, but considering that sqlite3 is
>> _already_ included with 2.5+ and it's a public domain licensed piece
>> of OSS software, would it make more sense to store packaging data with
>> a stable system like SQLite, especially when it would make removal a
>> trivial task?
>
> Why would using sqlite make remove a trivial task ? I don't see anything
> that sqlite would solve compared to a file for uninstall. I think the
> main problem of uninstall is that there are many possible installation
> locations, and for one package there may be multiple installations. I am
> afraid I don't see a reliable way to do this without python's help (one
> 'registry' / python installation).
>
> Although note setuptools needs to support python < 2.5 (a lot of
> platforms do not have python > 2.4, for example).
>
>>     About problem 3:
>>         Is the solution simple enough to solve using --build, --host,
>> and/or --target, like configure, or does more gross work need to be
>> done under the covers to make things all work?
>>
>
> If you want to do it as quickly as possible, then hacking something in
> distutils may be possible, but options handling is a bit of a pain in
> distutils (each command is independent and has its own option set).
>
> If you want something reliable, the only way is to bypass the build part
> IMHO (the build_* commands). It is not meant as a general thing, but you
> may find my project numscons helpful. It adds a scons distutils command,
> so that you can build your extensions with scons instead of distutils -
> you then have a sane build system. It is far from perfect, but it can
> build non trivial codebase on many platforms (windows included), and you
> can control flags, change compilers, etc... in scons.
>
> http://github.com/cournape/numscons/tree/master
>
> the code to plug into distutils is ugly and here:
>
> http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/scons.py)

Thanks for the pointers. The only thing I'm concerned about is while
scons and waf are both Python based, they're still not accepted
packages in Python proper. That being said, neither is setuptools.

Has there been talk of a standard python packaging tool being
integrated into the interpreter suite?

Also, has any serious thought been put into maybe taking the package
name, producing specific mnemonic based .pth files for the particular
package, and just installing this way, e.g.:

pexpect -> pexpect.pth
nose -> nose.pth

etc. I have seen some packages do this and maybe this is the quicker /
dirtier route to do this, but it's also the simplest route to go
whilst avoiding collisions with packages, from my point of view, and
it's not incredibly complex at all. Furthermore, it kind of lends
itself to other packaging methods like pkg_install (FreeBSD),
pkgconfig, etc etc. This would be good especially because easy_install
doesn't allow multiple versions by default... The only real loss is
that the interpreter would have to open up a number of .pth files
which would potentially slow down the machine because of I/O access,
but the number of python packages on a heavily populated system should
be under 20~50 I'd think, so the number seems negligible

Any more thoughts?
-Garrett