[Distutils] How can I create prebuilt distributions?

[David Ascher]

Jack Jansen wrote:

> I'm interested. I'm working on a thing called the Package Install 
> Manager for Python (pimp) 

I'm going to say this just once.  Please change the name.  It's funny, but 
it's not 'serious'.  I told the Piddle folks the same thing, and they 
regretted not changing it after it was too late.  There, it's off my 
conscience =).

> right now. I was planning not to go into 
> design discussions until 2.3 is out and only at that time write up a PEP 
> or something, but as pimp is 90% done I guess I can spare the time. 

Funny -- not going into design discussions until after you're done =).

> But Mac users often don't have a development 
> environment, and even if they do having them type in distutils commands 
> isn't exactly considered good style. 

Absolutely.  Same problem w/ Windows

> So the idea behind pimp is that there is an online database of packages 
> that is specific to a (OS-version, python-version) combination and that 
> has a maintainer who is responsible for not adding only packages that 
> have been tested and tried. Preferably the database should have source 
> and binary version of all packages, so the end user can state a 
> preference for either. The database also has dependency information, and 
> bits of code so pimp can actively test whether a specific package is 
> installed into your python, etc etc. 

This is exactly what PPM was designed for, and it works (95% of the time) for 
Perl.  See e.g. http://aspn.activestate.com/ASPN/Downloads/ActivePerl/PPM/Packages

The way it works as a user is that if you have ActivePerl installed you can type

'ppm install Yada-Yada-Yada' and it will work (on Solaris only, according to 
the table above =).

We update that from CPAN regularly (now), using an automated build system.  We 
have two people dedicated to that at this time.  This is a non-trivial 
resource commitment, both in hardware and in personel.

The idea behind PyPPM is that we would do the same for Python.  The problems 
came about because, as everyone here known,

	- there is no CPAN (with the actual sources, not just a catalog)

	- for a long time, there was no distutils

	- even with distutils, the process is much harder than in Perl.
           I'm not 100% sure I understand why.  One hunch I have is that
           95% of CPAN is "strictly" modules.  No front-end tools, no
           Start Menu configurations, very little dependencies on third-party
           packages, etc.  (also, 90% of CPAN is junk IMO, but that's another
           problem ;-).

Regardless of what else you do, I suggest you consider the The Open Software 
Description Format: http://www.w3.org/TR/NOTE-OSD

PPD (the format used in PPM) is a slight derivative of that.

> I hope that people will appear who want to maintain the database, and 
> even if I have to do it myself it's still less of a bottleneck if I 
> don't have to do it at the same time of making sure everything in a new 
> Python release works.

That job is what ActiveState does for Perl for three platforms right now. 
Note that there's more to do than maintaining the database - there's building 
the packages and testing them (and in our case, serving them).

> If people want to have a look at pimp: it's in Lib/plat-mac/pimp.py. 
> It's probably MacOSX-specific right now, but that shouldn't be too 
> difficult to fix.

Famous last words =).

Seriously, though -- cross-platform setup.py's are hard to come by as soon as 
you deviate from pure Python modules.  They tend to fail if e.g. the versions 
of shared libraries installed are of different version.  Some setup.py's 
create scripts -- are those installed in Tools, in the base Python directory, 
where?  Are their shebang lines tweaked on install?  Do they even get 
mentioned in the manifest and put in the distribution directories?  Etc.

I'm not trying to criticize the authors of setup.py's, btw!  I just know from 
painful experience that building cross-platform build systems takes a _huge_ 
amount of time.  Unfortunately, the highest value Python packages tend to rely 
on extension modules (one exception to that general statement is platform.py, 
one of my favorite pure-python modules), and so it's the edge cases that cause 
the greatest grief on the user side.  For example, PIL is notoriously hard to 
build -- it's not because Fredrik is incompetent or mean -- it's because the 
number of combintions that have to be dealt with is large, and it's very hard 
to know when you're "done" except through slow, distributed, painful testing 
in the field.

Don't get me wrong -- I think it's great that you're doing this, but I want to 
suggest that you're not 90% done =).

--david