[Distutils] How can I create prebuilt distributions?

[David Ascher]

M.-A. Lemburg wrote:

> Ah, not being mean... it's just that I have asked so many
> times to make the code public and nothing ever happened.

The client code ships with ActivePython.  Feel free to look at it, and then 
tell me if you really want it =).  On the PPD-generation front, we basically 
define a new distutils target, and then do "python setup.py bdist_ppm" and 
then package the setup.py and the distributions subdirectories.  The details 
of bdist_ppm change periodically, and I know I've given that code to some 
people, I don't remember if I sent it to you.  Regardless, it's not a solid 
foundation for the future.

The fundamental problem is that people write setup.py's which aren't tested in 
the case where the "install" phase is done on a different machine than the 
build.  So the build phase may do lots of machine-specific checking and then 
the install phase breaks if e.g. Python is installed in a different directory, 
or it's built on win2k and installed on win9x, etc.  The way people manage 
'supplementary' files varies also a great deal between packages.  It's also 
very hard to do packages which install different things depending on what is 
available on the target machine.

I think that the "right" answer involves a _lot_ of work specifying and 
validating the definitions for the targets, _if that approach is to be used_. 
  I'm not 100% sure that that approach is the right one.  The bdist_wininst 
approach seems to work much better in practice, although I'm not sure that it 
covers all the cases that I wanted to cover (things getting installed in the 
Tools directory with proper shebang line tweaking, etc.).

> There has been some work put into a package directory recently
> (see http://www.python.org/peps/pep-0301.html) and registering
> .ppm like files in such a directory would sure make the Python
> experience an even better one :-)

Yes, I know about PyPI, and I'm definitely hoping that we'll be able to help 
provide PPM-style functionality for everything registered in PyPI.

> The system is missing the "get the stuff and install it" part
> (which is good, since its better to do this in small steps
> rather than coming up with big ideas and then losing interest).

One problem with how PyPPM has "happened" in the past has been that the people 
doing it at ActiveState either weren't knowledgeable enough about Python or 
distutils, or didn't have the bandwidth needed to effect the required changes 
in distutils or in people's setup.py's.  (I still think that shipping 
setup.pys as opposed to using a declarative syntax was a mistake, but that's 
water under the bridge, although maybe we could build a dam).

Distutils is a pretty scary beast, and build and installation is a nasty 
domain in general because there are so many different variations.  If I 
estimated how much time it would take to do PyPPM right from scratch, it would 
probably amount to six man-months of work of my best build engineer.  That's 
something that I need to justify with a business case, something that's harder 
than some people expect. =)  In the Perl world, PPM is a no-brainer for us to 
fund, because PPM is only practically available through ActivePerl, and we get 
a lot of business benefits by providing ActivePerl for free.  In the Python 
world, everytime we talk about doing PPM-style things with ActivePython, key 
people say "I don't care about ActivePython, make it open source so that we 
can add it to the core"  (note that ActivePython users don't say that =). 
While I completely understand the reaction, I'm sure you can understand that 
it makes the business case quite tricky.

The good news is that we're reviewing our PPM strategy, and that we may be 
able to approach the problem in more effective ways than we have in the past. 
  I think that PyPI will help as well because it provides a centralized point 
by which we can e.g. contact all of the maintainers.  We could also presumably 
add functionality to the PyPI server that validates whether one can make a PPD 
out of the package, etc.

I'm willing to spend some time talking about this, just not able to commit to 
spending significant development time or promise deliverables.

--david