[Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns

[Jeff Rush]

Phillip J. Eby wrote:
> At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote:
>> Are you open to giving certain others patch view/commit privileges to 
>> setuptools?
> 
> Jim Fulton has such already.  I'm open to extending that to others who 
> have a good grasp of the subtleties involved.
> 
> Truthfully, if we can just get 0.6 put to bed, I could probably open up 
> the trunk a lot wider.

What is needed to put 0.6 to bed?  How can we help accelerate this?


> Probably the most frustrating thing (or "chief amongst the most 
> frustrating things") about setuptools development is that it's a black 
> hole.  By which I mean that backward compatibility and cruft accretion 
> make it difficult to get out of.
> 
> In the beginning, there was the distutils.  Distutils begat setuptools, 
> and setuptools begat virtualenv and zc.buildout and source control 
> plugins.  Etc., etc.

I've found in the past a revisiting of basic principles and objectives, 
communicated in enhanced documentation, can help to clear out such black 
holes. ;-)  I'm pulling something together, from the recent emails and some 
archived threads -- it definitely is tangled though, I'll agree.


> What I think is really needed in the long run is to keep eggs, but get 
> rid of setuptools and the distutils in their current form.  There's a 
> lot of brokenness there, and also a lot of accumulated cruft.  We really 
> need a distutils 3000, and it needs to be built on a better approach.

That will require a lot of concensus building as well as collection of use 
cases so that the architecture team can encompass aspects they are not 
personally aware of.  As you've said, it's hard to address itches that are not 
your own.

It certainly is possible for someone to create a parallel packaging moduleset 
that uses the existing eggs format and PyPI but without the currently 
codebase, and then, once proven to work, lobby for it as distutils 3000.

Frankly I'd like to see setuptools exploded, with those parts of general use 
folded back into the standard library, the creation of a set of 
non-implementation-specific documents of the distribution formats and 
behavior, leaving a small core of one implementation of how to do it and the 
door open for others to compete with their own implementation.


> In truth, my *real* motivation for PEP 365's bootstrap tool isn't so 
> much to support the package management tools we have today, as it is to 
> support a new one tomorrow.  I have a few ideas for ways to shift the 
> paradigm of how individual projects get built, to incorporate many 
> scenarios that don't work well now.  But to implement those things in 
> such a next-generation tool, I will not want to be restricted to just 
> what's in the stdlib or what can be bundled in the tool.

You should document those ideas someplace and start getting community input. 
There are a lot of diverse opinions on the right way to do this and the way 
ahead is quite unclear.


> And I think it's probably getting close to time I stepped down from 
> day-to-day management of the codebase (which is more like month-to-month 
> or quarter-to-quarter for me lately).  It will probably be a lot easier 
> for me to step back and critique stuff that goes in, after the fact, 
> than to go over the stuff beforehand.  :)
> 
> I'm not sure exactly how to go about such a handoff though.  My guess is 
> that we need a bug/patch tracker, and a few people to review, test, and 
> apply.  Maybe a transitional period during which I just say yea or nay 
> and let others do the test and apply, before opening it up entirely.  
> That way, we can perhaps solidify a few principles that I'd like to have 
> stay in place.  (Like no arbitrary post-install code hooks.)

I'll see about a tracker and identify some people to help out.


> btw, offtopic question: are you by any chance the same Jeff Rush who 
> invented EchoMail?

Yep, that's me.  Not many remember the Fidonet days.  I designed EchoMail on a 
napkin during a DFW Sysop pizza party during a conversation on what to do with 
the unused capability of inter-BBS private file transfers.  It too escaped its 
ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-)

-Jeff