[Distutils] Python people want CPAN and how the latter came about

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Dec 24 04:11:55 CET 2009

On 23 Dec, 10:51 pm, sridharr at activestate.com wrote:
>On 12/23/2009 2:32 PM, Tarek Ziadé wrote:
>>On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz
>><glyph at twistedmatrix.com>  wrote:
>>> >
>>> >  On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote:
>>> >
>>> >  1/ Missing packages (eg: Twisted is not there)
>>> >
>>> >  The Twisted guys do not upload their packages to PyPI. I think 
>>> >  a mistake, but it's hardly PyPI's fault. There is no law saying 
>>> >  have to use CPAN either.
>>> >
>>> >  For what it's worth, we don't upload because it's a big pain, and 
>>> >  cares anyway.
>>> >  It's a big pain because there are two ways to upload, and neither 
>>>one works
>>> >  for us.  We can't use 'setup.py upload' because we don't use 
>>>'sdist' to
>>> >  produce our tarball releases (although a discussion of why 'sdist' 
>>> >  insufficient is a topic for another post).  The other way to 
>>> >  manually interacting with a form in a web browser, is annoying and 
>>>as far as
>>> >  I know it is hostile to automation.
>>Note that it wouldn't take long to override the upload command so it 
>>independantly from any other *dist command.
>>I could even add a --dist-file option to it so you can point an
>>existing archive to push at PyPI,
>>so running sdist or another *dist command wouldn't be mandatory 
>The advantage of sdist is that the metadata (PKG-INFO; .egg-info/*.txt) 
>is automatically included in the source distribution. For a manually 
>generated source distribution, this is not always the case. 
>Consequently, one is forced to run the `egg_info` command ..something 
>that is unacceptable if you do not want to run Python code in a server 
>that simply mirrors PyPI.
>It would be much interesting to hear arguments for why `sdist` is not 
>suited for Twisted releases and see if it can be fixed.

Release packages for Twisted are constructed using some extra file- 
finding logic that sdist doesn't provide.  Additionally, for years 
distutils was seen as a blind alley, so we didn't bother to try to 
create a distutils-friendly substitute.  Partially because it seems that 
distutils has turned a corner over the last year, we are actually 
(slowly, with difficulty) working towards a more distutils-integrated 
solution (we're going to try to override sdist with a command based on 
our existing custom file-finding code).  This may result in something we 
can use with "setup.py upload" (but this isn't currently the primary 
goal, it would just be a happy side effect).

A separate issue with "setup.py upload", though, is that it really wants 
one of two undesirable things:

  * the upload is done at the same time as the release package is 
  * the release package is generated twice

The former requires that proper credentials are available to whoever is 
creating the release package.  Historically for Twisted, this isn't how 
things have been set up.  We could probably deal with it, but it would 
be nice if it were not a requirement.

The latter, of course, introduces the possibility of skew between two 
release packages, doubling testing requirements, etc.


More information about the Distutils-SIG mailing list