[Python-Dev] setuptools in 2.5. (summary)

[M.-A. Lemburg]

Anthony Baxter wrote:
> In an attempt to help this thread reach some sort of resolution, 
> here's a collection of arguments against and in favour of setuptools 
> in 2.5. My conclusions are at the end.

Thanks for the summary. I'd like to add some important aspects
(for me at least) that are missing:

- setuptools should not change the standard distutils "install"
  command to install everything as eggs

  Eggs are just one distribution format out of many. They do
  server their purpose, just like RPMs, DEBs or Windows installers do.

  However, when running "python setup.py install" you are in fact
  installing from source, so there's no need to wrap things up
  again.

  The distutils default of actually installing things in the
  standard Python is good, has worked for years and should continue
  to do so.

  The extra information needed by the dependency checking can
  easily be added to the package directory of the installed package
  or stored elsewhere in a repository of installed packages or as
  separate egg-info directory if we want to stick with setuptools'
  way of using the path name for getting meta-information on a
  package.

  Placing the egg-files into the system as ZIP files should
  be an option, e.g. as separate "install_egg" command,
  not the default.

- setuptools includes many good additions to distutils

  We should remember that setuptools will get part of the stdlib,
  so instead of monkey-patching distutils, a lot of setuptools
  features can be ported to distutils directly.

  For things that don't fit distutils, proper hooks should be
  added, so that extending distutils in the setuptools way can
  be done by the standard distutils extension mechanisms.

  I think that if Phillip were more open to reconsidering design
  decisions in setuptools like the decision to replace the
  install command instead of just adding a new install_egg
  command (together with a bdist_egg), quite a lot of code
  could be directly imported into core distutils.

  Even for things like his alternative sdist implementation
  could go in, albeit using a different command name.

- arguments like "just works" simply don't count

  Why not ? Because "just works" is not well-defined. If an
  implementation makes too many magical decisions for the
  user, you end up with a tunnel - with only one way in and
  one way out.

  In reality there are too many things you'd need magical
  decisions for. A generic distribution mechanism will
  never be capable of making all the right decisions for
  a user. Adding endless lists of options doesn't make things
  easier either.

  Let's face it: there isn't the one right way to do software
  distribution.

  What's good about it: there doesn't need to be.

  The package authors will know which particular tweaks
  are needed to make installing their package a breeze
  and easy for the user. So the right way to get close
  the "just works" idea, is to leave a door open for
  module authors and packages to add such functionality.

  Freedom of choice is often enough a lot better than
  any variant of "just works". Explicit is better than
  implicit. Good defaults help.

- setuptools is not ready for prime-time

  This is not quite right. setuptools works and has been
  used by quite a few people in the past.

  However, just dumping setuptools into the stdlib is
  considered a less than ideal approach. Instead, setuptools
  should get *integrated* into the stdlib.

  This process will take longer than the current 2.5 release
  cycle. We should at least mark setuptools as it is
  now in the stdlib as experimental and leave open
  the possibility of making changes to it.

> The arguments against:
>
>  - Someone should instead just fix distutils. 
> 
>    Right. And the amount of yelling if distutils changed in a non-b/w 
> compat way wouldn't be high. There's also the basic problem that the 
> distutils code is _horrible_. 
> 
>  - It monkeypatches pydoc and distutils!
> 
>    It only monkeypatches pydoc when the separate setuptools installer 
> is used on older Pythons. How is this relevant for this discussion of 
> Python 2.5? The monkeypatching for distutils should be reduced - see 
> AMK's message for a breakdown of this.
>
>  - Documentation
> 
>    beaker% pydoc xmlcore.etree
>    no Python documentation found for 'xmlcore.etree'
> 
>    beaker% pydoc ctypes
>    no Python documentation found for 'ctypes'
> 
>    The documentation (of which there is plenty) can and will be folded 
> into the standard python docs. Most of the new modules in 2.5 went in 
> before their docs. 
> 
>  - Where's the PEP?
> 
>    I don't see the need. The stuff that could go into a PEP about 
> formats and the like should go into the existing Distutils 
> documentation. It's a far more useful place, and many more people are 
> likely to find it there than in a PEP.

The PEP is needed to document the current discussion, not
the setuptools technical documentation (this should of course
go into the standard Python documentation).

Your summary along with the opinions of the ones who have
commented it would be a good start. The PEP should also include
the lists of things Martin and Fredrik have put forward.

Finally, the goal of the PEP should be to list the steps needed
to *integrate* setuptools into the stdlib.

>  - It's a huge amount of code (or "ball of mud"), or, it adds too many 
> features.
> 
>    Most of these have been added over the last 2 years in response to 
> feedback and requests from people on distutils-sig. There's been an 
> obvious pent-up demand for a bunch of this work, and now that 
> someone's working on it, these can get done.
> 
>  - It will break existing setup.py scripts!
> 
>    No it won't. If you don't type the letters 'import setuptools' into 
> your setup.py, it won't be affected. 
>
>  - Rewriting from scratch is bad
> 
>    This isn't a rewrite - it's built on top of distutils. 
> (An aside, I don't buy the "never rewrite" argument. As I mentioned in 
> an earlier message, look at urllib2, twisted and email for starters. 
> In addition, look at Firefox, Windows XP, and Mac OSX. Hell, Linux 
> could be considered a rewrite of Minix, once upon a time.)
> 
>  - Eggs are inferior to distribution-specific packaging
> 
>    Not all operating systems have a decent packaging system. The ones 
> that do, don't support multiple versions of the same library. In 
> addition, there's no reason why existing packaging systems can't just 
> bundle up the code as they do now - if they also add a .egg-info file 
> to the packages, that would be even better! Finally, these don't 
> support user installation of software. This is particularly useful in 
> a hosting environment. 

I think you're misrepresenting the argument here: eggs are not
considered inferior. What packagers and application builders
gripe about is the fact that setuptools forces them to use eggs
instead of the standard Python way of installing packages.

I don't think anyone would be bothered by adding dependency
information to their packages and thus make easy_install and
friends work with regular Python packages as well.

> And now let's look at some of the stuff that setuptools gives us:
> 
>  - We have a CPAN-type system
> 
>    I do quite a number of Python talks, and this is _always_ one of 
> the most requested features. There's been many attempts to write this, 
> none have been completed until now. If you honestly don't see that 
> this is a big thing for Python, then I am very, very suprised. I 
> suspect that this will be the #1 new feature of Python 2.5 that the 
> users will notice and be happy about.
> 
>  - Multiple installs of different versions of the same package, 
> including per-user installs. 
> 
>    Again, as Python gets more widely used, this becomes a big issue.
> Sure, it's not necessarily a killer argument for python-dev, but stuff 
> that's added to Python shouldn't just be just for the use of 
> python-dev. The multiple installed versions feature also avoids the 
> CPAN dependency hell problem - back when I used to work with Perl, 
> this was a constant source of nightmarish problems. 

I've never had a need for having multiple versions of a single
package installed, so can't comment.

User installation is certainly possible with stock distutils.

The main problem with distutils is the lack of user
documentation, not so much the lack of features.

>  - The "develop" mode
> 
>    This makes life that bit less painful all-round. 

Does it ? I usually point PYTHONPATH at my development tree
and that's all it takes to do setup "develop" mode.

>  - The plugin/extension support
> 
>    Extending distutils currently is a total pain in the arse. 

Views differ on this one and most of this perception is due
to missing distutils documentation.

>  - Backwards compatibility
> 
>    easy_install even works with existing packages that use traditional 
> distutils, so long as they're in the Cheeseshop. Damn, this is nice. 
> If you don't want to do the work to change your installation code, 
> don't bother - it will still be useful. 
> 
> The conclusions:
> 
> I'm a little suprised by the amount of fear and loathing this has 
> generated. To me, there are such obvious benefits that I don't see 
> why people are so vehemently against setuptools. I haven't seen any 
> arguments that have convinced me that this isn't the right thing to 
> do. Yes, there's still work to be done - but hell, we've only 
> released the first alpha so far.
> 
> For inclusion in the standard library, the usual benchmark is that the 
> code offers useful functionality, and that it be the "best of breed". 
> setuptools clearly meets these two criteria. (Ok, it's really "only of 
> breed", but that also makes it "best", by default <wink>). It's also 
> been under development for over 2 years - according to svn, 0.0.1 was 
> checked into svn back in March 2004.
> 
> I'm also suprised by how much some people seem to think that the 
> current state of distutils functionality is acceptable or desirable. 
> It's not - it's a mess. 
> 
> Finally, I'd like to point out that I think some of the hostility 
> towards Phillip's work has been excessive. He's done an amazing 
> amount of work on this (look at the distutils-sig archive for the 
> last two years for more), and produced something that's very very 
> useful. 
> 
> He deserves far more credit for this than he seems to have been 
> getting here.

He does and there's no doubt about it.

It would help a lot, though, if Phillip would be willing to
reconsider some defaults he's using in setuptools to make them
compatible to Python's default import mechanism.

A lot of the perceived hostility would go away if he'd play
nice with standards that have existed in Python for years.

If Phillip thinks that we should change Python to always
import packages from egg-files, then this would be a major
change in the way Python works and thus require a complete
PEP process.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 20 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::