[Python-3000] Proposal: No more standard library additions

[Phillip J. Eby]

At 01:41 PM 10/15/2006 +0200, Martin v. Löwis wrote:
>Phillip J. Eby schrieb:
> > The difference between the two is that an extensible system is one that
> > follows the same rules for extenders as it does for its core
> > developers.  For example, in the distutils, there are two ways to
> > register commands: one for extenders, and one for core developers.
> > (Actually, there are three, as there is yet another way added in Python
> > 2.4 for third parties to add commands as well.)
>
>It's only a terminology issue, but I think most people would assume
>a different definition for "extensible". I'd call a system extensible
>if it supports being extended, period. By my definition, distutils
>is extensible; by your definition, it's not.

Yes, and the people who complain about the distutils' design are using my 
definition or something like it.  Even by your definition it's sometimes 
very difficult to extend, because there is a tendency in the distutils to 
have page-long methods that encode an entire sequence of operations that 
cannot be customized without copying and modifying the whole block of 
code.  Command methods also often have side effects, which is something 
that the distutils encourages by its architecture.

(The architecture is also poor from a testability perspective, which limits 
our ability to simply refactor and improve the architecture.)

Last, but not least, the distutils suffer from an excess of policy 
flexibility in areas where flexibility is undesirable from a community 
perspective.  There should be "one obvious way" to do a lot of things that 
the distutils provide a ridiculous number of ways to do (e.g. insane 
project directory layouts including package-by-package directory 
mapping).  This is not something that affects the distutils themselves 
much, but it essentially prevents the distutils from improving in certain 
ways because there are so few assumptions one can make about its use.


>Actually, it is possible to add a new compiler to some Python
>installation, so that users can pass it to the --compiler option,
>and distutils packages don't have to be modified for that.
>You just have to extend the dictionary
>distutils.ccompiler.compiler_class.

Yes, and how do you accomplish *that*?  You have to either modify some 
Python code (e.g. sitecustomize) or use a big -c to python when running the 
setup script.

With entry point extension, it is only necessary to have an extension 
installed on sys.path; it is not necessary to run some code to do the 
registration.

In the setuptools case, if you want to add setup commands to your Python 
installation, you simply install distributions containing those commands, 
and they become available automatically for any setup script using 
setuptools.  If you want to add setup() arguments, same thing.  Revision 
control plugins, likewise.  Egg metadata file generator plugins 
also.  Essentially, anything that someone can think of a use case to allow 
plugins for, it is a simple change to add.


>I don't object to improving the design. I object to dropping distutils
>and replacing it with something else. If certain functionality is
>missing in distutils (e.g. means for more transparent extension), then
>distutils should be enhanced to provide this functionality.

Adding setuptools-style plugin ability to the distutils is very simple, and 
doesn't require a wholesale reorganization.  After all, setuptools manages 
to do it!  However, there are areas of build functionality that the 
distutils does not cover or provides incomplete coverage for, such as 
configuration and build of more complex C libraries and programs, shared 
libraries, documentation production, and so on, that the existing 
architecture of the distutils doesn't really encompass.

The current design of the distutils has essentially two layers: one very 
high-level layer dealing with command objects, and a very low-level layer 
that deals with copying files or trees thereof, making archives, and 
various other simple tasks.  The low-level layer is often infested with 
assumptions, however, that are propagated from higher layers.  Meanwhile, 
the higher-level layer is based on the idea of dependency between 
sequentially invoked commands, rather than dependency between things being 
built.  This makes it difficult to deal with concepts like having a bunch 
of reST release note docs combined with a LaTeX documentation build.

So, distutils' extensibility *could* be improved dramatically in the 2.x 
line by adding setuptools-style plugin entry points.  However, adding any 
*new* build functionality to the distutils is probably going to be better 
off being based on something that offers an extensible object-dependency 
layer, ala zc.buildout.

Could that be done while leaving the main distutils structure largely 
intact?  Perhaps.  I'm just not convinced that that's what should happen 
for Py3K.  There's a ton of other obsolete or nearly-so stuff happening in 
the existing distutils, like the specialized option parser, crazy bytecode 
compiling hacks, and so on.

(Which reminds me.  Does anybody have any idea why the distutils compile 
modules in build/?  There seems to be no point to this, since any 
installation of the modules is going to need to recompile them.  Even bdist 
operations end up recompiling them in a different location!)