[Python-3000] Proposal: No more standard library additions
Phillip J. Eby
pje at telecommunity.com
Sun Oct 15 18:01:30 CEST 2006
At 01:41 PM 10/15/2006 +0200, Martin v. Löwis wrote:
>Phillip J. Eby schrieb:
> > The difference between the two is that an extensible system is one that
> > follows the same rules for extenders as it does for its core
> > developers. For example, in the distutils, there are two ways to
> > register commands: one for extenders, and one for core developers.
> > (Actually, there are three, as there is yet another way added in Python
> > 2.4 for third parties to add commands as well.)
>It's only a terminology issue, but I think most people would assume
>a different definition for "extensible". I'd call a system extensible
>if it supports being extended, period. By my definition, distutils
>is extensible; by your definition, it's not.
Yes, and the people who complain about the distutils' design are using my
definition or something like it. Even by your definition it's sometimes
very difficult to extend, because there is a tendency in the distutils to
have page-long methods that encode an entire sequence of operations that
cannot be customized without copying and modifying the whole block of
code. Command methods also often have side effects, which is something
that the distutils encourages by its architecture.
(The architecture is also poor from a testability perspective, which limits
our ability to simply refactor and improve the architecture.)
Last, but not least, the distutils suffer from an excess of policy
flexibility in areas where flexibility is undesirable from a community
perspective. There should be "one obvious way" to do a lot of things that
the distutils provide a ridiculous number of ways to do (e.g. insane
project directory layouts including package-by-package directory
mapping). This is not something that affects the distutils themselves
much, but it essentially prevents the distutils from improving in certain
ways because there are so few assumptions one can make about its use.
>Actually, it is possible to add a new compiler to some Python
>installation, so that users can pass it to the --compiler option,
>and distutils packages don't have to be modified for that.
>You just have to extend the dictionary
Yes, and how do you accomplish *that*? You have to either modify some
Python code (e.g. sitecustomize) or use a big -c to python when running the
With entry point extension, it is only necessary to have an extension
installed on sys.path; it is not necessary to run some code to do the
In the setuptools case, if you want to add setup commands to your Python
installation, you simply install distributions containing those commands,
and they become available automatically for any setup script using
setuptools. If you want to add setup() arguments, same thing. Revision
control plugins, likewise. Egg metadata file generator plugins
also. Essentially, anything that someone can think of a use case to allow
plugins for, it is a simple change to add.
>I don't object to improving the design. I object to dropping distutils
>and replacing it with something else. If certain functionality is
>missing in distutils (e.g. means for more transparent extension), then
>distutils should be enhanced to provide this functionality.
Adding setuptools-style plugin ability to the distutils is very simple, and
doesn't require a wholesale reorganization. After all, setuptools manages
to do it! However, there are areas of build functionality that the
distutils does not cover or provides incomplete coverage for, such as
configuration and build of more complex C libraries and programs, shared
libraries, documentation production, and so on, that the existing
architecture of the distutils doesn't really encompass.
The current design of the distutils has essentially two layers: one very
high-level layer dealing with command objects, and a very low-level layer
that deals with copying files or trees thereof, making archives, and
various other simple tasks. The low-level layer is often infested with
assumptions, however, that are propagated from higher layers. Meanwhile,
the higher-level layer is based on the idea of dependency between
sequentially invoked commands, rather than dependency between things being
built. This makes it difficult to deal with concepts like having a bunch
of reST release note docs combined with a LaTeX documentation build.
So, distutils' extensibility *could* be improved dramatically in the 2.x
line by adding setuptools-style plugin entry points. However, adding any
*new* build functionality to the distutils is probably going to be better
off being based on something that offers an extensible object-dependency
layer, ala zc.buildout.
Could that be done while leaving the main distutils structure largely
intact? Perhaps. I'm just not convinced that that's what should happen
for Py3K. There's a ton of other obsolete or nearly-so stuff happening in
the existing distutils, like the specialized option parser, crazy bytecode
compiling hacks, and so on.
(Which reminds me. Does anybody have any idea why the distutils compile
modules in build/? There seems to be no point to this, since any
installation of the modules is going to need to recompile them. Even bdist
operations end up recompiling them in a different location!)
More information about the Python-3000