On 4/18/06, Barry Warsaw <barry@python.org> wrote:

On Wed, 2006-04-19 at 14:57 +1000, Anthony Baxter wrote:

...

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either.

Correct me if I'm wrong, but I don't think any of the above are currently documented in the stdlib.

Ok, I will. :-) cProfile is documented in libprofile.tex. None of the others are documented, also msilib doesn't appear to have any doc. I've updated the PEP to mention these issues. Thanks for pointing it out! Note that there is probably a bunch of doc, but it all needs to be incorporated into the python repo and format. What are the doc plans for these modules: + * ctypes + * ElementTree/cElementTree + * msilib + * pysqlite n

On Wed, Apr 19, 2006 at 10:54:09AM +0200, Gerhard Häring wrote:

We should probably check my docs in soon even in a preliminary state, so they can be reviewed/improved.

There's a group of volunteers who will help fix the LaTeX markup, so you certainly don't need to have everything working (or even learn much LaTeX) before the docs get committed. --amk

Neal Norwitz wrote:

On 4/18/06, Barry Warsaw <barry@python.org> wrote:

...

On Wed, 2006-04-19 at 14:57 +1000, Anthony Baxter wrote:

...

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either. Correct me if I'm wrong, but I don't think any of the above are currently documented in the stdlib.

Ok, I will. :-) cProfile is documented in libprofile.tex.

None of the others are documented, also msilib doesn't appear to have any doc. I've updated the PEP to mention these issues. Thanks for pointing it out!

Note that there is probably a bunch of doc, but it all needs to be incorporated into the python repo and format.

What are the doc plans for these modules: + * ctypes + * ElementTree/cElementTree + * msilib + * pysqlite

n I'm now happy with the tool that converts the ctypes tutorial from reST to LaTeX, I will later (today or tomorrow) commit that into Python SVN.

Thomas

Gerhard Häring wrote:

Thomas Heller wrote:

...

[...] I'm now happy with the tool that converts the ctypes tutorial from reST to LaTeX, I will later (today or tomorrow) commit that into Python SVN.

Did you commit that already? Alternatively, can you send it to me, please?

-- Gerhard

I meant the ctypes tutorial in latex format, not the tool itself. Anyway, the tool is mkydoc.py, in combination with missing.py: http://svn.python.org/view/external/ctypes-0.9.9.6/docs/manual/ I'm not sure it is ready for public consumption ;-). Thomas

engelbert.gruber@ssg.co.at wrote:

On Fri, 28 Apr 2006, Thomas Heller wrote:

...

Gerhard Häring wrote:

...

Thomas Heller wrote:

...

[...] I'm now happy with the tool that converts the ctypes tutorial from reST to LaTeX, I will later (today or tomorrow) commit that into Python SVN.

Did you commit that already? Alternatively, can you send it to me, please?

-- Gerhard

I meant the ctypes tutorial in latex format, not the tool itself.

Anyway, the tool is mkydoc.py, in combination with missing.py:

http://svn.python.org/view/external/ctypes-0.9.9.6/docs/manual/

I'm not sure it is ready for public consumption ;-).

from what i saw it is not, because paths are hardcoded,

Sure, but it should only require some small changes to adapt it to other rest sources.

but aside from this it should be usable and i am willing to take it up and into docutils, but this requires feedback

I must work on the docs themselves, so I currently have only two things: - the table in the ctypes tutorial has a totally different look than the other tables in the docs. Compare http://docs.python.org/dev/lib/ctypes-simple-data-types.html with http://docs.python.org/dev/lib/module-struct.html - feature request: it would be very nice if it were possible to generate links into the index for functions and types from the rest sources. Thomas

On Fri, 28 Apr 2006, Thomas Heller wrote:

I must work on the docs themselves, so I currently have only two things:

- the table in the ctypes tutorial has a totally different look than the other tables in the docs. Compare http://docs.python.org/dev/lib/ctypes-simple-data-types.html with http://docs.python.org/dev/lib/module-struct.html

could you try docutils.sf.net sandbox pydoc-writer checkout (svn or the snapshot or tomorrow http://docutils.sourceforge.net/sandbox/pydoc-writer/mkpydoc.py)

- feature request: it would be very nice if it were possible to generate links into the index for functions and types from the rest sources.

into the index means what ? into the module index ? cheers -- --- Engelbert Gruber -------+ SSG Fintl,Gruber,Lassnig / A6170 Zirl Innweg 5b / Tel. ++43-5238-93535 ---+

At 01:33 AM 4/19/2006 -0400, Barry Warsaw wrote:

On Wed, 2006-04-19 at 14:57 +1000, Anthony Baxter wrote:

...

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either.

Agreed. If modules like these have a solid history of use outside the stdlib I don't think we need all the formality of a PEP to pull them in. I /do/ think however that we need to be diligent in documenting them so that people who don't follow python-dev (or the packages own development forums) will become aware of what they are and how to use them. Correct me if I'm wrong, but I don't think any of the above are currently documented in the stdlib.

I thought that ctypes doc had been added, but I guess they're still in-progress. The setuptools docs are definitely on my plan for conversion to Pythondoc format, as per my earlier post today asking where they should go in the overall doc layout.

...

I don't have a problem at all with setuptools going into the standard library. It adds a whole pile of extremely useful functionality (easy_install, in particular, is something that people have been asking for, constantly, for YEARS). Making it an additional install is just silly.

I agree. My one stupid nit is that I don't like the name 'easy_install'. I wish a better, non-underscored word could be found.

The long term plan is for a tool called "nest" to be offered, which will offer a command-line interface similar to that of the "yum" package manager, with commands to list, uninstall, upgrade, and perform other management functions on installed packages. It's not likely to be available in time for the Python 2.5, release, but when it *is* available you'll just "python -m easy_install --upgrade setuptools" to get it. :)

At 08:51 AM 4/19/2006 +0200, Fredrik Lundh wrote:

do you expect linux and bsd packagers to switch to your stuff for all their python needs,

Heck no, which is why setuptools tries hard to be compatible with bdist_* commands. As long as they use --root or --single-version-externally-managed, setuptools should play nice with them.

or are you building a parallel universe ? if so, why ?

Many Python users are unable to use existing packaging systems - if for example they're not root, or their platform doesn't have one (Windows and Mac). People developing Python applications often need to switch between different versions of Python packages, and manage what packages are distributed with their application, while sometimes using bleeding edge versions. They can't use a system packaging tool either. People distributing Python libraries don't want to have to build packages for each and every Linux flavor, plus BSDs and other platforms. People creating extensible applications and frameworks (Zope, TurboGears, Chandler, Trac, ...) can't use system packagers to install and manage their plugins, or the libraries used by their plugins, especially if they're end-user applications or are run in shared hosting space. And there are probably other reasons that are escaping me at the moment. Mainly, though, setuptools offers developers relief from having to support myriad packaging systems in order to be able to depend on other Python packages. They specify their dependencies in terms of PyPI, not in terms of distribution X's name for a system package that installs that facility. That and plugin support are setuptools' core raison d'etre.

At 02:57 PM 4/19/2006 +1000, Anthony Baxter wrote:

Sure, it's possible that some people with extremely complicated distutils scripts may find they need to update them.

...if and *only* if they want setuptools' features, or their users do. Sorry to seize on this point out of context, Anthony. I just want to prevent anybody from seizing on this as an objection on "backward compatibility" grounds. Nobody will be forced to use setuptools in 2.5, any more than anyone will be forced to use ctypes or elementtree or sqlite.

On Wednesday 19 April 2006 16:22, Walter Dörwald wrote:

If I'm not calling shared libraries from Python I can ignore ctypes. If I'm not doing XML, I can ignore elementtree. If I'm not doing SQL I can ignore pysqlite and if I'm not interested in profiling I can ignore cProfile. But setuptools will potentially affect anyone that uses third-party modules/packages.

Sure. It might mean people can automatically install something like TurboGears and all it's dependencies, without stuffing their existing required versions (thanks to eggs) and without having to fetch 16 different packages. Oh, the horror! <wink>

And ctypes, elementtree and pysqlite are mature packages. setuptools isn't even finished yet.

Neither is distutils. setuptools at least is likely to be "finished", whatever that means.

Wouldn't I need at least have to change "from distutils.core import setup" to "from setuptools import setup"? Or to something like:

Nope, only if you want to use the new, nicer functionality. If you want to stick with the status quo, you're quite welcome to. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

At 09:11 AM 4/19/2006 +0200, Walter Dörwald wrote:

With setuptools this doesn't work, because the package is distributed over multiple egg-directories. AFAICR setuptools has a solution for this, but only if the package __init__.py is empty (because setuptools generates it). But I'd like to put at least a useful docstring into this __init__.py.

Actually, I believe that recent versions of setuptools actually do support your use case. In fact, I seem to recall having emailed you once I got it to work with your scenario, and you told me you'd already decided not to use it. But it's late at night now and I'm tired and should stop answering these emails for now anyway.

At 08:22 AM 4/19/2006 +0200, Walter Dörwald wrote:

Anthony Baxter wrote:

...

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either.

If I'm not calling shared libraries from Python I can ignore ctypes. If I'm not doing XML, I can ignore elementtree. If I'm not doing SQL I can ignore pysqlite and if I'm not interested in profiling I can ignore cProfile.

And if you're not using setuptools, or any packages that do, you can ignore it as well.

But setuptools will potentially affect anyone that uses third-party modules/packages.

Setuptools already exists, so it already affects "anyone that uses third-party modules/packages". That genie isn't going back in the metaphorical bottle. However, if setuptools is in the stdlib, it means that those people won't have to install setuptools *first* in order to use a package that's distributed using setuptools. Thus, having it in the stdlib arguably *reduces* the impact of setuptools on such users.

...

Sure, it's possible that some people with extremely complicated distutils scripts may find they need to update them.

Wouldn't I need at least have to change "from distutils.core import setup" to "from setuptools import setup"? Or to something like:

try: import ez_setup except ImportError: import distutils.core as tools else: ez_setup.use_setuptools() import setuptools as tools

for backwards compatibility reasons?

If, and *only* if, you want to use setuptools, or your users hound you enough to make you. :) Nobody is *forced* to use setuptools for a package they are distributing, any more than they are forced to use ctypes. The "setuptools takes over everything" argument is purest FUD and nonsense. Developers use setuptools because they want the features, or because their users want them to. It's true that sometimes users demand setuptools support, when all they really want is the ability to build eggs. For example, some folks were bugging Greg Ewing to add it to Pyrex, and I had to point out that as long as Pyrex has a PyPI entry and a vanilla setup.py, there is no need for him to start using setuptools. So he created a PyPI entry, making it now possible for easy_install users to do this: $ python2.5 -m easy_install Pyrex Searching for Pyrex Reading http://www.python.org/pypi/Pyrex/ Reading http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ Best match: Pyrex 0.9.4 Downloading http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/Pyrex-0.9.4.tar.gz Processing Pyrex-0.9.4.tar.gz ... zip_safe flag not set; analyzing archive contents... ... Adding Pyrex 0.9.4 to easy-install.pth file Installing pyrexc script to /home/pje/bin Installed /home/pje/lib/python2.5/site-packages/Pyrex-0.9.4-py2.5.egg Processing dependencies for Pyrex $ So, nobody was forced to import setuptools in order to offer any features to end users. Setuptools is perfectly capable of running setup scripts and making them into eggs on its own, within reasonable levels of distutils customization.

On Wed, 2006-04-19 at 18:26 +1200, Greg Ewing wrote:

I'd like to see a different approach taken to the design altogether, something more along the lines of Scons. Maybe it could even be an extension of Scons.

As much as I like Scons, there's too much unpythonic magic going on there that I wouldn't want emulated. -Barry

At 07:46 PM 4/19/2006 +0200, Martin v. Löwis wrote:

Greg Ewing wrote:

...

...

I started refactoring some of the ugliness out of the internals of distutils last year, but was completely stymied by the demand that no existing setup.py scripts be broken.

Instead of trying to fix distutils, maybe it would be better to start afresh with a new package and a new name, then there wouldn't be any backwards-compatibility issues to hold it back.

It is *precisely* my concern that this happens. For whatever reason, writing packaging-and-deployment software is totally unsexy.

I can tell you the reasons, no need to guess: 1. It's bloody hard work. Honestly, if I really knew what I was in for by doing this, I might not have started. That doesn't mean I'm going to *stop*, just that I'd have thought twice before starting. 2. Everybody thinks they can do it better, and isn't afraid to tell you so. 3. People complain that things didn't magically work the way they expected, but rarely provide any information about what they did or didn't do or how they think it should've figured out what they mean. 4. When it works, nobody notices or cares. When it doesn't work *once*, people blog that it's crap and should never be used -- but don't report the problem or ask for help, and don't change their judgment of "crap" even if the problem is fixed in a few days through somebody sending me mail about the blog article.

If distutils is now abandoned and replaced with something else, the same story will happen again: the developers will run away, the package gets abandoned, and, after a few years of sadness, a new, smart developer will come along and provide a super replacement. And that will repeat in cycles of roughly 10 years.

We have to stop this. If distutils has flaws, fix them.

I agree with you, which is why setuptools fixes distutils' flaws by subclassing where possible and monkeypatching only where necessary to ensure compatibility. (Only three classes are monkeypatched: Distribution, Command, and Extension.) It would be better to simply integrate those changes into the distutils in the long run, rather than maintaining two layers. The problem in the short term, however, is backward compatibility: sometimes people are relying on internals or implementation details of the existing distutils. I would suggest that setuptools be considered "distutils2" for all practical purposes, with the intention of being merged into the distutils proper for Py3K. Whether setuptools gets included in Python 2.5 or not, it should be in *some* 2.x release, and merge fully in 3.x.

At 04:15 PM 4/19/2006 -0400, A.M. Kuchling wrote:

At least some of these changes to Distutils seem unobjectionable for inclusion.

For example, the changes to Command just allow keyword arguments on two methods and adds a class attribute; they seem unlikely to break any existing users of the class.

The Extension change replaces .pyx source files with .c; I'm not sure what the purpose of this change is, but it clearly might cause problems for distributions with Pyrex source files.

The purpose of the extension change is that it first checks whether Pyrex's distutils extension is available. If it is not available, it changes the source file names to .c, so that any pre-processed Pyrex files will be processed. This allows Pyrex-using packages to include pre-translations from Pyrex to C, and non-Pyrex users can still compile the package. For example, PyProtocols ships with a _speedups.pyx file, and a _speedups.c file, produced locally with Pyrex. Let's say a user is building PyProtocols from source. If they have Pyrex installed, setuptools uses Pyrex to rebuild the .c from the .pyx. If they do *not* have Pyrex installed, setuptools tells distutils to just build directly from the .c file, so that the distutils isn't confused by all this '.pyx' business.

The Distribution changes add some more optional kw arguments -- no problem there -- and a bunch of egg-specific methods. This set would need some further study, and also bakes in .egg support; we'd have to be very sure that .eggs are the way to go, so this might need a PEP and/or BDFL pronouncement.

There's a slightly deeper issue there than just egg support itself. It's *plugin* support. Setuptools allows eggs to register plugins to provide new commands or keyword options that then apply either globally or only to specific setup scripts. Setuptools then uses this mechanism to register its own commands as distutils replacements. What this means is that if setuptools is installed, and you use setuptools' Distribution class, then setuptools' command plugins will apply -- which means that you now have setuptools behavior for those commands, instead of distutils default behavior. And this would be the case even if you never imported setuptools, if you just ported the Distribution class over. So, you can't actually activate the plugin support in setuptools' Distribution for distutils, or you'll be removing backward compatibility. There would have to be some way to specify whether setuptools or distutils should take precedence in the event that both define a given command. For example, a 'use_setuptools' argument that defaults to false, unless you first imported setuptools (for back/side compatibility w/existing setuptools scripts).

Obviously, applying changes to Distutils makes Phillip's maintenance of setuptools more difficult -- now he has to worry about monkeypatching or not depending on the Python version.

Yeah, there's already some of this to deal with for package data. Setuptools was added to the sandbox in 2004, and shortly afterwards, Fred Drake took on the task of adding some of its features to distutils for Python 2.4. Which was great, except that then setuptools had to start worrying about whether or not the stuff it was subclassing was in fact already doing what it was trying to do. So, there's a bunch of fragile conditional code in there, and I'm not thrilled at the idea of adding more, because it significantly increases the number of testing combinations required to ensure that things work when changes are made, as well as making it harder to tell what's going on in the code.

But at least some of the monkeypatching can be removed -- maybe all of it if the Distribution class proves amenable.

The Distribution class also adds the ability for individual commands to have positional argument parsing.

Martin v. Löwis wrote:

It is *precisely* my concern that this happens. For whatever reason, writing packaging-and-deployment software is totally unsexy.

for some reason, tools of this kind tend to reach the big ball of mud stage even before they reach the dogfood stage. and once you have a big ball of mud, you simply won't get much outside help -- not because it's unsexy, but because it's nearly impossible to contribute, at any level, unless you have days and days to spare... (distutils and setuptools are over 15000 lines of code, according to sloc- count. it took Python several years to accumulate a standard library that big ;-) </F>

On Thursday 20 April 2006 06:08, skip@pobox.com wrote:

Fredrik> for some reason, tools of this kind tend to reach the big ball Fredrik> of mud stage even before they reach the dogfood stage. and Fredrik> once you have a big ball of mud, you simply won't get much Fredrik> outside help

Not to mention many dogs won't eat mud...

Most dogs will eat _anything_ that might look like food. <wink>

Greg Ewing wrote:

Fredrik Lundh wrote:

...

(distutils and setuptools are over 15000 lines of code, according to sloc- count.

Ye cats! That's a *seriously* big ball of mud. I just checked, and the whole of Pyrex is only 17000 lines.

correction: it's actually only 14000 lines, but it's still the largest hand- written component we have in the standard library (when combined). ~90 modules, ~120 classes. (the mac library is larger, and so is encodings, but large parts of those are autogenerated from character databases and header files. and idle is larger than distutils, but smaller than distutils+setuptools) </F>

At 10:39 AM 4/20/2006 +1000, Anthony Baxter wrote:

On Thursday 20 April 2006 03:46, Martin v. Löwis wrote:

...

It is *precisely* my concern that this happens. For whatever reason, writing packaging-and-deployment software is totally unsexy. This is why setuptools is a one-man show, and this is why the original distutils authors ran away after they convinced everybody that distutils should be part of Python. If distutils is now abandoned and replaced with something else, the same story will happen again: the developers will run away,

Well, I've seen no indication that this is Phillip's plan. If it is, could he tell us now? <wink>

I don't know how widely the distutils were used *before* they became a part of Python, so it's possible the authors didn't know what they were in for. Of course, it's still possible that I don't know what I'm in for either. However, the amount of work I've been putting in wouldn't be possible without OSAF's indirect finanical support for the project. But, setuptools solves problems that commercial entities are often interested in solving. The Envisage folks inquired at Pycon about me possibly helping expand setuptools' scope to building and installing C dynamic link libraries, and this is also a feature potentially desired for Chandler's build system as well. What this means is that there is a potential basis for supporting the work going forward, at least through what I'm hoping will be the "1.0" version. And I suspect that there are other improvements possible that other commercial entities would be interested in funding. How much any of that could have also applied to the distutils at one time, I don't know. My point is merely that setuptools has enough commercial value to make me believe that sponsorship for features shouldn't be that hard to come by, and there are certainly worse things I could do with my work days. This shouldn't be construed as saying I'll only work on setuptools if paid; it merely means that the huge gobs of time I've been putting into it at times are only *possible* because it aligns with OSAF goals enough to let me put a substantial chunk of my work days and nights into it.

I started looking at this. The number of complaints I got when I started on this that it would break the existing distutils based installers totally discouraged me. In addition, the existing distutils codebase is ... not good.

You know, after my experience doing setuptools, I'm a lot less inclined to criticize anybody's code, if it works and solves a problem. Sometimes, the structure of the problems are just not amenable to beautiful solutions. Or more to the point, beautiful solutions are not always economical to implement while you're supporting actual users and backwards compatibility is a constant requirement. So my attitude to the distutils code has mellowed considerably; I view it as a collection of things that I at least don't have to think about. C compilers, for example, are something I don't have to touch at all, except for some recent experiments in implementing shared library building. So, I don't really view the distutils as broken per se; it's just that they lack a lot of desirable features that are hard to implement without breaking backward compatibility. That's not quite the same thing as "the code sucks and we should rewrite it". There have been a lot of calls to scrap it and rewrite it throughout the years I've been working on setuptools, but I can't imagine who could possibly *afford* such an undertaking (as opposed to talking about it).

Yes. I remember that piece. In particular, he wrote the original rant about this about Mozilla/Firefox. How did that work out again? Oh, that's right - we now have a much, much more successful and usable browser.

Well, the analogy isn't great anyway, because setuptools is hardly a rewrite of the distutils. I didn't rewrite a single thing I could get away with calling or subclassing. Setuptools is all about adding new features, not "cleaning up" the distutils code base, so arguing about whether rewrites are good or not is a straw man holding a red herring. :)

Anthony Baxter wrote:

...

http://www.joelonsoftware.com/articles/fog0000000069.html

Yes. I remember that piece. In particular, he wrote the original rant about this about Mozilla/Firefox. How did that work out again? Oh, that's right - we now have a much, much more successful and usable browser.

his point isn't that it cannot be done, it's that it is horridly expensive and time consuming. exactly how much money and time has gone into getting firefox to the point where it is today? (insert obligatory bastiat reference here) </F>

Anthony Baxter wrote:

And I would reply that sometimes a rewrite is necessary. I doubt firefox would be at the state it is today if it was still based on the ancient netscape codebase.

It's off-topic here certainly: but firefox is actually not a complete rewrite; it still has tons of "ancient netscape codebase" in it. The could not have completed it otherwise. Regards, Martin

On Thursday 20 April 2006 15:53, Martin v. Löwis wrote:

I don't know about Phillip's plans, but I do see many indications that people stop using distutils, and use setuptools instead.

Surely that's an indication that it _should_ become part of Python? If there's an obvious demand for the features. In addition, it's available for 2.3 and 2.4 as a separate package.

...

I started looking at this. The number of complaints I got when I started on this that it would break the existing distutils based installers totally discouraged me.

I believe this is a myth. Some changes might cause breakage, yes, but others wouldn't. For example, introducing additional parameters to setup is just fine: existing packages won't break; new packages using these parameters won't build on older Python releases, of course.

Something as simple as replacing the distutils.fancy_getopt with optparse couldn't be done in a completely backwards compatible way, because of some of the funkiness in fancy_getopt that people use today.

...

In addition, the existing distutils codebase is ... not good.

Then it shouldn't have become part of Python in the first place. Can you please elaborate what specific aspects of distutils you dislike?

It's extremely awkward to plug in functionality that _should_ be more obvious and extensible. At least, unless you want to make every single setup.py include a magic bit of patching to Do The Right Thing. There are a number of places where the code is obviously not finished, when Greg clearly got fed up with the project <wink>. There was no decent documentation of the internals.

...

It is flatly not possible to "fix" distutils and preserve backwards compatibility.

Why?

Because there are too many people that have poked too deeply into the innards with their setup.py scripts. Partly this is a function of the lack of documentation - until I sat down and spent a couple of days on it a year or so ago, there wasn't a library reference. There's still no decent "here's the Approved Way to poke about inside distutils" doc.

If I had the time, I would question each of these. Yes, it is easier for the author of the new package to build "in the green", but it is (nearly) never necessary, and almost always bad for the project.

I guess I disagree almost entirely. The new email package is vastly, vastly better than the old hodge-podge of stuff that was written back in python's Dark Ages. Trying to fix those would have been almost impossible. In addition, we'd be stuck with the need to maintain backwards compatibility. Something like spambayes or almost anything else handling modern email traffic would then not be possible in Python.

Yes, and everybody has to rewrite their code, because the "old" modules don't see fixes, and get obsoleted and eventually removed. Users get the impression that Python breaks their APIs for no good reason every now and then.

distutils doesn't _get_ fixes. Well, hardly any. Compare that to a 'svn log' in sandbox/setuptools. I know I much prefer an actively maintained and developed codebase to some ancient but inert existing system. Heck, most of the recent work on distutils was an offshoot of setuptools. As far as "breaks their APIs" - this is a pretty rare occurance, it might have happened half a dozen times so far. And the old module still keeps working fine. If there's bugs that someone logs, they will get addressed exactly the same as any other bug. I don't see people closing patches off as "that's an old module, I'm not going to apply it". It also takes _years_ for older modules to go through the process of deprecation, moving to lib-old, and final removal. I mean, heck, rfc822 still doesn't issue any deprecation warnings. I suspect C++ or Java change faster <wink>. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On Thu, Apr 20, 2006, A.M. Kuchling wrote:

I don't mind rewriting much, but hate leaving the original code in place; this is confusing to new users, even if it's convenient for existing users. How many HTML parsers are in the core now? (My gut feeling is that that Python's adoption curve has flattened, so it's probably now more important to keep existing users happy, so the time for jettisoning modules may be past.)

There's a lot of truth for that with Python 2.x, but that's what 3.0 is for. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach

While we're on the subject of distutils revision, here are a few things I've encountered about distutils which seem less than desirable. * There doesn't seem to be a way of supplying options on the command line for anything except the top-level command. Sometimes, e.g. I want to do an "install" but override some options for the "build_ext" that gets invoked by the install. But distutils won't let me, because those options are not recognised by the "install" command itself. If I try to get around this by doing a "build_ext" explicitly, then "install", the build_ext gets done again the second time around, without my options. I know I can do this using a config file, but the details of that don't seem to fit in my brain, and I have to go looking up the docs. Come to that, almost none of distutils seems to fit in my brain. Whenever I go to write a new setup.py I have to go and read the docs to remind myself what it's supposed to look like. * It seems to be next to impossible to override some of the options that get passed to the C compiler. For Pyrex I'd very much like to be able to turn off the -Wall that it seems to insist on using, but I haven't been able to find a way. Last time I tried, I got lost in a maze of twisty method calls, all alike. * The mechanisms for extending it seem clumsy. Since there's an Extension class, the you'd think the obvious way to add support for Pyrex would be to define a PyrexExtension type that embodied all the necessary knowledge to deal with a .pyx file. But that's not the way it seems to work. The current Pyrex distutils extension (which I didn't write) works by hijacking a pre-existing method designed for processing swig files. If that's really the cleanest way it can be done, something is badly wrong with the design. This is what I meant when I said I'd like a more Scons-like approach. Someone complained that Scons was too magical. I don't necessarily want it to behave exactly like Scons, but just to have the functionality divided into independent parts that can be composed in the manner of Scons, so that I could add a component for processing .pyx -> .c that builds on the existing components for dealing with .c files, and do it in an obvious and straight- forward way (and hopefully without having to be excessively Dutch in order to see the obviousness of it:-). -- Greg

Paul Moore wrote:

I do things like

python setup.py build --compiler=mingw32 bdist_wininst

which seem to work for me. Is that any help?

Possibly. I'll have to try it next time I have the problem and see. BTW, does that do anything different from python setup.py build --compiler=mingw32 python setup.py bdist_wininst ? If so, that's rather unintuitive and could do with documenting more clearly. -- Greg

Thomas Heller wrote:

The best solution is to configure the mingw32 compiler in the distutils configuration file.

That's the same conclusion I came to. But it's unintuitive that you can't also do the same thing using command line options, or if you can, it's not obvious how to do it. -- Greg

Anthony Baxter wrote:

...

http://www.joelonsoftware.com/articles/fog0000000069.html

From what I remember, he didn't actually say that you should never rewrite anything, but that if you do, you need to be prepared for a very long period of time when nothing new is working, and that *if you are a commercial company* this is a bad idea. But Python is not dependent on getting new features out the door before its competitors in order to survive, so this doesn't really apply. It doesn't matter if people have to make do with the old distutils for a while until something better is available. -- Greg

Guido van Rossum wrote:

I'd rather recommend the approach that Joel suggests for truly large systems: refactoring smaller components while keeping the overall structure intact.

That's fine as long as the overall structure isn't the very thing that's wrong and needs to be fixed. Incremental refactoring is a hill-climbing approach, and there's a well-known problem with hill-climbing algorithms -- they have a tendency to get stuck on the top of a small hill and fail to find the big hill. I appreciate the loss-of-knowledge problem, I really do. But maybe there is a way of dealing with it in those cases where a change of structure really is needed. Instead of throwing away the old code, go over it line by line, sucking out the knowledge, digesting it, and incorporating it into the relevant places in the new structure. When you've finished, and all that's left of the old code is a dry, empty husk, then you can throw it away. Perhaps this process could be called "pupation" -- breaking down the old structure and building something new out of the parts, without wasting any of the original material. I'm not claiming that this necessarily has to be done with distutils, but if it did need to be done, I think it could be done, with sufficient effort. I do think that the fact nobody but the now-absent author seems to understand the internals of distutils is a very serious problem, and something needs to be done to make it easier to understand and maintain. -- Greg

A.M. Kuchling wrote:

The 2.1 requirement was originally imposed because Greg Ward would make standalone Distutil releases. Greg is too busy working at his job, going camping on weekends, and being best man [1] to make new releases these days.

from python-announce: From: Greg Ward Subject: ANNOUNCE: Optik 1.5.1 Date: 2006-04-20 01:46:20 GMT Optik 1.5.1 is now available, just 16 months after I first planned to release it (sigh). ... </F>

Martin v. Löwis wrote:

If distutils is now abandoned and replaced with something else, the same story will happen again: the developers will run away, the package gets abandoned,

Seems to me that if we had something with a clean design that was easy to understand, maintain and extend, that this wouldn't be so much of a problem. If the original author ran away, others would more easily be able to take over the task.

We have to stop this. If distutils has flaws, fix them. Never ever even think about rewriting software:

Usually this is good advice, but it is possible for something to be so badly broken that the only reasonable way to fix it is to throw it away and start over. I'm not sure whether distutils is really that badly broken. But an earlier poster seemed to be saying that he had great difficulty finding anything that could be changed without breaking something that someone relied on. It's hard to fix something if you can't change it at all. I'd be happy to discuss ways of evolving distutils into something better, but first we have to decide that it is actually permissible to change it and possibly break stuff that's relying on its internals. -- Greg

Anthony Baxter wrote:

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either.

That's because they're all trivial building blocks, not all-consuming world views. Any programmer who's ever used a C compiler/linker, xml loader, or embedded database will understand how to use them, and what they do, right from the start. There's not a trace of "you don't need to under- stand this" or "won't anybody think of the newbies!" in any of them. They're also all relatively close to the simplest thing that could possibly work. setuptools, in contrast, appears to be an overengineered kitchen sink with tons of magic in-phillips-head-only features, all of which may have unforseen effects on all parts of your Python installation. I've skimmed the PEAK documentation, and all I find is bullet-point feature lists and endless lists of configuration options. It's like reading Microsoft documentation. </F>

Anthony Baxter wrote:

I'm not sure how people would prefer this be handled. I don't think we need to have a PEP for it - I don't see PEPs for ctypes, elementtree, pysqlite or cProfile, either.

I see a significant procedural difference between what happened for ctypes, elementtree, and pysqlite, as opposed to setuptools. For all these packages, there was 1. a desire of users to include it 2. an indication from the package maintainer that it's ok to include the package, and that he is willing to maintain it 3. some discussion on python-dev, which resulted only in support and no objection 4. some (other) committer who "approved" incorporation of the library. In essence, that committer is a "second" for the package inclusion. setuptools has 1 and 2, but fails on 3 and 4 so far. There is discussion now after the fact, but it results in objection. I'd like to point out the importance of 4: Fredrik Lundh originally asked "who approved that change", which really meant "who can I blame for it". I feel that I approved inclusion of ctypes and elementtree: I talked with the developers on how precisely it should happen, and I checked then that everything that I thought should happened indeed happened. And I did the majority of the communication on python-dev. So the package authors can get all the praise, and I happily take all the blame. The same didn't happen for setuptools: there is no second, so Phillip Eby takes all the praise *and* the blame. It's still a one-man show. Now, I know that Neal Norwitz had asked him what the status is and when it will happen, but he apparently did not want to *approve* inclusion of that package. Likewise, Guido van Rossum (apparently) did not want to approve it, either (he just would not object). If you (Anthony) want to act as a second for setuptools, I would feel much happier - because I can then blame you for whatever problems that decision causes five years from now. Regards, Martin

Anthony> I don't think it's fair to say that Phillip just checked this Anthony> in off on his own. In addition, since he did the development in the Python sandbox, his checkins all along have been there for everyone to see. It's not like he did the work in Outer Mongolia then showed up one day checked it in while everyone was sleeping. I see mention of "setuptools" in the distutils-sig archive at least as far back as May 2005 and mention of "eggs" back to at least April 2005. Skip