[Distutils] Improving distutils vs redesigning it (was people want CPAN)

Glyph Lefkowitz glyph at twistedmatrix.com
Thu Nov 12 07:10:29 CET 2009

On Nov 12, 2009, at 12:02 AM, David Cournapeau wrote:

> Glyph Lefkowitz wrote:
>> There are probably a dozen other ways that you *could* work on distutils and benefit more immediately from your efforts than the next Python release.  To think otherwise is a simply a failure of imagination.  Now, if you think it's *too hard* to do that, it might be interesting to hear why you think that, and what exactly the effort would be; a nebulous assertion that it's just too hard and we should throw our hands up (while I can definitely understand the impulse to make such an assertion) serves only to discourage everyone.
> I am trying to understand what is 'nebulous' about our claims. We have
> given plenty of hard and concrete examples of things which are
> problematic in distutils.

I'm sorry if I gave the impression that I was contesting that particular assertion.  We all agree that distutils has deep problems.

And, I don't think that everything that has been said is overgeneral or unhelpful.  Before I dive into more criticism, let me just say that I agree 100% with Robert Kern's message where he says:

> In order to integrate this with setuptools' develop command (...) we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it.

> This is nuts.

This is completely correct.  I've done stuff like this, we've all probably done stuff like this.  Conditional monkeypatching and dynamic subclassing is all over the place in distutils extension code, and it is *completely* nuts.

Still, it would have been more helpful to point out how exactly this problem could be solved, and to present (for example) a description of similar objects politely interacting and delegating responsibility to one another to accomplish the same task.

I would definitely characterize these assertion from Robert as "nebulous", given that the prior messages in the thread (as far as I can tell) do not describe the kind of massive-overhaul changes which would fix things, only the problems that currently exist:

> In our considered opinion, piecemeal changes probably aren't going to solve the significant problems that we face.

Why not?  The whole of computer history is the story of piecemeal improvements of one kind or another; despite perennial claims that, for example, hierarchical filesystems or bit-mapped displays "fundamentally" cannot support one type of data or another, here we are.

Or this one, also from Robert:

> Mostly because I'm entirely uninterested in helping you make incremental improvements that are going to break all the hard work we've already done just to get things working as it is.

Why do incremental improvements have to break all the hard work that has already been done?  Surely this is what a compatibility policy is about.

Or this one, from you, which is more specific to a topic, but still doesn't really say anything useful that I can discern:

> I think the compiler class and the likes should simply be removed ... There should not be any objects / classes for compilers, it is not flexible enough ... You cannot obtain this with classes and objects (especially when you start talking about performance: ...).

It's clear to me from the responses in this thread that I'm not the only one who is only getting vague hints of what you're actually talking about from language like this.

"classes and objects" have been used in many high-performance systems.  Personally I find "classes and objects" fairly flexible as well.  In fact, if *I* were to make a nebulous claim about distutils' design structure, it would be that the parsimony with creating whole new classes and instantiating multiple objects is the problem; there should be more classes, more objects, less inheritance and fewer methods.  So why can't 

> The major progress in our build issues have
> been achieved by dropping distutils. Up to now, the only people who have
> claimed that distutils can solve our problems are the people who are not
> involved at all with our projects, and the people who claim distutils
> cannot solve our problems are the people involved with it. That's odd to
> say the least.

I'm not asserting that distutils can fix your problems; I don't know enough about your problems to say for sure.  Certainly it seems clear that present-day distutils cannot.  I just know that there are many people on this list who are committed to a particular approach to evolving distutils, and while there is a lot of value in clearly explaining problems with that approach so they can be addressed, it's unhelpful to keep asserting (as has been done many times in this thread) that incremental evolution cannot address these problems.  It's a religious belief either way: my experience suggests that rewrites are never, ever a good idea, but others' experience may differ.

However, I feel compelled to repeat that it is a matter of historical fact and, I suspect, a corollary of the Church-Turing thesis that pretty much any software system *can* be changed into just about any other software system through a series of evolutionary steps where the system does something useful at each step; it is a question of whether you believe this approach requires an unreasonable amount of effort, and how big the steps need to be.  If you believe the effort required would be unreasonable, then let's see if we can find a radical, incompatible change to distutils that we all agree would be an improvement, and see if we also agree that the effort would be impractical.  Right now I can't find any such change looking through the recent history of this discussion, just a smattering of half-expressed ideas about how certain things in distutils are crummy and should look totally different than they are.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091112/5700c518/attachment-0001.htm>

More information about the Distutils-SIG mailing list