Andrew Kuchling <akuchlin@mems-exchange.org>:

...

1. The curses module is commented out in the default Modules/Setup file. This is not good, as it may lead careless distribution builders

It always has been commented out. Good distributions ship with most of the available modules enabled; I can't say if RH7.0 counts as a good distribution or not (still on 6.2).

I think this needs to change. If curses is a core facility now, the default build should tread it as one. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> If a thousand men were not to pay their tax-bills this year, that would ... [be] the definition of a peaceable revolution, if any such is possible. -- Henry David Thoreau

Thomas Wouters <thomas@xs4all.net>:

...

1. The curses module is commented out in the default Modules/Setup file. This is not good, as it may lead careless distribution builders to ship Python 2.0s that will not be able to support the curses front end in CML2. Supporting CML2 (and thus getting Python the "design win" of being involved in the Linux kernel build) was the major point of integrating the curses module into the Python core. It is possible that one little "#" may have blown that.

Note that Tkinter is off by default too. And readline. And ssl. And the use of shared libraries.

IMO ssl isn't an issue because it's not documented as being in the standard module set. Readline is a minor issue because raw_input()'s functionality changes somewhat if it's not linked, but I think we can live with this -- the change isn't visible to calling programs. Hm. It appears tkinter isn't documented in the standard set of modules either. Interesting. Technically this means I don't have a problem with it not being built in by default, but I think there is a problem here... My more general point is that right now Pyjthon has three classes of modules: 1. Documented as being in the core and built in by default. 2. Not documented as being in the core and not built in by default. 3. Documented as being in the core but not built in by default. My more general claim is that the existence of class 3 is a problem, because it compromises the "batteries are included" effect -- it means Python users don't have a bright-line test for what will be present in every Python (or at least every Python on an operating system theoretically feature-compatible with theirs). My struggle to get CML2 adopted brings this problem into particularly sharp focus because the kernel group is allergic to big footprints or having to download extension modules to do a build. But the issue is really broader than that. I think we ought to be migrating stuff out of class 3 into class 1 where possible and to class 2 only where unavoidable.

We *can't* enable the cursesmodule by default, because we don't know what the system's curses library is called. We'd have to auto-detect that before we can enable it (and lots of other modules) automatically, and that's a lot of work. I personally favour autoconf for the job, but since amk is already busy on using distutils, I'm not going to work on that.

Yes, we need to do a lot more autodetection -- this is a large part of my point. I have nothing against distutils, but I don't see how it solves this problem unless we assume that we'll always have Python already available on any platform where we're building Python. I'm willing to put my effort where my mouth is on this. I have a lot of experience with autoconf; I'm willing to write some of these nasty config tests.

...

2.The default Modules/Setup file assumes that various Tkinter-related libraries are in /usr/local. But /usr would be a more appropriate choice under most circumstances. Most Linux users now install their Tcl/Tk stuff from RPMs or .deb packages that place the binaries and libraries under /usr. Under most other Unixes (e.g. Solaris) they were there to begin with.

This is nonsense. The line above it specifically states 'edit to reflect where your Tcl/Tk headers are'. And besides from the issue whether they are usually found in /usr (I don't believe so, not even on Solaris, but 'my' Solaris box doesn't even have tcl/tk,) /usr/local is a perfectly sane choice, since /usr is already included in the include-path, but /usr/local usually is not.

Is it? That is not clear from the comment. Perhaps this is just a documentation problem. I'll look again.

...

3. The configure machinery could be made to deduce more about the contents of Modules/Setup than it does now. In particular, it's silly that the person building Python has to fill in the locations of X librasries when configure is in principle perfectly capable of finding them.

In principle, I agree. It's a lot of work, though. For instance, Debian stores the Tcl/Tk headers in /usr/include/tcl<version>, which means you can compile for more than one tcl version, by just changing your include path and the library you link with. And there are undoubtedly several other variants out there.

As I said to Guido, I think it is exactly our job to deal with this sort of grottiness. One of Python's major selling points is supposed to be cross-platform consistency of the API. If we fail to do what you're describing, we're failing to meet Python users' reasonable expectations for the language.

Should we really make the Setup file default to Linux, and leave other operating systems in the dark about what it might be on their system ? I think people with Linux and without clue are the least likely people to compile their own Python, since Linux distributions already come with a decent enough Python. And, please, lets assume the people assembling those know how to read ?

Please note that I am specifically *not* advocating making the build defaults Linux-centric. That's not my point at all.

Maybe we just need a HOWTO document covering Setup ?

That would be a good idea.

(Besides, won't this all be fixed when CML2 comes with a distribution, Eric ? They'll *have* to have working curses/tkinter then :-)

I'm concerned that it will work the other way around, that CML2 won't happen if the core does not reliably include these facilities. In itself CML2 not happening wouldn't be the end of the world of course, but I'm pushing on this because I think the larger issue of class 3 modules is actually important to the health of Python and needs to be attacked seriously. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The Bible is not my book, and Christianity is not my religion. I could never give assent to the long, complicated statements of Christian dogma. -- Abraham Lincoln

Thomas Wouters <thomas@xs4all.net>:

...

My more general claim is that the existence of class 3 is a problem, because it compromises the "batteries are included" effect -- it means Python users don't have a bright-line test for what will be present in every Python (or at least every Python on an operating system theoretically feature-compatible with theirs).

It depends on your definition of 'being in the core'. Some of the things that are 'in the core' are simply not possible on all platforms. So if you want really portable code, you don't want to use them. Other features are available on all systems that matter [to you], so you don't really care about it, just use them, and at best document that they need feature X.

I understand. We can't, for example, guarantee to duplicate the Windows- specific stuff in the Unix port (nor would we want to in most cases :-)). However, I think "we build in curses/Tkinter everywhere the corresponding libraries exist" is a guarantee we can and should make. Similarly for other modules presently in class 3.

There is also the subtle difference between a Python user and a Python compiler/assembler (excuse my overloading of the terms, but you know what I mean).

Yes. We have three categories here: 1. People who use python for applications (what I've been calling users) 2. People who configure Python binary packages for distribution (what you call a "compiler/assembler" and I think of as a "builder"). 3. People who hack Python itself. Problem is that "developer" is very ambiguous in this context...

People who choose to compile their own Python should realize that they might disable or misconfigure some parts of it. I personally trust most people that assemble OS distributions to compile a proper Python binary + modules, but I think a HOWTO isn't a bad idea -- unless we autoconf everything.

I'd like to see both things happen (HOWTO and autoconfing) and am willing to work on both.

I apologize for the tone of my previous post, and the above snippet.

No offense taken at all, I assure you.

I'm not trying to block progress here ;) I'm actually all for autodetecting as much as possible, and more than willing to put effort into it as well (as long as it's deemed useful, and isn't supplanted by a distutils variant weeks later.) And I personally have my doubts about the distutils variant, too, but that's partly because I have little experience with distutils. If we can work out a deal where both autoconf and distutils are an option, I'm happy to write a few, if not all, autoconf tests for the currently disabled modules.

I admit I'm not very clear on the scope of what distutils is supposed to handle, and how. Perhaps amk can enlighten us?

So, Eric, let's split the work. I'll do Tkinter if you do curses. :)

You've got a deal. I'll start looking at the autoconf code. I've already got a fair idea how to do this. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> No one who's seen it in action can say the phrase "government help" without either laughing or crying.

Guido van Rossum <guido@digicool.com>:

Not all the world is Linux. CML2 isn't the only Python application that matters. Python world domination is not a goal. There is no Eric conspiracy! :-)

Perhaps I misunderstood you, then. I thought you considered CML2 an potentially important design win, and that was why curses didn't get dropped from the core. Have you changed your mind about this? If Python world domination is not a goal then I can only conclude that you haven't had your morning coffee yet :-). There's a more general question here about what it means for something to be in the core language. Developers need to have a clear, bright-line picture of what they can count on to be present. To me this implies that it's the job of the Python maintainers to make sure that a facility declared "core" by its presence in the standard library documentation is always present, for maximum "batteries are included" effect. Yes, dealing with cross-platform variations in linking curses is a pain -- but dealing with that kind of pain so the Python user doesn't have to is precisely our job. Or so I understand it, anyway. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Conservatism is the blind and fear-filled worship of dead radicals.

Guido van Rossum <guido@digicool.com>:

Supporting CML2 was one of the reasons to keep curses in the core, but not the only one. Linux kernel configuration is so far removed from my daily use of computers that I don't have a good way to judge its importance in the grand scheme of things. Since you obviously consider it very important, and since I generally trust your judgement (except on the issue of firearms :-), your plea for keeping, and improving, curses support in the Python core made a difference in my decision. And don't worry, I don't expect to change that decision -- though I personally still find it curious that curses is so important. I find curses-style user interfaces pretty pathetic, and wished that Linux migrated to a real GUI for configuration. (And the linuxconf approach does *not* qualify as a a real GUI. :-)

Thank you, that makes your priorities much clearer. Actually I agree with you that curses interfaces are mostly pretty pathetic. A lot of people still like them, though, because they tend to be fast and lightweight. Then, too, a really well-designed curses interface can in fact be good enough that the usability gain from GUIizing is marginal. My favorite examples of this are mutt and slrn. The fact that GUI programs have failed to make much headway against this is not simply due to user conservatism, it's genuinely hard to see how a GUI interface could be made significantly better. And unfortunately, there is a niche where it is still important to support curses interfacing independently of anyone's preferences in interface style -- early in the system-configuration process before one has bootstrapped to the point where X is reliably available. I hasten to add that this is not just *my* problem -- one of your more important Python constituencies in a practical sense is the guys who maintain Red Hat's installer.

I was totally serious though: my personal satisfaction doesn't come from Python world domination. Others seem have that goal, and if it doesn't inconvenience me too much I'll play along, but in the end I've got some goals in my personal life that are much more important.

There speaks the new husband :-). OK. So what *do* you want from Python? Personally, BTW, my goal is not exactly Python world domination either -- it's that the world should be dominated by the language that has the least tendency to produce grotty fragile code (remember that I tend to obsess about the software-quality problem :-)). Right now that's Python. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The people of the various provinces are strictly forbidden to have in their possession any swords, short swords, bows, spears, firearms, or other types of arms. The possession of unnecessary implements makes difficult the collection of taxes and dues and tends to foment uprisings. -- Toyotomi Hideyoshi, dictator of Japan, August 1588

Andrew Kuchling <akuchlin@mems-exchange.org>:

...

1. The curses module is commented out in the default Modules/Setup file. This is not good, as it may lead careless distribution builders

It always has been commented out. Good distributions ship with most of the available modules enabled; I can't say if RH7.0 counts as a good distribution or not (still on 6.2).

I think this needs to change. If curses is a core facility now, the default build should tread it as one. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> If a thousand men were not to pay their tax-bills this year, that would ... [be] the definition of a peaceable revolution, if any such is possible. -- Henry David Thoreau

Thomas Wouters <thomas@xs4all.net>:

...

1. The curses module is commented out in the default Modules/Setup file. This is not good, as it may lead careless distribution builders to ship Python 2.0s that will not be able to support the curses front end in CML2. Supporting CML2 (and thus getting Python the "design win" of being involved in the Linux kernel build) was the major point of integrating the curses module into the Python core. It is possible that one little "#" may have blown that.

Note that Tkinter is off by default too. And readline. And ssl. And the use of shared libraries.

IMO ssl isn't an issue because it's not documented as being in the standard module set. Readline is a minor issue because raw_input()'s functionality changes somewhat if it's not linked, but I think we can live with this -- the change isn't visible to calling programs. Hm. It appears tkinter isn't documented in the standard set of modules either. Interesting. Technically this means I don't have a problem with it not being built in by default, but I think there is a problem here... My more general point is that right now Pyjthon has three classes of modules: 1. Documented as being in the core and built in by default. 2. Not documented as being in the core and not built in by default. 3. Documented as being in the core but not built in by default. My more general claim is that the existence of class 3 is a problem, because it compromises the "batteries are included" effect -- it means Python users don't have a bright-line test for what will be present in every Python (or at least every Python on an operating system theoretically feature-compatible with theirs). My struggle to get CML2 adopted brings this problem into particularly sharp focus because the kernel group is allergic to big footprints or having to download extension modules to do a build. But the issue is really broader than that. I think we ought to be migrating stuff out of class 3 into class 1 where possible and to class 2 only where unavoidable.

We *can't* enable the cursesmodule by default, because we don't know what the system's curses library is called. We'd have to auto-detect that before we can enable it (and lots of other modules) automatically, and that's a lot of work. I personally favour autoconf for the job, but since amk is already busy on using distutils, I'm not going to work on that.

Yes, we need to do a lot more autodetection -- this is a large part of my point. I have nothing against distutils, but I don't see how it solves this problem unless we assume that we'll always have Python already available on any platform where we're building Python. I'm willing to put my effort where my mouth is on this. I have a lot of experience with autoconf; I'm willing to write some of these nasty config tests.

...

2.The default Modules/Setup file assumes that various Tkinter-related libraries are in /usr/local. But /usr would be a more appropriate choice under most circumstances. Most Linux users now install their Tcl/Tk stuff from RPMs or .deb packages that place the binaries and libraries under /usr. Under most other Unixes (e.g. Solaris) they were there to begin with.

This is nonsense. The line above it specifically states 'edit to reflect where your Tcl/Tk headers are'. And besides from the issue whether they are usually found in /usr (I don't believe so, not even on Solaris, but 'my' Solaris box doesn't even have tcl/tk,) /usr/local is a perfectly sane choice, since /usr is already included in the include-path, but /usr/local usually is not.

Is it? That is not clear from the comment. Perhaps this is just a documentation problem. I'll look again.

...

3. The configure machinery could be made to deduce more about the contents of Modules/Setup than it does now. In particular, it's silly that the person building Python has to fill in the locations of X librasries when configure is in principle perfectly capable of finding them.

In principle, I agree. It's a lot of work, though. For instance, Debian stores the Tcl/Tk headers in /usr/include/tcl<version>, which means you can compile for more than one tcl version, by just changing your include path and the library you link with. And there are undoubtedly several other variants out there.

As I said to Guido, I think it is exactly our job to deal with this sort of grottiness. One of Python's major selling points is supposed to be cross-platform consistency of the API. If we fail to do what you're describing, we're failing to meet Python users' reasonable expectations for the language.

Should we really make the Setup file default to Linux, and leave other operating systems in the dark about what it might be on their system ? I think people with Linux and without clue are the least likely people to compile their own Python, since Linux distributions already come with a decent enough Python. And, please, lets assume the people assembling those know how to read ?

Please note that I am specifically *not* advocating making the build defaults Linux-centric. That's not my point at all.

Maybe we just need a HOWTO document covering Setup ?

That would be a good idea.

(Besides, won't this all be fixed when CML2 comes with a distribution, Eric ? They'll *have* to have working curses/tkinter then :-)

I'm concerned that it will work the other way around, that CML2 won't happen if the core does not reliably include these facilities. In itself CML2 not happening wouldn't be the end of the world of course, but I'm pushing on this because I think the larger issue of class 3 modules is actually important to the health of Python and needs to be attacked seriously. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The Bible is not my book, and Christianity is not my religion. I could never give assent to the long, complicated statements of Christian dogma. -- Abraham Lincoln

Thomas Wouters <thomas@xs4all.net>:

...

My more general claim is that the existence of class 3 is a problem, because it compromises the "batteries are included" effect -- it means Python users don't have a bright-line test for what will be present in every Python (or at least every Python on an operating system theoretically feature-compatible with theirs).

It depends on your definition of 'being in the core'. Some of the things that are 'in the core' are simply not possible on all platforms. So if you want really portable code, you don't want to use them. Other features are available on all systems that matter [to you], so you don't really care about it, just use them, and at best document that they need feature X.

I understand. We can't, for example, guarantee to duplicate the Windows- specific stuff in the Unix port (nor would we want to in most cases :-)). However, I think "we build in curses/Tkinter everywhere the corresponding libraries exist" is a guarantee we can and should make. Similarly for other modules presently in class 3.

There is also the subtle difference between a Python user and a Python compiler/assembler (excuse my overloading of the terms, but you know what I mean).

Yes. We have three categories here: 1. People who use python for applications (what I've been calling users) 2. People who configure Python binary packages for distribution (what you call a "compiler/assembler" and I think of as a "builder"). 3. People who hack Python itself. Problem is that "developer" is very ambiguous in this context...

People who choose to compile their own Python should realize that they might disable or misconfigure some parts of it. I personally trust most people that assemble OS distributions to compile a proper Python binary + modules, but I think a HOWTO isn't a bad idea -- unless we autoconf everything.

I'd like to see both things happen (HOWTO and autoconfing) and am willing to work on both.

I apologize for the tone of my previous post, and the above snippet.

No offense taken at all, I assure you.

I'm not trying to block progress here ;) I'm actually all for autodetecting as much as possible, and more than willing to put effort into it as well (as long as it's deemed useful, and isn't supplanted by a distutils variant weeks later.) And I personally have my doubts about the distutils variant, too, but that's partly because I have little experience with distutils. If we can work out a deal where both autoconf and distutils are an option, I'm happy to write a few, if not all, autoconf tests for the currently disabled modules.

I admit I'm not very clear on the scope of what distutils is supposed to handle, and how. Perhaps amk can enlighten us?

So, Eric, let's split the work. I'll do Tkinter if you do curses. :)

You've got a deal. I'll start looking at the autoconf code. I've already got a fair idea how to do this. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> No one who's seen it in action can say the phrase "government help" without either laughing or crying.

Guido van Rossum <guido@digicool.com>:

Not all the world is Linux. CML2 isn't the only Python application that matters. Python world domination is not a goal. There is no Eric conspiracy! :-)

Perhaps I misunderstood you, then. I thought you considered CML2 an potentially important design win, and that was why curses didn't get dropped from the core. Have you changed your mind about this? If Python world domination is not a goal then I can only conclude that you haven't had your morning coffee yet :-). There's a more general question here about what it means for something to be in the core language. Developers need to have a clear, bright-line picture of what they can count on to be present. To me this implies that it's the job of the Python maintainers to make sure that a facility declared "core" by its presence in the standard library documentation is always present, for maximum "batteries are included" effect. Yes, dealing with cross-platform variations in linking curses is a pain -- but dealing with that kind of pain so the Python user doesn't have to is precisely our job. Or so I understand it, anyway. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Conservatism is the blind and fear-filled worship of dead radicals.

Guido van Rossum <guido@digicool.com>:

Supporting CML2 was one of the reasons to keep curses in the core, but not the only one. Linux kernel configuration is so far removed from my daily use of computers that I don't have a good way to judge its importance in the grand scheme of things. Since you obviously consider it very important, and since I generally trust your judgement (except on the issue of firearms :-), your plea for keeping, and improving, curses support in the Python core made a difference in my decision. And don't worry, I don't expect to change that decision -- though I personally still find it curious that curses is so important. I find curses-style user interfaces pretty pathetic, and wished that Linux migrated to a real GUI for configuration. (And the linuxconf approach does *not* qualify as a a real GUI. :-)

Thank you, that makes your priorities much clearer. Actually I agree with you that curses interfaces are mostly pretty pathetic. A lot of people still like them, though, because they tend to be fast and lightweight. Then, too, a really well-designed curses interface can in fact be good enough that the usability gain from GUIizing is marginal. My favorite examples of this are mutt and slrn. The fact that GUI programs have failed to make much headway against this is not simply due to user conservatism, it's genuinely hard to see how a GUI interface could be made significantly better. And unfortunately, there is a niche where it is still important to support curses interfacing independently of anyone's preferences in interface style -- early in the system-configuration process before one has bootstrapped to the point where X is reliably available. I hasten to add that this is not just *my* problem -- one of your more important Python constituencies in a practical sense is the guys who maintain Red Hat's installer.

I was totally serious though: my personal satisfaction doesn't come from Python world domination. Others seem have that goal, and if it doesn't inconvenience me too much I'll play along, but in the end I've got some goals in my personal life that are much more important.

There speaks the new husband :-). OK. So what *do* you want from Python? Personally, BTW, my goal is not exactly Python world domination either -- it's that the world should be dominated by the language that has the least tendency to produce grotty fragile code (remember that I tend to obsess about the software-quality problem :-)). Right now that's Python. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The people of the various provinces are strictly forbidden to have in their possession any swords, short swords, bows, spears, firearms, or other types of arms. The possession of unnecessary implements makes difficult the collection of taxes and dues and tends to foment uprisings. -- Toyotomi Hideyoshi, dictator of Japan, August 1588