I've written up a list of things that need to get done before 1.6 is finished. This is my vision of what needs to be done, and doesn't have an official stamp of approval from GvR or anyone else. So it's very probably wrong. http://starship.python.net/crew/amk/python/1.6-jobs.html Here's the list formatted as text. The major outstanding things at the moment seem to be sre and Distutils; once they go in, you could probably release an alpha, because the other items are relatively minor. Still to do * XXX Revamped import hooks (or is this a post-1.6 thing?) * Update the documentation to match 1.6 changes. * Document more undocumented modules * Unicode: Add Unicode support for open() on Windows * Unicode: Compress the size of unicodedatabase * Unicode: Write \N{SMILEY} codec for Unicode * Unicode: the various XXX items in Misc/unicode.txt * Add module: Distutils * Add module: Jim Ahlstrom's zipfile.py * Add module: PyExpat interface * Add module: mmapfile * Add module: sre * Drop cursesmodule and package it separately. (Any other obsolete modules that should go?) * Delete obsolete subdirectories in Demo/ directory * Refurbish Demo subdirectories to be properly documented, match modern coding style, etc. * Support Unicode strings in PyExpat interface * Fix ./ld_so_aix installation problem on AIX * Make test.regrtest.py more usable outside of the Python test suite * Conservative garbage collection of cycles (maybe?) * Write friendly "What's New in 1.6" document/article Done Nothing at the moment. After 1.7 * Rich comparisons * Revised coercions * Parallel for loop (for i in L; j in M: ...), * Extended slicing for all sequences. * GvR: "I've also been thinking about making classes be types (not as huge a change as you think, if you don't allow subclassing built-in types), and adding a built-in array type suitable for use by NumPy." --amk
Andrew Kuchling <akuchlin@mems-exchange.org>:
* Drop cursesmodule and package it separately. (Any other obsolete modules that should go?)
Annoyingly enough, I may need this to stay in, for use by the new Linux-kernel configuration system I'm writing. Why is it on the hit list? -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> Still, if you will not fight for the right when you can easily win without bloodshed, if you will not fight when your victory will be sure and not so costly, you may come to the moment when you will have to fight with all the odds against you and only a precarious chance for survival. There may be a worse case. You may have to fight when there is no chance of victory, because it is better to perish than to live as slaves. --Winston Churchill
AMK> I've written up a list of things that need to get done before 1.6 AMK> is finished. This is my vision of what needs to be done, and AMK> doesn't have an official stamp of approval from GvR or anyone else. AMK> So it's very probably wrong. Might I suggest moving robotparser.py from Tools/webchecker to Lib? Modules of general usefulness (this is at least generally useful for anyone writing web spiders ;-) shouldn't live in Tools, because it's not always available and users need to do extra work to make them available. I'd be happy to write up some documentation for it and twiddle the module to include doc strings. -- Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/
Deal. Soon as we get the docs we'll move it to Lib. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
Might I suggest moving robotparser.py from Tools/webchecker to Lib? Modules Deal. Soon as we get the docs we'll move it to Lib.
What about putting it in a package like 'www' or 'web'? Packagizing the existing library is hard because of backward compatibility, but there's no such constraint for new modules. -- A.M. Kuchling http://starship.python.net/crew/amk/ One need not be a chamber to be haunted; / One need not be a house; / The brain has corridors surpassing / Material place. -- Emily Dickinson, "Time and Eternity"
On Fri, 24 Mar 2000, Andrew M. Kuchling wrote:
Or in the "network" package that was suggested a month ago? And why *can't* we start on repackaging old module? I think the only reason that somebody came up with to NOT do it was "well, if we don't repackage the whole thing, then we should repackage nothing." Which, IMO, is totally bogus. We'll never get anywhere operating under that principle. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
Or in the "network" package that was suggested a month ago?
+1
That doesn't bother me, but I tend to be a little conservative (though usually not as conservative as Guido on such matters). I *would* like to decided theat 1.7 will be fully packagized, and not wait until 2.0. As long as 1.7 is a "testing the evolutionary path" release, I think that's the right thing to do. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
Greg Stein writes:
Or in the "network" package that was suggested a month ago?
[Fred]
+1
Which reminds me of another reason to wait: coming up with the right package hierarchy is hard. (E.g. I find network too long; plus, does htmllib belong there?)
Agreed. At the SD conference I gave a talk about the future of Python, and there was (again) a good suggestion about forwards compatibility. Starting with 1.7 (if not sooner), several Python 3000 features that necessarily have to be incompatible (like 1/2 yielding 0.5 instead of 0) could issue warnings when (or unless?) Python is invoked with a compatibility flag. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
htmllib does not go there. Where does it go? Dunno. Leave it unless/until somebody comes up with a place for it. We package up obvious ones. We don't have to design a complete hierarchy. There seemed to be a general "good feeling" around some kind of network (protocol) package. Call it "net" if "network" is too long. Cheers, -g -- Greg Stein, http://www.lyra.org/
One thing you can definitely do now which breaks no code: propose a package hierarchy for the standard library.
On Fri, 24 Mar 2000, Barry A. Warsaw wrote:
One thing you can definitely do now which breaks no code: propose a package hierarchy for the standard library.
I already did! http://www.python.org/pipermail/python-dev/2000-February/003761.html *grumble* -g -- Greg Stein, http://www.lyra.org/
You've got to be kidding. That's not a package hierarchy proposal, it's just one package (network). Without a comprehensive proposal I'm against a partial reorganization: without a destination we can't start marching. Naming things is very contentious -- everybody has an opinion. To pick the right names you must see things in perspective. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
Not kidding at all. I said before that I don't think we can do everything all at once. I *do* think this is solvable with a greedy algorithm rather than waiting for some nebulous completion point.
Naming things is very contentious -- everybody has an opinion. To pick the right names you must see things in perspective.
Sure. And those diverse opinions are why I don't believe it is possible to do all at once. The task is simply too large to tackle in one shot. IMO, it must be solved incrementally. I'm not even going to attempt to try to define a hierarchy for all those modules. I count 137 on my local system. Let's say that I *do* try... some are going to end up "forced" rather than obeying some obvious grouping. If you do it a chunk at a time, then you get the obvious, intuitive groupings. Try for more, and you just bung it all up. For discussion's sake: can you provide a rationale for doing it all at once? In the current scenario, modules just appear at some point. After a partial reorg, some modules appear at a different point. "No big whoop." Just because module A is in a package doesn't imply that module B must also be in a package. Cheers, -g -- Greg Stein, http://www.lyra.org/
[Greg writes]
I agree with Greg - every module will not fit into a package. But I also agree with Guido - we _should_ attempt to go through the 137 modules and put the ones that fit into logical groupings. Greg is probably correct with his selection for "net", but a general evaluation is still a good thing. A view of the bigger picture will help to quell debates over the structure, and only leave us with the squabbles over the exact spelling :-) +2 on ... err .... -1 on ... errr - awww - screw that-<grin>-ly, Mark.
On Sun, 26 Mar 2000, Mark Hammond wrote:
But I also agree with Guido - we _should_ attempt to go through the 137
Where did you come up with that number? I counted much more -- not quite sure, but certainly more. Well, here's a tentative suggestion I worked out today. This is just to have something to quibble about. In the interest of rushing it out of the door, there are a few modules (explicitly mentioned) which I have said nothing about. net httplib ftplib urllib cgi gopherlib imaplib poplib nntplib smptlib urlparse telnetlib server BaseHTTPServer CGIHTTPServer SimpleHTTPServer SocketServer asynchat asyncore text sgmllib htmllib htmlentitydefs xml whatever the xml-sig puts here mail rfc822 mime MimeWriter mimetools mimify mailcap mimetypes base64 quopri mailbox mhlib binhex parse string re regex reconvert regex_syntax regsub shlex ConfigParser linecache multifile netrc bin gzip zlib aifc chunk image imghdr colorsys imageop imgfile rgbimg yuvconvert sound sndhdr toaiff audiodev sunau sunaudio wave audioop sunaudiodev db anydbm whichdb bsddb dbm dbhash dumbdbm gdbm math bisect fpformat random whrandom cmath math crypt fpectl fpetest array md5 mpz rotor sha time calendar time tzparse sched timing interpreter new py_compile code codeop compileall keyword token tokenize parser dis bdb pdb profile pyclbr tabnanny symbol pstats traceback rlcompleter security Bastion rexec ihooks file dircache path -- a virtual module which would do a from <something>path import * dospath posixpath macpath nturl2path ntpath macurl2path filecmp fileinput StringIO cStringIO glob fnmatch posixfile stat statcache statvfs tempfile shutil pipes popen2 commands dl fcntl serialize pickle cPickle shelve xdrlib copy copy_reg threads thread threading Queue mutex ui curses Tkinter cmd getpass internal _codecs _locale _tkinter pcre strop posix users pwd grp nis exceptions os types UserDict UserList user site locale sgi al cd cl fl fm gl misc (what used to be sgimodule.c) sv unicode codecs unicodedata unicodedatabase ========== Modules not handled ============ formatter getopt pprint pty repr tty errno operator pure readline resource select signal socket struct syslog termios Well, if you got this far, you certainly deserve... congratualtions-ly y'rs, Z. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
This made me think of one issue which is worth considering -- is there a mechanism for third-party packages to hook into the standard naming hierarchy? It'd be weird not to have the oracle and sybase modules within the db toplevel package, for example. --david ascher
On Sat, 25 Mar 2000, David Ascher wrote:
My position is that any 3rd party module decides for itself where it wants to live -- once we formalized the framework. Consider PyGTK/PyGnome, PyQT/PyKDE -- they should live in the UI package too...
That sounds good in theory, but I can see possible problems down the line: 1) The current mapping between package names and directory structure means that installing a third party package hierarchy in a different place on disk than the standard library requires some work on the import mechanisms (this may have been discussed already) and a significant amount of user education. 2) We either need a 'registration' mechanism whereby people can claim a name in the standard hierarchy or expect conflicts. As far as I can gather, in the Perl world registration occurs by submission to CPAN. Correct? One alternative is to go the Java route, which would then mean, I think, that some core modules are placed very high in the hierarchy (the equivalent of the java. subtree), and some others are deprecated to lower subtree (the equivalent of com.sun). Anyway, I agree with Guido on this one -- naming is a contentious issue wrought with long-term implications. Let's not rush into a decision just yet. --david
On Sat, 25 Mar 2000, David Ascher wrote:
Ummmm.... 1.a) If the work of the import-sig produces something (which I suspect it will), it's more complicated -- you could have JAR-like files with hierarchies inside. 1.b) Installation is the domain of the distutils-sig. I seem to remember Greg Ward saying something about installing packages.
Yes. But this is no worse then the current situation, where people pick a toplevel name <wink>. I agree a registration mechanism would be helpful.
Personally, I *hate* the Java mechanism -- see Stallman's position on why GNU Java packages use gnu.* rather then org.gnu.* for some of the reasons. I really, really, like the Perl mechanism, and I think we would do well to think if something like that wouldn't suit us, with minor modifications. (Remember that lwall copied the Pythonic module mechanism, so Perl and Python modules are quite similar)
I agree. That's why I pushed out the straw-man proposal. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> Personally, I *hate* the Java mechanism -- see Stallman's MZ> position on why GNU Java packages use gnu.* rather then MZ> org.gnu.* for some of the reasons. Actually, it's Per Bothner's position: http://www.gnu.org/software/java/why-gnu-packages.txt and I agree with him. I kind of wished that JimH had chosen simply `python' as JPython's top level package heirarchy, but that's too late to change now. -Barry
The biggest modification which I think is needed to a Perl-like organization is that IMO there is value in knowing what packages are 'blessed' by Guido. In other words, some sort of Q/A mechanism would be good, if it can be kept simple. [Alternatively, let's not put a Q/A mechanism in place and my employer can make money selling that information, the way they do for Perl! =)]
(Remember that lwall copied the Pythonic module mechanism, so Perl and Python modules are quite similar)
That's stretching things a bit (the part after the 'so' doesn't follow from the part before), as there is a lot more to the nature of module systems, but the point is well taken. --david
On Sat, 25 Mar 2000, David Ascher wrote:
You got a point. Anyone knows how the perl-porters decide what modules to put in source.tar.gz? -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MH" == Mark Hammond <mhammond@skippinet.com.au> writes:
MH> [Greg writes]
MH> I agree with Greg - every module will not fit into a package. Sure. No one is arguing with that :-). Where I disagree with Greg, is that we shouldn't approach this piecemeal. A greedy algorithm can lead to a locally optimal solution that isn't the right for the whole library. A name or grouping might make sense on its own, but isn't sufficiently clear when taking all 137 odd modules into account. MH> But I also agree with Guido - we _should_ attempt to go through MH> the 137 modules and put the ones that fit into logical MH> groupings. Greg is probably correct with his selection for MH> "net", but a general evaluation is still a good thing. A view MH> of the bigger picture will help to quell debates over the MH> structure, and only leave us with the squabbles over the exact MH> spelling :-) x1.5 on this. I'm not sure which direction you ended up thinking this was (+ or -), but which ever direction it was I like it. Jeremy
I guess I was making a request for a more comprehensive list. People are asking to packagize the entire directory, so I'd like to know what organization they'd propose for all the modules. -Barry
Guido> Which reminds me of another reason to wait: coming up with the Guido> right package hierarchy is hard. (E.g. I find network too long; Guido> plus, does htmllib belong there?) Ah, another topic for python-dev. Even if we can't do the packaging right away, we should be able to hash out the structure. Skip
[Greg]
The reason is backwards compatibility. Assume we create a package "web" and move all web related modules into it: httplib, urllib, htmllib, etc. Now for backwards compatibility, we add the web directory to sys.path, so one can write either "import web.urllib" or "import urllib". But that loads the same code twice! And in this (carefully chosen :-) example, urllib actually has some state which shouldn't be replicated. Plus, it's too much work -- I'd rather focus on getting 1.6 out of the door, and there's a lot of other stuff I need to do besides moving modules around. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
We don't add it to the path. Instead, we create new modules that look like: ---- httplib.py ---- from web.httplib import * ---- The only backwards-compat issue with this approach is that people who poke values into the module will have problems. I don't believe that any of the modules were designed for that, anyhow, so it would seem an acceptable to (effectively) disallow that behavior.
Stuff that *you* need to do, sure. But there *are* a lot of us who can help here, and some who desire to spend their time moving modules. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
Stuff that *you* need to do, sure. But there *are* a lot of us who can help here, and some who desire to spend their time moving modules.
Would it make sense for one of these people with time on their hands to propose a specific mapping from old->new names? I think that would be a good first step, regardless of the implementation timing. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
Hm. Moving modules requires painful and arcane CVS manipulations that can only be done by the few of us here at CNRI -- and I'm the only one left who's full time on Python. I'm still not convinced that it's a good plan. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
Fair enough.
There are a number of ways to do this, and I'm familiar with all of them. It is a continuing point of strife in the Apache CVS repositories :-) But... it is premised on accepting the desire to move them, of course. Cheers, -g -- Greg Stein, http://www.lyra.org/
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> OK, that's reasonable. I'll have to invent a different GvR> reason why I don't want this -- because I really don't! Tim's Fifth Enlightenment is all the reason you'd need, /if/ you can't be persuaded to change your mind :) -Barry
On Fri, 24 Mar 2000, Barry A. Warsaw wrote:
Maybe i'm just a slave to my organization mania, but i'd suggest the following order change of 5 and 6, plus an addition; from: 5 now: Flat is better than nested. 6 now: Sparse is better than dense. to: 5 Sparse is better than dense. 6 Flat is better than nested 6.5 until it gets too dense. or-is-it-me-that-gets-too-dense'ly yrs, ken klm@digicool.com (And couldn't the humor page get hooked up a bit better? That was definitely a fun part of maintaining python.org...)
[Guido]
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
[Barry]
Tim's Fifth Enlightenment is all the reason you'd need, /if/ you can't be persuaded to change your mind :)
No no no no no: "namespaces are one honking great idea ..." is the controlling one here: Guido really *does* want this! It's a question of timing, in the sense of "never is often better than *right* now", but to be eventually modified by "now is better than never". These were carefully designed to support any position whatsoever, you know <wink>. although-in-any-particular-case-there's-only-one-true-interpretation-ly y'rs - tim
Guido wrote:
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
I'm glad this organize-the-library-in-packages initiative seems to be moving towards concentrating on the organization, rather than just starting to put obvious things in the obvious places. Personally, i *crave* sensible, discoverable organization. The only thing i like less than complicated disorganization is complicated misorganization - and i think that just diving in and doing the "obvious" placements would have the terrible effect of making it harder, not easier, to move eventually to the right arrangement. Ken klm@digicool.com
[GregS proposes a partial packaging of std modules for 1.6, Guido objects on spurious grounds, GregS refutes that, Guido agrees]
I'll have to invent a different reason why I don't want this -- because I really don't!
This one's easy! It's why I left the 20th of the 20 Pythonic Theses for you to fill in <wink>. All you have to do now is come up with a pithy way to say "if it's something Guido is so interested in that he wants to be deeply involved in it himself, but it comes at a time when he's buried under prior commitments, then tough tulips, it waits". shades-of-the-great-renaming-ly y'rs - tim
On Fri, 24 Mar 2000, Tim Peters wrote:
No need for Pythonic Theses. I don't see anybody disagreeing with the end goal. The issue comes up with *how* to get there. I say "do it incrementally" while others say "do it all at once." Personally, I don't think it is possible to do all at once. As a corollary, if you can't do it all at once, but you *require* that it be done all at once, then you have effectively deferred the problem. To put it another way, Guido has already invented a reason to not do it: he just requires that it be done all at once. Result: it won't be done. [ not saying this was Guido's intent or desire... but this is how I read the result :-) ] Cheers, -g -- Greg Stein, http://www.lyra.org/
Bullshit, Greg. (I don't normally like to use such strong words, but since you're being confrontational here...) I'm all for doing it incrementally -- but I want the plan for how to do it made up front. That doesn't require all the details to be worked out -- but it requires a general idea about what kind of things we will have in the namespace and what kinds of names they get. An organizing principle, if you like. If we were to decide later that we go for a Java-like deep hierarchy, the network package would have to be moved around again -- what a waste. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Sat, 25 Mar 2000, Guido van Rossum wrote:
Fair enough, and point accepted. Sorry. I will say, tho, that you've taken this slightly out of context. The next paragraph explicitly stated that I don't believe you had this intent. I just felt that coming up with a complete plan before doing anything would be prone to failure. You asked to invent a new reason :-), so I said you had one already :-) Confrontational? Yes, guilty as charged. I was a bit frustrated.
All righty. So I think there is probably a single question that I have here: Moshe posted a large breakdown of how things could be packaged. He and Ping traded a number of comments, and more will be coming as soon as people wake up :-) However, if you are only looking for a "general idea", then should python-dev'ers nit pick the individual modules, or just examine the general breakdown and hierarchy? thx, -g -- Greg Stein, http://www.lyra.org/
On Sun, 26 Mar 2000, Greg Stein wrote:
Just a general comment -- it's so much fun to live in a different zone then all of you guys. just-wasting-time-ly y'rs, Z. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
On Fri, 24 Mar 2000, Guido van Rossum wrote:
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
Here's a reason: there shouldn't be changes we'll retract later -- we need to come up with the (more or less) right hierarchy the first time, or we'll do a lot of work for nothing.
Hmmmmm....this is a big problem. Maybe we need to have more people with access to the CVS? -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> Hmmmmm....this is a big problem. Maybe we need to have more MZ> people with access to the CVS? To make changes like this, you don't just need write access to CVS, you need physical access to the repository filesystem. It's not possible to provide this access to non-CNRI'ers. -Barry
On Sat, 25 Mar 2000, Barry A. Warsaw wrote:
Unless the CVS repository was moved to, say, SourceForge. :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
On Sat, 25 Mar 2000 bwarsaw@cnri.reston.va.us wrote:
Me neither, ergo the smiley :-) Just felt inclined to mention it, and I think the conversation stopped last time at that point; not sure it ever was "hashed" :-). But it is only a discussion to raise if checkins-via-CNRI-guys becomes a true bottleneck. Which it hasn't and doesn't look to be. Constrained? Yes. Bottleneck? No. Cheers, -g -- Greg Stein, http://www.lyra.org/
On Fri, 24 Mar 2000, Greg Stein wrote:
I just want to empahisize this point some more. Python 1.6 has a defined timeline, with a defined set of minimal requirements. However! I don't believe that a corollary of that says we MUST ignore everything else. If those other options fit within the required timeline, then why not? (assuming we have adequate testing and doc to go with the changes) There are ample people who have time and inclination to contribute. If those contributions add positive benefit, then I see no reason to exclude them (other than on pure merit, of course). Note that some of the problems stem from CVS access. Much Guido-time could be saved by a commit-then-review model, rather than review-then-Guido- commits model. Fred does this very well with the Doc/ area. Cheers, -g -- Greg Stein, http://www.lyra.org/
Actually, I'm experimenting with this already: Unicode, list.append() and socket.connect() are done in this way! For renames it is really painful though, even if someone else at CNRI can do it. I'd like to see a draft package hierarchy please? Also, if you have some time, please review the bugs in the bugs list. Patches submitted with a corresponding PR# will be treated with priority! --Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Stein writes:
Note that some of the problems stem from CVS access. Much Guido-time could be saved by a commit-then-review model, rather than review-then-Guido-
This is a non-problem; I'm willing to do the arcane CVS manipulations if the issue is Guido's time. What I will *not* do is do it piecemeal without a cohesive plan that Guido approves of at least 95%, and I'll be really careful to do that last 5% when he's not in the office. ;)
commits model. Fred does this very well with the Doc/ area.
Thanks for the vote of confidence! The model that I use for the Doc/ area is more like "Fred reviews, Fred commits, and Guido can read it on python.org like everyone else." Works for me! ;) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On Fri, 24 Mar 2000, Skip Montanaro wrote:
You're right, but I'd like this to be a 1.7 change. It's just that I plan to suggest a great-renaming-fest for 1.7 modules, and then namespace wouldn't be cluttered when you don't need it. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
Andrew Kuchling <akuchlin@mems-exchange.org>:
* Drop cursesmodule and package it separately. (Any other obsolete modules that should go?)
Annoyingly enough, I may need this to stay in, for use by the new Linux-kernel configuration system I'm writing. Why is it on the hit list? -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> Still, if you will not fight for the right when you can easily win without bloodshed, if you will not fight when your victory will be sure and not so costly, you may come to the moment when you will have to fight with all the odds against you and only a precarious chance for survival. There may be a worse case. You may have to fight when there is no chance of victory, because it is better to perish than to live as slaves. --Winston Churchill
AMK> I've written up a list of things that need to get done before 1.6 AMK> is finished. This is my vision of what needs to be done, and AMK> doesn't have an official stamp of approval from GvR or anyone else. AMK> So it's very probably wrong. Might I suggest moving robotparser.py from Tools/webchecker to Lib? Modules of general usefulness (this is at least generally useful for anyone writing web spiders ;-) shouldn't live in Tools, because it's not always available and users need to do extra work to make them available. I'd be happy to write up some documentation for it and twiddle the module to include doc strings. -- Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/
Deal. Soon as we get the docs we'll move it to Lib. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
Might I suggest moving robotparser.py from Tools/webchecker to Lib? Modules Deal. Soon as we get the docs we'll move it to Lib.
What about putting it in a package like 'www' or 'web'? Packagizing the existing library is hard because of backward compatibility, but there's no such constraint for new modules. -- A.M. Kuchling http://starship.python.net/crew/amk/ One need not be a chamber to be haunted; / One need not be a house; / The brain has corridors surpassing / Material place. -- Emily Dickinson, "Time and Eternity"
On Fri, 24 Mar 2000, Andrew M. Kuchling wrote:
Or in the "network" package that was suggested a month ago? And why *can't* we start on repackaging old module? I think the only reason that somebody came up with to NOT do it was "well, if we don't repackage the whole thing, then we should repackage nothing." Which, IMO, is totally bogus. We'll never get anywhere operating under that principle. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
Or in the "network" package that was suggested a month ago?
+1
That doesn't bother me, but I tend to be a little conservative (though usually not as conservative as Guido on such matters). I *would* like to decided theat 1.7 will be fully packagized, and not wait until 2.0. As long as 1.7 is a "testing the evolutionary path" release, I think that's the right thing to do. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
Greg Stein writes:
Or in the "network" package that was suggested a month ago?
[Fred]
+1
Which reminds me of another reason to wait: coming up with the right package hierarchy is hard. (E.g. I find network too long; plus, does htmllib belong there?)
Agreed. At the SD conference I gave a talk about the future of Python, and there was (again) a good suggestion about forwards compatibility. Starting with 1.7 (if not sooner), several Python 3000 features that necessarily have to be incompatible (like 1/2 yielding 0.5 instead of 0) could issue warnings when (or unless?) Python is invoked with a compatibility flag. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
htmllib does not go there. Where does it go? Dunno. Leave it unless/until somebody comes up with a place for it. We package up obvious ones. We don't have to design a complete hierarchy. There seemed to be a general "good feeling" around some kind of network (protocol) package. Call it "net" if "network" is too long. Cheers, -g -- Greg Stein, http://www.lyra.org/
One thing you can definitely do now which breaks no code: propose a package hierarchy for the standard library.
On Fri, 24 Mar 2000, Barry A. Warsaw wrote:
One thing you can definitely do now which breaks no code: propose a package hierarchy for the standard library.
I already did! http://www.python.org/pipermail/python-dev/2000-February/003761.html *grumble* -g -- Greg Stein, http://www.lyra.org/
You've got to be kidding. That's not a package hierarchy proposal, it's just one package (network). Without a comprehensive proposal I'm against a partial reorganization: without a destination we can't start marching. Naming things is very contentious -- everybody has an opinion. To pick the right names you must see things in perspective. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
Not kidding at all. I said before that I don't think we can do everything all at once. I *do* think this is solvable with a greedy algorithm rather than waiting for some nebulous completion point.
Naming things is very contentious -- everybody has an opinion. To pick the right names you must see things in perspective.
Sure. And those diverse opinions are why I don't believe it is possible to do all at once. The task is simply too large to tackle in one shot. IMO, it must be solved incrementally. I'm not even going to attempt to try to define a hierarchy for all those modules. I count 137 on my local system. Let's say that I *do* try... some are going to end up "forced" rather than obeying some obvious grouping. If you do it a chunk at a time, then you get the obvious, intuitive groupings. Try for more, and you just bung it all up. For discussion's sake: can you provide a rationale for doing it all at once? In the current scenario, modules just appear at some point. After a partial reorg, some modules appear at a different point. "No big whoop." Just because module A is in a package doesn't imply that module B must also be in a package. Cheers, -g -- Greg Stein, http://www.lyra.org/
[Greg writes]
I agree with Greg - every module will not fit into a package. But I also agree with Guido - we _should_ attempt to go through the 137 modules and put the ones that fit into logical groupings. Greg is probably correct with his selection for "net", but a general evaluation is still a good thing. A view of the bigger picture will help to quell debates over the structure, and only leave us with the squabbles over the exact spelling :-) +2 on ... err .... -1 on ... errr - awww - screw that-<grin>-ly, Mark.
On Sun, 26 Mar 2000, Mark Hammond wrote:
But I also agree with Guido - we _should_ attempt to go through the 137
Where did you come up with that number? I counted much more -- not quite sure, but certainly more. Well, here's a tentative suggestion I worked out today. This is just to have something to quibble about. In the interest of rushing it out of the door, there are a few modules (explicitly mentioned) which I have said nothing about. net httplib ftplib urllib cgi gopherlib imaplib poplib nntplib smptlib urlparse telnetlib server BaseHTTPServer CGIHTTPServer SimpleHTTPServer SocketServer asynchat asyncore text sgmllib htmllib htmlentitydefs xml whatever the xml-sig puts here mail rfc822 mime MimeWriter mimetools mimify mailcap mimetypes base64 quopri mailbox mhlib binhex parse string re regex reconvert regex_syntax regsub shlex ConfigParser linecache multifile netrc bin gzip zlib aifc chunk image imghdr colorsys imageop imgfile rgbimg yuvconvert sound sndhdr toaiff audiodev sunau sunaudio wave audioop sunaudiodev db anydbm whichdb bsddb dbm dbhash dumbdbm gdbm math bisect fpformat random whrandom cmath math crypt fpectl fpetest array md5 mpz rotor sha time calendar time tzparse sched timing interpreter new py_compile code codeop compileall keyword token tokenize parser dis bdb pdb profile pyclbr tabnanny symbol pstats traceback rlcompleter security Bastion rexec ihooks file dircache path -- a virtual module which would do a from <something>path import * dospath posixpath macpath nturl2path ntpath macurl2path filecmp fileinput StringIO cStringIO glob fnmatch posixfile stat statcache statvfs tempfile shutil pipes popen2 commands dl fcntl serialize pickle cPickle shelve xdrlib copy copy_reg threads thread threading Queue mutex ui curses Tkinter cmd getpass internal _codecs _locale _tkinter pcre strop posix users pwd grp nis exceptions os types UserDict UserList user site locale sgi al cd cl fl fm gl misc (what used to be sgimodule.c) sv unicode codecs unicodedata unicodedatabase ========== Modules not handled ============ formatter getopt pprint pty repr tty errno operator pure readline resource select signal socket struct syslog termios Well, if you got this far, you certainly deserve... congratualtions-ly y'rs, Z. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
This made me think of one issue which is worth considering -- is there a mechanism for third-party packages to hook into the standard naming hierarchy? It'd be weird not to have the oracle and sybase modules within the db toplevel package, for example. --david ascher
On Sat, 25 Mar 2000, David Ascher wrote:
My position is that any 3rd party module decides for itself where it wants to live -- once we formalized the framework. Consider PyGTK/PyGnome, PyQT/PyKDE -- they should live in the UI package too...
That sounds good in theory, but I can see possible problems down the line: 1) The current mapping between package names and directory structure means that installing a third party package hierarchy in a different place on disk than the standard library requires some work on the import mechanisms (this may have been discussed already) and a significant amount of user education. 2) We either need a 'registration' mechanism whereby people can claim a name in the standard hierarchy or expect conflicts. As far as I can gather, in the Perl world registration occurs by submission to CPAN. Correct? One alternative is to go the Java route, which would then mean, I think, that some core modules are placed very high in the hierarchy (the equivalent of the java. subtree), and some others are deprecated to lower subtree (the equivalent of com.sun). Anyway, I agree with Guido on this one -- naming is a contentious issue wrought with long-term implications. Let's not rush into a decision just yet. --david
On Sat, 25 Mar 2000, David Ascher wrote:
Ummmm.... 1.a) If the work of the import-sig produces something (which I suspect it will), it's more complicated -- you could have JAR-like files with hierarchies inside. 1.b) Installation is the domain of the distutils-sig. I seem to remember Greg Ward saying something about installing packages.
Yes. But this is no worse then the current situation, where people pick a toplevel name <wink>. I agree a registration mechanism would be helpful.
Personally, I *hate* the Java mechanism -- see Stallman's position on why GNU Java packages use gnu.* rather then org.gnu.* for some of the reasons. I really, really, like the Perl mechanism, and I think we would do well to think if something like that wouldn't suit us, with minor modifications. (Remember that lwall copied the Pythonic module mechanism, so Perl and Python modules are quite similar)
I agree. That's why I pushed out the straw-man proposal. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> Personally, I *hate* the Java mechanism -- see Stallman's MZ> position on why GNU Java packages use gnu.* rather then MZ> org.gnu.* for some of the reasons. Actually, it's Per Bothner's position: http://www.gnu.org/software/java/why-gnu-packages.txt and I agree with him. I kind of wished that JimH had chosen simply `python' as JPython's top level package heirarchy, but that's too late to change now. -Barry
The biggest modification which I think is needed to a Perl-like organization is that IMO there is value in knowing what packages are 'blessed' by Guido. In other words, some sort of Q/A mechanism would be good, if it can be kept simple. [Alternatively, let's not put a Q/A mechanism in place and my employer can make money selling that information, the way they do for Perl! =)]
(Remember that lwall copied the Pythonic module mechanism, so Perl and Python modules are quite similar)
That's stretching things a bit (the part after the 'so' doesn't follow from the part before), as there is a lot more to the nature of module systems, but the point is well taken. --david
On Sat, 25 Mar 2000, David Ascher wrote:
You got a point. Anyone knows how the perl-porters decide what modules to put in source.tar.gz? -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MH" == Mark Hammond <mhammond@skippinet.com.au> writes:
MH> [Greg writes]
MH> I agree with Greg - every module will not fit into a package. Sure. No one is arguing with that :-). Where I disagree with Greg, is that we shouldn't approach this piecemeal. A greedy algorithm can lead to a locally optimal solution that isn't the right for the whole library. A name or grouping might make sense on its own, but isn't sufficiently clear when taking all 137 odd modules into account. MH> But I also agree with Guido - we _should_ attempt to go through MH> the 137 modules and put the ones that fit into logical MH> groupings. Greg is probably correct with his selection for MH> "net", but a general evaluation is still a good thing. A view MH> of the bigger picture will help to quell debates over the MH> structure, and only leave us with the squabbles over the exact MH> spelling :-) x1.5 on this. I'm not sure which direction you ended up thinking this was (+ or -), but which ever direction it was I like it. Jeremy
I guess I was making a request for a more comprehensive list. People are asking to packagize the entire directory, so I'd like to know what organization they'd propose for all the modules. -Barry
Guido> Which reminds me of another reason to wait: coming up with the Guido> right package hierarchy is hard. (E.g. I find network too long; Guido> plus, does htmllib belong there?) Ah, another topic for python-dev. Even if we can't do the packaging right away, we should be able to hash out the structure. Skip
[Greg]
The reason is backwards compatibility. Assume we create a package "web" and move all web related modules into it: httplib, urllib, htmllib, etc. Now for backwards compatibility, we add the web directory to sys.path, so one can write either "import web.urllib" or "import urllib". But that loads the same code twice! And in this (carefully chosen :-) example, urllib actually has some state which shouldn't be replicated. Plus, it's too much work -- I'd rather focus on getting 1.6 out of the door, and there's a lot of other stuff I need to do besides moving modules around. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
We don't add it to the path. Instead, we create new modules that look like: ---- httplib.py ---- from web.httplib import * ---- The only backwards-compat issue with this approach is that people who poke values into the module will have problems. I don't believe that any of the modules were designed for that, anyhow, so it would seem an acceptable to (effectively) disallow that behavior.
Stuff that *you* need to do, sure. But there *are* a lot of us who can help here, and some who desire to spend their time moving modules. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
Stuff that *you* need to do, sure. But there *are* a lot of us who can help here, and some who desire to spend their time moving modules.
Would it make sense for one of these people with time on their hands to propose a specific mapping from old->new names? I think that would be a good first step, regardless of the implementation timing. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
Hm. Moving modules requires painful and arcane CVS manipulations that can only be done by the few of us here at CNRI -- and I'm the only one left who's full time on Python. I'm still not convinced that it's a good plan. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 24 Mar 2000, Guido van Rossum wrote:
Fair enough.
There are a number of ways to do this, and I'm familiar with all of them. It is a continuing point of strife in the Apache CVS repositories :-) But... it is premised on accepting the desire to move them, of course. Cheers, -g -- Greg Stein, http://www.lyra.org/
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> OK, that's reasonable. I'll have to invent a different GvR> reason why I don't want this -- because I really don't! Tim's Fifth Enlightenment is all the reason you'd need, /if/ you can't be persuaded to change your mind :) -Barry
On Fri, 24 Mar 2000, Barry A. Warsaw wrote:
Maybe i'm just a slave to my organization mania, but i'd suggest the following order change of 5 and 6, plus an addition; from: 5 now: Flat is better than nested. 6 now: Sparse is better than dense. to: 5 Sparse is better than dense. 6 Flat is better than nested 6.5 until it gets too dense. or-is-it-me-that-gets-too-dense'ly yrs, ken klm@digicool.com (And couldn't the humor page get hooked up a bit better? That was definitely a fun part of maintaining python.org...)
[Guido]
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
[Barry]
Tim's Fifth Enlightenment is all the reason you'd need, /if/ you can't be persuaded to change your mind :)
No no no no no: "namespaces are one honking great idea ..." is the controlling one here: Guido really *does* want this! It's a question of timing, in the sense of "never is often better than *right* now", but to be eventually modified by "now is better than never". These were carefully designed to support any position whatsoever, you know <wink>. although-in-any-particular-case-there's-only-one-true-interpretation-ly y'rs - tim
Guido wrote:
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
I'm glad this organize-the-library-in-packages initiative seems to be moving towards concentrating on the organization, rather than just starting to put obvious things in the obvious places. Personally, i *crave* sensible, discoverable organization. The only thing i like less than complicated disorganization is complicated misorganization - and i think that just diving in and doing the "obvious" placements would have the terrible effect of making it harder, not easier, to move eventually to the right arrangement. Ken klm@digicool.com
[GregS proposes a partial packaging of std modules for 1.6, Guido objects on spurious grounds, GregS refutes that, Guido agrees]
I'll have to invent a different reason why I don't want this -- because I really don't!
This one's easy! It's why I left the 20th of the 20 Pythonic Theses for you to fill in <wink>. All you have to do now is come up with a pithy way to say "if it's something Guido is so interested in that he wants to be deeply involved in it himself, but it comes at a time when he's buried under prior commitments, then tough tulips, it waits". shades-of-the-great-renaming-ly y'rs - tim
On Fri, 24 Mar 2000, Tim Peters wrote:
No need for Pythonic Theses. I don't see anybody disagreeing with the end goal. The issue comes up with *how* to get there. I say "do it incrementally" while others say "do it all at once." Personally, I don't think it is possible to do all at once. As a corollary, if you can't do it all at once, but you *require* that it be done all at once, then you have effectively deferred the problem. To put it another way, Guido has already invented a reason to not do it: he just requires that it be done all at once. Result: it won't be done. [ not saying this was Guido's intent or desire... but this is how I read the result :-) ] Cheers, -g -- Greg Stein, http://www.lyra.org/
Bullshit, Greg. (I don't normally like to use such strong words, but since you're being confrontational here...) I'm all for doing it incrementally -- but I want the plan for how to do it made up front. That doesn't require all the details to be worked out -- but it requires a general idea about what kind of things we will have in the namespace and what kinds of names they get. An organizing principle, if you like. If we were to decide later that we go for a Java-like deep hierarchy, the network package would have to be moved around again -- what a waste. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Sat, 25 Mar 2000, Guido van Rossum wrote:
Fair enough, and point accepted. Sorry. I will say, tho, that you've taken this slightly out of context. The next paragraph explicitly stated that I don't believe you had this intent. I just felt that coming up with a complete plan before doing anything would be prone to failure. You asked to invent a new reason :-), so I said you had one already :-) Confrontational? Yes, guilty as charged. I was a bit frustrated.
All righty. So I think there is probably a single question that I have here: Moshe posted a large breakdown of how things could be packaged. He and Ping traded a number of comments, and more will be coming as soon as people wake up :-) However, if you are only looking for a "general idea", then should python-dev'ers nit pick the individual modules, or just examine the general breakdown and hierarchy? thx, -g -- Greg Stein, http://www.lyra.org/
On Sun, 26 Mar 2000, Greg Stein wrote:
Just a general comment -- it's so much fun to live in a different zone then all of you guys. just-wasting-time-ly y'rs, Z. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
On Fri, 24 Mar 2000, Guido van Rossum wrote:
OK, that's reasonable. I'll have to invent a different reason why I don't want this -- because I really don't!
Here's a reason: there shouldn't be changes we'll retract later -- we need to come up with the (more or less) right hierarchy the first time, or we'll do a lot of work for nothing.
Hmmmmm....this is a big problem. Maybe we need to have more people with access to the CVS? -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> Hmmmmm....this is a big problem. Maybe we need to have more MZ> people with access to the CVS? To make changes like this, you don't just need write access to CVS, you need physical access to the repository filesystem. It's not possible to provide this access to non-CNRI'ers. -Barry
On Sat, 25 Mar 2000, Barry A. Warsaw wrote:
Unless the CVS repository was moved to, say, SourceForge. :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
On Sat, 25 Mar 2000 bwarsaw@cnri.reston.va.us wrote:
Me neither, ergo the smiley :-) Just felt inclined to mention it, and I think the conversation stopped last time at that point; not sure it ever was "hashed" :-). But it is only a discussion to raise if checkins-via-CNRI-guys becomes a true bottleneck. Which it hasn't and doesn't look to be. Constrained? Yes. Bottleneck? No. Cheers, -g -- Greg Stein, http://www.lyra.org/
On Fri, 24 Mar 2000, Greg Stein wrote:
I just want to empahisize this point some more. Python 1.6 has a defined timeline, with a defined set of minimal requirements. However! I don't believe that a corollary of that says we MUST ignore everything else. If those other options fit within the required timeline, then why not? (assuming we have adequate testing and doc to go with the changes) There are ample people who have time and inclination to contribute. If those contributions add positive benefit, then I see no reason to exclude them (other than on pure merit, of course). Note that some of the problems stem from CVS access. Much Guido-time could be saved by a commit-then-review model, rather than review-then-Guido- commits model. Fred does this very well with the Doc/ area. Cheers, -g -- Greg Stein, http://www.lyra.org/
Actually, I'm experimenting with this already: Unicode, list.append() and socket.connect() are done in this way! For renames it is really painful though, even if someone else at CNRI can do it. I'd like to see a draft package hierarchy please? Also, if you have some time, please review the bugs in the bugs list. Patches submitted with a corresponding PR# will be treated with priority! --Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Stein writes:
Note that some of the problems stem from CVS access. Much Guido-time could be saved by a commit-then-review model, rather than review-then-Guido-
This is a non-problem; I'm willing to do the arcane CVS manipulations if the issue is Guido's time. What I will *not* do is do it piecemeal without a cohesive plan that Guido approves of at least 95%, and I'll be really careful to do that last 5% when he's not in the office. ;)
commits model. Fred does this very well with the Doc/ area.
Thanks for the vote of confidence! The model that I use for the Doc/ area is more like "Fred reviews, Fred commits, and Guido can read it on python.org like everyone else." Works for me! ;) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On Fri, 24 Mar 2000, Skip Montanaro wrote:
You're right, but I'd like this to be a 1.7 change. It's just that I plan to suggest a great-renaming-fest for 1.7 modules, and then namespace wouldn't be cluttered when you don't need it. -- Moshe Zadka <mzadka@geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com
participants (15)
-
Andrew Kuchling -
Andrew M. Kuchling
-
Barry A. Warsaw -
bwarsaw＠cnri.reston.va.us
-
David Ascher -
Eric S. Raymond -
Fred L. Drake, Jr. -
Greg Stein -
Guido van Rossum -
Jeremy Hylton -
Ken Manheimer -
Mark Hammond -
Moshe Zadka -
Skip Montanaro -
Tim Peters