Mailman 3 December 2002 - Python-Dev

Re: [Python-Dev] make install fails with current Python CVS
by Michael Hudson 12 Dec '02

12 Dec '02

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > Martin v. L=F6wis wrote: > >>>>>Sorry: MemoryError: > >> > >>Could this error message be made more helpful :-) > >=20 > > I don't know. I tried, and failed. I did not try too hard, though. > >=20 > > I just knew what the problem was as I investigated this a couple of > > times before, to track it down to the codecs lookup facility not > > working. > > Could you tell me where this error is generated ? I imagine it's somewhere inside PyUnicode_AsEncodedString which is called from compile.c:parsestr. If you want to expose the latter to Python while you're at it, be my guest (sf feature request 617979). Cheers, M. -- I've even been known to get Marmite *near* my mouth -- but never actually in it yet. Vegamite is right out. UnicodeError: ASCII unpalatable error: vegamite found, ham expected -- Tim Peters, comp.lang.python

1 0

make install fails with current Python CVS
by M.-A. Lemburg 12 Dec '02

12 Dec '02

I've tried to run a "make install" with todays CVS version, but this fails on my Linux box with the following error: Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ... Sorry: MemoryError: Is that just my setup or reproducable on other machines as well ? The above file looks pretty innocent to me: #! -*- coding: koi8-r -*- assert u"<russian for python>".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' assert u"\<russian p>".encode("utf-8") == '\\\xd0\x9f' -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

3 6

os.path.walk generator
by Edward C. Jones 12 Dec '02

12 Dec '02

I feel that "The Right Way" to walk through a directory tree is with a generator. Therefore I suggest that a generator walk2(topdir, hidden=None) be added to os.path. It yields all the files and directories under (and including) "topdir". If "hidden" is true, then hidden files and directories are included. Thanks, Ed Jones

4 3

Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
by Greg Ward 12 Dec '02

12 Dec '02

On 11 December 2002, Just van Rossum said: > This checkin makes textwrap fail for me when running python -S ("don't run > site.py"): > > Python 2.3a0 (#42, Dec 11 2002, 10:33:42) > [GCC 3.1 20020420 (prerelease)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import textwrap > Traceback (most recent call last): > File "<stdin>", line 1, in ? > File "/Users/just/code/python_cvs/Lib/textwrap.py", line 15, in ? > class TextWrapper: > File "/Users/just/code/python_cvs/Lib/textwrap.py", line 56, in TextWrapper > unicode_whitespace_trans[ord(unicode(c))] = ord(u' ') > UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 0: ordinal > not in range(128) > >>> Weird. What the heck is 0xa0 doing in string.whitespace? Oh, waitasec, 0xa0 is non-breaking-space in ISO-8859-1. But no, string.whitespace is hardcoded to the US-ASCII whitespace chars, all of which are in range(128) -- your locale shouldn't matter. Just, can you run the following commands and tell me what you get: python -c "import string; print [hex(ord(c)) for c in string.whitespace]" python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]" LANG=C python -c "import string; print [hex(ord(c)) for c in string.whitespace]" LANG=C python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]" (On my Linux box where LANG=en_CA normally, I always get the six US-ASCII whitespace chars.) Greg -- Greg Ward <gward(a)python.net> http://www.gerg.ca/ This quote intentionally left blank.

5 9

Re: [Python-Dev] make install fails with current Python CVS
by Michael Hudson 12 Dec '02

12 Dec '02

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > Michael Hudson wrote: > > "M.-A. Lemburg" <mal(a)lemburg.com> writes: > > > > > >>Martin v. L=F6wis wrote: > >> > >>>>I've tried to run a "make install" with todays CVS version, but > >>>>this fails on my Linux box with the following error: > > > > [...] > > > >>Looking at the output of ./python -vv it appears as if the struct > >>cannot be loaded (which is imported by codecs.py). > > > > > > Is this an out-of-tree build? > > This is a CVS checkout from today. I then tried the usual > configure; make install dance and the byte-code compile > stopped with the error I posted. Oh. Then maybe it's something else. > > I have a feeling you may be being bitten by their current breakage. > > It's still broken ? I though you cleared that problem out > last week ? No, I think it's still broken. I handed the problem off to AMK -- don't /think/ he's fixed it yet... Cheers, M. -- My hat is lined with tinfoil for protection in the unlikely event that the droid gets his PowerPoint presentation working. -- Alan W. Frame, alt.sysadmin.recovery

3 4

Re: [Python-Dev] make install fails with current Python CVS
by Michael Hudson 12 Dec '02

12 Dec '02

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > Martin v. L=F6wis wrote: > >>I've tried to run a "make install" with todays CVS version, but > >>this fails on my Linux box with the following error: [...] > Looking at the output of ./python -vv it appears as if the struct > cannot be loaded (which is imported by codecs.py). Is this an out-of-tree build? I have a feeling you may be being bitten by their current breakage. Weird symptom, though. Cheers, M. -- 59. In English every word can be verbed. Would that it were so in our programming languages. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

2 1

Re: [Python-Dev] make install fails with current Python CVS
by Michael Hudson 12 Dec '02

12 Dec '02

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > I've tried to run a "make install" with todays CVS version, but > this fails on my Linux box with the following error: > > Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ... > Sorry: MemoryError: > > Is that just my setup or reproducable on other machines as well ? I saw that too, but thought it must have been my fault. > The above file looks pretty innocent to me: > > #! -*- coding: koi8-r -*- > assert u"<russian for python>".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' > assert u"\<russian p>".encode("utf-8") == '\\\xd0\x9f' It might well be the only file that actually uses the pep263 stuff to affect the interpretation of a unicode string. That would seem to be the place to start looking -- though it imports & compiles file when run as part of the test suite... Hmm... >>> exec open("../Lib/test/test_pep263.py").read() sigh, that was one easy guess. Cheers, M. -- Every now and then, Google doesn't throw up what I need so I start checking Altavista, Yahoo, etc. In almost every single case, I am brutally reminded why I use Google in the first place. -- John Riddoch, asr

1 0

RE: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11
by Delaney, Timothy 12 Dec '02

12 Dec '02

> From: M.-A. Lemburg [mailto:mal@lemburg.com] > > I remember Tim opposing such changes (and Greg Stein insisting > on them ;-). AFAIK, this breaks third party code since some > compilers simply reject casting char* to const char*. I must admit I caused this recent constifying mess ... someone complained on python-list and I told him to submit a bug report (in the hope that it would get rejected because I knew the problems it would cause). I submit myself for a spanking. Tim Delaney

8 11

trace.py in std library?
by Jeremy Hylton 11 Dec '02

11 Dec '02

There's a trace.py script sitting in Tools/scripts that works when imported. I used it today to add code coverage to Zope's test runner script. It took about 30 min. (mostly fixing bugs in trace.py) and seems to work pretty well. Why don't we put this script in the std library so it can actually be imported? I can guess at some of the reasons. The code looks very complicated and has been updated piecemeal by many people of the years. The last update has a bunch of comments with Zooko's name on them and a date. They seem like XXX comments. It might be nice to clean it up before putting in the library, but I'd be in favor of having some minimally functional tool in the library regardless. Jeremy

4 4

RE: [Python-Dev] trace.py in std library?
by Delaney, Timothy 11 Dec '02

11 Dec '02

> From: Jeremy Hylton [mailto:jeremy@alum.mit.edu] > > >>>>> "TD" == Timothy Delaney <Delaney> writes: > > TD> But I don't think it's a candidate yet. Among other things, the > TD> tracing functionality is not hooked into each new thread. Not an > TD> issue for my current project (which must run single-threaded due > TD> to being invoked inside something that must run in a single > TD> thread) but it is an issue for a version in the standard > TD> library. > > Maybe you should checkin what you have now. If it's at least as > functional as trace.py, then it's an improvement over what we've got > in the std library now. You/we can improve it as 2.3 development > proceeds. Ooh - believe me - you don't want to see the code as it is now. Yuk! Needs some major cleanup (esp. after the hacking I've done to make it work with Jython). No way I would check something that disgusting in where other people could see it ;) Unfortunately, I'm really busy right now. Maybe I'll get to spend some time on it over my 2-week break coming up. Plus as I said, I need to check with my employer - it has been partially developed on company time, and for a company project. Maybe I can get it into my goals and objectives ;) I should use the sandbox though. Unfortunately, I can't compile 2.3 at home ... MinGW still doesn't work, and I don't have a BSD or Linux box set up (the only available machine is currently serving as my internet gateway until I fix another machine). Tim Delaney

2 1

RE: [Python-Dev] trace.py in std library?
by Delaney, Timothy 11 Dec '02

11 Dec '02

> From: barry(a)python.org [mailto:barry@python.org] > > >>>>> "JH" == Jeremy Hylton <jeremy(a)alum.mit.edu> writes: > > JH> trace.py is Skip's tool. > > So I guess that means they're similar. I think it's premature to put it into the std library - not least because I've got a version I've been working on for a while which I'd eventually like to get in there if I can ;) Of course, part of that will rest on my employer ... The major problems I had with trace.py are that it is *slow* and that the data it collects is very incomplete. If we are going to have a standard tracing module, I think it needs to be *fast* and the data it collects needs to be complete, and easily transformable to many types of reports. My main use case is automated code coverage reports. If my unit tests take 30 minutes to run with no coverage tool, but 15 hours to run under a coverage tool, the coverage tool is useless for my nightly builds and automated testing. The one I've been working on is a *lot* faster (though still in pure python), dropping from about 3000% performance overhead to about 50%. It also collects pretty much all the data it can (calls, lines, which lines belong to which classes/functions/methods), resolves modules down to a single, most-qualified instance (for example, if import a and import b.a end up importing the same module, both will be represented in the report as b.a) and various other things. It also determines which source lines are executable (for example, multi-line strings will only have one of the lines as executable). It also *almost* works with Jython (doesn't work out executable lines properly). It *also* allows outputting of tracing lines as they are executed (and damn that slows things down). But I don't think it's a candidate yet. Among other things, the tracing functionality is not hooked into each new thread. Not an issue for my current project (which must run single-threaded due to being invoked inside something that must run in a single thread) but it is an issue for a version in the standard library. Tim Delaney

3 2

Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11
by M.-A. Lemburg 11 Dec '02

11 Dec '02

loewis(a)users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Include > In directory sc8-pr-cvs1:/tmp/cvs-serv15966/Include > > Modified Files: > compile.h parsetok.h pyerrors.h pythonrun.h symtable.h > Log Message: > Constify filenames and scripts. Fixes #651362. I remember Tim opposing such changes (and Greg Stein insisting on them ;-). AFAIK, this breaks third party code since some compilers simply reject casting char* to const char*. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

6 8

Modules/tclNotify.c
by Martin v. Löwis 11 Dec '02

11 Dec '02

Is this file still in use? I stumbled over it while investigating support for threaded Tcl. If it is not used anymore, can it be deleted? Regards, Martin

3 2

Re: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
by Michael Hudson 11 Dec '02

11 Dec '02

Just van Rossum <just(a)letterror.com> writes: > I have _no_ idea what's going on here. I do: readline. I wouldn't want to live without readline, but some of the things it does call for the application of thumbscrews. Cheers, M. -- FORD: Just pust the fish in your ear, come on, it's only a little one. ARTHUR: Uuuuuuuuggh! -- The Hitch-Hikers Guide to the Galaxy, Episode 1

4 5

anyone tried to use optparse with something else than sys.argv?
by Erik Heneryd 10 Dec '02

10 Dec '02

hi snuck a peek at the newly included optparse module yesterday and it seems option parsing will be a breeze in the future. good work! however, imho, it seems a bit too focused on initial sys.argv parsing. for anything else, you basically have to subclass OptionParser etc to do anything useful. for example, you can supply your own argument list to the parse_args() method, but the autohelp/version/error reporters still use sys.argv[0] as command name (probably not what you want). the way optparse gets the command name (a global function) indicates that you're not supposed to override it? also, even if you tell OptionParser to use your user defined Option class (as explained in the docs: op = OptionParser(option_class=MyOption)), it still adds standard Option instances to handle autohelp/version. per default, the autohelp/version/error handlers call sys.exit(). the docs says that if you want to change this behaviour, simply subclass OptionParser and/or Option and override/extend the corresponding methods. ofcourse, this has no effect, since standard instances are used. also, supressing the actual printing of autohelp/version/error requires subclassing OptionParser and Option (or sys.stdout = StringIO fiddling). so? some minor glitches and (imho) annoyances, whats the deal? well, to me, it's pretty obvious that the design effort on optparse has pretty much been focused strictly on initial sys.argv parsing and while it's extendable, other use cases (embedded consoles etc) seem to have been given little or no real thought. the things i mention is probably easy to fix: * make parse_args() take the whole "sys.argv" list and use the first element as the command name. breaks backward compatibility, i know. why was sys.argv[1:] used in the first place? the getopt legacy? * add parse_args_ex() that raises an exception containing the help/error message instead of SystemExit (SystemExit containing something means something went wrong, don't it?) * (simple bug) dynamically populate the OptionParser using the option_class and not some pre-made Option. * (simple bug) replace all %prog in help texts etc, not just the first occurance. erik

3 2

Proposed changes to linuxaudiodev
by Greg Ward 10 Dec '02

10 Dec '02

Hi all -- I've been hacking sporadically on the linuxaudiodev module for the last couple of days. Initial results are in patch #645786 (www.python.org/sf/645786). The main goal of this patch is to make linuxaudiodev objects a thinner wrapper around the underlying OSS device driver, while still providing some highish-level convenience methods -- see my comments in the patch for details. There's a slight theoretical risk of backwards incompatibility, though. Since this module has never been documented, and since its current behaviour is such that doing anything really funky is severely curtailed, I very much doubt this will be a problem. Is anyone doing anything with linuxaudiodev more sophisticated than playing silly beep sounds? Should I ask around on python-list to be sure I won't ruin anyone's day with this change? Oh yeah: someone noted in the CVS history that the module is misnamed. This is true; it should probably have been called ossaudiodev -- OSS is the current standard audio API used by Linux and, I think, various BSDs. It's also available for a bunch of commercial Unices. Greg -- Greg Ward <gward(a)python.net> http://www.gerg.ca/ Just because you're paranoid doesn't mean they *aren't* out to get you.

3 6

Complexity of import.c
by James C. Ahlstrom 10 Dec '02

10 Dec '02

I see that my patch to import.c has been criticized as adding to the complexity of an already complex module. I don't entirely agree. If you want to have the standard library available as a zip module, you must have zip file import available in C. The patch I proposed in PEP 273 was a minimalist patch; it adds as little as possible while still keeping package import and preserving all import semantics. IMHO, the real complexity of import.c comes not from the double loop for path in sys.path: for suffix in (".pyc", ".pyo",...): fopen(...) and not from the caching of zip archive names. It comes from the use of import.c to perform package imports. Package imports are performed by creating recursive calls to functions that would otherwise be flat, and by replacing sys.path with the package path. This is darned confusing. At the time PEP 273 was discussed, I proposed moving package imports out of import.c into a Python module such as Gordon's iu.py or Greg's imputil.py. This was rejected because package users feared that package imports would be slowed down. The speed of Python startup and import was a concern. I understand that people want to generalize zip imports, and create a better import hook mechanism. Some feel that now is the time to do it. This is my humble opinion: Replacing import.c plus zip import with an equally complex import.c is a fundamental mistake. The current import.c is understood (by few), and the zip file import patch adds the minimum required. If a different import.c is accepted, then it should be MUCH simpler, and support only flat directory searches and zip directories. Package imports and import hooks should be moved to a Python module such as Gordon's iu.py. And import.c must bootstrap iu.py and all its imports from either a directory search or a zip file. An improved import hook mechanism should have its own PEP. I tried my best not to break the Macintosh port, which has a lot of special code. A replacement import.c should do the same. My patch modified many other Python .c and .py files to solve several difficult bootstrap problems. If you replace just import.c you will get a painful lesson in bootstrapping. You probably need these other patches even if you reject my import.c patch. Jim Ahlstrom

4 6

Re: [Patches] Patch for xmlrpc encoding
by M.-A. Lemburg 10 Dec '02

10 Dec '02

Ragnar Kjørstad wrote: > Hi > > The dumps-method in xmlrpclib has the following comment: > All 8-bit strings in the data structure are assumed to use the > packet encoding. Unicode strings are automatically converted, > where necessary. > > > This doesn't work very well. In our particular case we're using latin_1 > as our default encoding, and we're using UTF-8 for the packet encoding. > We can't really change the default encoding, because the sql-modules > transfer latin_1 encoded data and we can't change the packet encoding to > latin_1 because the xmlrpc-client (php) doesn't work with that. > > The attached patch changes xmlrpclib to convert strings to unicode using > the default encoding, and then convert them back to strings with the > packet encoding. If unicode is not available it falls back to the old > behaviour. I believe this is overkill. If you need this behaviour, subclass the Marshaller in xmlrpclib and add your feature to that subclass. Then replace the Marshaller class in xmlrpclib with your subclass. Aside: xmlrpclib should support subclassing the Marshaller and Unmarshaller more transparently. Currently, the two are hard-coded into the rest of xmlrpclib without the possibility to provide your own subclasses without tweaking xmlrpclib from the outside. > I guess for performance it could check if the defaultencoding is the > same as the packet-encoding, but my guess is that it hardly ever is, so > no reason to optimize for it. > > Note; I'm not at all sure this is the best way to fix the problem. If > it's not, please feel free to ignore this patch, or even better - tell > me what the preferable fix is. Please post patches using the SourceForge patch manager. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

6 7

zipimport, round 4
by Just van Rossum 09 Dec '02

09 Dec '02

Here's a new patch. Changes include: - pyc time stamp checking (thanks Skip and </F>!) - better python -v output - also works when zipimport is built dynamically I've written a note about various aspects of the patch (pasted below) but I'm not sure it's PEP-ready yet. Comments are more than welcome! Just --------------------- This note is in a way an addendum to PEP 273. I fully agree the points and goals of the PEP, except for the sections "Efficiency", "Directory Imports" and "Custom Imports". However, I disagree strongly with much of the design and implementation of PEP 273. This note presents an alternative, with a matching implementation. A brief history of import.c -- digging through its cvs log. When Python was brand new, there were only builtin modules and .py files. Then .pyc support was added and about half a year later support for dynamically loaded C extension was implemented. Then came frozen modules. Then Guido rewrote much of import.c, introducing the filedescr struct {suffix, mode, type}, allowing for some level of (builtin) extension of the import mechanism. This was just before Python 1.1 was released. Since then, the only big change has been package support (in 1997), which added a lot of complexity. (The __import__ hook was quietly added in 1995, it's not even mentioned in the log entry of ceval.c r2.69, I had to do a manual binary search to find it...) All later import extensions were either implemented using the filedescr mechanism and/or hardcoded in find_module and load_module. This ranges from reading byte code from Macintosh resources to Windows registry-based imports. Every single time this involved another test in find_module() and another branch in the load_module() switch. "This has to stop." The PEP 273 implementation. Obviously the PEP 273 implementation has to add *something* to import.c, but it makes a few mistakes: - it's badly factored (for example it adds the full implementation of reading zip files to import.c.) - it adds a questionable new feature: directory caching. The original author claimed this was needed for zip imports, but instead solving the problem locally for zip files the feature is added to the builtin import mechanism as a whole. Although this causes some speedup (especially for network file system imports), this is bad, for several reasons: - it's not strictly *needed* for builtin import - it's not a frequent feature request from users (as far as I know) - it makes import.c even more complicated than it already is (note that I say "complicated", not "complex") - it changes semantics: if a module is added to the file system *after* the directory contents has been cached, it will not be found. This might only be a real issue for an IDE that runs code inside the IDE process, but still. A different approach. An __import__ hook is close to useless for the problem at hand, as it needs to reimplement much of import.c from scratch. This can be witnessed in Guido's old ihooks.py, Greg Stein's imputils.py and Gordon McMillan's iu.py, each of which are failry complex, and not compatible with any other. So we either have to add just another import type to import.c for zip archives, or we can add a more general import hook. Let's assume for a moment we want to do the *former*. The most important goal is for zip file names on sys.path and PYTHONPATH to "just work" -- as if a zip archive is just another directory. So when traversing sys.path, each item must be checked for zip-file-ness, and if it is, the zip file's file index needs to be read so we can determine whether the module being imported is in there. I went for an OO approach, and represent a zip file with an instance of the zipimporter class. Obviously it's quite expensive to read the zip file index again and again, so we have to maintain a cache of zipimporter objects. The most Pythonic approach would be to use a dict, using the sys.path item as the key. This cache could be private to the zip import mechanism, but it makes sense to also cache the fact that a sys.path item is *not* a zip file. A simple solution is to map such a path item to None. By now it makes more sense to have this cache available in import.c. The zipimporter protocol. The zipimporter's constructor takes one argument: a path to a zip archive. It will raise an exception if the file is not found or if it's not a zip file. The import mechanism works in two steps: 1) find the module, 2) if found, load the module. The zipimporter object follows this pattern, it has two methods: find_module(name): Returns None if the module wasn't found, or the zipimporter object itself if it was. load_module(fullname): Load the module, return it (or propagate an exception). The main path traversing loop in import.c will then look like this (omitting the caching mechanics for brevity): def find_module(name, path): if isbuiltin(name): return builtin_filedescr if isfrozen(name): return frozen_filedescr if path is None: path = sys.path for p in sys.path: try: v = zipimporter(p) except ZipImportError: pass else: w = v.find_module(name) if w is not None: return w ...handle builtin file system import... Packages. Paths to subdirectories of the zip archive must also work, on sys.path for one, but most importantly for pkg.__path__. For example: "Archive.zip/distutils/". Such a path will most likely be added *after* "StdLib.zip" has been read (after all, the parent package is *always* loaded before any submodules), so all I need to do is strip the sub path, and look up the bare .zip path in the cache. A *new* zipimporter instance is then created, which references the same (internal, but visible) file directory info as the "bare" zipimporter object. A .prefix contains the sub path: >>> from zipimport import zipimporter >>> z = zipimporter("Archive.zip/distutils") >>> z.archive 'Archive.zip' >>> z.prefix 'distutils/' >>> Beyond zipimport. So there we are, zipimport works, with just a relatively minor impact on import.c. The obvious next question is: what about other import types, whether future needs for the core, or third party needs? It turns out the above approach is *easily* generalized to handle *arbitrary* path-based import hooks. Instead of just checking for zip-ness, it can check a list of candidates (again, caching cruft omitted): def find_module(name, path): if isbuiltin(name): return builtin_filedescr if isfrozen(name): return frozen_filedescr if path is None: path = sys.path for p in sys.path: v = None for hook in sys.import_hooks: try: v = hook(p) except ImportError: pass else: break if v is not None: w = v.find_module(name) if w is not None: return w ...handle builtin file system import... Now, one tiny step further, and we have something that fairly closely mimics Gordon McMillan's iu.py. That tiny step is what Gordon calls the "metapath". It works like this: def find_module(name, path): for v in sys.meta_path: w = v.find_module(name, path) if w is not None: return w # fall through to builtin behavior if isbuiltin(name): return builtin_filedescr [ rest same as above ] An item on sys.meta_path can override *anything*, and does not need an item on sys.path to get invoked. The find_module() method of such an object has an extra argument: path. It is None or parent.__path__. If it's None, sys.path is implied. The Patch. I've modified import.c to support all of the above. Even the path handler cache is exposed: sys.path_importers. (This is what Gordon calls "shadowpath"; I'm not happy with either name, I'm open to suggestions.) The sys.meta_path addition isn't strictly neccesary, but it's a useful feature and I think generalizes and exposes the import mechanism to the maximum of what is possible with the current state of import.c. The protocol details are open to discussion. They are partly based on what's relatively easily doable in import.c. Other than that I've tried to follow common sense as to what is practical for writing import hooks. The patch is not yet complete, especially regarding integration with the imp module: you can't currently use the imp module to invoke any import hook. I have some ideas on how to do this, but I'd like to focus on the basics first. Also: the reload() function is currently not supported. This will be easy to fix later. I *thought* about allowing objects on sys.path (which would then work as an importer object) but for now I've not done it as sys.meta_path makes it somewhat redundant. It would be easy to do, though: it would add another 10 lines or so to import.c. I've tested the zipimporter module both as a builtin and as a shared lib: it works for me in both configurations. But when building it dynamically: it _has_ to be available on sys.path *before* site.py is run. When running from the build dir on unix: add the appropriate build/lib.* dir to your PYTHONPATH and it should work.

1 0

zipimport, round 3 (or would that be that 37?)
by Just van Rossum 09 Dec '02

09 Dec '02

Here's the current state of my stuff. I'd rather post it now than to fiddle more: I'm too tired, I'd only break stuff... I currently build the zipimport module as a builtin to avoids some bootstrapping pitfalls (*). It should still work as a shared lib *provided* it is available on sys.path *before* site.py is run. The same goes more or less for zlib unless you're using uncompressed archives. The test suite ran fine with the std lib packaged as a zip file (minus the test suite it, as it globs for *.py in Lib/test/), apart from three tests that do stuff with module.__file__ (eg. email can't find it's data files ;-). Highlights: - zip file names on sys.path work - zip file names PYTHONPATH work - sys.path still contains strings - module.__file__ is still a regular string, looking like this: '/path/to/archive.zip/path/to/module.pyc' - pkg.__path__ still contains a regular string, that looks like this: '/path/to/archive.zip/path/to/packagedir/' I'll write a bit about the mechanics tomorrow, but in the meantime I would like to see some test results/bug reports/bug fixes from Windows developers. Please don't start with the full monty (ie. the std lib), try a small test archive first please ;-) There's still plenty to do regarding imp module integration, minor things like reload() not being supported yet, etc. Nevertheless I think the import.c patch is remarkably small for what it accomplishes. Needless to say, the zipimport module itself also needs work. Fun, fun! Just *) It's mostly a problem when you're running Python from the source directory, when site.py adds stuff to sys.path to get a running system: by then it's too late to find zipimport. It should be less of a problem with a regular install, but I haven't tried.

6 16

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 09 Dec '02

09 Dec '02

From: Gordon McMillan [mailto:gmcm@hypernet.com] > BTW, iu does allow non string objects in sys.path. > If it exposes "getmod(name)", that will be tried > in preference to iu's own machinery. I just tried this out. As far as I can see, iu works perfectly with a simple getmod() method. Why does your patch require find_module() and load_module()? I'm not criticising, just trying to understand the requirements... (Anything to make writing an import hook easier!!!) You'll notice I'm wavering over my dislike of having objects on sys.path :-) My interest in import hooks stems from wanting to be able to implement something like Java's 'executable jar file' support, where an application, data and all, can be packaged up as a single file. In theory, there's no real problem with this (I can write import hooks to load from a zip file right now), but in practice, the bootstrapping problems are quite nasty. The more in-core (or at least standard library) support there is for these features, the less of a bootstrap problem I have... Paul. """ Example code for iu (adds an object to sys.path which satisfies *all* requests for modules and packages with a new empty module...) """ import iu import imp import sys iu.ImportManager().install() class Tester: def getmod(self, name): print "Requested", name, locals() m = imp.new_module(name) # Either set __path__ to contain an object, # or set __path__ to contain a path, and # __importsub__ to the object... m.__path__ = [ Tester() ] sys.modules[name] = m return m sys.path.append(Tester())

5 9

Zip imports, PEP 273, and the .zip extension
by Paul Moore 09 Dec '02

09 Dec '02

PEP 273 does not mandate a particular file extension for zip files. However, both current implementations (Jim Ahlstrom's patch and Just's import hooks) recognise zip files via the extension ".zip". I'd like to suggest that this should not be the case, and that zipfiles should be recognised by the embedded signature, as is done by zipfile.is_zipfile(). This will (slightly) slow down the initial checking for zipfiles, as it requires a file read, but if the result of the check is cached the performance impact will be minimal. The main advantage of this is in generality. However, I have a specific usage in mind - namely, concatenating a Python script with its supporting code as a zip archive. (Zip archives can have arbitrary data prepended without affecting their validity). Then, simply by executing sys.path.append(sys.argv[0]) the script can access its own embedded modules. One problem with getting this to work is that at present, the Python parser will choke on data appended to the script. This can be avoided under Windows by a system-specific hack, of embedding a ^Z character in the script (which text-mode I/O treats as EOF) just before the appended data. In the longer term, a language-mandated "stop parsing here" token, like Perl's __END__, might be useful. But that's not something I'm proposing right now (unless an expert in Python's parser steps up and says "oh, that's easy - I can implement that in a couple of lines of code" :-)) Any comments? Paul. -- This signature intentionally left blank

4 3

Import compromise
by Aahz 09 Dec '02

09 Dec '02

Guido, I call for a Pronouncement: I'm seeing some convergence on basic criteria needed for a Grand Import Redesign, but I'm not seeing convergence on the actual design itself. Time is running short for 2.3, and people making hasty decisions make poor decisions. OTOH, the complaints raised about the current patch for PEP 273 have merit, I think. On the gripping hand, we really want PEP 273 in some form for 2.3. So here's my proposed compromise: Split the PEP 273 patch into two parts. The first part is the absolutely critical modifications to import.c, which basically boils down to making a function call every time you access a '*.zip' string on sys.path. Everything else goes into zipimport.c, a compiled-in module like socket. (I'd even support making zipimport.c an include file for import.c) Yes, this means we lose the caching for non-ZIP files for 2.3 -- I vote YAGNI. Yes, this makes more work over the long haul, but it saves stress and time now. Yes, it's more work than just accepting the current PEP 273 patch, but I think there are too many good arguments against that. zipimport will *NOT* be a publicly exposed library for 2.3, possibly never. -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy(a)lion.austin.ibm.com

9 14

__getstate__() not inherited when __slots__ present
by Greg Ward 09 Dec '02

09 Dec '02

If a class and its superclass both define __slots__, it appears that __getstate__() is not inherited from the superclass. Example: """ class Base (object): __slots__ = [] def __getstate__ (self): return tuple([getattr(self, attr) for attr in self.__slots__]) class Thing (Base): __slots__ = ['a', 'b'] def __init__ (self): self.a = 42 self.b = "boo!" abstract = Base() thing = Thing() print abstract.__getstate__ print thing.__getstate__ """ When I run this with a not-quite-current CVS Python: <bound method Base.__getstate__ of <__main__.Base object at 0x401f73b8>> <built-in function __getstate__> The upshot of this is that I can't just define one __getstate__() in the superclass of a bunch of __slots__-using classes -- I guess I'll have to set __getstate__ manually for each class. ;-( If this is a feature, is it documented anywhere? (BTW, I see the same behaviour with Python 2.2.2.) Greg -- Greg Ward <gward(a)python.net> http://www.gerg.ca/ "Very funny, Scotty. Now beam my *clothes* down."

3 2

Re: Another approach for the import mechanism
by Chermside, Michael 09 Dec '02

09 Dec '02

Gustavo Niemeyer writes: > - Don't have to change path to use compressed packages (at least > not if you want to provide compressed packages, individual > compressed modules or the standard library). > > - Don't have to specify the compression type hardcoded. > > - Allows one to ship a package inside a zip file, without asking > the user to change his path, and without hacking the package. > > - Allows one to compress a single file (foobar.py.bz2). > > I belive that my propose is quite clear now. If there are no > additional supporters, there's no reason to go on. I think this proposal would make sense IF compression were an important goal here. But to me, it isn't. Zip does two things... it aggregates into a single file (maintaining directory structure) and it compresses. Of the two, I find the aggregation important and the compression a mere side effect. This is why I really don't care much about switching to a different compression (or aggregation) format -- one standard way to do it is more useful to me than a BETTER way. It is why I don't care about compressing a single file. Basically, disk space is cheap, but effort to keep track of (and distribute) complex file hierarchies isn't. > Thanks to everyone who discussed. And thanks for your contributions too. I don't realize exactly what I was looking for from .zip until you explained so clearly the benefits of your proposal. -- Michael Chermside

4 3