Mailman 3 December 2002 - Python-Dev

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 04 Dec '02

04 Dec '02

> My current sys.import_hooks patch implements #3 and I think > that's the most logical thing to do; after all sys.path may not > be meaningful for a hook. #2 makes little sense I guess. #1 > could be an option, but, given the current state of import.c, > would be a lot harder to implement. I also think that #3 is the > most efficient scheme, as the hook can save some local state > while traversing sys.path. Have you looked at Gordon McMillan's import utilities? He handles this by having a "shadowpath" in the manager which maps sys.path entries to "Owners", where each Owner is responsible for loading from one location (directory, zipfile, whatever). I'm convinced you're reinventing the wheel here, at least in part. (But reinventing it in C, rather than Python, so it's not a completely bad thing :-)) Paul.

2 1

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 04 Dec '02

04 Dec '02

From: Martin v. Löwis [mailto:martin@v.loewis.de] > Just van Rossum <just(a)letterror.com> writes: >> I'm working hard on a zipimport module that uses >> sys.import_hooks. It's going to be SO much cleaner than >> the existing zip import patch. > Ok. I then won't apply the existing patch until > a) your zipfile patch comes along to review, or > b) significant time has passed after 2.3a1 (for some > reasonable value for "significant") The only reservation I have about this is if the existing patch starts going "out of date" again. I can't get CVS to work on my slow dial-up connection, for anything other than a straight cvs checkout, so I'm reduced to full downloads and manually updating the patch. This is both tedious and error-prone. Just promised his patch in "a couple of days". If that's the timescale, I'm fine with this plan. But if it's a couple of weeks, then I'm going to be annoyed if the reintegration turns out to be complicated. Paul.

1 0

Re: [Python-Dev] extension module .o files wind up in the wrong place
by Michael Hudson 04 Dec '02

04 Dec '02

Michael Hudson <mwh(a)python.net> writes: > The first out-of-tree build I did today put one .o file in the wrong > place (bsddbmodule.o, I think). Now all of them are ending up in the > wrong place. Not sure how I managed that one, but I've found the offending checkin: ---------------------------- revision 1.52 date: 2002/11/14 01:29:00; author: akuchling; state: Exp; lines: +3 -3 [Bug #599248] ext module generation problem If you have source files srcdir1/foo.c and srcdir2/foo.c, the temporary .o for both files is written to build/temp.<platform>/foo.o. This patch sets strip_dir to false for both calls to object_filename, so now the object files are written to temp.<platform>/srcdir1/foo.o and .../srcdir2/foo.o. 2.2 bugfix candidate ---------------------------- Now to see if patching it up is easy... Cheers, M. -- ARTHUR: Ford, you're turning into a penguin, stop it. -- The Hitch-Hikers Guide to the Galaxy, Episode 2

1 0

Re: [Python-Dev] extension module .o files wind up in the wrong place
by Michael Hudson 04 Dec '02

04 Dec '02

Jack Jansen <Jack.Jansen(a)cwi.nl> writes: > I am tempted to think that this problem has always existed, but has=20 > gone unnoticed until now. It's definitely gotten worse recently. > Now that I know about the problem it clears up some strange behaviour=20 > I've seen in the past. > I would love to see this fixed, but I'm afraid I'm not the person to do=20= > > it. Distutils > never ceases to amaze me, and it's path-mangling tricks are beyond me.=20= > > I'm pretty sure that > if we somehow stuff "../Modules/regexmodule.c" (assuming we're building=20= > > in a subdir of the main dir) > into the list of module sources we'll get something like=20 > build/tmp.blabla/../Modules/regexmodule.o > for the object filename. > > Does anyone else feel up to fixing this? I'll have a look, but so far I'm only confusing myself. The first out-of-tree build I did today put one .o file in the wrong place (bsddbmodule.o, I think). Now all of them are ending up in the wrong place. Cheers, M. -- Very clever implementation techniques are required to implement this insanity correctly and usefully, not to mention that code written with this feature used and abused east and west is exceptionally exciting to debug. -- Erik Naggum on Algol-style "call-by-name"

1 0

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 04 Dec '02

04 Dec '02

From: Just van Rossum [mailto:just@letterror.com] > Neal Norwitz wrote: > > > On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote: > > > > > My view is that import.c is, indeed, very complex and > > > difficult code to follow. I agree with Just's comments that > > > a cleanup in import.c would be useful. > > > > I agree with both of these statements. I have reviewed the > > patch. While it's quite involved, I don't think it's worth > > rejecting I would really like to see a cleanup, though > > I'm working hard on a zipimport module that uses > sys.import_hooks. It's going to be SO much cleaner than the > existing zip import patch. That's great news, as I've said before. Can I ask that you consider a few issues, though. I don't have particular use cases in mind, but I believe that PEP 273 had a number of implicit assumptions, which your proposal does not incorporate. I'd like to make the differences explicit, just so that people get a chance to comment on whether they care. 1. Your import hooks seem to act at a level "above" sys.path, in the sense that the normal sys.path search becomes just another hook. With that in mind, I don't see a way for a user to add a zipfile in the *middle* of sys.path. 2. The syntax needed to add a zipfile search is different from the familiar one for sys.path additions. This isn't an issue as such, but may make the feature a little less accessible. 3. By using a non-sys.path approach, you make it impossible for users to set PYTHONPATH to include zipfiles. Can I again ask you to review Gordon McMillan's import manager. I really feel that you're reinventing the wheel here, and Gordon's code is well tested, as it's part of his Installer package. Maybe Gordon's code (which is pure Python) isn't appropriate to use, but I'm sure the design is worth looking at. > > Rejecting the patch would be worthwhile if the interface > > would need to change after incorporating this patch. I don't > > believe this is the case. Whether this patch is accepted > > or not wouldn't change future patches that Just wants to > > incorporate. > > I personally reject the patch because it modifies import.c > beyond repair. Once this stuff is in, there's NO way we can go > back without a huge effort, meaning it won't happen. I think you're exaggerating. It makes things worse, certainly, but I'd strongly dispute your claim of "beyond repair". I simply don't think the change is that major. Either it's already beyond repair, or the patched version is still fixable. And nobody is proposing "going back". A refactoring is "going forward". Whether it's a refactoring of code that *already* supports zipfiles or a refactoring which then has to be changed to add zipfile support shouldn't matter. > It adds > a whole directory listing caching scheme that is a) totally > unneeded and b) has (IMO) _nothing_ to do with zip importing > per se. That ExternalNames/External names stuff? I don't > understand why it is needed just to enable zip imports. It's a > lot of stuff! I've explained this a couple of times now. Please read the PEP for details, but Jim claims that zipfile imports need this if they are not to be too slow. A directory lookup in a zipfile is basically a linear search through the list of all files in the zipfile. For a large zipfile (such as the Python library) this is going to be *slow*. Please benchmark your implementation on a zipped version of the full Python library, replacing the directory version, and compare with Jim's benchmarks. While speed isn't the be-all and end-all of this patch, I'd hate to lose a significant speed gain... And the cache has a nice side-effect in that it speeds up general imports, particularly on networked drives. > > > I think that the zipfile patch is close to being > > > acceptable. I don't have a problem if it gets rejected, but > > > can I make a plea - if it does, can it be "Rejected because > > > patch XXX does the job better" rather than "Rejected > > > because this can be implemented better using feature YYY". > > > > Agreed. > > And that's ecxactly what I intend to say in a couple of days if > not sooner. Great! Will your code address all of the above points, and the issues raised on python-dev in Oct/Nov 2001 (see below for references)? > [ ... ] > > If Just incorporates his import patch, perhaps the zipfile > > code could be refactored to use the more generic framework. I > > don't view these two patches as mutually exclusive. > > I *totally* do. > > > If all of these issues are addressed, the resulting code > > could be clearer, even if intermediate versions are not. > > A rewrite now is MUCH easier than cleaning it up later. It's > going to be much less code. Wait and see. Here are some references that anyone looking at this stuff should read. There was a *lot* of discussion on this patch in Oct/Nov 2001. Start of the PEP 273 thread: http://mail.python.org/pipermail/python-dev/2001-October/018158.html Thread on caching of lookups http://mail.python.org/pipermail/python-dev/2001-November/018346.html Thomas Heller proposing Gordon McMillan's code for the standard library: http://mail.python.org/pipermail/python-dev/2001-October/017750.html As I've said, I'm not against rejecting the current patch - but I'd be concerned if all of the work that went into it (ie, Jim's design and discussions on python-dev, *not* the specific code implementing it) was just lost. Paul.

7 15

Re: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
by Michael Hudson 04 Dec '02

04 Dec '02

Tim Peters <tim.one(a)comcast.net> writes: > [Michael Hudson] > > Presumably there's a possibility of an optimization for pickling > > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > > > Hard to say whether it would be worth it, though. > > I don't really care about lists of date objects. The intent was just to see > how much of the administrative pickle bloat could be saved by pickle's > internal memo facility when multiple date objects appear in a structure for > *whatever* reason (be it a list or tuple of dates, or a dict keyed by dates, > or an object with multiple data-value attributes, or ...). OK. > The administrative pickle bloat for a single date instance is severe: > > pickling 2000-12-13 via Python -- pickle length 80 > pickling 2000-12-13 via C -- pickle length 43 Yow. > The internal pickle memo saves a lot if more than one data instance appears, > via "remembering" parts of the overhead scaffolding. But in the end a data > object has a 4-byte state string, and there's still a lot more overhead than > state stored in the pickles. Seems so. > > ... > > Here's a fairly simple minded patch to the pickling side of pickle.py: > > it seems to save about 6 bytes per object in the good cases. > > > > with: > > list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj > > > > without: > > list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj > > The "without" number is most curious. When I run the checked-in code, I get > > list of 100 dates via C -- 1533 bytes, 15.33 bytes/obj > > This was w/ CVS Python, Win2K and MSVC6. Ah! You must have fiddled the > script to use pickle instead of cPickle. Yep. > I get 1871 bytes then. I'm surprised they're so different. I know cPickle does some tricks if refcounts are 1 that aren't duplicated in pickle.py. > > I'm not going to pursue this further unless someone thinks it's a > > worthwhile move. > > I don't personally have large pickles of homogeneous lists, so hard to say. Me neither. > The C implementation of pickle would also need to be fiddled. That was part of "pursuing this further" (as is, e.g., unpickling support). > Cutting the pickle overheads for instances of new-style classes > would more clearly be worthwhile. Hard to see how to do that. You could reduce the overhead by giving _datetime_unpickler a shorter name :-) In general, though, any improvements we can add here are likely to be blown away by ramming the pickle through zlib, so I'm not sure this is worth worrying about too much. Cheers, M. -- And then the character-only displays went away (leading to increasingly silly graphical effects and finally to ads on web pages). -- John W. Baxter, comp.lang.python

1 0

RE: [Python-Dev] Re: New and Improved Import Hooks
by Delaney, Timothy 04 Dec '02

04 Dec '02

> From: Shane Holloway (IEEE) [mailto:shane.holloway@ieee.org] > > Expanding upon Tim > Delaney's desires to include importing from tar files and internet > references, Hey - I didn't say *I* wanted this (they came from other posts ;) but it's an obvious extension ... > consider being able to extend this mechanism to importing > from an XML document, database of modules, bz2 files, and > even encrypted > collections, As are all of these. I think if the import mechanism is going to be changed, it should be changed so as to be flexible. The fact that such a change is also (IMO) a much cleaner approach is icing. Tim Delaney

1 0

Re: New and Improved Import Hooks
by Shane Holloway (IEEE) 04 Dec '02

04 Dec '02

As it seems I am supposed to introduce myself on my first post, I am a professional software engineer in Colorado, USA, blessed with the chance to implement my solutions in Python. I've a fairly diverse background in languages, but the most experience with C++. I keep working on the RuneBlade project <http://www.runeblade.com>, which helps me deliver my Python solutions just that much faster. Seaching for a better language, Python has satisfied my thirst, seeming to combine my best experiences from Lisp, Pascal, C++ and others. Thanks for all of your hard work! Now it's time for me to try and help! IMO, Just's approach seems to be flexable enough to solve more than the problem at hand. Not only does the framework help to implement PEP 273, allowing for import from zip files, but it provides a very logical place for other import mechanisms to be implemented. Expanding upon Tim Delaney's desires to include importing from tar files and internet references, consider being able to extend this mechanism to importing from an XML document, database of modules, bz2 files, and even encrypted collections, The key, as I see it, is the flexability of the approach. One can even add and remove import stratigies on the fly without prior knowledge of all the import hooks! :-) As for keeping the import order of the sys.path, Tim seems to have hit the mark with the default importer supporting "sub-hooks" for the different path entry types. The best part about the import_hooks approach is that it leaves enough flexability with developer to implement creative approaches, while maintaining a commonly recognized standard for import hook interfaces. Even module-caching import hooks can use this model. One could write a simple hook that would log all imports, or all imports that could not be found, regardless of others import hooks. (Well, at least, if they behave properly ;) -Shane Holloway

1 0

RE: [Python-Dev] New and Improved Import Hooks
by Delaney, Timothy 03 Dec '02

03 Dec '02

> From: Just van Rossum [mailto:just@letterror.com] > > Regarding sys.import_hooks, I see three possibilities: > > #1 > for p in sys.path: > for hook in sys.import_hooks: > hook(p, ...) > > #2 > for hook in sys.import_hooks: > for p in sys.path: > hook(p, ...) > > #3 > for hook in sys.import_hooks: > # the hook can traversing sys.path if needed. > hook(...) It seems to me there needs to be two separate sets of import hooks. 1. Module import hooks. 2. Source import hooks. Module import hooks are the type being discussed for sys.import_hooks - a sequence of callables which are called one after the other until a module is successfully imported. Source import hooks provide the exact mechanism for doing an import from a particular data source. Doing this, in effect we would have the following algorithm ... for hook in sys.import_hooks: hook(...) and the algorithm for the builtin import hook would be ... for p in sys.path: for hook in sys.source_import_hooks: hook(p, ...) which would end up being the default behaviour. Individual module import hooks could (and probably should) use the appropriate source import hooks. I think this would suit all requirements, whilst being both general and clean. It would provide a simple mechanism for allowing .tar files in sys.path, internet imports, etc. Tim Delaney

1 0

Re: New and Improved Import Hooks
by Chermside, Michael 03 Dec '02

03 Dec '02

> > 1. Your import hooks seem to act at a level "above" sys.path, > > in the sense that the normal sys.path search becomes just > > another hook. With that in mind, I don't see a way for a user > > to add a zipfile in the *middle* of sys.path. > > True. There would be ways around that, but it's currently not easy. I don't feel > this is a particularly important feature, though. Hmm... Not sure I agree. I think this is one area where we'd be wise to follow Java's example. Normally I think it's wise for Python to do its own design instead of imitating the errors of other languages, but this is the exception. (1) Java (AFAIK) originated the idea of mixing .zip files as part of the path. (2) Their approach makes sense (in my mind, it would make sense to say "zip archives appear in path and are treated as if they were subdirectories", and it would ALSO make sense to say "the list of zip archives is separate from the path". But some mix like "zip archives appear in the path, but we treat them as if they were at the end even though they aren't" doesn't make sense). (3) It makes sense for Jython (for obvious reasons!). Because of (1), along with the popularity of java, putting .zip's on the path will seem "natural and familiar" to lots of Python programmers. Because of (2) it isn't a BAD idea. And (3) is just another good motivation... let's keep c-python and jython as similar as possible! I sympathize about this being "not currently easy", but... -- Michael Chermside PS: I like the import-hooks idea, and I think the zip importer is an excellent test case to really see how well the concept plays in practice.

1 0