Mailman 3 December 2002 - Python-Dev

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul Dec. 4, 2002

Dec. 4, 2002

> My current sys.import_hooks patch implements #3 and I think > that's the most logical thing to do; after all sys.path may not > be meaningful for a hook. #2 makes little sense I guess. #1 > could be an option, but, given the current state of import.c, > would be a lot harder to implement. I also think that #3 is the > most efficient scheme, as the hook can save some local state > while traversing sys.path. Have you looked at Gordon McMillan's import utilities? He handles … [View More]

2 1

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul Dec. 4, 2002

Dec. 4, 2002

From: Martin v. Löwis [mailto:martin@v.loewis.de] > Just van Rossum <just(a)letterror.com> writes: >> I'm working hard on a zipimport module that uses >> sys.import_hooks. It's going to be SO much cleaner than >> the existing zip import patch. > Ok. I then won't apply the existing patch until > a) your zipfile patch comes along to review, or > b) significant time has passed after 2.3a1 (for some > reasonable value for "significant") The only … [View More]

1 0

Re: [Python-Dev] extension module .o files wind up in the wrong place
by Michael Hudson Dec. 4, 2002

Dec. 4, 2002

Michael Hudson <mwh(a)python.net> writes: > The first out-of-tree build I did today put one .o file in the wrong > place (bsddbmodule.o, I think). Now all of them are ending up in the > wrong place. Not sure how I managed that one, but I've found the offending checkin: ---------------------------- revision 1.52 date: 2002/11/14 01:29:00; author: akuchling; state: Exp; lines: +3 -3 [Bug #599248] ext module generation problem If you have source files srcdir1/foo.c and … [View More]

1 0

Re: [Python-Dev] extension module .o files wind up in the wrong place
by Michael Hudson Dec. 4, 2002

Dec. 4, 2002

Jack Jansen <Jack.Jansen(a)cwi.nl> writes: > I am tempted to think that this problem has always existed, but has=20 > gone unnoticed until now. It's definitely gotten worse recently. > Now that I know about the problem it clears up some strange behaviour=20 > I've seen in the past. > I would love to see this fixed, but I'm afraid I'm not the person to do=20= > > it. Distutils > never ceases to amaze me, and it's path-mangling tricks are beyond me.=20= > … [View More]

1 0

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul Dec. 4, 2002

Dec. 4, 2002

From: Just van Rossum [mailto:just@letterror.com] > Neal Norwitz wrote: > > > On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote: > > > > > My view is that import.c is, indeed, very complex and > > > difficult code to follow. I agree with Just's comments that > > > a cleanup in import.c would be useful. > > > > I agree with both of these statements. I have reviewed the > > patch. While it's quite involved, I don't think it'… [View More]s worth > > rejecting I would really like to see a cleanup, though > > I'm working hard on a zipimport module that uses > sys.import_hooks. It's going to be SO much cleaner than the > existing zip import patch. That's great news, as I've said before. Can I ask that you consider a few issues, though. I don't have particular use cases in mind, but I believe that PEP 273 had a number of implicit assumptions, which your proposal does not incorporate. I'd like to make the differences explicit, just so that people get a chance to comment on whether they care. 1. Your import hooks seem to act at a level "above" sys.path, in the sense that the normal sys.path search becomes just another hook. With that in mind, I don't see a way for a user to add a zipfile in the *middle* of sys.path. 2. The syntax needed to add a zipfile search is different from the familiar one for sys.path additions. This isn't an issue as such, but may make the feature a little less accessible. 3. By using a non-sys.path approach, you make it impossible for users to set PYTHONPATH to include zipfiles. Can I again ask you to review Gordon McMillan's import manager. I really feel that you're reinventing the wheel here, and Gordon's code is well tested, as it's part of his Installer package. Maybe Gordon's code (which is pure Python) isn't appropriate to use, but I'm sure the design is worth looking at. > > Rejecting the patch would be worthwhile if the interface > > would need to change after incorporating this patch. I don't > > believe this is the case. Whether this patch is accepted > > or not wouldn't change future patches that Just wants to > > incorporate. > > I personally reject the patch because it modifies import.c > beyond repair. Once this stuff is in, there's NO way we can go > back without a huge effort, meaning it won't happen. I think you're exaggerating. It makes things worse, certainly, but I'd strongly dispute your claim of "beyond repair". I simply don't think the change is that major. Either it's already beyond repair, or the patched version is still fixable. And nobody is proposing "going back". A refactoring is "going forward". Whether it's a refactoring of code that *already* supports zipfiles or a refactoring which then has to be changed to add zipfile support shouldn't matter. > It adds > a whole directory listing caching scheme that is a) totally > unneeded and b) has (IMO) _nothing_ to do with zip importing > per se. That ExternalNames/External names stuff? I don't > understand why it is needed just to enable zip imports. It's a > lot of stuff! I've explained this a couple of times now. Please read the PEP for details, but Jim claims that zipfile imports need this if they are not to be too slow. A directory lookup in a zipfile is basically a linear search through the list of all files in the zipfile. For a large zipfile (such as the Python library) this is going to be *slow*. Please benchmark your implementation on a zipped version of the full Python library, replacing the directory version, and compare with Jim's benchmarks. While speed isn't the be-all and end-all of this patch, I'd hate to lose a significant speed gain... And the cache has a nice side-effect in that it speeds up general imports, particularly on networked drives. > > > I think that the zipfile patch is close to being > > > acceptable. I don't have a problem if it gets rejected, but > > > can I make a plea - if it does, can it be "Rejected because > > > patch XXX does the job better" rather than "Rejected > > > because this can be implemented better using feature YYY". > > > > Agreed. > > And that's ecxactly what I intend to say in a couple of days if > not sooner. Great! Will your code address all of the above points, and the issues raised on python-dev in Oct/Nov 2001 (see below for references)? > [ ... ] > > If Just incorporates his import patch, perhaps the zipfile > > code could be refactored to use the more generic framework. I > > don't view these two patches as mutually exclusive. > > I *totally* do. > > > If all of these issues are addressed, the resulting code > > could be clearer, even if intermediate versions are not. > > A rewrite now is MUCH easier than cleaning it up later. It's > going to be much less code. Wait and see. Here are some references that anyone looking at this stuff should read. There was a *lot* of discussion on this patch in Oct/Nov 2001. Start of the PEP 273 thread: http://mail.python.org/pipermail/python-dev/2001-October/018158.html Thread on caching of lookups http://mail.python.org/pipermail/python-dev/2001-November/018346.html Thomas Heller proposing Gordon McMillan's code for the standard library: http://mail.python.org/pipermail/python-dev/2001-October/017750.html As I've said, I'm not against rejecting the current patch - but I'd be concerned if all of the work that went into it (ie, Jim's design and discussions on python-dev, *not* the specific code implementing it) was just lost. Paul. [View Less]

7 15

Re: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
by Michael Hudson Dec. 4, 2002

Dec. 4, 2002

Tim Peters <tim.one(a)comcast.net> writes: > [Michael Hudson] > > Presumably there's a possibility of an optimization for pickling > > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > > > Hard to say whether it would be worth it, though. > > I don't really care about lists of date objects. The intent was just to see > how much of the administrative pickle bloat could be saved by pickle's > internal memo facility when … [View More]multiple date objects appear in a structure for > *whatever* reason (be it a list or tuple of dates, or a dict keyed by dates, > or an object with multiple data-value attributes, or ...). OK. > The administrative pickle bloat for a single date instance is severe: > > pickling 2000-12-13 via Python -- pickle length 80 > pickling 2000-12-13 via C -- pickle length 43 Yow. > The internal pickle memo saves a lot if more than one data instance appears, > via "remembering" parts of the overhead scaffolding. But in the end a data > object has a 4-byte state string, and there's still a lot more overhead than > state stored in the pickles. Seems so. > > ... > > Here's a fairly simple minded patch to the pickling side of pickle.py: > > it seems to save about 6 bytes per object in the good cases. > > > > with: > > list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj > > > > without: > > list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj > > The "without" number is most curious. When I run the checked-in code, I get > > list of 100 dates via C -- 1533 bytes, 15.33 bytes/obj > > This was w/ CVS Python, Win2K and MSVC6. Ah! You must have fiddled the > script to use pickle instead of cPickle. Yep. > I get 1871 bytes then. I'm surprised they're so different. I know cPickle does some tricks if refcounts are 1 that aren't duplicated in pickle.py. > > I'm not going to pursue this further unless someone thinks it's a > > worthwhile move. > > I don't personally have large pickles of homogeneous lists, so hard to say. Me neither. > The C implementation of pickle would also need to be fiddled. That was part of "pursuing this further" (as is, e.g., unpickling support). > Cutting the pickle overheads for instances of new-style classes > would more clearly be worthwhile. Hard to see how to do that. You could reduce the overhead by giving _datetime_unpickler a shorter name :-) In general, though, any improvements we can add here are likely to be blown away by ramming the pickle through zlib, so I'm not sure this is worth worrying about too much. Cheers, M. -- And then the character-only displays went away (leading to increasingly silly graphical effects and finally to ads on web pages). -- John W. Baxter, comp.lang.python [View Less]

1 0

RE: [Python-Dev] Re: New and Improved Import Hooks
by Delaney, Timothy Dec. 4, 2002

Dec. 4, 2002

> From: Shane Holloway (IEEE) [mailto:shane.holloway@ieee.org] > > Expanding upon Tim > Delaney's desires to include importing from tar files and internet > references, Hey - I didn't say *I* wanted this (they came from other posts ;) but it's an obvious extension ... > consider being able to extend this mechanism to importing > from an XML document, database of modules, bz2 files, and > even encrypted > collections, As are all of these. I think if the import … [View More]

1 0

Re: New and Improved Import Hooks
by Shane Holloway (IEEE) Dec. 4, 2002

Dec. 4, 2002

As it seems I am supposed to introduce myself on my first post, I am a professional software engineer in Colorado, USA, blessed with the chance to implement my solutions in Python. I've a fairly diverse background in languages, but the most experience with C++. I keep working on the RuneBlade project <http://www.runeblade.com>, which helps me deliver my Python solutions just that much faster. Seaching for a better language, Python has satisfied my thirst, seeming to combine my … [View More]

1 0

RE: [Python-Dev] New and Improved Import Hooks
by Delaney, Timothy Dec. 3, 2002

Dec. 3, 2002

> From: Just van Rossum [mailto:just@letterror.com] > > Regarding sys.import_hooks, I see three possibilities: > > #1 > for p in sys.path: > for hook in sys.import_hooks: > hook(p, ...) > > #2 > for hook in sys.import_hooks: > for p in sys.path: > hook(p, ...) > > #3 > for hook in sys.import_hooks: > # the hook can traversing sys.path if needed. > hook(...) It seems to … [View More]

1 0

Re: New and Improved Import Hooks
by Chermside, Michael Dec. 3, 2002

Dec. 3, 2002

> > 1. Your import hooks seem to act at a level "above" sys.path, > > in the sense that the normal sys.path search becomes just > > another hook. With that in mind, I don't see a way for a user > > to add a zipfile in the *middle* of sys.path. > > True. There would be ways around that, but it's currently not easy. I don't feel > this is a particularly important feature, though. Hmm... Not sure I agree. I think this is one area where we'd be wise to … [View More]

1 0