Mailman 3 December 2002 - Python-Dev

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 05 Dec '02

05 Dec '02

From: Guido van Rossum [mailto:guido@python.org] > I still think I'd like sys.path to be a real list of real strings > (not necessarily representing files or directories though), with > a configurable set of handlers that are given a chance to deal > with each string in turn. That's exactly what I am arguing for. Jim Ahlstrom's existing patch makes this happen, for the extremely specific case of zip files only. Gordon McMillan's iu.py code offers this in a general way, but in Python - and hence with a (fairly small!) efficiency hit, and no obvious way of writing hooks in C. I believe that taking Gordon's design and implementing it in C, along with a standardised zip import hook, satisfies both Just's requirements and the requirements of PEP 273. But if it's not possible to do this in the available timescale, we should just go with Jim's patch for now (and maybe add iu.py to the standard library, as an interim pure-python solution), and look at import hooks in the longer term. I see no obvious value in going through the design process again, from scratch. Paul.

2 1

Re: [Python-Dev] New Import Hook & New Zip Importer
by Moore, Paul 05 Dec '02

05 Dec '02

On Thu, 5 Dec 2002 02:00:23 +0100, in you wrote: >Phew, it works. > >The attached tgz file contains: > zipper.patch: patches setup.py, Python/import.c, Python/importdl.h > zipimport.c: zipimport module, move to Modules/ > >Do I need to mention it's a rough patch <wink>? How do I build on Windows? I don't know what's needed to build the zipimport module... (Sorry, I'm not being awkward here, I have no idea how to set up the necessary Visual Studio stuff so that I can build it - and someone will need to, before the patch is complete). >import.c changes: > - Allow arbitrary objects on sys.path. I still have reservations about this. Code will break... (if you mean "string subclasses", then I feel slightly better about it) > - If a path entry ends in ".zip", replace that entry with > a zipimporter instance. (obviously, the zipimport module > must be available for this to work.) This seems like a gross hack. How do I write a similar hook for strings ending in ".tar"? This isn't a YAGNI, it's a design constraint which I don't believe is reasonable. > - The search algo is: > [handle builtin modules and frozen] > for p on sys.path: > if hasattr(p, "find_module"): > handler = p.find_module(name) > if handler is None: > continue > if isinstance(p, str): > [do builtin import] Who handles checking for pyc/pyo, looking at timestamps, saving compiled source (if possible) etc? Note that Jim Ahlstrom's patch modified the search order so that *both* of pyc and pyo were considered. (This is to allow python -O runs to get compiled modules from a zipfile when only .pyc were available in the zipfile. There was a discussion at the time about why this was good - I can't recall the details just now). Your hooks don't seem to support modified search orders like this. >import.c todo/notes: > - clean up, refactor, debug, document Too right! I don't see any benefits in terms of code readability for this patch (compared to Jim's code). In both cases, import.c is still pretty incomprehensible :-) That's about all the comments I can make, without building the code and trying it out. As a point of reference, could you possibly try the benchmark code I sent you? If for no other reason than to see if it runs under your patch... Paul.

2 1

New Import Hook & New Zip Importer
by Just van Rossum 05 Dec '02

05 Dec '02

Phew, it works. The attached tgz file contains: zipper.patch: patches setup.py, Python/import.c, Python/importdl.h zipimport.c: zipimport module, move to Modules/ Do I need to mention it's a rough patch <wink>? import.c changes: - Allow arbitrary objects on sys.path. - If a path entry ends in ".zip", replace that entry with a zipimporter instance. (obviously, the zipimport module must be available for this to work.) - The search algo is: [handle builtin modules and frozen] for p on sys.path: if hasattr(p, "find_module"): handler = p.find_module(name) if handler is None: continue if isinstance(p, str): [do builtin import] zipimport.c implements: - The zipimporter object; explicit use example: sys.path.append(zipimport.zipimporter("myarchive.zip")) this is effectively equivalent to sys.path.append("myarchive.zip") - the zipimporter has two methods: find_module(subname) -> None or self load_module(fullname) -> module import.c todo/notes: - clean up, refactor, debug, document The patch adds two new static functions, find_module2() and load_module2(). Ideally these should _replace_ find_module and load_module, but since the signature of the new ones is different I decided to keep the originals (which are reimplemented to call the new ones) to keep the patch small for easier review. zipimport.c todo: - rework the zipimporter class as a subclass of str - doco - add a get_source(fullname) method (linecache.py could use that!) Have fun! Just

2 3

Mac OSX issues
by Guido van Rossum 05 Dec '02

05 Dec '02

I have temporary access to a Mac OSX box. I find the following problems: - test_re crashes unless I do ulimit -s 2000 -- haven't tried other values, but the default of 512 (KB) is insufficient. I know the README file explains this, but now that I've experienced this myself, I wonder if we shouldn't hack main() to increase the stack size to 2 MB, inside an #ifdef darwin or something like that. - test_socket fails with errno 3, 'Unknown server error' on a socket.gethostbyaddr() call. Does anybody know what that is about? - test_locale fails with a complaint about "1,024" vs. "1024". Since this is probably a libc bug (though wouldn't this also occur on other BSD systems?) I can live with it. - test_largefile takes a *very* long time. Perhaps it actually creates a truly large file (the Mac OSX filesystem is case insensitive, so I suppose it may be so different that it doesn't support files with holes in them). Maybe the test should disable itself (or part of itself) unless a specific resource is requested? --Guido van Rossum (home page: http://www.python.org/~guido/)

10 18

Can someone look at dummy_thread (#622537)?
by Brett Cannon 05 Dec '02

05 Dec '02

There is a slight chance some of you remember the discussion that Zack Weinberg brought up here back in September about his experiences in rewriting ``tempfile``. One of the things he brought up was having to write his own fake lock so that the code would work when ``thread`` was not available. Guido suggested writing a module called ``dummy_thread`` that was API compatible with ``thread`` but was available on all platforms; he said he would check it in if it was written. Well, I wrote it. But Guido has told me that he is too busy to check it in and so I am emailing you guys to see if someone can look at it and check it in. I would like to get it in before 2.3a so as to get the extra testing the alpha will get. The patch is #622537 ( http://www.python.org/sf/622537 ). It is complete to the best of my abilities. I have docs and the module and the tests and such. Should be pretty straight-forward to apply assuming I didn't muck up the docs (first attempt at doing any docs from scratch). The only thing beyond quality of code and documentation that might have to be dealt with is how far to integrate ``dummy_thread`` into the stdlib. I have a patch to have ``Queue``, ``threading``, and ``tempfile`` all use ``dummy_thread`` when ``thread`` is not available. Now I tested all of them with and without ``thread`` available and they all pass. But of course I wonder if this will break any code. The problem is that since ``dummy_thread`` basically just executes thread calls serially there is a problem when sommething like blocking I/O is used that blocks waiting for another thread (test_asynchat and test_socket do that, but they require ``thread`` so there is no issue there outright). So there is a chance that some code, when trying to run using ``threading`` will lock up. But I don't think this is an issue. I say in the docs that if you want to guarantee true threading to run ``import thread; del thread`` before importing ``threading``. But the main reason I don't think it will be an issue is that all other code that runs fine (but slowly) using ``dummy_thread`` now can be run where threads are not available. And those programs where deadlock would occur do not lose or gain anything since they couldn't run on a platform without ``thread`` anyway. The only issue is that if they don't check for ``thread`` someone trying to run the program will have a deadlocked program (although they shouldn't have been trying to run it in the first place). This is the only real questionable thing. Obviously the module can be checked in with no problem without touching the other modules (although ``Queue`` and ``tempfile`` should be patched regardless) if this seems to dicey to do. -Brett

5 17

PEP 301: Package Index and Metadata for Distutils
by Richard Jones 05 Dec '02

05 Dec '02

I've implemented all I believe is necessary for the package index to be pronounced accepted or not. This includes the last few missing pieces: - library reference patch against latest CVS - distutils/dist.py patch against latest CVS - use of distutils.version.LooseVersion sorting in the index database Where to now? Is there more to do? Should I stop now because it's not worth continuing? Richard

2 2

extension module .o files wind up in the wrong place
by Skip Montanaro 05 Dec '02

05 Dec '02

I notice that when building Python in a subdirectory of the src dir that the .o files for extension modules are written in the wrong directory. Graphically, if my source tree looks like head/dist/src Makefile.pre.in /Modules /Python ... build Makefile /Modules /Python ... If I run cd build ../configure --prefix=/Users/skip/local make extension module .o files wind up in .../src/Modules instead of .../src/build/Modules. I don't have time to look into this right now. I only just noticed it while I was trying to do parallel builds in build.dbg and build.opt directories. The debug .o files got written first, so when the optimized make came along, there were already .o files, which had been compiled with --with-pydebug, got linked improperly. Skip

5 7

Make universal newlines non-optional
by martin＠v.loewis.de 05 Dec '02

05 Dec '02

At the time the universal newline code was committed, there was discussion whether or not it should be optional. I'd like to propose that it doesn't need to be optional: having a configuration option for it complicates the build documentation, and the space savings that one may gain from disabling it are minimal. So I volunteer to remove the #ifdef machinery around it. Any objections? Regards, Martin

3 4

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 04 Dec '02

04 Dec '02

From: Just van Rossum [mailto:just@letterror.com] > I think this is more or less the same as iu.py, > except that there's no need for a "shadowpath" Thinking further, what is the advantage of your code over iu.py, if this is true? Performance? Do you have figures? Paul. PS I attach a small test script I used to get a feel for the performance of Jim's patch. Unzip this file, then do cd test zip -0r ../test0 * zip -9r ../test9 * Create a copy of the standard library as Lib, and do cd Lib zip -0r ../stdlib0 * zip -9r ../stdlib9 * Then run test.py. Figures for me, with CVS + Jim's patch, on a 450MHz Athlon with 256M RAM running Windows 2000, were: Lib [[['Lib\\email']]]: 0.309 stdlib0.zip [[['stdlib0.zip\\email']]]: 0.866 stdlib9.zip [[['stdlib9.zip\\email']]]: 1.154 test [[test\testCR.pyc]]: 0.344 test0.zip [[test0.zip\testCR.py]]: 0.204 test9.zip [[test9.zip\testCR.py]]: 0.348 So import from zip is noticeably slower than directory imports on big zips, but *faster* on small ones. Compressed zipfiles are slower than uncompressed.

2 1

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul 04 Dec '02

04 Dec '02

From: Just van Rossum [mailto:just@letterror.com] > In the meantime things have changed: death to sys.import_hooks, > and yay to arbitrary objects on sys.path. I think this is more > or less the same as iu.py, except that there's no need for a > "shadowpath" (if not a string, the sys.path item _is_ the > "owner"). How do I put a zipfile on my PYTHONPATH environment variable? Paul.

2 1