[Jeremy on python-checkins list, PEP 283: Python 2.3 release schedule]
Planned features for 2.3 Here are a few PEPs that I know to be under consideration. [...] S 273 Import Modules from Zip Archives Ahlstrom
I haven't participated in the discussion of PEP 273, IIRC it was mostly about implementation details... Wouldn't it be the right time now, instead of complicating the builtin import mechanism further, to simplify the builtin import code, and use it as the foundation of a Python coded implementation - imputil, or better Gordon's iu.py, or whatever? Thomas
Thomas Heller wrote:
[Jeremy on python-checkins list, PEP 283: Python 2.3 release schedule]
Planned features for 2.3 Here are a few PEPs that I know to be under consideration. [...] S 273 Import Modules from Zip Archives Ahlstrom
I haven't participated in the discussion of PEP 273, IIRC it was mostly about implementation details...
Wouldn't it be the right time now, instead of complicating the builtin import mechanism further, to simplify the builtin import code, and use it as the foundation of a Python coded implementation - imputil, or better Gordon's iu.py, or whatever?
This would be nice to have, but how do you bootstrap the importer if it's written in Python ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
From: "M.-A. Lemburg" <mal@lemburg.com>
Thomas Heller wrote:
Wouldn't it be the right time now, instead of complicating the builtin import mechanism further, to simplify the builtin import code, and use it as the foundation of a Python coded implementation - imputil, or better Gordon's iu.py, or whatever?
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
Have you looked at imputil? It bootstraps itself only from builtin modules (which may be the only mechanism to be in the core). Probably everything else, even packages can be implemented outside. How did ni do it? Also I think Gordon's rimport and aimport are good ideas. Thomas
Thomas Heller wrote:
From: "M.-A. Lemburg" <mal@lemburg.com>
Thomas Heller wrote:
Wouldn't it be the right time now, instead of complicating the builtin import mechanism further, to simplify the builtin import code, and use it as the foundation of a Python coded implementation - imputil, or better Gordon's iu.py, or whatever?
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
Have you looked at imputil? It bootstraps itself only from builtin modules (which may be the only mechanism to be in the core).
Sure, but for finding imputil itself you still need the C import mechanism. Even worse: if Python can't find imputil (for some reason), it would be completely broken. My only gripe with the existing C implementation is that I would like to have more hooks available. Currently, you have to replace the complete API in order to add new features -- not exactly OO :-/ BTW, how is progress on the ZIP import patch doing ? Perhaps Jim should just check in what he has so that the code gets a little more code review... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
M.-A. Lemburg wrote:
Sure, but for finding imputil itself you still need the C import mechanism. Even worse: if Python can't find imputil (for some reason), it would be completely broken.
The other objection raised at the time was the possible slow down of imports. I think the existing C module search code is basically good, although I wouldn't mind moving module import into a Python method. But since the C works, I have little motivation to replace it.
My only gripe with the existing C implementation is that I would like to have more hooks available. Currently, you have to replace the complete API in order to add new features -- not exactly OO :-/
My code uses os.listdir to cache directory contents, but defers its use until the os module can be imported using the C import code. I think a similar trick could be used to replace imports with a new module. This would make it easy to replace imports. But this would not make it easy to add features unless a module were available which implemented the current import semantics in Python.
BTW, how is progress on the ZIP import patch doing ? Perhaps Jim should just check in what he has so that the code gets a little more code review...
The code is "done" has been in Source Forge patch 492105 for some time. I am leaving for Panama tomorrow for 8 days, so if I seem to disappear, that's why. I would be happy to work hard on this after I get back, because I think it is an important addition for Python. JimA
On 28 Feb 2002 at 9:44, James C. Ahlstrom wrote:
The other objection raised at the time was the possible slow down of imports.
imputil was 30 to 40% slower than C import. iu is about 10 to 15% slower under normal usage, but can be faster if you use archives and arrange sys.path intelligently.
I think the existing C module search code is basically good, although I wouldn't mind moving module import into a Python method. But since the C works, I have little motivation to replace it.
It works because its implementation is the definition of what works. Note that while the import namespace (pkg.submodule.module) is mapped to the filesystem, the two namespaces are not isomorphic.
My code uses os.listdir to cache directory contents, but defers its use until the os module can be imported using the C import code.
A win over some threshold of number of hits on that directory; a loss under that threshold.
I think a similar trick could be used to replace imports with a new module. This would make it easy to replace imports. But this would not make it easy to add features unless a module were available which implemented the current import semantics in Python.
The only incompatibility I'm aware of in iu.py is that it doesn't have a import lock. -- Gordon http://www.mcmillan-inc.com/
Gordon McMillan wrote:
On 28 Feb 2002 at 9:44, James C. Ahlstrom wrote:
The other objection raised at the time was the possible slow down of imports.
imputil was 30 to 40% slower than C import. iu is about 10 to 15% slower under normal usage, but can be faster if you use archives and arrange sys.path intelligently.
I think I can add iu.py as the standard Python importer to my import.c patches. That is, if iu.py can be imported (using C), then it takes over imports. Note that the C code changes to import.c are still required. Also note that iu.py may be in a zip file, and so the import.c changes are still required.
My code uses os.listdir to cache directory contents, but defers its use until the os module can be imported using the C import code.
A win over some threshold of number of hits on that directory; a loss under that threshold.
Exactly correct. It is tradeoff between the OS caching directory hits from fopen() versus using a Python cache and os.listdir(). Dramatic gains are obtained when importing from network file systems, an important case. JimA
On 28 Feb 2002 at 12:16, James C. Ahlstrom wrote:
I think I can add iu.py as the standard Python importer to my import.c patches. That is, if iu.py can be imported (using C), then it takes over imports. Note that the C code changes to import.c are still required. Also note that iu.py may be in a zip file, and so the import.c changes are still required.
Thanks, but I don't want iu.py to be used instead of c import in normal Python installations. I'll use it that way in Installer, but since that's an embedding app, it's not hard to bootstrap. In the context of python-dev, iu is, I think, useful because it (a) emulates nearly exactly Python's import rules and (b) it does so in a nicely OO framework with some interesting facilities. In other words, as a model of what some future revamp of c import might be. -- Gordon http://www.mcmillan-inc.com/
"James C. Ahlstrom" wrote:
M.-A. Lemburg wrote:
BTW, how is progress on the ZIP import patch doing ? Perhaps Jim should just check in what he has so that the code gets a little more code review...
The code is "done" has been in Source Forge patch 492105 for some time.
I am leaving for Panama tomorrow for 8 days, so if I seem to disappear, that's why. I would be happy to work hard on this after I get back, because I think it is an important addition for Python.
Great ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
On Mon, 11 Mar 2002 09:37:46 -0500 "James C. Ahlstrom" <jim@interet.com> wrote:
James C. Ahlstrom wrote:
I am leaving for Panama tomorrow for 8 days, so if I
I am back. If someone could start installing the patches, I can correct problems as they arise.
It looks like the PEP points to an obsolete implementation patch. Could you update it? Jeremy
On Tue, 12 Mar 2002 16:59:05 -0500 "James C. Ahlstrom" <jim@interet.com> wrote:
Jeremy Hylton wrote:
It looks like the PEP points to an obsolete implementation patch. Could you update it?
OK, but it will take a couple days because so much changed. I have downloaded the current CVS tree, and I will work from that.
Jim, I was actually asking a simpler question: Can you change the PEP to point to the most recent implementation patch? Even if the patch doesn't apply, I'd like to take a quick look at how you implemented it. Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it. Jeremy
Jeremy Hylton wrote:
I was actually asking a simpler question: Can you change the PEP to point to the most recent implementation patch? Even if the patch doesn't apply, I'd like to take a quick look at how you implemented it.
I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it.
The updated patch is in file jim.diff on SourceForge 492105. Also, Kalle Svensson send me his version of the patch, and it agrees. Thanks Kalle. JimA
I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Done. --Guido van Rossum (home page: http://www.python.org/~guido/)
[M.-A. Lemburg]
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
The response to "let's revamp C import" is "Oh no, we need a chance to play with it in Python first." [Thomas Heller]
Have you looked at imputil? It bootstraps itself only from builtin modules (which may be the only mechanism to be in the core).
True when Greg wrote it, but strop is now depecrated, and not necessarily builtin. It's still the best route, because strop has no dependencies, while string does.
Sure, but for finding imputil itself you still need the C import mechanism. Even worse: if Python can't find imputil (for some reason), it would be completely broken.
If Python can't find the std lib, it's broken. No change there.
My only gripe with the existing C implementation is that I would like to have more hooks available.
All the more reason to try it in Python first. There's never been agreement about what hooks should be available. The import-sig was founded so ihooks defenders could hash it out with imputil defenders (the ihooks camp has never said a word). It's my observation that most import hacks these days are really namespace hacks anyway (that is, they do a relatively normal import, and then alter the way it's exposed so that "replace dots with slashes and look in the filesystem" no longer applies). -- Gordon http://www.mcmillan-inc.com/
Gordon> [Thomas Heller] >> > Have you looked at imputil? It bootstraps itself only from builtin >> > modules (which may be the only mechanism to be in the core). Gordon> True when Greg wrote it, but strop is now depecrated, and not Gordon> necessarily builtin. It's still the best route, because strop Gordon> has no dependencies, while string does. What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now. Skip
On 28 Feb 2002 at 10:15, Skip Montanaro wrote:
What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now.
Oops, you're right. iu doesn't use strop. Just sys, imp and marshal (and optionally Win32api if required and found). -- Gordon http://www.mcmillan-inc.com/
Skip Montanaro wrote:
Gordon> [Thomas Heller] >> > Have you looked at imputil? It bootstraps itself only from builtin >> > modules (which may be the only mechanism to be in the core).
Gordon> True when Greg wrote it, but strop is now depecrated, and not Gordon> necessarily builtin. It's still the best route, because strop Gordon> has no dependencies, while string does.
What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now.
The real problem isn't the string module, it is the os module. Any importer will need this. The usual hack is to duplicate its logic in the my_importer module. That is, the selection of the correct builtin os functions. And MAL's point that you need a C importer to import your Python importer is inescapable. And suppose the whole Python library is in a zip file? You must have additional C code to extract and load your Python importer as well as the modules it imports. It seems to me that the correct solution is to use the C importer to import the my_importer Python module, plus all the imports that my_importer needs. Then you switch to resolving imports with my_importer.py. Something like this is already in my import.c patch. I don't think this discussion should hold up installing my zip import patches. I believe these patches are required, and can be the basis of a subsequent patch to add an external Python importer. JimA
On 28 Feb 2002 at 12:02, James C. Ahlstrom wrote:
The real problem isn't the string module, it is the os module. Any importer will need this. The usual hack is to duplicate its logic in the my_importer module. That is, the selection of the correct builtin os functions.
getpath.c has to invent the same filesystem primitives, since it runs before builtins are loaded.
And MAL's point that you need a C importer to import your Python importer is inescapable.
Everybody has the same bootstrap problem.
And suppose the whole Python library is in a zip file? You must have additional C code to extract and load your Python importer as well as the modules it imports.
Right. Primitives have to come from somewhere.
It seems to me that the correct solution is to use the C importer to import the my_importer Python module, plus all the imports that my_importer needs. Then you switch to resolving imports with my_importer.py. Something like this is already in my import.c patch.
Which is what almost everybody does, the exception being macPython. They use resources a lot, and most of the import extensions are built in at a very low level.
I don't think this discussion should hold up installing my zip import patches.
Not at all. Getting zip files onto sys.path is a very good thing. -- Gordon http://www.mcmillan-inc.com/
[bootstrapping an importer]
What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now.
The real problem isn't the string module, it is the os module. Any importer will need this. The usual hack is to duplicate its logic in the my_importer module. That is, the selection of the correct builtin os functions.
Couldn't this be solved in the same way that made the string module obsolete? I mean, move the functionality from the os module into methods of file objects (or class methods of the file type)? Example: file.stat(pathname) instead of os.stat(pathname) Thomas
I mean, move the functionality from the os module into methods of file objects (or class methods of the file type)?
Example: file.stat(pathname) instead of os.stat(pathname)
Cool -- for Python 3000. Write a PEP! --Guido van Rossum (home page: http://www.python.org/~guido/)
True when Greg wrote it, but strop is now depecrated, and not necessarily builtin. It's still the best route, because strop has no dependencies, while string does.
Have a look at the code. It no longer import strop -- it uses string methods now. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Gordon McMillan wrote:
[M.-A. Lemburg]
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
The response to "let's revamp C import" is "Oh no, we need a chance to play with it in Python first."
I think you misunderstood my request: I *don't* want to revamp import.c, I would just like some extra hooks to be able to only replace those few parts which I'd like to extend from time to time, e.g. instead of replacing the complete __import__ machinery, it would be nice to have a callback hook in the finder and another one in the module loader. All this has nothing to do with the PEP, though :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
M.-A. Lemburg wrote:
I think you misunderstood my request: I *don't* want to revamp import.c, I would just like some extra hooks to be able to only replace those few parts which I'd like to extend from time to time, e.g. instead of replacing the complete __import__ machinery, it would be nice to have a callback hook in the finder and another one in the module loader.
I have a some rough code that does this. I've stuck it on my web site at: http://arctrix.com/nas/python/cimport-20020228.tar.gz if anyone is interested. I found that for my application (importing .ptl modules that need to be compiled with a different compiler), imputil did not have the right kind of hooks. ihooks was better but still kunky and slow. Neil
On Thu, 28 Feb 2002, M.-A. Lemburg wrote:
My only gripe with the existing C implementation is that I would like to have more hooks available. Currently, you have to replace the complete API in order to add new features -- not exactly OO :-/
It might be time to consider, rather than a special case for zip files only, adding an extensible import mechanism ( something like the protocol or mime-type handlers for browsers ). If there's a zipfile in sys.path, then import calls the zipfile handler to search it, if there's a URL in the path, it calls a handler for that, etc. ( Maybe even a url for some sort of directory service that finds the module for you. ) -- Steve [Obviously thinking about TimBL's talk...]
participants (9)
-
Gordon McMillan -
Guido van Rossum -
James C. Ahlstrom -
Jeremy Hylton -
M.-A. Lemburg -
Neil Schemenauer -
Skip Montanaro -
Steven Majewski -
Thomas Heller