
[Jeremy on python-checkins list, PEP 283: Python 2.3 release schedule]
I haven't participated in the discussion of PEP 273, IIRC it was mostly about implementation details... Wouldn't it be the right time now, instead of complicating the builtin import mechanism further, to simplify the builtin import code, and use it as the foundation of a Python coded implementation - imputil, or better Gordon's iu.py, or whatever? Thomas

Thomas Heller wrote:
This would be nice to have, but how do you bootstrap the importer if it's written in Python ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

From: "M.-A. Lemburg" <mal@lemburg.com>
Have you looked at imputil? It bootstraps itself only from builtin modules (which may be the only mechanism to be in the core). Probably everything else, even packages can be implemented outside. How did ni do it? Also I think Gordon's rimport and aimport are good ideas. Thomas

Thomas Heller wrote:
Sure, but for finding imputil itself you still need the C import mechanism. Even worse: if Python can't find imputil (for some reason), it would be completely broken. My only gripe with the existing C implementation is that I would like to have more hooks available. Currently, you have to replace the complete API in order to add new features -- not exactly OO :-/ BTW, how is progress on the ZIP import patch doing ? Perhaps Jim should just check in what he has so that the code gets a little more code review... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

M.-A. Lemburg wrote:
The other objection raised at the time was the possible slow down of imports. I think the existing C module search code is basically good, although I wouldn't mind moving module import into a Python method. But since the C works, I have little motivation to replace it.
My code uses os.listdir to cache directory contents, but defers its use until the os module can be imported using the C import code. I think a similar trick could be used to replace imports with a new module. This would make it easy to replace imports. But this would not make it easy to add features unless a module were available which implemented the current import semantics in Python.
The code is "done" has been in Source Forge patch 492105 for some time. I am leaving for Panama tomorrow for 8 days, so if I seem to disappear, that's why. I would be happy to work hard on this after I get back, because I think it is an important addition for Python. JimA

On 28 Feb 2002 at 9:44, James C. Ahlstrom wrote:
The other objection raised at the time was the possible slow down of imports.
imputil was 30 to 40% slower than C import. iu is about 10 to 15% slower under normal usage, but can be faster if you use archives and arrange sys.path intelligently.
It works because its implementation is the definition of what works. Note that while the import namespace (pkg.submodule.module) is mapped to the filesystem, the two namespaces are not isomorphic.
A win over some threshold of number of hits on that directory; a loss under that threshold.
The only incompatibility I'm aware of in iu.py is that it doesn't have a import lock. -- Gordon http://www.mcmillan-inc.com/

Gordon McMillan wrote:
I think I can add iu.py as the standard Python importer to my import.c patches. That is, if iu.py can be imported (using C), then it takes over imports. Note that the C code changes to import.c are still required. Also note that iu.py may be in a zip file, and so the import.c changes are still required.
Exactly correct. It is tradeoff between the OS caching directory hits from fopen() versus using a Python cache and os.listdir(). Dramatic gains are obtained when importing from network file systems, an important case. JimA

On 28 Feb 2002 at 12:16, James C. Ahlstrom wrote:
Thanks, but I don't want iu.py to be used instead of c import in normal Python installations. I'll use it that way in Installer, but since that's an embedding app, it's not hard to bootstrap. In the context of python-dev, iu is, I think, useful because it (a) emulates nearly exactly Python's import rules and (b) it does so in a nicely OO framework with some interesting facilities. In other words, as a model of what some future revamp of c import might be. -- Gordon http://www.mcmillan-inc.com/

"James C. Ahlstrom" wrote:
Great ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Tue, 12 Mar 2002 16:59:05 -0500 "James C. Ahlstrom" <jim@interet.com> wrote:
Jim, I was actually asking a simpler question: Can you change the PEP to point to the most recent implementation patch? Even if the patch doesn't apply, I'd like to take a quick look at how you implemented it. Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it. Jeremy

Jeremy Hylton wrote:
I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it.
The updated patch is in file jim.diff on SourceForge 492105. Also, Kalle Svensson send me his version of the patch, and it agrees. Thanks Kalle. JimA

I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Done. --Guido van Rossum (home page: http://www.python.org/~guido/)

[M.-A. Lemburg]
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
The response to "let's revamp C import" is "Oh no, we need a chance to play with it in Python first." [Thomas Heller]
True when Greg wrote it, but strop is now depecrated, and not necessarily builtin. It's still the best route, because strop has no dependencies, while string does.
If Python can't find the std lib, it's broken. No change there.
My only gripe with the existing C implementation is that I would like to have more hooks available.
All the more reason to try it in Python first. There's never been agreement about what hooks should be available. The import-sig was founded so ihooks defenders could hash it out with imputil defenders (the ihooks camp has never said a word). It's my observation that most import hacks these days are really namespace hacks anyway (that is, they do a relatively normal import, and then alter the way it's exposed so that "replace dots with slashes and look in the filesystem" no longer applies). -- Gordon http://www.mcmillan-inc.com/

Gordon> [Thomas Heller] >> > Have you looked at imputil? It bootstraps itself only from builtin >> > modules (which may be the only mechanism to be in the core). Gordon> True when Greg wrote it, but strop is now depecrated, and not Gordon> necessarily builtin. It's still the best route, because strop Gordon> has no dependencies, while string does. What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now. Skip

On 28 Feb 2002 at 10:15, Skip Montanaro wrote:
Oops, you're right. iu doesn't use strop. Just sys, imp and marshal (and optionally Win32api if required and found). -- Gordon http://www.mcmillan-inc.com/

Skip Montanaro wrote:
The real problem isn't the string module, it is the os module. Any importer will need this. The usual hack is to duplicate its logic in the my_importer module. That is, the selection of the correct builtin os functions. And MAL's point that you need a C importer to import your Python importer is inescapable. And suppose the whole Python library is in a zip file? You must have additional C code to extract and load your Python importer as well as the modules it imports. It seems to me that the correct solution is to use the C importer to import the my_importer Python module, plus all the imports that my_importer needs. Then you switch to resolving imports with my_importer.py. Something like this is already in my import.c patch. I don't think this discussion should hold up installing my zip import patches. I believe these patches are required, and can be the basis of a subsequent patch to add an external Python importer. JimA

On 28 Feb 2002 at 12:02, James C. Ahlstrom wrote:
getpath.c has to invent the same filesystem primitives, since it runs before builtins are loaded.
And MAL's point that you need a C importer to import your Python importer is inescapable.
Everybody has the same bootstrap problem.
Right. Primitives have to come from somewhere.
Which is what almost everybody does, the exception being macPython. They use resources a lot, and most of the import extensions are built in at a very low level.
I don't think this discussion should hold up installing my zip import patches.
Not at all. Getting zip files onto sys.path is a very good thing. -- Gordon http://www.mcmillan-inc.com/

[bootstrapping an importer]
Couldn't this be solved in the same way that made the string module obsolete? I mean, move the functionality from the os module into methods of file objects (or class methods of the file type)? Example: file.stat(pathname) instead of os.stat(pathname) Thomas

Cool -- for Python 3000. Write a PEP! --Guido van Rossum (home page: http://www.python.org/~guido/)

Have a look at the code. It no longer import strop -- it uses string methods now. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Gordon McMillan wrote:
I think you misunderstood my request: I *don't* want to revamp import.c, I would just like some extra hooks to be able to only replace those few parts which I'd like to extend from time to time, e.g. instead of replacing the complete __import__ machinery, it would be nice to have a callback hook in the finder and another one in the module loader. All this has nothing to do with the PEP, though :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

M.-A. Lemburg wrote:
I have a some rough code that does this. I've stuck it on my web site at: http://arctrix.com/nas/python/cimport-20020228.tar.gz if anyone is interested. I found that for my application (importing .ptl modules that need to be compiled with a different compiler), imputil did not have the right kind of hooks. ihooks was better but still kunky and slow. Neil

On Thu, 28 Feb 2002, M.-A. Lemburg wrote:
It might be time to consider, rather than a special case for zip files only, adding an extensible import mechanism ( something like the protocol or mime-type handlers for browsers ). If there's a zipfile in sys.path, then import calls the zipfile handler to search it, if there's a URL in the path, it calls a handler for that, etc. ( Maybe even a url for some sort of directory service that finds the module for you. ) -- Steve [Obviously thinking about TimBL's talk...]

Thomas Heller wrote:
This would be nice to have, but how do you bootstrap the importer if it's written in Python ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

From: "M.-A. Lemburg" <mal@lemburg.com>
Have you looked at imputil? It bootstraps itself only from builtin modules (which may be the only mechanism to be in the core). Probably everything else, even packages can be implemented outside. How did ni do it? Also I think Gordon's rimport and aimport are good ideas. Thomas

Thomas Heller wrote:
Sure, but for finding imputil itself you still need the C import mechanism. Even worse: if Python can't find imputil (for some reason), it would be completely broken. My only gripe with the existing C implementation is that I would like to have more hooks available. Currently, you have to replace the complete API in order to add new features -- not exactly OO :-/ BTW, how is progress on the ZIP import patch doing ? Perhaps Jim should just check in what he has so that the code gets a little more code review... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

M.-A. Lemburg wrote:
The other objection raised at the time was the possible slow down of imports. I think the existing C module search code is basically good, although I wouldn't mind moving module import into a Python method. But since the C works, I have little motivation to replace it.
My code uses os.listdir to cache directory contents, but defers its use until the os module can be imported using the C import code. I think a similar trick could be used to replace imports with a new module. This would make it easy to replace imports. But this would not make it easy to add features unless a module were available which implemented the current import semantics in Python.
The code is "done" has been in Source Forge patch 492105 for some time. I am leaving for Panama tomorrow for 8 days, so if I seem to disappear, that's why. I would be happy to work hard on this after I get back, because I think it is an important addition for Python. JimA

On 28 Feb 2002 at 9:44, James C. Ahlstrom wrote:
The other objection raised at the time was the possible slow down of imports.
imputil was 30 to 40% slower than C import. iu is about 10 to 15% slower under normal usage, but can be faster if you use archives and arrange sys.path intelligently.
It works because its implementation is the definition of what works. Note that while the import namespace (pkg.submodule.module) is mapped to the filesystem, the two namespaces are not isomorphic.
A win over some threshold of number of hits on that directory; a loss under that threshold.
The only incompatibility I'm aware of in iu.py is that it doesn't have a import lock. -- Gordon http://www.mcmillan-inc.com/

Gordon McMillan wrote:
I think I can add iu.py as the standard Python importer to my import.c patches. That is, if iu.py can be imported (using C), then it takes over imports. Note that the C code changes to import.c are still required. Also note that iu.py may be in a zip file, and so the import.c changes are still required.
Exactly correct. It is tradeoff between the OS caching directory hits from fopen() versus using a Python cache and os.listdir(). Dramatic gains are obtained when importing from network file systems, an important case. JimA

On 28 Feb 2002 at 12:16, James C. Ahlstrom wrote:
Thanks, but I don't want iu.py to be used instead of c import in normal Python installations. I'll use it that way in Installer, but since that's an embedding app, it's not hard to bootstrap. In the context of python-dev, iu is, I think, useful because it (a) emulates nearly exactly Python's import rules and (b) it does so in a nicely OO framework with some interesting facilities. In other words, as a model of what some future revamp of c import might be. -- Gordon http://www.mcmillan-inc.com/

"James C. Ahlstrom" wrote:
Great ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Tue, 12 Mar 2002 16:59:05 -0500 "James C. Ahlstrom" <jim@interet.com> wrote:
Jim, I was actually asking a simpler question: Can you change the PEP to point to the most recent implementation patch? Even if the patch doesn't apply, I'd like to take a quick look at how you implemented it. Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it. Jeremy

Jeremy Hylton wrote:
I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Of course, updating the patch is necessary to get it accepted. So it would be great if you could update it.
The updated patch is in file jim.diff on SourceForge 492105. Also, Kalle Svensson send me his version of the patch, and it agrees. Thanks Kalle. JimA

I can't seem to update the PEP. Could someone please change the PEP to reference SourceForge patch 492105?
Done. --Guido van Rossum (home page: http://www.python.org/~guido/)

[M.-A. Lemburg]
This would be nice to have, but how do you bootstrap the importer if it's written in Python ?
The response to "let's revamp C import" is "Oh no, we need a chance to play with it in Python first." [Thomas Heller]
True when Greg wrote it, but strop is now depecrated, and not necessarily builtin. It's still the best route, because strop has no dependencies, while string does.
If Python can't find the std lib, it's broken. No change there.
My only gripe with the existing C implementation is that I would like to have more hooks available.
All the more reason to try it in Python first. There's never been agreement about what hooks should be available. The import-sig was founded so ihooks defenders could hash it out with imputil defenders (the ihooks camp has never said a word). It's my observation that most import hacks these days are really namespace hacks anyway (that is, they do a relatively normal import, and then alter the way it's exposed so that "replace dots with slashes and look in the filesystem" no longer applies). -- Gordon http://www.mcmillan-inc.com/

Gordon> [Thomas Heller] >> > Have you looked at imputil? It bootstraps itself only from builtin >> > modules (which may be the only mechanism to be in the core). Gordon> True when Greg wrote it, but strop is now depecrated, and not Gordon> necessarily builtin. It's still the best route, because strop Gordon> has no dependencies, while string does. What do strop or string provide that string methods don't? It's likely that if you needed to import either in the past, you don't need to now. Skip

On 28 Feb 2002 at 10:15, Skip Montanaro wrote:
Oops, you're right. iu doesn't use strop. Just sys, imp and marshal (and optionally Win32api if required and found). -- Gordon http://www.mcmillan-inc.com/

Skip Montanaro wrote:
The real problem isn't the string module, it is the os module. Any importer will need this. The usual hack is to duplicate its logic in the my_importer module. That is, the selection of the correct builtin os functions. And MAL's point that you need a C importer to import your Python importer is inescapable. And suppose the whole Python library is in a zip file? You must have additional C code to extract and load your Python importer as well as the modules it imports. It seems to me that the correct solution is to use the C importer to import the my_importer Python module, plus all the imports that my_importer needs. Then you switch to resolving imports with my_importer.py. Something like this is already in my import.c patch. I don't think this discussion should hold up installing my zip import patches. I believe these patches are required, and can be the basis of a subsequent patch to add an external Python importer. JimA

On 28 Feb 2002 at 12:02, James C. Ahlstrom wrote:
getpath.c has to invent the same filesystem primitives, since it runs before builtins are loaded.
And MAL's point that you need a C importer to import your Python importer is inescapable.
Everybody has the same bootstrap problem.
Right. Primitives have to come from somewhere.
Which is what almost everybody does, the exception being macPython. They use resources a lot, and most of the import extensions are built in at a very low level.
I don't think this discussion should hold up installing my zip import patches.
Not at all. Getting zip files onto sys.path is a very good thing. -- Gordon http://www.mcmillan-inc.com/

[bootstrapping an importer]
Couldn't this be solved in the same way that made the string module obsolete? I mean, move the functionality from the os module into methods of file objects (or class methods of the file type)? Example: file.stat(pathname) instead of os.stat(pathname) Thomas

Cool -- for Python 3000. Write a PEP! --Guido van Rossum (home page: http://www.python.org/~guido/)

Have a look at the code. It no longer import strop -- it uses string methods now. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Gordon McMillan wrote:
I think you misunderstood my request: I *don't* want to revamp import.c, I would just like some extra hooks to be able to only replace those few parts which I'd like to extend from time to time, e.g. instead of replacing the complete __import__ machinery, it would be nice to have a callback hook in the finder and another one in the module loader. All this has nothing to do with the PEP, though :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

M.-A. Lemburg wrote:
I have a some rough code that does this. I've stuck it on my web site at: http://arctrix.com/nas/python/cimport-20020228.tar.gz if anyone is interested. I found that for my application (importing .ptl modules that need to be compiled with a different compiler), imputil did not have the right kind of hooks. ihooks was better but still kunky and slow. Neil

On Thu, 28 Feb 2002, M.-A. Lemburg wrote:
It might be time to consider, rather than a special case for zip files only, adding an extensible import mechanism ( something like the protocol or mime-type handlers for browsers ). If there's a zipfile in sys.path, then import calls the zipfile handler to search it, if there's a URL in the path, it calls a handler for that, etc. ( Maybe even a url for some sort of directory service that finds the module for you. ) -- Steve [Obviously thinking about TimBL's talk...]
participants (9)
-
Gordon McMillan
-
Guido van Rossum
-
James C. Ahlstrom
-
Jeremy Hylton
-
M.-A. Lemburg
-
Neil Schemenauer
-
Skip Montanaro
-
Steven Majewski
-
Thomas Heller