
Dev-er's After numerous false-starts and dead-ends, I've come up with two modules that do what imputil and modulefinder do, but better. Code and detailed descriptions are available here: http://www.mcmillan-inc.com/importhooks.html I would like to propose these (or something quite like them) as replacements for the official versions. The code is quite similar (in fact, the modulefinder code could have been written by subclassing the imputil stuff, but I wrote them the other way 'round). If the charter of the Import-SIG is not as dead as the list is, I would promote the basic structure as a potential model for a reformed import. For now, though, it's enough to consider the code. The differences are too extreme to consider these patches, but the subject hardly seems PEPable so I bring it up here. - Gordon

From: "Gordon McMillan" <gmcm@hypernet.com>
Gordon, I've read your descriptions and IMO this is a great design for a new import-utility and a new module-finder. Some comments: To fully simulate python's behaviour for import, the case of the filename must be checked on windows. I found only two possibilities to do this - the first one is to use os.listdir() which is probably expensive, but it returns filenames with the actual case. Second would be to use Mark's win32api.FindFiles(), but this may not be available. Your metapath models Python's import policy. One part of the policy is that already loaded modules are fetched from sys.modules instead of imported again. Should this behaviour also be modeled by a SysModulesCacheDirector on your metapath? What about reload? Shouldn't a new import-util module also implement a reload-replacement?
It seems noone cares about this. imputil is in the distribution, but is it really 'official'?
Thomas

Thomas,
True. Sigh.
That's already done by the ImportManager as soon as he's got a candidate fqname.
What about reload? Shouldn't a new import-util module also implement a reload-replacement?
Yes, it should.
The entire subject seems to have dropped off radar, after starting out as highly controversial (the Import-SIG was started so the ihooks-partisans could hash it out with the imputil- partisans). Import hacks are more common than ever, but they're all home-grown now. - Gordon

[Gordon]
[Thomas]
You thought of replacing the import hacks by custom import policies implemented with imputil or iu4? No chance, IMO.
That's what ihooks was written for, and nobody ever used it, either.
Several reasons I can think of: - Import hacks have this cool touch...
Ironically, they usually are justified as "making it easier on the user" (where they really mean programmer). Which is crap, because when the programmer has trouble with X.A.fribble(), he goes looking for the source. But there is no X/A.py.
- Import hacks start out to help the developers, but they never get cleaned up for the poor user (pmw being an exception!)
And they help constrain Python to being a developer-only language. The existing import mechanism is poorly understood (just see how many wrong answers an import question on c.l.py gets) and home to some major warts (such as the fact that relative and absolute imports are spelled the same way). Greg wrote imputil as a first step towards fixing those kinds of problems. under-the-spreading-entropy-ly y'rs - Gordon

The existing import mechanism is poorly understood (just see how many wrong answers an import question on c.l.py gets) ...
Note that this question has recently been asked by Michael Abbot on c.l.p, for now there are no answers: Is there up to date documentation for package support in Python 2? Section 6.11 of the "Python Reference Manual" has the following nice quote: [XXX Can't be bothered to spell this out right now; see the URL http://www.python.org/doc/essays/packages.html for more details, also about how the module search works from inside a package.] and the referred URL documents Python 1.5. I presume that nothing significant has changed recently, but it's certainly disconcerting for something as fundamental as module importing to not actually be part of the core language documentation! Thomas

It seems noone cares about this. imputil is in the distribution, but is it really 'official'?
I want to care, but I'm already spread too thin. I don't find 'imputil' particularly official -- but Greg Stein pushed hard to get it in the distribution. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, Oct 05, 2001 at 09:06:32AM -0400, Guido van Rossum wrote:
Don't blame me :-) I thought it was much better than ihooks, and a good number of people wanted to see it in the distro. I wrote imputil to solve some of my problems, and it was a great step forward over the ihooks stuff. It was reasonably clean, very easy to extend, and helped to clarify the existing import and package mechanisms (by virtue of needing to replace them). Gordon picked up imputil and did a lot more work with it than I ever did. At this point, if he says that iu4 is better than imputil, then I'd be a mind to trust that :-) I'm just happy that my imputil was able to show a way for even better improvements. Personally, I don't have a particular attachment to preferring it over Gordon's new iu4. iu4 probably needs a bit wider usage to see whether *other* people can use it like Gordon :-), but it looks like he has documented it quite well. Cheers, -g -- Greg Stein, http://www.lyra.org/

Thomas Heller wrote:
I care about it. I am using imputil to implement "jar" files, zip files of Python libraries. Greg worked hard to get imputil into the distribution, and I appreciate his effort. But I always wondered if imputil was 'official'. Users on c.l.p frequently want to ship a stand-alone Python. The current answer to this is freeze, but this requires a C compiler and some work. A standard jar file implementation would satisfy this need. What I am thinking of is documentation which says something like: "Zip up the Python libraries into pylibs.zip and put that file in the same directory as python[.exe] and everything Just Works." My interest in imputil and ImportManager is to implement jar files, and neither does this well. The problem is always how to get the SpecialImporter.py imported first, and how to hack the imports that the SpecialImporter.py may need to import. In iu4.py we see the use of posix.stat to replace the os module, and the import of only built-in modules sys, imp, and marshal. The same is true of imputil.py. I admit I don't really understand package imports. But I feel they are an unsolved problem, because package authors always seem to write custom ones. A first-class solution would have these features IMHO: 1) Access to libraries in zip files is implemented in C (I volunteer). 2) All Python library modules can be imported by C code, either from directories or zip files. 3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library. 4) If an import of A.B is attempted, and a function A.MyCustomImporter() exists, then that is used instead of the default. 5) More or better ideas from you all. Jim Ahlstrom

From: "James C. Ahlstrom" <jim@interet.com> py2exe basically builds a zip-file containing the libary modules, appends them to a special python.exe. This special exe 'boots' itself by using imputil, and exposes a 'get_code' function into the __main__ namespace. This function is called by a module name contained in the (embedded) zip, and returns the stored bytes. The installed imputil code then creates a module from these bytes. Gordon's installer works very similar, except that he uses a custom archive format (IIRC).
You could take the code from py2exe...
BTW: I remember I having seen a ZipFileImporter using imputil, but cannot find it any more. Am I dreaming? Thomas

Thomas Heller wrote:
Do you know about py2exe and installer? This is a variation of what they do.
I've heard of it, but where is it? I never searched out the URL. I just use freeze. Anyway, I think we something in the core.
BTW: I remember I having seen a ZipFileImporter using imputil, but cannot find it any more. Am I dreaming?
Either Greg Stein or I (I don't remember) wrote a ZipFileImporter for imputil, and that is what I am currently using. Jim

From: "James C. Ahlstrom" <jim@interet.com>
Installer: http://www.mcmillan-inc.com/install1.html Um, BTW: py2exe's code originated from bdist_wininst (the distutils windows installer), which also contains C-code to unpack zip-files: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/distutils/misc/ Thomas

"James C. Ahlstrom" wrote:
Huh ? The builtin mechanism in Python works just fine for packages. I think that custom importers are not very common these days. Still, I do agree that the current implementation is a gross hack and should in the long run be replaced by something more OO-style. For the short-term however, I have a feeling that there are not that many features actually being requested out there -- the most wanted one is certainly the ability to import modules from some form of archive, so perhaps we should start thinking about providing this kind of support in the core ?!
A first-class solution would have these features IMHO: 1) Access to libraries in zip files is implemented in C (I volunteer).
Great !
2) All Python library modules can be imported by C code, either from directories or zip files.
Right.
3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library.
Why ? I don't see a need for that and it would slow down package imports which are becoming increasingly popular as the Python code base grows.
Better postpone this one...
5) More or better ideas from you all.
One question: should these ZIP-archives filenames be placed in sys.path or should Python scan for ZIP-archives within the dirs on sys.path ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

On 05 October 2001, M.-A. Lemburg said:
For the record, I agree with you on all points: there's no need to rewrite package import in Python, but it would be nice to have "import from archive" available in the core. Quite possibly adding this feature would silence 80% of the desire for genreic import hooks. The only import hook I'm familiar with is Quixote, where we have defined a Python dialect called PTL (Python Template Language, used to embed HTML [or other text] in Python). Install Quixote's import hook, and you can import .ptl files just like .py files. It's very handy. Out of curiosity, does anyone know of any other import hooks like this out there -- ie. import something that is not strictly Python. (It sounds like most import hooks/hacks deal with the location of the .py files to import. Quixote doesn't touch that, but it adds the ability to import not-quite-Python source files.)
I think the archive file should be listed in sys.path. Note that my experience of this in Java was largely negative, because Java doesn't have a standard way of putting .class files in the filesystem (AFAIK) -- so everything has to be in a .jar file, and those .jar files can be anywhere you please. So you end up with a mile-long CLASSPATH that's very fragile and forever needing fixing. As long as most Python modules are accessed the ordinary way (files in a directory), then Python won't have a problem. But if somebody makes a Python installation with "stdlib.zip", "distutils.zip", "mxDateTime.zip", etc. etc., then the poor users will be in the same boat as Java users. The main concerns I have with scanning sys.path directories for ZIP files are performance and transparence. (It ain't necessarily obvious where a particular module will come from if sys.path can be searched in two ways.) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ Pointers are Arrays; Code is Data; Time is Money

On Friday 05 October 2001 12:19 pm, Greg Ward wrote:
On 05 October 2001, M.-A. Lemburg said:
The dpython interpreter I wrote this summer would tokenize .dp files that were in the Python path. If the alternative suffix was detected the Python interpreter would use the dpython tokenizer. The dpython tokenizer would generate decimal numbers instead of binary numbers as the default for float and int literals. I didn't use the import hooks. I built the .dp file recognition into the interpreter at the point where it evaluates the suffix of the file name and compairs it to the filedesc struct. When I discussed the implementation of dpython at the ZPUG meeting I joked about extending the concept to add bytecode generators for JavaScript and Visual Basic modules. The work required to add JavaScript would probably be reasonable. It might help get Python used anywhere JavaScript is used. There has also been some interest at NIST and NASA in building modules to parse meta languages such as XML Schema, KIF, and Express and directly generate Python bytecodes.

Hi [Greg Ward]
the latest jython code base (2.1a3) allow to put jar/zip in the sys.path or reference of the form jarfile!pkgdir1/pkgdir2 This are used to set meaningful __path__ attrs for the packages and should allow to deal with code that change __path__ pointing to some relative subdir according to some context property.
I don't get the point :(
I agree. regards.

Greg Ward wrote:
On 05 October 2001, M.-A. Lemburg said:
I think that is correct. At one time there was talk of sys.path containing importer instances. But currently (please check me here) sys.path is purely strings, and these strings must be directory names. Python depends on directories and subdirectories for its import semantics. So any zip archive must support subdirectories to support general imports. Therefore, a string in sys.path which is a zip file is equivalent to hanging a nameless subdirectory at that point. "Nameless" means zip archive files *.pyc occur at the top level, and the zip file may contain relative subdirectories with files sub1/file1.pyc, etc. So a zip file can always substitute for an arbitrary subdirectory tree, and is functionally equivalent to a directory name at the same place in sys.path. JimA

[Greg Ward]
I know of import hooks like that, but in 1000+ emails from Installer users, no one has ever attempted to distribute an app that uses one. If the hook writes out a .pyc, then the hook is pretty much developer-only, anyway.
Huh? Jar files don't date from day 1 - for .class files it works almost like Python.
I think the only difference is that Python goes to considerable effort (and expense) to work out a sys.path before considering PYTHONPATH. - Gordon

"M.-A. Lemburg" wrote:
Well, I looked at import.c and adding zip files with subdirectories will be a mess. The path searches would have to be extended to zip files. Moving out the package code would be a great simplification. Or maybe I am lazy. It doesn't help that I don't really understand packages. Your point about efficiency is a good one though, but I am not sure of the magnitude of the slow down. Anyone else have an opinion? JimA

[James C. Ahlstrom]
Here's my bootstrap code (archive_rt knows how to import from .pyz archives): import archive_rt, iu4 iu4._globalownertypes.insert(0, archive_rt.PYZOwner) iu4.ImportManager().install() Now a .pyz on sys.path will be searched. Nothing hard about doing this for a .zip. MacPython & Jython do it.
Most of what I see is namespace trickery, not custom importers.
With Thomas's changes to PyImport_ImportModuleEx, this is largely taken care of (it gets routed through __import__). So with the above bootstrap code, C code will import from my .pyz as well.
3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library.
The rules aren't documented, but the essentials are pretty straightforward. I'd like to see them changed to draw a distinction between relative imports and absolute imports (since that's a constant source of gotcha's). However, there are import hacks out there that rely on undocumented (and probably accidental) features of implementation.
In iu4, A is welcome to take over all imports below A. If A.__init__ does not do so, the standard mechanism is used.
5) More or better ideas from you all.
- Gordon

Gordon McMillan wrote:
I am not sure if you are saying that code already exists to import from zip archives, or that iu4.py should be used as the basis. I think the best approach is for me to write up a PEP based on the discussions so far, and let everyone criticize. The proposal is only to import from zip archives, not whether to replace imputil.py with iu4.py. I will submit a PEP after I write some code. I often find that my initial design is poor once I implement it. So expect the PEP in a week or so. This won't make it into 2.2, so there is no rush, right? Jim

[Gordon]
Neither. I thought you were referring to the Py 1.5-timeframe problem of C code using PyImport_ImportModule* and bypassing all hooks.
Does "importing from zip archives" really need a PEP? It's already quasi-official in that Jython and MacPython implement it. The problem it exposes (the fact that the C code makes it difficult to do for packages) is certainly a good topic for discussion and is part of the reason Greg wrote imputil and I wrote iu4. - Gordon

Hi.
Personally, I think so, a PEP would be indeed a good thing, mostly to fix the details. AFAIK MacPython and Jython are already different about the details, at least in Jython the feature is still experimental so there is space for changes, don't know in general. regards, Samule Pedroni.

+1. A PEP is the only way to get this to be a standard feature. --Guido van Rossum (home page: http://www.python.org/~guido/)

From: "Gordon McMillan" <gmcm@hypernet.com>
Gordon, I've read your descriptions and IMO this is a great design for a new import-utility and a new module-finder. Some comments: To fully simulate python's behaviour for import, the case of the filename must be checked on windows. I found only two possibilities to do this - the first one is to use os.listdir() which is probably expensive, but it returns filenames with the actual case. Second would be to use Mark's win32api.FindFiles(), but this may not be available. Your metapath models Python's import policy. One part of the policy is that already loaded modules are fetched from sys.modules instead of imported again. Should this behaviour also be modeled by a SysModulesCacheDirector on your metapath? What about reload? Shouldn't a new import-util module also implement a reload-replacement?
It seems noone cares about this. imputil is in the distribution, but is it really 'official'?
Thomas

Thomas,
True. Sigh.
That's already done by the ImportManager as soon as he's got a candidate fqname.
What about reload? Shouldn't a new import-util module also implement a reload-replacement?
Yes, it should.
The entire subject seems to have dropped off radar, after starting out as highly controversial (the Import-SIG was started so the ihooks-partisans could hash it out with the imputil- partisans). Import hacks are more common than ever, but they're all home-grown now. - Gordon

[Gordon]
[Thomas]
You thought of replacing the import hacks by custom import policies implemented with imputil or iu4? No chance, IMO.
That's what ihooks was written for, and nobody ever used it, either.
Several reasons I can think of: - Import hacks have this cool touch...
Ironically, they usually are justified as "making it easier on the user" (where they really mean programmer). Which is crap, because when the programmer has trouble with X.A.fribble(), he goes looking for the source. But there is no X/A.py.
- Import hacks start out to help the developers, but they never get cleaned up for the poor user (pmw being an exception!)
And they help constrain Python to being a developer-only language. The existing import mechanism is poorly understood (just see how many wrong answers an import question on c.l.py gets) and home to some major warts (such as the fact that relative and absolute imports are spelled the same way). Greg wrote imputil as a first step towards fixing those kinds of problems. under-the-spreading-entropy-ly y'rs - Gordon

The existing import mechanism is poorly understood (just see how many wrong answers an import question on c.l.py gets) ...
Note that this question has recently been asked by Michael Abbot on c.l.p, for now there are no answers: Is there up to date documentation for package support in Python 2? Section 6.11 of the "Python Reference Manual" has the following nice quote: [XXX Can't be bothered to spell this out right now; see the URL http://www.python.org/doc/essays/packages.html for more details, also about how the module search works from inside a package.] and the referred URL documents Python 1.5. I presume that nothing significant has changed recently, but it's certainly disconcerting for something as fundamental as module importing to not actually be part of the core language documentation! Thomas

It seems noone cares about this. imputil is in the distribution, but is it really 'official'?
I want to care, but I'm already spread too thin. I don't find 'imputil' particularly official -- but Greg Stein pushed hard to get it in the distribution. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, Oct 05, 2001 at 09:06:32AM -0400, Guido van Rossum wrote:
Don't blame me :-) I thought it was much better than ihooks, and a good number of people wanted to see it in the distro. I wrote imputil to solve some of my problems, and it was a great step forward over the ihooks stuff. It was reasonably clean, very easy to extend, and helped to clarify the existing import and package mechanisms (by virtue of needing to replace them). Gordon picked up imputil and did a lot more work with it than I ever did. At this point, if he says that iu4 is better than imputil, then I'd be a mind to trust that :-) I'm just happy that my imputil was able to show a way for even better improvements. Personally, I don't have a particular attachment to preferring it over Gordon's new iu4. iu4 probably needs a bit wider usage to see whether *other* people can use it like Gordon :-), but it looks like he has documented it quite well. Cheers, -g -- Greg Stein, http://www.lyra.org/

Thomas Heller wrote:
I care about it. I am using imputil to implement "jar" files, zip files of Python libraries. Greg worked hard to get imputil into the distribution, and I appreciate his effort. But I always wondered if imputil was 'official'. Users on c.l.p frequently want to ship a stand-alone Python. The current answer to this is freeze, but this requires a C compiler and some work. A standard jar file implementation would satisfy this need. What I am thinking of is documentation which says something like: "Zip up the Python libraries into pylibs.zip and put that file in the same directory as python[.exe] and everything Just Works." My interest in imputil and ImportManager is to implement jar files, and neither does this well. The problem is always how to get the SpecialImporter.py imported first, and how to hack the imports that the SpecialImporter.py may need to import. In iu4.py we see the use of posix.stat to replace the os module, and the import of only built-in modules sys, imp, and marshal. The same is true of imputil.py. I admit I don't really understand package imports. But I feel they are an unsolved problem, because package authors always seem to write custom ones. A first-class solution would have these features IMHO: 1) Access to libraries in zip files is implemented in C (I volunteer). 2) All Python library modules can be imported by C code, either from directories or zip files. 3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library. 4) If an import of A.B is attempted, and a function A.MyCustomImporter() exists, then that is used instead of the default. 5) More or better ideas from you all. Jim Ahlstrom

From: "James C. Ahlstrom" <jim@interet.com> py2exe basically builds a zip-file containing the libary modules, appends them to a special python.exe. This special exe 'boots' itself by using imputil, and exposes a 'get_code' function into the __main__ namespace. This function is called by a module name contained in the (embedded) zip, and returns the stored bytes. The installed imputil code then creates a module from these bytes. Gordon's installer works very similar, except that he uses a custom archive format (IIRC).
You could take the code from py2exe...
BTW: I remember I having seen a ZipFileImporter using imputil, but cannot find it any more. Am I dreaming? Thomas

Thomas Heller wrote:
Do you know about py2exe and installer? This is a variation of what they do.
I've heard of it, but where is it? I never searched out the URL. I just use freeze. Anyway, I think we something in the core.
BTW: I remember I having seen a ZipFileImporter using imputil, but cannot find it any more. Am I dreaming?
Either Greg Stein or I (I don't remember) wrote a ZipFileImporter for imputil, and that is what I am currently using. Jim

From: "James C. Ahlstrom" <jim@interet.com>
Installer: http://www.mcmillan-inc.com/install1.html Um, BTW: py2exe's code originated from bdist_wininst (the distutils windows installer), which also contains C-code to unpack zip-files: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/distutils/misc/ Thomas

"James C. Ahlstrom" wrote:
Huh ? The builtin mechanism in Python works just fine for packages. I think that custom importers are not very common these days. Still, I do agree that the current implementation is a gross hack and should in the long run be replaced by something more OO-style. For the short-term however, I have a feeling that there are not that many features actually being requested out there -- the most wanted one is certainly the ability to import modules from some form of archive, so perhaps we should start thinking about providing this kind of support in the core ?!
A first-class solution would have these features IMHO: 1) Access to libraries in zip files is implemented in C (I volunteer).
Great !
2) All Python library modules can be imported by C code, either from directories or zip files.
Right.
3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library.
Why ? I don't see a need for that and it would slow down package imports which are becoming increasingly popular as the Python code base grows.
Better postpone this one...
5) More or better ideas from you all.
One question: should these ZIP-archives filenames be placed in sys.path or should Python scan for ZIP-archives within the dirs on sys.path ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

On 05 October 2001, M.-A. Lemburg said:
For the record, I agree with you on all points: there's no need to rewrite package import in Python, but it would be nice to have "import from archive" available in the core. Quite possibly adding this feature would silence 80% of the desire for genreic import hooks. The only import hook I'm familiar with is Quixote, where we have defined a Python dialect called PTL (Python Template Language, used to embed HTML [or other text] in Python). Install Quixote's import hook, and you can import .ptl files just like .py files. It's very handy. Out of curiosity, does anyone know of any other import hooks like this out there -- ie. import something that is not strictly Python. (It sounds like most import hooks/hacks deal with the location of the .py files to import. Quixote doesn't touch that, but it adds the ability to import not-quite-Python source files.)
I think the archive file should be listed in sys.path. Note that my experience of this in Java was largely negative, because Java doesn't have a standard way of putting .class files in the filesystem (AFAIK) -- so everything has to be in a .jar file, and those .jar files can be anywhere you please. So you end up with a mile-long CLASSPATH that's very fragile and forever needing fixing. As long as most Python modules are accessed the ordinary way (files in a directory), then Python won't have a problem. But if somebody makes a Python installation with "stdlib.zip", "distutils.zip", "mxDateTime.zip", etc. etc., then the poor users will be in the same boat as Java users. The main concerns I have with scanning sys.path directories for ZIP files are performance and transparence. (It ain't necessarily obvious where a particular module will come from if sys.path can be searched in two ways.) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ Pointers are Arrays; Code is Data; Time is Money

On Friday 05 October 2001 12:19 pm, Greg Ward wrote:
On 05 October 2001, M.-A. Lemburg said:
The dpython interpreter I wrote this summer would tokenize .dp files that were in the Python path. If the alternative suffix was detected the Python interpreter would use the dpython tokenizer. The dpython tokenizer would generate decimal numbers instead of binary numbers as the default for float and int literals. I didn't use the import hooks. I built the .dp file recognition into the interpreter at the point where it evaluates the suffix of the file name and compairs it to the filedesc struct. When I discussed the implementation of dpython at the ZPUG meeting I joked about extending the concept to add bytecode generators for JavaScript and Visual Basic modules. The work required to add JavaScript would probably be reasonable. It might help get Python used anywhere JavaScript is used. There has also been some interest at NIST and NASA in building modules to parse meta languages such as XML Schema, KIF, and Express and directly generate Python bytecodes.

Hi [Greg Ward]
the latest jython code base (2.1a3) allow to put jar/zip in the sys.path or reference of the form jarfile!pkgdir1/pkgdir2 This are used to set meaningful __path__ attrs for the packages and should allow to deal with code that change __path__ pointing to some relative subdir according to some context property.
I don't get the point :(
I agree. regards.

Greg Ward wrote:
On 05 October 2001, M.-A. Lemburg said:
I think that is correct. At one time there was talk of sys.path containing importer instances. But currently (please check me here) sys.path is purely strings, and these strings must be directory names. Python depends on directories and subdirectories for its import semantics. So any zip archive must support subdirectories to support general imports. Therefore, a string in sys.path which is a zip file is equivalent to hanging a nameless subdirectory at that point. "Nameless" means zip archive files *.pyc occur at the top level, and the zip file may contain relative subdirectories with files sub1/file1.pyc, etc. So a zip file can always substitute for an arbitrary subdirectory tree, and is functionally equivalent to a directory name at the same place in sys.path. JimA

[Greg Ward]
I know of import hooks like that, but in 1000+ emails from Installer users, no one has ever attempted to distribute an app that uses one. If the hook writes out a .pyc, then the hook is pretty much developer-only, anyway.
Huh? Jar files don't date from day 1 - for .class files it works almost like Python.
I think the only difference is that Python goes to considerable effort (and expense) to work out a sys.path before considering PYTHONPATH. - Gordon

"M.-A. Lemburg" wrote:
Well, I looked at import.c and adding zip files with subdirectories will be a mess. The path searches would have to be extended to zip files. Moving out the package code would be a great simplification. Or maybe I am lazy. It doesn't help that I don't really understand packages. Your point about efficiency is a good one though, but I am not sure of the magnitude of the slow down. Anyone else have an opinion? JimA

[James C. Ahlstrom]
Here's my bootstrap code (archive_rt knows how to import from .pyz archives): import archive_rt, iu4 iu4._globalownertypes.insert(0, archive_rt.PYZOwner) iu4.ImportManager().install() Now a .pyz on sys.path will be searched. Nothing hard about doing this for a .zip. MacPython & Jython do it.
Most of what I see is namespace trickery, not custom importers.
With Thomas's changes to PyImport_ImportModuleEx, this is largely taken care of (it gets routed through __import__). So with the above bootstrap code, C code will import from my .pyz as well.
3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library.
The rules aren't documented, but the essentials are pretty straightforward. I'd like to see them changed to draw a distinction between relative imports and absolute imports (since that's a constant source of gotcha's). However, there are import hacks out there that rely on undocumented (and probably accidental) features of implementation.
In iu4, A is welcome to take over all imports below A. If A.__init__ does not do so, the standard mechanism is used.
5) More or better ideas from you all.
- Gordon

Gordon McMillan wrote:
I am not sure if you are saying that code already exists to import from zip archives, or that iu4.py should be used as the basis. I think the best approach is for me to write up a PEP based on the discussions so far, and let everyone criticize. The proposal is only to import from zip archives, not whether to replace imputil.py with iu4.py. I will submit a PEP after I write some code. I often find that my initial design is poor once I implement it. So expect the PEP in a week or so. This won't make it into 2.2, so there is no rush, right? Jim

[Gordon]
Neither. I thought you were referring to the Py 1.5-timeframe problem of C code using PyImport_ImportModule* and bypassing all hooks.
Does "importing from zip archives" really need a PEP? It's already quasi-official in that Jython and MacPython implement it. The problem it exposes (the fact that the C code makes it difficult to do for packages) is certainly a good topic for discussion and is part of the reason Greg wrote imputil and I wrote iu4. - Gordon

Hi.
Personally, I think so, a PEP would be indeed a good thing, mostly to fix the details. AFAIK MacPython and Jython are already different about the details, at least in Jython the feature is still experimental so there is space for changes, don't know in general. regards, Samule Pedroni.

+1. A PEP is the only way to get this to be a standard feature. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (10)
-
Andrew Kuchling
-
Gordon McMillan
-
Greg Stein
-
Greg Ward
-
Guido van Rossum
-
James C. Ahlstrom
-
M.-A. Lemburg
-
Michael McLay
-
Samuele Pedroni
-
Thomas Heller