[Python-Dev] Towards a Python based import scheme

[Greg Stein]

M.-A. Lemburg wrote:
>...
> The original version of imputil I fetched from Greg's page
> did work out of the box (from...import... hassles) and

"did not work", I presume. From my original testing, I thought
from...import worked. With more testing, I found that something of the
form "from xml.dom import builder" did not work.

I discovered why it failed (xml.dom was imported by Importer instance I1
but I2 thought it could handle the from...import, and this barfed a
check). I've fixed this by delegating to the proper importer (I1 in my
example) to complete the import. Your solution to check the __importer__
variable in the globals is probably incorrect. If I read/eval it
correctly, that would mean that a module imported by IMP1 could not use
modules imported by IMP2. In other words, a package module could not
import a top-level module defined by a different importer.
(note also that your globals.get() could fail if globals is None)

> obviously did not support in-package shared libs. I've added

I did not fold this in. Your change isn't "in the spirit" of the
Importer mechanism. The "Right Way" to do this is to create a
BuiltinImporter and add that to the chain of importers. The
DirectoryImporter should only import from directories -- no reason for
it to know about builtin stuff. As a result, I did not accept the new
methods on Importer for handling builtins/special modules -- those would
go in the BuiltinImporter.
[BuiltinImporter should be written and included in imputil.py; I don't
really have the time at the moment to write the thing... 7am and time
for sleep...]

However, your change here did raise a very important design issue:
get_code() needs to be able to return a loaded module, rather than just
a code object. I've folded in your patches for that.

I also folded in many of your extended doc/comments (at least in
concept; not necessarily verbatim). You and Gordon are recognized in the
header now, and I've added a "proper" author notice and licensing
(public domain).

I did not include the "misses" feature that you added to the
DirectoryImporter. I would hate to see a miss-cache get loaded, a module
dropped into the filesystem, and the user never being able to import the
thing.

I didn't fold in your indentation changes or name changes. I liked mine
:-). The __main__ thing at the bottom didn't make much sense to me,
though, since the call to _test_dir() followed by an exit doesn't really
do anything. And yes, I recognize that you can use "python -i
imputil.py" but I'd rather just see "python" followed by "import imputil
; imputil._test_dir()".

Of course, please feel free to generate a new patch if I've missed
something (thinking about it, I missed the OSError thing).

> both features so that the test script in DateTime can run
> successfully.
> 
> Things that remain are:
> · the win32 registry stuff (needs C code)

And a new Importer to use it.

> · the Mac fork stuff (needs C code)

Ditto.

> · a working __path__ implementation (is anyone using this attribute
>   which only is available in packages ?)

Per the private mail that I sent to you: I explicitly punted on the
__path__ attribute. It can lead to *way* too much confusion. It is also
unavailable for frozen packages (boy oh boy did the win32com get some
ugliness in there to compensate for being frozen w.r.t. its use of
__path__).

The DirectoryImporter can insert the attribute, but it definitely
wouldn't go into the Importer itself. The __path__ attribute is specific
to loading from a filesystem, yet Importer is generic.

> · probably a whole bunch of other quirks
> · some speedups (there currently are too many stat()s)

Yes. I recognize that the "misses" feature was intended to remedy this.
I don't have an immediate answer to the stat() issue. Does the Importer
mechanism actually perform more stats on an import than Python itself?
(it looks like it does one for the isdir() plus two for fetching file
timestamps)

And a big thanx: I appreciate the patches to imputil! The new module is
now available in its "official" location at
http://www.lyra.org/greg/python/

Cheers,
-g

--
Greg Stein, http://www.lyra.org/