[Python-Dev] Import redesign [LONG]

[Greg Stein]

On Thu, 2 Dec 1999, Guido van Rossum wrote:
>...
> Sometime, Greg Stein wrote:
>...
> > On Thu, 18 Nov 1999, Guido van Rossum wrote:
>...
> > > Agreed.  I like some of imputil's features, but I think the API
> > > need to be redesigned.
> > 
> > It what ways? It sounds like you've applied some thought. Do you have any
> > concrete ideas yet, or "just a feeling" :-)  I'm working through some
> > changes from JimA right now, and would welcome other suggestions. I think
> > there may be some outstanding stuff from MAL, but I'm not sure (Marc?)
> 
> I actually think that the way the PVM (Python VM) calls the importer
> ought to be changed.  Assigning to __builtin__.__import__ is a crock.
> The API for __import__ is a crock.

Something like sys.set_import_hook() ?

The other alternative that I see would be to have the C code scan
sys.importers, assuming each are callable objects, and call them with the
appropriate params (e.g. module name). Of course, to move this scanning
into Python would require something like sys.set_import_hook() unless
Python looks for a hard-coded module and entrypoint.

>...
> > Which APIs are you referring to? The "imp" module? The C functions? The
> > __import__ and reload builtins?
> 
> > I'm guessing some of imp, the two builtins, and only one or two C
> > functions.
> 
> All of those.

We can provide Python code to provide compatibility for "imp" and the two
hooks. Nothing we can do to the C code, though. I'm not sure what the
import API looks like from C, and whether they could all stay. A brief
glance looks like most could stay.
[ removing any would change Python's API version, which might be "okay" ]

>...
> > > - load .py/.pyc/.pyo files and shared libraries from files
> > 
> > No problem. Again, a function is needed for platform-specific loading of
> > shared libraries.
> 
> Is it useful to expose the platform differences?  The current
> imp.load_dynamic() should suffice.

This comes up several times throughout this message, and in some off-list
mail Guido and I have exchanged. Namely, "should dynamic loading be part
of the core, or performed via a module?"

I would rather see it become a module, rather than inside the core
(despite the fact that the module would have to be compiled into the
interpreter). I believe this provides more flexibility for people looking
to replace/augment/update/fix dynamic loading on various architectures.
Rather than changing the core, a person can just drop in another module.
The isolation between the core and modules is nicer, aesthetically, to me.

The modules would also be exposing Just Another Importer Function, rather
than a specialized API in the builtin imp module. Also note that it is
easier to keep a module *out* of a Python-based application, than it is to
yank functions out of the core of Python. Frozen apps, embedded apps, etc
could easily leave out dynamic loading.

Are there strict advantages? Not any that I can think of right now (beyond
a bit of ease-of-use mentioned above). It just feels better to me.

>...
> > > - sys.path and sys.modules should still exist; sys.path might
> > > have a slightly different meaning
> > 
> > I would suggest that both retain their *exact* meaning. We introduce
> > sys.importers -- a list of importers to check, in sequence. The first
> > importer on that list uses sys.path to look for and load modules. The
> > second importer loads builtins and frozen code (i.e. modules not on
> > sys.path).
> 
> This is looking like the redesign I was looking for.  (Note that
> imputil's current chaining is not good since it's impossible to remove
> or reorder importers, which I think is a required feature; an explicit
> list would solve this.)

The chaining is an aspect of the current, singular import hook that Python
uses. In the past, I've suggested the installation of a "manager" that
maintains a list. sys.importers is similar in practice.

Note that this Manager would be present with the sys.set_import_hook()
scheme, while the Manager is implied if the core scans sys.importers.

> Actually, the order is the other way around, but by now you should
> know that.  It makes sense to have separate ones for builtin and
> frozen modules -- these have nothing in common.

Yes, JimA pointed this out. The latest imputil has corrected this.

I combined the builtin and frozen Importers because they were just so
similar. I didn't want to iterate over two Importers when a single one
sufficed quite well.

*shrug* Could go either way, really.

> There's another issue, which isn't directly addressed by imputil,
> although with clever use of inheritance it might be doable.  I'd like
> more support for this however.  Quite orthogonally to the issue of
> having separate importers, I might want to recognize new extensions.

Correct: while imputil doesn't address this, the standard/default Importer
classes *definitely* can.

>...
> the directory/directories with .isl files are placed.)  This requires
> an ugly modification to the _fs_import() function.  (Which should have
> been a method, by the way, to make overriding it in a subclass of
> PathImporter easier!)

I yanked that code out of the DirectoryImporter so that the PathImporter
could use it. I could see a reorg that creates a FileSystemImporter that
defines the method, and the other two just subclass from that.

> I've been thinking here along the lines of a strategy where the
> standard importer (the one that walks sys.path) has a set of hooks
> that define various things it could look for, e.g. .py files, .pyc
> files, .so or .dll files.  This list of hooks could be changed to
> support looking for .isl files.

Agreed. It should be easy to have a mapping of extension to handler.

One issue: should there be an ordering to the extensions? Exercise for the
reader to alter the data structures...

> There's an old, subtle issue that could be solved through this as
> well: whether or not a .pyc file without a .py file should be accepted
> or not.  Long ago (in Python 0.9.8) a .pyc file alone would never be
> loaded.  This was changed at the request of a small but vocal minority
> of Python developers who wanted to distribute .pyc files without .py
> files.  It has occasionally caused frustration because sometimes
> developers move .py files around but forget to remove the .pyc files,
> and then the .pyc file is silently picked up if it occurs on sys.path
> earlier than where the .py was moved to.

I think, "too bad for them."  :-)

Having just a .pyc is a very nice feature. But how can you tell whether it
was meant to be a plain .pyc or a mis-ordered one? To truly resolve that,
you would need to scan the whole path, looking for a .py. However, maybe
somebody put the .pyc there on purpose, to override the .py!

--- begin slightly-off-topic ---

Here is a neat little Bash script that allows you to use a .pyc as a CGI
(to avoid parse overhead). Normally, you can't just drop a .pyc into the
cgi-bin directory because the OS doesn't know how to execute it. Not a
problem, I say... just append your .pyc to the following Bash script and
execute! :-)

#!/bin/bash
exec - 3< $0 ; exec python -c 'import os,marshal ; f = os.fdopen(3, "rb")
; f.readline() ; f.readline() ; f.seek(8, 1) ; _c = marshal.load(f) ; del
os, marshal, f ; exec _c' $@

(the script should be two lines; and no... you can't use readlines(2))

The above script will preserve stdin, stdout, and stderr. If the caller
also use 3< ... well, that got overridden :-)

The script doesn't work on Windows for two reasons, though: 1) Bash, 2)
the "rb" mode followed by readline()

Detailed info at the bottom of http://www.lyra.org/greg/python/

--- end of off-topic ---

> Having a set of hooks for various extensions would make it possible to
> have a default where lone .pyc files are ignored, but where one can
> insert a .pyc importer in the list of hooks that does the right thing
> here.  (Of course, it may be possible that this whole feature of lone
> .pyc files should be replaced since the same need is easily taken care
> of by zip importers.

Maybe. I'd still like to see plain .pyc files, but I know I can work
around any change you might make here :-)

(i.e. whatever you'd like to do... go for it)

> I also want to support (Jim A notwithstanding :-) a feature whereby
> different things besides directories can live on sys.path, as long as
> they are strings -- these could be added from the PYTHONPATH env
> variable.  Every piece of code that I've ever seen that uses sys.path
> doesn't care if a directory named in sys.path doesn't exist -- it may
> try to stat various files in it, which also don't exist, and as far as
> it is concerned that is just an indication that the requested module
> doesn't live there.

I'm not in favor of this, but it is more-than-doable. Again: your
discretion...

> Again, we would have to dissect imputil to support various hooks that
> deal with different kind of entities in sys.path.  The default hook
> list would consist of a single item that interprets the name as a
> directory name; other hooks could support zip files or URLs.  Jack's
> "magic cookies" could also be supported nicely through such a
> mechanism.

Specifically, the PathImporter would get "dissected" :-). No problem.

> > Users can insert/append new importers or alter sys.path as before.
> > 
> > sys.modules continues to record name:module mappings.
> 
> Yes.
> 
> Note that the interpretation of __file__ could be problematic.  To
> what value do you set __file__ for a module loaded from a zip archive?

You don't (certainly in a way that is nice/compatible for modules that
refer to it). This is why I don't like __file__ and __path__. They just
don't make sense in archives or frozen code. Python code that relies on
them will create problems when that code is placed into different
packaging mechanisms.

>...
> > > (I wouldn't mind a splitting up of importdl.c into several
> > > platform-specific files, one of which is chosen by the configure
> > > script; but that's a bit of a separate issue.)
> > 
> > Easy enough. The standard importer can select the appropriate
> > platform-specific module/function to perform the load. i.e. these can move
> > to Modules/ and be split into a module-per-platform.
> 
> Again: what's the advantage of exposing the platform specificity?

See above.

>...
> Probably more support is required from the other end: once it's common
> for modules to be imported from zip files, the distutil code needs to
> support the creation and installation of such zip files.  Also, there
> is a need for the install phase of distutil to communicate the
> location of the zip file to the Python installation.

I'm quite confident that something can be designed that would satisfy the
needs here. Something akin to .pth files that a zip importer could read.

>...
> > > - Standard import from zip or jar files, in two ways:
> > > 
> > >   (1) an entry on sys.path can be a zip/jar file instead of a directory;
> > >       its contents will be searched for modules or packages
> 
> Note that this is what I mention above for distutil support.
> 
> > While this could easily be done, I might argue against it. Old
> > apps/modules that process sys.path might get confused.
> 
> Above I argued that this shouldn't be a problem.

For most code, no, but as Fred mentioned (and I surmise), there are things
out there assuming that sys.path contains strings which specify
directories.

Sure, we can do this (your discretion), but my feeling is to avoid it.

> > If compatibility is not an issue, then "No problem."
> > 
> > An alternative would be an Importer instance added to sys.importers that
> > is configured for a specific archive (in other words, don't add the zip
> > file to sys.path, add ZipImporter(file) to sys.importers).
> 
> This would be harder for distutil: where does Python get the initial
> list of importers?

Default is just the two: BuiltinImporter and PathImporter. Adding
ZipImporters (or anything else) at startup is TBD, but shouldn't pose a
problem.

>...
> > >   (2) a file in a directory that's on sys.path can be a zip/jar file;
> > >       its contents will be considered as a package (note that this is
> > >       different from (1)!)
> > 
> > No problem. This will slow things down, as a stat() for *.zip and/or *.jar
> > must be done, in addition to *.py, *.pyc, and *.pyo.
> 
> Fine, this is where the caching comes in handy.

IFF caching is enabled for the particular platform and installation.

>...
> > The Importer class is already designed for subclassing (and its interface 
> > is very narrow, which means delegation is also *very* easy; see
> > imputil.FuncImporter).
> 
> But maybe it's *too* narrow; some of the hooks I suggest above seem to
> require extra interfaces -- at least in some of the subclasses of the
> Importer base class.

Correct -- the *subclasses*. I still maintain the imputil design of a
single hook (get_code) is Right.

I'll make a swipe at PathImporter in the next few weeks to add the
capability for new extensions.

> Note: I looked at the doc string for get_code() and I don't understand
> what the difference is between the modname and fqname arguments.  If I
> write "import foo.bar", what are modname and fqname?  Why are both
> present?  Also, while you claim that the API is narrow, the multiple
> return values (also the different types for the second item) make it
> complicated.

Gordon detailed this in another note...

Yes, the multiple return values make it a bit more complicated, but I
can't think of any reasonable alternatives.

A bit more doc should do the trick, I'd guess.

>...
> > >   - a hook to auto-generate .py files from other filename
> > >     extensions (as currently implemented by ILU)
> > 
> > No problem at all.
> 
> See above -- I think this should be more integrated with sys.path than
> you are thinking of.  The more I think about it, the more I see that
> the problem is that for you, the importer that uses sys.path is a
> final subclass of Importer (i.e. it is itself not further subclassed).
> Several of the hooks I want seem to require additional hooks in the
> PathImporter rather than new importers.

Correct -- I've currently designed/implemented PathImporter as "final".

I don't forsee a problem turning it into something that can be hooked at
run-time, or subclassed at code-time. A detailing of the features needed 
would be handy:

* allow alternative file suffixes, with functions or subclasses to map the
  file into a code/module object.

>...
> > > - Note that different kinds of hooks should (ideally, and within
> > >   reason) properly combine, as follows: if I write a hook to recognize
> > >   .spam files and automatically translate them into .py files, and you
> > >   write a hook to support a new archive format, then if both hooks are
> > >   installed together, it should be possible to find a .spam file in an
> > >   archive and do the right thing, without any extra action.  Right?
> > 
> > Ack. Very, very difficult.
> 
> Actually, I take most of this back.  Importers that deal with new
> extension types often have to go through a file system to transform
> their data to .py files, and this is just too complicated.  However it
> would be still nice if there was code sharing between the code that
> looks for .py and .pyc files in a zip archive and the code that does
> the same in a filesystem.  Hm, maybe even that shouldn't be necessary,
> the zip file probably should contain only .pyc files...

Gordon replies to this... All of the archives that myself, Gordon, and
JimA have been using only store .pyc files. I don't see much code sharing
between the filesystem and archive import code.

>...
> > All is not lost, however. I can easily envision the get_code() hook as
> > allowing any kind of return type. If it isn't a code or module object,
> > then another hook is called to transform it.
> > [ actually, I'd design it similarly: a *series* of hooks would be called
> >   until somebody transforms the foo.spam into a code/module object. ]
> 
> OK.  This could be a feature of a subclass of Importer.

That would be my preference, rather than loading more into the Importer
base class itself.

>...
> > > - It should be possible to write hooks in C/C++ as well as Python
> > 
> > Use FuncImporter to delegate to an extension module.
> 
> Maybe not so great, since it sounds like the C code can't benefit from
> any of the infrastructure that imputil offers.  I'm not sure about
> this one though.

There isn't any infrastructure that needs to be accessed. get_code() is
the call-point, and there is no mechanism provided to the callee to call
back into the imputil system.

> > This is one of the benefits of imputil's single/narrow interface.
> 
> Plus its vague specs? :-)

Ouch. I thought I was actually doing quite a bit better than normal with
that long doc-string on get_code :-(

>...
> > For a restricted execution app, it might install an Importer that loads
> > files from *one* directory only which is configured from a specific
> > Win32 Registry entry. That importer could also refuse to load shared
> > modules. The BuiltinImporter would still be present (although the app
> > would certainly omit all but the necessary builtins from the build).
> > Frozen modules could be excluded.
> 
> Actually there's little reason to exclude frozen modules or any
> .py/.pyc modules -- by definition, bytecode can't be dangerous.  It's
> the builtins and extensions that need to be censored.
> 
> We currently do this by subclassing ihooks, where we mask the test for
> builtins with a comparison to a predefined list of names.

True. My concern is an invader misusing one "type" of module for another.
For example, let's say you've provided a selection of modules each
exporting function FOO, and the user can configure which module to use.
Can they do damage if some unrelated, frozen module also exports FOO?

Minor issue, anyhow. All the functionality is there.

>...
> > I posited once before that the cost of import is mostly I/O rather than
> > CPU, so using Python should not be an issue. MAL demonstrated that a good
> > design for the Importer classes is also required. Based on this, I'm a
> > *strong* advocate of moving as much as possible into Python (to get
> > Python's ease-of-coding with little relative cost).
> 
> Agreed.  However, how do you explain the slowdown (from 9 to 13
> seconds I recall) though?  Are you a lousy coder? :-)

Heh :-)

I have not spent *any* time working on optimization. Currently, each
Importer in the chain redoes some work of the prior Importer. A bit of
restructuring would split the common work out to a Manager, which then
calls a method in the Importer (and passes all the computed work). Of
course, a bit of profiling wouldn't hurt either. Some of the "imp"
interfaces could possibly be refined to better support the BuiltinImporter
or the dynamic load features.

The question is still valid, though -- at the moment, I can't explain it
because I haven't looked into it.

> > The (core) C code should be able to search a path for a module and import
> > it. It does not require dynamic loading or packages. This will be used to
> > import exceptions.py, then imputil.py, then site.py.

Note: after writing this, I realized there is really no need for the core
to do the imputil import. site.py can easily do that.

> It does, however, need to import builtin modules.  imputil currently

Correct.

> imports imp, sys, strop and __builtin__, struct and marshal; note that
> struct can easily be a dynamic loadable module, and so could strop in
> theory.  (Note that strop will be unnecessary in 1.6 if you use string
> methods.)

I knew about strop, but imputil would be harder to use today if it relied
on the string methods. So... I've delayed that change.

The struct module is used in a couple teeny cases, dealing with
constructing a network-order, 4-byte, binary integer value. It would be
easy enough to just do that with a bit of Python code instead.

> I don't think that this chicken-or-egg problem is particularly
> problematic though.

Right.

In my ideal world, the core couldn't do a dynamic load, so that would need
to be considered within the bootstrap process.

>...
> > site.py can complete the bootstrap by setting up sys.importers with the
> > appropriate Importer instances (this is where an application can define
> > its own policy). sys.path was initially set by the import.c bootstrap code
> > (from the compiled-in path and environment variables).
> 
> I thing that algorithm (currently in getpath.c / getpathp.c) might
> also be moved to Python code -- imported frozen.  Sadly, rebuilding
> with a new version of a frozen module might be more complicated than
> rebuilding with a new version of a C module, but writing and
> maintaining this code in Python would be *sooooooo* much easier that I
> think it's worth it.

I think we can find a better way to freeze modules and to use them.
Especially for the cases where we have specific "core" functions
implemented in Python. (e.g. freezing parsers, compilers, and/or the
read-eval loop)

I don't forsee an issue that the build process becomes more complicated.
If we nuke "makesetup" in favor of a Python script, then we could create a
stub Python executable which runs the build script which writes the Setup
file and the getpath*.c file(s).

> > Note that imputil.py would not install any hooks when it is loaded. That
> > is up to site.py. This implies the core C code will import a total of
> > three modules using its builtin system. After that, the imputil mechanism
> > would be importing everything (site.py would .install() an Importer which
> > then takes over the __import__ hook).
> 
> (Three not counting the builtin modules.)

Correct, although I'll modify my statement to "two plus the builtins".

> > Further note that the "import" Python statement could be simplified to use
> > only the hook. However, this would require the core importer to inject
> > some module names into the imputil module's namespace (since it couldn't
> > use an import statement until a hook was installed). While this
> > simplification is "neat", it complicates the run-time system (the import
> > statement is broken until a hook is installed).
> 
> Same chicken-or-egg.  We can be pragmatic.
> 
> For a developer, I'd like a bit of robustness (all this makes it
> rather hard to debug a broken imputil, and that's a fair amount of
> code!).

True. I threw that out as an alternative, and then presented the counter
argument :-)

>...
> > Therefore, the core C code must also support importing builtins. "sys" and
> > "imp" are needed by imputil to bootstrap.
> > 
> > The core importer should not need to deal with dynamic-load modules.
> 
> Same question.  Since that all has to be coded in C anyway, why not?

It simplifies the core's import code to not deal with that stuff at all.

> > To support frozen apps, the core importer would need to support loading
> > the three modules as frozen modules.
> 
> I'd like to see a description of how someone like Jim A would build a
> single-file application using the new mechanism.  This could
> completely replace freeze.  (Freeze currently requires a C compiler;
> that's bad.)

The portable mechanism for freezing will always need a compiler. Platform
specific mechanisms (e.g. append to the .EXE, or use the linker to create
a new ELF section) can optimize the freeze process in different ways.

I don't have a design in my head for the freeze issues -- I've been
considering that the mechanism would remain about the same. However, I can
easily see that different platforms may want to use different freeze
processes... hmm...

>...
> > Yes. I don't see this as a requirement, though. We wouldn't start to use
> > these by default, would we? Or insist on zlib being present? I see this as
> > more along the lines of "we have provided a standardized Importer to do
> > this, *provided* you have zlib support."
> 
> Agreed.  Zlib support is easy to get, but there are probably platforms
> where it's not.  (E.g. maybe the Mac?  I suppose that on the Mac,
> there would be some importer classes to import from a resource fork.)

Exactly. And importer classes to load from a Win32 resources (modifying a
.EXE's resources post-link is cleaner than the append solution)

>...
> > My outline above does not freeze anything. Everything resides in the
> > filesystem. The C code merely needs a path-scanning loop and functions to
> > import .py*, builtin, and frozen types of modules.
> 
> Good.  Though I think there's also a need for freezing everything.
> And when we go the route of the zip archive, the zip archive handling
> code needs to be somewhere -- frozen seems to be a reasonable choice.

Sure.

> > If somebody nukes their imputil.py or site.py, then they return to Python
> > 1.4 behavior where the core interpreter uses a path for importing (i.e. no
> > packages). They lose dynamically-loaded module support.
> 
> But if the path guessing is also done by site.py (as I propose) the
> path will probably be wrong.  A warning should be printed.

All right. Doesn't Python already print a warning if it can't find
site.py?

> > > Let's first complete the requirements gathering.  Are these
> > > requirements reasonable?  Will they make an implementation too
> > > complex?  Am I missing anything?
> > 
> > I'm not a fan of the compositing due to it requiring a change to semantics
> > that I believe are very useful and very clean. However, I outlined a
> > possible, clean solution to do that (a secondary set of hooks for
> > transforming get_code() return values).
> 
> As you may see from my responses, I'm a big fan of having several
> different sets of hooks.

Yes. However, I've only recognized one so far. Propose more... I'm
confident we can update the PathImporter design to accomodate (and retain
the underlying imputil paradigm).

> I do withdraw the composition requirement
> though.

:-)

>...
> > Once you hit site.py, you have a "full" environment and can easily detect
> > and import a read-eval-print loop module (i.e. why return to Python? just 
> > start things up right there).
> 
> You mean "why return to C?"  I agree.  It would be cool if somehow

Heh. Yah, that's what I meant :-)

> IDLE and Pythonwin would also be bootstrapped using the same
> mechanisms.  (This would also solve the question "which interactive
> environment am I using?" that some modules and apps want to see
> answered because they need to do things differently when run under
> IDLE,for example.)

Haven't thought on this. Should be doable, I'd think.

> > site.py can also install new optimizers as desired, a new Python-based
> > parser or compiler, or whatever...  If Python is built without a parser or
> > compiler (I hope that's an option!), then the three startup modules would
> > simply be frozen into the executable.
> 
> More power to hooks!

:-) You betcha!

I believe my next order of business:

* update PathImporter with the file-extension hook
* dynload C code reorg, per the other email
* create new-model site.py and trash import.c
* review freeze mechanisms and process
* design mechanism for frozen core functionality (eg. getpath*.c)
  (coding and building design)
* shift core functions to Python, using above design

I'll just plow ahead, but also recognize that any/all may change. ie. I'll
build examples/finals/prototypes and Guido can pick/choose/reimplement/etc
as needed. I'm out next week, but should start on the above items by the
end of the month (will probably do another mod_dav release in there
somewhere).

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/