[Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)

[Moore, Paul]

From: Guido van Rossum [mailto:guido@python.org]

> I find string subclasses an ugly hack that should not be used.

Agreed.

> The meta hook (or whatever it's called now) should do this.

Why bother?

> >   - strings which don't have a path structure wil break
>
> We can fix that by skipping anything for which
> os.path.isdir(dir) [for the original value of dir] isn't true.

Agreed. This makes extend_path "do the right thing" for anything
that can currently be on a path, and punt to an extension hook
for anything else. That sounds reasonable (but see below).

> > (and of course, objects other than string subclasses don't
> > get "extended").
>
> We can use some kind of extension hook to deal with those.

It depends where you envisage that hook. I'd be unhappy with
requiring something in the import hook implementation - two
reasons, first because the current hook mechanism is nice and
simple, and hence easy to document and use, and second because
people simply won't implement the path-extension hook, and nearly
everything will work fine - the breakage only occurs when the
hook is used in conjunction with pkgutil. At that point, saying
"I told you so" isn't much help...

I've seen this sort of thing happen too often before to be
comfortable with it. A simple protocol, with an obscure little
requirement which is to support a relatively rare usage. People
*never* implement that correctly, and the result is annoying
breakages.

> The important point is that the idiom placed in __init__.py
> can remain the same (the two lines quoted above) while the
> implementation of pkgutil.extend_path() can evolve or be
> modified (even monkey-patched if you're desperate) as the
> import hooks evolve.

I'd say the extension mechanism should be external to both
the import hook and pkgutil. It then becomes the user's
responsibility, when they combine a particular hook with the
use of pkgutil, to write the glue.

   pkgutil.register_hook(whatever)

Hook authors can, of course, implement a convenience method,
to be used as

   pkgutil.register_hook(ihook.pkgutil_hook)

but if they fail to, at least the user isn't left up the creek
without the proverbial paddle :-)

(Hope that made sense - I was hand-waving a bit too much...)

> > I don't think that either of these is even remotely a
> > showstopper, but they do need to be documented - specifically
> > because as a module, this code can be imported by anybody,
> > and so the constraints on the user should be made clear.
>
> Of course. Though with the current state of documentation,
> I give less priority to documenting every little detail of
> this code than to getting it in the code base (with *some*
> documentation, for sure).

At the moment, I'm playing with the hook mechanism, with the
intent of being able to provide some of the documentation. At
the very least, I'd like to be able to do a tutorial-style "how
to write a simple import hook" document.

But yes, a lot of this area is only minimally documented right
now, so we don't need to aim for complete perfection just yet :-)

> OTOH I expect that most things you'd want to place on sys.path
> *do* have a natural string representation. After all, if
> something doesn't have a name, it's hard to talk about. And if
> it has a name, it can be placed in PYTHONPATH (with the minor
> quibble that URLs use ':' which -- at least on Unix -- is also
> used as the separator in *PATH variables. :-( )

Hmm. Arguable. I take your point, but I'm looking at it from a
different point of view. A "place to store modules" could be
something as complex as a database (TCL Starkits use MetaKit).
I *could* define a string representation encoding the table name,
columns to use, etc etc, but why bother? Just use a database
importer object directly. You can't put it on PYTHONPATH, but
you can easily use something like this in a packaged application
context.

Let's wait and see what real use brings on this one.

> Maybe importer objects should have a standard API that lets you
> ask for an extended object.

This way lies a VFS implementation. I'd say we should write a
general import hook based on a VFS implementation, rather than the
other way around. And let's leave a full VFS alone for now :-)

Paul.