[Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)

[Guido van Rossum]

> > > (and of course, objects other than string subclasses don't
> > > get "extended").
> >
> > We can use some kind of extension hook to deal with those.
> 
> It depends where you envisage that hook. I'd be unhappy with
> requiring something in the import hook implementation - two
> reasons, first because the current hook mechanism is nice and
> simple, and hence easy to document and use, and second because
> people simply won't implement the path-extension hook, and nearly
> everything will work fine - the breakage only occurs when the
> hook is used in conjunction with pkgutil. At that point, saying
> "I told you so" isn't much help...
> 
> I've seen this sort of thing happen too often before to be
> comfortable with it. A simple protocol, with an obscure little
> requirement which is to support a relatively rare usage. People
> *never* implement that correctly, and the result is annoying
> breakages.

Only annoying if you need the rare usage.  It turns out that running
from a zipfile isn't a priority for Zope3 anyway, so I'll call this a
YAGNI.

> > The important point is that the idiom placed in __init__.py
> > can remain the same (the two lines quoted above) while the
> > implementation of pkgutil.extend_path() can evolve or be
> > modified (even monkey-patched if you're desperate) as the
> > import hooks evolve.
> 
> I'd say the extension mechanism should be external to both
> the import hook and pkgutil. It then becomes the user's
> responsibility, when they combine a particular hook with the
> use of pkgutil, to write the glue.
> 
>    pkgutil.register_hook(whatever)
> 
> Hook authors can, of course, implement a convenience method,
> to be used as
> 
>    pkgutil.register_hook(ihook.pkgutil_hook)
> 
> but if they fail to, at least the user isn't left up the creek
> without the proverbial paddle :-)
> 
> (Hope that made sense - I was hand-waving a bit too much...)

Yes, except you failed to specify what pkgutil.register_hook(whatever)
is supposed to do, and you failed to specify the signature for
'whatever'...  IOW, please think more about the implementation and
provide a patch to the latest pkgutil.py (I posted it here earlier).

> > > I don't think that either of these is even remotely a
> > > showstopper, but they do need to be documented - specifically
> > > because as a module, this code can be imported by anybody,
> > > and so the constraints on the user should be made clear.
> >
> > Of course. Though with the current state of documentation,
> > I give less priority to documenting every little detail of
> > this code than to getting it in the code base (with *some*
> > documentation, for sure).
> 
> At the moment, I'm playing with the hook mechanism, with the
> intent of being able to provide some of the documentation. At
> the very least, I'd like to be able to do a tutorial-style "how
> to write a simple import hook" document.
> 
> But yes, a lot of this area is only minimally documented right
> now, so we don't need to aim for complete perfection just yet :-)

You're talking about Just's import hooks, right?

> > OTOH I expect that most things you'd want to place on sys.path
> > *do* have a natural string representation. After all, if
> > something doesn't have a name, it's hard to talk about. And if
> > it has a name, it can be placed in PYTHONPATH (with the minor
> > quibble that URLs use ':' which -- at least on Unix -- is also
> > used as the separator in *PATH variables. :-( )
> 
> Hmm. Arguable. I take your point, but I'm looking at it from a
> different point of view. A "place to store modules" could be
> something as complex as a database (TCL Starkits use MetaKit).
> I *could* define a string representation encoding the table name,
> columns to use, etc etc, but why bother? Just use a database
> importer object directly. You can't put it on PYTHONPATH, but
> you can easily use something like this in a packaged application
> context.

A packaged application knows whether it needs to support pkgutil or
not.  So it can take whatever measures necessary (including overriding
pkgutil.py) to make it work.

> Let's wait and see what real use brings on this one.

Yes please.

> > Maybe importer objects should have a standard API that lets you
> > ask for an extended object.
> 
> This way lies a VFS implementation.

I'm not sure what you mean by that.

> I'd say we should write a general import hook based on a VFS
> implementation, rather than the other way around. And let's leave a
> full VFS alone for now :-)

The old ihooks was trying to do something like a vfs (it has a bunch
of APIs that provide hooks into all filesystem operations).  But it
wasn't a big success...

--Guido van Rossum (home page: http://www.python.org/~guido/)