[Python-Dev] Writing importers and path hooks

[Brett Cannon]

On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 28 March 2013 13:42, Brett Cannon <brett at python.org> wrote:
> >> importer, as I wanted to try a proof of concept importer based on the
> >> new importlib stuff (which is intended to make writing custom
> >> importers easier), and I really struggled to get something working.
> >
> > Struggling how? With the finder? The loader? What exactly were you
> trying to
> > accomplish and how were you deviating from the standard import system?
>
> What I was trying to do was to write a path hook that would allow me
> to add a string to sys.path which contained a base-64 encoded zipfile
> (plus some sort of marker so it could be distinguished from a normal
> path entry) and as a result the contents of that embedded zip file
> would be available as if I'd added an actual zip file with that
> content to sys.path.
>
> I got in a complete mess as I tried to strip out the (essentially
> non-interesting) zipfile handling to get to a dummy "do nothing,
> everything is valid" type of example. But I don't think I would have
> fared much better if I'd stuck to the original full requirement.
>
> >> It seems to me that the importlib documentation doesn't help much for
> >> people trying to import path hooks.
> >
> > There is a bug to clarify the docs to have more geared towards writing
> new
> > importers instead of just documenting what's available:
> > http://bugs.python.org/issue15867
>
> Thanks. I'll keep an eye on that.
>
> >> But it might be just me. Does
> >> anyone have an example of a simple importlib-based finder/loader?
> >
> > Define simple. =) I would argue importlib itself is easy enough to read.
>
> :-) Fair point. I guess importlib is not *that* hard to read, but the
> only case that implements packages in the filesystem one, and that
> also deals with C extensions and other complexities that I don't have
> a need for. I'll try again to have a deeper look at it, but I didn't
> find it easy to extract the essentials when I looked before.
>
> >> That
> >> would be a huge help for me. In return for any pointers, I'll look at
> >> putting together a doc patch to clarify how to use importlib to build
> >> your own path hooks :-)
> >
> > Do you specifically mean the path hook aspect or the whole package of
> hook,
> > finder, and loader?
>
> OK, after some more digging, it looks like I misunderstood the process
> somewhat. Writing a loader that inherits from *both* FileLoader and
> SourceLoader,


You only need SourceLoader since you are dealing with Python source. You
don't need FileLoader since you are not reading from disk but an in-memory
zipfile.


> and then implementing get_data (and module_repr - why do
> I need that, couldn't the ABC offer a default implementation?)


http://bugs.python.org/issue17093 and http://bugs.python.org/issue17566


> does
> the job for that.
>

You should be implementing get_data, get_filename, and path_stats for
SourceLoader.


>
> But the finder confuses me. I assume I want a PathEntryFinder and
> hence I should implement find_loader().


Yes since you are pulling from sys.path.


> The documentation on what I
> need to return from there is very sparse... In the end I worked out
> that for a package, I need to return (MyLoader(modulename,
> 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
> example).


The second argument should just be None: "An empty list can be used for
portion to signify the loader is not part of a [namespace] package".
Unfortunately a key word is missing in that sentence.
http://bugs.python.org/issue17567


> In essence, PathEntryFinder really has to implement some
> form of virtual filesystem mount point, and preserve the standard
> filesystem semantics of modules having a filename of .../__init__.py.
>

Well, if your zip file decided to create itself with a different file
extension then it wouldn't be required, but then other people's code might
break if they don't respect module abstractions (i.e. looking at
__package__/__name__ or __path__ to see if something is a package).


>
> So I managed to work out what was needed in the end, but it was a lot
> harder than I'd expected. On reflection, getting the finder semantics
> right (and in particular the path entry finder semantics) was the hard
> bit.
>

Yep, that bit has had the least API tweaks as most people don't muck with
finders but with loaders.


>
> I'm now 100% sure that some cookbook examples would help a lot. I'll
> see what I can do.
>

I plan on writing a pure Python zip importer for Python 3.4 which should be
fairly minimal and work out as a good example chunk of code.  And no one
need bother writing it as I'm going to do it myself regardless to make sure
I plug any missing holes in the API. If you really want something to try
for fun go for a sqlite3-backed setup (don't see it going in the stdlib but
it would be a project to have).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130328/7463eeea/attachment.html>