[Python-Dev] Writing importers and path hooks

[Paul Moore]

On 28 March 2013 13:42, Brett Cannon <brett at python.org> wrote:
>> importer, as I wanted to try a proof of concept importer based on the
>> new importlib stuff (which is intended to make writing custom
>> importers easier), and I really struggled to get something working.
>
> Struggling how? With the finder? The loader? What exactly were you trying to
> accomplish and how were you deviating from the standard import system?

What I was trying to do was to write a path hook that would allow me
to add a string to sys.path which contained a base-64 encoded zipfile
(plus some sort of marker so it could be distinguished from a normal
path entry) and as a result the contents of that embedded zip file
would be available as if I'd added an actual zip file with that
content to sys.path.

I got in a complete mess as I tried to strip out the (essentially
non-interesting) zipfile handling to get to a dummy "do nothing,
everything is valid" type of example. But I don't think I would have
fared much better if I'd stuck to the original full requirement.

>> It seems to me that the importlib documentation doesn't help much for
>> people trying to import path hooks.
>
> There is a bug to clarify the docs to have more geared towards writing new
> importers instead of just documenting what's available:
> http://bugs.python.org/issue15867

Thanks. I'll keep an eye on that.

>> But it might be just me. Does
>> anyone have an example of a simple importlib-based finder/loader?
>
> Define simple. =) I would argue importlib itself is easy enough to read.

:-) Fair point. I guess importlib is not *that* hard to read, but the
only case that implements packages in the filesystem one, and that
also deals with C extensions and other complexities that I don't have
a need for. I'll try again to have a deeper look at it, but I didn't
find it easy to extract the essentials when I looked before.

>> That
>> would be a huge help for me. In return for any pointers, I'll look at
>> putting together a doc patch to clarify how to use importlib to build
>> your own path hooks :-)
>
> Do you specifically mean the path hook aspect or the whole package of hook,
> finder, and loader?

OK, after some more digging, it looks like I misunderstood the process
somewhat. Writing a loader that inherits from *both* FileLoader and
SourceLoader, and then implementing get_data (and module_repr - why do
I need that, couldn't the ABC offer a default implementation?) does
the job for that.

But the finder confuses me. I assume I want a PathEntryFinder and
hence I should implement find_loader(). The documentation on what I
need to return from there is very sparse... In the end I worked out
that for a package, I need to return (MyLoader(modulename,
'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
example). In essence, PathEntryFinder really has to implement some
form of virtual filesystem mount point, and preserve the standard
filesystem semantics of modules having a filename of .../__init__.py.

So I managed to work out what was needed in the end, but it was a lot
harder than I'd expected. On reflection, getting the finder semantics
right (and in particular the path entry finder semantics) was the hard
bit.

I'm now 100% sure that some cookbook examples would help a lot. I'll
see what I can do.

Thanks,
Paul