[Python-Dev] Writing importers and path hooks
brett at python.org
Thu Mar 28 17:08:29 CET 2013
On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 28 March 2013 13:42, Brett Cannon <brett at python.org> wrote:
> >> importer, as I wanted to try a proof of concept importer based on the
> >> new importlib stuff (which is intended to make writing custom
> >> importers easier), and I really struggled to get something working.
> > Struggling how? With the finder? The loader? What exactly were you
> trying to
> > accomplish and how were you deviating from the standard import system?
> What I was trying to do was to write a path hook that would allow me
> to add a string to sys.path which contained a base-64 encoded zipfile
> (plus some sort of marker so it could be distinguished from a normal
> path entry) and as a result the contents of that embedded zip file
> would be available as if I'd added an actual zip file with that
> content to sys.path.
> I got in a complete mess as I tried to strip out the (essentially
> non-interesting) zipfile handling to get to a dummy "do nothing,
> everything is valid" type of example. But I don't think I would have
> fared much better if I'd stuck to the original full requirement.
> >> It seems to me that the importlib documentation doesn't help much for
> >> people trying to import path hooks.
> > There is a bug to clarify the docs to have more geared towards writing
> > importers instead of just documenting what's available:
> > http://bugs.python.org/issue15867
> Thanks. I'll keep an eye on that.
> >> But it might be just me. Does
> >> anyone have an example of a simple importlib-based finder/loader?
> > Define simple. =) I would argue importlib itself is easy enough to read.
> :-) Fair point. I guess importlib is not *that* hard to read, but the
> only case that implements packages in the filesystem one, and that
> also deals with C extensions and other complexities that I don't have
> a need for. I'll try again to have a deeper look at it, but I didn't
> find it easy to extract the essentials when I looked before.
> >> That
> >> would be a huge help for me. In return for any pointers, I'll look at
> >> putting together a doc patch to clarify how to use importlib to build
> >> your own path hooks :-)
> > Do you specifically mean the path hook aspect or the whole package of
> > finder, and loader?
> OK, after some more digging, it looks like I misunderstood the process
> somewhat. Writing a loader that inherits from *both* FileLoader and
You only need SourceLoader since you are dealing with Python source. You
don't need FileLoader since you are not reading from disk but an in-memory
> and then implementing get_data (and module_repr - why do
> I need that, couldn't the ABC offer a default implementation?)
http://bugs.python.org/issue17093 and http://bugs.python.org/issue17566
> the job for that.
You should be implementing get_data, get_filename, and path_stats for
> But the finder confuses me. I assume I want a PathEntryFinder and
> hence I should implement find_loader().
Yes since you are pulling from sys.path.
> The documentation on what I
> need to return from there is very sparse... In the end I worked out
> that for a package, I need to return (MyLoader(modulename,
> 'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my
The second argument should just be None: "An empty list can be used for
portion to signify the loader is not part of a [namespace] package".
Unfortunately a key word is missing in that sentence.
> In essence, PathEntryFinder really has to implement some
> form of virtual filesystem mount point, and preserve the standard
> filesystem semantics of modules having a filename of .../__init__.py.
Well, if your zip file decided to create itself with a different file
extension then it wouldn't be required, but then other people's code might
break if they don't respect module abstractions (i.e. looking at
__package__/__name__ or __path__ to see if something is a package).
> So I managed to work out what was needed in the end, but it was a lot
> harder than I'd expected. On reflection, getting the finder semantics
> right (and in particular the path entry finder semantics) was the hard
Yep, that bit has had the least API tweaks as most people don't muck with
finders but with loaders.
> I'm now 100% sure that some cookbook examples would help a lot. I'll
> see what I can do.
I plan on writing a pure Python zip importer for Python 3.4 which should be
fairly minimal and work out as a good example chunk of code. And no one
need bother writing it as I'm going to do it myself regardless to make sure
I plug any missing holes in the API. If you really want something to try
for fun go for a sqlite3-backed setup (don't see it going in the stdlib but
it would be a project to have).
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev