<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore <span dir="ltr"><<a href="mailto:p.f.moore@gmail.com" target="_blank">p.f.moore@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 28 March 2013 13:42, Brett Cannon <<a href="mailto:brett@python.org">brett@python.org</a>> wrote:<br>
>> importer, as I wanted to try a proof of concept importer based on the<br>
>> new importlib stuff (which is intended to make writing custom<br>
>> importers easier), and I really struggled to get something working.<br>
><br>
> Struggling how? With the finder? The loader? What exactly were you trying to<br>
> accomplish and how were you deviating from the standard import system?<br>
<br>
</div>What I was trying to do was to write a path hook that would allow me<br>
to add a string to sys.path which contained a base-64 encoded zipfile<br>
(plus some sort of marker so it could be distinguished from a normal<br>
path entry) and as a result the contents of that embedded zip file<br>
would be available as if I'd added an actual zip file with that<br>
content to sys.path.<br>
<br>
I got in a complete mess as I tried to strip out the (essentially<br>
non-interesting) zipfile handling to get to a dummy "do nothing,<br>
everything is valid" type of example. But I don't think I would have<br>
fared much better if I'd stuck to the original full requirement.<br>
<div class="im"><br>
>> It seems to me that the importlib documentation doesn't help much for<br>
>> people trying to import path hooks.<br>
><br>
> There is a bug to clarify the docs to have more geared towards writing new<br>
> importers instead of just documenting what's available:<br>
> <a href="http://bugs.python.org/issue15867" target="_blank">http://bugs.python.org/issue15867</a><br>
<br>
</div>Thanks. I'll keep an eye on that.<br>
<div class="im"><br>
>> But it might be just me. Does<br>
>> anyone have an example of a simple importlib-based finder/loader?<br>
><br>
> Define simple. =) I would argue importlib itself is easy enough to read.<br>
<br>
</div>:-) Fair point. I guess importlib is not *that* hard to read, but the<br>
only case that implements packages in the filesystem one, and that<br>
also deals with C extensions and other complexities that I don't have<br>
a need for. I'll try again to have a deeper look at it, but I didn't<br>
find it easy to extract the essentials when I looked before.<br>
<div class="im"><br>
>> That<br>
>> would be a huge help for me. In return for any pointers, I'll look at<br>
>> putting together a doc patch to clarify how to use importlib to build<br>
>> your own path hooks :-)<br>
><br>
> Do you specifically mean the path hook aspect or the whole package of hook,<br>
> finder, and loader?<br>
<br>
</div>OK, after some more digging, it looks like I misunderstood the process<br>
somewhat. Writing a loader that inherits from *both* FileLoader and<br>
SourceLoader,</blockquote><div><br></div><div style>You only need SourceLoader since you are dealing with Python source. You don't need FileLoader since you are not reading from disk but an in-memory zipfile.</div><div>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> and then implementing get_data (and module_repr - why do<br>
I need that, couldn't the ABC offer a default implementation?)</blockquote><div><br></div><div><a href="http://bugs.python.org/issue17093">http://bugs.python.org/issue17093</a> and <a href="http://bugs.python.org/issue17566">http://bugs.python.org/issue17566</a><br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> does<br>
the job for that.<br></blockquote><div><br></div><div style>You should be implementing get_data, get_filename, and path_stats for SourceLoader.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
But the finder confuses me. I assume I want a PathEntryFinder and<br>
hence I should implement find_loader().</blockquote><div><br></div><div style>Yes since you are pulling from sys.path.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The documentation on what I<br>
need to return from there is very sparse... In the end I worked out<br>
that for a package, I need to return (MyLoader(modulename,<br>
'foo/__init__.py'), ['foo']) (here, "foo" is my dummy marker for my<br>
example).</blockquote><div><br></div><div style>The second argument should just be None: "An empty list can be used for portion to signify the loader is not part of a [namespace] package". Unfortunately a key word is missing in that sentence. <a href="http://bugs.python.org/issue17567">http://bugs.python.org/issue17567</a></div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> In essence, PathEntryFinder really has to implement some<br>
form of virtual filesystem mount point, and preserve the standard<br>
filesystem semantics of modules having a filename of .../__init__.py.<br></blockquote><div><br></div><div style>Well, if your zip file decided to create itself with a different file extension then it wouldn't be required, but then other people's code might break if they don't respect module abstractions (i.e. looking at __package__/__name__ or __path__ to see if something is a package).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
So I managed to work out what was needed in the end, but it was a lot<br>
harder than I'd expected. On reflection, getting the finder semantics<br>
right (and in particular the path entry finder semantics) was the hard<br>
bit.<br></blockquote><div><br></div><div style>Yep, that bit has had the least API tweaks as most people don't muck with finders but with loaders.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm now 100% sure that some cookbook examples would help a lot. I'll<br>
see what I can do.<br></blockquote><div><br></div><div style>I plan on writing a pure Python zip importer for Python 3.4 which should be fairly minimal and work out as a good example chunk of code. And no one need bother writing it as I'm going to do it myself regardless to make sure I plug any missing holes in the API. If you really want something to try for fun go for a sqlite3-backed setup (don't see it going in the stdlib but it would be a project to have).</div>
</div></div></div>