[Import-SIG] Proposed design for importlib.resources()

[Brett Cannon]

On Mon, 23 Nov 2015 at 13:46 Barry Warsaw <barry at python.org> wrote:

> On Nov 20, 2015, at 09:23 PM, Brett Cannon wrote:
>
> >I have created a Jupyter Notebook to explain my thinking on what
> >importlib.resources() should be (at least initially).
>
> Just a few thoughts based on a review of two projects' use of
> pkg_resources.
> +1 on getting *something* into Python 3.6.
>
> Module API vs package API.  Doesn't pkg_resources actually support
> something
> similar to both, with the module function providing a convenience API?


Yes, but that doesn't sway me. This isn't a "pkg_resources++" but a "make
reading data from a package make sense in a modern import world". IOW I'm
purposefully not using pkg_resources as a template but simply as a
motivating factor.


>   I like
> that a lot because the convenience API is so darn... convenient!  You just
> give it the Python dotted-path and the resource and it does the rest.
>

I don't see how that's any different than the other approach since you're
still providing the exact same data; no more, no less.


> Generally I don't care about caching the results of the search; these calls
> are almost never in performance critical code.
>

Unfortunately for you the poll liked the other approach and TOOWTDI. So
either convince me that resources.read_bytes(pkg, path) is better than
resources(pkg).read_bytes(path) or consider the bike shed painted. :)


>
> resource_filename().  Doesn't pkg_resources already have a strategy for the
> temporary file that sometimes has to be created?


Yes and I don't like it. :) Basically you either create an instance or
implicitly use a global instance of a class that stores the references and
registers with atexit a cleanup function to be executed.


>   Maybe it doesn't work so
> well on some platforms (I've never noticed a problem on *nix).  A context
> manager as proposed seems like the most reasonable approach.  We definitely
> need this API though.  I see plenty of examples where e.g. test data files
> have to be shutil.copy()'d, passed to subprocess command line arguments,
> etc.
>

OK, between you and Donald saying you have real needs for the API you can
rest assured that it will be in the initial version, especially since I
already coded up the tempfile implementation.


>
> read_bytes().  Thank you for the truth in advertising!  IIRC in Python 3,
> pkg_resource.resource_string() actually returns bytes. from-import-as to
> the
> rescue.  An actual resource_string() would have to accept an encoding
> argument
> (as would any resource-based open() method).
>

Yes, which is why I don't think it's worth it to provide a
resource_strinng() since calling decode isn't difficult (and is something
you must know in Python 3).


>
> resource_stream().  IIRC, the pkg_resource's version is not a context
> manager
> so it has to be closed explicitly (or wrapped in contextlib.closing()).  We
> can do better.
>

I'm not convinced it's necessary to provide an equivalent open() yet; if
you have an API that requires a file-like object then io.BytesIO to the
rescue for read_bytes(). There is nothing tricky to get right like with a
file path that may or may not be backed by a temporary file. This is a
somewhat low-level API and if people want to provide convenience wrappers
that's fine but I don't want to start guessing at needs beyond core APIs or
ones that are hard to get right and allow for composability to higher APIs
like file-like objects which others can handle.


>
> I do have one use of resource_listdir() which is used to find importable
> plugin modules at runtime.  It's handy.
>

I'm going to punt on this for as long as possible because it's asking for
trouble to get right. For example, if I do resources(pkg).listdir(), then I
will end up returning relative paths, but if you disassociate those paths
from pkg then you have lost proper context. You could return tuples of
(pkg, relative_path), but that just doesn't seem satisfactory either. I'm
just not convinced yet it is needed enough to support (at least initially).

-Brett


>
> That's all for now.
> -Barry
> _______________________________________________
> Import-SIG mailing list
> Import-SIG at python.org
> https://mail.python.org/mailman/listinfo/import-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20151124/63c489fd/attachment.html>