[Import-SIG] Loading Resources From a Python Module/Package

[Brett Cannon]

On Sat Jan 31 2015 at 11:13:08 AM Donald Stufft <donald at stufft.io> wrote:

>
> > On Jan 31, 2015, at 10:54 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > On 31 January 2015 at 15:47, Donald Stufft <donald at stufft.io> wrote:
> >>> It's certainly possible to add a new API that loads resources based on
> >>> a relative name, but you'd have to specify relative to *what*.
> >>> get_data explicitly ducks out of making that decision.
> >>
> >> data = __loader__.get_bytes(__name__, “logo.gif”)
> >
> > Quite possibly. It needs a bit of fleshing out to make sure it doesn't
> > prohibit sharing of loaders, etc, in the way Brett mentions. Also, the
> > fact that it needs __name__ in there feels wrong - a bit like the old
> > version of super() needing to be told which class it was being called
> > from. But in principle I don't object to finding a suitable form of
> > this.
>
> To be clear, I think using __name__ is massively better than using
> __file__,
> for one even though PEP 302 states that __file__ must be set, it actually
> doesn’t have to be set and PEP 420 doesn’t set it. Even if it did set it
> that pattern is only actually really usable for non namespace packages (of
> any type).
>

So you're starting to get into the murky corners of import. =) PEP 420
actually supercedes PEP 302, but that doesn't mean it negates it. For
backwards-compatibility importlib still sets __file__, but you're right it
isn't necessary as long as __spec__ is set.

-Brett


>
> The namespace package way of doing that is basically:
>
> for path in __path__:
>     try:
>         data = __loader__.get_data(os.path.join(path, “logo.gif”))
>     except FileNotFoundError:
>         pass
>     else:
>         break
> else:
>     raise Exception(“Cannot Find the file ‘logo.gif’”)
>
>
> Either way if a Loader isn’t specific to a particular importable name and
> can
> be re-used between them then you need a way to specify what module it’s
> relative
> to and it seems to me the *obvious* way to load a resource that is
> relative to
> a module is to tell Python you want to load a particular resource from a
> particular
> module, not to construct some (pseudo) file path that says all that
> information
> as well but requires you to know if the thing you’re importing is a Python
> module, a python package, or a namespace package.
>
> In order to make a function like pkgutil.get_data that actually works in
> all
> situations that you’d have to do something like:
>
> def get_data(package, resource):
>     mod = importlib.import_module(package)
>     if hasattr(mod, “__path__”):
>         for path in __path__:
>             try:
>                 return mod.__loader__.get_data(os.path.join(path,
> resource))
>             except FileNotFoundError:
>                 pass
>     if hasattr(mod, "__file__"):
>         d = os.path.dirname(__file__)
>         try:
>             return mod.__loader__.get_data(os.path.join(d, resource))
>         except FileNotFoundError:
>             pass
>
> This is compared to the situation where the Loaders encapsulate that logic
> for you:
>
> def get_data(package, resource):
>     mod = importlib.import_module(package)
>     try:
>         mod.__loader__.get_bytes(package, resource)
>     except FileNotFoundError:
>         pass
>
> Obviously the logic in the first function still exists, it’s just moved
> away
> from the caller needing to handle it and instead the Loader handles it,
> just
> like the loader abstracts away the __file__ location for importing a
> particular
> module.
>
> Although looking closer at the Loader().exec_module implementation, It
> appears
> that it expects something other than a string to be passed to it. So if it
> makes
> sense possibly Loader().get_bytes() etc should also expect something other
> than
> a string to identify the module as well (whatever it actually wants, I
> can’t tell).
> Then the utility functions in pkgutil or importlib.resources or whatever
> will do
> the logic to translate from a string to whatever the Loader itself wants.
>
>
> >
> > And I like the name get_bytes - much more explicit in these Python 3
> > days of explicit str/bytes distinctions :-)
> > Paul
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20150131/cbae5723/attachment.html>