[Import-SIG] Loading Resources From a Python Module/Package

[Nick Coghlan]

On 3 Feb 2015 00:18, "Brett Cannon" <brett at python.org> wrote:
>
>
>
> On Sun Feb 01 2015 at 12:28:46 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> * I don't think we should couple the concept of resource anchors
>> directly to package names (as discussed, it doesn't work for namespace
>> packages, for example). I think we *should* be able to *look up*
>> resource anchors by package name, although this may fail in some cases
>> (such as namespace packages), and that the top level API should do
>> that lookup implicitly (allowing package names to be passed wherever
>> an anchor is expected). A module object should also be usable as its
>> own anchor. I believe we should disallow the use of filesystem paths
>> as resource anchors, as that breaks the intended abstraction (looking
>> resources up relative to the related modules), and the API behaviour
>> is clearer if strings are always assumed to be referring to
>> package/module names.
>
>
> Not quite following here. So are you saying we should define the location
as ('foo.bar', 'baz/file.txt') or as ('foo.bar.baz', 'file.txt')? You say
you "don't think we should couple the concept of resource anchors directly"
but then say "we should disallow the use of filesystem paths".

I like Donald's resource anchor/resource key terminology suggestion here.

The "no filesystem paths" comment refers only to specifying the anchor
point in the module namespace, not to the key within that anchor.

>> * I *don't* think it's a good idea to incorporate this idea directly
>> onto the existing module Loader API. Better to create a new
>> "ResourceLoader" abstraction, such that we can easily provide a
>> default LocationResourceLoader. Reusing module Loader instances across
>> modules would still be permitted, reusing ResourceLoader instances
>> *would not*. This allows the resource anchor to be specified when
>> creating the resource loader, rather than on every call.
>
> You do realize that importlib.abc.ResourceLoader already exists, right?
Otherwise I'm rather confused by the terminology. =)

*blinks* Apparently I missed that. OK, guess I need a different name :)

> And are you saying that we should have special rules for
LocationResourceLoader instances such that you can not have to specify the
anchoring package and thus force loader creators to provide unique
instances per package? Or are you talking about some new thing that is tied
to specs?

Yes, I'm proposing each module will need its own resource reader, they
won't be shareable the way module loaders are. I don't think the memory
savings from sharing are worth the extra complexity.

>> * As a consequence of the previous point, the ResourceLoader instance
>> would be linked *from the module spec* (and perhaps from the module
>> globals), rather than from the module loader instance. (This is how we
>> would support using a module as its own anchor). Having a resource
>> loader defined in the spec would be optional, making it clear that
>> namespace modules (for example), don't provide a resource access API -
>> if you want to store resources inside a namespace package, you need to
>> create a submodule or self-contained subpackage to serve as the
>> resource anchor.
>
>
> So are you suggesting we add a new attribute to specs which would store a
certain ABC subclass which implements an API for loading resources?

Correct (although it would be "reading resources" with your suggested
terminology tweak).

>> * As a consequence of making a suitably configured resource loader
>> available through the module spec as part of the module finding
>> process it would become possible to access module relative resources
>> *without actually loading the module itself*.
>
> OK, you are suggesting adding a new object type and attribute to specs.
Can we call them "resource readers" so we don't conflate the "loader" term?

Yep, that sounds like a good improvement to me.

> And doing it through specs also means that the overhead of requiring the
file name not have any directory parts is not extra overhead.

I don't follow this part. I'm OK with resource keys having path separators
in them.

>> * If the import system gets a module spec where "spec.has_location" is
>> set and Loader.get_data is available, but the new
>> "spec.resource_loader" attribute is set to None, then it will set it
>> to "LocationResourceLoader(spec.origin)", which will rely solely on
>> Loader.get_data() for content access
>
>
> This is a little finicky. Are we going to simply say that we assume
spec.origin is some path that works with os.path functions? Will Windows be
okay if someone decided to standardize on / as a path separator instead of
\ ? I get this buys us support from older loader implementations but I just
want to make sure that it will work 80% of the time before we add more
implicit magic to importlib.

We'd only be assuming that loader.get_data(os.path.join(spec.origin,
resource_key) works.

It will fail in the same cases where using __file__ currently fails, but
with a potential way to fix it (i.e. providing a custom resource reader
when populating the module spec)

>> * We'd also provide an optimised FilesystemResourceLoader for use with
>> actual installed packages where the resources already exist on disk
>> and don't need to be copied to memory or a temporary directory to
>> provide a suitable API.
>>
>> * For abstract data access at the ResourceLoader API level, I like
>> "get_anchor()" (returning a suitably descriptive string such that
>> "os.path.join(anchor, <relative path>)" will work with get_data() on
>> the corresponding module Loader),
>
>
> I would rather call it get_location() since get_anchor() using 'anchor'
seems to conflate what an anchor is representing.

Yeah, I confused myself while writing that. I like anchor for the user
facing API, location for the plugin level.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20150203/2f448e56/attachment.html>