[Import-SIG] making it feasible to rely on loaders for reading intra-package data files

[PJ Eby]

On Sat, Feb 1, 2014 at 1:44 PM, Brett Cannon <brett at python.org> wrote:
> Over on distutils-sig it came up that getting people to not simply assume
> that __file__ points to an actual file and thus avoid using open() directly
> to read intra-package files is an issue. In order to make using a loader's
> get_data reasonable (let alone set_data), there needs to be a clear
> specification of how things are expected to work and make sure that
> everything that people need is available.
>
> The docs for importlib.ResourceLoader.get_data
> (http://docs.python.org/3.4/library/importlib.html#importlib.abc.ResourceLoader.get_data)
> say that things are expected to be based off of __file__, and with Python
> 3.4 using only absolute paths (except for __main__) that means all paths
> would be absolute by default. As long as people stick to pathlib/os.path and
> don't use non-standard path separators then this should just work.

Wait, what?  How can you define an "absolute path" when __file__ might
not be a filesystem path?  ISTM this *must* be loader-defined.

pkg_resources' loader-to-resources adapter framework specifically
abstracts a "generate an appropriate get_data() path" operation (the
'_fn()' method of ResourceProvider objects) specifically to handle the
possibility that a particular loader class handles this differently.
I don't see how this can work properly without a higher-level API for
resource management, ala pkg_resources or distlib.

This seems to me like a place where an API should be provided, rather
than have every program have to keep track of available loader
implementations.