[Import-SIG] making it feasible to rely on loaders for reading intra-package data files

[Brett Cannon]

Over on distutils-sig it came up that getting people to not simply assume
that __file__ points to an actual file and thus avoid using open() directly
to read intra-package files is an issue. In order to make using a loader's
get_data reasonable (let alone set_data), there needs to be a clear
specification of how things are expected to work and make sure that
everything that people need is available.

The docs for importlib.ResourceLoader.get_data (
http://docs.python.org/3.4/library/importlib.html#importlib.abc.ResourceLoader.get_data)
say that things are expected to be based off of __file__, and with Python
3.4 using only absolute paths (except for __main__) that means all paths
would be absolute by default. As long as people stick to pathlib/os.path
and don't use non-standard path separators then this should just work.

But what if people don't do that? I honestly say that it should either be
explicitly undefined or that it's an IOError. IOW either we say "use
absolute paths or else you're on your own" or "use absolute paths, period".
That prevents having to make a decision as to whether a relative path is
relative to the module the loader is attached to or relative to the package
(e.g. easier for pre-module loaders or per-package loaders, respectively).
The former is more backwards-compatible so I say the docs get updated to
say that relative paths are undefined behaviour.

The second issue is whether get_data/set_data are enough or if something
else is needed, e.g. a listdir-like method. Since this is meant for
handling intra-package data my assumption is that it isn't really necessary
as chances are you know what files you included in your distribution (or at
least what the possible names are). I know some have asked for a
listdir-like API to help discover what modules are available so as to
provide a plugin API, but I view that as a separate thing and potentially
more appropriate on finders. Remember, the smaller the API service for the
common case the better for the stdlib.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140201/c69535cc/attachment.html>