[Import-SIG] making it feasible to rely on loaders for reading intra-package data files

[Paul Moore]

On 1 February 2014 18:44, Brett Cannon <brett at python.org> wrote:
> The second issue is whether get_data/set_data are enough or if something
> else is needed, e.g. a listdir-like method. Since this is meant for handling
> intra-package data my assumption is that it isn't really necessary as
> chances are you know what files you included in your distribution (or at
> least what the possible names are). I know some have asked for a
> listdir-like API to help discover what modules are available so as to
> provide a plugin API, but I view that as a separate thing and potentially
> more appropriate on finders. Remember, the smaller the API service for the
> common case the better for the stdlib.

One immediate use case for a listdir-type function is virtualenv.
Admittedly, listdir is only half of the problem (and the other half -
the receiving code needs a real file, not a resource - is harder to
adderss) but it is an example of where the API might be helpful.

Basically, virtualenv uses a virtualenv_support directory (package -
in that it has an __init__.py) to hold the wheels for pip and
setuptools that are to be loaded into the virtualenv being created.
While in theory we know the names of those wheels, the problem is that
wheel names encode the version of the package, so rather than have to
update the code every time we look for setuptools*.whl and pip*.whl.
Also, we support users replacing the supplied wheels with newer
versions, so the actual filenames aren't in our control anyway.

As I say, to actually use the wheels we need to have them in the
filesystem, so there are other issues that would prevent us from
removing the filesystem assumption in the short term. But we couldn't
even start without a listdir API.

BTW, an unrelated issue is that if we did go down this route with
virtualenv, we'd be looking at having a resource that is the content
of a zipfile that we'd want to put on sys.path. There's no support in
Python for putting in-memory zipfiles on sys.path. We could, and
probably would, dump the data to a temporary file in the first
instance and put that on sys.path, but in the light of this thread, is
putting in-memory zipfiles onto sys.path something that we should be
supporting?

Paul