[Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

[Paul Moore]

On 1 February 2014 17:26, Brett Cannon <brett at python.org> wrote:
> PEP 302 tried to unify this with get_data() and set_data() on loaders, but
> prior to Python 3.3 you just didn't have any guarantee that __loader__ would
> even be set, let alone have a loader with those methods. Paul can tell me if
> my hunch is off, but I assume the dream was that people would do
> `__loader__.get_data('asset.txt')` instead of
> `os.path.join(os.path.dirname(__file__), 'asset.txt')` to read something
> bundled with your package. But as Brian has pointed out, people just have
> habits at this point of assuming unpacked files and working off of __file__.

That's right. Unfortunately, (a) people have habits, as you say, (b)
the loader API was too limited - for good reason, we couldn't add any
more without breaking more than we gained at the time, and (c) things
like ssl that need files as Donald points out are always a problem.

Setuptools (and distlib more recently) tried to address this with
unpack-on-demand tricks, but honestly non-filesystem loaders are still
such an uncommon use case that most people don't bother. It's just a
very, very slow process of change.

> I know Daniel has said he wanted some concept of a listdir() on loaders so
> that they could basically act as a virtual file system for packages, but it
> would also require locking down what relative vs. absolute paths meant in
> get_data/set_data (i.e. is it relative to the loader in the package or the
> package itself? My pref is the latter for the case of reused loaders) and
> really pushing people to use the loader APIs for reading intra-package
> "files".

That is probably the most annoying omission from PEP 302. IIRC, we did
want to add it, but couldn't. But I can't recall the reason now. Maybe
it could be remedied - it really is a pain.

Paul