[Python-Dev] PEP 376

[Tarek Ziadé]

2009/7/3 Paul Moore <p.f.moore at gmail.com>:
> This is a good point. Distutils only installs files in the filesystem
> - it has no facilities for installing packages based on any other sort
> of PEP 302 based importers. Hence, PEP 376 in principle should only
> relate to filesystem-based distributions. But it also mentions
> zipfile-based distributions: "Notice that the API is organized in five
> classes that work with directories and Zip files (so it works with
> files included in Zip files, see PEP 273 for more details [8])."
>
> This is wrong. The PEP should either (1) restrict itself to filesystem
> based implementations (leaving the problem of other PEP 302 loaders to
> systems that manage these) or (2) be defined in a sufficiently general
> way that it can be implemented for *any* PEP 302 based loader - which
> probably means extending the PEP 302 protocols - and supplying zipfile
> functions as an example of how this is used.
>
> I believe that (1) is unlikely to be sufficient for real world use.
> Zip files (eggs, py2exe embedded modules, etc) are far too important a
> real world use case to ignore. The problem with (2) is that it
> requires significant extra work. But special-casing zip files (as the
> present implementation appears to do) will break as soon as any other
> PEP 302 compliant format becomes popular.
>> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the
>> ``pkg_resources`` approach where we don't enforce resources to reside on a
>> traditional filesystem.
>
> On the other hand, pkgutil.get_data is the standard library means of
> reading resources from a package. This is PEP 302 compliant now. This
> new PEP doesn't affect that.
>

Right. While it would be feasible to make pgutil works with PEP 302 loaders,
we would still need to define in a generic way the content of the RECORD files.

Right now it works for directory and zipped files since it's expressed
with '/' separated paths.

And if I understand PEP 302 right, any backend would be able to handle
these paths
no matter how they are stored, as long as the implement APIs like get_data()

> What PEP 302 doesn't provide is package management. But Python itself
> doesn't provide package management, except in the form of distutils
> setup.py install - which is solely filesystem based.
>
> Maybe there's a case for extending PEP 302 and distutils to allow
> integrated management of other forms of importer format, but that's a
> huge other project, which no-one to my knowledge is even looking at.

Sounds like a fully-featured packaging managment system, which is
imho, out of scope.

And I don't see PEP 376 making it impossible for someone to build such a
packaging system on the top of distutils.

I've started one myself for the sake of experimentation, with built-in
multiversion
support, for a full replacement of site-packages.

> Eggs are fundamentally a PEP 302 zip file format. There are some extra
> bits of metadata for setuptools/easy_install in there (as I understand
> things) but essentially they are zip files. When you say "decoupling
> the egg format", I assume you mean "decoupling the egg metadata" -
> which is fine, but to properly decouple, you need API level access to
> the metadata. PEP 376 offers read-only access, but as you rightly
> point out, it is only for filesystem data (and some form of zip file,
> which appears to be limited in some way, as it isn't PEP 302 based,
> and the actual format isn't defined anywhere).

And also PEP 376 goal is to define a single standard location of
egg-info files for filesystem data.

The zip form was built so it could work with zipped site-packages directories,
like what the py2app project uses.

>
> The basic point here is that PEP 376 needs to define precisely how
> pkgutil.get_distributions() scans sys.path looking for ".egg-info
> directories". What does it do for sys.path entries that don't
> correspond to filesystem directories? (Note - these may or may not be
> zip files. Even if they are zip files, an earlier entry on
> sys.path_hooks could have taken precedence. At the very least, you
> should only process path entries as zip files if their importer - in
> sys.path_importer_cache or via an explicit path hook scan - is a
> zipimporter object.).

I'll add more details on that part. right now it visits directories
and zip files.

Tarek