[Distutils] PEP 426: proposed metadata caching convention

[Paul Moore]

On 25 February 2013 14:57, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Feb 26, 2013 at 12:45 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> We don't really need everything to be in a single file, surely?
>
> Yes, I want the metadata to map cleanly to a single data structure so
> it can be more easily managed through things that *aren't* file
> systems (such as finally getting the installation database to support
> import hooks and also for potential metadata publication through TUF).

Fair point. OK, I can accept that the metadata stays in one file.

> However, decomposing it for efficient runtime access and backwards
> compatibility reasons makes sense.

I'm not entirely sure what code will be responsible for that
decomposition, though. In theory, it's obvious ("the installer") but
real life is more complex. Consider the following toolchain (which is
a real-life example of something I'm fiddling with at the moment):

1. setup.py -> use this to build a project and install it into a
temporary location (this may be pure distutils, or it may be
setuptools/distribute based)
2. distlib -> collect the data from the temporary location and put it
into a wheel
3. pip -> unpack the wheel and install it into a virtualenv

In this case, (1) is an "installer" and so should write the decomposed
files. But (2) doesn't want them and must then delete or otherwise
skip them. Then (3) recreates them. That's both wasteful, and
potentially complex/error-prone (distlib doesn't do the skipping of
decomposed metadata files, that's in user code as things stand at the
moment).

Maybe the simplest solution is to say that setup.py install is not,
technically an installer - it's used as a component of a lot of
builder-type toolchains (another example is bdist_wininst). So maybe
setup.py install needs to grow an option to *not* create the
decomposed files (at the same time as it gains the ability to write
Metadata 2.0). But this may impact setuptools, not just the stdlib (I
don't know if setuptools overrides the install command, but I suspect
so).

Handling of metadata format conversion (egg-info to dist-info,
ignoring or copying extra files, when to decompose and ignore cached
metadata) is fast becoming the most complicated bit of the whole
process (I say this from experience of writing installers and
converters using distlib). I'm getting very sick of writing variations
on convert_egg_info functions with a variety of different subtle
issues. But I suppose that's just saying that we need a transition
plan, which you know.

Paul.