[Distutils] PEP 426: proposed metadata caching convention

[Daniel Holth]

On Wed, Feb 27, 2013 at 4:48 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Mon, Feb 25, 2013 at 9:39 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> (This probably belongs in a successor to PEP 376, but I'll leave it
>> under the PEP 426 umbrella for now)
>>
>> One of the points raised regarding PEP 426's integrated metadata
>> format is the potential for runtime issues with pkg_resources as it
>> reads and processes the metadata during startup, particularly if it
>> needs to process any environment markers. While I acknowledge the
>> suggestions I have received that we should really be moving away from
>> the current filesystem based distributed installation information to a
>> real database that properly handle import hooks, I'm looking for
>> something simpler that will make it easier for setuptools and
>> distribute to consume the new metadata format (and thus hopefully make
>> them more amenable to generating it as well)
>>
>> Assuming we add an Entry-Points field as I have proposed in another
>> message, I'd like to propose that installers generate three additional
>> cache files as part of the installation process:
>>
>>     <dist-info-dir>/__cache__/version.txt
>>     <dist-info-dir>/__cache__/requires-dist.txt
>>     <dist-info-dir>/__cache__/entry-points.txt
>>
>> version.txt would just be the version of the installed distribution
>> (no need to parse the main metadata file just to read the version
>> field)
>>
>> requires-dist.txt would be similar to the pkg_resources requires.txt
>> format, but use PEP 426 version specifiers. It would:
>> - only contain runtime requirements where the environment markers
>> match the current system
>> - be split into sections based on the "extras" definition needed to
>> get the environment marker to pass
>>
>> entry-points.txt would be the same format as the pkg_resources entry_points.txt
>>
>> Cheers,
>> Nick.
>
> Since this isn't going to be backwards-compatible anyway, may I suggest that:
>
> 1. The caching algorithm be fixed and defined as part of the extension machinery
> 2. The caching consists of simply copying the data to a file, whose
> name is programmatically based on the extension/field name.
> 3. Environment markers are not processed - that's up to the tool
> consuming the cached data
>
> This way, if e.g. entry points are defined as an extension, then the
> Builder making a wheel doesn't need to "understand" entry points, it
> just has to copy fields to a file.  It allows other resource types
> (like i18n/l10n resources) to be defined in the metadata and cached
> for runtime use, without needing a metadata version upgrade or any
> tool rewrites.  And not processing environment markers means that
> pure-Python wheels can still be used by just placing them on sys.path.

My aim is to provide a hook mechanism that specifically does not say
anything about the way the cache is stored or even whether the hook
produces a cache at all. It will just run when pip is done.