[Distutils] Comments on PEP 426

[Nick Coghlan]

On 20 August 2013 23:25, Antoine Pitrou <antoine at python.org> wrote:
>
> Hello,
>
> Some comments about PEP 426:
>
>> The information defined in this PEP is serialised to pydist.json files for
> some > use cases. These are files containing UTF-8 encoded JSON metadata.
>
> Perhaps add that on-disk pydist.json files may/should be generated in printed
> form with sorted keys, to ease direct inspection by users and developers?

Good point: https://bitbucket.org/pypa/pypi-metadata-formats/issue/7/require-sorted-keys-when-serialising

>> Source labels MUST be unique within each project and MUST NOT match any
>> defined version for the project.
>
> Is there a motivation for the "not matching any defined version"?
> AFAICT it makes it necessary to have two different representation
> schemes, e.g. "X.Y.Z" for source labels and "vX.Y.Z" for versions.


>> For source archive references, an expected hash value may be specified by
>> including a ``<hash-algorithm>=<expected-hash>`` entry as part of the URL
>> fragment.
>
> Why only source archive references (and not e.g. binary)?

The core metadata is for the source distribution (something that can
be used to rebuild the software on a trusted build server). wheel (PEP
427) is the format for built binaries. PEP 440 does cover direct
references to binary archives as dependencies, though.

>>    "project_urls": {
>>      "Documentation": "https://distlib.readthedocs.org"
>>      "Home": "https://bitbucket.org/pypa/distlib"
>>      "Repository": "https://bitbucket.org/pypa/distlib/src"
>>      "Tracker": "https://bitbucket.org/pypa/distlib/issues"
>>    }
>
> This example lacks commas.

Fixed.

>> An abbreviation of "metadistribution requires". This is a list of
>> subdistributions that can easily be installed and used together by
>> depending on this metadistribution.
>
> I don't understand what it means :-) Care to explain and/or clarify
> the purpose?
>
> (for me, "meta-requires" sounds like something that setup.py depends
> on for its own operation, but that the installed software doesn't need)
>
> (edit: I now see this is clarified in Appendix C. The section ordering
> in the PEP makes it look like "meta_requires" are the primary type of
> requires, though, while according to that appendix they're a rather
> exotic use case. Would be nice to spell that out *before* the appendices :-)).

https://bitbucket.org/pypa/pypi-metadata-formats/issue/9/move-meta_requires-after-requires

>> * MAY allow direct references
>
> What is a direct reference?

Explained in PEP 440. Will likely add some more cross references to
that when I next do a full editing pass.

>> Automated tools MUST NOT allow strict version matching clauses or direct
>> references in this field - if permitted at all, such clauses should appear
>> in ``meta_requires`` instead.
>
> Why so?

Because version pinning is almost always the wrong thing to do in
published projects, while *not* pinning is wrong if you're
redistributing an exact version of something as part of an umbrella
project. Separating the two use cases (ordinary dependencies and
redistribution) into separate fields gives the tools the info they
need to provide suitable warnings.

>
> [test requires]
>> Public index servers SHOULD NOT allow strict version matching clauses or
>> direct references in this field.
>
> Again, why? Is it important for public index servers that test
> dependencies be not pinned?

Yes. Overzealous pinning sets up dependency hell.

>> Note that while these are build dependencies for the distribution being
>> built, the installation is a *deployment* scenario for the dependencies.
>
> But there are no deployment requires, right? :)
> (or is what "meta requires" are for?)

deployment = runtime = run_requires and meta_requires

I'm currently fairly inconsistent in whether I refer to deployment or
runtime, though.

>> For example, multiple projects might supply
>> PostgreSQL bindings for use with SQL Alchemy: each project might declare
>> that it provides ``sqlalchemy-postgresql-bindings``, allowing other
>> projects to depend only on having at least one of them installed.
>
> But the automated installer wouldn't be able to suggest the various
> packages providing ``sqlalchemy-postgresql-bindings`` if none is
> installed, which should IMO discourage such a scheme.
>
>> To handle this case in a way that doesn't allow for name hijacking, the
>> authors of the distribution that first defines the virtual dependency
>> should
>> create a project on the public index server with the corresponding name,
>> and
>> depend on the specific distribution that should be used if no other
>> provider
>> is already installed. This also has the benefit of publishing the default
>> provider in a way that automated tools will understand.
>
> But then the alternatives needn't provide the "virtual dependency".
> They can just provide the "default provider", which saves the time and
> hassle of defining a well-known virtual dependency for all similar
> projects.

No, because the default provider probably won't be implemented by the
project defining the virtual dependency and may change over time.
Think of it as "symlinks for dependencies".

Using a direct dependency would only make sense if the same group of
people were developing both projects.

>> A string that indicates that this project is no longer being developed.
>> The
>> named project provides a substitute or replacement.
>
> How about a project that is no longer being developed but has no
> direct substitution? :)
> Can it use an empty string (or null / None perhaps?)

Hmm, I'll think about that one.

https://bitbucket.org/pypa/pypi-metadata-formats/issue/10/allow-obsoleted_by-to-be-set-to-null-none

>> Examples indicating supported operating systems::
>>
>>    # Windows only
>>    "supports_environments": ["sys_platform == 'win32'"]
>
> Hmm, which syntax is it exactly? In a previous section, you used
> the following example:
>
>>        "environment": "sys.platform == 'win32'"
>
> (note dot vs. underscore)

Must have missed updating those from the previous PEP. Fixed.

>>     "modules": ["chair", "chair.cushions", (...)]
>
> The example is a bit intriguing. Is it expected that both "chair" and
> "chair.cushions" be specified there, or is "chair" sufficient?

If only "chair" was declared, you wouldn't be able to search the index
server for the provider of "chair.cushions". It gets more interesting
once namespace packages get involved.

>> When installing from an sdist, source archive or VCS checkout,
>> installation
>> tools SHOULD create a binary archive using ``setup.py bdist_wheel`` and
>> then install binary archive normally (including invocation of any install
>> hooks). Installation tools SHOULD NOT invoke ``setup.py install``
>> directly.
>
> Interesting. Is "setup.py install" meant to die, or will it be redefined
> as "bdist_wheel + install_wheel"?
> (also, why is this mentioned in the postinstall hooks section, or
> even in a metadata-related PEP?)

Because it's all been co-evolving in an attempt to make sure existing
use cases are covered adequately in the new metadata. It needs to be
split out, though, and will be eventually.

>> Installation tools SHOULD treat an exception thrown by a preuninstall
>> hook as an indication the removal of the distribution should be aborted.
>
> I hope a "--force" option will be provided by such tools. Failure to
> uninstall because of buggy uninstall tools is a frustrating experience.
>
>> Extras are additional dependencies that enable an optional aspect
>> of the distribution
>
> I am confused. To me, extras look like additional provides, not
> additional dependencies. I.e. in:
>
>     "requires": ["ComfyChair[warmup]"]
>         -> requires ``ComfyChair`` and ``SoftCushions``
>
> "warmup" is an additional provide of ComfyChair, and it depends on
> SoftCushions.

All of the ComfyChair code needed for "ComfyChair[warmup]" is
installed when you install "ComfyChair". It just won't work properly
without the extra dependencies. So the extra names really only refer
to the optional dependencies, not to optional components of the
ComfyChair code base.

>>    "requires": ["ComfyChair[*]"]
>>        -> requires ``ComfyChair`` and ``SoftCushions``, but will also
>>           pick up any new extras defined in later versions
>
> This one confuses me (again :-)). What does "pick up" mean? Accept?
> Require?

If a latter versions defines another extra, this one will depend on those.

>>    pip install ComfyChair[-,:*:,*]
>>        -> installs the full set of development dependencies, but avoids
>>           installing ComfyChair itself
>
> Are all these possibilities ("-", ":*:", "*") useful in real life?

Yes. The five examples given are meant to illustrate that.

>> Environment markers
>> ===================
>
> In this section, there still are inconsistencies in the format examples
> ("sys.platform" vs. "sys_platform").
>
>> * ``platform_python_implementation``: ``platform.python_implementation()``
>> * ``implementation_name````: ``sys.implementation.name``
>
> Why two different ways to spell nearly the same thing:
>
>>>> platform.python_implementation()
> 'CPython'
>>>> sys.implementation.name
> 'cpython'
>
> (also, look at how platform.python_implementation() is implemented :-))

Older versions of Python don't have sys.implementation.

> Also, do the ordering operators ("<=", ">=", etc.) operate logically or
> lexicographically on version values?

Purely string based. Environment markers are intended to be really dumb.

>> Build labels
>> ------------
>>
>> See PEP 440 for the rationale behind the addition of this field.
>
> I can't see anything named "Build label" in PEP 426. Did you mean "source
> label"?

The rationale section is lagging a bit. It'll get some TLC next time I
do a full editing pass.

>> This version of the metadata specification continues to use ``setup.py``
>> and the distutils command syntax to invoke build and test related
>> operations on a source archive or VCS checkout.
>
> I don't really understand how Metadata 2.0 is dependent on the distutils
> command scheme. Can you elaborate?

Given an sdist, how do you build a wheel?

Given a source checkout or raw tarball, how do you build an sdist or
generate the distribution's metadata?

The whole problem of building from source is currently woefully
underspecified, and there's a lot to be said in favour of
standardising a subset of the existing setuptools command line API
(since we'll have to do that as the legacy fallback option anyway).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia