[Distutils] Upcoming changes to PEP 426/440

[Nick Coghlan]

On 30 June 2013 18:53, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>
>> No, because the semantic dependencies form a Cartesian product with
>> the extras. You can define :meta:, :run:, :test:, :build: and :dev:
>> dependencies for any extra. So if, for example, you have a "localweb"
>> extra, then you can define extra test dependencies for that.
>>
>> The semantic specifiers determine *which sets of dependencies* you're
>> interested in, while the explicit extras define optional subsets of
>> those.
>
> Isn't that the same as having an additional field in the dependency mapping?
> It seems like that's how one would organise it at the RDBMS level, anyway.
>
> {
>   "install": "localweb-test-util [win] (>= 1.0)",
>   "extra": "localweb",
>   "environment": "sys_platform == 'win32'",
>   "kind": ":test:"
> }

You certainly *could* define it like that, but no existing dependency
system I'm aware of does it that way. If they allow for anything other
than runtime dependencies in the first place, they define a different
top level field:

* setuptools has requires and install_requires
* PEP 346 has Requires-Dist and Setup-Requires-Dist
* RPM has Requires and BuildRequires
* npm has dependencies and devDependencies

The different kinds of semantic dependency make fundamentally
different statements about the distributions being referenced. In
particular, automated tools (especially PyPI) may place different
constraints on the kind of specifier they allow in each field. PEP 426
explicitly *requires* that runtime dependencies be split between
:meta: (which requires exact version specifiers) and :run: (which
disallows them) so tools can issue appropriate warnings or errors when
someone pins a specific version without explicitly stating their
intent to monitor that dependency responsibly. That kind of difference
in what's appropriate indicates they're only the "same thing" at a
superficial syntactic level.

I decided to merge the *_requires and *_may_require fields because
they had syntactic differences that made it hard to do unified
processing on them. That's what Donald noticed and pointed out to me
off-list - the need for two distinct code paths just to deal with the
fact that *_requires used strings while *_may_require used a mapping
with a nested list, even when the natural model in the database for an
unconditional dependency is the same as a conditional dependency with
a NULL extra and environment field. Having them split as they
currently are in the PEP is also directly responsible for the odd
"must define extra, environment or both" constraint on the
"*_may_require" field entries.

Merging the fields eliminates all that complexity - the only required
subfield is now "install", with the two conditional fields both
optional. This means unconditional dependencies can be read and
written using the same code path as conditional dependencies, since
the conditional dependency code already needed to cope with the fact
the "extra" or "environment" subfields might be missing.

We don't get the same level of payoff by switching to a "kind"
subfield, because all five dependency fields already use the same
internal syntax. Whether you're keying off the top level field name,
or keying off a "kind" subfield, the processing code will still be
identical across all five kinds of dependency.

As far as data modelling goes, Warehouse actually splits the different
kinds of dependency as distinct ORM classes. This allows various
useful details (like the descriptive names for each kind of
dependency) to be handled directly in the data model, rather than
needing to be tracked separately.

While the two forms are functionally equivalent, I still prefer
multiple top level fields, as I consider it both easier to document
and more consistent with the approach used by other packaging systems.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia