[Distutils] Comments on PEP 426 and 459

Nick Coghlan ncoghlan at gmail.com
Wed Apr 9 12:41:07 CEST 2014


On 8 April 2014 08:44, Daniel Holth <dholth at gmail.com> wrote:
> On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 7 Apr 2014 23:15, "Daniel Holth" <dholth at gmail.com> wrote:
>> Meta distribution is an essential part of encouraging decoupled component
>> design, without increasing the complexity of consuming an overall "release".
>
> Is the entire justification for this feature not "we don't trust you
> to put == in run_requires", while exposing the same now-frustated
> users to the same problem once they rename run_ to meta_? Instead of
> having a separate but nearly identical top-level key, the == in
> run_requires and not providing any actual modules could cause a
> meta-distribution, and unwanted == in poorly written upstream
> dependencies could be handled by a "patch" directive in the install
> tool (like in buildout).

There's a reason I refer to the new dependency model as "semantic
dependencies" - the five categories are based on what the dependencies
*mean*, moreso than when they're relevant. Those meanings then
translate to whether or not pinning is appropriate on a public index
server.

With dev vs build vs run/meta vs test, the distinctions are obvious,
because they're dependencies that are needed for *specific
activities*:

- run the software (run+meta)
- run the automated test suite (test+run+meta)
- build the software (build)
- work on the software (dev+build+test+run+meta)

With run vs meta, the distinction is a little more subtle, as it
relates to the *commitments being made to users*, even though they
both apply to running the software.

The normal case is to use "run" - that means you're operating in a
pure publisher role, and integration and responding in a timely manner
to security updates for dependencies is the responsibility of the
software consumer. This is what you're going to want in almost all
cases when publishing to a public index server like PyPI.

The "meta" case is different, as that represents a *pre-integrated*
bundle of software - you're operating in a hybrid publisher/integrator
role, and hence taking on the responsibility for monitoring for and
responding to security updates in the pinned dependencies yourself. On
a public index server, this is designed to cover cases where projects
are developed and released in a coordinated fashion, and are able to
be consumed independently (so they don't depend on each other), but
when you combine them together, they must use consistent versions. I
just checked, and it turns out I haven't published a version of PEP
459 with the "constraints" extension defined at all yet (see
https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd3394216882626e66d97bcc9),
but there's a case to be made for allowing version compatibility
constraints as part of that extension (it is currently aimed at
killing off the "Supports Environments" field in the main metadata
spec, as well as allowing the scientific community to handle ABI
compatibility constraints for NumPy).

Meta-requirements are also a good fit when distributing *applications*
(rather than libraries or frameworks), and that's a use case more
likely to arise on private index servers (although it's not impossible
to do on a public index server - it would potentially make sense for
open source server applications like graphite or sentry, for example,
to make selective use of meta-requirements).

By explicitly splitting meta requirements from normal runtime
requirements, that allows us to change the affordances of the tools to
nudge less experienced users in the right direction (as run_requires
is more intuitive than meta_requires, so a lot of folks will simply
believe version pinning isn't allowed on PyPI), while still supporting
the more advanced use cases where version pinning actually makes sense
(via meta_requires). The split also makes subsequent dependency audits
easier, as it makes it clearer where the responsibility for responding
to security updates in dependencies lies (run requires = with the
software consumer, meta requires = with the software publisher).

I'll likely add a bit more on this topic to the rationale section for
the new semantic dependency system in PEP 426 (or perhaps just a
reference back to this post).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Distutils-SIG mailing list