Re: [Distutils] Comments on PEP 426 and 459

On 8 April 2014 08:44, Daniel Holth wrote:

On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan wrote:

...

On 7 Apr 2014 23:15, "Daniel Holth" wrote: Meta distribution is an essential part of encouraging decoupled component design, without increasing the complexity of consuming an overall "release".

Is the entire justification for this feature not "we don't trust you to put == in run_requires", while exposing the same now-frustated users to the same problem once they rename run_ to meta_? Instead of having a separate but nearly identical top-level key, the == in run_requires and not providing any actual modules could cause a meta-distribution, and unwanted == in poorly written upstream dependencies could be handled by a "patch" directive in the install tool (like in buildout).

There's a reason I refer to the new dependency model as "semantic dependencies" - the five categories are based on what the dependencies *mean*, moreso than when they're relevant. Those meanings then translate to whether or not pinning is appropriate on a public index server. With dev vs build vs run/meta vs test, the distinctions are obvious, because they're dependencies that are needed for *specific activities*: - run the software (run+meta) - run the automated test suite (test+run+meta) - build the software (build) - work on the software (dev+build+test+run+meta) With run vs meta, the distinction is a little more subtle, as it relates to the *commitments being made to users*, even though they both apply to running the software. The normal case is to use "run" - that means you're operating in a pure publisher role, and integration and responding in a timely manner to security updates for dependencies is the responsibility of the software consumer. This is what you're going to want in almost all cases when publishing to a public index server like PyPI. The "meta" case is different, as that represents a *pre-integrated* bundle of software - you're operating in a hybrid publisher/integrator role, and hence taking on the responsibility for monitoring for and responding to security updates in the pinned dependencies yourself. On a public index server, this is designed to cover cases where projects are developed and released in a coordinated fashion, and are able to be consumed independently (so they don't depend on each other), but when you combine them together, they must use consistent versions. I just checked, and it turns out I haven't published a version of PEP 459 with the "constraints" extension defined at all yet (see https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd33...), but there's a case to be made for allowing version compatibility constraints as part of that extension (it is currently aimed at killing off the "Supports Environments" field in the main metadata spec, as well as allowing the scientific community to handle ABI compatibility constraints for NumPy). Meta-requirements are also a good fit when distributing *applications* (rather than libraries or frameworks), and that's a use case more likely to arise on private index servers (although it's not impossible to do on a public index server - it would potentially make sense for open source server applications like graphite or sentry, for example, to make selective use of meta-requirements). By explicitly splitting meta requirements from normal runtime requirements, that allows us to change the affordances of the tools to nudge less experienced users in the right direction (as run_requires is more intuitive than meta_requires, so a lot of folks will simply believe version pinning isn't allowed on PyPI), while still supporting the more advanced use cases where version pinning actually makes sense (via meta_requires). The split also makes subsequent dependency audits easier, as it makes it clearer where the responsibility for responding to security updates in dependencies lies (run requires = with the software consumer, meta requires = with the software publisher). I'll likely add a bit more on this topic to the rationale section for the new semantic dependency system in PEP 426 (or perhaps just a reference back to this post). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia