Comments on PEP 426 and 459
I read through the latest versions of PEP 426 and 459 "Metadata 2.0 and extensions". Here are my comments. The PEP suggests "setup.py dist_info" is a thing. Only "setup.py egg_info" works. It might make sense to refactor bdist_wheel to include a dist_info command, but if done badly it would break distributions that rely on the pluggable nature of .egg-info generation (most every part of the egg-info directory is written by a separate plugin). Why do we care about "Source labels MUST be unique within each project and MUST NOT match any defined version for the project."? I still think meta distributions are unnecessary and hard to understand. It's either a lot of trouble to try to discourage == dependencies while still allowing them sometimes, or an overcomplicated way to try to define requirements.txt 2.0 by allowing distributions that aren't exactly distributions. This sounds like a pain: " To avoid malicious hijacking of names, when interpreting metadata retrieved from a public index server, automated tools MUST NOT pay any attention to ``"provides"`` entries that do not correspond to a published distribution. " I'm not convinced that console scripts should not continue to be a specially named entry point / export rather than their own very similar thing. Overall both peps are quite good. We need to be able to put them to use to advance our goal of providing a transparently backwards-compatible Python packaging system. My peps repository on bitbucket has an updated json schema.
On 7 Apr 2014 23:15, "Daniel Holth"
I read through the latest versions of PEP 426 and 459 "Metadata 2.0 and extensions". Here are my comments.
The PEP suggests "setup.py dist_info" is a thing. Only "setup.py egg_info" works. It might make sense to refactor bdist_wheel to include a dist_info command, but if done badly it would break distributions that rely on the pluggable nature of .egg-info generation (most every part of the egg-info directory is written by a separate plugin).
Why do we care about "Source labels MUST be unique within each project and MUST NOT match any defined version for the project."?
Hmm, I think that stems from an earlier versions that allowed requiring a source label. I'll double check.
I still think meta distributions are unnecessary and hard to understand. It's either a lot of trouble to try to discourage == dependencies while still allowing them sometimes, or an overcomplicated way to try to define requirements.txt 2.0 by allowing distributions that aren't exactly distributions.
PyObjC at least needs them, and things like OpenStack releases would need it (if published through PyPI). Meta distribution is an essential part of encouraging decoupled component design, without increasing the complexity of consuming an overall "release".
This sounds like a pain: " To avoid malicious hijacking of names, when interpreting metadata
retrieved
from a public index server, automated tools MUST NOT pay any attention to ``"provides"`` entries that do not correspond to a published distribution. "
Yup. Necessary security feature for virtual provides in an untrusted environment, though.
I'm not convinced that console scripts should not continue to be a specially named entry point / export rather than their own very similar thing.
This is mostly a usability issue, but also relates to the stricter requirements for installers.
From a usability perspective, I don't believe users should have to learn about the arbitrary entry point support just to request install time script generation. "scripts are just a specific kind of plugin" is a case of confusing one possible implementation model with the semantic model uses actually care about (where scripts and plugins are different things).
From an installer perspective, the key difference is that the *installer itself* is one of the intended consumers of the metadata, so if the installer doesn't know how to handle script metadata, it should fail, rather than silently doing the wrong thing. By contrast, most other entry points are only relevant at *runtime* - the installer doesn't need to know anything about them.
Overall both peps are quite good. We need to be able to put them to use to advance our goal of providing a transparently backwards-compatible Python packaging system.
Yeah, agreed - we're to the point now where finalising these PEPs is pretty high on the packaging agenda :)
My peps repository on bitbucket has an updated json schema.
Sweet, thanks. Cheers, Nick.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan
On 7 Apr 2014 23:15, "Daniel Holth"
wrote: I read through the latest versions of PEP 426 and 459 "Metadata 2.0 and extensions". Here are my comments.
The PEP suggests "setup.py dist_info" is a thing. Only "setup.py egg_info" works. It might make sense to refactor bdist_wheel to include a dist_info command, but if done badly it would break distributions that rely on the pluggable nature of .egg-info generation (most every part of the egg-info directory is written by a separate plugin).
Why do we care about "Source labels MUST be unique within each project and MUST NOT match any defined version for the project."?
Hmm, I think that stems from an earlier versions that allowed requiring a source label. I'll double check.
I still think meta distributions are unnecessary and hard to understand. It's either a lot of trouble to try to discourage == dependencies while still allowing them sometimes, or an overcomplicated way to try to define requirements.txt 2.0 by allowing distributions that aren't exactly distributions.
PyObjC at least needs them, and things like OpenStack releases would need it (if published through PyPI).
Meta distribution is an essential part of encouraging decoupled component design, without increasing the complexity of consuming an overall "release".
Is the entire justification for this feature not "we don't trust you to put == in run_requires", while exposing the same now-frustated users to the same problem once they rename run_ to meta_? Instead of having a separate but nearly identical top-level key, the == in run_requires and not providing any actual modules could cause a meta-distribution, and unwanted == in poorly written upstream dependencies could be handled by a "patch" directive in the install tool (like in buildout).
Is the entire justification for this feature not "we don't trust you to put == in run_requires",
I'm confused - where would the == which you speak of go? IIUC a run_requires looks like "run_requires": [ { "requires": ["SoftCushions"], "extra": "warmup" } ] where each dictionary in the list could have "requires", "extra" and "environment" keys. The "environment" value might have "==" as part of a marker expression. Are you talking about something else? Regards, Vinay Sajip
On Tue, Apr 8, 2014 at 12:12 PM, Vinay Sajip
Is the entire justification for this feature not "we don't trust you to put == in run_requires",
I'm confused - where would the == which you speak of go? IIUC a run_requires looks like
"run_requires": [ { "requires": ["SoftCushions"], "extra": "warmup" } ]
where each dictionary in the list could have "requires", "extra" and "environment" keys. The "environment" value might have "==" as part of a marker expression. Are you talking about something else?
Regards,
Vinay Sajip
That would be the == in "requires": ["SoftCushions == 4"] which IIUC in the current PEP would be allowed in meta_ but disallowed in run_requirements. Nick has said in the past that he thinks it's easier to teach the 4 specific concepts without having to [first] explain the general case. Most of our disagreements have been about this issue: whether a distinction is important enough to deserve special treatment in the spec, or whether it can be expressed when needed as a consequence of having a more general format.
On 8 April 2014 08:44, Daniel Holth
On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan
wrote: On 7 Apr 2014 23:15, "Daniel Holth"
wrote: Meta distribution is an essential part of encouraging decoupled component design, without increasing the complexity of consuming an overall "release". Is the entire justification for this feature not "we don't trust you to put == in run_requires", while exposing the same now-frustated users to the same problem once they rename run_ to meta_? Instead of having a separate but nearly identical top-level key, the == in run_requires and not providing any actual modules could cause a meta-distribution, and unwanted == in poorly written upstream dependencies could be handled by a "patch" directive in the install tool (like in buildout).
There's a reason I refer to the new dependency model as "semantic dependencies" - the five categories are based on what the dependencies *mean*, moreso than when they're relevant. Those meanings then translate to whether or not pinning is appropriate on a public index server. With dev vs build vs run/meta vs test, the distinctions are obvious, because they're dependencies that are needed for *specific activities*: - run the software (run+meta) - run the automated test suite (test+run+meta) - build the software (build) - work on the software (dev+build+test+run+meta) With run vs meta, the distinction is a little more subtle, as it relates to the *commitments being made to users*, even though they both apply to running the software. The normal case is to use "run" - that means you're operating in a pure publisher role, and integration and responding in a timely manner to security updates for dependencies is the responsibility of the software consumer. This is what you're going to want in almost all cases when publishing to a public index server like PyPI. The "meta" case is different, as that represents a *pre-integrated* bundle of software - you're operating in a hybrid publisher/integrator role, and hence taking on the responsibility for monitoring for and responding to security updates in the pinned dependencies yourself. On a public index server, this is designed to cover cases where projects are developed and released in a coordinated fashion, and are able to be consumed independently (so they don't depend on each other), but when you combine them together, they must use consistent versions. I just checked, and it turns out I haven't published a version of PEP 459 with the "constraints" extension defined at all yet (see https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd33...), but there's a case to be made for allowing version compatibility constraints as part of that extension (it is currently aimed at killing off the "Supports Environments" field in the main metadata spec, as well as allowing the scientific community to handle ABI compatibility constraints for NumPy). Meta-requirements are also a good fit when distributing *applications* (rather than libraries or frameworks), and that's a use case more likely to arise on private index servers (although it's not impossible to do on a public index server - it would potentially make sense for open source server applications like graphite or sentry, for example, to make selective use of meta-requirements). By explicitly splitting meta requirements from normal runtime requirements, that allows us to change the affordances of the tools to nudge less experienced users in the right direction (as run_requires is more intuitive than meta_requires, so a lot of folks will simply believe version pinning isn't allowed on PyPI), while still supporting the more advanced use cases where version pinning actually makes sense (via meta_requires). The split also makes subsequent dependency audits easier, as it makes it clearer where the responsibility for responding to security updates in dependencies lies (run requires = with the software consumer, meta requires = with the software publisher). I'll likely add a bit more on this topic to the rationale section for the new semantic dependency system in PEP 426 (or perhaps just a reference back to this post). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Apr 9, 2014 at 6:41 AM, Nick Coghlan
On 8 April 2014 08:44, Daniel Holth
wrote: On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan
wrote: On 7 Apr 2014 23:15, "Daniel Holth"
wrote: Meta distribution is an essential part of encouraging decoupled component design, without increasing the complexity of consuming an overall "release". Is the entire justification for this feature not "we don't trust you to put == in run_requires", while exposing the same now-frustated users to the same problem once they rename run_ to meta_? Instead of having a separate but nearly identical top-level key, the == in run_requires and not providing any actual modules could cause a meta-distribution, and unwanted == in poorly written upstream dependencies could be handled by a "patch" directive in the install tool (like in buildout).
There's a reason I refer to the new dependency model as "semantic dependencies" - the five categories are based on what the dependencies *mean*, moreso than when they're relevant. Those meanings then translate to whether or not pinning is appropriate on a public index server.
With dev vs build vs run/meta vs test, the distinctions are obvious, because they're dependencies that are needed for *specific activities*:
- run the software (run+meta) - run the automated test suite (test+run+meta) - build the software (build) - work on the software (dev+build+test+run+meta)
With run vs meta, the distinction is a little more subtle, as it relates to the *commitments being made to users*, even though they both apply to running the software.
The normal case is to use "run" - that means you're operating in a pure publisher role, and integration and responding in a timely manner to security updates for dependencies is the responsibility of the software consumer. This is what you're going to want in almost all cases when publishing to a public index server like PyPI.
The "meta" case is different, as that represents a *pre-integrated* bundle of software - you're operating in a hybrid publisher/integrator role, and hence taking on the responsibility for monitoring for and responding to security updates in the pinned dependencies yourself. On a public index server, this is designed to cover cases where projects are developed and released in a coordinated fashion, and are able to be consumed independently (so they don't depend on each other), but when you combine them together, they must use consistent versions. I just checked, and it turns out I haven't published a version of PEP 459 with the "constraints" extension defined at all yet (see https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd33...), but there's a case to be made for allowing version compatibility constraints as part of that extension (it is currently aimed at killing off the "Supports Environments" field in the main metadata spec, as well as allowing the scientific community to handle ABI compatibility constraints for NumPy).
Meta-requirements are also a good fit when distributing *applications* (rather than libraries or frameworks), and that's a use case more likely to arise on private index servers (although it's not impossible to do on a public index server - it would potentially make sense for open source server applications like graphite or sentry, for example, to make selective use of meta-requirements).
By explicitly splitting meta requirements from normal runtime requirements, that allows us to change the affordances of the tools to nudge less experienced users in the right direction (as run_requires is more intuitive than meta_requires, so a lot of folks will simply believe version pinning isn't allowed on PyPI), while still supporting the more advanced use cases where version pinning actually makes sense (via meta_requires). The split also makes subsequent dependency audits easier, as it makes it clearer where the responsibility for responding to security updates in dependencies lies (run requires = with the software consumer, meta requires = with the software publisher).
I'll likely add a bit more on this topic to the rationale section for the new semantic dependency system in PEP 426 (or perhaps just a reference back to this post).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I can live with it.
participants (3)
-
Daniel Holth
-
Nick Coghlan
-
Vinay Sajip