Adding Provides-Extra as an edit to PEP 345
Hopefully this will be a more useful spec. This begs for parenthesis in environment markers, something that is easy to do by compiling markers to Python functions using the ast as `markerlib` does. Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be a valid Python identifier. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' ... Full changeset at https://bitbucket.org/dholth/python-peps/changeset/4dc72bbf9ed2
On Wed, Jul 4, 2012 at 7:40 AM, Daniel Holth
Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be a valid Python identifier. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf'
+1
Add Provides-Extra: to PEP 345 and clarify Requires-Dist: name (1) is the same as Requires-Dist: name (==1)
+1
Great. How do we get https://bitbucket.org/dholth/python-peps merged into hg.python.org/peps/ ?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/06/2012 10:32 AM, Daniel Holth wrote:
Add Provides-Extra: to PEP 345 and clarify Requires-Dist: name (1) is the same as Requires-Dist: name (==1)
+1
Great. How do we get https://bitbucket.org/dholth/python-peps merged into hg.python.org/peps/ ?
Given that PEP 345 is already in 'Accepted' state, I'm not sure we can just merge it in. We might need to create a new PEP which extends PEP 345 and adds the feature (it likely needs to bump the 'Metadata-Version', too). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/2+PQACgkQ+gerLs4ltQ57KwCfWR5W+gL0bgLOLOcyyOlXCI7e KTIAoLDLUTjNQjw9VA9Hx8R4rYbW5XqZ =pl62 -----END PGP SIGNATURE-----
Hi,
Hopefully this will be a more useful spec. Thanks for the proposal, but it is not really a spec. You’re suggesting name and syntax for a new field, but you don’t describe what it means and what it is used for. From what I understand, the goal is similar to build dependencies and optional dependencies in systems like Debian’s dpkg; for the build dependencies, there were multiple threads about adding Setup-Requires-Dist (like setuptools’ setup_requires) and Test-Requires-Dist (like tests_require, or whatever the spelling is—I always get it wrong :), for which we were near to agreement. See https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-... and https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-...
This begs for parenthesis in environment markers, something that is easy to do by compiling markers to Python functions using the ast as `markerlib` does. distutils2 supports Python 2.5, which has _ast but not ast IIRC. If you have other suggestions for improvements like this please open bug reports though! Feedback and patches are welcome.
Regards
Given that PEP 345 is already in 'Accepted' state, I'm not sure we can just merge it in. We might need to create a new PEP which extends PEP 345 and adds the feature (it likely needs to bump the 'Metadata-Version', too).
I would have thought so too, but then I downloaded 16,984 of the newest sdists hosted on pypi, parsed their PKG-INFO or METADATA files, and discovered adoption is quite low. Also, it is not Final. 'Metadata-Version: 1.0': 15384, 'Metadata-Version: 1.02': 1, 'Metadata-Version: 1.1': 1383, 'Metadata-Version: 1.2': 7,
On Fri, Jul 6, 2012 at 11:01 AM, Éric Araujo
Hopefully this will be a more useful spec.
Thanks for the proposal, but it is not really a spec. You’re suggesting name and syntax for a new field, but you don’t describe what it means and what it is used for. From what I understand, the goal is similar to build dependencies and optional dependencies in systems like Debian’s dpkg; for the build dependencies, there were multiple threads about adding Setup-Requires-Dist (like setuptools’ setup_requires) and Test-Requires-Dist (like tests_require, or whatever the spelling is—I always get it wrong :), for which we were near to agreement. See https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-... and https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-...
It is exactly the same as setuptools' "extras" feature, and I didn't quote the entire edit on-list. Also, there is a reference implementation at https://bitbucket.org/dholth/distribute/ in the DistInfoDistribution class. The 'test' and 'doc' reserved names are not an essential part of the Provides-Extra feature. Provides-Extra: is primarily for optional run-time (not build-time) dependencies.
This begs for parenthesis in environment markers, something that is easy to do by compiling markers to Python functions using the ast as `markerlib` does.
distutils2 supports Python 2.5, which has _ast but not ast IIRC. If you have other suggestions for improvements like this please open bug reports though! Feedback and patches are welcome.
markerlib is not required for Provides-Extra: It was just a convenient way to add the additional 'extra' variable name to environment markers. markerlib uses a few helpers from ast.py. Python 2.5 support would mean copying those helpers into markerlib.
You're right, the example could be better. It works like so: beaglevote's METADATA: Provides-Extra: quux Provides-Extra: pdf Requires-Dist: doghouse Requires-Dist: sphinx; extra == 'quux' Requires-Dist: reportlab; extra == 'pdf' "install beaglevote" -> evaluate Requires-Dist: with extra == None: -> requires doghouse "install beaglevote[pdf]" -> set(Requires-Dist: with extra == None) + set(Requires-Dist: with extra == 'pdf') -> requires doghouse, reportlab "install beaglevote[quux]" -> set(Requires-Dist: with extra == None) + set(Requires-Dist: with extra == 'quux') -> requires doghouse, sphinx "install beaglevote[pearl]" -> error, 'pearl' is not mentioned in Provides-Extra: Extras are only supposed to add requirements. To make things as predictable as possible, extra is a string, not a list, requirements are always evaluated with extra == None first, and the requirements are re-evaluated for each requested extra individually. This is to avoid mischief like Requires-Dist: something; 'foo' in extra and not 'bar' in extra Requires-Dist: another; extra != 'quux' Parenthesis would be desirable so you can say: Requires-Dist: splort; (python_version < '2.6' or sys.platform == 'win32') and extra == 'blurble'
On Fri, Jul 06, 2012 at 11:01:33AM -0400, Éric Araujo wrote:
Hi,
Hopefully this will be a more useful spec. Thanks for the proposal, but it is not really a spec. You’re suggesting name and syntax for a new field, but you don’t describe what it means and what it is used for. From what I understand, the goal is similar to build dependencies and optional dependencies in systems like Debian’s dpkg;
You can also think of extras as similar to Debian's metapackages. For example, looking at the zope.component packaging: http://packages.debian.org/source/sid/zope.component The zope.component [zcml] extra is translated to metapackage (python-zope.component-zcml) that depends on python-zope.component and python-zope.configuration. -- Brian Sutherland
And some stats: of the 16k sdists I surveyed, 9484 would need to use 'Requires-Dist:', and 1624 define what would be a Provides-Extra: line. I found 483 unique extra names. So Provides-Extra: really has nothing to do with tests or docs. I'm sorry I included that distracting remark in the specification. Provides-Extra: is all about requiring dependencies that enable an optional feature of a distribution. Some of the most popular extra names: [(5, 'kid'), (5, 'lxml'), (5, 'oracle'), (5, 'pylons'), (5, 'simplejson'), (5, 'wsgi'), (5, 'xml'), (6, 'all'), (6, 'django'), (6, 'jinja2'), (6, 'nose'), (7, 'sphinx'), (8, 'mysql'), (9, 'doc'), (10, 'zcml'), (12, 'app'), (12, 'zopesupport'), (13, 'dev'), (20, 'genshi'), (22, 'mako'), (41, 'testing'), (60, 'docs'), (305, 'tests'), (1053, 'test')]
On Fri, Jul 6, 2012 at 11:20 AM, Daniel Holth
It is exactly the same as setuptools' "extras" feature, and I didn't quote the entire edit on-list. Also, there is a reference implementation at https://bitbucket.org/dholth/distribute/ in the DistInfoDistribution class.
The 'test' and 'doc' reserved names are not an essential part of the Provides-Extra feature. Provides-Extra: is primarily for optional run-time (not build-time) dependencies.
Though in the interest of not having too many different ways to do a similar thing, could this not also be used for tests/docs? Perhaps even "Provides-Extra: tests", and "Provides-Extra: docs" should be reserved names as you suggested. I envision something like: Provides-Extra: tests Requires-Dist: nose; extra == 'tests' Provides-Extra: docs Requires-Dist: sphinx; extra == 'docs' $ pysetup test Tests require the 'nose' package. Please run `pysetup install mypackage[tests]` to install the test dependencies. Ditto for building the docs. In a sense, being able to run the tests, or build the docs could be considered 'extra' features. One could even have: Provides-Extra: setup Requires-Dist: mysetuphooks; extra == 'setup' for build-time dependencies. Although in that case, pysetup (or any other installer) would explicitly have to check for this in the metadata before doing anything else. So maybe this example stretches this syntax too far? Plus build-time dependencies are not exactly "extras". It's very much required to do anything with the package. Erik
Though in the interest of not having too many different ways to do a similar thing, could this not also be used for tests/docs? Perhaps even "Provides-Extra: tests", and "Provides-Extra: docs" should be reserved names as you suggested. I envision something like:
...
Provides-Extra: setup Requires-Dist: mysetuphooks; extra == 'setup'
for build-time dependencies. Although in that case, pysetup (or any other installer) would explicitly have to check for this in the metadata before doing anything else. So maybe this example stretches this syntax too far? Plus build-time dependencies are not exactly "extras". It's very much required to do anything with the package.
Pre-built or binary packages won't want to install the build-time dependencies. The check for a particular Provides-Extra is trivial, just a list membership test. I expect to see a lot of bugs in packages that require some of their build-time dependencies at run time.
On Mon, Jul 9, 2012 at 1:27 PM, Daniel Holth
Though in the interest of not having too many different ways to do a similar thing, could this not also be used for tests/docs? Perhaps even "Provides-Extra: tests", and "Provides-Extra: docs" should be reserved names as you suggested. I envision something like:
...
Provides-Extra: setup Requires-Dist: mysetuphooks; extra == 'setup'
for build-time dependencies. Although in that case, pysetup (or any other installer) would explicitly have to check for this in the metadata before doing anything else. So maybe this example stretches this syntax too far? Plus build-time dependencies are not exactly "extras". It's very much required to do anything with the package.
Pre-built or binary packages won't want to install the build-time dependencies. The check for a particular Provides-Extra is trivial, just a list membership test.
I expect to see a lot of bugs in packages that require some of their build-time dependencies at run time.
Maybe, but wouldn't that be the developer's problem? setup_requires has the same issue--it does not install the setup requirements into site-packages. Instead it just does an egg install into the source directory and activates it on the path. Outside of future setup.py runs, the setup_requires distributions are not available (you would have to add them to install_requires too if you need them at runtime). Under a scheme like this one would have to list that dependency under Requires-Dist twice: with and without the 'extra' marker. I might still prefer extending the metadata format to add a Setup-Requires-Dist or the like. Erik
I expect to see a lot of bugs in packages that require some of their build-time dependencies at run time.
Maybe, but wouldn't that be the developer's problem? setup_requires has the same issue--it does not install the setup requirements into site-packages. Instead it just does an egg install into the source directory and activates it on the path. Outside of future setup.py runs, the setup_requires distributions are not available (you would have to add them to install_requires too if you need them at runtime).
OK, it sounds like this "bug" has been mostly anticipated and taken care of already.
Under a scheme like this one would have to list that dependency under Requires-Dist twice: with and without the 'extra' marker. I might still prefer extending the metadata format to add a Setup-Requires-Dist or the like.
I don't follow. Does anyone build/install packages without also first installing the run-time requirements? Surely Setup-Requires-Dist would have exactly the same problem. It seems like two ways to say exactly the same thing. Let's just try to focus on getting Metadata 1.2 out the door and implemented before its 8th birthday, preferably during Julython... The only thing I need is to be able to represent the wildly popular "extras" feature in METADATA to benefit from PEP 345 and 376 without rewriting thousands of Python packages. That is all. Save the debate about reserved "extra" names for setup / build / test / doc dependencies for Metadata 1.3.
On Mon, Jul 9, 2012 at 4:37 PM, Daniel Holth
I expect to see a lot of bugs in packages that require some of their build-time dependencies at run time.
Maybe, but wouldn't that be the developer's problem? setup_requires has the same issue--it does not install the setup requirements into site-packages. Instead it just does an egg install into the source directory and activates it on the path. Outside of future setup.py runs, the setup_requires distributions are not available (you would have to add them to install_requires too if you need them at runtime).
OK, it sounds like this "bug" has been mostly anticipated and taken care of already.
Under a scheme like this one would have to list that dependency under Requires-Dist twice: with and without the 'extra' marker. I might still prefer extending the metadata format to add a Setup-Requires-Dist or the like.
I don't follow. Does anyone build/install packages without also first installing the run-time requirements? Surely Setup-Requires-Dist would have exactly the same problem. It seems like two ways to say exactly the same thing.
The difference is that setup/build dependencies generally should be downloaded, extracted, and added to the path before another commands are run, because the packaging framework allows for the behavior of built-in commands themselves to be modified, as well as certain metadata. The usefulness of this has been discussed in past threads. See http://mail.python.org/pipermail/distutils-sig/2012-May/018627.html and the thread Eric Araujo pointed out at: https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-...
Let's just try to focus on getting Metadata 1.2 out the door and implemented before its 8th birthday, preferably during Julython... The only thing I need is to be able to represent the wildly popular "extras" feature in METADATA to benefit from PEP 345 and 376 without rewriting thousands of Python packages. That is all. Save the debate about reserved "extra" names for setup / build / test / doc dependencies for Metadata 1.3.
I don't have data on the "popularity" of setup_requires--is there some easy way to find that out without downloading an entire PyPI mirror? That said, I'm anticipating that it would be very useful for packaging. As it is, setup.py files tend to contain a lot of repetitive boilerplate, especially for non-pure-Python packages. With the elimination of setup.py, more packages are going to require setup hook-script modules to be shipped with their code, and I could see some of these being highly reusable. I'm certain this is a useful feature--just not so certain whether it's desirable needed to modify the Metadata spec to account for it. In the worst case, I could imagine writing some kind of "bootstrap" module that can be dropped into a project and used via a setup-hook that is then possible of downloading other build-time dependencies. But that's exactly the sort of thing I'm trying to get away from--if I have to maintain a bootstrap script (or another module containing setup-hooks, custom commands, etc.) I have to update every project that uses it every time those scripts are updated. Erik
Under a scheme like this one would have to list that dependency under Requires-Dist twice: with and without the 'extra' marker. I might still prefer extending the metadata format to add a Setup-Requires-Dist or the like.
The difference is that setup/build dependencies generally should be downloaded, extracted, and added to the path before another commands are run, because the packaging framework allows for the behavior of built-in commands themselves to be modified, as well as certain metadata. The usefulness of this has been discussed in past threads.
Setup requirements are certainly necessary. I see what you mean now. 'Setup-Requires-Dist' is a different feature than 'Provides-Extra', since with 'Provides-Extra' there isn't an elegant way to figure out what "just the setup requirements are". Provides-Extra is only useful if you want to install the mandatory runtime requirements at the same time (which is exactly what you want to do for 'extras'). You could make the argument that you only need setup_requires_dist in setup.cfg... since it will generate METADATA the first time (when downloading from revision control) anyway. But it can also go in METADATA, you will get no complaint from me. Of course I did download PyPI, that's the way to tell. Just under 80% of the newest sdists for each package on PyPI contain PKG-INFO. How many phases does a build + install have? Is this about right? 1. Install setup requirements "Setup-Requires-Dist" from .dist-info/METADATA 2. Run setup "dist_info" (pip always runs egg_info) to get a .dist-info directory with METADATA inside 3. Read up-to-date runtime requirements "Requires-Dist" 4. Install those 5. Install package setup "install"
New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068 Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist: (full text below) Setup-Requires-Dist lives because it makes sense to list packages only needed for setup.py to run before install is possible. The reserved extra names test and doc live. No one should mind installing a distribution and its dependencies in order to run the unit tests or generate the documentation. Setup-Requires-Dist (multiple use) :::::::::::::::::::::::::::::::::: Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time.
On Mon, Jul 9, 2012 at 10:28 PM, Daniel Holth
New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068
Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist:
To review, Setup-Requires-Dist (only installed for setup.py to run, called build requirements in some systems) is a completely different feature than Provides-Extra (for example when code uses 'try: import somepackage; except: don't offer a feature; called optional_features in CPAN; similar to Debian 'Suggests' dependencies).
On Mon, Jul 9, 2012 at 10:28 PM, Daniel Holth
New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068
Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist: (full text below)
Setup-Requires-Dist lives because it makes sense to list packages only needed for setup.py to run before install is possible.
The reserved extra names test and doc live. No one should mind installing a distribution and its dependencies in order to run the unit tests or generate the documentation.
Setup-Requires-Dist (multiple use) ::::::::::::::::::::::::::::::::::
Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to generate a manifest from version control.
Examples::
Setup-Requires-Dist: custom_setup_command
Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time.
+1 To this. I agree Setup-Requires and Provides-Extra are different use cases and should both be added to the spec. I think that filling in these two missing features will benefit packaging in the long run. If this idea gains some amount of acceptance I'm happy to work on an implementation of the setup-requires end of things for packaging. Erik
+1 To this. I agree Setup-Requires and Provides-Extra are different use cases and should both be added to the spec. I think that filling in these two missing features will benefit packaging in the long run. If this idea gains some amount of acceptance I'm happy to work on an implementation of the setup-requires end of things for packaging.
One last addition, explicitly allow X- headers. IMO the technical purpose of each tag is solid. I would like to get this implemented by the end of July so that it can be used. After that I would like to open PEP 346 "Metadata 1.3". Metadata 1.3 could add: Parenthesis in environment markers (make official "a subset of Python without most of the good stuff?") The ~> operator for version specs "~> 1.2" is sugar for ">= 1.2, < 2.0". (Not allowed in environment markers). Thanks dstufft. Metadata-Is-Static: true / false (name to be determined; means "does Requires-Dist reliably use environment markers, or does it depend on Python code in setup.py)
On Thu, Jul 12, 2012 at 12:03 PM, Daniel Holth
+1 To this. I agree Setup-Requires and Provides-Extra are different use cases and should both be added to the spec. I think that filling in these two missing features will benefit packaging in the long run. If this idea gains some amount of acceptance I'm happy to work on an implementation of the setup-requires end of things for packaging.
One last addition, explicitly allow X- headers. IMO the technical purpose of each tag is solid. I would like to get this implemented by the end of July so that it can be used.
If there are no further comments, I will ask Benjamin Peterson to merge this into python-peps this weekend. To avoid NameError, distutils2 should define extra == None by default until it supports extras (simply not installing any extra dependencies), or markerlib should just be folded in. For markerlib's Python 2.4 problem a fallback that doesn't require _ast is either the current distutils2 marker implementation, or to evaluate empty markers as True, and non-empty markers as False, so only installing unconditional dependencies; I don't suppose many Python 2.4 users are clamoring to adopt .dist-info & Metadata 1.2... Thanks, Daniel Holth
participants (6)
-
Brian Sutherland -
Daniel Holth -
Erik Bray -
PJ Eby -
Tres Seaver -
Éric Araujo