PEP 426 updated based on last round of discussion
I actually pushed this to python.org on the weekend but forgot to announce it on the list. The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ It was a net deletion of content despite going into more depth in a few areas, so I'm counting that as a win for clarity :) Change details are at http://hg.python.org/peps/rev/067d3c3c1351 Significant changes: * serialisation prefix changed to "pydist". This metadata is the metadata that exists at the sdist level. Wheels and installation may add other metadata files, and PyPI will publish additional metadata extracted from the archive contents, so I decided "pymeta" didn't feel write (the name of the schema file changed as well, but I did that in a separate commit so the diff stayed readable) * all the *_may_require fields are gone (as previously discussed) * rather than "install specifiers" I went with "requirement specifiers" (install turned out not to read very well) * accordingly, the subfields of dependency specifiers are "requires", "extra" and "environment" * the abbreviated form (which has "requires" as a list) was easy enough to specify, so that's what I used. The unpacked form (where multiple entries in the same dependency list have the same extra/environment combination) is explicitly disallowed in order to encourage consistent formatting. * clarified that internal whitespace is permitted in requirement specifiers (there may be a simpler way to specify this, such as "all whitespace in requirement specifiers is ignored") * I made the change to explicitly distrust "Provides" data retrieved from a public index server, and noted that projects defining a virtual dependency should claim that name to define the default provider. * noted that Debian packagers may want to map extras to Recommended or Suggested packages * noted some possible use cases for metadata extensions * fixed and clarified various things in the reference copy of the JSON schema (it could still do with an audit against the current PEP text, though) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 16, 2013, at 9:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I actually pushed this to python.org on the weekend but forgot to announce it on the list.
The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/
It was a net deletion of content despite going into more depth in a few areas, so I'm counting that as a win for clarity :)
Change details are at http://hg.python.org/peps/rev/067d3c3c1351
Significant changes:
* serialisation prefix changed to "pydist". This metadata is the metadata that exists at the sdist level. Wheels and installation may add other metadata files, and PyPI will publish additional metadata extracted from the archive contents, so I decided "pymeta" didn't feel write (the name of the schema file changed as well, but I did that in a separate commit so the diff stayed readable)
* all the *_may_require fields are gone (as previously discussed)
* rather than "install specifiers" I went with "requirement specifiers" (install turned out not to read very well)
* accordingly, the subfields of dependency specifiers are "requires", "extra" and "environment"
* the abbreviated form (which has "requires" as a list) was easy enough to specify, so that's what I used. The unpacked form (where multiple entries in the same dependency list have the same extra/environment combination) is explicitly disallowed in order to encourage consistent formatting.
So to be clear, this means it's { "requires": [ "foo", "bar" ] } ? And it means that having multiple combinations of the same extra/envs is disallowed so I'm going to have to collapse everything back down since it's not stored that way at all?
* clarified that internal whitespace is permitted in requirement specifiers (there may be a simpler way to specify this, such as "all whitespace in requirement specifiers is ignored")
* I made the change to explicitly distrust "Provides" data retrieved from a public index server, and noted that projects defining a virtual dependency should claim that name to define the default provider.
* noted that Debian packagers may want to map extras to Recommended or Suggested packages
* noted some possible use cases for metadata extensions
* fixed and clarified various things in the reference copy of the JSON schema (it could still do with an audit against the current PEP text, though)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
So to be clear, this means it's
{ "requires": [ "foo", "bar" ] }
?
And it means that having multiple combinations of the same extra/envs is disallowed so I'm going to have to collapse everything back down since it's not stored that way at all?
I posted a working example [1] showing how there's no need to have the same structure at the RDBMS layer and the JSON layer. I asked for more information about modelling difficulties you said you had encountered, but didn't hear anything more about it. AFAICT the code you were talking about isn't public - at least, I couldn't see it in the branches on your GitHub repo. As my example shows, it's possible to have a sensible RDBMS structure which interoperates with multiple entries in "requires". If I've misunderstood something, please let me know what it is. Regards, Vinay Sajip [1] https://gist.github.com/vsajip/5929707
On Jul 16, 2013, at 1:54 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
So to be clear, this means it's
{ "requires": [ "foo", "bar" ] }
?
And it means that having multiple combinations of the same extra/envs is disallowed so I'm going to have to collapse everything back down since it's not stored that way at all?
I posted a working example [1] showing how there's no need to have the same structure at the RDBMS layer and the JSON layer. I asked for more information about modelling difficulties you said you had encountered, but didn't hear anything more about it. AFAICT the code you were talking about isn't public - at least, I couldn't see it in the branches on your GitHub repo.
As my example shows, it's possible to have a sensible RDBMS structure which interoperates with multiple entries in "requires". If I've misunderstood something, please let me know what it is.
Regards,
Vinay Sajip
The dependency models are located at https://github.com/dstufft/warehouse/blob/f438bdcb17a5ee9de8e209d3eb6c93cc4a... It's completely possible and if I came across as saying it wasn't then I failed to clarify myself properly. My point was that it was simpler using a single list of dictionaries, not a list of dictionaries itself containing lists because there was less support code required to transform between them. Every additional piece of code comes with an overhead in the form of tests, mental overhead, potential bugs etc. I was trying to advocate for less required code because it makes things simpler :) I was asking for clarification here because my original plan if things were required to be a list was to make single entry lists, again to limit the need to include additional support code. It appears that this plan isn't inline with the current iteration of the PEP but I was making sure :) I have a preference for not introducing more nesting, and making things match the modeling better but I'll make it work either way. I hardly think PEP426 will fail if it's using deeper nesting even if I dislike it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 17 Jul 2013 05:34, "Donald Stufft" <donald@stufft.io> wrote:
On Jul 16, 2013, at 1:54 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
So to be clear, this means it's
{ "requires": [ "foo", "bar" ] }
?
And it means that having multiple combinations of the same extra/envs is disallowed so I'm going to have to collapse everything back down since it's not stored that way at all?
I posted a working example [1] showing how there's no need to have the
structure at the RDBMS layer and the JSON layer. I asked for more information about modelling difficulties you said you had encountered, but didn't hear anything more about it. AFAICT the code you were talking about isn't public - at least, I couldn't see it in the branches on your GitHub repo.
As my example shows, it's possible to have a sensible RDBMS structure which interoperates with multiple entries in "requires". If I've misunderstood something, please let me know what it is.
Regards,
Vinay Sajip
The dependency models are located at https://github.com/dstufft/warehouse/blob/f438bdcb17a5ee9de8e209d3eb6c93cc4a...
It's completely possible and if I came across as saying it wasn't then I failed to clarify myself properly. My point was that it was simpler using a single list of dictionaries, not a list of dictionaries itself containing
same lists because there was less support code required to transform between them. Every additional piece of code comes with an overhead in the form of tests, mental overhead, potential bugs etc. I was trying to advocate for less required code because it makes things simpler :)
I was asking for clarification here because my original plan if things
were required to be a list was to make single entry lists, again to limit the need to include additional support code. It appears that this plan isn't inline with the current iteration of the PEP but I was making sure :)
I have a preference for not introducing more nesting, and making things
match the modeling better but I'll make it work either way. I hardly think PEP426 will fail if it's using deeper nesting even if I dislike it. Yes, in this case I think improving the brevity of the serialisation format will be an aid to debuggability, even though the primary purpose of the format remains communicating between tools. I should add a section discussing this decision in "Rejected Design Ideas", though. Cheers, Nick.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On 16 July 2013 14:40, Nick Coghlan <ncoghlan@gmail.com> wrote:
The latest version of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/
Just looking at the "Build requires" section I found myself wondering: is there any way to say that e.g. a C compiler is required for building, or a Fortran compiler or any other piece of software that isn't a "Python distribution"? The example shows Cython which is commonly built and used with MinGW on Windows. I guess that it would be possible to create a pypi distribution that would install MinGW and set it up as part of a Python installation so that a project such as Cython could depend on it with e.g.: "name": "Cython", "build_requires": [ { "requires": ["pymingw"], "environment": "sys.platform == 'win32'" } ] "run_requires": [ { "requires": ["pymingw"], "environment": "sys.platform == 'win32'" } ] But it would be unfortunate to depend on MinGW in the event that the user actually has the appropriate MSVC version. Or perhaps there could be a meta-distribution called "CCompiler" that installs MinGW only if the the appropriate MSVC version is not available. Or could there be an environment marker to indicate the presence of particularly common requirements such as having a C compiler? Oscar
On 17 July 2013 12:01, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 16 July 2013 14:40, Nick Coghlan <ncoghlan@gmail.com> wrote:
The latest version of PEP 426 is up at
http://www.python.org/dev/peps/pep-0426/
Just looking at the "Build requires" section I found myself wondering: is there any way to say that e.g. a C compiler is required for building, or a Fortran compiler or any other piece of software that isn't a "Python distribution"?
[...]
Or perhaps there could be a meta-distribution called "CCompiler" that installs MinGW only if the the appropriate MSVC version is not available. Or could there be an environment marker to indicate the presence of particularly common requirements such as having a C compiler?
I can't imagine it's practical to auto-install a C compiler - or even to check for one before building. But I can see it being useful for introspection purposes to know about this type of requirement. (A C compiler could be necessary, or optional for speedups, a particular external library could be needed, etc) The data would likely only be as good as what project developers provide, but nevertheless having standard places to record the data could encourage doing so... OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need to get what we have now out of the door rather than continually adding extra capabilities. Paul
On 17 July 2013 12:10, Paul Moore <p.f.moore@gmail.com> wrote:
I can't imagine it's practical to auto-install a C compiler
Why not?
- or even to check for one before building.
But I can see it being useful for introspection purposes to know about this type of requirement. (A C compiler could be necessary, or optional for speedups, a particular external library could be needed, etc)
Perhaps instead the installer tool would give you a way to clarify that you do have a C compiler and to warn if not. Alternatively a meta-package could be used to indicate (when installed) that a compatible C-compiler is available and then other distributions could depend on it for building.
The data would likely only be as good as what project developers provide, but nevertheless having standard places to record the data could encourage doing so...
OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need to get what we have now out of the door rather than continually adding extra capabilities.
I wasn't proposing to hold anything up or add new capabilities. I'm just trying to see how far these changes go towards making non-pure Python software automatically installable. Everything I would want to build "build requires" software that is not on pypi. It would be great if e.g. the instructions for installing Cython on Windows could just be "pip install cython" instead of this: http://wiki.cython.org/InstallingOnWindows Oscar
On 17 July 2013 21:45, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 17 July 2013 12:10, Paul Moore <p.f.moore@gmail.com> wrote:
I can't imagine it's practical to auto-install a C compiler
Why not?
- or even to check for one before building.
But I can see it being useful for introspection purposes to know about this type of requirement. (A C compiler could be necessary, or optional for speedups, a particular external library could be needed, etc)
Perhaps instead the installer tool would give you a way to clarify that you do have a C compiler and to warn if not.
Alternatively a meta-package could be used to indicate (when installed) that a compatible C-compiler is available and then other distributions could depend on it for building.
The data would likely only be as good as what project developers provide, but nevertheless having standard places to record the data could encourage doing so...
OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need to get what we have now out of the door rather than continually adding extra capabilities.
I wasn't proposing to hold anything up or add new capabilities. I'm just trying to see how far these changes go towards making non-pure Python software automatically installable. Everything I would want to build "build requires" software that is not on pypi.
It would be great if e.g. the instructions for installing Cython on Windows could just be "pip install cython" instead of this: http://wiki.cython.org/InstallingOnWindows
External dependencies are platform specific, so the task of defining them has been deliberately left to metadata extensions. Depending on a Windows exe or MSI, an OS X app bundle, a Fedora/RHEL/EPEL/Suse RPM or a Debian/Ubuntu DEB are all very different things not easily accommodated in a cross platform standard. That said, the new metadata standard does deliberately include a few pieces intended to make such things easier to define: 1. The extensions concept - using a structured data format like JSON makes it much easier for platform specific tools (or even pip itself) to say "declare this metadata, and we will run these commands automatically" 2. The "direct reference" format as a tool for linking a name with a specific URL 3. The "install hooks" system for running arbitrary additional commands post-install (if the installation tool is instructed to allow that) 4. A more robust mechanism for defining supported platforms 5. The extras system extended across all the different kinds of dependency (so if you don't want to build optional C accelerators, you can express that clearly and skip all the associated dependencies) Essentially, the standard tries to provide a better toolkit for solving these kinds of problems, without trying to solve them directly. It's still going to be up to other projects like zc.buildout and conda to provide a way to package that kind of thing up at least somewhat neatly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17 July 2013 13:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
That said, the new metadata standard does deliberately include a few pieces intended to make such things easier to define:
1. The extensions concept - using a structured data format like JSON makes it much easier for platform specific tools (or even pip itself) to say "declare this metadata, and we will run these commands automatically"
Okay, so this is where you can put the "I need a [specific] C-compiler" information. Then a pip alternative (or a future pip) that knew more about C compilation could respond appropriately. The PEP doesn't explicitly say anything about how a tool should handle unrecognised metadata extensions; it seems fairly obvious to me that they are supposed to be ignored but perhaps this should be explicitly stated. On the other hand it would be useful to be able to say: if you don't understand my "fortran" metadata extension then you don't know how to install/build this distribution. Is there a way e.g. to indicate a build/install dependency on the tool understanding some section in the extension metadata, or that an extension is compulsory somehow? Then a user could do: $ pip install autocont Error installing "autocont": required extension "fortran" not understood. See http://pypa.org/list_of_known_extensions.htm for more information. Oscar
On 17 July 2013 22:40, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 17 July 2013 13:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
That said, the new metadata standard does deliberately include a few pieces intended to make such things easier to define:
1. The extensions concept - using a structured data format like JSON makes it much easier for platform specific tools (or even pip itself) to say "declare this metadata, and we will run these commands automatically"
Okay, so this is where you can put the "I need a [specific] C-compiler" information. Then a pip alternative (or a future pip) that knew more about C compilation could respond appropriately.
The PEP doesn't explicitly say anything about how a tool should handle unrecognised metadata extensions; it seems fairly obvious to me that they are supposed to be ignored but perhaps this should be explicitly stated.
On the other hand it would be useful to be able to say: if you don't understand my "fortran" metadata extension then you don't know how to install/build this distribution. Is there a way e.g. to indicate a build/install dependency on the tool understanding some section in the extension metadata, or that an extension is compulsory somehow?
Yes, you can include a build_requires on something that understands the extension and make setup.py a shim that invokes that build system rather than setuptools/distutils (similar to the way d2to1 allows setup.cfg based projects work with tools that are expecting a setup.py file) It's not as elegant as a proper metabuild system (since there's a fair bit of legacy cruft in the setup.py CLI), but it works :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17 July 2013 12:45, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
OTOH, maybe this is metadata 3.0 stuff - I feel like at the moment we need to get what we have now out of the door rather than continually adding extra capabilities.
I wasn't proposing to hold anything up or add new capabilities. I'm just trying to see how far these changes go towards making non-pure Python software automatically installable. Everything I would want to build "build requires" software that is not on pypi.
Sorry - my comment was more directed at myself suggesting extra capabilities in the metadata, than at you for asking the question. The more questions we ask about "how would this work in practice, for such-and-such use case" the better... Paul
participants (5)
-
Donald Stufft -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Vinay Sajip