proposing Python package index upload API spec (potential PEP)
As a new Twine maintainer I've been running into questions like:
* Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).
I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.
I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.
Per Nick Coghlan's formulation, my specific goal here would be close to:
Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.
After a little conversation here on distutils-sig, I believe my steps would be:
1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance
Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 .
I like this practice of writing specifications to better document interfaces that already exist. However, in this case I wonder if it would be better to spend the time defining a new, simpler API instead? I think we're currently in a position where most tools could use a new upload API relatively quickly once it was defined.
On Tue, May 8, 2018, at 7:09 PM, Sumana Harihareswara wrote:
As a new Twine maintainer I've been running into questions like:
- Now that Warehouse doesn't use "register" anymore, can we deprecate it
from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311
- It would be nice if Twine could depend on a package index providing an
HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).
I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.
I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.
Per Nick Coghlan's formulation, my specific goal here would be close to:
Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.
After a little conversation here on distutils-sig, I believe my steps would be:
- start a very early PEP draft with lots of To Be Determined blanks,
submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance
Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/W...
On 9 May 2018 at 04:09, Sumana Harihareswara sh@changeset.nyc wrote:
As a new Twine maintainer I've been running into questions like:
- Now that Warehouse doesn't use "register" anymore, can we deprecate it
from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311
- It would be nice if Twine could depend on a package index providing an
HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).
I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/ guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.
I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.
Per Nick Coghlan's formulation, my specific goal here would be close to:
Documenting what the current upload API between twine & warehouse
actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.
That's effectively what PEP 517 did for the legacy setup.py-centric sdist format, with just a single paragraph referencing the previous de facto standard and giving that version a name: https://www.python.org/dev/peps/pep-0517/#source-trees
The equivalent here would be to call the existing upload interface the "v1 upload API", and cite the relevant Warehouse endpoint URL as the reference implementation for it.
While I initially wasn't a fan of that idea, the effectiveness of the approach in PEP 517 now makes me agree it could work well here, too.
Cheers, Nick.
I move to define PyPA and Warehouse HTTP APIs with OpenAPI definitions.
https://swagger.io/specification/ https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0....
OPENAPI SPECIFICATION
The OpenAPI specification (formerly known as the Swagger Specification)
is a powerful definition format to describe RESTful APIs. The specification creates a RESTful interface for easily developing and consuming an API by effectively mapping all the resources and operations associated with it. It’s easy-to-learn, language agnostic, and both human and machine readable.
Introduction
The OpenAPI Specification (OAS) defines a standard, language-agnostic
interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.
An OpenAPI definition can then be used by documentation generation tools
to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.
- It's possible to generate WAF (Web Application Firewall) rules from a Swagger/OpenAPI definition. - There's an online editor. - There are many integrations for various languages and frameworks.
An OpenAPI spec/schema can be written by hand as YAML or JSON, autogenerated from e.g. decorators or docstrings or framework integration.
https://swagger.io/docs/specification/basic-structure/ has a YAML example.
Here's the online editor with validation: https://swagger.io/swagger-editor/
https://swagger.io/open-source-integrations/ #python
- pyramid-swagger https://github.com/striglia/pyramid_swagger
Convenient tools for using Swagger to define and validate your interfaces
in a Pyramid webapp.
Request/response validation do have overhead and may be best done in a WAF, but there's no reason not to serve e.g. /swagger.json or /swagger.yml with the API spec for this app.
OTOH, DevPi and Pulp Project support the v1 upload API.
An initial OpenAPI schema definition document for just the v1 upload API e.g. path, request, and response shouldn't take much time at all.
On Wednesday, May 9, 2018, Nick Coghlan ncoghlan@gmail.com wrote:
On 9 May 2018 at 04:09, Sumana Harihareswara sh@changeset.nyc wrote:
As a new Twine maintainer I've been running into questions like:
- Now that Warehouse doesn't use "register" anymore, can we deprecate it
from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311
- It would be nice if Twine could depend on a package index providing an
HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).
I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.
I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.
Per Nick Coghlan's formulation, my specific goal here would be close to:
Documenting what the current upload API between twine & warehouse
actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.
That's effectively what PEP 517 did for the legacy setup.py-centric sdist format, with just a single paragraph referencing the previous de facto standard and giving that version a name: https://www.python.org/dev/ peps/pep-0517/#source-trees
The equivalent here would be to call the existing upload interface the "v1 upload API", and cite the relevant Warehouse endpoint URL as the reference implementation for it.
While I initially wasn't a fan of that idea, the effectiveness of the approach in PEP 517 now makes me agree it could work well here, too.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I don't have time to work on this and hope someone else can pick it up - https://github.com/pypa/packaging-problems/issues/128 is a good issue to use to keep track.
participants (4)
-
Nick Coghlan -
Sumana Harihareswara -
Thomas Kluyver -
Wes Turner