proposing Python package index upload API spec (potential PEP)

As a new Twine maintainer I've been running into questions like: * Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request). I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload. I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc. Per Nick Coghlan's formulation, my specific goal here would be close to:
Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this. After a little conversation here on distutils-sig, I believe my steps would be: 1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

I like this practice of writing specifications to better document interfaces that already exist. However, in this case I wonder if it would be better to spend the time defining a new, simpler API instead? I think we're currently in a position where most tools could use a new upload API relatively quickly once it was defined. On Tue, May 8, 2018, at 7:09 PM, Sumana Harihareswara wrote:

On 9 May 2018 at 04:09, Sumana Harihareswara <sh@changeset.nyc> wrote:
That's effectively what PEP 517 did for the legacy setup.py-centric sdist format, with just a single paragraph referencing the previous de facto standard and giving that version a name: https://www.python.org/dev/peps/pep-0517/#source-trees The equivalent here would be to call the existing upload interface the "v1 upload API", and cite the relevant Warehouse endpoint URL as the reference implementation for it. While I initially wasn't a fan of that idea, the effectiveness of the approach in PEP 517 now makes me agree it could work well here, too. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I move to define PyPA and Warehouse HTTP APIs with OpenAPI definitions. https://swagger.io/specification/ https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0....
- It's possible to generate WAF (Web Application Firewall) rules from a Swagger/OpenAPI definition. - There's an online editor. - There are many integrations for various languages and frameworks. An OpenAPI spec/schema can be written by hand as YAML or JSON, autogenerated from e.g. decorators or docstrings or framework integration. https://swagger.io/docs/specification/basic-structure/ has a YAML example. Here's the online editor with validation: https://swagger.io/swagger-editor/ https://swagger.io/open-source-integrations/ #python - pyramid-swagger https://github.com/striglia/pyramid_swagger
Request/response validation do have overhead and may be best done in a WAF, but there's no reason not to serve e.g. /swagger.json or /swagger.yml with the API spec for this app. OTOH, DevPi and Pulp Project support the v1 upload API. An initial OpenAPI schema definition document for just the v1 upload API e.g. path, request, and response shouldn't take much time at all. On Wednesday, May 9, 2018, Nick Coghlan <ncoghlan@gmail.com> wrote:

I don't have time to work on this and hope someone else can pick it up - https://github.com/pypa/packaging-problems/issues/128 is a good issue to use to keep track.

I like this practice of writing specifications to better document interfaces that already exist. However, in this case I wonder if it would be better to spend the time defining a new, simpler API instead? I think we're currently in a position where most tools could use a new upload API relatively quickly once it was defined. On Tue, May 8, 2018, at 7:09 PM, Sumana Harihareswara wrote:

On 9 May 2018 at 04:09, Sumana Harihareswara <sh@changeset.nyc> wrote:
That's effectively what PEP 517 did for the legacy setup.py-centric sdist format, with just a single paragraph referencing the previous de facto standard and giving that version a name: https://www.python.org/dev/peps/pep-0517/#source-trees The equivalent here would be to call the existing upload interface the "v1 upload API", and cite the relevant Warehouse endpoint URL as the reference implementation for it. While I initially wasn't a fan of that idea, the effectiveness of the approach in PEP 517 now makes me agree it could work well here, too. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I move to define PyPA and Warehouse HTTP APIs with OpenAPI definitions. https://swagger.io/specification/ https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0....
- It's possible to generate WAF (Web Application Firewall) rules from a Swagger/OpenAPI definition. - There's an online editor. - There are many integrations for various languages and frameworks. An OpenAPI spec/schema can be written by hand as YAML or JSON, autogenerated from e.g. decorators or docstrings or framework integration. https://swagger.io/docs/specification/basic-structure/ has a YAML example. Here's the online editor with validation: https://swagger.io/swagger-editor/ https://swagger.io/open-source-integrations/ #python - pyramid-swagger https://github.com/striglia/pyramid_swagger
Request/response validation do have overhead and may be best done in a WAF, but there's no reason not to serve e.g. /swagger.json or /swagger.yml with the API spec for this app. OTOH, DevPi and Pulp Project support the v1 upload API. An initial OpenAPI schema definition document for just the v1 upload API e.g. path, request, and response shouldn't take much time at all. On Wednesday, May 9, 2018, Nick Coghlan <ncoghlan@gmail.com> wrote:

I don't have time to work on this and hope someone else can pick it up - https://github.com/pypa/packaging-problems/issues/128 is a good issue to use to keep track.
participants (4)
-
Nick Coghlan
-
Sumana Harihareswara
-
Thomas Kluyver
-
Wes Turner