[Distutils] Re: proposing Python package index upload API spec (potential PEP)

I move to define PyPA and Warehouse HTTP APIs with OpenAPI definitions. https://swagger.io/specification/ https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0....

OPENAPI SPECIFICATION

The OpenAPI specification (formerly known as the Swagger Specification) is a powerful definition format to describe RESTful APIs. The specification creates a RESTful interface for easily developing and consuming an API by effectively mapping all the resources and operations associated with it. It’s easy-to-learn, language agnostic, and both human and machine readable.

Introduction

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

- It's possible to generate WAF (Web Application Firewall) rules from a Swagger/OpenAPI definition. - There's an online editor. - There are many integrations for various languages and frameworks. An OpenAPI spec/schema can be written by hand as YAML or JSON, autogenerated from e.g. decorators or docstrings or framework integration. https://swagger.io/docs/specification/basic-structure/ has a YAML example. Here's the online editor with validation: https://swagger.io/swagger-editor/ https://swagger.io/open-source-integrations/ #python - pyramid-swagger https://github.com/striglia/pyramid_swagger

Convenient tools for using Swagger to define and validate your interfaces in a Pyramid webapp. http://pyramid-swagger.readthedocs.org/

Request/response validation do have overhead and may be best done in a WAF, but there's no reason not to serve e.g. /swagger.json or /swagger.yml with the API spec for this app. OTOH, DevPi and Pulp Project support the v1 upload API. An initial OpenAPI schema definition document for just the v1 upload API e.g. path, request, and response shouldn't take much time at all. On Wednesday, May 9, 2018, Nick Coghlan wrote:

On 9 May 2018 at 04:09, Sumana Harihareswara wrote:

...

As a new Twine maintainer I've been running into questions like:

* Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).

I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.

I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.

Per Nick Coghlan's formulation, my specific goal here would be close to:

...

Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.

Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.

That's effectively what PEP 517 did for the legacy setup.py-centric sdist format, with just a single paragraph referencing the previous de facto standard and giving that version a name: https://www.python.org/dev/ peps/pep-0517/#source-trees

The equivalent here would be to call the existing upload interface the "v1 upload API", and cite the relevant Warehouse endpoint URL as the reference implementation for it.

While I initially wasn't a fan of that idea, the effectiveness of the approach in PEP 517 now makes me agree it could work well here, too.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia