[Python-ideas] PyPI JSON Metadata Standardization for Mirrors

[Brett Cannon]

The proper list for this would be distutils-sig as that's where
packaging-related discussions typically occur.

On Wed, 9 Aug 2017 at 21:22 Cooper Ry Lees <lists at cooperlees.com> wrote:

> Hi all,
>
> First time emailer, so please be kind. Also, if this is not the right
> mailing list for PyPA talk, I apologize. Please point me in the right
> direction if so. The main reason I have emailed here is I believe it may be
> PEP time to standardize the JSON metadata that PyPI makes available, like
> what was done for the `'simple API` described in PEP503.
>
> I've been doing a bit of work on `bandersnatch` (I didn't name it), which
> is a PEP 381 mirroring package and wanted to enhance it to also mirror the
> handy JSON metadata PyPI generates and makes available @
> https://pypi.python.org/pypi/PKG_NAME/json.
>
> I've done a PR on bandersnatch as a POC that mirrors both the PyPI
> directory structure (URL/pypi/PKG_NAME/json) and created a standardizable
> URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some
> other proxy). I'm also contemplating naming the directory 'metadata' rather
> than JSON so if some new hotness / we want to change the format down the
> line we're not stuck with json as the dirname. This PR can be found here:
> https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json-metadata-to-mirror
>
> My main use case is to write a very simple async 'verifier' tool that will
> crawl all the JSON files and then ensure the packages directory on each of
> my internal mirrors (I have a mirror per region / datacenter) have all the
> files they should. I sync centrally (to save resource on the PyPI
> infrastructure) and then rsync out all the diffs to each region /
> datacenter, and under some failure scenarios I could miss a file or many.
> So I feel using JSON pulled down from the authoritative source will allow
> an async job to verify the MD5 of all the package files on each mirror.
>
> What are peoples thoughts here? Is it worth a PEP similar to PEP503 going
> forward? Can people enhance / share some thoughts on this idea.
>
> Thanks,
> Cooper Lees
> me at cooperlees.com <me at copperlees.com>
> https://cooperlees.com/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170810/ea74c0f0/attachment.html>