[Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Wes Turner]

On Tuesday, August 23, 2016, Nathaniel Smith <njs at pobox.com> wrote:

> On Aug 23, 2016 12:57 PM, "Donald Stufft" <donald at stufft.io
> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
> >
> [...]
> > However, PyPI does need
> > to do work when a file is uploaded to PyPI. For instance, it needs
> > to verify that the file being uploaded is valid, it needs to ensure
> > that it’s for the project it claims to be for, etc. To do this, PyPI
> > has to know things about the file format itself, and what it can
> > expect from it. One bug that has cropped up from time to time again
> > is people accidentally uploading a package that inside it contains
> > version say “1.0”, but when they registered it with PyPI they told
> > PyPI it was version “1.0a1” or something like that, which causes a lot
> > of the tooling to do subtly weird and broken things. PyPI should be
> > double checking the internal metadata of these files, but it can’t
> > do that unless it can expect that metadata to exist in those files
> > and it has to implement it for each file type (and then, that has to
> > be maintained).
>
> Am I understanding correctly that PyPI needs to start peeking inside
> sdists but hasn't started doing that yet? If that's correct, then I just
> want to double check that the cost of implementing this upcoming feature
> has been factored into the .zip-vs-.tar.gz discussion, because code for
> peeking inside .tar.gz files is presumably harder to write and more
> expensive to run than code for peeking inside .zip files. (But maybe only
> negligibly harder, I haven't tried writing such code myself, and uploads
> are relatively rare compared to downloads.) I guess the worst case would be
> if it turns out pypi needs to look at multiple files inside each sdist,
> where .tar.gz access becomes quadratic unless you're very clever.
>
-1 on .tar.gz as the one true sdist.
  - "You require 7zip to open >< 3.4 pkgs":
     http://www.7-zip.org/download.html

- This should probably be done with containers and tasks? (Inspecting and
parsing the compressed metadata)

- .tar.gz optionally persists +x, setuid, setgid permission bits (!?)

- zipimport and wheel already depend on ZIP

Also, regarding legacy support and future maintenance costs,
- who added this functionality? Can they be contacted to modernize it?
- it's not like we're going to be able to eliminate the code paths
entirely, so "yeah that's not maintained anymore" is worse better than
"someone should make those slow tests faster"

+1 on pushing for just .tar.gz as an extension






> -n
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/e8161024/attachment-0001.html>