[Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Nick Coghlan]

On 24 August 2016 at 02:46, Donald Stufft <donald at stufft.io> wrote:
> Abstract
> ========
>
> This PEP recommends deprecating, and ultimately removing, support for uploading
> certain unused or under used file types and extensions to PyPI. In particular
> it recommends disallowing further uploads of any files of the types
> ``bdist_dumb``, ``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and
> ``bdist_wininst``, leaving PyPI to only accept new uploads of the ``sdist``,
> ``bdist_wheel``, and ``bdist_egg`` file types.

+1 on this part. To address MAL's concerns, you may want to state
explicitly that this proposal is based around a notion of Python
package publication as a multi-level distribution network where:

* PyPI itself only supports publication in venv-compatible
Python-specific formats (sdist, wheel, egg)
* PyPA (the development community) actively encourages
community-driven and commercial conversions to downstream formats for
particular target environments:

- the conda cross-platform data analysis ecosystem (conda-forge)
- the deb based Linux ecosystem (Debian, Ubuntu, etc)
- the RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
- the homebrew, MacPorts and fink ecosystems for Mac OS X
- the Windows Package Management ecosystem (NuGet, Chocalatey, etc)
- 3rd party creation of Windows MSIs and installers (e.g. Christoph
Gohlke's work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
- other commercial redistribution formats (ActiveState's PyPM,
Enthought Canopy, etc)
- other open source community redistribution formats (Nix, Gentoo,
Arch, *BSD, etc)

This division of responsibility means that we (PyPA) don't need to be
experts in every platform management system out there, and neither do
folks wanting to publish software through PyPI. However, we also
remain open to enhancement proposals for the standard upstream formats
that allow for easier automation of the conversions to downstream
formats.

> In addition, this PEP proposes removing support for new uploads of sdists using
> the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``,
> ``.tbz``, and any other extension besides ``.tar.gz``.

I think in this case, campaigning to remove ".zip" support will prove
to be a false economy, as anything you'd gain from simplifying the
toolchain implementation you'll lose arguing with folks that would
prefer for .zip to be the default instead of .tar.gz.

So while I agree with your argument that settling on a single sdist
archive format would be better, I think the political cost of pushing
for that will be too high (and overshadow the other benefits of this
proposal).

By contrast, allowing both .tar.gz and .zip still achieves the goal of
minimising external binary dependencies (since they both just need
zlib), and we can also still keep the change where setuptools and
distutils default to generating .tar.gz regardless of platform (so we
optimise for size reduction by default, but folks that really prefer
to publish their sdists as zip archives will remain free to do so).

I also see a concrete *practical* benefit to pursuing this option
(mainly prompted by Robert Collins's observations in the other
thread): when we get around to defining sdist 2.0, we're likely to
make that a zip based format for consistency with wheel, so we don't
really want people to be actively dropping zip support from their
toolchains. If we continue to allow both .tar.gz and .zip, we
essentially get that for free, and then at some point in the future,
we can push for convergence on a new ".sdist" format for distributing
annotated source archives (which could even explore ideas like a
tarball-in-zip approach that offers random access to the metadata
files, but still focuses on compression for the source archive
itself).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia