[Distutils] Deprecating little used file types/extensions on PyPI?

Tue Aug 16 11:15:52 EDT 2016

Thanks Daniel,

A few corrections and considerations below:

On 16 August 2016 at 09:50, Daniel Holth <dholth at gmail.com> wrote:

> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>

Specifically, buildout right now has setuptools as its only dependence, and
buildout uses it to locate, download and build sdist dependencies, or
directly download and use .egg dependencies, which are important for
Windows platforms in the communities that use buildout and depend on
compiled extensions.

It also uses setuptools for reading entry points of packages for internal
use (buildout recipes and extensions), and for figuring out console scripts
to install in "bin"

(this last part is actually the job of the `zc.recipe.egg` recipe, which is
buildout's pip replacement that predates pip by a few years, but it's the
only recipe that is developed in the same repo as buildout itself, so we'll
treat them as one for this discussion).

This also means that so far buildout has been missing out on wheels. It
hasn't been much of a problem since almost all packages also have an sdist
that can be turned into an egg by setuptools, but if the trend of
sdist-less wheels increases, this could quickly become a problem.

It also means that, buildout is missing out on manylinux...

A wheel2egg might be nice, but unless it's integrated into setuptools
proper, it would mean adding another dependency to buildout and the
developers would rather depend on a single library for talking to PyPI and
making packages appear as "directories addable to `sys.path`.

The main reason wheel did not support these use cases from the beginning is
> that it was quicker not to. Also, it is a bit confusing for 'unzipped
> directory in the (egg/wheel) format', 'archive', and 'plugin you add to
> PYTHONPATH' to all have the same name 'eggs'.
>
> We would need to work with the buildout developers on this. Buildout uses
> a flat directory full of unzipped eggs
>

Not necessarily unzipped. Although buildout unzips all sdists and eggs it
installs, if it's in the "eggs" directory, has the right name (e.g.
suds_jurko-0.6-py2.7.egg), and can be added to sys.path, that's all
buildout cares about, but if it's not yet there, buildout will use
setuptools to install it as an "unzipped egg" (there used to be a switch to
install eggs as zipped, or rather, as unzipped, zipped being the default,
but nobody liked the eggs zipped so they threw that foot gun away).

> , not exactly the uploading to pypi case, and when installing
> console_scripts, makes sure the necessary eggs are added to sys.path in the
> script itself. The exact mechanism varies depending on the version of
> buildout used.
>

This hasn't changed for a while. The mechanism is, basically, to create a
script in "bin/" that appends all (recursive) dependencies of the script
from the "eggs" directory into `sys.path` then call the entry point
declared in the `console_scripts`.

The buildout configuration can specify both the name that the console
scripts will be installed, and also specify some code to be run immediately
before the entry point,  which is useful for pre-setting command line
arguments, for example. All of this is done using setuptools APIs.

This and the fact that buildout has a plethora of extensions and powerful
declarative configuration language that allows for dependency management
between different "parts" of the configuration (e.g. coordinating ports and
addresses between services like Varnish and Plone, or adding the paths of
the script and configuation file from Plone into a supervisord
configuration) means that, like many things coming out of the brilliant
mind of Jim Fulton, it's considered overly complex by many, but those who
use it would find it rather difficult to replace if it's no longer
functional.

zope.interface is an important package with up-to-date eggs.
> https://pypi.python.org/pypi/zope.interface
>

And it's used by Plone and Pyramid, to mention two mainstream communities
that would be affected by a lack of binary eggs on Windows, for example.

To sumarize, if we could convince setuptools to treat wheels on PyPI for
download and installation the same way it treats eggs, or if we could
replace setuptools with something that did, then buildout would be onboard
the deprecation of eggs.

Cheers,

PS: In the buildout community, we never really understood the impetus for
replacing egg as a format, which is really not all that complex and not all
the different from wheel as it stands, instead of just formalizing it in a
PEP. We got the feeling that, in the movement of getting rid of setuptools
as an "installation" dependency, we ended up throwing out the baby with the
bathwater...

On Tue, Aug 16, 2016 at 5:41 AM Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 15 August 2016 at 20:09, Donald Stufft <donald at stufft.io> wrote:
>> >
>> > First off, we currently allow people to upload sdist, bdist_wheel,
>> bdist_egg,
>> > bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However
>> I think
>> > that we should try to get rid of support for most of these. Just for
>> reference
>> > currently the number of files uploaded for each type of file looks like:
>> >
>> > * sdist: 506,585
>> > * bdist_wheel: 81,207
>> > * bdist_egg: 48,282
>> > * bdist_wininst: 14,002
>> > * bdist_dumb: 5,502
>> > * bdist_msi: 497
>> > * bdist_rpm: 464
>> > * bdist_dmg: 45
>> >
>> > Out of all of these, I think that we can easily remove bdist_dmg,
>> bdist_rpm,
>> > and bdist_dumb. I also believe that there is a strong case for removing
>> > bdist_msi and bdist_wininst. I also think we should consider removing
>> > bdist_egg.
>>
>> Overall +1 from me for this.
>>
>> Some thoughts:
>>
>> 1. It would be nice to provide some transition process for the
>> bdist_wininst case, as these are automatically convertible (with some
>> caveats) to wheel using "wheel convert". I'm not suggesting that we do
>> an auto-convert, but a way of getting the message out to projects that
>> use wininst that there is a migration path might be worth it (although
>> it's equally possible that the projects in question are all
>> unmaintained, in which case there's not much point in bothering).
>> 2. Do we understand the remaining use cases for eggs? AIUI, buildout
>> uses it, and I get the impression that there are other specialist
>> groups that still use the egg format (the plone community?). While I
>> don't think having two binary distribution formats is good for the
>> ecosystem (it's confusing for users, and splits effort), I think we
>> should be make sure we have those use cases covered (or at least have
>> a plan on how we handle the situation) before deprecating the egg
>> format.
>>
>> Paul
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/acfdb66d/attachment-0001.html>