[Distutils] distlib and wheel metadata

[C Anthony Risinger]

On Thu, Feb 23, 2017 at 1:56 PM, Xavier Fernandez <xav.fernandez at gmail.com>
wrote:

> +1 also.
> This whole double requirement feels over-complicated for what seems like a
> rather small usecase: it would be interesting to have a few stats on the
> number of packages concerned by this pinning (maybe just scan all the last
> uploaded wheels of each package ?).
>

FWIW, an application packaging tool a wrote several years ago used to hit
into dependency solver problems quite often. The tool leaned on distlib
(which is a pretty nice library, but strict, as noted in OP) because there
is/was no interface to pip. IIRC we upstreamed a few patches related to
this and for sure carried some local patches.

The distlib solver would bind up from impossible constraints, yet every
time, pip found a way to "power through" the exact same configuration
despite blatantly incompatible metadata at times. I never looked into it
further on pip's side (though probably someone here can confirm/deny this)
but I suspect poor metadata is more widespread than pip makes visible.

I had a dump from 2014 of the distlib data at red-dove.com and I ran a
quick script against it:

https://gist.github.com/anthonyrisinger/f9140191009fb1ec1434cb0585a4a75c

total_projects:    41228
total_projects_eq: 182
% affected:        0.44%
total_files:       285248
total_files_eq:    1276
% affected:        0.45%
total_reqs:        642447
total_reqs_bare:   460080
% affected:        71.61%

I know the distlib data (from 2014 at least) is imperfect, but this would
suggest not many projects use "==" explicitly. Maybe the bigger problem is
that 75% of requirements have no version specifier at all. I know for us
this specifically contributed to our solver problems because distlib was
eager about choosing a version for such requirements, even though a later
package might fulfill the requirement. Maybe this has since changed, but we
needed to patch it at the time [1].

We really have to figure out this distribution stuff friends. Existing
files, new metadata files, old PEPs, new PEPs... it's looking a bit "broken
windows theory" for the principal method used to share Python with the
world, and the outsized lens though which Python is perceived. Maybe this
means hard or opinionated decisions, but I really can't stress enough how
much of a drag it is to an otherwise reasonably solid Python experience.
There is a real perception that it's more trouble than it's worth,
especially with many other good options at the table.

[1]
https://github.com/anthonyrisinger/zippy/commit/1c5d34d89805c47188a18cfbe17cfc39a9cb4480#diff-a533aaf4eec84e7c5d85d2129e10514fR1168

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170223/7efa06cf/attachment.html>