[Python-Dev] Yearly PyPI breakage

Thu May 5 19:07:52 EDT 2016

> On May 5, 2016, at 6:22 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> I know you're not happy with myself and the other distutils-sig folks
>> regarding the decision to deprecate and remove automatic link
>> spidering,
> 
> More accurately: Not happy with the removal of the checksummed "explicit"
> mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD
> has been used in huge and secure installations, so the I don't buy the
> reliability and security arguments that are used in favor of centralization.
> But centralization seems to be a goal in and of itself these days (and
> that isn't limited to Python).

Let me say outright that there is nothing wrong with a well designed, ports
like system. However, PyPI was not that and the vast bulk of packages were not
using it in that fashion. At the time of PEP 470 being written a mere 59 out of
65,232 (at the time) projects were utilizing the feature that allowed people to
safely do a ports like system. That's roughly 0.09% of all of the projects on
PyPI which meant those projects were in a very serious minority. Thus it is
only natural that the expectations end users had when installing software was
that it was coming from PyPI, not from some external host and they made
decisions based around those expectations and when those expectations were
violated they came to the people operating PyPI and developing pip and were
at best very confused, and at worst irate at *us*.

It's also true that there are benefits and cons to both a repository centric
approach and a ports style approach, however since we were attempting to do
both in one repository it typically meant that we got all of the cons of both
systems, but we couldn't ever take advantage of any of the unique benefits of
either system since it would only apply to some fraction of the projects. This
meant that improvements that would help people installing 99.11% of projects
would end up getting blocked by 0.09% of projects.

At the end of the day, attempting to both things in one repository just wasn't
working out very well in the general case, so we needed (IMO) to pick one way
and stop supporting the other way. I personally think it's a pretty obvious
choice to go the way that we did, given that the overwhelming majority of
projects were already being used that way.

That being said, I don't think the characterization of "centralizing" is really
accurate here. People are still free to host their projects wherever they want
and pip (and easy_install) can be used to install them. All it requires is
telling pip where it needs to look in addition to the default location (PyPI),
which can be configured via the CLI, environment variables, or a number of
configuration files at the system, user, and virtual environment level. You're
free to continue providing a page on PyPI to enable discovery of your project
and you can include in that page instructions on how to install your project.

> 
> 
>> nor with the PSF regarding the current Terms of Service
>> around uploads to PyPI, but that doesn't make it OK to start off-topic
>> threads on python-dev just because you're a CPython core developer in
>> addition to being a PyPI user.
> 
> Alexander thought otherwise:
> 
>  https://mail.python.org/pipermail/python-dev/2015-October/141840.html
> 
> "Given that ensurepip is part of stdlib, I am not sure this is an accurate
> statement.  Even if it was, did you make any effort to discuss the proposal
> outside of a small group subscribed to distutils ML?"
> 
> I completely agree with that.

We've sort of hijacked the PEP process to do something that it wasn't really
meant to do. Changes like PEP 470 have nothing to do with what Python itself
does and thus it's really outside of the realm of where python-dev really needs
a say or notification. In Alexander's example of PEP 495 it was talking about
adding a new API to Python itself, thus it makes sense that python-dev should
be told about that PEP, and in cases where we needed to change the standard
library we've done that discussion on python-dev (such as with PEP 453).

The python-dev list is not the list to notify people of changes to things in
all corners of the Python ecosystem. If people want to get involved or keep
abreast of the changes in the external projects of PyPI, pip, setuptools, etc
then they should subscribe to the places that those changes get discussed--
which is distutils-sig. Packaging already has enough problems with getting
bogged down in "why wasn't I consulted" and bikeshedding, I don't think that
inviting folks (and nothing against python-dev here) a chance to try and
re-legislate an agreed upon change after the fact is going to provide much in
the way of benefits and will instead just contribute further to stymying
efforts to move packaging forward. If it was decided that we really need all
PEPs announced on python-dev and python-dev given a chance to weigh in, I would
advocate for packaging to stop using the PEP process all together and create
our own process to mirror the PEP process because again, we're not using the
PEP process because these things naturally fall under what the PEP process was
designed to cover, we're using it because it was there already and close enough
that we could hijack it and it was moderately easier to just do that. If the
downsides of using the PEP process become greater than the effort to come up
with our own process then I think it's a no brainer to just do our own thing.

> 
> 
> Fredrik Lundh is also affected (and might not have received any mail,
> same as me):
> 
>  https://pypi.python.org/pypi/PIL

I've personally emailed the PIL offer several times through various email
addresses over several years and have never gotten a response. At this point I
can only assume either they are willfully ignoring those messages or they've
completely changed their contact information in a way that I have not been able
to find updated contacted information for at all.

> Technically, he
> didn't gloat, but suddenly legal advice was apparently readily available.
> 
>  https://pypi.python.org/pypi/m3-cdecimal
> 

I'm not going to respond to the rest of this, because I don't think it's
possible for you and me to have a constructive discussion on these other
issues. However, I do want to point out that the PSF is the legal entity behind
PyPI itself and Van is both a board member and a lawyer. I don't recall what
exactly was discussed with who about m3-cdecimal, but it's completely
reasonable that when a legal question comes up with PyPI that we consult the
PSF board members, particularly the ones who are lawyers. I do recall that the
last time m3-cdecimal came up [1] you (again) brought up issues you had with
PyPI in an inappropriate venue and as far as I know, you never actually used
any of the contact channels that you were offered to try and resolve the
issue that you had with a project on PyPI.

[1] https://bugs.python.org/issue22483

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/881c096a/attachment.sig>