On May 3, 2016, at 1:42 PM, Stefan Krah <stefan@bytereef.org> wrote:

I don't fully understand your explanation. Judging by the link that Donald posted (thanks!) it seems that PEP 470 introduced extra work for him that would not have been present had things been left in place.

IIRC the PyPI maintainers were constantly nagged about “PyPI reliability issues” that were instead external hosting issues. Everybody was affected every now and then whenever tummy.com or other external servers for popular packages were down. Or at least I know *I was*. Way too often.

But if the majority prefers PyPI that way, I'll stop arguing.

I’m not sure what you mean here but if you want to argue for reverting PEP 470, I wouldn’t hold my breath. -- Ł

On May 3, 2016, at 2:38 PM, Stefan Krah <stefan@bytereef.org> wrote:

But making them completely unreachable does not increase reliability. :)

But it does increase security. The other motivation, besides reliability, listed in this section <https://www.python.org/dev/peps/pep-0470/#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that>, is that: "transparently including external links [is] a security hazard (given that in most cases it allowed a MITM to execute arbitrary Python code on the end users machine)". And, indeed, the URL presently listed on PyPI for the cdecimal upload is an unverified http URL. This means that any evil barista with access to a coffee-shop wifi router could instantly execute user-privileged code on any Python programmer's laptop if they were to `pip install´ this externally hosted package, which is one of the reasons why neither `pip´ nor `pypi´ allow such a thing any more. Please believe me when I say I do not mean the following to be insulting - information security is incredibly confusing, difficult, and rapidly evolving, and I don't blame you for getting it wrong - but maintaining a popular package in this way is dangerously irresponsible. There are solid social reasons to centralize the control of the default package repository in the hands of dedicated experts who can scale their security expertise to a large audience, so that package authors like you and I don't need to do this in order to prevent Python from gaining a reputation as a vector for malware; this package is a case in point. Separately from the issue of how PyPI works, even if you have some reason you need to host it externally (which I seriously doubt), please take the trouble to set up a server with properly verified TLS, or use a '.github.io' hostname that can be verified that way. In the meanwhile, just to demonstrate that it's a trivial amount of work to just host it on PyPI, I checked out this package via a verified mechanism ("git clone https://github.com/bytereef/bytereef.github.io") and created a new pypi-cdecimal package <https://pypi.python.org/pypi/pypi-cdecimal <https://pypi.python.org/pypi/pypi-cdecimal>>, via editing the setup.py to change the name, 'python setup.py register', 'python setup.py sdist', 'pip wheel' (for some reason direct 'python setup.py bdist_wheel' didn't work), and 'twine upload'. `pip install pypi-cdecimal´ should now work and get you an importable `cdecimal´, and if you happen to be lucky enough to run the same OS version I am, you won't even need to build C code. cdecimal users may wish to retrieve it via this mechanism until there's a secure way to get the proper upstream distribution. If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term. -glyph

(Is cdecimal substantially different from the _decimal added in 3.5?)

AFAICT, they are unrelated codebases that do about the same thing with the same amount of performance, with the main exception that _decimal in 3.5 does not require one to change their import (or to compile the package themselves.)

[cut overlong post]

Glyph, nice sneaky way to try to divert from the original issue. Your whole post is invalidated by the simple fact that the URL was protected by a hash (which I repeatedly asked to be upgraded to sha256). This was the official scheme promoted by PEP-438, which you should know. But of course your actual intention here is character assassination, pretending to "rescue" cdecimal and trying to divert from the fact that the transition to PEP 470 was handled suboptimally. The very reason for this thread is that the security was silently disabled WITHOUT me getting a notification. What is on PyPI *now* is not what I configured! Please believe me when I say I do not mean the following to be insulting -- people who have done *actual* cryptography to varying degrees often tend to focus on the important parts and aren't impressed by regurgitating catch phrases like SSL and man-in-the-middle: http://cr.yp.to/ecdh.html The amount of security "experts" in the Python community that pontificate on any occasion is pretty annoying. What do you think djb thinks of Twisted?

If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term.

That apparently all you can do: Move bits from place A to place B and not care how long it took to produce them. You are a real hero. Stefan Krah

<tritium-list <at> sdamon.com> writes:

You posted a mean-spirited complaint about a policy that is nearly exactly two years old, to the wrong list, and call out the people calmly trying to explain what happened and why, and how you can mitigate it for your own work

Yeah, right. This is distutils-sig level. Read Glyph's "calm" and "nice" post again. Also note that *I* had already bowed out of the thread when Glyph unnecessarily decided to spread complete misinformation. Stefan Krah

On Thu, 5 May 2016 at 16:54 David Wilson <dw+python-dev@hmmz.org> wrote:

This is mostly just observational, and not meant primarily as criticism of the fabulous work of Donald and others (ignoring pypa, also the prompt, reliable, and skilled support responses common on such places as IRC), however I can't help but notice that PyPI governance seems to come under fire vastly more often than similar and much more popular packaging systems, and some choices that have been made particularly in recent years have caused a noticeable amount of dissent with what might be considered the typical developer.

When a contributor to the core language is having repeat issues maintaining some basic element of the function of the packaging system, might it be fair to reflect on how changes to those functions are being managed?

There are PEPs covering a great deal of the work done to PyPI recently, but, and I say this as someone who has bumped into friction with the packaging tooling in the relatively recent past, even I despite my motivations to the contrary, I have not read most of them. It seems the current process is observed by few, does not sufficiently address the range of traditional use cases that were possible in the past, and the first the common user learns of a change is when pip (after insisting it must be upgraded) fails to function as it previously did.

The usual course then is some complaint, that leads to distutils-sig, which ultimately leads to pointing at some design work that was only observed by perhaps 50 people max that turns out had some edge cases that hurt in a common use case.

Is there something to contemplate in here? I dislike posting questions instead of answers, but it seems apparent there is a problem here and it continues to remain unaddressed.

This is whole thread is off-topic precisely because all of this is discussed -- in the open -- on distutils-sig and decided there. If people feel changes need to be made like broadcasting to a wider audience when a change occurs, then please bring it up on distutils-sig. But if people choose not to participate then they are implicitly delegating decision powers to those who do participate (which is totally fine and I'm not implying that if people don't participate they are doing something wrong).

On May 5, 2016, at 8:35 PM, David Wilson <dw+python-dev@hmmz.org> wrote:

On Fri, May 06, 2016 at 12:03:48AM +0000, Brett Cannon wrote:

...

Is there something to contemplate in here? I dislike posting questions instead of answers, but it seems apparent there is a problem here and it continues to remain unaddressed.

...

This is whole thread is off-topic precisely because all of this is discussed -- in the open -- on distutils-sig and decided there. If people feel changes need to be made like broadcasting to a wider audience when a change occurs, then please bring it up on distutils-sig.

I respectfully disagree, as this has been the default response applied for years, and it seems friction and dissemination have not been improved by it. Packaging is not some adjunct technicality, anyone learning Python in the past few years at least has been taught pip within the first week.

...

But if people choose not to participate then they are implicitly delegating decision powers to those who do participate

I believe this is also practically rhetorical in nature. I've watched the wars on distutils-sig for many years now, and the general strategy is that beyond minor outside influence, the process there is occupied by few individuals who are resistant to outside change. Outside influence is regularly met with essay-length reponses and tangential minutia until the energy of the challenge is expended.

Honestly, I just don't think this is an honest characterization. It is true that in general there are few people who bother to put in the effort to take a proposal from start to finish including actually writing the code to make such a thing happen. The problem of packaging is a particularly hard one where it's difficult to make trade offs because unlike other systems, people are sort of locked into using whatever the popular system is. If asyncio doesn't suite everyone that's fine it's not hard for them to go and switch and use Twisted, Tornado, gevent, eventlet, curio, etc etc but it's not very realistic for someone to opt out of the packaging ecosystem. In addition, it's much like something like HTTP or SMTP or the such where once you add some feature, it becomes incredibly difficult to ever remove it if you end up needing to (case in point, this thread and off site hosting), so we tend to over scrutinize changes wherever we can to try and make sure we *really* understand what the tradeoffs we're making are. Just as an example, it took a year and a half for PEP 440 to be standardized which is for something as bite sized as version numbers and PEP 440 was a continuation of the stalled PEP 386. The most time consuming part of PEP 440 was trying our new rules against every single version that existed on PyPI and minimizing the breakage. Even then, we've had several recent PEPs go through (manylinux1, environment markers, requirements) from people willing to do so.

As an example, one common argument is that "Donald is overworked", however as an example, I offered a very long time ago to implement full text indexing for PyPI search. At the time I belive I was told such things weren't necessary, only to learn a few years later that Donald himself implemented the same function, and it suffers from huge latency and accuracy issues in the meantime. The solution to those problems is of course the ever-delayed rewrite.

I don't remember the specific details around your proposal, but I'm pretty sure that *I* never told you that "such things were not necessary" since I've never held that opinion to my memory. What I have found [1][2] (to try and refresh my memory) are threads from you that went largely unanswered in April of 2013, which was right at the time I was heavily focused on getting PyPI setup behind Fastly and wasn't really paying attention to much else. I don't see anything else from you about it until September of 2015 when you offered your help again and at that point I told you that we had already switched to using Elasticsearch. In those two years since your initial offer, and then your follow up in 2015 I do not see any pull request to the PyPI code base from you (which, assuming they were reasonable would have been merged), however I do see the PR [3] from Ernest (No, I didn't implement the search) which got merged and deployed. My experience is that people are often willing to *offer* help with PyPI, but then they quickly disappear once they start to actually try and hack on it's code base and realize how difficult it is to work with. That experience tends to mean that I don't really get super excited when someone shows interest in helping because it rarely actually manifests. I think we've had more people contribute to Warehouse in the last year than we've *ever* had contribute to the legacy PyPI code base, which I think says a lot about the decision to switch.

Over on distutils-sig, one will learn that a large amount of effort has been poured into a rewrite of PyPI (an effort going on years now), however the original codebase was not far from rescue (I had a local copy almost entirely ported to Flask in a few days). There is no reason why this effort nor any other (like full text search) should be used, as it often is, as an argument in the decisionmaking process that largely governs how PyPI and pip have worked in the recent years, yet it only takes a few glances at the archives to demonstrate that it regularly is.

I find the statement that "the original code base is not far from rescue" a bit interesting, since you had previously stated [2] that: Again PyPI has been growing organically for a very long time, and dumping even more features in there doesn't seem a great idea. I looked at retrofitting PyPI with Flask, but there is simply too much custom code to be sure things won't be broken by doing it in a hurry. In any case, yes we're rewriting PyPI and it's been something that I've put a lot of effort into. That also means that I'm not particularly enthused about spending a bunch of time and effort working on legacy PyPI because doing so is incredibly demotivating to me to the point that the more time I spent in that code base the more I want to quit all together. Even given that I still have and would review PRs to that code base, it's just that very few people ever suffer through that code base long enough to actually submit one. I don't believe we've ever told someone that something can't happen because of Warehouse, only that *I* won't implement something until after Warehouse. That often times means that something won't happen until after Warehouse because of the severe shortage of people with enough time and motivation to work on this stuff but if someone did step up more things would get done. [1] https://mail.python.org/pipermail/distutils-sig/2013-April/020553.html [2] https://groups.google.com/d/msg/pypa-dev/ZjUNkczsKos/SzwNOckisXUJ [3] https://bitbucket.org/pypa/pypi/pull-requests/81/implement-an-elasticsearch-... ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 6, 2016, at 1:11 PM, A.M. Kuchling <amk@amk.ca> wrote:

On Thu, May 05, 2016 at 10:31:48PM -0400, Donald Stufft wrote:

...

I don't believe we've ever told someone that something can't happen because of Warehouse, only that *I* won't implement something until after Warehouse. That often times means that something won't happen until after Warehouse because of the severe shortage of people with enough time and motivation to work on this stuff but if someone did step up more things would get done.

Could the PSF help with this, whether by paying for Donald's (or a consultant's) time to whatever do final implementation, polishing, testing, or sysadmin work is required?

Personally, my time is already paid for by HPE to work on this stuff but I’m only one person. The PSF could pay for others to help with that though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA