[Distutils] PyPI Migrated to New Infrastructure with some Breakage

Nick Coghlan ncoghlan at gmail.com
Sun Jan 26 01:40:45 CET 2014

On 26 Jan 2014 09:51, "Richard Jones" <richard at python.org> wrote:
> Thanks everyone who helped make this happen.

Indeed - fine work! :)

> From my perspective* I believe the ssh upload mechanism was added to
address security issues around the basic-auth-over-http method used
historically. Now uploads *may* be done over https, and those using the ssh
method can move over to using twine or pip upload, I think that it's
reasonable to discontinue support for ssh uploads.

Yes, I agree that pointing the (very few) pypissh users towards twine as a
replacement is the most reasonable option at this point - we should chat to
MvL about putting a notice to that effect in the pypissh README (IIRC, MvL
is the creator of that upload option).


>      Richard
> * as the guy who will be hassled if its loss is noticed ;)
> On 26 January 2014 10:38, Donald Stufft <donald at stufft.io> wrote:
>> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new
>> infrastructure.
>> The old infrastructure was:
>> - a single database server managed by OSUOSL
>> - a pair of load balancers shared by all of the python.org services
hosted on
>> - a single backend VM that served as everything else for PyPI
>> The VM that was acting as the backend server from PyPI was partially hand
>> configured and partially setup to be managed by chef. Additionally it
had an
>> issue that caused it to kernel panic every so often which had been the
cause of
>> a number of downtimes in the last few months. Because it was primarily
>> configured and administered by hand and because the way it was set up it
>> not feasible to have any sort of failover or spare.
>> The new infrastructure is:
>> - 2 Web VMs
>> - 2 Database servers in Master/Slave Configuration
>> - 2 PgPool Servers pooling connections to the database servers and load
>>   balancing reads across them.
>> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file
>>   for package and package docs
>> - 1 metrics server to handle updating the download counts as they come
in from
>>   Fastly
>> All of the VMs are hosted on Rackspace’s Public Cloud and have their
>> configuration completely controlled and managed using Salt. Going
forward this
>> will allow us to easily scale out as required or kill malfunctioning
>> and spin up new ones easily. Additionally the setup has been setup so
>> where possible there is two servers performing the same role, ideally in
>> Active/Active configuration but at least in a Master/Slave
configuration. This
>> should allow PyPI to be far more stable moving forward and make
downtimes much
>> easier to recover from.
>> The services are still fronted by Fasty’s CDN and in the new
>> we’ve removed our load balancer and have replaced it by having Fastly
>> the load balancing for us. Additionally we’ve recently setup a static
mirror of
>> PyPI that is updated once every minute. This is hosted on Rackspace
cloud as
>> well but in a separate data center from the rest of PyPI. Fastly is
>> to fall back to this static mirror in the case that neither of the two
>> heads are functioning. This should ensure that even in the event of a
>> catastrophic failure of the PyPI service that the bulk of package
>> should hopefully remain working.
>> The bad news, (and the “Breakage” from the subject) is that while the new
>> infrastructure was being planned out, built, and migrated to the
>> package was forgotten. The pypissh package is an alternative way to
>> packages to PyPI however it is very difficult, because of the way it
works, to
>> provide HA support for it as we’ve set up for everything else. We don’t
>> any numbers for how many people are actively using this package but
>> at a roughly 2 week chunk of time in PyPI’s download history, the pypissh
>> package was downloaded 7 times by a browser, and 7 times by pip. All
>> downloads were caused by the mirroring system.
>> As of right now pypissh is non functional and due to the difficulty in
>> and monitoring the current setup and because it is apparently has a very
>> small set of users we would like to effectively kill off this particular
>> service. Additionally the benefits of pypissh have been reduced now that
>> is available over a TLS connection with a well trusted certificate. My
>> to you is, is this something that distutils-sig is willing to have
happen? If
>> we are to re-enable pypissh we’ll need to write a new solution to doing
it that
>> can be properly HA’d and we’d prefer to put our efforts into improving
>> for a much larger set of people.
>> So yea, PyPI should be loads more stable and more reliable now.
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140126/385b93e5/attachment.html>

More information about the Distutils-SIG mailing list