[Catalog-sig] Rewrite PyPI for App Engine?

M.-A. Lemburg mal at egenix.com
Fri Jun 25 10:39:45 CEST 2010

Ian Bicking wrote:
> On Thu, Jun 24, 2010 at 5:16 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Almir Karic wrote:
>>> i would like to help out with the move.
>>> is anyone actually opposed to moving to GAE (either moving the current
>>> code base or re-write, whichever seems more appropriate)?
>> I don't think people are opposed to having a PyPI clone on GAE,
>> but moving the existing installation to GAE is something we would
>> have to discuss separately.
>> I for one would not welcome such a change, since we then completely
>> lose control over service availability.
> I don't really understand what this means.  Services become unavailable
> sometimes.  A computer breaks, a company shuts down, an agreement ends.  We
> don't necessarily have "control" over these situations, but we can respond
> to them.  If App Engine goes down and the App Engine team is all like
> "whatever, we'll get around to fixing stuff sometime" then sure it's a
> problem.  But it's not a plausible problem.  The plausible problem is that
> App Engine goes down, as it has from time to time, and we have to wait for
> them to figure out what's wrong and fix it.  *We* don't have to fix it, we
> only have to *wait for someone else to do it*.  I don't see any reason why
> *we* are any better at fixing issues than the App Engine team would be.
> Also presumably when there is a failure we want for the failure to be
> understood and avoided in the future.  The App Engine team does that.  And
> they do that *for us*.

I hear you, but don't agree that putting the runtime into the
hands of the GAE would get us an overall better service :-)

The point is that with GAE you only have control over the code
that you post there. Everything else is under control of the GAE
team (and their automatic administration systems), i.e. whether
your data is available and whether there are
proper backups, whether the site is reachable or not, whether
the performance is available and meets your requirements, whether
the service is accessible, fast enough and has low latency, etc.

So if something breaks, you can only fix it, if the problem
is caused by a bug in the code. For all other situations, you
have to wait for the GAE team to go in and do whatever is needed.

I'm not saying that the GAE team would be doing a poor job,
but just sitting there waiting for them to fix it in any
of the typical problem situations (apart from a bug in the
code), is asking a bit much, IMHO.

We have to find a middle ground, where we can still apply the
necessary hand holding ourselves, if we like to, while leaving
most of the day-to-day tasks to automatic tools or other service
providers to deal with.

Since PyPI is becoming a central piece of Python community
infrastructure, we need to make sure that we can provide a very
good uptime of the service and fast access to the data,
esp. for the automatic download tools.

Fortunately, those tools only use static data, so focusing on
making that highly available will get us a much better service
uptime with little extra effort.

> In some catastrophic case we could move the site to another server, use
> TyphoonAE to move the code over (or simply require that there is a
> sufficient abstraction layer to allow for a more normal environment) and
> bring the site up.  We control the domain, we can ultimately control where
> it is hosted.  This kind of failure seems like it would be far more likely
> given our current situation than on App Engine, but moving to App Engine
> would not somehow make this kind of move impossible.

True, but do you really want to go through all that trouble
just because GAE is down or too slow to be usable again ?

If we were to go for a cloud service to deploy the PyPI runtime, I'd
much rather like to see a standard virtualized server approach
being used.

With that approach, moving (virtual) servers would take
at most 5 minutes, if needed at all - you can rather easily setup
virtual servers as high availability cluster and then have
them manage the failover all by themselves.

BTW: Here's a nice blog on the subject of downtimes:

>> Someone would also have to do some math to calculate the monthly
>> costs for the PSF:
>>    http://code.google.com/appengine/docs/quotas.html
>>    http://code.google.com/appengine/docs/billing.html
>>    http://code.google.com/appengine/business/
> It seems unlikely we'd have to pay for the service.

Perhaps, but then someone will have to get that information as well.

Marc-Andre Lemburg

Professional Python Services directly from the Source  (#1, Jun 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
2010-07-19: EuroPython 2010, Birmingham, UK                23 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Catalog-SIG mailing list