[Catalog-sig] PyPI down again...

[Tarek Ziadé]

2010/6/13 "Martin v. Löwis" <martin at v.loewis.de>:
>> I think it overlaps a bit the PEP goal, which is to set up a network of
>> mirrors,
>> and have them listed in the PyPI DNS  so clients can switch from one
>> mirror
>> to another.(and even do geoloc!)
>
> JFTR, this already exists. a.mirrors.pypi.python.org and
> b.mirrors.pypi.python.org are already there and could be used by clients.

I wasn't aware of these mirrors. Do you maintain them ? how are they
synchronized ?
Do you get the statistics if we use them ?

If so, we could start to use them in all clients asap. (as fallbacks
if PyPI gets down)

>
>> Well maybe this is the best path to follow right now, as it will be done
>> faster,
>> without having to interact with much people to set it up, so it's a quick
>> win.
>
> My main worry (besides the client integration) is statistics: I do want to
> get download statistics. So anybody implementing it would have to find a way
> of fetching the download numbers from Amazon.
>
>> But it will probably kill the mirroring protocol idea from the PEP in
>> the process,
>> which I think is superior in the long term since it provides a
>> standardized ground
>> for the community to set up mirrors independently from pypi.python.org.
>
> I also remain skeptical that this cloud idea is useful at all. Amazon
> Cloudfront is a *beta* service. So they aren't sure themselves whether it
> works correctly - and there have been reports about two-day outages of EC2,
> for bitbucket.org. There also have been complaints about the available
> bandwidth. So I'm not sure whether replacing a single point of failure with
> a different one is actually improving anything.

ISTM that the workload is the same, whether a cloud or a regular mirror is used,
because of the statistics.  FIY, the work to be done for the mirrors,
beside the PEP
editing consist of :

- implementing the extra pages generation + stats builder in a package
like z3c.pypimirror
  (http://pypi.python.org/pypi/z3c.pypimirror) which is used by several mirrors.

- adding the client-side code in a project like Distribute or Pp


Regards
Tarek



-- 
Tarek Ziadé | http://ziade.org