[Catalog-sig] High-availability for PyPI, mirroring infrastructures?
tarek.ziade at ingeniweb.com
Wed Aug 13 19:22:18 CEST 2008
2008/8/12 Phillip J. Eby <pje at telecommunity.com>
> At 02:13 PM 8/12/2008 +0200, Tarek Ziade wrote:
>> 2008/8/12 Jim Fulton <<mailto:jim at zope.com>jim at zope.com>
>> On Aug 12, 2008, at 4:41 AM, Tarek Ziade wrote:
>> Since mirrors for PyPI and alternative egg servers are being , what about
>> making the "index" variable in
>> zc.buildout accept multiple values, like what find-links does ?
>> Buildout uses and tries hard to be compatible with setuptools. In
>> particular, it relies on the setuptools.package_index.PackageIndex class,
>> which only allows a single index to be provided.
>> Yes, I was thinking of handling several PackageIndex instances on
>> zc.buildout side as a first thaught.
> What you really want/need is to make the PackageIndex support retrieval
> from multiple index urls; the PackageIndex itself aggregates available
> packages from sources such as the local file system, -f urls, and an
> underlying package index. So having multiple aggregators would duplicate
> processing, and deprive you of a global ordering of package precedences.
Is this a feature you would like to see in setuptools ? If so I can write a
Besides this feature, there's one feature we started to add on zc.buildout
but could be put in setuptools's PackageIndex as well I believe :
Adding timeouts for url openings, to speed up the processing. For instance,
when several urls are visited for
a given package, and when a server pointed by one of the url is down, it can
urlib2.urlopen does not have a timeout option yet, even if it has been
discussed for Python 2.6 inclusion, 2 years ago 
So we used *socket*.*setdefaulttimeout*() , which looks safe to change in
this use case.
At zc.buildout level, we could wait for up to 5 minutes to finish a buildout
and setting the timeout to a few seconds solved the problem efficiently.
In a shape of a decorator, that could be :
# five seconds
TIMEOUT = 5
def _timedout(*args, **kw):
return func(*args, **kw)
Tarek Ziadé - Directeur Technique
INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632
Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage
92210 Saint Cloud - France
Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04
http://www.ingeniweb.com - une société du groupe Alter Way
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Catalog-SIG