Re: [Distutils] [Catalog-sig] High-availability for PyPI, mirroring infrastructures?

2008/8/20 Phillip J. Eby pje@telecommunity.com
At 07:22 PM 8/13/2008 +0200, Tarek Ziade wrote:
2008/8/12 Phillip J. Eby <mailto:pje@telecommunity.com pje@telecommunity.com> What you really want/need is to make the PackageIndex support retrieval from multiple index urls; the PackageIndex itself aggregates available packages from sources such as the local file system, -f urls, and an underlying package index. So having multiple aggregators would duplicate processing, and deprive you of a global ordering of package precedences.
Is this a feature you would like to see in setuptools ? If so I can write a patch,
Just be aware that I'm likely to be rather picky about how it works. :)
I bet, I'll give a try and submit it :)
Besides this feature, there's one feature we started to add on zc.buildout
but could be put in setuptools's PackageIndex as well I believe :
Having the option to set a short timeout for *connections* would be useful, I think, just as long as it doesn't end up cutting off slow downloads. I'd prefer it to be controllable from the command line, nonetheless.
As far as I know, the socket timeout will not apply on a download that is beeing processed, but only on 'sleeping' connections. For instance, if I download Firefox with urlopen(), it takes up to 1 second to read() it and the timeout won't occur in that case :
import socket socket.setdefaulttimeout(1) import urllib2 g = urllib2.urlopen('
http://www.gtlib.gatech.edu/pub/mozilla.org/firefox/releases/3.0.1/mac/fr/Fi... ')
g.read()
xxxxxxx I get firefox dmg file here
Although I don't know how it behaves on stale connections, I will digg on this
Tarek

On Thu, Aug 21, 2008 at 2:59 PM, Tarek Ziade tarek.ziade@ingeniweb.comwrote:
2008/8/20 Phillip J. Eby pje@telecommunity.com
At 07:22 PM 8/13/2008 +0200, Tarek Ziade wrote:
2008/8/12 Phillip J. Eby <mailto:pje@telecommunity.com pje@telecommunity.com> What you really want/need is to make the PackageIndex support retrieval from multiple index urls; the PackageIndex itself aggregates available packages from sources such as the local file system, -f urls, and an underlying package index. So having multiple aggregators would duplicate processing, and deprive you of a global ordering of package precedences.
Is this a feature you would like to see in setuptools ? If so I can write a patch,
Just be aware that I'm likely to be rather picky about how it works. :)
I bet, I'll give a try and submit it :)
Ok I have submitted a patch here http://bugs.python.org/setuptools/issue32
I have added a test where I played with two indexes: Pypi and another custom one, that hold a package version PyPI don't.
I changed the code accordingly, and recorded all urllib2.urlopen input/output into a mapping saved in a shelve file, so the test can work without external calls. (maybe this could be pushed in a helper in the sandbox)
I have also added a deprecation warning, but I don't know if this is something you want..
I have tested it successfully with several buildouts and I will test it more this week, together with your latest bug fixes.
Tarek
participants (2)
-
Tarek Ziade
-
Tarek Ziadé