[Catalog-sig] Mirror list detection/construction - PEP 381

Jesus Cea jcea at jcea.es
Thu Jul 22 02:52:51 CEST 2010


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/07/10 21:20, P.J. Eby wrote:
> At 01:10 PM 7/21/2010 +0100, Paul Nasrat wrote:
>> Given the fragility of this it seems that we might want to consider
>> alternative mirrorlist discovery mechanism.
> 
> As a fallback, you can of course probe addresses on a binary search, or
> even just select a random mirror in the first place.  (If you overshoot
> the list, you just decrease the max on your binary lookup.)
> 
> Non-random selection is tougher to implement, since you'd need to keep
> some kind of history to make it work effectively.  Determining the
> length of the list is a trivial problem by comparison.

Well a random shuffle is a standard operation in "random" module :-).

What I usually do is to pick a random server and my previous selection
(the first time, you choose a second random server). Then do some
operations in BOTH and choose the faster. Complete the operation on it
and keep it for the next time.

This way you always go the fast route, but randomly and gently probe
other nodes trying to find even other faster.

Since using pypi is critical but not very frequent (nobody is going to
install a hundred modules daily), being smart is, maybe, overkill. Could
be enough just pick 5 nodes at random, check them simultaneously
(threads!), choose one and complete the download from there. If the
download fails, repeat from the very beginning a few times.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTEeWY5lgi5GaxT1NAQI4/gP+Ibqmsqel2tiEN6zdmWWZIKJsfoKS+u8x
B9XEHvrnF54YFU+GJOqcEmRRsHndy2DKMcWpH0t1wzMSIZlGbdOqtqG47b+KJfnD
8Qun5f5oWvyQXPoPSUqngQHQx7mPOQhTTUP75oeduTjMm6NOBzsbIDhKTDfiSP9g
RzYQo+QK3u8=
=ENnz
-----END PGP SIGNATURE-----


More information about the Catalog-SIG mailing list