[Distutils] Sourceforge mirrors, again
Phillip J. Eby
pje at telecommunity.com
Wed Sep 20 00:01:42 CEST 2006
At 04:49 PM 9/19/2006 -0400, Kevin Dangoor wrote:
>On Sep 19, 2006, at 11:51 AM, Phillip J. Eby wrote:
> > Well, it looks like Sourceforge has found yet another way to mess with
> > easy_install's ability to download from their mirrors. :
> > ( Specifically,
> > they are not keeping the dl.sourceforge.net "A" list up-to-date, so
> > easy_install's attempts to just use simple round-robin DNS aren't
> > always
> > working. Several IPs in the round robin "A" list are not
> > responding, and
> > some new mirrors haven't been added to it. At this rate, the current
> > approach will become unusable in a relatively short timeframe. :(
> > It seems as though there is no way to auto-discover the mirrors
> > themselves;
> > I had hoped that perhaps a zone transfer on the dl.sourceforge.net
> > zone
> > might work to obtain a list of the actual mirrors, but I haven't
> > been able
> > to successfully obtain one.
> > What I'm wondering at this point is if perhaps the only sane thing
> > to do is
> > to publish our own mirror list via DNS, so that at least when
> > there's a
> > problem it can still be fixed. The idea would be to replace
> > easy_install's
> > current DNS lookup of 'dl.sourceforge.net' IPs with something like
> > 'dl.sfmirrors.telecommunity.com' (for example).
>That seems like a good idea. One other possibility is if we want to
>future-proof against changes to SF.net's download pages, there could
>be a web service that figures out where to send the user to a file.
>That way, only that service needs to change if they change their
>download page format. That is, of course, a lot heavier of a solution.
>Doing the DNS change seems like a good idea.
I've implemented a proof of concept as 'sf-mirrors.telecommunity.com', with
a cron job that scrapes the mirror names via HTTP and then updates the zone
file. For the moment, it's set up to automatically halt if there's any
change in the mirror names or the number of mirrors, so I can make sure the
change isn't due to SF changing their UI again. If there are no changes
and the script is successful in pulling the current IPs for the named
mirrors, it updates the zone file.
Anybody want to give it a try? Just change all references to
'dl.sourceforge.net' in setuptools/package_index.py with references to
I'm not sure what I think about it, exactly. One issue is that it makes it
look like the software is "phoning home" to me, or that downloads are
coming from my servers, even though they are unrelated. It's also possible
that some mirrors might freak when they receive a 'Host:' header that
points to telecommunity.com!
So, I'm not 100% sure this can work reliably yet. Maybe it would be better
to just encourage SF to fix their broken DNS records. :(
More information about the Distutils-SIG