[Distutils] Sourceforge mirrors, again

[Phillip J. Eby]

At 04:49 PM 9/19/2006 -0400, Kevin Dangoor wrote:
>On Sep 19, 2006, at 11:51 AM, Phillip J. Eby wrote:
>
> > Well, it looks like Sourceforge has found yet another way to mess with
> > easy_install's ability to download from their mirrors.  :
> > (  Specifically,
> > they are not keeping the dl.sourceforge.net "A" list up-to-date, so
> > easy_install's attempts to just use simple round-robin DNS aren't
> > always
> > working.  Several IPs in the round robin "A" list are not
> > responding, and
> > some new mirrors haven't been added to it.  At this rate, the current
> > approach will become unusable in a relatively short timeframe.  :(
> >
> > It seems as though there is no way to auto-discover the mirrors
> > themselves;
> > I had hoped that perhaps a zone transfer on the dl.sourceforge.net
> > zone
> > might work to obtain a list of the actual mirrors, but I haven't
> > been able
> > to successfully obtain one.
> >
> > What I'm wondering at this point is if perhaps the only sane thing
> > to do is
> > to publish our own mirror list via DNS, so that at least when
> > there's a
> > problem it can still be fixed.  The idea would be to replace
> > easy_install's
> > current DNS lookup of 'dl.sourceforge.net' IPs with something like
> > 'dl.sfmirrors.telecommunity.com' (for example).
>
>That seems like a good idea. One other possibility is if we want to
>future-proof against changes to SF.net's download pages, there could
>be a web service that figures out where to send the user to a file.
>That way, only that service needs to change if they change their
>download page format. That is, of course, a lot heavier of a solution.
>
>Doing the DNS change seems like a good idea.

I've implemented a proof of concept as 'sf-mirrors.telecommunity.com', with 
a cron job that scrapes the mirror names via HTTP and then updates the zone 
file.  For the moment, it's set up to automatically halt if there's any 
change in the mirror names or the number of mirrors, so I can make sure the 
change isn't due to SF changing their UI again.  If there are no changes 
and the script is successful in pulling the current IPs for the named 
mirrors, it updates the zone file.

Anybody want to give it a try?  Just change all references to 
'dl.sourceforge.net' in setuptools/package_index.py with references to 
'sf-mirrors.telecommunity.com'.

I'm not sure what I think about it, exactly.  One issue is that it makes it 
look like the software is "phoning home" to me, or that downloads are 
coming from my servers, even though they are unrelated.  It's also possible 
that some mirrors might freak when they receive a 'Host:' header that 
points to telecommunity.com!

So, I'm not 100% sure this can work reliably yet.  Maybe it would be better 
to just encourage SF to fix their broken DNS records.  :(