Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks.
Phillip J. Eby <pje@telecommunity.com> wrote:
Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks.
This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files? -- Giovanni Bajo
On 1/17/06, Giovanni Bajo <rasky@develer.com> wrote:
Phillip J. Eby <pje@telecommunity.com> wrote:
Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks.
This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files?
The problem is that a machine doesn't click on ads. (Or, at least, isn't supposed to :) Kevin
Kevin Dangoor wrote:
This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files?
The problem is that a machine doesn't click on ads. (Or, at least, isn't supposed to :)
Since they have a 5-second delay in downloading, I assume they really are trying to get ad clicks (since 1 or 0 seconds would do just as well, and direct links would of course be better for everyone). OTOH, it never hurts to ask; maybe someone can open a ticket over on SF? If anyone does, please note that here. Hell, a stable API with a request that API users show a text ad (that would be delivered as part of the API) would be fine with me (as a user of easy_install). But right now I think it is more useful to encourage people to use the cheese shop for file distribution -- SF is also really hard to upload to, so the cheese shop is better on both ends. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 02:09 PM 01/17/2006 -0600, Ian Bicking wrote:
Since they have a 5-second delay in downloading, I assume they really are trying to get ad clicks (since 1 or 0 seconds would do just as well, and direct links would of course be better for everyone). OTOH, it never hurts to ask; maybe someone can open a ticket over on SF? If anyone does, please note that here. Hell, a stable API with a request that API users show a text ad (that would be delivered as part of the API) would be fine with me (as a user of easy_install).
I checked what the NetBSD "pkgsrc" system does, and it uses the fact that there is a .dl.sourceforge.net subdomain for mirrors. I investigated further and found that dl.sourceforge.net is a round-robin (or random?) DNS for each of the mirrors in the subdomain, so there's a simple transformation from the user-visible download pages to the actual download address. You can see this in the new fix_sf_url() function I added to setuptools.package_index.
But right now I think it is more useful to encourage people to use the cheese shop for file distribution -- SF is also really hard to upload to, so the cheese shop is better on both ends.
Yeah, uploading to cheeseshop is great for new projects. But practically speaking, SF *is* used to distribute a lot of things, especially big things that mirrors are more useful for. So, it's definitely good for stuff to work. :)
Phillip J. Eby wrote:
I checked what the NetBSD "pkgsrc" system does, and it uses the fact that there is a .dl.sourceforge.net subdomain for mirrors. I investigated further and found that dl.sourceforge.net is a round-robin (or random?) DNS for each of the mirrors in the subdomain, so there's a simple transformation from the user-visible download pages to the actual download address. You can see this in the new fix_sf_url() function I added to setuptools.package_index.
I just got a Connection Refused error, but it worked on the second try. As I remember, the port system typically has a set of links, and frequently fails over from one link to the next, so I expect that this error should be expected. When it is encountered, setuptools should just try again until it works. I have noticed in the past that SF mirrors go up and down quite frequently, probably too fast for DNS to keep up. -- Ian Bicking | ianb@colorstudy.com | http://blog.ianbicking.org
On 1/18/06, Ian Bicking <ianb@colorstudy.com> wrote:
Phillip J. Eby wrote:
I checked what the NetBSD "pkgsrc" system does, and it uses the fact that there is a .dl.sourceforge.net subdomain for mirrors. I investigated further and found that dl.sourceforge.net is a round-robin (or random?) DNS for each of the mirrors in the subdomain, so there's a simple transformation from the user-visible download pages to the actual download address. You can see this in the new fix_sf_url() function I added to setuptools.package_index.
I just got a Connection Refused error, but it worked on the second try. As I remember, the port system typically has a set of links, and frequently fails over from one link to the next, so I expect that this error should be expected. When it is encountered, setuptools should just try again until it works. I have noticed in the past that SF mirrors go up and down quite frequently, probably too fast for DNS to keep up.
Does the pypi download url need to be updated to be compatible with svn setuptools? I am assuming no, but I am constantly getting 404's from easy_install, yet I can paste the dl.sf.net link into a browser and it works?! These files have been on sf long enough to be propgated around, and constantly ~= 20 attempts on the command line. I am seeing this with matplotlib, btw. Thanks, Charlie
At 07:54 PM 01/17/2006 +0100, Giovanni Bajo wrote:
Phillip J. Eby <pje@telecommunity.com> wrote:
Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks.
This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files?
They have already done it, and I've changed setuptools to use it now (as of r42088). I've now removed all of the code originally contributed by Ian to parse SourceForge's "human user" HTML and simply preprocess prdownloads.sf.net URLs to a similar path on dl.sourceforge.net, which is a round-robin DNS for the various mirrors. This cuts two page fetches out of the process and eliminates any HTML scraping dependencies. As long as SF doesn't get rid of the *.dl system, this should continue to work no matter how much the HTML frontend changes.
Phillip J. Eby wrote:
Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks.
This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files?
They have already done it, and I've changed setuptools to use it now (as of r42088). I've now removed all of the code originally contributed by Ian to parse SourceForge's "human user" HTML and simply preprocess prdownloads.sf.net URLs to a similar path on dl.sourceforge.net, which is a round-robin DNS for the various mirrors. This cuts two page fetches out of the process and eliminates any HTML scraping dependencies. As long as SF doesn't get rid of the *.dl system, this should continue to work no matter how much the HTML frontend changes.
So far my experiences downloading this way have been very poor -- it fails more often than not, often stalling indefinitely. What are other people's experiences? If my experience is typical, I think it would be better to go back to the regex way with a fixed regex -- it may be fragile in the long term, but it works much better for the moment. To deal with this for Paste, I even asked Tavis to put a Cheetah package on the Cheese Shop page, but since the SF link is longer easy_install prefers it :( -- Ian Bicking | ianb@colorstudy.com | http://blog.ianbicking.org
At 08:34 PM 1/26/2006 -0600, Ian Bicking wrote:
So far my experiences downloading this way have been very poor -- it fails more often than not, often stalling indefinitely. What are other people's experiences? If my experience is typical, I think it would be better to go back to the regex way with a fixed regex -- it may be fragile in the long term, but it works much better for the moment.
What platform are you using, and what revision of easy_install? The first version of the dl.sf.net trick could end up using the same IP over and over depending on local DNS cache settings. Windows machines were particularly prone to getting stuck with a bad IP. The current version explicitly requests *all* of the IPs, then selects from them at random.
To deal with this for Paste, I even asked Tavis to put a Cheetah package on the Cheese Shop page, but since the SF link is longer easy_install prefers it :(
There's probably a different reason why. All else being equal, easy_install prefers *shorter* URLs for the same target version of a package.
participants (5)
-
Charlie Moad -
Giovanni Bajo -
Ian Bicking -
Kevin Dangoor -
Phillip J. Eby