[Catalog-sig] PyPI reverse download
noah at coderanger.net
Tue Jul 27 23:38:15 CEST 2010
> -----Original Message-----
> From: catalog-sig-bounces+noah=coderanger.net at python.org
> [mailto:catalog-sig-bounces+noah=coderanger.net at python.org] On Behalf
> Of "Martin v. Löwis"
> Sent: Tuesday, July 27, 2010 2:07 PM
> To: M.-A. Lemburg
> Cc: catalog-sig
> Subject: Re: [Catalog-sig] PyPI reverse download
> Am 27.07.2010 22:46, schrieb M.-A. Lemburg:
> > "Martin v. Löwis" wrote:
> >> I'll be implementing a feature for PyPI where you can POST
> >> to a certain action (revdownload), and then PyPI will POST
> >> the file requested to an URL that was passed; this is need
> >> to make blobs work on AppEngine.
> >> Any objections?
> > Could you provide more detail on how this would work and why
> > this is needed for AppEngine ?
> Ok, here is the long story.
> First, I tried to use the approach of pypione, using blobs for
> distributions. That won't work because blobs are limited to 1MB.
> Then I tried using lists of blobs instead. That won't work because
> the HTTP response size in urlfetch is limited to 10GB.
> Then I tried using Range: headers to mirror large files in pieces.
> That won't work because I then wouldn't be able to serve the files
> to setuptools, unless that would also start to use Range: headers.
> The only way to serve files larger than 10GB is the blobstore.
> However, apps can neither read from nor write to the blobstore.
> The only way to read from it is to serve the file, and the only
> way to write to it is through a POST from the outside.
> BTW, Google has kindly granted the app access to the blobstore
> (which is a for-fees feature only), and also kindly increased
> the store quota (which is 1GB in the free service, when a PyPI
> mirror needs about 15GB).
I've been able to write to it using urlfetch internally. You just craft a
request with the POST from the app and send it to the URL you get from the
More information about the Catalog-SIG