[Catalog-sig] PyPI reverse download

["Martin v. Löwis"]

Am 27.07.2010 22:46, schrieb M.-A. Lemburg:
> "Martin v. Löwis" wrote:
>> I'll be implementing a feature for PyPI where you can POST
>> to a certain action (revdownload), and then PyPI will POST
>> the file requested to an URL that was passed; this is need
>> to make blobs work on AppEngine.
>>
>> Any objections?
> 
> Could you provide more detail on how this would work and why
> this is needed for AppEngine ?

Ok, here is the long story.

First, I tried to use the approach of pypione, using blobs for
distributions. That won't work because blobs are limited to 1MB.

Then I tried using lists of blobs instead. That won't work because
the HTTP response size in urlfetch is limited to 10GB.

Then I tried using Range: headers to mirror large files in pieces.
That won't work because I then wouldn't be able to serve the files
to setuptools, unless that would also start to use Range: headers.

The only way to serve files larger than 10GB is the blobstore.

However, apps can neither read from nor write to the blobstore.
The only way to read from it is to serve the file, and the only
way to write to it is through a POST from the outside.

BTW, Google has kindly granted the app access to the blobstore
(which is a for-fees feature only), and also kindly increased
the store quota (which is 1GB in the free service, when a PyPI
mirror needs about 15GB).

Regards,
Martin