[Distutils] Buildout can't talk to PyPI any more?

[Donald Stufft]

On May 29, 2013, at 5:39 AM, Donald Stufft <donald at stufft.io> wrote:

> 
> On May 29, 2013, at 5:18 AM, Marius Gedminas <marius at pov.lt> wrote:
> 
>> On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote:
>>> On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas <marius at pov.lt> wrote:
>>>> Could it be that recent PyPI CDN changes broke zc.buildout somehow?
>>>> Witness this:
>>>> 
>>>>  $ wget http://downloads.buildout.org/2/bootstrap.py
>>>> 
>>>>  $ cat buildout.cfg
>>>>  [buildout]
>>>>  parts = zodbbrowser
>>>>  [zodbbrowser]
>>>>  recipe = zc.recipe.egg
>>>> 
>>>>  $ python bootstrap.py
>>>> 
>>>>  $ bin/buildout
>>>>  Error: Couldn't find a distribution for 'zodbbrowser'.
>>> 
>>> Works for me.
>> 
>> Works for me too, now.
>> 
>> 
>> Interesting detail: when buildout was failing, and I tried
>> 
>>  curl http://pypi.python.org/simple/zodbbrowser/
>> 
>> all I got was a bunch of binary garbage.  To be fair, curl -I showed
>> Content-encoding: gzip, and piping curl to zcat showed the HTML just
>> fine.  But the interesting thing is that when I try this now, I get raw
>> HTML.
>> 
>> Can it be that buildout doesn't handle Content-Encoding: gzip?
>> 
>> I was also looking at raw HTTP requests with ngrep, and saw that
>> buildout issued something like
>> 
>>  GET /simple/zodbbrowser HTTP/forgotwhichversionsadly
>>  Host: ...
>>  Accept-Encoding: identity
>>  User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib
>> 
>> That "Accept-Encoding" header kind of hints that distribute can't handle
>> gzipped index pages.  So maybe Varnish or nginx or something on the CDN
>> side was misconfigured and ignored the Accept-Encoding?  But why would
>> pip/easy_install work then -- they use distribute too!?  (Or was I
>> accidentally using a virtualenv that had setuptools instead of
>> distribute?  Python packaging aargh.)
>> 
>> 
>> Another possibly interesting detail: I made a release of zodbbrowser
>> 0.11.0 this morning.  Buildout was unable to get it right after that,
>> and this condition lasted for several hours (from 07:40 UTC until some
>> point between 08:20 and 09:00 UTC).
>> 
>> The Zope Windows buildbot had a couple of similar intermittent errors
>> (buildout was unable to find a distribution of some package) that went
>> away when the build job was retried after 24 hours.  Could it be that
>> something about the PyPI CDN causes new package releases to be gzipped
>> for an hour or so, and then revert to plain HTML?
>> 
>> I'm tempted to do a test, something like
>> 
>>  curl -I http://pypi.python.org/simple/somepackage/
>>  cd ~/src/somepackage && fullrelease # zest.releaser FTW
>>  curl -I http://pypi.python.org/simple/somepackage/
>> 
>> if I can find some package ready for a new release.
>> 
>> Marius Gedminas
>> -- 
>> We'll show a small example here with one user calling another. (Under
>> international treaties describing technical papers these users must be called
>> "Alice" and "Bob".)
>>        -- Anthony Baxter
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> The cache timeout is an hour unless modifications are made to the package which triggers a cache invalidation.
> 
> I bet there is a missing Vary for the encoding. I'll check and get back to you.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


There was indeed no Vary headers, I've turned them on and purged the cache. Please let me know if this behavior doesn't go away.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130529/77b73cf9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130529/77b73cf9/attachment.pgp>