[Distutils] Buildout can't talk to PyPI any more?

[Marius Gedminas]

On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote:
> On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas <marius at pov.lt> wrote:
> > Could it be that recent PyPI CDN changes broke zc.buildout somehow?
> > Witness this:
> >
> >   $ wget http://downloads.buildout.org/2/bootstrap.py
> >
> >   $ cat buildout.cfg
> >   [buildout]
> >   parts = zodbbrowser
> >   [zodbbrowser]
> >   recipe = zc.recipe.egg
> >
> >   $ python bootstrap.py
> >
> >   $ bin/buildout
> >   Error: Couldn't find a distribution for 'zodbbrowser'.
> 
> Works for me.

Works for me too, now.


Interesting detail: when buildout was failing, and I tried

  curl http://pypi.python.org/simple/zodbbrowser/

all I got was a bunch of binary garbage.  To be fair, curl -I showed
Content-encoding: gzip, and piping curl to zcat showed the HTML just
fine.  But the interesting thing is that when I try this now, I get raw
HTML.

Can it be that buildout doesn't handle Content-Encoding: gzip?

I was also looking at raw HTTP requests with ngrep, and saw that
buildout issued something like

  GET /simple/zodbbrowser HTTP/forgotwhichversionsadly
  Host: ...
  Accept-Encoding: identity
  User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib

That "Accept-Encoding" header kind of hints that distribute can't handle
gzipped index pages.  So maybe Varnish or nginx or something on the CDN
side was misconfigured and ignored the Accept-Encoding?  But why would
pip/easy_install work then -- they use distribute too!?  (Or was I
accidentally using a virtualenv that had setuptools instead of
distribute?  Python packaging aargh.)


Another possibly interesting detail: I made a release of zodbbrowser
0.11.0 this morning.  Buildout was unable to get it right after that,
and this condition lasted for several hours (from 07:40 UTC until some
point between 08:20 and 09:00 UTC).

The Zope Windows buildbot had a couple of similar intermittent errors
(buildout was unable to find a distribution of some package) that went
away when the build job was retried after 24 hours.  Could it be that
something about the PyPI CDN causes new package releases to be gzipped
for an hour or so, and then revert to plain HTML?

I'm tempted to do a test, something like

  curl -I http://pypi.python.org/simple/somepackage/
  cd ~/src/somepackage && fullrelease # zest.releaser FTW
  curl -I http://pypi.python.org/simple/somepackage/

if I can find some package ready for a new release.

Marius Gedminas
-- 
We'll show a small example here with one user calling another. (Under
international treaties describing technical papers these users must be called
"Alice" and "Bob".)
        -- Anthony Baxter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130529/a8a4ea93/attachment.pgp>