Deflate with urllib2...

Sam samslists at
Fri Sep 19 04:29:30 CEST 2008

On Sep 18, 2:10 pm, "Gabriel Genellina" <gagsl-... at>
> En Tue, 16 Sep 2008 21:58:31 -0300, Sam <samsli... at> escribió:
> The code is correct - try with another server. I tested it with a  
> LightHTTPd server and worked fine.


I found a bunch of servers to test it on.  It fails on every server I
could find (sans one).

Here's the ones it fails on:

I did manage to find one webserver it succeeded on---that is --- a domain squatter with a typoed domain of one of my
favorite photographer's websites (the actual website should be

This squatter's site is indeed running lighttpd---but it appears to be
an earlier version, because the official lighttpd site fails on this

We have all the major web servers failing the test:
* Apache 1.3
* Apache 2.2
* Microsoft-IIS/6.0
* lighttpd/1.5.0

So I think it's the python side that is wrong, regardless of what the
standard is.

What should I do next?

I've rewritten the code to make it easier to test.  Just run it as is
and it will try all my test cases; or pass in a site on the command
line, and it will try just that.


#!/usr/bin/env python
"""Put the site you want to test as a command line parameter.
Otherwise tests the list of defaults."""

import urllib2
import zlib
import sys

opener = urllib2.build_opener()
opener.addheaders = [('Accept-encoding', 'deflate')]

    sites = [sys.argv[1]]
except IndexError:
    sites = ['', '',
             '', '',
             '', '']

for site in sites:
    print "Trying: ", site
    stream =
    data =
    encoded = stream.headers.get('Content-Encoding')
    server = stream.headers.get('Server')

    print "  %s - %s (%s)" % (site, server, encoded)

    if encoded == 'deflate':
        before = len(data)
            data = zlib.decompress(data)
            after = len(data)
            print "  Able to decompress...went from %i to %i." %
(before, after)
        except zlib.error:
            print "  Errored out on this site."
        print "  Data is not deflated."

More information about the Python-list mailing list