Deflate with urllib2... (solved)

Gabriel Genellina gagsl-py2 at
Fri Sep 19 17:25:09 CEST 2008

En Thu, 18 Sep 2008 23:29:30 -0300, Sam <samslists at> escribió:

> On Sep 18, 2:10 pm, "Gabriel Genellina" <gagsl-... at>
> wrote:
>> En Tue, 16 Sep 2008 21:58:31 -0300, Sam <samsli... at> escribió:
>> The code is correct - try with another server. I tested it with a  
>> LightHTTPd server and worked fine.
> Gabriel...
> I found a bunch of servers to test it on.  It fails on every server I
> could find (sans one).
> Here's the ones it fails on:
> I did manage to find one webserver it succeeded on---that is
> --- a domain squatter with a typoed domain of one of my
> favorite photographer's websites (the actual website should be
> This squatter's site is indeed running lighttpd---but it appears to be
> an earlier version, because the official lighttpd site fails on this
> test.
> We have all the major web servers failing the test:
> * Apache 1.3
> * Apache 2.2
> * Microsoft-IIS/6.0
> * lighttpd/1.5.0
> So I think it's the python side that is wrong, regardless of what the
> standard is.

I've found the problem. The zlib header is missing (2 bytes), data begins  
right with the compressed stream. You may decode it if you pass a negative  
value for wsize:

         data = zlib.decompress(data)
       except zlib.error:
         data = zlib.decompress(data, -zlib.MAX_WBITS)

Note that this is clearly in violation of RFC 1950: the header is *not*  

BTW, the curl developers had this same problem some time ago  
<> and the proposed solution  
is the same as above.

This is the output from your test script modified as above. (Note that in  
some cases, the compressed stream is larger than the uncompressed data):

Trying: - Apache/1.3.41 (Unix) mod_perl/1.31-rc4 (deflate)  
late)=73174 len(gzip)=73208
   Able to decompress...went from 73174 to 73073.

Trying: - Microsoft-IIS/6.0 (deflate) len(deflate)=1609  
   Able to decompress...went from 1609 to 3969.

Trying: - Microsoft-IIS/6.0 (deflate) len(deflate)=40646  
   Able to decompress...went from 40646 to 157141.

Trying: - Apache/2.2.8 (Unix) PHP/5.2.5 (deflate)  
862 len(gzip)=52880
   Able to decompress...went from 52862 to 52786.

Trying: - lighttpd/1.5.0 (deflate) len(deflate)=5669  
   Able to decompress...went from 5669 to 15746.

Trying: - lighttpd (deflate) len(deflate)=414  
   Able to decompress...went from 414 to 744.

Gabriel Genellina

More information about the Python-list mailing list