Decompressing gzip over FTP

Albert Hopkins marduk at letterboxes.org
Sat Aug 22 10:59:55 EDT 2009


On Fri, 2009-08-21 at 18:15 -0700, SeanMon wrote:
> Is there a way to decompress a large (2GB) gzipped file being
> retrieved over FTP on the fly?
> 
> I'm using ftplib.FTP to open a connection to a remote server, and I
> have had no success connecting retrbinary to gzip without using an
> intermediate file.
> 
> Is there any way to get a file-like object describing the remote file,
> or a way to utilize gzip's decompression without providing it a file-
> like object?

I tried to solve your problem with StringIO and .truncate().  I also
tried to solve it with us.pipe(), but was unsuccessful with either.  The
problem is the gzip module uses .seek() which you can't do with pipes...

You could probably get away with just using the compress module (as zlib
does) but you will have to deal with handling the headers, footers, CRC
checks, etc. on your own.

The cheap alternative is to just have your script open a pipe to "gunzip
-c" and read/write from that pipe.

hth,
-a





More information about the Python-list mailing list