popen2() gives broken pipe for large files - is it fixed yet?

[Frank Gibbons]

Hi,

I'm trying to allow users of my CGI script to download rather large data 
files by clicking a button. The solution which worked for small test files 
was this:

- Create a button whose action is to print the formatted data with the 
appropriate content-type header, so that the desired application will open.
- Create another button whose action is to open a pipe to gzip (using 
popen2()), write the same data into the pipe, then write the output from 
the pipe to the browser, again with appropriate content-type, to say that 
it's a zipped file.

It worked fine for a small test file, but I get a broken pipe for realistic 
sizes (1MB). I've written a little test script that shows the same behavior 
(see below). I've searched the newsgroups, and find that popen2() has a 
history of breaking the pipe on large files. Does anyone know if this has 
been fixed? Is popen3 any better? I've also read that the os module now has 
a popen function (as of Python 2.x, forget which x). Is this any better?

TIA,

Frank Gibbons

#!/usr/bin/python

import os
import popen2
import string
GZIP = "/bin/gzip"
filename = "ChIP_Squared_Template_FR2.txt"
f = open(filename)
data = f.readlines()
f.close()

print "Read %d lines" % len(data)
zip_output, zip_input = popen2.popen2("%s -c  "% GZIP)
data = string.join(data, "")
print "LEN(data) = %d" % len(data)
zip_input.write(data)
#zip_input.close()

zipfile = open("popen.gz", "w+")
zipfile.write(zip_output.read())
zipfile.flush()
zipfile.close()

PhD, Computational Biologist,
Harvard Medical School BCMP/SGM-322, 250 Longwood Ave, Boston MA 02115, USA.
Tel: 617-432-3555       Fax: 
617-432-3557       http://llama.med.harvard.edu/~fgibbons