popen2 with large input

[Eric Brunel]

cherico wrote:
> from popen2 import popen2
> 
> r, w = popen2 ( 'tr "[A-Z]" "[a-z]"' )
> w.write ( t ) # t is a text file of around 30k bytes
> w.close ()
> text = r.readlines ()
> print text
> r.close ()
> 
> This simple script halted on 
> 
> w.write ( t )
> 
> Anyone knows what the problem is?

Yep: deadlock... Pipes are synchronized: you can't read from (resp. write to) a 
pipe if the process at the other end does not write to (resp. read from) it. If 
you try the command "tr '[A-Z]' '[a-z]'" interactively, you'll see that 
everytime tr receives a line, it outputs *immediately* the converted line. So if 
you write a file having several lines to the pipe, on the first \n, tr will try 
to write to its output, and will be stuck since your program is not reading from 
it. So it won't read on its input anymore, so your program will be stuck because 
it can't write to the pipe. And they'll wait for each other until the end of 
times...

If you really want to use the "tr" command for this stuff, you'd better send 
your text lines by lines and read the result immediatly, like in:

text = ''
for line in text.splitlines(1):
   w.write(line)
   w.flush()   # Mandatory because of output bufferization - see below
   text += r.readline()
w.close()
r.close()

It *may* work better, but you cannot be sure: in fact, you just can't know 
exactly when tr will actually output the converted text. Even worse: since 
output is usually buffered, you'll only see the output from tr when its standard 
output is flushed, and you can't know when that will be...

(BTW, the script above does not work on my Linux box: the first r.readline() 
never returns...)

So the conclusion is: don't use pipes unless you're really forced to. They're a 
hell to use, since you never know how to synchronize them.

BTW, if the problem you posted is your real problem, why on earth don't you do:
text = t.lower()
???

HTH
-- 
- Eric Brunel <eric dot brunel at pragmadev dot com> -
PragmaDev : Real Time Software Development Tools - http://www.pragmadev.com