managing stdout and stderr neatly while doing parallel processing

[Jeff Epler]

Well, the problem is that stderr messages are still going the same
place as before (eg the terminal) instead of being redirected by the
popen().  You could use "exec 2>&1;" (if my shell-fu doesn't fail me,
anyway) at the beginning of your popen command to merge them and read
both from the file returned by popen.

However, even when you use the 2>&1 trick, stdio buffering can reverse
the order between stdout and stderr messages:
$ (exec 2>&1; python -c 'import sys; sys.stdout.write("hi there\n"); sys.stderr.write("this is the error, written second\n");') | cat
this is the error, written second
hi there

I'm not sure what tricks to use to fix this problem, and my knowledge of
how Unix works gets iffy.  Getting output to be char- or line-buffered
by stdio in the subprocess would seem to be the ticket, but I don't how
to do this for programs that don't support it (python has 'python -u' to
do it).  If you set things up to run in a pty, that should get you
terminal-like behavior, including the "expected" ordering of messages
from a single process, but here both my Unix and Python knowledge fail
me.

good luck,
Jeff