[Chicago] capturing output from subprocesses

[Noel Thomas Taylor]

Hi Jess,

Thanks for this great example. I've been experimenting with it, and it 
could be the answer to my prayers. One question about forking: I know that 
when you do a fork() call, the code gets duplicated in memory, the child 
gets its own pid as the return value of the fork, and the parent gets 
nothing.

But how much code gets duplicated, and can a single fork call 
significantly impact your memory? In your "child_process.py" for 
example, does the whole module get duplicated? If this function were 
just one in a giant file thousands of lines long, would that whole file 
get duplicated? If your code has a call to fork() in it, does that mean 
you should isolate it into a smaller module which you then import, or does 
that make a difference?

Or maybe the duplication is virtual and the two processes are really 
occupying the same memory space?

I'm sorry I can't make the meeting tonight.

with thanks,

Noel Taylor



On Tue, 8 Nov 2005, Jess Balint wrote:

> I made a prototype you can use. It's a simple combination of creating
> pipe()s for stdin and stderr, then dup2 them into the streams after a
> fork(), before an exec(). I've attach the python and a test shell
> script (that kills itself). You should be able to see it capture the
> output. (If there is a problem with the attachments, I will put them
> on a web site or something.)
>
> Jess
>
> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>> Noel Thomas Taylor wrote:
>>> Hi Ian,
>>>
>>> I could try that, but in the case of the real application whose output I
>>> want to capture, I have no control over how much output it produces.
>>
>> I thought it would be an interesting test to understand exactly what is
>> going on, even if it isn't exactly what you are expecting to receive.
>>
>>> Do you have any thoughts about recapturing the output of an aborted child
>>> process before the memory that is buffering its output gets blown away?
>>
>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>> know; I would suggest starting with a test, then finding something that
>> passes that test.
>>
>> --
>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
>