[Chicago] capturing output from subprocesses

[Noel Thomas Taylor]

The program should never be running more than a single child-process at a 
time.

On Thu, 10 Nov 2005, Jess Balint wrote:

> That's a valid concern. Initially there will be an increase in memory.
> On most systems, this is pretty efficient and all parent process
> memory is copy-on-write in the child. However, since you exec() so
> soon the memory will be replaced by that of the image of the other
> program. So in effect the end result when you run the other program is
> no more memory than system() (or possibly even less due to no /bin/sh
> overhead if it's not a shell script). How many processes do predict
> you will be running at one time?
>
> Jess
>
> On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>
>> Hi Jess,
>>
>> Thanks for this great example. I've been experimenting with it, and it
>> could be the answer to my prayers. One question about forking: I know that
>> when you do a fork() call, the code gets duplicated in memory, the child
>> gets its own pid as the return value of the fork, and the parent gets
>> nothing.
>>
>> But how much code gets duplicated, and can a single fork call
>> significantly impact your memory? In your "child_process.py" for
>> example, does the whole module get duplicated? If this function were
>> just one in a giant file thousands of lines long, would that whole file
>> get duplicated? If your code has a call to fork() in it, does that mean
>> you should isolate it into a smaller module which you then import, or does
>> that make a difference?
>>
>> Or maybe the duplication is virtual and the two processes are really
>> occupying the same memory space?
>>
>> I'm sorry I can't make the meeting tonight.
>>
>> with thanks,
>>
>> Noel Taylor
>>
>>
>>
>> On Tue, 8 Nov 2005, Jess Balint wrote:
>>
>>> I made a prototype you can use. It's a simple combination of creating
>>> pipe()s for stdin and stderr, then dup2 them into the streams after a
>>> fork(), before an exec(). I've attach the python and a test shell
>>> script (that kills itself). You should be able to see it capture the
>>> output. (If there is a problem with the attachments, I will put them
>>> on a web site or something.)
>>>
>>> Jess
>>>
>>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>>>> Noel Thomas Taylor wrote:
>>>>> Hi Ian,
>>>>>
>>>>> I could try that, but in the case of the real application whose output I
>>>>> want to capture, I have no control over how much output it produces.
>>>>
>>>> I thought it would be an interesting test to understand exactly what is
>>>> going on, even if it isn't exactly what you are expecting to receive.
>>>>
>>>>> Do you have any thoughts about recapturing the output of an aborted child
>>>>> process before the memory that is buffering its output gets blown away?
>>>>
>>>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>>>> know; I would suggest starting with a test, then finding something that
>>>> passes that test.
>>>>
>>>> --
>>>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>>>> _______________________________________________
>>>> Chicago mailing list
>>>> Chicago at python.org
>>>> http://mail.python.org/mailman/listinfo/chicago
>>>>
>>>
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>