[Chicago] capturing output from subprocesses
Noel Thomas Taylor
nttaylor at uchicago.edu
Thu Nov 10 19:35:09 CET 2005
The program should never be running more than a single child-process at a
On Thu, 10 Nov 2005, Jess Balint wrote:
> That's a valid concern. Initially there will be an increase in memory.
> On most systems, this is pretty efficient and all parent process
> memory is copy-on-write in the child. However, since you exec() so
> soon the memory will be replaced by that of the image of the other
> program. So in effect the end result when you run the other program is
> no more memory than system() (or possibly even less due to no /bin/sh
> overhead if it's not a shell script). How many processes do predict
> you will be running at one time?
> On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>> Hi Jess,
>> Thanks for this great example. I've been experimenting with it, and it
>> could be the answer to my prayers. One question about forking: I know that
>> when you do a fork() call, the code gets duplicated in memory, the child
>> gets its own pid as the return value of the fork, and the parent gets
>> But how much code gets duplicated, and can a single fork call
>> significantly impact your memory? In your "child_process.py" for
>> example, does the whole module get duplicated? If this function were
>> just one in a giant file thousands of lines long, would that whole file
>> get duplicated? If your code has a call to fork() in it, does that mean
>> you should isolate it into a smaller module which you then import, or does
>> that make a difference?
>> Or maybe the duplication is virtual and the two processes are really
>> occupying the same memory space?
>> I'm sorry I can't make the meeting tonight.
>> with thanks,
>> Noel Taylor
>> On Tue, 8 Nov 2005, Jess Balint wrote:
>>> I made a prototype you can use. It's a simple combination of creating
>>> pipe()s for stdin and stderr, then dup2 them into the streams after a
>>> fork(), before an exec(). I've attach the python and a test shell
>>> script (that kills itself). You should be able to see it capture the
>>> output. (If there is a problem with the attachments, I will put them
>>> on a web site or something.)
>>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>>>> Noel Thomas Taylor wrote:
>>>>> Hi Ian,
>>>>> I could try that, but in the case of the real application whose output I
>>>>> want to capture, I have no control over how much output it produces.
>>>> I thought it would be an interesting test to understand exactly what is
>>>> going on, even if it isn't exactly what you are expecting to receive.
>>>>> Do you have any thoughts about recapturing the output of an aborted child
>>>>> process before the memory that is buffering its output gets blown away?
>>>> Since it is OS buffering, shouldn't the OS handle that for you? I don't
>>>> know; I would suggest starting with a test, then finding something that
>>>> passes that test.
>>>> Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
>>>> Chicago mailing list
>>>> Chicago at python.org
>> Chicago mailing list
>> Chicago at python.org
> Chicago mailing list
> Chicago at python.org
More information about the Chicago