[Chicago] capturing output from subprocesses

Jess Balint jbalint at gmail.com
Thu Nov 10 18:18:29 CET 2005


That's a valid concern. Initially there will be an increase in memory.
On most systems, this is pretty efficient and all parent process
memory is copy-on-write in the child. However, since you exec() so
soon the memory will be replaced by that of the image of the other
program. So in effect the end result when you run the other program is
no more memory than system() (or possibly even less due to no /bin/sh
overhead if it's not a shell script). How many processes do predict
you will be running at one time?

Jess

On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>
> Hi Jess,
>
> Thanks for this great example. I've been experimenting with it, and it
> could be the answer to my prayers. One question about forking: I know that
> when you do a fork() call, the code gets duplicated in memory, the child
> gets its own pid as the return value of the fork, and the parent gets
> nothing.
>
> But how much code gets duplicated, and can a single fork call
> significantly impact your memory? In your "child_process.py" for
> example, does the whole module get duplicated? If this function were
> just one in a giant file thousands of lines long, would that whole file
> get duplicated? If your code has a call to fork() in it, does that mean
> you should isolate it into a smaller module which you then import, or does
> that make a difference?
>
> Or maybe the duplication is virtual and the two processes are really
> occupying the same memory space?
>
> I'm sorry I can't make the meeting tonight.
>
> with thanks,
>
> Noel Taylor
>
>
>
> On Tue, 8 Nov 2005, Jess Balint wrote:
>
> > I made a prototype you can use. It's a simple combination of creating
> > pipe()s for stdin and stderr, then dup2 them into the streams after a
> > fork(), before an exec(). I've attach the python and a test shell
> > script (that kills itself). You should be able to see it capture the
> > output. (If there is a problem with the attachments, I will put them
> > on a web site or something.)
> >
> > Jess
> >
> > On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
> >> Noel Thomas Taylor wrote:
> >>> Hi Ian,
> >>>
> >>> I could try that, but in the case of the real application whose output I
> >>> want to capture, I have no control over how much output it produces.
> >>
> >> I thought it would be an interesting test to understand exactly what is
> >> going on, even if it isn't exactly what you are expecting to receive.
> >>
> >>> Do you have any thoughts about recapturing the output of an aborted child
> >>> process before the memory that is buffering its output gets blown away?
> >>
> >> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
> >> know; I would suggest starting with a test, then finding something that
> >> passes that test.
> >>
> >> --
> >> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> >> _______________________________________________
> >> Chicago mailing list
> >> Chicago at python.org
> >> http://mail.python.org/mailman/listinfo/chicago
> >>
> >
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>


More information about the Chicago mailing list