[IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process
MinRK
benjaminrk at gmail.com
Fri Oct 3 14:26:52 EDT 2014
You don’t need to read the whole output at once. You can get one line at a
time. Or, you can use pexpect <http://pexpect.readthedocs.org/en/latest/>
to create the subprocess, which offers nice timeout, pattern-expecting
waiting for output. Depending on context, IPython will use one or the other
of these for !.
-MinRK
On Fri, Oct 3, 2014 at 11:22 AM, Andy Davidson <
Andy at santacruzintegration.com> wrote:
> Hi MinRK
>
> I am new to Python system programming. In my first experiments I use
> subprocess. Reading the document it appears subprocess it great for a
> setting up traditional unix style pipe like for example calling “ls”
>
> In the notebook I aussem
> ! Uname –a
>
> Works using subprocess.
>
> The subprocess documentation suggests that you need to call communicate()
> and that this function will not return until the child sends EOF or
> terminates. My child will never exit or send EOF. it will keep sending me
> real time data. I want to process the data one line at a time. Here is my
> psudo code
>
> While (True):
> Data = readALineFromChild() # this should block until data becomes
> available
> Graph(data)
>
>
>
> Thanks
>
> Andy
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/ipython-dev/attachments/20141002/aefaa3fc/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Thu, 2 Oct 2014 17:19:18 -0700
> From: MinRK <benjaminrk at gmail.com>
> Subject: Re: [IPython-dev] ERROR can not use fork(), exec(), dup2() to
> communicate with child process
> To: IPython developers list <ipython-dev at scipy.org>
> Message-ID:
> <CAHNn8BXxCej6VL4XkjguOSXtgnxpsNCJXHhKGNLSBoYbiu48-g at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> why use execlp instead of subprocess.Popen, which allows capturing stdout,
> stderr, etc.?
> ?
>
> On Thu, Oct 2, 2014 at 4:17 PM, Andy Davidson <
> Andy at santacruzintegration.com
>
> wrote:
>
>
> Hi
>
>
> I want to write a notebook that receives data from another process. Bellow
> is my python code. It forks a child process and reads the data from the
> child using a pipe. This code works if I run it in python. When I try to
> execute from iPython notebook I get the following error. Any idea what the
> problems is ?
>
>
> If pipes do not work, can I use a socket? The other process would send me
> data by writing, my notebook needs behave like a server.
>
>
> Any comments or suggestions would be greatly appreciated.
>
>
> Thanks
>
>
> Andy
>
>
> P.s. I am running headless on a remote machine and use an ssh tunnel to
> run the notebook from my local browser. My other notebooks works fine
>
>
>
> [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline
> --no-browser --port=7000" /root/spark/bin/pyspark
>
> /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify
> limit: Operation not permitted
>
> [NotebookApp] Using existing profile dir:
> u'/home/ec2-user/.ipython/profile_default'
>
> [NotebookApp] Serving notebooks from /home/ec2-user
>
> [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/
>
> [NotebookApp] Use Control-C to stop this server and shut down all kernels.
>
> [NotebookApp] Using MathJax from CDN:
> http://cdn.mathjax.org/mathjax/latest/MathJax.js
>
> [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b
>
> [NotebookApp] Connecting to: tcp://127.0.0.1:54119
>
> [NotebookApp] Connecting to: tcp://127.0.0.1:50649
>
> [NotebookApp] Connecting to: tcp://127.0.0.1:35516
>
> [IPKernelApp] To connect another client to this kernel, use:
>
> [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json
>
> Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79)
>
> [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to
> respond to heartbeat
>
>
>
> import sys
> import os
>
> def child(pipeout):
> os.dup2(pipeout, sys.stdout.fileno())
> os.close(pipeout)
> os.execlp("./test.sh", "abc")
>
>
> def parent():
> pipein, pipeout = os.pipe()
> if os.fork() == 0:
> os.close(pipein)
> child(pipeout)
> else:
> os.close(pipeout)
> counter = 1
> pipein = os.fdopen(pipein)
> while True:
> json = pipein.readline()[:-1]
> print 'parent: %s' % (json)
>
> parent()
>
> $ cat test.sh
>
> #!/bin/bash
>
>
> script_name=$0
>
> args=$@
>
> current=`pwd`
>
> echo $script_name $args
>
> echo "XYZ: "$XYZ
>
> echo "PATH: "$PATH
>
> echo "current directory: $current"
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141003/e56ec670/attachment.html>
More information about the IPython-dev
mailing list