Python module to: Talk to stdout/stdin for 64-bit Windows/Linux C++ application?
Hi, Background / Goal: My team's application does regressions by controlling our 64-bit Windows/Linux C++ application (SimNow, www.amd.com/simnow) via stdout/stdin with ("pexpect" for unix) and (python COM objects) for Windows (b/c popen doesn't work). However, I am trying to replace this with a unified windows/linux solution. The idea is to extend Python with a SimNow-control module. My sole goal in extending a python module is so our python regressions scripts can use python to control SimNow via SimNow's stdin/stdout interface. Sample Final results: import simnow simnow.start() # start simnow executable output1 = simnow.sendCommand("load platform x") # blocking call, sends simnow cmd, returns the output string Disutils: Does extending python with Disutils sound like the simple and clean way for my goal? Is there a simple cookbook solution for my goal? (Btw, someone on my team already embedded python into SimNow. But I think we actually want to extend python, so to control SimNow from our python regressions code/infrastructure.) --Thanks, any response is appreciated! Peter Mowry ("Pem") SimNow Team, Software Engineer AMD Organization
-->"Peter" == Mowry, Peter <peter.mowry@amd.com> writes: Peter> My team's application does regressions by controlling our Peter> 64-bit Windows/Linux C++ application (SimNow, Peter> www.amd.com/simnow) via stdout/stdin with ("pexpect" for unix) Peter> and (python COM objects) for Windows (b/c popen doesn't work). Peter> However, I am trying to replace this with a unified Peter> windows/linux solution. have you looked at the subprocess module, part of the standard library in Python 2.4 and later? Peter> Does extending python with Disutils sound like the simple and Peter> clean way for my goal? Is there a simple cookbook solution for Peter> my goal? i don't think distutils is what you're after. distutils is about producing installers for Python applications. d
David, The subprocess module doesn't work for interactive programs (I think) (see appended pexpect faq). So instead, I was trying to get something like the following working: import simnow # import complex simnow C++ application simnow.start() # spawn simnow's main method output1 = simnow.sendCommand("load platform x") # blocking call to a single simnow C++ functions # This is our real current goal, but if I get this working, we might consider exposing other simnow functionality to the simnow python module. disutils is listed in the extend/embed python doc (http://www.python.org/doc/ext/ext.html). This doc seems to be saying that disutils is a cross-plaform way to build a python module ("extend python"). Chapter 3 - disultils cross-platform build. Chapter 4 - do it yourself windows build. Chapter 5 - do it yourself unix build. Chapter 4 says, "Module authors are encouraged to use the distutils approach for building extension modules, instead of the one described in this section." --Thanks subprocess - "appended comments" # This doesn't work self.simnow = Popen(executable=(os.getcwd() + '/simnow'), args=simnowCmd, stdin=PIPE, stdout=PIPE, stderr=PIPE) self.simnow.stdin.write("open stuff") print self.simnow.stdout.read() Why doesn't it work? The following is from the pexect FAQ Q: Why not just use a pipe (popen())? A: A pipe works fine for getting the output to non-interactive programs. If you just want to get the output from ls, uname, or ping then this works. Pipes do not work very well for interactive programs and pipes will almost certainly fail for most applications that ask for passwords such as telnet, ftp, or ssh. There are two reasons for this. First an application may bypass stdout and print directly to its controlling TTY. Something like SSH will do this when it asks you for a password. This is why you cannot redirect the password prompt because it does not go through stdout or stderr. The second reason is because most applications are built using the C Standard IO Library (anything that uses #include <stdio.h>). One of the features of the stdio library is that it buffers all input and output. Normally output is line buffered when a program is printing to a TTY (your terminal screen). Everytime the program prints a line-feed the currently buffered data will get printed to your screen. The problem comes when you connect a pipe. The stdio library is smart and can tell that it is printing to a pipe instead of a TTY. In that case it switches from line buffer mode to block buffered. In this mode the currently buffered data is flushed when the buffer is full. This causes most interactive programs to deadlock. Block buffering is more efficient when writing to disks and pipes. Take the situation where a program prints a message "Enter your user name:\n" and then waits for you type type something. In block buffered mode, the stdio library will not put the message into the pipe even though a linefeed is printed. The result is that you never receive the message, yet the child application will sit and wait for you to type a response. Don't confuse the stdio lib's buffer with the pipe's buffer. The pipe buffer is another area that can cause problems. You could flush the input side of a pipe, whereas you have no control over the stdio library buffer. http://pexpect.sourceforge.net/#faq <http://pexpect.sourceforge.net/%23faq> What's wrong with pexpect? It doesn't work on Windows. Python COM is an alternative, but we are trying to find a clean unified cross-platform solution. Peter Mowry ("Pem") SimNow Team, Software Engineer AMD Organization -----Original Message----- From: David Arnold [mailto:david@mantara.com] Sent: Saturday, May 12, 2007 10:00 PM To: Mowry, Peter Cc: distutils-sig@python.org Subject: Re: [Distutils] Python module to: Talk to stdout/stdin for 64-bit Windows/Linux C++ application? -->"Peter" == Mowry, Peter <peter.mowry@amd.com> writes: Peter> My team's application does regressions by controlling our Peter> 64-bit Windows/Linux C++ application (SimNow, Peter> www.amd.com/simnow) via stdout/stdin with ("pexpect" for unix) Peter> and (python COM objects) for Windows (b/c popen doesn't work). Peter> However, I am trying to replace this with a unified Peter> windows/linux solution. have you looked at the subprocess module, part of the standard library in Python 2.4 and later? Peter> Does extending python with Disutils sound like the simple and Peter> clean way for my goal? Is there a simple cookbook solution for Peter> my goal? i don't think distutils is what you're after. distutils is about producing installers for Python applications. d
Mowry, Peter wrote:
David,
The subprocess module doesn't work for interactive programs (I think) (see appended pexpect faq).
So instead, I was trying to get something like the following working:
import simnow # import complex simnow C++ application
simnow.start() # spawn simnow's main method
output1 = simnow.sendCommand("load platform x") # blocking call to a single simnow C++ functions
# This is our real current goal, but if I get this working, we might consider exposing other simnow functionality to the simnow python module.
disutils is listed in the extend/embed python doc (http://www.python.org/doc/ext/ext.html). This doc seems to be saying that disutils is a cross-plaform way to build a python module ("extend python").
Distutils is kind of like "make" for Python. It's there to help you build and distribute Python modules, but that's basically all it does. There's nothing in distutils to help with cross-platform process IO. Based on the pexpect page it looks like it should work with Cygwin on Windows, so you could consider using that instead. -- Matt Good
Ever seen this error? Build Error Description: python2.5 setup.py demo.C build That creates a demo.so, and when I try to import it, it gives me the error: ImportError: ./demo.so: undefined symbol: __gxx_personality_v0 Hack to fix it: I found one single helpful post about this on google: http://www.intevation.de/pipermail/thuban-list/2003-September/000230.htm l suggests: Do the link step of gdalwarp manually by copying the commandline printed by setup.py and replacing the gcc with g++ So I tried this, and I saw that setup.py prints the following 2 commands: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wstrict-prototypes -fpermissive -fPIC -I/tool/pandora64/.package/python-2.5/include/python2.5 -c demo.C -o build/temp.linux-x86_64-2.5/demo.o gcc -pthread -shared build/temp.linux-x86_64-2.5/demo.o -o build/lib.linux-x86_64-2.5/demo.so Then I simply manually rerun just the second command with g++: g++ -pthread -shared build/temp.linux-x86_64-2.5/demo.o -o build/lib.linux-x86_64-2.5/demo.so And now the import works and so does the python spam_system() function. Non-Hack to fix it?: Is there a non-hack way to fix this? I've seen this question on the internet but no answers so far. I did check --help-compiler, and it only lists --compiler=unix, not --compiler=g++ for ex. --Thanks Peter Mowry ("Pem") SimNow Team, Software Engineer AMD Organization
participants (3)
-
David Arnold -
Matt Good -
Mowry, Peter