[Twisted-Python] Processes in Twisted, only possible to run external prog?
![](https://secure.gravatar.com/avatar/275dbf027f6e280894a60f83a480e083.jpg?s=120&d=mm&r=g)
Hello everyone, Here's the problem : I need to run a windows executable (proprietary, they won't give us the code) with wine to do some audio compression. I need to do different processing before (like calling sox, or parsing files, etc) so I wrote some functions that do everything. The windows exec creates a tmp file named "_tmp" in the working dir, so if I want to run several in parallel, I have to create a tmp dir with a unique name and change the working dir, if not then the multiple executions will overwrite each other's tmp file. So when I call my function that does the whole process, I first create a tmp dir, then pass it the tmp dir and the input filename, then once inside the function, it does an os.chDir() into the tmp dir and at the end it does it again to go back to the original current dir. Then I run that function in threads. The problem is that threads share the current dir : http://www.biais.org/blog/index.php/2007/01/23/19-python-threads-and-oschdir http://bugs.python.org/issue1367 so I need to use processes instead, as they each have their own current dir. The thing is that from what I understand, Twisted's process API is intended to be used with external executables only, correct? I wasn' t able to fins anything that allowed me to do stuff like I would using fork(). Should I just use python's fock() instead? Is it safe with Twisted? Thank you, Gabriel
![](https://secure.gravatar.com/avatar/275dbf027f6e280894a60f83a480e083.jpg?s=120&d=mm&r=g)
Maarten ter Huurne wrote:
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script. I have stuff to do before the conversion,preparations, those could be done in the script, and after : parsing output, filter and modify the data & then write the new data in binary form -- which is harder to do from a non-python shell script. Maybe I could use a standalone python shell script? Is there no other solution? bye, Gabriel
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 09:06 am, gabriel.rossetti@arimaz.com wrote:
As it happens, "python" is an "external executable". Here's an example of spawning it in such a way as to make sure that sys.path is set properly: http://divmod.org/trac/browser/trunk/Epsilon/epsilon/process.py#L7 However, Wine is also an external process. Perhaps you just want to spawn the Wine command-line using spawnProcess using the 'path' parameter set to a directory that you have created in the main process, then clean it up when the process exits? http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IRe... Using fork() in Python has a lot of confusing issues associated with it unless you're going to exec() immediately afterwards. You probably want to stick to one of these other approaches.
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Fri, 29 Aug 2008 13:35:24 -0000, glyph@divmod.com wrote:
You also need to do some other stuff before you call fork(), like close file descriptors and disable garbage collection. And then re-enable it in the parent after the fork(). There's probably even some more things that are necessary that no one has realized yet. Jean-Paul
![](https://secure.gravatar.com/avatar/275dbf027f6e280894a60f83a480e083.jpg?s=120&d=mm&r=g)
Maarten ter Huurne wrote:
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script. I have stuff to do before the conversion,preparations, those could be done in the script, and after : parsing output, filter and modify the data & then write the new data in binary form -- which is harder to do from a non-python shell script. Maybe I could use a standalone python shell script? Is there no other solution? bye, Gabriel
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 09:06 am, gabriel.rossetti@arimaz.com wrote:
As it happens, "python" is an "external executable". Here's an example of spawning it in such a way as to make sure that sys.path is set properly: http://divmod.org/trac/browser/trunk/Epsilon/epsilon/process.py#L7 However, Wine is also an external process. Perhaps you just want to spawn the Wine command-line using spawnProcess using the 'path' parameter set to a directory that you have created in the main process, then clean it up when the process exits? http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IRe... Using fork() in Python has a lot of confusing issues associated with it unless you're going to exec() immediately afterwards. You probably want to stick to one of these other approaches.
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Fri, 29 Aug 2008 13:35:24 -0000, glyph@divmod.com wrote:
You also need to do some other stuff before you call fork(), like close file descriptors and disable garbage collection. And then re-enable it in the parent after the fork(). There's probably even some more things that are necessary that no one has realized yet. Jean-Paul
participants (4)
-
Gabriel Rossetti
-
glyph@divmod.com
-
Jean-Paul Calderone
-
Maarten ter Huurne