[Twisted-Python] Processes in Twisted, only possible to run external prog?
Hello everyone, Here's the problem : I need to run a windows executable (proprietary, they won't give us the code) with wine to do some audio compression. I need to do different processing before (like calling sox, or parsing files, etc) so I wrote some functions that do everything. The windows exec creates a tmp file named "_tmp" in the working dir, so if I want to run several in parallel, I have to create a tmp dir with a unique name and change the working dir, if not then the multiple executions will overwrite each other's tmp file. So when I call my function that does the whole process, I first create a tmp dir, then pass it the tmp dir and the input filename, then once inside the function, it does an os.chDir() into the tmp dir and at the end it does it again to go back to the original current dir. Then I run that function in threads. The problem is that threads share the current dir : http://www.biais.org/blog/index.php/2007/01/23/19-python-threads-and-oschdir http://bugs.python.org/issue1367 so I need to use processes instead, as they each have their own current dir. The thing is that from what I understand, Twisted's process API is intended to be used with external executables only, correct? I wasn' t able to fins anything that allowed me to do stuff like I would using fork(). Should I just use python's fock() instead? Is it safe with Twisted? Thank you, Gabriel
On Friday 29 August 2008, Gabriel Rossetti wrote:
Here's the problem : I need to run a windows executable (proprietary, they won't give us the code) with wine to do some audio compression. I need to do different processing before (like calling sox, or parsing files, etc) so I wrote some functions that do everything. The windows exec creates a tmp file named "_tmp" in the working dir, so if I want to run several in parallel, I have to create a tmp dir with a unique name and change the working dir, if not then the multiple executions will overwrite each other's tmp file. So when I call my function that does the whole process, I first create a tmp dir, then pass it the tmp dir and the input filename, then once inside the function, it does an os.chDir() into the tmp dir and at the end it does it again to go back to the original current dir. Then I run that function in threads. The problem is that threads share the current dir :
Maybe you could bypass this problem by generating a short shell script that does "cd" before it starts Wine? Since the shell will be a separate process, it has its own working dir. Bye, Maarten
Maarten ter Huurne wrote:
On Friday 29 August 2008, Gabriel Rossetti wrote:
Here's the problem : I need to run a windows executable (proprietary, they won't give us the code) with wine to do some audio compression. I need to do different processing before (like calling sox, or parsing files, etc) so I wrote some functions that do everything. The windows exec creates a tmp file named "_tmp" in the working dir, so if I want to run several in parallel, I have to create a tmp dir with a unique name and change the working dir, if not then the multiple executions will overwrite each other's tmp file. So when I call my function that does the whole process, I first create a tmp dir, then pass it the tmp dir and the input filename, then once inside the function, it does an os.chDir() into the tmp dir and at the end it does it again to go back to the original current dir. Then I run that function in threads. The problem is that threads share the current dir :
Maybe you could bypass this problem by generating a short shell script that does "cd" before it starts Wine? Since the shell will be a separate process, it has its own working dir.
Bye, Maarten
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script. I have stuff to do before the conversion,preparations, those could be done in the script, and after : parsing output, filter and modify the data & then write the new data in binary form -- which is harder to do from a non-python shell script. Maybe I could use a standalone python shell script? Is there no other solution? bye, Gabriel
On Friday 29 August 2008, Gabriel Rossetti wrote:
Maarten ter Huurne wrote:
Maybe you could bypass this problem by generating a short shell script that does "cd" before it starts Wine? Since the shell will be a separate process, it has its own working dir.
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script.
Do the other processing steps also require the working directory to be changed? Most unix programs work fine from any dir as long as you provide absolute paths for the input and output files. Bye, Maarten
Maarten ter Huurne wrote:
On Friday 29 August 2008, Gabriel Rossetti wrote:
Maarten ter Huurne wrote:
Maybe you could bypass this problem by generating a short shell script that does "cd" before it starts Wine? Since the shell will be a separate process, it has its own working dir.
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script.
Do the other processing steps also require the working directory to be changed? Most unix programs work fine from any dir as long as you provide absolute paths for the input and output files.
Bye, Maarten
Hmmm, good question, let me investigate that, I'll keep you updated. Bye, Gabriel
Maarten ter Huurne wrote:
On Friday 29 August 2008, Gabriel Rossetti wrote:
Maarten ter Huurne wrote:
Maybe you could bypass this problem by generating a short shell script that does "cd" before it starts Wine? Since the shell will be a separate process, it has its own working dir.
I'd rather not, because that would mean implementing everything (the whole conversion process) in the script.
Do the other processing steps also require the working directory to be changed? Most unix programs work fine from any dir as long as you provide absolute paths for the input and output files.
Bye, Maarten
Maarten, It worked, I was able to do the pre & post processing using abs paths (thus from outside of the working dir) and the encoding was done from a script that changed the working directory. Thank you, Gabriel
On 09:06 am, gabriel.rossetti@arimaz.com wrote:
http://www.biais.org/blog/index.php/2007/01/23/19-python-threads-and- oschdir http://bugs.python.org/issue1367
so I need to use processes instead, as they each have their own current dir. The thing is that from what I understand, Twisted's process API is intended to be used with external executables only, correct? I wasn' t able to fins anything that allowed me to do stuff like I would using fork(). Should I just use python's fock() instead? Is it safe with Twisted?
As it happens, "python" is an "external executable". Here's an example of spawning it in such a way as to make sure that sys.path is set properly: http://divmod.org/trac/browser/trunk/Epsilon/epsilon/process.py#L7 However, Wine is also an external process. Perhaps you just want to spawn the Wine command-line using spawnProcess using the 'path' parameter set to a directory that you have created in the main process, then clean it up when the process exits? http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IRe... Using fork() in Python has a lot of confusing issues associated with it unless you're going to exec() immediately afterwards. You probably want to stick to one of these other approaches.
On Fri, 29 Aug 2008 13:35:24 -0000, glyph@divmod.com wrote:
On 09:06 am, gabriel.rossetti@arimaz.com wrote:
[snip] Should I just use python's fock() instead? Is it safe with Twisted?
[snip]
Using fork() in Python has a lot of confusing issues associated with it unless you're going to exec() immediately afterwards. You probably want to stick to one of these other approaches.
You also need to do some other stuff before you call fork(), like close file descriptors and disable garbage collection. And then re-enable it in the parent after the fork(). There's probably even some more things that are necessary that no one has realized yet. Jean-Paul
glyph@divmod.com wrote:
On 09:06 am, gabriel.rossetti@arimaz.com wrote:
http://www.biais.org/blog/index.php/2007/01/23/19-python-threads-and- oschdir http://bugs.python.org/issue1367
so I need to use processes instead, as they each have their own current dir. The thing is that from what I understand, Twisted's process API is intended to be used with external executables only, correct? I wasn' t able to fins anything that allowed me to do stuff like I would using fork(). Should I just use python's fock() instead? Is it safe with Twisted?
As it happens, "python" is an "external executable". Here's an example of spawning it in such a way as to make sure that sys.path is set properly: http://divmod.org/trac/browser/trunk/Epsilon/epsilon/process.py#L7
Ok, thanks
However, Wine is also an external process. Perhaps you just want to spawn the Wine command-line using spawnProcess using the 'path' parameter set to a directory that you have created in the main process, then clean it up when the process exits?
http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IRe...
mmmm, yes, that sounds nice, I'll try that, thanks!
Using fork() in Python has a lot of confusing issues associated with it unless you're going to exec() immediately afterwards. You probably want to stick to one of these other approaches.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (4)
-
Gabriel Rossetti -
glyph＠divmod.com -
Jean-Paul Calderone -
Maarten ter Huurne