I wonder why you decided to use Ampoule to manage a single child process  (that apparently shouldn't be restarted when it exits)?  Mostly because Ampoule handles quite a bit of stuff behind the scenes. I didn’t want to use it per se—to not add extra complexity to the stack, chiefly—but it was the only way I could get it to work. It also gives me a very nice AMP interface which I am already familiar with, and that allows me to communicate with my frontend very liberally. I think Ampoule's purpose is to make it easy to run many processes and  automatically restart them from time to time.  I agree. Though it’s also very useful as an drop–in implementation of a Twisted spawnProcess for one single other process. It sounds like the basic process support included in Twisted,  reactor.spawnProcess, could address your requirements.  I have just found your answer here:  https://stackoverflow.com/questions/6678763/simple-example-with-spawnprocess which seems useful. If nothing else works, I will try to reduce complexity by converting this to bare spawnProcess.  Any ideas on what might be going wrong in my current stack? After all, processes being repeatedly created is just a symptom of something going wrong in the process very early, so ampoule tries to spawn another one. I am mainly looking to find a way to see the exception and where it happens. Twisted’s logging module seem to kick in too late.  Best, Burak From: exarkun@twistedmatrix.com exarkun@twistedmatrix.com Reply: Twisted general discussion twisted-python@twistedmatrix.com Date: March 3, 2014 at 4:21:42 AM To: Twisted general discussion twisted-python@twistedmatrix.com Subject:  Re: [Twisted-Python] Freezing Ampoule or the Invasion of Infinite Zombie Processes On 06:34 am, burak@nehbit.net wrote:

Hi everyone,

It seems that I ran into a rather inextricable problem, would love to see what you think.

As you might remember, I have been working on a distributed, anonymous community platform(s) called Aether ( www.getaether.net ) which is largely a Twisted codebase in its networking system. After releasing the app, I ended up with performance problems, in some part due to my programming incompetence, and some part because I was also using it to drive a GUI with qtreactor, which ran its own Javascript interpreter.

The codebase is largely cleaned up now and the current architecture of the program involves two processes that talk to each other via Ampoule: a default reactor core, which handles network events, and a qtreactor which is solely concerned with the user interface. It works very well. Stress induced by networking does not slow down the UI and vice versa.

I wonder why you decided to use Ampoule to manage a single child process (that apparently shouldn't be restarted when it exits)? I think Ampoule's purpose is to make it easy to run many processes and automatically restart them from time to time. It sounds like the basic process support included in Twisted, reactor.spawnProcess, could address your requirements. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

spawnProcess is somewhat low-level.  It would be very nice indeed if Twisted had some sort of Service object you could easily use for spawning worker processes, along the lines of Ampoule, that would take care of all the platform-specific nonsense of setting up communication with subprocesses, spawning them, monitoring them, figuring out the right Python to spawn, the right environment variables to get sys.path correct, and other things, such as... Ampoule is very close. Any chance of polishing and absorbing it into Twisted? Parallel computing is the weakest point of entire Python ecosystem, and Ampoule is so very close to the perfection, unless, like me, you end up getting stuck on a corner case.  For this, I am almost entirely sure that the imports are failing, most probably because it’s trying to import the system python, not the one PyInstaller packages up. I have no idea how to fix it though. I’m feeling rather guilty about it, Aether v1.1 is a vastly improved version, but  I can’t package it, so the release is off for a while. For personal reasons (Moved to SF, started a new full–time job) I am unlikely to find enough time to debug such a complex issue in the near future, while the code is fading from my memory and rotting in drive. Such a shame.   Best, Burak From: Glyph Lefkowitz glyph@twistedmatrix.com Reply: Twisted general discussion twisted-python@twistedmatrix.com Date: March 10, 2014 at 11:39:36 AM To: Twisted general discussion twisted-python@twistedmatrix.com Subject:  Re: [Twisted-Python] Freezing Ampoule or the Invasion of Infinite Zombie Processes On Mar 9, 2014, at 12:31 AM, Burak Nehbit <burak@nehbit.net> wrote: I agree. Though it’s also very useful as an drop–in implementation of a Twisted spawnProcess for one single other process. spawnProcess is somewhat low-level.  It would be very nice indeed if Twisted had some sort of Service object you could easily use for spawning worker processes, along the lines of Ampoule, that would take care of all the platform-specific nonsense of setting up communication with subprocesses, spawning them, monitoring them, figuring out the right Python to spawn, the right environment variables to get sys.path correct, and other things, such as... It sounds like the basic process support included in Twisted,  reactor.spawnProcess, could address your requirements.  I have just found your answer here:  https://stackoverflow.com/questions/6678763/simple-example-with-spawnprocess which seems useful. If nothing else works, I will try to reduce complexity by converting this to bare spawnProcess.  Any ideas on what might be going wrong in my current stack? After all, processes being repeatedly created is just a symptom of something going wrong in the process very early, so ampoule tries to spawn another one. I am mainly looking to find a way to see the exception and where it happens. Twisted’s logging module seem to kick in too late.  Twisted's logging typically gets initialized after all modules are imported.  A tool like Ampoule really ought to set up logging as early as possible in its child, and handle the case where it's not getting set up on the parent side. The important thing to do is to make sure you have an errReceived callback that actually prints or logs the stderr it receives; there's likely a traceback in there which you may have missed if the process is just crashing. -glyph - smime.p7s, 6 KB _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Mar 12, 2014, at 4:32 AM, exarkun@twistedmatrix.com wrote:

Considering the amount of maintenance Ampoule seems to have gotten over the last four years (~1 commit) I doubt you can count on someone else picking this up just because it would be great, though.

The licenses are compatible and the maintainer is amenable to the idea though, so if you (and by "you" I mean Burak, not exarkun) wanted to start the work of merging in Ampoule, you could go right ahead. The one caveat I will give you is that Ampoule is probably not quite good enough as-is. It's been a while, but I definitely remember discussing some of its deficiencies with people who had used it; you will probably get requests to change stuff in code review. That's OK though, that's the whole point of code review. This actually means that the fact that it's unmaintained is a good thing - not a lot of duplication of effort to improve it :). -glyph