
On 11:37 pm, clemesha@gmail.com wrote:
Would you say that this method of using a separate process instead of threads to do work also is possibly good for database operations?
Sure.
I'm always reading 'beware of threads' ;) with respect to Twisted, so using a very simple control protocol to manage a separate dedicate database process might be better than relying on the threading that 'adbapi' uses?
It would definitely work. The main reason adbapi itself isn't done this way is that setting up a process pool is unfortunately more work than spawning a thread - you have to decide on a proper set of environment variables, locate the python interpreter, locate the script that you're going to run, make sure that script is installed by setup.py, decide on a control protocol, wait for the subprocess to start up before sending it messages, shut down the process appropriately, catch process termination and restart (or not), etc etc etc. Twisted should really have a nice API that does all that stuff for you, though, and it's a shame that it doesn't. The other reason is that adbapi is old. If we were going to implement something like ADBAPI today, we'd probably write a process pool first, but adbapi was written as a quick hack to get some database integration a long time ago. The only caveat is that additional Python interpreters take up more RAM than additional threads. If the database processes are doing any heavy CPU lifting though, this cost could well be worth it.