[Twisted-Python] Correct pattern to do SSH forwarding in a GUI
I am using twisted in a GUI application to do port forwarding over an SSH connection. All the network connections of the application (database, etc...) are routed through this connection, to simplify server configuration and secure things. My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong. So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ?
On 04:19 am, luper.rouch@gmail.com wrote:
My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong.
These errors sound like you are making Twisted API calls from threads other than the reactor thread by accident. There's no obvious fix except for "don't do that" :). My suggestion would be to move the SQLAlchemy/psycopg2 into threads, with e.g. deferToThread, rather than putting the reactor into a thread. Then the reactor won't freeze while the blocking SQL stuff happens.
So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ?
There's no reason that running the reactor in a thread shouldn't work; it's just generally tricky (especially if you're working with an existing codebase) to completely avoid calling Twisted APIs from other threads.
glyph@divmod.com wrote:
On 04:19 am, luper.rouch@gmail.com wrote:
My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong.
These errors sound like you are making Twisted API calls from threads other than the reactor thread by accident. There's no obvious fix except for "don't do that" :).
Yes, I did not pay attention to separate Twisted from the main thread at all, apart from putting reactor.run() in another thread.
My suggestion would be to move the SQLAlchemy/psycopg2 into threads, with e.g. deferToThread, rather than putting the reactor into a thread. Then the reactor won't freeze while the blocking SQL stuff happens.
So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ?
There's no reason that running the reactor in a thread shouldn't work; it's just generally tricky (especially if you're working with an existing codebase) to completely avoid calling Twisted APIs from other threads.
This implies too much modifications in my code, I think I will put the whole Twisted stuff in a separate process and interact with it via simple IPC, as I don't use that much of Twisted in the GUI code anyway.
On 04:19 am, luper.rouch@gmail.com wrote:
My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/ psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong.
These errors sound like you are making Twisted API calls from threads other than the reactor thread by accident. There's no obvious fix except for "don't do that" :).
My suggestion would be to move the SQLAlchemy/psycopg2 into threads, with e.g. deferToThread, rather than putting the reactor into a thread. Then the reactor won't freeze while the blocking SQL stuff happens.
So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ?
There's no reason that running the reactor in a thread shouldn't work; it's just generally tricky (especially if you're working with an existing codebase) to completely avoid calling Twisted APIs from other threads.
Just wanted to thank you, I isolated the Twisted part of my application using the 'multiprocessing' module, and all the weird issues are gone, it seems I was mixing threads somewhere ... So executing the reactor in a child process was a good solution for my problem, no need to use deferToThread everywhere or to do tedious research about what is called from where. Cheers, Luper
participants (3)
-
flupke -
glyph＠divmod.com -
Luper Rouch