[Twisted-Python] Perspective Broker with blocking, non-thread-safe C api?
I'd like to use Perspective Broker to serve a set of objects remotely. However, these objects wrap a non-thread-safe, blocking C api. I'm new to Twisted, but I believe I understand correctly that the blocking C api needs to be accessed via threads in Twisted. Given this, the only way I can think to use Perspective Broker would be to use a special PBServerFactory/Protocol that limited the server to one connected client. For the use I would be putting the server to, this is maybe not as horrible as it sounds, but it is probably as ugly as it sounds. I'm hoping there's something in PB that I haven't come across that can solve this problem in a more elegant way. Any help or pointers would be greatly appreciated. Thanks, Steve _________________________________________________________________ http://newlivehotmail.com
On Wed, 2007-07-11 at 00:34 +0000, s. w. wrote:
I'd like to use Perspective Broker to serve a set of objects remotely. However, these objects wrap a non-thread-safe, blocking C api. I'm new to Twisted, but I believe I understand correctly that the blocking C api needs to be accessed via threads in Twisted.
Given this, the only way I can think to use Perspective Broker would be to use a special PBServerFactory/Protocol that limited the server to one connected client. For the use I would be putting the server to, this is maybe not as horrible as it sounds, but it is probably as ugly as it sounds.
I'm hoping there's something in PB that I haven't come across that can solve this problem in a more elegant way. Any help or pointers would be greatly appreciated.
It's not specific to PB - this applied to all of Twisted, and all of Twisted has the same solution. See "deferToThread" for examples of ways to call blocking APIs inside a thread, and when the call returns the deferred will fire.
Thanks, Steve
_________________________________________________________________ http://newlivehotmail.com
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Things are a bit more complicated that just calling deferToThread.
The reason is that is you call a blocking C api in the thread, the
Python interpreter will not be able to switch threads until the
blocking C calls....unless the C code releases the Global Interpreter
Lock. Thus you have two choices:
1) Make sure that the C code releases the GIL and call it using
deferToThread. This is probably the best solution.
2) If you can't release the GIL - if for instance the C code makes
calls to the Python C API, then you have to do the following. You can
simply call the block C code in the main thread. This will block the
Twisted event loop while the C code runs. To mask this blocking
behavior, you will need to have a client that takes this into account.
You basically need a way of making sure that while that code is
running no other client makes a call to the server running the C code.
You also have to make sure that the client that initiated the call
doesn't make any further calls until the first is done.
The way we usually handle this is to make a separate process that
manages a queue of commands to be executed on the server. Clients
then connect to this manager, which in turns submits them to the
actual PB server at the appropriate time. This is a pain, but it
works extremely well.
Hope that helps.
Brian
On 7/11/07, Phil Mayers
On Wed, 2007-07-11 at 00:34 +0000, s. w. wrote:
I'd like to use Perspective Broker to serve a set of objects remotely. However, these objects wrap a non-thread-safe, blocking C api. I'm new to Twisted, but I believe I understand correctly that the blocking C api needs to be accessed via threads in Twisted.
Given this, the only way I can think to use Perspective Broker would be to use a special PBServerFactory/Protocol that limited the server to one connected client. For the use I would be putting the server to, this is maybe not as horrible as it sounds, but it is probably as ugly as it sounds.
I'm hoping there's something in PB that I haven't come across that can solve this problem in a more elegant way. Any help or pointers would be greatly appreciated.
It's not specific to PB - this applied to all of Twisted, and all of Twisted has the same solution.
See "deferToThread" for examples of ways to call blocking APIs inside a thread, and when the call returns the deferred will fire.
Thanks, Steve
_________________________________________________________________ http://newlivehotmail.com
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 03:25 pm, ellisonbg.net@gmail.com wrote:
The reason is that is you call a blocking C api in the thread, the Python interpreter will not be able to switch threads until the blocking C calls....unless the C code releases the Global Interpreter Lock. Thus you have two choices: (1 and 2 snipped) The way we usually handle this is to make a separate process that manages a queue of commands to be executed on the server. Clients then connect to this manager, which in turns submits them to the actual PB server at the appropriate time. This is a pain, but it works extremely well.
That sounds like a viable "third choice", and I have a fourth ;). Have the PB server itself spawn a subprocess (or multiple subprocesses!) dedicated to this particular API. Then simply block in that (those) subprocess(es), and return Deferreds from your PB methods that wait for those requests to complete. That way the PB server remains self-contained and you don't add the administrative overhead of setting up a separate server.
participants (4)
-
Brian Granger -
glyph＠divmod.com -
Phil Mayers -
s. w.