[Twisted-Python] Perspective Broker with blocking, non-thread-safe C api?
I'd like to use Perspective Broker to serve a set of objects remotely. However, these objects wrap a non-thread-safe, blocking C api. I'm new to Twisted, but I believe I understand correctly that the blocking C api needs to be accessed via threads in Twisted. Given this, the only way I can think to use Perspective Broker would be to use a special PBServerFactory/Protocol that limited the server to one connected client. For the use I would be putting the server to, this is maybe not as horrible as it sounds, but it is probably as ugly as it sounds. I'm hoping there's something in PB that I haven't come across that can solve this problem in a more elegant way. Any help or pointers would be greatly appreciated. Thanks, Steve _________________________________________________________________ http://newlivehotmail.com
On Wed, 2007-07-11 at 00:34 +0000, s. w. wrote:
It's not specific to PB - this applied to all of Twisted, and all of Twisted has the same solution. See "deferToThread" for examples of ways to call blocking APIs inside a thread, and when the call returns the deferred will fire.
Things are a bit more complicated that just calling deferToThread. The reason is that is you call a blocking C api in the thread, the Python interpreter will not be able to switch threads until the blocking C calls....unless the C code releases the Global Interpreter Lock. Thus you have two choices: 1) Make sure that the C code releases the GIL and call it using deferToThread. This is probably the best solution. 2) If you can't release the GIL - if for instance the C code makes calls to the Python C API, then you have to do the following. You can simply call the block C code in the main thread. This will block the Twisted event loop while the C code runs. To mask this blocking behavior, you will need to have a client that takes this into account. You basically need a way of making sure that while that code is running no other client makes a call to the server running the C code. You also have to make sure that the client that initiated the call doesn't make any further calls until the first is done. The way we usually handle this is to make a separate process that manages a queue of commands to be executed on the server. Clients then connect to this manager, which in turns submits them to the actual PB server at the appropriate time. This is a pain, but it works extremely well. Hope that helps. Brian On 7/11/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 03:25 pm, ellisonbg.net@gmail.com wrote:
That sounds like a viable "third choice", and I have a fourth ;). Have the PB server itself spawn a subprocess (or multiple subprocesses!) dedicated to this particular API. Then simply block in that (those) subprocess(es), and return Deferreds from your PB methods that wait for those requests to complete. That way the PB server remains self-contained and you don't add the administrative overhead of setting up a separate server.
On Wed, 2007-07-11 at 00:34 +0000, s. w. wrote:
It's not specific to PB - this applied to all of Twisted, and all of Twisted has the same solution. See "deferToThread" for examples of ways to call blocking APIs inside a thread, and when the call returns the deferred will fire.
Things are a bit more complicated that just calling deferToThread. The reason is that is you call a blocking C api in the thread, the Python interpreter will not be able to switch threads until the blocking C calls....unless the C code releases the Global Interpreter Lock. Thus you have two choices: 1) Make sure that the C code releases the GIL and call it using deferToThread. This is probably the best solution. 2) If you can't release the GIL - if for instance the C code makes calls to the Python C API, then you have to do the following. You can simply call the block C code in the main thread. This will block the Twisted event loop while the C code runs. To mask this blocking behavior, you will need to have a client that takes this into account. You basically need a way of making sure that while that code is running no other client makes a call to the server running the C code. You also have to make sure that the client that initiated the call doesn't make any further calls until the first is done. The way we usually handle this is to make a separate process that manages a queue of commands to be executed on the server. Clients then connect to this manager, which in turns submits them to the actual PB server at the appropriate time. This is a pain, but it works extremely well. Hope that helps. Brian On 7/11/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 03:25 pm, ellisonbg.net@gmail.com wrote:
That sounds like a viable "third choice", and I have a fourth ;). Have the PB server itself spawn a subprocess (or multiple subprocesses!) dedicated to this particular API. Then simply block in that (those) subprocess(es), and return Deferreds from your PB methods that wait for those requests to complete. That way the PB server remains self-contained and you don't add the administrative overhead of setting up a separate server.
participants (4)
-
Brian Granger -
glyph＠divmod.com -
Phil Mayers -
s. w.