[Twisted-Python] Reactors and New Asynchronous APIs
Hello, I have several questions about twisted reactors and extending the twisted framework with new asynchronous APIs. 1- What is the status of the IOCP reactor? Is it stable? If it is, is there any reason for not selecting it as the default reactor for win32 NT platforms? 2- I am considering the integration of a custom database API (like Berkeley DB) in a twisted application. I could easily wrap the API in a C++ python module and access it synchronously from twisted, at least in a prototyping phase. Obviously, if the blocking calls are too long it would make the whole framework completely useless. Then, the calls could be made from twisted managed threads. Assuming the module code was written to minimize GIL contention it would probably greatly improve the situation. The most efficient solution would be to plug the asynchronous API directly into the reactor: Is it possible to extend reactors with new asynchronous API implemented in separate client modules? With my little knowledge of the twisted framework and quick glances at the selectreactor and iocpreactor implementation, I think the answer is no. The practical solution would be to duplicate the reactor code and extend it with the new API calls. I do not know if it would be easy or not to provide a pluggable asynchronous framework *at C module level*. Basically, the reactor is nothing else than a thread-safe event queue with a (post/get)-event API. If this abstraction can be implemented at C level, then plugging new asynchronous handlers would be clearly doable, _iocp.c is a good example for that. Now, I am not really knowledgeable (yet) about python C module coding, and I guess it is not exactly easy to bind C callbacks from one python C module into another using normal python code as glue. In the end, this is just another plugin/component system like COM to write so... Any thoughts? Patrick Mézard
On 5/5/05, Patrick Mézard <pmezard@gmail.com> wrote:
The IOCP reactor doesn't support spawnProcess, among other things, which is why I currently can't use it. The author (PenguinOf in IRC) was also going to submit a list of bugs he knows about. Here are the notes I jotted notes down in IRC. 1. always resets connection instead of closing it nicely 2. need half-close support 3. need fancy optimizations, like multiple threads and multiple outstanding read requests per socket 4. redesign proactor 5. reactor.shutdown doesn't cancel 2- I am considering the integration of a custom database API (like Berkeley
On May 5, 2005, at 5:52 AM, Patrick Mézard wrote:
No, that's fundamentally impossible. Unless you have some resource the API you're using understands (i.e. a file descriptor), then there's no way you're going to be able to wake up a blocking call to select(), WaitForMultipleObjects, or whatever. What you probably want to do is run your code in a separate thread, and notify the main thread when something happens. (e.g. callInThread, etc.). See threadedselectreactor (in svn) for an inversion of this idea -- which runs (the blocking part of) Twisted in a separate thread and notifies your foreign event loop when something happens.
If the reactor just polled until something happened and spun 100% CPU, then sure, you could integrate whatever the hell "asynchronous" API you wanted. That's not how anything (should) work in a multi- tasking operating system. That's not even driven. -bob
On 5/5/05, Patrick Mézard <pmezard@gmail.com> wrote:
The IOCP reactor doesn't support spawnProcess, among other things, which is why I currently can't use it. The author (PenguinOf in IRC) was also going to submit a list of bugs he knows about. Here are the notes I jotted notes down in IRC. 1. always resets connection instead of closing it nicely 2. need half-close support 3. need fancy optimizations, like multiple threads and multiple outstanding read requests per socket 4. redesign proactor 5. reactor.shutdown doesn't cancel 2- I am considering the integration of a custom database API (like Berkeley
On May 5, 2005, at 5:52 AM, Patrick Mézard wrote:
No, that's fundamentally impossible. Unless you have some resource the API you're using understands (i.e. a file descriptor), then there's no way you're going to be able to wake up a blocking call to select(), WaitForMultipleObjects, or whatever. What you probably want to do is run your code in a separate thread, and notify the main thread when something happens. (e.g. callInThread, etc.). See threadedselectreactor (in svn) for an inversion of this idea -- which runs (the blocking part of) Twisted in a separate thread and notifies your foreign event loop when something happens.
If the reactor just polled until something happened and spun 100% CPU, then sure, you could integrate whatever the hell "asynchronous" API you wanted. That's not how anything (should) work in a multi- tasking operating system. That's not even driven. -bob
participants (3)
-
Bob Ippolito -
Justin Johnson -
Patrick Mézard