Hi. I am hoping you can clarify for me whether it is possible to run multiple independent main loops in Twisted. It is my understanding that this would mean separate independent reactors.
Googling, I found this patch but after looking into the repository, it was never committed. I would appreciate any comments on the patch or attempts people have made on the integration of a second twisted loop.
At the present time I am using twisted with wxpython with wxpython as main loop and also running a selectreactor instance. Here there are two separate loops, interoperating and operating independently, however they are not both twisted loops.
Zope currently runs using a reactor instance for each zope instance but I would like to consider a separate independent loop for each client to synchronize to a server without paying a penalty. Zope's ZEO (basically a synchronization server) currently uses a simple zrpc protocol and runs independently but interoperates with a separate ansyncore main loop. The current system works well for the exception that it would be great to simplify this as well as improve security using twisted's built in capabilities. Many thanks.
Regards David
Index: twisted/internet/default.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/default.py,v retrieving revision 1.79 diff -u -r1.79 default.py --- twisted/internet/default.py 10 May 2003 12:52:37 -0000 1.79 +++ twisted/internet/default.py 11 May 2003 23:11:26 -0000 @@ -367,8 +367,6 @@
# global state for selector -reads = {} -writes = {}
def win32select(r, w, e, timeout=None): @@ -403,13 +401,18 @@
__implements__ = (PosixReactorBase.__implements__, IReactorFDSet)
+ def __init__(self): + PosixReactorBase.__init__(self) + self.reads = {} + self.writes = {} + def _preenDescriptors(self): log.msg("Malformed file descriptor found. Preening lists.") - readers = reads.keys() - writers = writes.keys() - reads.clear() - writes.clear() - for selDict, selList in ((reads, readers), (writes, writers)): + readers = self.reads.keys() + writers = self.writes.keys() + self.reads.clear() + self.writes.clear() + for selDict, selList in ((self.reads, readers), (self.writes, writers)): for selectable in selList: try: select.select([selectable], [selectable], [selectable], 0) @@ -419,19 +422,19 @@ selDict[selectable] = 1
- def doSelect(self, timeout, - # Since this loop should really be as fast as possible, - # I'm caching these global attributes so the interpreter - # will hit them in the local namespace. - reads=reads, - writes=writes, - rhk=reads.has_key, - whk=writes.has_key): + def doSelect(self, timeout): """Run one iteration of the I/O monitor loop.
This will run all selectables who had input or output readiness waiting for them. """ + # Since this loop should really be as fast as possible, + # I'm caching these global attributes so the interpreter + # will hit them in the local namespace. + reads=self.reads + writes=self.writes + rhk=reads.has_key + whk=writes.has_key while 1: try: r, w, ignored = _select(reads.keys(), @@ -502,33 +505,33 @@ def addReader(self, reader): """Add a FileDescriptor for notification of data available to read. """ - reads[reader] = 1 + self.reads[reader] = 1
def addWriter(self, writer): """Add a FileDescriptor for notification of data available to write. """ - writes[writer] = 1 + self.writes[writer] = 1
def removeReader(self, reader): """Remove a Selectable for notification of data available to read. """ - if reads.has_key(reader): - del reads[reader] + if self.reads.has_key(reader): + del self.reads[reader]
def removeWriter(self, writer): """Remove a Selectable for notification of data available to write. """ - if writes.has_key(writer): - del writes[writer] + if self.writes.has_key(writer): + del self.writes[writer]
def removeAll(self): """Remove all readers and writers, and return list of Selectables.""" - readers = reads.keys() + readers = self.reads.keys() for reader in readers: - if reads.has_key(reader): - del reads[reader] - if writes.has_key(reader): - del writes[reader] + if self.reads.has_key(reader): + del self.reads[reader] + if self.writes.has_key(reader): + del self.writes[reader] return readers
Hi,
I'm not sure you are talking about the same topic. But I have just been thinking about a threaded reactor.
Because my biggest itch with twisted is, that it can not really make use multi-processor systems. Especially nowadays with dual-core or even multi-core processors hitting the market this seems like a must-have.
In theory, it would be possible to start as many threads as you have processors. All threads would process 'events' of the main loop. See for example the apache event MPM for a better explaination of the algorithm.
Beside allowing only one thread at a time to call select(), this would have other implications for twisted. I'm not sure about what they are, but I can imagine it's a non-trivial goal.
How about todays python thread implementation? Did it improve over the years? Is it worth using? What must be considered to get a reasonably good performance using python threads?
..just some ideas. I'd be thankfull for hints, though.
Regards
Markus
On Tue, 2006-05-02 at 12:00 -0300, David Pratt wrote:
Hi. I am hoping you can clarify for me whether it is possible to run multiple independent main loops in Twisted. It is my understanding that this would mean separate independent reactors.
Googling, I found this patch but after looking into the repository, it was never committed. I would appreciate any comments on the patch or attempts people have made on the integration of a second twisted loop.
At the present time I am using twisted with wxpython with wxpython as main loop and also running a selectreactor instance. Here there are two separate loops, interoperating and operating independently, however they are not both twisted loops.
Zope currently runs using a reactor instance for each zope instance but I would like to consider a separate independent loop for each client to synchronize to a server without paying a penalty. Zope's ZEO (basically a synchronization server) currently uses a simple zrpc protocol and runs independently but interoperates with a separate ansyncore main loop. The current system works well for the exception that it would be great to simplify this as well as improve security using twisted's built in capabilities. Many thanks.
Regards David
<SNIPPED the attached patch>
On Tue, 02 May 2006 17:50:16 +0200, Markus Schiltknecht markus@bluegap.ch wrote:
Because my biggest itch with twisted is, that it can not really make use multi-processor systems. Especially nowadays with dual-core or even multi-core processors hitting the market this seems like a must-have.
s/Twisted/Python/g
Python maintains a global interpreter lock which prevents the interpreter from running on more than one CPU at a time. If you want to take advantage of multiple CPUs with Twisted, you have to spawn multiple communicating processes, not threads.
Even if you *could* take advantage of multiple processors by using threads, making Twisted generally thread-safe would be a HUGE undertaking, probably spanning years, and I cannot imagine anyone on the current development team would care about this as a feature, with so many other major issues looming.
On Tue, 02 May 2006 12:00:53 -0300, David Pratt fairwinds@eastlink.ca wrote:
Hi. I am hoping you can clarify for me whether it is possible to run multiple independent main loops in Twisted. It is my understanding that this would mean separate independent reactors.
Yep. This is not currently possible. Most importantly there are no tests for it, so we would have no idea if it worked. Writing tests for race conditions is notoriously hard.
Googling, I found this patch but after looking into the repository, it was never committed. I would appreciate any comments on the patch or attempts people have made on the integration of a second twisted loop.
This patch was deliberately left uncommitted. It definitely reverses an optimization that was demonstrated to improve performance on the critical path; it only maybe makes it possible to initialize multiple reactors. If you want to write a comprehensive set of unit tests for multiple reactors, we might consider doing something like this, otherwise the reactor will be left in its current state in order to discourage people from attempting to start multiple reactors concurrently.
At the present time I am using twisted with wxpython with wxpython as main loop and also running a selectreactor instance. Here there are two separate loops, interoperating and operating independently, however they are not both twisted loops.
wxpython is buggy and has terrible event-loop support. Consider using PyGTK instead. If you really must use wx, there are a variety of mostly-working ways to integrate the mainloop with Twisted without resorting to multiple reactors.
glyph@divmod.com wrote:
On Tue, 02 May 2006 12:00:53 -0300, David Pratt fairwinds@eastlink.ca wrote:
Hi. I am hoping you can clarify for me whether it is possible to run multiple independent main loops in Twisted. It is my understanding that this would mean separate independent reactors.
Yep. This is not currently possible. Most importantly there are no tests for it, so we would have no idea if it worked. Writing tests for race conditions is notoriously hard.
Hi glyph. Have there been previous attempts to run more than a single reactor together? Does any of this exist anywhere in the history of the your repository.
Googling, I found this patch but after looking into the repository, it was never committed. I would appreciate any comments on the patch or attempts people have made on the integration of a second twisted loop.
This patch was deliberately left uncommitted. It definitely reverses an optimization that was demonstrated to improve performance on the critical path; it only maybe makes it possible to initialize multiple reactors. If you want to write a comprehensive set of unit tests for multiple reactors, we might consider doing something like this, otherwise the reactor will be left in its current state in order to discourage people from attempting to start multiple reactors concurrently.
I understand what you are saying however there is some appeal in exploring this possibility (if there is a possibility for it to work).
At the present time I am using twisted with wxpython with wxpython as main loop and also running a selectreactor instance. Here there are two separate loops, interoperating and operating independently, however they are not both twisted loops.
wxpython is buggy and has terrible event-loop support. Consider using PyGTK instead. If you really must use wx, there are a variety of mostly-working ways to integrate the mainloop with Twisted without resorting to multiple reactors.
I have been using wxpython and twisted together for about a year. They seem to perform well together and I would have no problem recommending wxpython for general cross platform gui development. I am using wxpython only to illustrate that multiple independent loops can work cooperatively. The target for the multiple reactor possibility is zope client instances where currently an asyncore loop has been integrated for zeo support.
Regards, David
David Pratt wrote:
I have been using wxpython and twisted together for about a year. They seem to perform well together and I would have no problem recommending wxpython for general cross platform gui development. I am using wxpython
I'm afraid at least on this list you're in a minority then. It's not widely liked, if for no other reason than it's highly uncooperative event loop.
only to illustrate that multiple independent loops can work cooperatively. The target for the multiple reactor possibility is zope client instances where currently an asyncore loop has been integrated for zeo support.
I'm not entirely certain I understand what you want to do - could you explain the zope/zeo/zrpc issue(s) you're trying to solve?
On Wed, 03 May 2006 09:26:21 +0100, Phil Mayers p.mayers@imperial.ac.uk wrote:
David Pratt wrote:
I have been using wxpython and twisted together for about a year. They seem to perform well together and I would have no problem recommending wxpython for general cross platform gui development. I am using wxpython
I'm afraid at least on this list you're in a minority then. It's not widely liked, if for no other reason than it's highly uncooperative event loop.
Actually, it seems like a lot of people use WX and Twisted, and some people like the combination quite a lot. I think OSAF is still using this combination for Chandler and I don't think they're unhappy.
The reason you hear so much vitriol about it on this list is that 90% of the time, wx + networking appears to work in the lab, but when released into the wild will explode in unexpected ways; many people here who don't like wx are nevertheless maintaining applications which use it. If you know what you're doing with both event loops you can prevent this, though.
glyph@divmod.com wrote:
On Wed, 03 May 2006 09:26:21 +0100, Phil Mayers p.mayers@imperial.ac.uk wrote:
David Pratt wrote:
I have been using wxpython and twisted together for about a year. They seem to perform well together and I would have no problem recommending wxpython for general cross platform gui development. I am using wxpython
I'm afraid at least on this list you're in a minority then. It's not widely liked, if for no other reason than it's highly uncooperative event loop.
Actually, it seems like a lot of people use WX and Twisted, and some people like the combination quite a lot. I think OSAF is still using this combination for Chandler and I don't think they're unhappy.
We are using Twisted and wxPython and so far have been very pleased - but that doesn't mean we haven't had many of the issues that glyph is warning about. We've just built suitable walls so the two parts don't really realize they are in the same app.
bear build/release grunt for Open Source Applications Foundation (OSAF)
Hi Phil. I am pleased about the zope and twisted integrated in zope3. I believe there can be a tighter integration and additional benefits. Zope currently uses a twisted loop to run the application and an asyncore loop for zeo communication. These are both initialized at start up. What I would like to see, is ansyncore replaced with a second twisted loop. I would also like the zeo server to be a twisted app that runs zrpc. Twisted's ssh and ssl support would be used to secure client / server communication.
Regards, David
Phil Mayers wrote:
David Pratt wrote:
I have been using wxpython and twisted together for about a year. They seem to perform well together and I would have no problem recommending wxpython for general cross platform gui development. I am using wxpython
I'm afraid at least on this list you're in a minority then. It's not widely liked, if for no other reason than it's highly uncooperative event loop.
only to illustrate that multiple independent loops can work cooperatively. The target for the multiple reactor possibility is zope client instances where currently an asyncore loop has been integrated for zeo support.
I'm not entirely certain I understand what you want to do - could you explain the zope/zeo/zrpc issue(s) you're trying to solve?
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, 03 May 2006 20:32:26 -0300, David Pratt fairwinds@eastlink.ca wrote:
Hi Phil. I am pleased about the zope and twisted integrated in zope3. I believe there can be a tighter integration and additional benefits. Zope currently uses a twisted loop to run the application and an asyncore loop for zeo communication. These are both initialized at start up. What I would like to see, is ansyncore replaced with a second twisted loop. I would also like the zeo server to be a twisted app that runs zrpc. Twisted's ssh and ssl support would be used to secure client / server communication.
Why is a second event-loop necessary here?
glyph@divmod.com wrote:
On Wed, 03 May 2006 20:32:26 -0300, David Pratt fairwinds@eastlink.ca wrote:
Hi Phil. I am pleased about the zope and twisted integrated in zope3. I believe there can be a tighter integration and additional benefits. Zope currently uses a twisted loop to run the application and an asyncore loop for zeo communication. These are both initialized at start up. What I would like to see, is ansyncore replaced with a second twisted loop. I would also like the zeo server to be a twisted app that runs zrpc. Twisted's ssh and ssl support would be used to secure client / server communication.
Why is a second event-loop necessary here?
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Glyph. I believe that performance would suffer using a single loop. I am not aware of any evaluation to provide evidence of the extent of the impact this would have. zrpc has never been refactored in twisted to date, though I have recently suggested this. My general feeling is that a different syncing solution is most likely needed. The existing zeo client / server relationship has limitations, one of these being the granularity desired in the communication longer term. My inclination more recently has been to leave zeo alone and work with something different that might present an alternative.
The thought of using the existing reactor for communication and serving has not escaped me. Once I have a syncing server working in twisted with whatever protocol I use, I will have the chance to evaluate performance on a couple of levels. This discussion for me has a bit more to do with possibilities and exploring boundaries at this point. Googling for information, I came across the patch that also suggested the possibility of multiple independent reactors. This possibility is interesting to me, even outside of zope.
Regards, David
i have erver try to patch twisted to let it use threadpool,so that do_select will not be blocked,and let more than one thread execute the event handle,but seeming as above mentioned,that undertaking is non_trivival goal.we have to take care of condition race.so currently for multiple threaded main loop,it is not possible
glyph@divmod.com wrote:
Why is a second event-loop necessary here?
Hi Glyph. I believe that performance would suffer using a single loop. I am not aware of any evaluation to provide evidence of the extent of the impact this would have.
Fair enough.
The thought of using the existing reactor for communication and serving has not escaped me. Once I have a syncing server working in twisted with whatever protocol I use, I will have the chance to evaluate performance on a couple of levels. This discussion for me has a bit more to do with possibilities and exploring boundaries at this point. Googling for information, I came across the patch that also suggested the possibility of multiple independent reactors. This possibility is interesting to me, even outside of zope.
Despite the fact that I don't have a similar use-case on a real server or client, it's also interesting to me. It would definitely make certain test-cases easier to write, and easier testing == more tests. :)
I previously suggested that perhaps you could write tests for that patch, to verify that multiple reactors could actually run simultaneously. That was a serious suggestion: if you'd like Twisted to support this feature, you can almost certainly get it accepted if there are accompanying test cases. I'll commit to doing the reviews if you submit the patches.
glyph@divmod.com wrote:
I previously suggested that perhaps you could write tests for that patch, to verify that multiple reactors could actually run simultaneously. That was a serious suggestion: if you'd like Twisted to support this feature, you can almost certainly get it accepted if there are accompanying test cases. I'll commit to doing the reviews if you submit the patches.
Hi Glyph. The first thing I'm looking at is putting together a sync server. This will take some time. Once I have something, I'll be looking to measure the impact on zope. This should eliminate some speculatation and provide a clearer picture of performance.
At this point I'm hoping to have something a bit more concrete to discuss. I may be surprised - who knows. I am hoping this effort will result in something that will work well enough to use on zope's single reactor (if only for lighter use) in the interim.
If performance is not so hot, I will be revisit the multiple reactor issue. There is another person that I have been in contact with that is also interested in helping. But there is no point in going this direction just yet. Not without at least a demonstration server and client, and any proof a significant performance impact.
Regards, David
David Pratt wrote:
currently uses a twisted loop to run the application and an asyncore loop for zeo communication. These are both initialized at start up. What I would like to see, is ansyncore replaced with a second twisted loop. I
Why in the world wouldn't you use the first twisted loop? Why a second?
participants (6)
-
bear -
David Pratt -
glyph＠divmod.com -
Markus Schiltknecht -
Phil Mayers -
xuezhu aerodew