[Twisted-Python] Re: [Twisted-commits] CVS: Twisted/twisted/internet poll.py,NONE,1.1
Itamar Shtull-Trauring wrote:
Update of /cvs/Twisted/twisted/internet In directory zaibach:/tmp/cvs-serv28575
Added Files: poll.py Log Message: preliminary poll()-based event loop
What I learned - the addReader/addWriter/etc. API is far too select() oriented. Every other alternative (poll, kqeue, windows event thing) works by registering and unregistering objects - correct me if I'm wrong. So instead of having to emulate a select()-like API, we need a select() wrapper that acts like poll(). In other words, addReader and friends must go when we refactor the event loop code.
On Wed, Nov 21, 2001 at 04:51:51PM +0200, Itamar Shtull-Trauring wrote:
What I learned - the addReader/addWriter/etc. API is far too select() oriented. Every other alternative (poll, kqeue, windows event thing) works by registering and unregistering objects - correct me if I'm wrong.
So instead of having to emulate a select()-like API, we need a select() wrapper that acts like poll(). In other words, addReader and friends must go when we refactor the event loop code.
I'm confused. In what way is "adding/removing a reader/writer" not "registering/unregistering an object"? In fact, I chose that API specifically because I didn't think it was like select(). -- ______ you are in a maze of twisted little applications, all | |_\ remarkably consistent. | | -- glyph lefkowitz, glyph @ twisted matrix . com |_____| http://www.twistedmatrix.com/
Glyph Lefkowitz wrote:
I'm confused. In what way is "adding/removing a reader/writer" not "registering/unregistering an object"?
In fact, I chose that API specifically because I didn't think it was like select().
Well, in poll() you don't register writing and reading separately - you have to register for them together (bu actually the limitation is in the Python API, not the C API). Further research shows that the current API might match other systems, so I was probably wrong about that.
Glyph Lefkowitz wrote:
I'm confused. In what way is "adding/removing a reader/writer" not "registering/unregistering an object"?
Here's one issue - removeWriter is called every time the buffer is epmtied out. In the lifetime of a connection, the transport keeps on adding and removing, adding and removing itself as a writer because of this. In select() this isn't an issue, but in poll() it is, since removing and adding the writer is an expensive operation (in fact it adds a %33 overhead, in tests under high loads). I'll see what I can do to fix the poll()-based implementation, but meanwhile the select event loop is much faster under high loads.
On Thu, Nov 22, 2001 at 03:27:20PM +0200, Itamar Shtull-Trauring wrote:
Glyph Lefkowitz wrote:
I'm confused. In what way is "adding/removing a reader/writer" not "registering/unregistering an object"?
Here's one issue - removeWriter is called every time the buffer is epmtied out. In the lifetime of a connection, the transport keeps on adding and removing, adding and removing itself as a writer because of this. In select() this isn't an issue, but in poll() it is, since removing and adding the writer is an expensive operation (in fact it adds a %33 overhead, in tests under high loads).
I'll see what I can do to fix the poll()-based implementation, but meanwhile the select event loop is much faster under high loads.
What I don't understand is that frequently, in an implementation where there is no distinction between registering for reads and registering for writes, you won't care if a socket is writable, only if it's readable. What does the higher-level implementation do then? Just early-out of the callback if the connection isn't "really" registerd for writes? Even though I am not sure I understand the problem :), perhaps the right answer to this question, since the common case is that all data can be written at once, is that the write should happen immediately, only being added to the select/poll loop if the write would block. This would probably add a little complexity to the code but save a bunch of performance on small-write heavy protocols. -- ______ you are in a maze of twisted little applications, all | |_\ remarkably consistent. | | -- glyph lefkowitz, glyph @ twisted matrix . com |_____| http://www.twistedmatrix.com/
Glyph Lefkowitz wrote:
Even though I am not sure I understand the problem :), perhaps the right answer to this question, since the common case is that all data can be written at once, is that the write should happen immediately, only being added to the select/poll loop if the write would block. This would probably add a little complexity to the code but save a bunch of performance on small-write heavy protocols.
Well, I tried this. The result was a 25-50 percent improvement in read benchmarks on my code. Web performance seems slightly worse in the one test I did. And it's probably got some issues detecting closed connections... I have another behcmark that takes twice as long. But as you can see there is some room for improvement, even if this patch specifically is buggy. Index: twisted/internet/abstract.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/abstract.py,v retrieving revision 1.11 diff -c -r1.11 abstract.py *** twisted/internet/abstract.py 2001/10/30 23:55:39 1.11 --- twisted/internet/abstract.py 2001/11/29 16:26:29 *************** *** 106,111 **** --- 106,117 ---- if not self.connected: return if data: + if not self.unsent: + l = self.writeSomeData(data) + if l == len(data) or l < 0: + return + else: + data = data[l:] self.unsent = self.unsent + data if self.producer is not None: if len(self.unsent) > self.bufferSize:
participants (2)
-
Glyph Lefkowitz -
Itamar Shtull-Trauring