On Mon, Feb 21, 2005 at 04:21:06PM +1100, Andrew Bennetts wrote:

On Mon, Feb 21, 2005 at 12:08:43AM -0500, James Y Knight wrote: [...]

...

NetstringReceiver implementing it). LineReceiver's pauseProducing does exactly what you want: 1) stops the socket from being read more and 2) doesn't call lineReceived any more even if there are more lines already buffered.

Ooh, I didn't know LineReceiver had that feature. Nice!

Very nice indeed.

On Mon, Feb 21, 2005 at 08:47:57PM +0000, glyph@divmod.com wrote:

On Mon, 21 Feb 2005 18:46:18 +0100, Andrea Arcangeli wrote:

...

Ok here we go, I even found a severe bug in the linereceiver code and I fixed it as well. The bug in linereceiver happens like this:

Any chance that you could include some unit tests for the behavior you're expecting? I think it might need to be fixed a different way. For example:

From the client send a flood of lines, and from the server call

Using the echoserver/client as base should be good enough to test this. pauseProducing() after the first line has been received. Then verify the server doesn't receive the other lines and that they sit in the tcp recv queue.

...

- while len(self.recvd) > 1: + while len(self.recvd) > 1 and not self.paused:

This seems like it could lead to some unpleasantly large (i.e. quadratic-time concatenation) buffers strings being left around if you pause one of these receivers and then forget to unpause it, or even

Won't the data be freed along with the protocol when the connection is closed? The protocol should go away when the connection is closed.

just if your peer happens to be sending data very rapidly.

The high bound of in flight receive data for each int32 protocol is set to 99999 (protocol.MAX_LENGTH), and the reactor will read with a limited buffer too. So I don't see any problem as long as transport.pauseProducing() does what's expected to do (i.e. to stop polling the socket and to stop receiving data from the wire). Anyway if there's still a problem, let's ignore my problem, and let's only focus on the LineReceiver implementation that is already in current twisted. As long as LineReceiver works my patch is going to be fine. There was a reentrancy bug in LineReceiver but I already fixed it.

On Tue, Feb 22, 2005 at 03:33:22AM +0000, Jp Calderone wrote:

That's not a unit test. Unit tests are described here:

http://c2.com/cgi/wiki?UnitTest

I've no time to write it sorry.

On Tue, Feb 22, 2005 at 03:35:40PM +1100, Andrew Bennetts wrote:

I've filed http://twistedmatrix.com/bugs/issue899 so this issue doesn't get forgotten.

Thanks a lot! What the pauseProducing does is fine for my objective of stopping the client and to restart it from both errback/callback, but what it does is possibly too aggressive. On the lines of glyph's email, sometime what could be needed is an option to _only_ mask the POLLIN, not necessairly all other bitflags, so that I could still receive a disconnect event. In some ways it might be good not to get a disconnect during a deferred, but OTOH the fact it doesn't notice a disconnect, prevents to leave the pauseProducing enabled for an extended period of time, while I could really do that in theory (I can't do that or I would lose the disconnect event with pauseProducing). But overall this would be an additional (minor) feature, quite orthogonal with the "hard" approach of pauseProducing. And having the API of int32/int16 in sync with linereceiver sounds a better approach for the short term, plus the reentrancy bugfix sounds good to have. Thanks everyone for the help and sorry for not having much time to provide unit test.

On Sun, 13 Mar 2005 20:57:09 +0100, Andrea Arcangeli wrote:

This patch adds a check for transport.disconnecting to the int16/32 protocols, so that they don't keep firing stringReceived callbacks after calling transport.loseConnection (in line with the linereceiver protocol). This also adds the methods for pauseproducer in line with the linereceiver protocol (as per previous patch in this thread).

If these two fixes to int32/16 are correct please apply to SVN.

transport.disconnecting is horrible (it's an implementation detail, it doesn't work on a lot of transports). LineReceiver shouldn't be using it. The feature LineReceiver uses it to implement may be reasonable, though, which is why its code is still how it is. If the feature is desired somewhere else, the best way to add it is to either find another way that doesn't rely on transport.disconnecting (and then we can fix LineReceiver too), or factor the code such that all protocols can provide the feature without having more than one method that relies on transport.disconnecting (then we can fix them all at once someday). Jp

On Tue, 2005-03-15 at 14:41 +0100, Andrea Arcangeli wrote:

On Mon, Mar 14, 2005 at 12:58:47PM +0000, Jp Calderone wrote:

...

If the feature is desired somewhere else, the best way to add it is to either find another way that doesn't rely on transport.disconnecting (and then we can fix LineReceiver too), or factor the code such that all protocols can provide the feature without having more than one method that relies on transport.disconnecting (then we can fix them all at once someday).

If there's no real generic API and you're not planning on adding it soon, perhaps we could check it with hasattr and spawn a warning if the transport doens't provide .disconnecting?

We probably ought to add an API, since the functionality appears to be useful.

On Tue, 15 Mar 2005 19:14:52 +0100, Andrea Arcangeli wrote:

On Tue, Mar 15, 2005 at 12:29:35PM -0500, Itamar Shtull-Trauring wrote:

...

We probably ought to add an API, since the functionality appears to be useful.

What about just adding .disconnecting to the other transports too then? That already exists and I don't see anything fundamentally wrong with it (especially with the property feature of python that won't leave us locked into a fixed value). Then incidentally my patch will become correct too ;).

The right way would be to add a 'state' attribute to all transports and have one possible value of it be 'disconnecting' (or DISCONNECTING, whatever ;). The disconnecting attribute is problematic as a symptom of a larger internal problem (a lack of clarity as to the possible states and state transitions a transport can be in and go through). Jp

On Mon, Feb 21, 2005 at 04:13:53PM +1100, Andrew Bennetts wrote:

On Mon, Feb 21, 2005 at 05:50:16AM +0100, Andrea Arcangeli wrote: [...]

...

The example in the docs is like this:

------------------------- class FingerProtocol(basic.LineReceiver): def lineReceived(self, user): self.factory.getUser(user ).addErrback(lambda _: "Internal error in server" ).addCallback(lambda m: (self.transport.write(m+"\r\n"), self.transport.loseConnection())) [..] we would have to fetch result from a remote Oracle? Or from the web? -------------------------

But as far as I can tell the above also is broken and it won't wait for the deferred to fire, so two of those queries could be invoked if the client side is malicious and it sends two lines in one packet and that could screwup the server. It'll work only as long as the clients aren't malicious.

I don't think it matters if the two lines arrive in the same packet or not; if the lines arrive before the getUser deferred fires, it will fire multiple queries, regardless of the exact packets involved. In this case maybe a flag should be in the protocol after the first lineReceived event... any

Well, it will raise an exception in transport.write at the very least, in theory unhandled exceptions shouldn't happen. But the reason I suggested to fix it, it to get people get the right mindset, unless they know what they're doing, they should always call LineReceiver.pauseProducing before executing the getUser(user), otherwise a _real_ app that copied the above code, may get wrong for more than just an exception.

I think the transport's pauseProducing call will do what you want, here. And resumeProducing when you're ready again.

Yes, if I were using a unbuffered input. Thanks a lot for the help!