[Twisted-Python] A few simple questions
I'm adding Twisted support to some existing communication code and I have a few questions which I hope are simple. I've been reading the API documentation, but if the answers are there I'm missing them. What happens if one tries to write to a protocol.transport that is not connected or in an error state? (I'm hoping that this case has defined behavior and that it will raise an exception). Is there some way to query a protocol to see if it is in a good state (connected and no errors)? I realize with callbacks this is not usually necessary, but I have a case where it would be helpful (though I can probably manage without it). When reading raw bytes, is there any way to get the bytes from the protocol? I assume not: that if one wants to buffer the data one must do that in the callback. If so, does Twisted provide a suitable buffer class? -- Russell
On Thursday, July 12, 2012, Russell E. Owen wrote:
I'm adding Twisted support to some existing communication code and I have a few questions which I hope are simple. I've been reading the API documentation, but if the answers are there I'm missing them.
What happens if one tries to write to a protocol.transport that is not connected or in an error state? (I'm hoping that this case has defined behavior and that it will raise an exception).
Not connected usually means that protocol.transport is None
Is there some way to query a protocol to see if it is in a good state (connected and no errors)? I realize with callbacks this is not usually necessary, but I have a case where it would be helpful (though I can probably manage without it).
Can you elaborate on the case?
When reading raw bytes, is there any way to get the bytes from the protocol? I assume not: that if one wants to buffer the data one must do that in the callback. If so, does Twisted provide a suitable buffer class?
The protocol gets dataReceivrd called. What kind of buffering do you want to do and why? Would a simple StringIO suffice?
-- Russell
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <javascript:;> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- cheers lvh
On 04:02 pm, _@lvh.cc wrote:
On Thursday, July 12, 2012, Russell E. Owen wrote:
I'm adding Twisted support to some existing communication code and I have a few questions which I hope are simple. I've been reading the API documentation, but if the answers are there I'm missing them.
What happens if one tries to write to a protocol.transport that is not connected or in an error state? (I'm hoping that this case has defined behavior and that it will raise an exception).
Not connected usually means that protocol.transport is None
I don't think you'll find this to be true. Not connected means makeConnection hasn't yet been called or connectionLost has been called. Anything else you want, you need to build yourself (and given those two callbacks, it should be obvious how to build something simple to tell you this). Jean-Paul
In article <CAE_Hg6bDAiL_HUF1D7ZfigGv-FgXTDqsriq2RT44tAqweiBBDg@mail.gmail.com>, Laurens Van Houtven <_@lvh.cc> wrote:
On Thursday, July 12, 2012, Russell E. Owen wrote:
I'm adding Twisted support to some existing communication code and I have a few questions which I hope are simple. I've been reading the API documentation, but if the answers are there I'm missing them.
What happens if one tries to write to a protocol.transport that is not connected or in an error state? (I'm hoping that this case has defined behavior and that it will raise an exception).
Not connected usually means that protocol.transport is None
Is there some way to query a protocol to see if it is in a good state (connected and no errors)? I realize with callbacks this is not usually necessary, but I have a case where it would be helpful (though I can probably manage without it).
Can you elaborate on the case?
I've found that Twisted sometimes swallows errors unless I am extremely careful. I would like to be able to check a protocol to make sure it is operational (connected and happy) as a means of assuring that I've not missed an error.
When reading raw bytes, is there any way to get the bytes from the protocol? I assume not: that if one wants to buffer the data one must do that in the callback. If so, does Twisted provide a suitable buffer class?
The protocol gets dataReceived called. What kind of buffering do you want to do and why? Would a simple StringIO suffice?
I read the LineReceiver code (should have done that before asking) and realize it's buffering using a simple string. I just rewrote that simple class to meet my needs and am quite happy. -- Russell
On Jul 17, 2012, at 9:25 AM, Russell E. Owen <rowen@uw.edu> wrote:
Can you elaborate on the case?
I've found that Twisted sometimes swallows errors unless I am extremely careful. I would like to be able to check a protocol to make sure it is operational (connected and happy) as a means of assuring that I've not missed an error.
This isn't really elaborating. What error does Twisted swallow? In what manner do you have to be "careful"? What characterizes a protocol's "happiness" beyond its momentary connected-ness? How would you "miss" an error? Twisted is event-driven, so pretty much all the state changes you're interested in are delivered as events (method calls on your object). If you are "missing" them because Twisted isn't calling them, that sounds like a really serious bug we should investigate. If you're missing them for some other reason then you should just fix your code so it doesn't miss them any more :). -glyph
In article <DB49C8B0-451D-46E4-9B3B-58519CDAA75A@twistedmatrix.com>, Glyph <glyph@twistedmatrix.com> wrote:
On Jul 17, 2012, at 9:25 AM, Russell E. Owen <rowen@uw.edu> wrote:
Can you elaborate on the case?
I've found that Twisted sometimes swallows errors unless I am extremely careful. I would like to be able to check a protocol to make sure it is operational (connected and happy) as a means of assuring that I've not missed an error.
This isn't really elaborating.
What error does Twisted swallow? In what manner do you have to be "careful"? What characterizes a protocol's "happiness" beyond its momentary connected-ness? How would you "miss" an error?
Twisted is event-driven, so pretty much all the state changes you're interested in are delivered as events (method calls on your object). If you are "missing" them because Twisted isn't calling them, that sounds like a really serious bug we should investigate. If you're missing them for some other reason then you should just fix your code so it doesn't miss them any more :).
Regarding swallowing errors: I'm not claiming Twisted has bugs in this area (though I found and reported one obscure case that may be a bug: errors in tk command ::tk::Mac::quit are silently ignored). However, I found it rather easy to make coding errors that cause Twisted to not report errors. I have heard the same complaint from colleagues. At this point I think my code is robust, though I will be more confident once I finish my unit tests. If there is a "best practices for error handling" document I'd love to read it. I found an overview of deferreds that was helpful. It pointed out that addCallbacks is not the same as addCallback followed by addErrback and I'm not sure I'm handling the difference correctly. Right now I use addCallbacks(callback, errback). This does not call the errback if the callback fails. I do this intentionally because I want the errback to focus on problems with the connection, not my reaction to it, and the default error handler seems to report errors in the callback, which is fine. Regarding state: I'm used to event-driven frameworks that both offer callbacks and the ability to query state. I find it helpful for reporting errors as soon as possible (e.g. before writing check if the socket is connected; raise an exception if not). For a TCP/IP socket, I view a socket as having these states: - connecting - connected - disconnecting (with associated reason) - disconnected (with associated reason) TCP/IP Servers have a similar set of states, with connected -> listening However, it's certainly true that being able to query state is not essential; callbacks suffice. -- Russell
If you haven't seen it already, Dave Peticolas has a great introduction to twisted (it's a bit long, but well worth it) at http://krondo.com/?page_id=1327. In particular, until I saw this graphic on callback/errback chaining ( http://krondo.com/blog/wp-content/uploads/2009/10/deferred-2.png), I never properly understood what twisted was doing (linked from this http://krondo.com/?p=1825). Naveen On Wed, Jul 18, 2012 at 1:31 PM, Russell E. Owen <rowen@uw.edu> wrote:
In article <DB49C8B0-451D-46E4-9B3B-58519CDAA75A@twistedmatrix.com>, Glyph <glyph@twistedmatrix.com> wrote:
On Jul 17, 2012, at 9:25 AM, Russell E. Owen <rowen@uw.edu> wrote:
Can you elaborate on the case?
I've found that Twisted sometimes swallows errors unless I am extremely careful. I would like to be able to check a protocol to make sure it is operational (connected and happy) as a means of assuring that I've not missed an error.
This isn't really elaborating.
What error does Twisted swallow? In what manner do you have to be "careful"? What characterizes a protocol's "happiness" beyond its momentary connected-ness? How would you "miss" an error?
Twisted is event-driven, so pretty much all the state changes you're interested in are delivered as events (method calls on your object). If you are "missing" them because Twisted isn't calling them, that sounds like a really serious bug we should investigate. If you're missing them for some other reason then you should just fix your code so it doesn't miss them any more :).
Regarding swallowing errors:
I'm not claiming Twisted has bugs in this area (though I found and reported one obscure case that may be a bug: errors in tk command ::tk::Mac::quit are silently ignored).
However, I found it rather easy to make coding errors that cause Twisted to not report errors. I have heard the same complaint from colleagues.
At this point I think my code is robust, though I will be more confident once I finish my unit tests.
If there is a "best practices for error handling" document I'd love to read it. I found an overview of deferreds that was helpful. It pointed out that addCallbacks is not the same as addCallback followed by addErrback and I'm not sure I'm handling the difference correctly. Right now I use addCallbacks(callback, errback). This does not call the errback if the callback fails. I do this intentionally because I want the errback to focus on problems with the connection, not my reaction to it, and the default error handler seems to report errors in the callback, which is fine.
Regarding state:
I'm used to event-driven frameworks that both offer callbacks and the ability to query state. I find it helpful for reporting errors as soon as possible (e.g. before writing check if the socket is connected; raise an exception if not). For a TCP/IP socket, I view a socket as having these states: - connecting - connected - disconnecting (with associated reason) - disconnected (with associated reason) TCP/IP Servers have a similar set of states, with connected -> listening
However, it's certainly true that being able to query state is not essential; callbacks suffice.
-- Russell
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 07/18/2012 01:31 PM, Russell E. Owen wrote:
If there is a "best practices for error handling" document I'd love to read it. I found an overview of deferreds that was helpful. It pointed out that addCallbacks is not the same as addCallback followed by addErrback and I'm not sure I'm handling the difference correctly. Right now I use addCallbacks(callback, errback). This does not call the errback if the callback fails. I do this intentionally because I want the errback to focus on problems with the connection, not my reaction to it, and the default error handler seems to report errors in the callback, which is fine.
One thing to realize is that errors that end up in a Deferred are only logged if it gets garbage collected... which won't happen if you keep a reference to it. So: 1. You should always delete a reference to a Deferred when you do callback() or errback(). 2. In case it never gets GC'ed, always do .addErrback(log.err) as the last callback on the Deferred, if there's no other final error handler.
On 04:25 pm, rowen@uw.edu wrote:
In article <CAE_Hg6bDAiL_HUF1D7ZfigGv-FgXTDqsriq2RT44tAqweiBBDg@mail.gmail.com>, Laurens Van Houtven <_@lvh.cc> wrote:
On Thursday, July 12, 2012, Russell E. Owen wrote:
I'm adding Twisted support to some existing communication code and I have a few questions which I hope are simple. I've been reading the API documentation, but if the answers are there I'm missing them.
What happens if one tries to write to a protocol.transport that is not connected or in an error state? (I'm hoping that this case has defined behavior and that it will raise an exception).
Not connected usually means that protocol.transport is None
Is there some way to query a protocol to see if it is in a good
state
(connected and no errors)? I realize with callbacks this is not usually necessary, but I have a case where it would be helpful (though I can probably manage without it).
Can you elaborate on the case?
I've found that Twisted sometimes swallows errors unless I am extremely careful. I would like to be able to check a protocol to make sure it is operational (connected and happy) as a means of assuring that I've not missed an error.
Instead, make sure you add errbacks to all your Deferreds. Twisted does not swallow errors. However, a Deferred (that your application keeps a reference to, preventing it from being garbage collected) that has a Failure result and no more errbacks will sit on that Failure until you let the Deferred get garbage collected or add an errback to the Deferred. Jean-Paul
When reading raw bytes, is there any way to get the bytes from the protocol? I assume not: that if one wants to buffer the data one must do that in the callback. If so, does Twisted provide a suitable buffer class?
The protocol gets dataReceived called. What kind of buffering do you want to do and why? Would a simple StringIO suffice?
I read the LineReceiver code (should have done that before asking) and realize it's buffering using a simple string. I just rewrote that simple class to meet my needs and am quite happy.
-- Russell
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (6)
-
exarkun＠twistedmatrix.com -
Glyph -
Itamar Turner-Trauring -
Laurens Van Houtven -
Naveen Michaud-Agrawal -
Russell E. Owen