[Twisted-Python] Proposal: changes to reactor.clientTCP
Background: Ivo pointed out there's no way to differentiate between "no such hostname" and "connection refused". For that matter, connectionFailed has nothing to do with the protocol - why are we doing a callback on the protocol at all? It shouldn't be created if no connection was made. Proposal: reactor.clientTCP accepts a protocol factory instead of a protocol, and only makes a protocol instance if connection was made. Additionally, it accepts an additional argument, failureCallback, which will get called with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye. Comments?
On Sat, Jun 15, 2002 at 06:04:29PM +0200, Itamar Shtull-Trauring wrote:
Background: Ivo pointed out there's no way to differentiate between "no such hostname" and "connection refused". For that matter, connectionFailed has nothing to do with the protocol - why are we doing a callback on the protocol at all? It shouldn't be created if no connection was made.
Proposal: reactor.clientTCP accepts a protocol factory instead of a protocol, and only makes a protocol instance if connection was made. Additionally, it accepts an additional argument, failureCallback, which will get called with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye.
Comments?
Some people don't see a point in providing these errors, so I'll illustrate the issue with some more examples: when binding to a local port, I'd like to know if it fails due to the port already being in use (EBUSY), permission denied (i.e. a port < 1024 when not being root), or some system failure that may be difficult to track down without an appropriate error (i.e. running out of fd's) I agree with itamar that if these errors aren't handled in the protocol implementation, failure to connect shouldn't be there either. Ivo -- Drs. I.R. van der Wijk -=- Brouwersgracht 132 Amaze Internet Services V.O.F. 1013 HA Amsterdam, NL -=- T +31-20-4688336 F +31-20-4688337 Linux/Web/Zope/SQL/MMBase W http://www.amaze.nl E info@amaze.nl Network Solutions W http://vanderwijk.info E ivo@amaze.nl Consultancy PGP http://vanderwijk.info/pgp -=-
From: Ivo van der Wijk <ivo@amaze.nl> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: Sun, 16 Jun 2002 01:06:51 +0200
On Sat, Jun 15, 2002 at 06:04:29PM +0200, Itamar Shtull-Trauring wrote:
Background: Ivo pointed out there's no way to differentiate between "no such hostname" and "connection refused". For that matter, connectionFailed has nothing to do with the protocol - why are we doing a callback on the protocol at all? It shouldn't be created if no connection was made.
Proposal: reactor.clientTCP accepts a protocol factory instead of a protocol, and only makes a protocol instance if connection was made. Additionally, it accepts an additional argument, failureCallback, which will get called with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye.
You know, I'm tempted to disagree with this because backwards-compatibility at this point is a really major argument to me at this point ... however, you guys make some really good arguments. It may even be possible, given this description, to make __init__, and not makeConnection, be the real "initializer" of a Protocol at this point. The inability to report these errors has also been an annoyance on the client-interface end of things for some time. There is an asymmetry between client and server-side connecting in IReactorTCP. In addition, Itamar's proposal would simultaneously eliminate the confusion currently surrounding Connectors, too; having a wrapper protocol *factory* would not pose the same efficiency concerns as a wrapper Protocol instance, since indirection would only happen when opening and closing the connection.
Comments?
Some people don't see a point in providing these errors, so I'll illustrate the issue with some more examples:
Also, being able to report these errors is important to judge at what level networking is misconfigured for the end-user, when they can't connect.
I agree with itamar that if these errors aren't handled in the protocol implementation, failure to connect shouldn't be there either.
After hearing this, so do I. Thank you for the bug report. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
Glyph Lefkowitz wrote:
with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye.
The error may be fine as a string instead of a status code - depends on the requirements.
In addition, Itamar's proposal would simultaneously eliminate the confusion currently surrounding Connectors, too; having a wrapper protocol *factory* would not pose the same efficiency concerns as a wrapper Protocol instance, since indirection would only happen when opening and closing the connection.
You still need to deal with connectionLost, which is in the Protocol still. But, if all Protocols have a factory, we could have them report to their factory that they are closed as part of their interface definition. But note that currently we do *not* require that protocols refer to a factory - it's merely a common implementation. Some protocols only refer to their service (e.g. in words, IIRC).
Also, being able to report these errors is important to judge at what level networking is misconfigured for the end-user, when they can't connect.
Well, for servers, just logging errors when attempting to listen on a port it, as we do now is acceptable - most servers work this way, where errors are logged. It's connections that really need error reporting, for clients. Of course, some client applications (p2p, FTP) listen on ports as well...
Itamar Shtull-Trauring <twisted@itamarst.org> writes:
Proposal: reactor.clientTCP accepts a protocol factory instead of a protocol, and only makes a protocol instance if connection was made. Additionally, it accepts an additional argument, failureCallback, which will get called with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye.
Please do not clobber the information given by the kernel in errno. Translating errno to 2 different failures is not nice. -- tv@{{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}
From: Tommi Virtanen <tv@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: 18 Jun 2002 21:12:52 +0300
Itamar Shtull-Trauring <twisted@itamarst.org> writes:
Proposal: reactor.clientTCP accepts a protocol factory instead of a protocol, and only makes a protocol instance if connection was made. Additionally, it accepts an additional argument, failureCallback, which will get called with a status code (CONNECTION_REFUSED, NO_SUCH_HOSTNAME) or something when the connection fails. Protocol.connectionFailed goes bye-bye.
Please do not clobber the information given by the kernel in errno. Translating errno to 2 different failures is not nice.
It needs to be clobbered, at least in some sense, for portability. Jython does not have errno and win32's support for it is very weak. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
From: Glyph Lefkowitz <glyph@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: Tue, 18 Jun 2002 19:38:46 -0500 (CDT)
From: Tommi Virtanen <tv@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: 18 Jun 2002 21:12:52 +0300
Please do not clobber the information given by the kernel in errno. Translating errno to 2 different failures is not nice.
It needs to be clobbered, at least in some sense, for portability.
Jython does not have errno and win32's support for it is very weak.
Reading this a few hours later, I think my tone was wrong and I was altogether too brief here. :-). To rephrase in a more constructive manner: I believe that it needs to be clobbered, and some information lost, because I don't see any other way to retain portability. Also, we're losing information already, since errno is used to report things that aren't "errors" in the Twisted worldview (or at least, not errors that the user can deal with directly). Jython and win32 both have error-reporting mechanisms; Jython's is exceptions, win32 is a sort of wacked-out informal errno. Can you think of a few primitive errors that we should support, rather than just the 2 types? (I'd rather use Failure than just integer error codes, personally.) I think that if we're going to add a new error reporting mechanism we need to do it carefully so that we don't miss the same kind of information that we did the last time we architected this API. (Protocol.connectionLost used to take an integer error code which was one of these two errors; that was a bad idea, too). Thanks for your feedback. Does anybody have a solution? -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
Glyph Lefkowitz <glyph@twistedmatrix.com> writes:
Please do not clobber the information given by the kernel in errno. Translating errno to 2 different failures is not nice.
It needs to be clobbered, at least in some sense, for portability.
Jython does not have errno and win32's support for it is very weak.
Reading this a few hours later, I think my tone was wrong and I was altogether too brief here. :-). To rephrase in a more constructive manner:
I believe that it needs to be clobbered, and some information lost, because I don't see any other way to retain portability. Also, we're losing information already, since errno is used to report things that aren't "errors" in the Twisted worldview (or at least, not errors that the user can deal with directly). Jython and win32 both have error-reporting mechanisms; Jython's is exceptions, win32 is a sort of wacked-out informal errno. Can you think of a few primitive errors that we should support, rather than just the 2 types? (I'd rather use Failure than just integer error codes, personally.)
I think that if we're going to add a new error reporting mechanism we need to do it carefully so that we don't miss the same kind of information that we did the last time we architected this API. (Protocol.connectionLost used to take an integer error code which was one of these two errors; that was a bad idea, too).
Thanks for your feedback. Does anybody have a solution?
Allow portable programs to get a "sanitized version", but allow my POSIX-only programs get errno. For what kind of sanitization one might want, you probably want to look at the man page for say connect(2) for all the possible error cases, and categorize them based on local/remote, temporary/permanent, etc., and create a "high-level error" for each class, with perhaps the string corresponding to the errno as extra info meant for users (and not if statements). But please do _not_ loose any information in the translation. Say people shouldn't look at the error objects errno attribute to decide what to do, as not all platforms provide it. But don't hide the actual error. Remember my background; I'm a kernel hacker, and I know the UNIX libc/kernel API. If I run a program that tries to e.g. connect with TCP, and it fails without giving me a good reason, I _will_ run it under strace -e trace=connect just to see whether connect failed, and with what errno. Operating systems without BSD sockets just aren't, and I write my programs to use fork anyway. My point -- don't decide for me, let me decide on my own. -- tv@{{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}
From: Tommi Virtanen <tv@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: 21 Jun 2002 22:04:08 +0300
Allow portable programs to get a "sanitized version", but allow my POSIX-only programs get errno. For what kind of sanitization one might want, you probably want to look at the man page for say connect(2) for all the possible error cases, and categorize them based on local/remote, temporary/permanent, etc., and create a "high-level error" for each class, with perhaps the string corresponding to the errno as extra info meant for users (and not if statements).
But please do _not_ loose any information in the translation. Say people shouldn't look at the error objects errno attribute to decide what to do, as not all platforms provide it. But don't hide the actual error.
Upon consideration I strongly agree that no information should be lost. So, let's have such a high-level error, and a .platformError attribute on the connection error object which is an object specific to the "OS" that you're running under. On POSIX this will be (or contain) an errno error code, but on Jython and elsewhere it will be whatever's appropriate. Does that work for you?
Remember my background; I'm a kernel hacker, and I know the UNIX libc/kernel API. If I run a program that tries to e.g. connect with TCP, and it fails without giving me a good reason, I _will_ run it under strace -e trace=connect just to see whether connect failed, and with what errno. Operating systems without BSD sockets just aren't, and I write my programs to use fork anyway. My point -- don't decide for me, let me decide on my own.
The reason I'm resistent to this is that I would like to encourage writing portable programs with Twisted, especially in cases where platform specificity doesn't add any functionality. On the other hand, I don't want to be Java here and try to _force_ programmers to write their programmers in a portable way when they really know better than I do what they want. Problem is, programmers rarely know what they want. For example; if you're using Twisted, why use fork? You ought to be using reactor.spawnProcess for non-portability-related reasons anyway; Python's refcounting doesn't work reliably with COW memory. Of course, there are reasons not to do this; of course, there are times when you just _need_ to get down to the OS and do something in a very, very platform specific way (sometimes even specific to your particular OS vendor, not just POSIX). This should not be the default. Handling an errno error code specifically should also be considered a *very* esoteric operation. I personally can't come up with any need to use it except debugging under a particular environment; I would far rather that people used to the POSIX API learned how to use the Twisted APIs to do POSIX-y things, so that their programs would be more portable. Balancing this with the need not to piss people off is difficult :-). I can't think of a way to say "DON'T USE .platformError UNLESS YOU REALLY NEED IT, WHICH YOU DON'T" without making everybody read the documentation, which they won't anyway. I can imagine many programs made non-portable by someone looking for the answer to the question "how do I find the errno of what happened" rather than the correct question "how do I handle a connection error". So I guess I'll provide the platform specific information and put my faith in the worldview of applications programmers; please try to give them good examples to look at :) -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
On Fri, Jun 21, 2002 at 06:59:23PM -0500, Glyph Lefkowitz wrote:
From: Tommi Virtanen <tv@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: 21 Jun 2002 22:04:08 +0300
But please do _not_ loose any information in the translation. Say people shouldn't look at the error objects errno attribute to decide what to do, as not all platforms provide it. But don't hide the actual error.
Upon consideration I strongly agree that no information should be lost.
So, let's have such a high-level error, and a .platformError attribute on the connection error object which is an object specific to the "OS" that you're running under. On POSIX this will be (or contain) an errno error code, but on Jython and elsewhere it will be whatever's appropriate. Does that work for you?
I don't care about the errno, but I do think that the corresponding error string (e.g. 'No route to host') should be available, so that you have something meaningful to display to the user, write in error logs, etc. Is it guaranteed that a socket.error's .args attribute will always be a 2-tuple of (errno, description) on all platforms? If so, I think that would make sense as the .platformError. -Andrew.
Glyph Lefkowitz <glyph@twistedmatrix.com> writes:
So, let's have such a high-level error, and a .platformError attribute on the connection error object which is an object specific to the "OS" that you're running under. On POSIX this will be (or contain) an errno error code, but on Jython and elsewhere it will be whatever's appropriate. Does that work for you?
Yes.
Balancing this with the need not to piss people off is difficult :-). I can't think of a way to say "DON'T USE .platformError UNLESS YOU REALLY NEED IT, WHICH YOU DON'T" without making everybody read the documentation, which they won't anyway. I can imagine many programs made non-portable by someone looking for the answer to the question "how do I find the errno of what happened" rather than the correct question "how do I handle a connection error".
Don't document platformError outside the source code, and there point out that it should not be used, and point to the real solution. This is a social problem, and technical methods will not be able to solve it. And, if you want me to write code using the high-level API, the high-level API needs to be able to tell me atleast 1) if the error is local or remote 2) if the error is temporary or permanent 3) how to report the error to the user without losing information -- tv@{{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}
From: Tommi Virtanen <tv@twistedmatrix.com> Subject: Re: [Twisted-Python] Proposal: changes to reactor.clientTCP Date: 23 Jun 2002 00:57:31 +0300
Glyph Lefkowitz <glyph@twistedmatrix.com> writes:
... Does that work for you?
Yes.
Cool; sounds like a plan.
Balancing this with the need not to piss people off is difficult :-). ...
Don't document platformError outside the source code, and there point out that it should not be used, and point to the real solution. This is a social problem, and technical methods will not be able to solve it.
I agree. This is a good suggestion.
And, if you want me to write code using the high-level API, the high-level API needs to be able to tell me atleast
1) if the error is local or remote 2) if the error is temporary or permanent 3) how to report the error to the user without losing information
This is also a good list. Does anyone else want to add to it? From what I can tell, this complete, so I think I'll do the refactoring with these things in mind. I'd like to get another release out before we start on this, though. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
participants (5)
-
Andrew Bennetts -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Ivo van der Wijk -
Tommi Virtanen