[Twisted-Python] Thread Cancelled
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error
log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload))
deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCall
def _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco robert.difalco@gmail.com wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved.
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection. https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau colin.dunklau@gmail.com wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco robert.difalco@gmail.com wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s... https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155
The method above is "doRead". Which calls this: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2... https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239
I guess if If socket.rcv() returns an empty string it simply closes the connection. https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2... https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau <colin.dunklau@gmail.com mailto:colin.dunklau@gmail.com> wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco <robert.difalco@gmail.com mailto:robert.difalco@gmail.com> wrote: Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.: def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com mailto:Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph glyph@twistedmatrix.com wrote:
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau colin.dunklau@gmail.com wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco robert.difalco@gmail.com wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
If you're dealing with lots of clients on the public internet, sometimes this is just gonna happen, for a variety of reasons; it's normal. We would welcome better error reporting for this scenario so it doesn't require the kind of debugging you just did :-).
-g
On Jan 24, 2021, at 2:58 PM, Robert DiFalco robert.difalco@gmail.com wrote:
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph <glyph@twistedmatrix.com mailto:glyph@twistedmatrix.com> wrote: While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco <robert.difalco@gmail.com mailto:robert.difalco@gmail.com> wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s... https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155
The method above is "doRead". Which calls this: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2... https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239
I guess if If socket.rcv() returns an empty string it simply closes the connection. https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2... https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau <colin.dunklau@gmail.com mailto:colin.dunklau@gmail.com> wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco <robert.difalco@gmail.com mailto:robert.difalco@gmail.com> wrote: Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.: def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com mailto:Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com mailto:Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com mailto:Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Well, I've dealt with this issue with other languages. Not sure how to deal with it in Klein/Twisted. This operation is idempotent so I suppose what I'd like to happen is have the whole chain of deferred's succeed but then just not be able to write the response to the socket -- but not interrupt the chain. Unfortunately, what I really need is my operations to be atomic -- introduce a two phase commit or some such. But that's too big of a job on this legacy code base for now. Right now I'd be content if I could save the database record and then send the SQS message to AWS and not have that SQS send interrupted if the client closed the socket.. Is there a simple way to achieve that? If I got the request body I'm good, I don't care if I can't write the response.
One other thing that would be nice is to know why a deferred was canceled. If the client close a connection I might like to ignore the cancel, but I think what is happening is that Twisted is pretty smart. So it either knows I'm going to make a write to SQS using Boto that is inbound, so it is somehow able to cancel that operation -- perhaps more likely the client closed the connection and the connection canceled defers after I called deferToThread but before a thread was available to run on. Either way, I'd like them all to run, and just fail to write the final response from the end of the chain.
Sorry, too many words.
On Sun, Jan 24, 2021 at 3:22 PM Glyph glyph@twistedmatrix.com wrote:
If you're dealing with lots of clients on the public internet, sometimes this is just gonna happen, for a variety of reasons; it's normal. We would welcome better error reporting for this scenario so it doesn't require the kind of debugging you just did :-).
-g
On Jan 24, 2021, at 2:58 PM, Robert DiFalco robert.difalco@gmail.com wrote:
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph glyph@twistedmatrix.com wrote:
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau colin.dunklau@gmail.com wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco < robert.difalco@gmail.com> wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
So I have a simple question from all this. Is there a twisted idiom I can use so that deferred returned from a Klein route are not canceled if the client resets the connection? It's fine if it's before I've gotten the request body, but once I've gotten the request body I want all deferreds to succeed. I only wan't the streaming of the final result to fail.
On Sun, Jan 24, 2021 at 7:40 PM Robert DiFalco robert.difalco@gmail.com wrote:
Well, I've dealt with this issue with other languages. Not sure how to deal with it in Klein/Twisted. This operation is idempotent so I suppose what I'd like to happen is have the whole chain of deferred's succeed but then just not be able to write the response to the socket -- but not interrupt the chain. Unfortunately, what I really need is my operations to be atomic -- introduce a two phase commit or some such. But that's too big of a job on this legacy code base for now. Right now I'd be content if I could save the database record and then send the SQS message to AWS and not have that SQS send interrupted if the client closed the socket.. Is there a simple way to achieve that? If I got the request body I'm good, I don't care if I can't write the response.
One other thing that would be nice is to know why a deferred was canceled. If the client close a connection I might like to ignore the cancel, but I think what is happening is that Twisted is pretty smart. So it either knows I'm going to make a write to SQS using Boto that is inbound, so it is somehow able to cancel that operation -- perhaps more likely the client closed the connection and the connection canceled defers after I called deferToThread but before a thread was available to run on. Either way, I'd like them all to run, and just fail to write the final response from the end of the chain.
Sorry, too many words.
On Sun, Jan 24, 2021 at 3:22 PM Glyph glyph@twistedmatrix.com wrote:
If you're dealing with lots of clients on the public internet, sometimes this is just gonna happen, for a variety of reasons; it's normal. We would welcome better error reporting for this scenario so it doesn't require the kind of debugging you just did :-).
-g
On Jan 24, 2021, at 2:58 PM, Robert DiFalco robert.difalco@gmail.com wrote:
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph glyph@twistedmatrix.com wrote:
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau colin.dunklau@gmail.com wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco < robert.difalco@gmail.com> wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, Jan 25, 2021 at 5:14 PM Robert DiFalco robert.difalco@gmail.com wrote:
So I have a simple question from all this. Is there a twisted idiom I can use so that deferred returned from a Klein route are not canceled if the client resets the connection? It's fine if it's before I've gotten the request body, but once I've gotten the request body I want all deferreds to succeed. I only wan't the streaming of the final result to fail.
As far as I know, you can't make Klein not cancel that Deferred. However, you can return a different Deferred.
def ignore_cancellation(d):
ignore_d = Deferred()
d.chainDeferred(ignore_d)
return ignore_d
Wherever you're returning a Deferred to Klein now, if you return ignore_cancellation(that_deferred) instead then either:
- it will run to completion and deliver its result to ignore_d which will deliver it to Klein; or - Klein will cancel it and the cancellation will not propagate to the original Deferred so that operation can complete (when it does, it will deliver the result to the cancelled Deferred which will drop it on the floor)
The documentation for Deferred cancellation could be a little bit clearer on how cancellation works for Deferreds that are related in various ways, including this way. I'm not sure how you would discover this behavior except for reading the implementation or doing experiments. The way that might make sense to think about it, though, is that results only propagate in one direction down the chain - from d to ignore_d - they never flow the other way. So anything that happens to ignore_d cannot affect d.
Jean-Paul
On Sun, Jan 24, 2021 at 7:40 PM Robert DiFalco robert.difalco@gmail.com wrote:
Well, I've dealt with this issue with other languages. Not sure how to deal with it in Klein/Twisted. This operation is idempotent so I suppose what I'd like to happen is have the whole chain of deferred's succeed but then just not be able to write the response to the socket -- but not interrupt the chain. Unfortunately, what I really need is my operations to be atomic -- introduce a two phase commit or some such. But that's too big of a job on this legacy code base for now. Right now I'd be content if I could save the database record and then send the SQS message to AWS and not have that SQS send interrupted if the client closed the socket.. Is there a simple way to achieve that? If I got the request body I'm good, I don't care if I can't write the response.
One other thing that would be nice is to know why a deferred was canceled. If the client close a connection I might like to ignore the cancel, but I think what is happening is that Twisted is pretty smart. So it either knows I'm going to make a write to SQS using Boto that is inbound, so it is somehow able to cancel that operation -- perhaps more likely the client closed the connection and the connection canceled defers after I called deferToThread but before a thread was available to run on. Either way, I'd like them all to run, and just fail to write the final response from the end of the chain.
Sorry, too many words.
On Sun, Jan 24, 2021 at 3:22 PM Glyph glyph@twistedmatrix.com wrote:
If you're dealing with lots of clients on the public internet, sometimes this is just gonna happen, for a variety of reasons; it's normal. We would welcome better error reporting for this scenario so it doesn't require the kind of debugging you just did :-).
-g
On Jan 24, 2021, at 2:58 PM, Robert DiFalco robert.difalco@gmail.com wrote:
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph glyph@twistedmatrix.com wrote:
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau colin.dunklau@gmail.com wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco < robert.difalco@gmail.com> wrote:
Hi, I apologize this question is a little vague. I'm looking for pointers. I have a klein route that makes an underlying deferToThread call with a simple single thread (an IO based sync call I can't change, a boto3 sqs write). The thread pool is simple, just a couple of threads, nothing fancy.
VERY rarely it appears that Klein cancels the thread. What techniques can I use to figure out why my thread is being Canceled? There's nothing in the failure to tell me "who, why, or where" it was canceled. Also, I cannot get this down to a reproducible case, but here's the boto3 sqs wrapper, this fall back works fine, but it's a band-aide for an error I can't track down.:
def write(self, payload): """ Write message to SQS async from thread pool. If twisted cancels the thread, instead write synchronously.
def _retrySynchronously(error): if error.type != CancelledError: return error log.warn("Async SQS write cancelled. Calling synchronously.") return defer.succeed(self._writeSyncFallback(payload)) deferredCall = self._deferToThread(self.sqs.write, payload) deferredCall.addErrback(_retrySynchronously) return deferredCalldef _writeSyncFallback(self, payload): return self.sqs.write(payload)
The _deferToThread call just uses my own thread pool with 2 threads, but is otherwise stock.
Is there a level of logging I'm missing or some other thing that would tell me why the thread is being canceled? The retry works great and Klein does not return an error from the route.
Thanks in advance.
I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
This is great Jean-Paul, see so much for me to learn. I didn't realize that you could break the propagation of Cancel errors like that. thanks!
On Mon, Jan 25, 2021 at 3:51 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Mon, Jan 25, 2021 at 5:14 PM Robert DiFalco robert.difalco@gmail.com wrote:
So I have a simple question from all this. Is there a twisted idiom I can use so that deferred returned from a Klein route are not canceled if the client resets the connection? It's fine if it's before I've gotten the request body, but once I've gotten the request body I want all deferreds to succeed. I only wan't the streaming of the final result to fail.
As far as I know, you can't make Klein not cancel that Deferred. However, you can return a different Deferred.
def ignore_cancellation(d):
ignore_d = Deferred() d.chainDeferred(ignore_d) return ignore_dWherever you're returning a Deferred to Klein now, if you return ignore_cancellation(that_deferred) instead then either:
- it will run to completion and deliver its result to ignore_d which
will deliver it to Klein; or
- Klein will cancel it and the cancellation will not propagate to the
original Deferred so that operation can complete (when it does, it will deliver the result to the cancelled Deferred which will drop it on the floor)
The documentation for Deferred cancellation could be a little bit clearer on how cancellation works for Deferreds that are related in various ways, including this way. I'm not sure how you would discover this behavior except for reading the implementation or doing experiments. The way that might make sense to think about it, though, is that results only propagate in one direction down the chain - from d to ignore_d - they never flow the other way. So anything that happens to ignore_d cannot affect d.
Jean-Paul
On Sun, Jan 24, 2021 at 7:40 PM Robert DiFalco robert.difalco@gmail.com wrote:
Well, I've dealt with this issue with other languages. Not sure how to deal with it in Klein/Twisted. This operation is idempotent so I suppose what I'd like to happen is have the whole chain of deferred's succeed but then just not be able to write the response to the socket -- but not interrupt the chain. Unfortunately, what I really need is my operations to be atomic -- introduce a two phase commit or some such. But that's too big of a job on this legacy code base for now. Right now I'd be content if I could save the database record and then send the SQS message to AWS and not have that SQS send interrupted if the client closed the socket.. Is there a simple way to achieve that? If I got the request body I'm good, I don't care if I can't write the response.
One other thing that would be nice is to know why a deferred was canceled. If the client close a connection I might like to ignore the cancel, but I think what is happening is that Twisted is pretty smart. So it either knows I'm going to make a write to SQS using Boto that is inbound, so it is somehow able to cancel that operation -- perhaps more likely the client closed the connection and the connection canceled defers after I called deferToThread but before a thread was available to run on. Either way, I'd like them all to run, and just fail to write the final response from the end of the chain.
Sorry, too many words.
On Sun, Jan 24, 2021 at 3:22 PM Glyph glyph@twistedmatrix.com wrote:
If you're dealing with lots of clients on the public internet, sometimes this is just gonna happen, for a variety of reasons; it's normal. We would welcome better error reporting for this scenario so it doesn't require the kind of debugging you just did :-).
-g
On Jan 24, 2021, at 2:58 PM, Robert DiFalco robert.difalco@gmail.com wrote:
That makes sense, thank you. A timeout seems unlikely but maybe the client is closing the connection due to a network issue. This is an extremely rare occurrence.
On Sun, Jan 24, 2021 at 2:41 PM Glyph glyph@twistedmatrix.com wrote:
While a socket is open and receiving data, recv() will either give you a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if no bytes are ready. A result of zero bytes (the empty string) means "end of file" - the other end has closed the socket.
So what's happening here is your client is timing out or otherwise canceling its request by closing the socket, and this is the correct, intentional response to that scenario.
-g
On Jan 24, 2021, at 11:57 AM, Robert DiFalco robert.difalco@gmail.com wrote:
You're absolutely right, I meant "cancel the deferred". I don't grok server sockets very well so maybe someone can help. But apparently, klein does a .doRead from our server socket (getting the request from the client?). This returns a "why" of "connection done" so that closes the connection before we have written our response to the client, and that cancels the deferred SQS write.
https://github.com/racker/python-twisted-core/blob/master/twisted/internet/s...
The method above is "doRead". Which calls this:
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
I guess if If socket.rcv() returns an empty string it simply closes the connection.
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L2...
Is that normal? I mean I guess it must be but then why is the read getting an empty string and closing the connection? I can't really account for it? Some kind of back pressure due to load?
Thanks for any thoughts.
On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau < colin.dunklau@gmail.com> wrote:
On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco < robert.difalco@gmail.com> wrote:
> Hi, I apologize this question is a little vague. I'm looking for > pointers. I have a klein route that makes an underlying deferToThread call > with a simple single thread (an IO based sync call I can't change, a boto3 > sqs write). The thread pool is simple, just a couple of threads, nothing > fancy. > > VERY rarely it appears that Klein cancels the thread. What > techniques can I use to figure out why my thread is being Canceled? There's > nothing in the failure to tell me "who, why, or where" it was canceled. > Also, I cannot get this down to a reproducible case, but here's the boto3 > sqs wrapper, this fall back works fine, but it's a band-aide for an error I > can't track down.: > > def write(self, payload): > """ > Write message to SQS async from thread pool. If twisted cancels the > thread, instead write synchronously. > > def _retrySynchronously(error): > if error.type != CancelledError: > return error > > log.warn("Async SQS write cancelled. Calling synchronously.") > return defer.succeed(self._writeSyncFallback(payload)) > > deferredCall = self._deferToThread(self.sqs.write, payload) > deferredCall.addErrback(_retrySynchronously) > return deferredCall > > def _writeSyncFallback(self, payload): > return self.sqs.write(payload) > > The _deferToThread call just uses my own thread pool with 2 threads, > but is otherwise stock. > > Is there a level of logging I'm missing or some other thing that > would tell me why the thread is being canceled? The retry works great and > Klein does not return an error from the route. > > Thanks in advance. > > I think we'll need to see more code for this, specifically the caller of that `write` method, and its callers, etc. Note that the thread itself isn't being cancelled, the Deferred you get from _deferToThread is... so you'll most likely need to find out what code interacts with that object to progress in isolating this.
In my quick skim of the deferToThread and ThreadPool source, I can't find any explicit cancellations. While that certainly doesn't rule it out, it does make me think you're more likely to find the issue by inspecting the callers involved. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (4)
-
Colin Dunklau -
Glyph -
Jean-Paul Calderone -
Robert DiFalco