<div dir="ltr"><div>I'm seeing more of this today, just heads up in case you see miss-islington not working.</div><div><br></div>Seems like this is a known issue with celery + kombu 🙁<div><a href="https://github.com/celery/kombu/issues/1019">https://github.com/celery/kombu/issues/1019</a><br></div></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=abWFyaWF0dGEud2lqYXlhQGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=cd8d26f9-83f7-45e6-8fdd-2ca39e1d586a"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 10:02 AM Mariatta <<a href="mailto:mariatta@python.org">mariatta@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟<div>I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.</div><div><br></div><div>log:</div><div><br></div><div><pre style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">at=info method=POST path=<span style="color:rgb(0,128,128);font-weight:bold">"/" </span>host=<a href="http://miss-islington.herokuapp.com" target="_blank">miss-islington.herokuapp.com</a> request_id=<span style="color:rgb(0,0,255)">8</span>de611a2-<span style="color:rgb(0,0,255)">6112</span>-<span style="color:rgb(0,0,255)">4039</span>-a425-<span style="color:rgb(0,0,255)">97850</span>ac989b0 fwd=<span style="color:rgb(0,128,128);font-weight:bold">"140.82.115.15" </span>dyno=web<span style="color:rgb(0,0,255)">.1 </span>connect=<span style="color:rgb(0,0,255)">1</span>ms service=<span style="color:rgb(0,0,255)">1005</span>ms status=<span style="color:rgb(0,0,255)">200 </span>bytes=<span style="color:rgb(0,0,255)">168 </span>protocol=https<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: Traceback (most recent call last):<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py"</span>, line <span style="color:rgb(0,0,255)">600</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>send_packed_command<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self._sock.sendall(item)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: <span style="color:rgb(0,0,128)">TimeoutError</span>: [Errno <span style="color:rgb(0,0,255)">110</span>] Connection timed out<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: During handling of the above exception, another exception occurred:<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: Traceback (most recent call last):<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py"</span>, line <span style="color:rgb(0,0,255)">431</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>_reraise_as_library_errors<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: <span style="color:rgb(0,0,128);font-weight:bold">yield<br></span>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py"</span>, line <span style="color:rgb(0,0,255)">744</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>send_task<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self.backend.on_task_call(P, task_id)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py"</span>, line <span style="color:rgb(0,0,255)">265</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>on_task_call<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self.result_consumer.consume_from(task_id)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py"</span>, line <span style="color:rgb(0,0,255)">126</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>consume_from<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self._consume_from(task_id)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py"</span>, line <span style="color:rgb(0,0,255)">132</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>_consume_from<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self._pubsub.subscribe(key)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py"</span>, line <span style="color:rgb(0,0,255)">3096</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>subscribe<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: ret_val = self.execute_command(<span style="color:rgb(0,128,128);font-weight:bold">'SUBSCRIBE'</span>, *iterkeys(new_channels))<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py"</span>, line <span style="color:rgb(0,0,255)">3009</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>execute_command<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self._execute(connection, connection.send_command, *args)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py"</span>, line <span style="color:rgb(0,0,255)">3013</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>_execute<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: <span style="color:rgb(0,0,128);font-weight:bold">return </span>command(*args)<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py"</span>, line <span style="color:rgb(0,0,255)">620</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>send_command<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: self.send_packed_command(self.pack_command(*args))<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: File <span style="color:rgb(0,128,128);font-weight:bold">"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py"</span>, line <span style="color:rgb(0,0,255)">613</span>, <span style="color:rgb(0,0,128);font-weight:bold">in </span>send_packed_command<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: (errno, errmsg))<br>May <span style="color:rgb(0,0,255)">08 09</span>:<span style="color:rgb(0,0,255)">35</span>:<span style="color:rgb(0,0,255)">15 </span>miss-islington app/web<span style="color:rgb(0,0,255)">.1</span>: redis.exceptions.ConnectionError: Error <span style="color:rgb(0,0,255)">110 </span><span style="color:rgb(0,0,128);font-weight:bold">while </span>writing to socket. Connection timed out. </pre><pre style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt"><br></pre><pre style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt"><br></pre></div></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=abWFyaWF0dGEud2lqYXlhQGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=20f79ee7-3d5f-4203-90fa-287fa536478f"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith <<a href="mailto:greg@krypto.org" target="_blank">greg@krypto.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on <a href="https://github.com/python/cpython/pull/13192" target="_blank">https://github.com/python/cpython/pull/13192</a> assuming the usual backport latency is no more than a minute or two.<div><br></div><div>-gps</div></div>
_______________________________________________<br>
python-committers mailing list<br>
<a href="mailto:python-committers@python.org" target="_blank">python-committers@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-committers" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-committers</a><br>
Code of Conduct: <a href="https://www.python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">https://www.python.org/psf/codeofconduct/</a><br>
</blockquote></div>
</blockquote></div>