miss-islington backport pipeline is stalled?
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟 I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15" dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 600, in send_packed_command May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item) May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110] Connection timed out May 08 09:35:15 miss-islington app/web.1: During handling of the above exception, another exception occurred: May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors May 08 09:35:15 miss-islington app/web.1: yield May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P, task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call May 08 09:35:15 miss-islington app/web.1: self.result_consumer.consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe May 08 09:35:15 miss-islington app/web.1: ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009, in execute_command May 08 09:35:15 miss-islington app/web.1: self._execute(connection, connection.send_command, *args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute May 08 09:35:15 miss-islington app/web.1: return command(*args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 620, in send_command May 08 09:35:15 miss-islington app/web.1: self.send_packed_command(self.pack_command(*args)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 613, in send_packed_command May 08 09:35:15 miss-islington app/web.1: (errno, errmsg)) May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith greg@krypto.org wrote:
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
Would it make sense to work with the PSF infra staff so that miss-isslington is hooked up to the PSF Sentry account so folks can get email notifications and similar on unhandled exceptions?
Alex
On Wed, May 8, 2019 at 1:02 PM Mariatta mariatta@python.org wrote:
There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟 I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15" dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 600, in send_packed_command May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item) May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110] Connection timed out May 08 09:35:15 miss-islington app/web.1: During handling of the above exception, another exception occurred: May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors May 08 09:35:15 miss-islington app/web.1: yield May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P, task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call May 08 09:35:15 miss-islington app/web.1: self.result_consumer.consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe May 08 09:35:15 miss-islington app/web.1: ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009, in execute_command May 08 09:35:15 miss-islington app/web.1: self._execute(connection, connection.send_command, *args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute May 08 09:35:15 miss-islington app/web.1: return command(*args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 620, in send_command May 08 09:35:15 miss-islington app/web.1: self.send_packed_command(self.pack_command(*args)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 613, in send_packed_command May 08 09:35:15 miss-islington app/web.1: (errno, errmsg)) May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith greg@krypto.org wrote:
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
-- All that is necessary for evil to succeed is for good people to do nothing.
We don't have sentry (I will ask Ernest about it), but such errors are already sent to Zulip ( https://python.zulipchat.com/#narrow/stream/142203-workflow.2Fbot-api.20rate...) and core-workflow https://github.com/python/core-workflow/issues/257#issuecomment-490541831 But posting back to the PR itself will still be useful. ᐧ
On Wed, May 8, 2019 at 10:04 AM Alex Gaynor alex.gaynor@gmail.com wrote:
Would it make sense to work with the PSF infra staff so that miss-isslington is hooked up to the PSF Sentry account so folks can get email notifications and similar on unhandled exceptions?
Alex
On Wed, May 8, 2019 at 1:02 PM Mariatta mariatta@python.org wrote:
There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟 I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15" dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 600, in send_packed_command May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item) May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110] Connection timed out May 08 09:35:15 miss-islington app/web.1: During handling of the above exception, another exception occurred: May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors May 08 09:35:15 miss-islington app/web.1: yield May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P, task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call May 08 09:35:15 miss-islington app/web.1: self.result_consumer.consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe May 08 09:35:15 miss-islington app/web.1: ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009, in execute_command May 08 09:35:15 miss-islington app/web.1: self._execute(connection, connection.send_command, *args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute May 08 09:35:15 miss-islington app/web.1: return command(*args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 620, in send_command May 08 09:35:15 miss-islington app/web.1: self.send_packed_command(self.pack_command(*args)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 613, in send_packed_command May 08 09:35:15 miss-islington app/web.1: (errno, errmsg)) May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith greg@krypto.org wrote:
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
-- All that is necessary for evil to succeed is for good people to do nothing.
On Wed, May 8, 2019 at 1:04 PM Alex Gaynor alex.gaynor@gmail.com wrote:
Would it make sense to work with the PSF infra staff so that miss-isslington is hooked up to the PSF Sentry account so folks can get email notifications and similar on unhandled exceptions?
Yes it would. :)
-Brett
Alex
On Wed, May 8, 2019 at 1:02 PM Mariatta mariatta@python.org wrote:
There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟 I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15" dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 600, in send_packed_command May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item) May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110] Connection timed out May 08 09:35:15 miss-islington app/web.1: During handling of the above exception, another exception occurred: May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors May 08 09:35:15 miss-islington app/web.1: yield May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P, task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call May 08 09:35:15 miss-islington app/web.1: self.result_consumer.consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe May 08 09:35:15 miss-islington app/web.1: ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009, in execute_command May 08 09:35:15 miss-islington app/web.1: self._execute(connection, connection.send_command, *args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute May 08 09:35:15 miss-islington app/web.1: return command(*args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 620, in send_command May 08 09:35:15 miss-islington app/web.1: self.send_packed_command(self.pack_command(*args)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 613, in send_packed_command May 08 09:35:15 miss-islington app/web.1: (errno, errmsg)) May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith greg@krypto.org wrote:
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
-- All that is necessary for evil to succeed is for good people to do nothing.
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
I'm seeing more of this today, just heads up in case you see miss-islington not working.
Seems like this is a known issue with celery + kombu 🙁 https://github.com/celery/kombu/issues/1019 ᐧ
On Wed, May 8, 2019 at 10:02 AM Mariatta mariatta@python.org wrote:
There was an error from Redis. I think this is the first time I've seen it, so I don't have any resolution on how to fix it right now. 😟 I will look into handling the error and have miss-islington leave a comment in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15" dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 600, in send_packed_command May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item) May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110] Connection timed out May 08 09:35:15 miss-islington app/web.1: During handling of the above exception, another exception occurred: May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last): May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors May 08 09:35:15 miss-islington app/web.1: yield May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P, task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call May 08 09:35:15 miss-islington app/web.1: self.result_consumer.consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe May 08 09:35:15 miss-islington app/web.1: ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009, in execute_command May 08 09:35:15 miss-islington app/web.1: self._execute(connection, connection.send_command, *args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute May 08 09:35:15 miss-islington app/web.1: return command(*args) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 620, in send_command May 08 09:35:15 miss-islington app/web.1: self.send_packed_command(self.pack_command(*args)) May 08 09:35:15 miss-islington app/web.1: File "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line 613, in send_packed_command May 08 09:35:15 miss-islington app/web.1: (errno, errmsg)) May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith greg@krypto.org wrote:
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two.
-gps
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
participants (4)
-
Alex Gaynor -
Brett Cannon -
Gregory P. Smith -
Mariatta