reactor.running False in (some) callback/errback
This has been driving me crazy for a while -- for some reason reactor.stop() in the _error errback in example below raises error.ReactorNotRunning. In order to stop the reactor, I have to do reactor.callWhenRunning(reactor.stop) (or I did reactor.callLater(0, ...) until I discovered callWhenRunning). In the example, I bind to a low port to make sure error is triggered. #!/usr/local/bin/python3 from twisted.internet import reactor, defer, protocol from twisted.internet.protocol import Factory from twisted.internet.endpoints import TCP4ServerEndpoint def _port(port): print('got', port) def _error(err): print('got err', err) print('is reactor running?', reactor.running) print('is reactor running?', (lambda: reactor.running)()) reactor.stop() # reactor.callWhenRunning(reactor.stop) d = TCP4ServerEndpoint(reactor, 123).listen(Factory()) d.addCallback(_port) d.addErrback(_error) d.addErrback(print) reactor.run() OTOH, in the following example reactor.stop() is stoppig the reactor properly: #!/usr/local/bin/python3 from twisted.internet import reactor, defer def cb(res): print('running?', reactor.running) if res == 'bar': raise Exception() reactor.stop() def eb(err): print('running?', reactor.running) print(err) reactor.stop() d = defer.Deferred() d.addCallback(cb) d.addErrback(eb) #reactor.callWhenRunning(d.callback, 'foo') reactor.callWhenRunning(d.callback, 'bar') reactor.run() Any ideas?
In general, if you have to call reactor.stop() manually like this in a script, you're probably doing too much manual fiddling with its lifecycle. I have not fully debugged this example, but I suspect it's just because the error is synchronous before the reactor starts. However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup, consider using <https://docs.twistedmatrix.com/en/stable/api/twisted.internet.task.html#reac...> instead and putting all your code (other than your imports) into your main function. This will result in far fewer confusing scenarios. Don't stop the reactor, just complete your coroutine and let `react()` shut it down for you when you're done. All that said: there are probably a bunch of improvements to the API and documentation that could make this easier, and the reactor's starting/stopping state shouldn't be such an opaque mess. It might be nice to eventually put it into an Automat state machine, for example, with some more explicit queries and fewer flags. So it's not like there's no work to be done here. But even if we did all that, the right way to write 99% of applications would still be to use something like a `twist` plugin, or a callable passed to `react`. Hope this helps, -g
• Glyph [2025-01-22 18:25]:
The provided scripts are just minimal examples which illustrate the problem I am having in a bigger piece of a daemon, which sets up a number of listeners on multiple interfaces and ports: if it is not able to bind to any of those, reactor should stop. It it does not stop by calling reactor.stop(), only by wrapping it in callWhenRunning or callLater. At first I just thought that it's just the way it is, but after I started the question WHY, it bothers me a little bit.
However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup,
There is no synchronous code in the examples though, apart from setting up some things which would be called once reactor is running. E.g. the first two defs set up callback and erroback functions -- isn't reactor.running supposed to be True when callbacks/errorbacks attached to a deferred are called? as an experiment, here's example which uses react(): #!/usr/local/bin/python3 from twisted.internet import reactor, defer, protocol from twisted.internet.protocol import Factory from twisted.internet.endpoints import TCP4ServerEndpoint from twisted.internet.task import react def _port(port): print('got', port) def _error(err): print('got err', err) print('is reactor running in error?', reactor.running) print('is reactor running in lambda?', (lambda: reactor.running)()) reactor.stop() # <- this fails # reactor.callWhenRunning(reactor.stop) # <- this works async def main(reactor): print('running in main, huh?', reactor.running) d = TCP4ServerEndpoint(reactor, 123).listen(Factory()) d.addCallback(_port) d.addErrback(_error) d.addErrback(print) react(main) it outputs this: running in main, huh? False got err [Failure instance: Traceback: <class 'twisted.internet.error.CannotListenError'>: Couldn't listen on any:123: [Errno 13] Permission denied. /usr/local/lib/python3.12/site-packages/twisted/internet/defer.py:155:execute /usr/local/lib/python3.12/site-packages/twisted/internet/posixbase.py:366:listenTCP /usr/local/lib/python3.12/site-packages/twisted/internet/tcp.py:1360:startListening ] is reactor running in error? False is reactor running in lambda? False [Failure instance: Traceback: <class 'twisted.internet.error.ReactorNotRunning'>: Can't stop reactor that isn't running. /usr/local/lib/python3.12/site-packages/twisted/internet/defer.py:1088:_runCallbacks /home/km/./foo:15:_error /usr/local/lib/python3.12/site-packages/twisted/internet/base.py:790:stop ]
Are these listeners set up later than startup time, though? If you do something like: async def main(): try: await myEndpoint.listen(...) except CannotListenError: print("can't listen, goodbye") return serveForever = Deferred() await serveForever you will achieve that early-exit you want.
However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup,
There is no synchronous code in the examples though, apart from setting up some things which would be called once reactor is running.
I guess "synchronous" is a slippery word here, "eager" might be a better one, but the salient difference is that TCP4ServerEndpoint.listen is going to call socket.socket.listen() immediately when you invoke it, and then it is going to raise an error and fire its Deferred immediately, which will result in running those callbacks early.
E.g. the first two defs set up callback and erroback functions -- isn't reactor.running supposed to be True when callbacks/errorbacks attached to a deferred are called?
No, the reactor has nothing to do with Deferred. Now that we have managed to deprecate and remove the ill-conceived "Deferred.setTimeout" there is no interaction between the two. Consider this example, with no reactor anywhere:
reactor.running means something extremely subtle about the run-state of the reactor, which is only loosely related to whether you will get a ReactorNotRunning error calling stop() or not; the reactor continues running for a while after stop is called, during which period .running is True, but .stop() will raise an exception. I realize the naming of the exception here is not the best. The rule is "don't call reactor.stop() more than once", not "don't call reactor.stop() if reactor.running is True".
What is happening here is that you get called, the reactor starts, then you immediately synchronously return without awaiting anything, shutting down the reactor. If you're using react(), it's react()'s job to call reactor.stop; all you can safely do is fire the Deferred (or complete the coroutine) returned from main().
Yep, this code stops the reactor twice: once in the code that concludes react() and once in _error. If you manage to fully exit the reactor before that second call to reactor.stop() happens (which is what I think your callWhenRunning does), then you won't see an exception. But that is an accident of scheduling, so I would not rely on it for correctness here. -g
Thank you for detailed feedback! • Glyph [2025-01-26 23:21]:
So I understand that although I write code in a non-synchronous way, some parts trigger the code running immediately.
so ReactorNotRunning can be raised both when reactor is not yet running, but also when it is running, but stop has been called?
On Jan 28, 2025, at 8:23 AM, km@krot.org wrote:
Thank you for detailed feedback!
Thanks for the questions! These are nuances that are difficult to capture in the documentation, even if it were excellent (and it isn't).
So I understand that although I write code in a non-synchronous way, some parts trigger the code running immediately.
Exactly. This is a conscious decision, because it allows the state to be affected immediately, which can make certain sorts of testing easier, but a subtle one.
so ReactorNotRunning can be raised both when reactor is not yet running, but also when it is running, but stop has been called?
Yes. The 'running' flag is incredibly subtle; as it is described here,<https://docs.twistedmatrix.com/en/stable/api/twisted.internet.interfaces.IRe...>:
A bool which is True from during startup to during shutdown and False the rest of the time.
Now, `stop`'s documentation https://docs.twistedmatrix.com/en/stable/api/twisted.internet.interfaces.IRe... is missing the explanation of when ReactorNotRunning is raised, and we should fix that. But the intention behind this exception is that if you call .stop() when its behavior might not do anything or might do something unexpected, it raises an exception so you can debug this problem with lifecycle state. There are shutdown hooks that your code might expect to run (such as those with addSystemEventTrigger) and so managing these expectations and getting loud errors is important, because otherwise you get a lot of "the reactor just didn't exit" bugs that are harder to observe. Thanks again, -g
In general, if you have to call reactor.stop() manually like this in a script, you're probably doing too much manual fiddling with its lifecycle. I have not fully debugged this example, but I suspect it's just because the error is synchronous before the reactor starts. However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup, consider using <https://docs.twistedmatrix.com/en/stable/api/twisted.internet.task.html#reac...> instead and putting all your code (other than your imports) into your main function. This will result in far fewer confusing scenarios. Don't stop the reactor, just complete your coroutine and let `react()` shut it down for you when you're done. All that said: there are probably a bunch of improvements to the API and documentation that could make this easier, and the reactor's starting/stopping state shouldn't be such an opaque mess. It might be nice to eventually put it into an Automat state machine, for example, with some more explicit queries and fewer flags. So it's not like there's no work to be done here. But even if we did all that, the right way to write 99% of applications would still be to use something like a `twist` plugin, or a callable passed to `react`. Hope this helps, -g
• Glyph [2025-01-22 18:25]:
The provided scripts are just minimal examples which illustrate the problem I am having in a bigger piece of a daemon, which sets up a number of listeners on multiple interfaces and ports: if it is not able to bind to any of those, reactor should stop. It it does not stop by calling reactor.stop(), only by wrapping it in callWhenRunning or callLater. At first I just thought that it's just the way it is, but after I started the question WHY, it bothers me a little bit.
However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup,
There is no synchronous code in the examples though, apart from setting up some things which would be called once reactor is running. E.g. the first two defs set up callback and erroback functions -- isn't reactor.running supposed to be True when callbacks/errorbacks attached to a deferred are called? as an experiment, here's example which uses react(): #!/usr/local/bin/python3 from twisted.internet import reactor, defer, protocol from twisted.internet.protocol import Factory from twisted.internet.endpoints import TCP4ServerEndpoint from twisted.internet.task import react def _port(port): print('got', port) def _error(err): print('got err', err) print('is reactor running in error?', reactor.running) print('is reactor running in lambda?', (lambda: reactor.running)()) reactor.stop() # <- this fails # reactor.callWhenRunning(reactor.stop) # <- this works async def main(reactor): print('running in main, huh?', reactor.running) d = TCP4ServerEndpoint(reactor, 123).listen(Factory()) d.addCallback(_port) d.addErrback(_error) d.addErrback(print) react(main) it outputs this: running in main, huh? False got err [Failure instance: Traceback: <class 'twisted.internet.error.CannotListenError'>: Couldn't listen on any:123: [Errno 13] Permission denied. /usr/local/lib/python3.12/site-packages/twisted/internet/defer.py:155:execute /usr/local/lib/python3.12/site-packages/twisted/internet/posixbase.py:366:listenTCP /usr/local/lib/python3.12/site-packages/twisted/internet/tcp.py:1360:startListening ] is reactor running in error? False is reactor running in lambda? False [Failure instance: Traceback: <class 'twisted.internet.error.ReactorNotRunning'>: Can't stop reactor that isn't running. /usr/local/lib/python3.12/site-packages/twisted/internet/defer.py:1088:_runCallbacks /home/km/./foo:15:_error /usr/local/lib/python3.12/site-packages/twisted/internet/base.py:790:stop ]
Are these listeners set up later than startup time, though? If you do something like: async def main(): try: await myEndpoint.listen(...) except CannotListenError: print("can't listen, goodbye") return serveForever = Deferred() await serveForever you will achieve that early-exit you want.
However, rather than carefully managing the reactor's startup state like this where half your code runs synchronously before reactor startup,
There is no synchronous code in the examples though, apart from setting up some things which would be called once reactor is running.
I guess "synchronous" is a slippery word here, "eager" might be a better one, but the salient difference is that TCP4ServerEndpoint.listen is going to call socket.socket.listen() immediately when you invoke it, and then it is going to raise an error and fire its Deferred immediately, which will result in running those callbacks early.
E.g. the first two defs set up callback and erroback functions -- isn't reactor.running supposed to be True when callbacks/errorbacks attached to a deferred are called?
No, the reactor has nothing to do with Deferred. Now that we have managed to deprecate and remove the ill-conceived "Deferred.setTimeout" there is no interaction between the two. Consider this example, with no reactor anywhere:
reactor.running means something extremely subtle about the run-state of the reactor, which is only loosely related to whether you will get a ReactorNotRunning error calling stop() or not; the reactor continues running for a while after stop is called, during which period .running is True, but .stop() will raise an exception. I realize the naming of the exception here is not the best. The rule is "don't call reactor.stop() more than once", not "don't call reactor.stop() if reactor.running is True".
What is happening here is that you get called, the reactor starts, then you immediately synchronously return without awaiting anything, shutting down the reactor. If you're using react(), it's react()'s job to call reactor.stop; all you can safely do is fire the Deferred (or complete the coroutine) returned from main().
Yep, this code stops the reactor twice: once in the code that concludes react() and once in _error. If you manage to fully exit the reactor before that second call to reactor.stop() happens (which is what I think your callWhenRunning does), then you won't see an exception. But that is an accident of scheduling, so I would not rely on it for correctness here. -g
Thank you for detailed feedback! • Glyph [2025-01-26 23:21]:
So I understand that although I write code in a non-synchronous way, some parts trigger the code running immediately.
so ReactorNotRunning can be raised both when reactor is not yet running, but also when it is running, but stop has been called?
On Jan 28, 2025, at 8:23 AM, km@krot.org wrote:
Thank you for detailed feedback!
Thanks for the questions! These are nuances that are difficult to capture in the documentation, even if it were excellent (and it isn't).
So I understand that although I write code in a non-synchronous way, some parts trigger the code running immediately.
Exactly. This is a conscious decision, because it allows the state to be affected immediately, which can make certain sorts of testing easier, but a subtle one.
so ReactorNotRunning can be raised both when reactor is not yet running, but also when it is running, but stop has been called?
Yes. The 'running' flag is incredibly subtle; as it is described here,<https://docs.twistedmatrix.com/en/stable/api/twisted.internet.interfaces.IRe...>:
A bool which is True from during startup to during shutdown and False the rest of the time.
Now, `stop`'s documentation https://docs.twistedmatrix.com/en/stable/api/twisted.internet.interfaces.IRe... is missing the explanation of when ReactorNotRunning is raised, and we should fix that. But the intention behind this exception is that if you call .stop() when its behavior might not do anything or might do something unexpected, it raises an exception so you can debug this problem with lifecycle state. There are shutdown hooks that your code might expect to run (such as those with addSystemEventTrigger) and so managing these expectations and getting loud errors is important, because otherwise you get a lot of "the reactor just didn't exit" bugs that are harder to observe. Thanks again, -g
participants (3)
-
Glyph
-
Kirill Miazine
-
km@krot.org