[Twisted-Python] doWrite on twisted.internet.tcp.Port
Hey guys, Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port. I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I'm coming back to it. I'll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted: Referencing Jean-Paul's comment at the end of ticket 4759, I read you shouldn't fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective? Context: I only hit this problem on Linux, not Windows. The software project (github.com/opencontentplatform/ocp) has been a lot of fun, especially with walking the tight rope in using multi-processing, multi-threading, and Twisted reactors. The main controller process kicks off about 10 child processes, each doing different types of work. In general though, the child processes individually start a Twisted reactor, connect to Kafka, connect to a database, use a shared REST API, and some listen for connecting clients to accomplish work. I test on Linux about once a year, so too many changes to rollback and figure out that way. It was working on Linux 2 years ago, but last year's testing and current testing, receive the doWrite error. It continues running fine on Windows. I've gone back about 2 years of versions with Python3, Twisted, and dependent libs. on both Windows and Linux. Every version change yields the same result - continues to work on Windows and continues to hit the error on Linux. So something I added has caused Linux to throw that error. I'm not explicitly sharing much between the main process and the sub processes. They are spun up with (1) a shared multiprocessing.Event() - to have the main process shut the children down, (2) their own unique multiprocessing.Event() - to have the child processes notify back to the parent, and (3) a deep copy of a dictionary (containing a bunch of settings that remain constant). The main process uses twisted.logger, but for testing I strip that out to remove any twisted imports in the main process. So I'm not importing anything Twisted in the main process, and I don't believe I'm explicitly sharing something I shouldn't. Seems like something is implicitly being exposed/shared across Linux child processes, that aren't on Windows. The tracebacks come through on Linux (sometimes randomly), on the console of the parent controller process. No need to paste here, since it's the same as the ticket shows. I can't reliably reproduce the problem, but I know if I stop/start client connections (ServerFactory/twisted.internet.interfaces.IReactorTCP and twisted.internet.protocol.ReconnectingClientFactory) then it will eventually happen. I need to devote time at whittling down the code and attempting to create a reliable test case. if even possible. The error is slightly different when running HTTP vs HTTPS, but the story is the same. It cripples whatever child process that hits it, from doing much of anything thereafter. Not much luck with troubleshooting. The tracebacks do not include a calling function from my code, to tell me where to start. And it happens across different child process types, so not the same one each time. When I throw debuggers on the child processes, the problem seems to mask itself. Well, at least I didn't hit the problem over the last 3 days using pudb and stepping through code at breakpoints. I'm absolutely open to suggestions for troubleshooting, but first wanted to take a HUGE step back and ask a design question regarding Twisted and multiprocessing. Thanks!
On Fri, Sep 11, 2020 at 1:34 PM <chris@cmsconstruct.com> wrote:
Hey guys,
Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port.
I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I’m coming back to it. I’ll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted:
Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective?
To answer the asked question, I don't think there is rigorous (or even casual) testing of very much of Twisted in the context of "some Twisted code has been loaded into memory and then the process forked". So while it seems like a reasonable thing, I wouldn't say there's currently much effort being put towards making it a *supported* usage of Twisted. Of course this can change at almost any moment if someone decides to commit the effort. To dig a bit further into the specific problem, even if you only *import* the reactor in the parent process and then fork a child and try to start the reactor in the child, I strongly suspect epollreactor will break. This is because the epoll object is created by reactor instantiation (as opposed to being delayed until the reactor is run). epoll objects have a lot of weird behavior. See the *Questions and Answers* section of the epoll(7) man page. I don't know if this is the cause of your particular expression of these symptoms (it certainly doesn't apply to the *original* bug report which is on FreeBSD where there is no epoll) but it's at least *a possible* cause. This could probably be fixed in Twisted by only creating the epoll object when run is called. There's nothing particularly difficult about that change but it does involve touching a lot of the book-keeping logic since that all assumes it can register file descriptors before the reactor is started (think reactor.listenTCP(...); reactor.run()). I'm not sure but it may also be the case that only delaying creation of the *waker* until the reactor starts would also fix this. This is because as long as the epoll object remains empty a lot of the weird behavior is avoided and the waker is probably the only thing that actually gets added to it if you're just *importing* the reactor but not *running* it before forking. Alternatively, your application *should* be able to fix it by studiously avoiding the import of twisted.internet.reactor (directly or transitively, of course). You could add some kind of assertion about the state of sys.modules immediately before your forking code to develop some confidence you've managed this. And if this is really an epoll problem then switching to poll or select reactor would also presumably get rid of the issue. Jean-Paul
Context:
I only hit this problem on Linux, not Windows.
The software project (github.com/opencontentplatform/ocp) has been a lot of fun, especially with walking the tight rope in using multi-processing, multi-threading, and Twisted reactors. The main controller process kicks off about 10 child processes, each doing different types of work. In general though, the child processes individually start a Twisted reactor, connect to Kafka, connect to a database, use a shared REST API, and some listen for connecting clients to accomplish work.
I test on Linux about once a year, so too many changes to rollback and figure out that way. It was working on Linux 2 years ago, but last year’s testing and current testing, receive the doWrite error. It continues running fine on Windows. I’ve gone back about 2 years of versions with Python3, Twisted, and dependent libs… on both Windows and Linux. Every version change yields the same result - continues to work on Windows and continues to hit the error on Linux. So something I added has caused Linux to throw that error.
I’m not explicitly sharing much between the main process and the sub processes. They are spun up with (1) a shared multiprocessing.Event() - to have the main process shut the children down, (2) their own unique multiprocessing.Event() - to have the child processes notify back to the parent, and (3) a deep copy of a dictionary (containing a bunch of settings that remain constant). The main process uses twisted.logger, but for testing I strip that out to remove any twisted imports in the main process. So I’m not importing anything Twisted in the main process, and I don’t believe I’m explicitly sharing something I shouldn’t. Seems like something is implicitly being exposed/shared across Linux child processes, that aren’t on Windows.
The tracebacks come through on Linux (sometimes randomly), on the console of the parent controller process. No need to paste here, since it’s the same as the ticket shows. I can’t reliably reproduce the problem, but I know if I stop/start client connections (ServerFactory/twisted.internet.interfaces.IReactorTCP and twisted.internet.protocol.ReconnectingClientFactory) then it will eventually happen. I need to devote time at whittling down the code and attempting to create a reliable test case… if even possible.
The error is slightly different when running HTTP vs HTTPS, but the story is the same. It cripples whatever child process that hits it, from doing much of anything thereafter. Not much luck with troubleshooting. The tracebacks do not include a calling function from my code, to tell me where to start. And it happens across different child process types, so not the same one each time. When I throw debuggers on the child processes, the problem seems to mask itself. Well, at least I didn’t hit the problem over the last 3 days using pudb and stepping through code at breakpoints.
I’m absolutely open to suggestions for troubleshooting, but first wanted to take a HUGE step back and ask a design question regarding Twisted and multiprocessing.
Thanks! _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Jean-Paul, Thank you very much for the detailed answer. And my appologies for not providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD as the ticket discussed. Looks like Red Hat (EL 7.6) is using epoll reactor, and the Windows side is using the select reactor. Thanks for the direction on checking out sys.modules. To avoid the reactor being loaded in the parent process, I can presumably move twisted imports within the multiprocessing child modules (from top, down into the run() functions). I will see how far I need to go (e.g. if I can continue using Twisted’s JSON logging or if absolutely everything should be isolated until after child process startup). But knowing I need to head that direction for epoll or other potential reactor conflicts - is very helpful. Reminds me of the GI Joe cartoon in the early 1980’s that would end with, “knowing is half the battle.” -Chris From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Jean-Paul Calderone Sent: Friday, September 11, 2020 1:28 PM To: Twisted general discussion <twisted-python@twistedmatrix.com> Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port On Fri, Sep 11, 2020 at 1:34 PM <chris@cmsconstruct.com <mailto:chris@cmsconstruct.com> > wrote: Hey guys, Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port. I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I’m coming back to it. I’ll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted: Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective? To answer the asked question, I don't think there is rigorous (or even casual) testing of very much of Twisted in the context of "some Twisted code has been loaded into memory and then the process forked". So while it seems like a reasonable thing, I wouldn't say there's currently much effort being put towards making it a supported usage of Twisted. Of course this can change at almost any moment if someone decides to commit the effort. To dig a bit further into the specific problem, even if you only import the reactor in the parent process and then fork a child and try to start the reactor in the child, I strongly suspect epollreactor will break. This is because the epoll object is created by reactor instantiation (as opposed to being delayed until the reactor is run). epoll objects have a lot of weird behavior. See the Questions and Answers section of the epoll(7) man page. I don't know if this is the cause of your particular expression of these symptoms (it certainly doesn't apply to the original bug report which is on FreeBSD where there is no epoll) but it's at least a possible cause. This could probably be fixed in Twisted by only creating the epoll object when run is called. There's nothing particularly difficult about that change but it does involve touching a lot of the book-keeping logic since that all assumes it can register file descriptors before the reactor is started (think reactor.listenTCP(...); reactor.run()). I'm not sure but it may also be the case that only delaying creation of the waker until the reactor starts would also fix this. This is because as long as the epoll object remains empty a lot of the weird behavior is avoided and the waker is probably the only thing that actually gets added to it if you're just importing the reactor but not running it before forking. Alternatively, your application should be able to fix it by studiously avoiding the import of twisted.internet.reactor (directly or transitively, of course). You could add some kind of assertion about the state of sys.modules immediately before your forking code to develop some confidence you've managed this. And if this is really an epoll problem then switching to poll or select reactor would also presumably get rid of the issue. Jean-Paul
[Closing the loop for future contextual searches] This solved the problem. Moving all Twisted reactor-related imports inside overloaded multiprocessing.Process.run() functions, allows a single controller process to manage many Twisted reactors running in child processes. Thanks again. -Chris From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of chris@cmsconstruct.com Sent: Saturday, September 12, 2020 12:07 PM To: 'Twisted general discussion' <twisted-python@twistedmatrix.com> Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port Hi Jean-Paul, Thank you very much for the detailed answer. And my appologies for not providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD as the ticket discussed. Looks like Red Hat (EL 7.6) is using epoll reactor, and the Windows side is using the select reactor. Thanks for the direction on checking out sys.modules. To avoid the reactor being loaded in the parent process, I can presumably move twisted imports within the multiprocessing child modules (from top, down into the run() functions). I will see how far I need to go (e.g. if I can continue using Twisted’s JSON logging or if absolutely everything should be isolated until after child process startup). But knowing I need to head that direction for epoll or other potential reactor conflicts - is very helpful. Reminds me of the GI Joe cartoon in the early 1980’s that would end with, “knowing is half the battle.” -Chris From: Twisted-Python <twisted-python-bounces@twistedmatrix.com <mailto:twisted-python-bounces@twistedmatrix.com> > On Behalf Of Jean-Paul Calderone Sent: Friday, September 11, 2020 1:28 PM To: Twisted general discussion <twisted-python@twistedmatrix.com <mailto:twisted-python@twistedmatrix.com> > Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port On Fri, Sep 11, 2020 at 1:34 PM <chris@cmsconstruct.com <mailto:chris@cmsconstruct.com> > wrote: Hey guys, Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port. I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I’m coming back to it. I’ll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted: Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective? To answer the asked question, I don't think there is rigorous (or even casual) testing of very much of Twisted in the context of "some Twisted code has been loaded into memory and then the process forked". So while it seems like a reasonable thing, I wouldn't say there's currently much effort being put towards making it a supported usage of Twisted. Of course this can change at almost any moment if someone decides to commit the effort. To dig a bit further into the specific problem, even if you only import the reactor in the parent process and then fork a child and try to start the reactor in the child, I strongly suspect epollreactor will break. This is because the epoll object is created by reactor instantiation (as opposed to being delayed until the reactor is run). epoll objects have a lot of weird behavior. See the Questions and Answers section of the epoll(7) man page. I don't know if this is the cause of your particular expression of these symptoms (it certainly doesn't apply to the original bug report which is on FreeBSD where there is no epoll) but it's at least a possible cause. This could probably be fixed in Twisted by only creating the epoll object when run is called. There's nothing particularly difficult about that change but it does involve touching a lot of the book-keeping logic since that all assumes it can register file descriptors before the reactor is started (think reactor.listenTCP(...); reactor.run()). I'm not sure but it may also be the case that only delaying creation of the waker until the reactor starts would also fix this. This is because as long as the epoll object remains empty a lot of the weird behavior is avoided and the waker is probably the only thing that actually gets added to it if you're just importing the reactor but not running it before forking. Alternatively, your application should be able to fix it by studiously avoiding the import of twisted.internet.reactor (directly or transitively, of course). You could add some kind of assertion about the state of sys.modules immediately before your forking code to develop some confidence you've managed this. And if this is really an epoll problem then switching to poll or select reactor would also presumably get rid of the issue. Jean-Paul
On Friday, 11 September 2020 19:28:13 BST Jean-Paul Calderone wrote:
On Fri, Sep 11, 2020 at 1:34 PM <chris@cmsconstruct.com> wrote:
Hey guys,
Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port.
I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I’m coming back to it. I’ll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted:
Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective?
To answer the asked question, I don't think there is rigorous (or even casual) testing of very much of Twisted in the context of "some Twisted code has been loaded into memory and then the process forked". So while it seems like a reasonable thing, I wouldn't say there's currently much effort being put towards making it a *supported* usage of Twisted. Of course this can change at almost any moment if someone decides to commit the effort.
To dig a bit further into the specific problem, even if you only *import* the reactor in the parent process and then fork a child and try to start the reactor in the child, I strongly suspect epollreactor will break. This is because the epoll object is created by reactor instantiation (as opposed to being delayed until the reactor is run). epoll objects have a lot of weird behavior. See the *Questions and Answers* section of the epoll(7) man page.
Suspect no longer. It breaks very badly. I guess by weird you mean that the epoll uses an FD, sounds like a great feature to me, the state is managed in the kernel. Of course you have to be aware of this and that the FD is inherited into children where the reactor will fail in various ways. You now have each child that adds and removes FD's is doing it on one shared epoll FD, but events get delivered to all the children. What we did is use the PollReactor not the EPollReactor. And before we fork we call reactor.removeReader on each of the ports (sockets) we setup in the parent. Then in the children we add the ports back in with reactor.addReader. After that the children run just fine. We do this so that we can open priv'ed ports that the children will share. We drop priv's after the priv'ed ports are opened. If you do not need to listen on sockets in the parent then simply avoid importing reactor before you fork. Barry
I don't know if this is the cause of your particular expression of these symptoms (it certainly doesn't apply to the *original* bug report which is on FreeBSD where there is no epoll) but it's at least *a possible* cause.
This could probably be fixed in Twisted by only creating the epoll object when run is called. There's nothing particularly difficult about that change but it does involve touching a lot of the book-keeping logic since that all assumes it can register file descriptors before the reactor is started (think reactor.listenTCP(...); reactor.run()).
I'm not sure but it may also be the case that only delaying creation of the *waker* until the reactor starts would also fix this. This is because as long as the epoll object remains empty a lot of the weird behavior is avoided and the waker is probably the only thing that actually gets added to it if you're just *importing* the reactor but not *running* it before forking.
Alternatively, your application *should* be able to fix it by studiously avoiding the import of twisted.internet.reactor (directly or transitively, of course). You could add some kind of assertion about the state of sys.modules immediately before your forking code to develop some confidence you've managed this.
And if this is really an epoll problem then switching to poll or select reactor would also presumably get rid of the issue.
Jean-Paul
Context:
I only hit this problem on Linux, not Windows.
The software project (github.com/opencontentplatform/ocp) has been a lot of fun, especially with walking the tight rope in using multi-processing, multi-threading, and Twisted reactors. The main controller process kicks off about 10 child processes, each doing different types of work. In general though, the child processes individually start a Twisted reactor, connect to Kafka, connect to a database, use a shared REST API, and some listen for connecting clients to accomplish work.
I test on Linux about once a year, so too many changes to rollback and figure out that way. It was working on Linux 2 years ago, but last year’s testing and current testing, receive the doWrite error. It continues running fine on Windows. I’ve gone back about 2 years of versions with Python3, Twisted, and dependent libs… on both Windows and Linux. Every version change yields the same result - continues to work on Windows and continues to hit the error on Linux. So something I added has caused Linux to throw that error.
I’m not explicitly sharing much between the main process and the sub processes. They are spun up with (1) a shared multiprocessing.Event() - to have the main process shut the children down, (2) their own unique multiprocessing.Event() - to have the child processes notify back to the parent, and (3) a deep copy of a dictionary (containing a bunch of settings that remain constant). The main process uses twisted.logger, but for testing I strip that out to remove any twisted imports in the main process. So I’m not importing anything Twisted in the main process, and I don’t believe I’m explicitly sharing something I shouldn’t. Seems like something is implicitly being exposed/shared across Linux child processes, that aren’t on Windows.
The tracebacks come through on Linux (sometimes randomly), on the console of the parent controller process. No need to paste here, since it’s the same as the ticket shows. I can’t reliably reproduce the problem, but I know if I stop/start client connections (ServerFactory/twisted.internet.interfaces.IReactorTCP and twisted.internet.protocol.ReconnectingClientFactory) then it will eventually happen. I need to devote time at whittling down the code and attempting to create a reliable test case… if even possible.
The error is slightly different when running HTTP vs HTTPS, but the story is the same. It cripples whatever child process that hits it, from doing much of anything thereafter. Not much luck with troubleshooting. The tracebacks do not include a calling function from my code, to tell me where to start. And it happens across different child process types, so not the same one each time. When I throw debuggers on the child processes, the problem seems to mask itself. Well, at least I didn’t hit the problem over the last 3 days using pudb and stepping through code at breakpoints.
I’m absolutely open to suggestions for troubleshooting, but first wanted to take a HUGE step back and ask a design question regarding Twisted and multiprocessing.
Thanks! _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sep 15, 2020, at 10:05 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
We do this so that we can open priv'ed ports that the children will share. We drop priv's after the priv'ed ports are opened.
A better way to do this might be to use something like the systemd activation endpoint, so you never need privileges in your code in the first place: https://twistedmatrix.com/documents/current/core/howto/systemd.html#socket-a... <https://twistedmatrix.com/documents/current/core/howto/systemd.html#socket-a...> -glyph
On Tuesday, 15 September 2020 19:37:41 BST Glyph wrote:
On Sep 15, 2020, at 10:05 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
We do this so that we can open priv'ed ports that the children will share. We drop priv's after the priv'ed ports are opened.
A better way to do this might be to use something like the systemd activation endpoint, so you never need privileges in your code in the first place:
https://twistedmatrix.com/documents/current/core/howto/systemd.html#socket-a... <https://twistedmatrix.com/documents/current/core/howto/systemd.html#socket-a...>
Yep, but at the moment I'm on Centos 6 (no systemd) porting to Centos 8 at the moment. I'm not sure socket activation is the way forward, but there are a lot of other tricks in systemd that should help. Also there is the idea to pass the prived FD's over sockets into the non priv'ed code etc. Barry
participants (4)
-
Barry Scott -
chris＠cmsconstruct.com -
Glyph -
Jean-Paul Calderone