Hey guys,
Last year I hit a condition discussed in this ticket: https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a twisted.internet.tcp.Port.
I ignored it at the time since it was just on Linux, and my main platform was Windows. Now I’m coming back to it. I’ll add context on the problem below, but first I want to ask a high-level, design-type question with multiprocessing and Twisted:
Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t fork a process (multiprocessing module) that already has a Twisted reactor. Understood. But what about a parent process (not doing anything Twisted) forking child processes, where each child process starts their own Twisted reactor? Is that intended to work from the Twisted perspective?
_______________________________________________
Context:
I only hit this problem on Linux, not Windows.
The software project (github.com/opencontentplatform/ocp) has been a lot of fun, especially with walking the tight rope in using multi-processing, multi-threading, and Twisted reactors. The main controller process kicks off about 10 child processes, each doing different types of work. In general though, the child processes individually start a Twisted reactor, connect to Kafka, connect to a database, use a shared REST API, and some listen for connecting clients to accomplish work.
I test on Linux about once a year, so too many changes to rollback and figure out that way. It was working on Linux 2 years ago, but last year’s testing and current testing, receive the doWrite error. It continues running fine on Windows. I’ve gone back about 2 years of versions with Python3, Twisted, and dependent libs… on both Windows and Linux. Every version change yields the same result - continues to work on Windows and continues to hit the error on Linux. So something I added has caused Linux to throw that error.
I’m not explicitly sharing much between the main process and the sub processes. They are spun up with (1) a shared multiprocessing.Event() - to have the main process shut the children down, (2) their own unique multiprocessing.Event() - to have the child processes notify back to the parent, and (3) a deep copy of a dictionary (containing a bunch of settings that remain constant). The main process uses twisted.logger, but for testing I strip that out to remove any twisted imports in the main process. So I’m not importing anything Twisted in the main process, and I don’t believe I’m explicitly sharing something I shouldn’t. Seems like something is implicitly being exposed/shared across Linux child processes, that aren’t on Windows.
The tracebacks come through on Linux (sometimes randomly), on the console of the parent controller process. No need to paste here, since it’s the same as the ticket shows. I can’t reliably reproduce the problem, but I know if I stop/start client connections (ServerFactory/twisted.internet.interfaces.IReactorTCP and twisted.internet.protocol.ReconnectingClientFactory) then it will eventually happen. I need to devote time at whittling down the code and attempting to create a reliable test case… if even possible.
The error is slightly different when running HTTP vs HTTPS, but the story is the same. It cripples whatever child process that hits it, from doing much of anything thereafter. Not much luck with troubleshooting. The tracebacks do not include a calling function from my code, to tell me where to start. And it happens across different child process types, so not the same one each time. When I throw debuggers on the child processes, the problem seems to mask itself. Well, at least I didn’t hit the problem over the last 3 days using pudb and stepping through code at breakpoints.
I’m absolutely open to suggestions for troubleshooting, but first wanted to take a HUGE step back and ask a design question regarding Twisted and multiprocessing.
Thanks!
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python