Multiprocessing vs. concurrent.futures, Linux vs. Windows
John Ladasky
john_ladasky at sbcglobal.net
Mon May 4 15:26:35 EDT 2020
Several years ago I built an application using multiprocessing. It only needed to work in Linux. I got it working fine. At the time, concurrent.futures did not exist.
My current project is an application which includes a PyQt5 GUI, and a live video feed with some real-time image processing. Running all this in one process resulted in unacceptable performance, so I'm trying to move all of the heavy video work into its own process. I only need to define one child process. I don't need a Pool. The child Process can run indefinitely, and it will communicate multiple messages to the main process using a Pipe.
I built a working example in Linux, but it hangs in Windows. I built a minimum example. My problem has nothing to do with PyQt. In Windows, my example hangs when I try to create the child Process. Code:
import os
from multiprocessing import Pipe, Process
def child_app():
inlet.send("Child process id = {}".format(os.getpid()))
if __name__ == "__main__":
outlet, inlet = Pipe()
print("Parent process id =", os.getpid())
child_process = Process(target=child_app) # Windows hangs here
child_process.start()
print(outlet.recv())
child_process.join()
child_process.close()
print("Program complete.")
I'm working in Python 3.7.6 on Windows 10, and 3.7.5 on Ubuntu Linux 19.10.
I am reading through the multiprocessing documentation, and I'm guessing that I've run into a problem with spawning vs. forking a new Process. The former is the Windows default, and the latter is the Posix default.
Before I dig too deeply, I am wondering whether this cross-platform problem is something that concurrent.futures might handle automatically. The concurrent API looks foreign to me. But if it is meant to replace multiprocessing, I suppose this would be a good time for me to learn it.
Thanks for any advice and suggestions!
More information about the Python-list
mailing list