concurrent.futures.ProcessPoolExecutor(restart_workers=True)
Hi guys, I want to have a version of `concurrent.futures.ProcessPoolExecutor` in which every worker process shuts down after each task. I want this because I believe I have a memory leak in my program that I wasn't able to fix. (Possibly it's in C extensions that I use.) Since I'm going to use a process pool anyway, I'm hoping to sidestep this memory leak by launching every task on a worker process that shuts down after it's done. A few years back when I was working at Dell, they did something similar. They had a Python server that had a memory leak, and they worked around it by having a process that can be easily restarted, and then they restarted it often. The plan is that each new work item will run on a fresh process that has no memory leaks from previous work items. I looked at the `futures/process.py` file to see whether I could subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy this is not easy. Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`? Thanks, Ram.
billiard a multiprocessing py2 fork/backport has https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multi... with maxtasksperchild On Sat, 26 Jun 2021, 16:57 Ram Rachum, <ram@rachum.com> wrote:
Hi guys,
I want to have a version of `concurrent.futures.ProcessPoolExecutor` in which every worker process shuts down after each task. I want this because I believe I have a memory leak in my program that I wasn't able to fix. (Possibly it's in C extensions that I use.)
Since I'm going to use a process pool anyway, I'm hoping to sidestep this memory leak by launching every task on a worker process that shuts down after it's done.
A few years back when I was working at Dell, they did something similar. They had a Python server that had a memory leak, and they worked around it by having a process that can be easily restarted, and then they restarted it often.
The plan is that each new work item will run on a fresh process that has no memory leaks from previous work items.
I looked at the `futures/process.py` file to see whether I could subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy this is not easy.
Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
Thanks, Ram. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7Y... Code of Conduct: http://python.org/psf/codeofconduct/
And it's on Python stdlib 3 https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.... On Sat, 26 Jun 2021, 17:24 Thomas Grainger, <tagrain@gmail.com> wrote:
billiard a multiprocessing py2 fork/backport has https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multi... with maxtasksperchild
On Sat, 26 Jun 2021, 16:57 Ram Rachum, <ram@rachum.com> wrote:
Hi guys,
I want to have a version of `concurrent.futures.ProcessPoolExecutor` in which every worker process shuts down after each task. I want this because I believe I have a memory leak in my program that I wasn't able to fix. (Possibly it's in C extensions that I use.)
Since I'm going to use a process pool anyway, I'm hoping to sidestep this memory leak by launching every task on a worker process that shuts down after it's done.
A few years back when I was working at Dell, they did something similar. They had a Python server that had a memory leak, and they worked around it by having a process that can be easily restarted, and then they restarted it often.
The plan is that each new work item will run on a fresh process that has no memory leaks from previous work items.
I looked at the `futures/process.py` file to see whether I could subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy this is not easy.
Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
Thanks, Ram. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7Y... Code of Conduct: http://python.org/psf/codeofconduct/
Thank you Thomas, that's helpful. Though I really like the future interface, I've used it many times while I never used `multiprocessing.pool.Pool` or its `AsyncResult`. I worry that some of the expectations I've grown to have from `Future` objects would be broken here. It would be nice if this feature existed in `concurrent.futures`. Thanks, Ram. On Sat, Jun 26, 2021 at 7:28 PM Thomas Grainger <tagrain@gmail.com> wrote:
And it's on Python stdlib 3 https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool....
On Sat, 26 Jun 2021, 17:24 Thomas Grainger, <tagrain@gmail.com> wrote:
billiard a multiprocessing py2 fork/backport has https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multi... with maxtasksperchild
On Sat, 26 Jun 2021, 16:57 Ram Rachum, <ram@rachum.com> wrote:
Hi guys,
I want to have a version of `concurrent.futures.ProcessPoolExecutor` in which every worker process shuts down after each task. I want this because I believe I have a memory leak in my program that I wasn't able to fix. (Possibly it's in C extensions that I use.)
Since I'm going to use a process pool anyway, I'm hoping to sidestep this memory leak by launching every task on a worker process that shuts down after it's done.
A few years back when I was working at Dell, they did something similar. They had a Python server that had a memory leak, and they worked around it by having a process that can be easily restarted, and then they restarted it often.
The plan is that each new work item will run on a fresh process that has no memory leaks from previous work items.
I looked at the `futures/process.py` file to see whether I could subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy this is not easy.
Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
Thanks, Ram. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7Y... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, Jun 26, 2021 at 11:59 AM Ram Rachum <ram@rachum.com> wrote:
Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
*restart_workers* certainly seems worth further investigation and consideration. However, you may also be able to find a workaround using the *initialzer* argument for ProcessPoolExecutor. A detailed description is found in the docs for ThreadPoolExecutor: *initializer* is an optional callable that is called at the start of each worker thread; *initargs* is a tuple of arguments passed to the initializer. Not certain and could be decently involved, but figured it may be a lead worth sharing. :) With loving-kindness, On Sat, Jun 26, 2021 at 11:59 AM Ram Rachum <ram@rachum.com> wrote:
Hi guys,
I want to have a version of `concurrent.futures.ProcessPoolExecutor` in which every worker process shuts down after each task. I want this because I believe I have a memory leak in my program that I wasn't able to fix. (Possibly it's in C extensions that I use.)
Since I'm going to use a process pool anyway, I'm hoping to sidestep this memory leak by launching every task on a worker process that shuts down after it's done.
A few years back when I was working at Dell, they did something similar. They had a Python server that had a memory leak, and they worked around it by having a process that can be easily restarted, and then they restarted it often.
The plan is that each new work item will run on a fresh process that has no memory leaks from previous work items.
I looked at the `futures/process.py` file to see whether I could subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy this is not easy.
Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
Thanks, Ram. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7Y... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Kyle R. Stanley, Python Core Developer (what is a core dev? <https://devguide.python.org/coredev/>) *Pronouns: they/them **(why is my pronoun here?* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> )
participants (3)
-
Kyle Stanley -
Ram Rachum -
Thomas Grainger