Issues with multiprocessing
Hi, I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=.....)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp'ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process 'sleeps' for some time and continues. I am using a T5220, Solaris box with Solaris 10. Problem -there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned. Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it? Thanks and Regards, Yateen V. Joshi This email and any attachments may contain legally privileged and/or confidential information of Starent Networks, Corp. and is intended only for the individual or entity named in the message. The information transmitted may not be used to create or change any contractual obligations of Starent Networks, Corp. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this e-mail and its attachments by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify the sender immediately -- by replying to this message or by sending an email to postmaster@starentnetworks.com -- and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you.
On 09/12/2009 08:51, Joshi, Yateen wrote:
Hi,
I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=.....)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp'ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process 'sleeps' for some time and continues.
I am using a T5220, Solaris box with Solaris 10.
Problem --there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned.
Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it?
Hello Joshi, This email list is for the development of Python, not for developing with Python. A better place to try would be the python-list (comp.lang.python - which has both google groups and gmane gateways). All the best, Michael Foord
Thanks and Regards,
Yateen V. Joshi
This email and any attachments may contain legally privileged and/or confidential information of Starent Networks, Corp. and is intended only for the individual or entity named in the message. The information transmitted may not be used to create or change any contractual obligations of Starent Networks, Corp. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this e-mail and its attachments by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify the sender immediately -- by replying to this message or by sending an email to postmaster@starentnetworks.com -- and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog
On Wed, Dec 9, 2009 at 6:22 AM, Michael Foord
On 09/12/2009 08:51, Joshi, Yateen wrote:
Hi,
I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=…..)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp’ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process ‘sleeps’ for some time and continues.
I am using a T5220, Solaris box with Solaris 10.
Problem –there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned.
Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it?
Hello Joshi,
This email list is for the development of Python, not for developing with Python. A better place to try would be the python-list (comp.lang.python - which has both google groups and gmane gateways).
All the best,
Michael Foord
Additionally, if you suspect there is a reproducible box (preferably not just on solaris, which I do not have access to) please file it in the tracker.
participants (3)
-
Jesse Noller -
Joshi, Yateen -
Michael Foord