[issue18620] multiprocessing page leaves out important part of Pool example
New submission from Chris Curvey: on http://docs.python.org/2/library/multiprocessing.html, there is a bit about how to use a Pool properly, which looks like this pool = Pool(processes=4) # start 4 worker processes result = pool.apply_async(f, [10]) What this neglects to mention is that only one process will get any of the work. If you really want four processes in the pool to work, you have to call apply_async four times. For example: results = [] pool = Pool(processes=4) for i in xrange(4): results.append(pool.apply_async(f, [10])) hat tip to http://stackoverflow.com/questions/12483512/python-multiprocessing-apply-asy... ---------- assignee: docs@python components: Documentation messages: 194115 nosy: Chris.Curvey, docs@python priority: normal severity: normal status: open title: multiprocessing page leaves out important part of Pool example type: enhancement _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Mark Lawrence added the comment: Looks as if the v2 and v3 docs need changing. ---------- nosy: +BreamoreBoy type: enhancement -> behavior versions: +Python 2.7, Python 3.4, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Davin Potts <python@discontinuity.net>: ---------- nosy: +davin _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Davin Potts added the comment: Attached are proposed patches for 2.7, 3.4, and default (3.5) branches. In these patches, the 2 examples in the documentation that showcase the use of pool.apply_async have been modified to specifically highlight that a single invocation of apply_async will only lead to a single process doing any work (not all of the pool's processes). Specifically, in the first code example: * In the 3.x branches, the example is expanded slightly to demonstrate that only a single process is used by a single apply_async call and that to accomplish similar to the mapping examples just above it, multiple apply_async calls are necessary. * Also in the 3.x branches, the code deliberately causes an exception to be raised which will terminate the execution of this example when anyone attempts to run it. Rather than provide an example which terminates prematurely, the exception is now caught and a message displayed to demonstrate the exception did indeed occur, permitting the example to run through to a successful exit. * In the 2.7 branch, this first example is much smaller than what is in use in the 3.x branches -- I've expanded it to cover the same information as is being shared in the 3.x branches now. Specifically, in the second code example: * In the 3.x branches, the only change is to the comment to the right of the apply_async call which emphasizes that only a single process is being used. Also, for clarity, the reference to the exception being raised in the last line was changed to highlight that it's an exception defined in the multiprocessing library (multiprocessing.TimeoutError). * The 2.7 branch is the same -- there were no other differences on this example between branches. These updated examples have been tested with the latest from 2.7, 3.4, and default/3.5 branches on OSX 10.10 and Windows 7. ---------- keywords: +patch stage: -> patch review Added file: http://bugs.python.org/file38029/issue18620_py35.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Davin Potts <python@discontinuity.net>: Added file: http://bugs.python.org/file38030/issue18620_py34.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Davin Potts <python@discontinuity.net>: Added file: http://bugs.python.org/file38031/issue18620_py27.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: ---------- nosy: +jnoller, sbt _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Davin Potts <python@discontinuity.net>: ---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Davin Potts <python@discontinuity.net>: ---------- versions: +Python 3.6 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Changes by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: docs@python -> rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Roundup Robot added the comment: New changeset c89f4f59872f by Berker Peksag in branch '3.5': Issue #18620: Improve Pool examples in multiprocessing documentation https://hg.python.org/cpython/rev/c89f4f59872f New changeset 9f201578d8d9 by Berker Peksag in branch 'default': Issue #18620: Improve Pool examples in multiprocessing documentation https://hg.python.org/cpython/rev/9f201578d8d9 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Roundup Robot added the comment: New changeset 9480e217ade0 by Berker Peksag in branch '2.7': Issue #18620: Improve Pool examples in multiprocessing documentation https://hg.python.org/cpython/rev/9480e217ade0 ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
Berker Peksag added the comment: Thanks for the patch, Davin. I tweaked code style a bit and removed trailing whitespaces. ---------- nosy: +berker.peksag resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: -Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18620> _______________________________________
participants (7)
-
Berker Peksag
-
Chris Curvey
-
Davin Potts
-
Mark Lawrence
-
Raymond Hettinger
-
Roundup Robot
-
Serhiy Storchaka