[New-bugs-announce] [issue34134] multiprocessing memory huge usage

[Windson Yang]

New submission from Windson Yang <wiwindson at gmail.com>:

I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file) 

I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383)

    if not hasattr(iterable, '__len__'):
        iterable = list(iterable)

    if chunksize is None:
        chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
        if extra:
            chunksize += 1

When we convert an iterable to list(iterable), we lost the advantage of using the generator. I'm not sure how to fix it, maybe we can set a default value for an object don't have '__len__' attr, any ideas?

----------
files: test.py
messages: 321788
nosy: Windson Yang, zach.ware
priority: normal
severity: normal
status: open
title: multiprocessing memory huge usage
type: resource usage
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47698/test.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34134>
_______________________________________