Memory consumption of multiprocessing.Pool

[Wolodja Wentland]

Hi all,

I have a problem with the memory consumption of multiprocessing.Pool()'s
worker processes. I have a parent process that has to handle big data
structures and would like to use a pool of processes for computations.

The problem is, that all worker processes have the same memory
requirement as the parent one, although they do *not* use any the parent
processes data structures. The following snippet illustrates this
behaviour:

--- snip ---
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import multiprocessing
import time

def worker(el):
    time.sleep(10)

def main():
    print 'Init Pool 1'
    p_pool = multiprocessing.Pool()
    
    print 'Call pool.map()'
    p_pool.map(worker, range(multiprocessing.cpu_count()))

    print 'Allocate memory'
    eat_memory = range(1000000)
    
    print 'Call pool.map()'
    p_pool.map(worker, range(multiprocessing.cpu_count()))
    
    print 'Delete pool 1'
    del p_pool
    
    print 'Init Pool 2'
    p_pool = multiprocessing.Pool()
    
    print 'Call pool.map()'
    p_pool.map(worker, range(multiprocessing.cpu_count()))
    
    print 'Delete pool 2'
    del p_pool

if __name__ == '__main__':
    main()
--- snip ---

You will see that the memory consumption of the worker processes will be
roughly the same for the first two calls to p_pool.map(), but rise for
the third.

How can I make sure that 'eat_memory' does not use any memory in the
pool processes? This is important as I don't always know when a pool is
instanciated and the pool processes should *not* have the same memory
requirements as the parent process.

Am I missing something here?

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20091215/45677fbc/attachment-0001.sig>