problem with simple multiprocessing script on OS X

Darren Dale dsdale24 at
Mon Aug 23 15:58:16 CEST 2010

The following script runs without problems on Ubuntu and Windows 7.
h5py is a package wrapping the hdf5 library (

from multiprocessing import Pool
import h5py

def update(i):
    print i

def f(i):
    "hello foo"
    return i*i

if __name__ == '__main__':
    pool = Pool()
    for i in range(10):
        pool.apply_async(f, [i], callback=update)

On OS X 10.6 (tested using python-2.6.5 from MacPorts), I have to
comment out the as-yet unused h5py import, otherwise I get a

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/", line 532, in __bootstrap_inner
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/multiprocessing/", line 226, in _handle_tasks
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

I've searched that pickle error and found some references to pickling
a lambda, but I don't think that is the issue. There are no lambdas in
the h5py module, and the script runs fine on windows and linux. I need
access to both multiprocessing and h5py objects in the same module, so
I can register a callback that saves the results to an hdf5 file.

Are there any suggestions as to what could be the problem, or
suggestions on how I can track it down?


More information about the Python-list mailing list