problem with simple multiprocessing script on OS X

[Darren Dale]

On Aug 24, 4:32 pm, Thomas Jollans <tho... at jollybox.de> wrote:
> On Tuesday 24 August 2010, it occurred to Darren Dale to exclaim:
>
>
>
>
>
> > On Aug 23, 9:58 am, Darren Dale <dsdal... at gmail.com> wrote:
> > > The following script runs without problems on Ubuntu and Windows 7.
> > > h5py is a package wrapping the hdf5 library (http://code.google.com/p/
> > > h5py/):
>
> > > from multiprocessing import Pool
> > > import h5py
>
> > > def update(i):
> > >     print i
>
> > > def f(i):
> > >     "hello foo"
> > >     return i*i
>
> > > if __name__ == '__main__':
> > >     pool = Pool()
> > >     for i in range(10):
> > >         pool.apply_async(f, [i], callback=update)
> > >     pool.close()
> > >     pool.join()
>
> > > On OS X 10.6 (tested using python-2.6.5 from MacPorts), I have to
> > > comment out the as-yet unused h5py import, otherwise I get a
> > > traceback:
>
> What on earth is h5py doing there?  If what you're telling us is actually
> happening, and the code works 1:1 on Linux and Windows, but fails on OSX, and
> you're using the same versions of h5py and Python, then the h5py
> initialization code is not only enticing multiprocessing to try to pickle
> something other than usual, but it is also doing that due to some platform-
> dependent witchcraft, and I doubt there's very much separating the OSX
> versions from the Linux versions of anything involved.

I can't find anything in the source to suggest that h5py is doing any
platform-specific magic. Do you have an idea of how it would be
possible for initialization code to cause multiprocessing to try to
pickle something it normally would not?

> > This is a really critical bug for me, but I'm not sure how to proceed.
> > Can I file a bug report on the python bugtracker if the only code I
> > can come up with to illustrate the problem requires a lame import of a
> > third party module?
>
> I doubt this is an issue with Python. File a bug on the h5py tracker and see
> what they say. The people there might at least have some vague inkling of what
> may be going on.

Thanks for the suggestion. I was in touch with the h5py maintainer
before my original post. We don't have any leads.