[IPython-dev] drop-in replacement for multiprocessing

Michael McKerns mmckerns at caltech.edu
Mon Jun 22 15:19:18 EDT 2015


Yeah… a monkeypach is probably the right way to go...

For python 3.3+, it's an actual two-line change that tries to
import dill, and failing imports pickle.

For python 2.7, there's an additional two-line change in
_multiprocessing (the C lib), that does exactly the same
as the above.  I *think* multiprocessing only imports
_multiprocessing as _multiprocessing and not as
"multiprocessing._multiprocessing", so a runtime
monkeypatch would then still work.

The stdlib version is already nearly abstracted for the serializer,
so adding 2-3 more lines to pick whatever serializer one would
like should also be easy.

I'll see about submitting a patch for python 3.6.  Thanks all for
the feedback.


> On Mon, Jun 22, 2015 at 11:42 AM, Michael McKerns <mmckerns at caltech.edu>
> wrote:
>
> Yeah, basic hesitation in colliding with a stdlib module is the reason
>> I didn't do this a long time ago.  The most preferable solution would
>> be to have the underlying "two line patch" accepted to the stdlib that
>> allows a different serializer.  I'm not sure if a PEP was submitted for
>> that it would be accepted.  But maybe I'd get surprised.
>>
>> The only benefit overriding multiprocessing gives is that all packages
>> that depend on multiprocessing also get accelerated serialization
>> propagated to them.  That's really my motivation for making the
>> drop-in replacement.  So multiprocessing taking an alternate
>> serializer would work… and it's a simple as dirt patch with
>> apparently no loss of functionality.
>>
> Since it’s adding functionality, people can’t really assume that
> multiprocessing is your multiprocessing anyway, so they would need a
> check.
> That being the case, you could have a two line:
>
> import multidill
> multidill.patch_multiprocessing()
>
> If the patch is one that’s possible to make at runtime.
>
> Then you have:
>
>    - library with no collisions
>    - add functionality transparently to multiprocessing, but only
>    explicitly, not implicitly.
>
> -MinRK
>
>
>>
>>
>> > I didn't mean to start a naming discussion, just that I don't think
>> > colliding with the stdlib is a good plan.
>> >
>> > On Mon, Jun 22, 2015 at 11:07 AM, Thomas Kluyver <takowl at gmail.com>
>> wrote:
>> >
>> >> On 22 June 2015 at 10:44, MinRK <benjaminrk at gmail.com> wrote:
>> >>
>> >>> and go with multiprocess or multiprocessing2 or some such.
>> >>>
>> >> I'd suggest something more explicit like multiprocessing_dill. It
>> seems
>> >> presumptuous call a module 'thing2' without the blessing of the
>> 'thing'
>> >> maintainers. They might want to use that name one day, like the
>> Python
>> >> 2.7+
>> >> unittest package is backported as unittest2.
>> >>
>> >> 'multiprocess' sounds liable to cause confusion - if I saw that and
>> >> 'multiprocessing' in a tab completion list, I'd probably have to
>> check
>> >> which one was which.
>> >>
>> >> Thomas
>> >>
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >>
>> >>
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >
>>
>>
>> ---
>>
>> Mike McKerns
>> http://www.its.caltech.edu/~mmckerns
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>


---

Mike McKerns
http://www.its.caltech.edu/~mmckerns




More information about the IPython-dev mailing list