[Ironpython-users] multiprocessing?

Wed Feb 5 12:55:46 CET 2014

On 5 February 2014 09:57, Jeff Hardy <jdhardy at gmail.com> wrote:

> On Wed, Feb 5, 2014 at 9:38 AM, Olof Bjarnason <olof.bjarnason at gmail.com>
> wrote:
> >
> >
> >
> > On 5 February 2014 09:04, Jeff Hardy <jdhardy at gmail.com> wrote:
> >>
> >> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
> >> <olof.bjarnason at gmail.com> wrote:
> >> > Hi there, new to IronPython, but 5+ years user of CPython.
> >> >
> >> > I've done some googling already, but cannot seem to find whether or
> not
> >> > the
> >> > Python std lib module "multiprocessing" is intended to be used in
> >> > IronPython
> >> > or not.
> >>
> >> I would love to, but it's probably the most difficult module to
> >> support in the entire stdlib. It's also primarily (but not entirely) a
> >> workaround for CPython's GIL, which IronPython doesn't have. Because
> >> multiprocessing and threading implement the same API, in principle you
> >> should be able to use multiprocessing on CPython and threading on
> >> IronPython without much effort.
> >>
> >
> > I can see multiprocessing being one of the most advanced modules of
> > PyStdLib, yes.
>
> Not only that, but it's fork behaviour might not even be possible
> running on the .NET framework. Mono's documentation for fork() has all
> sorts of warnings - basically, if it breaks, you get to keep the
> pieces. I'm confident it's even possible (and I don't think Jython
> supports it either, for much the same reason).
>

OK. The multiprocessing.Pool is a readable way of doing parallellization in
CPython. Basically run a method and collect results in a list (can it be
simpler than that?). It seems concurrent.futures has a similar syntax using
the ThreadPool.

I had a vague hope threading module would have the same module available
(since their API is supposed to be 1:1), but cannot find it via from
threading import Pool (CPython 2.7.6).

For now I could just write a similar module in IPY myself, since I know
.NET Threads and the Pool api I need for my script. On more general note
it's a bit unfortunate that Pool isn't available in PyStdLib's threading
module.

Pool missing from threading might be more of a PyStdLib issue than
IronPython issue, but it affects me anyway. :)

> >
> > I guess I could just switch to the threading module then. But does it
> > support Pool e.g., so I can re-use scripts I already have written?
>
> There is a thread pool in the multiprocessing library that I believe
> is pure Python and should work on IronPython (perhaps with some small
> patches). I haven't checked to see if concurrent.futures works; it'll
> have to for 3.0, but I'm not sure about the 2.7 backport.
>

Is it's in multprocessing, that isn't importable into IPY. But possibly I
can just copy-patch it to be able to use Pools + threading in IPY.

> >
> > Is the semantics of thread the same as in Python, being "fake" threading,
> > not separate processes, in the IPY version of threading?
>
> Each Python thread is 1:1 with a .NET thread, which I'm pretty sure
> are 1:1 with OS threads. Only one process is used (.NET, not
> surprisingly, takes a lot from Windows' computation model, where
> threads are cheap and processes are expensive).
>

Makes sense nomenclature wise, but not semantically for Python in general;
the question to ask is "would the Python scrip parallell_execution.py give
the same computation result running under both CPython and IronPython?" and
the answer would in general be "No" since Python Threads are running
sequentially/interleaved while the IronPython execution model relies on
.NET threads which is actually not interleaved (at least not in the Python
GIL sense .. it might be in the windows process-thread sense). This is
getting complicated :)

> >
> > I guess a follow up on that is "why"? Shouldn't the semantics be as
> similar
> > as possible to the PyStdLib?
>
> Time and interest. There are other gaps in the stdlib (e.g. expat)
> that are more critical, and plenty of other bug fixes that will have a
> wider impact. If someone provides the patches to make it work I
> wouldn't turn it down, but so far I don't think anyone has the
> interest yet.
>

I see.

>
> >
> >
> >> >
> >> > 1. Is there a web page describing exactly what python standard modules
> >> > are
> >> > available in the IronPython environment?
> >>
> >> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
> >> create a branch of the Python documentation that contains
> >> IronPython-specific information. Last time I tried I got lost yak
> >> shaving Sphinx and never quite finished it
> >> (https://github.com/IronLanguages/ironpython-docs). Ideally it would
> >> list what version of IronPython things became available (or if they're
> >> not) and have completely rewritten sections for extending & embedding.
> >
> >
> > I can see this being a huge task to tackle, but it is also essential for
> > IronPython adoption by CPython devs (as me).
> >
> > It is definately need to have documentation of what is available and not
> > from the StdLib.
>
> I know. :( It's difficult to balance things that benefit the community
> (documentation) vs. things that I want to work on (Py3k features).
>

Know that balance, it's hard!

>
> >
> > Also, having a fork of the whole doc seems overkill - and a high
> maintenance
> > burden.
>
> Not any more work than having a fork of the standard library, which we
> already do.  I have some git tricks up my sleeve to make it easy to do
> a 3-way merge which will make it easy to bring in upstream changes
> while preserving IronPython changes.
>
> >
> > I'm asking for a list of modules and their status. Importable? Unit tests
> > passing? Sounds almost automatable to come with that list come to think
> of
> > it...
> >
> > Is there any CI mechanism for IronPython in place? TravisCI or similar?
>
> http://teamcity.codebetter.com/project.html?projectId=project110. Not
> in great shape right now, mainly because the IronPython tests are
> extremely difficult to run. Another thing on the 3.0 TODO list is
> getting that straightened out.
>

I get a login screen following that URL...

>
> - Jeff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/29256db8/attachment.html>