[Ironpython-users] multiprocessing?

[Jeff Hardy]

On Wed, Feb 5, 2014 at 9:38 AM, Olof Bjarnason <olof.bjarnason at gmail.com> wrote:
>
>
>
> On 5 February 2014 09:04, Jeff Hardy <jdhardy at gmail.com> wrote:
>>
>> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
>> <olof.bjarnason at gmail.com> wrote:
>> > Hi there, new to IronPython, but 5+ years user of CPython.
>> >
>> > I've done some googling already, but cannot seem to find whether or not
>> > the
>> > Python std lib module "multiprocessing" is intended to be used in
>> > IronPython
>> > or not.
>>
>> I would love to, but it's probably the most difficult module to
>> support in the entire stdlib. It's also primarily (but not entirely) a
>> workaround for CPython's GIL, which IronPython doesn't have. Because
>> multiprocessing and threading implement the same API, in principle you
>> should be able to use multiprocessing on CPython and threading on
>> IronPython without much effort.
>>
>
> I can see multiprocessing being one of the most advanced modules of
> PyStdLib, yes.

Not only that, but it's fork behaviour might not even be possible
running on the .NET framework. Mono's documentation for fork() has all
sorts of warnings - basically, if it breaks, you get to keep the
pieces. I'm confident it's even possible (and I don't think Jython
supports it either, for much the same reason).

>
> I guess I could just switch to the threading module then. But does it
> support Pool e.g., so I can re-use scripts I already have written?

There is a thread pool in the multiprocessing library that I believe
is pure Python and should work on IronPython (perhaps with some small
patches). I haven't checked to see if concurrent.futures works; it'll
have to for 3.0, but I'm not sure about the 2.7 backport.

>
> Is the semantics of thread the same as in Python, being "fake" threading,
> not separate processes, in the IPY version of threading?

Each Python thread is 1:1 with a .NET thread, which I'm pretty sure
are 1:1 with OS threads. Only one process is used (.NET, not
surprisingly, takes a lot from Windows' computation model, where
threads are cheap and processes are expensive).

>
> I guess a follow up on that is "why"? Shouldn't the semantics be as similar
> as possible to the PyStdLib?

Time and interest. There are other gaps in the stdlib (e.g. expat)
that are more critical, and plenty of other bug fixes that will have a
wider impact. If someone provides the patches to make it work I
wouldn't turn it down, but so far I don't think anyone has the
interest yet.

>
>
>> >
>> > 1. Is there a web page describing exactly what python standard modules
>> > are
>> > available in the IronPython environment?
>>
>> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
>> create a branch of the Python documentation that contains
>> IronPython-specific information. Last time I tried I got lost yak
>> shaving Sphinx and never quite finished it
>> (https://github.com/IronLanguages/ironpython-docs). Ideally it would
>> list what version of IronPython things became available (or if they're
>> not) and have completely rewritten sections for extending & embedding.
>
>
> I can see this being a huge task to tackle, but it is also essential for
> IronPython adoption by CPython devs (as me).
>
> It is definately need to have documentation of what is available and not
> from the StdLib.

I know. :( It's difficult to balance things that benefit the community
(documentation) vs. things that I want to work on (Py3k features).

>
> Also, having a fork of the whole doc seems overkill - and a high maintenance
> burden.

Not any more work than having a fork of the standard library, which we
already do.  I have some git tricks up my sleeve to make it easy to do
a 3-way merge which will make it easy to bring in upstream changes
while preserving IronPython changes.

>
> I'm asking for a list of modules and their status. Importable? Unit tests
> passing? Sounds almost automatable to come with that list come to think of
> it...
>
> Is there any CI mechanism for IronPython in place? TravisCI or similar?

http://teamcity.codebetter.com/project.html?projectId=project110. Not
in great shape right now, mainly because the IronPython tests are
extremely difficult to run. Another thing on the 3.0 TODO list is
getting that straightened out.

- Jeff