<html><head><title>Re: [Ironpython-users] multiprocessing?</title>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
</head>
<body>
<span style=" font-family:'Courier New'; font-size: 9pt;">The usages of multiprocessing.Process are few and far between in multiprocessing.Pool.<br>
This is the most prominent one:<br>
<br>
for i in range(self._processes - len(self._pool)):<br>
w = self.Process(target=worker,<br>
args=(self._inqueue, self._outqueue,<br>
self._initializer,<br>
self._initargs, self._maxtasksperchild)<br>
)<br>
self._pool.append(w)<br>
<br>
I guess you can just replace multiprocessing\process.py with a set of thunks,<br>
then happily import the stock mutiprocessing.Pool .<br>
<br>
-----Original Message-----<br>
From: Olof Bjarnason <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:olof.bjarnason@gmail.com">olof.bjarnason@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">><br>
Sent: Wednesday, February 5, 2014 15:55<br>
To: Jeff Hardy <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:jdhardy@gmail.com">jdhardy@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">><br>
Cc: "ironpython-users@python.org" <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:ironpython-users@python.org">ironpython-users@python.org</a><span style=" font-family:'Courier New'; font-size: 9pt;">><br>
Subject: [Ironpython-users] multiprocessing?<br>
<br>
<br>
<br>
<br>
On 5 February 2014 09:57, Jeff Hardy <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:jdhardy@gmail.com">jdhardy@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">> wrote:<br>
On Wed, Feb 5, 2014 at 9:38 AM, Olof Bjarnason <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:olof.bjarnason@gmail.com">olof.bjarnason@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">> wrote:<br>
><br>
><br>
><br>
> On 5 February 2014 09:04, Jeff Hardy <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:jdhardy@gmail.com">jdhardy@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">> wrote:<br>
>><br>
>> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason<br>
>> <</span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:olof.bjarnason@gmail.com">olof.bjarnason@gmail.com</a><span style=" font-family:'Courier New'; font-size: 9pt;">> wrote:<br>
>> > Hi there, new to IronPython, but 5+ years user of CPython.<br>
>> ><br>
>> > I've done some googling already, but cannot seem to find whether or not<br>
>> > the<br>
>> > Python std lib module "multiprocessing" is intended to be used in<br>
>> > IronPython<br>
>> > or not.<br>
>><br>
>> I would love to, but it's probably the most difficult module to<br>
>> support in the entire stdlib. It's also primarily (but not entirely) a<br>
>> workaround for CPython's GIL, which IronPython doesn't have. Because<br>
>> multiprocessing and threading implement the same API, in principle you<br>
>> should be able to use multiprocessing on CPython and threading on<br>
>> IronPython without much effort.<br>
>><br>
><br>
> I can see multiprocessing being one of the most advanced modules of<br>
> PyStdLib, yes.<br>
<br>
Not only that, but it's fork behaviour might not even be possible<br>
running on the .NET framework. Mono's documentation for fork() has all<br>
sorts of warnings - basically, if it breaks, you get to keep the<br>
pieces. I'm confident it's even possible (and I don't think Jython<br>
supports it either, for much the same reason).<br>
<br>
OK. The multiprocessing.Pool is a readable way of doing parallellization in CPython. Basically run a method and collect results in a list (can it be simpler than that?). It seems concurrent.futures has a similar syntax using the ThreadPool.<br>
<br>
I had a vague hope threading module would have the same module available (since their API is supposed to be 1:1), but cannot find it via from threading import Pool (CPython 2.7.6).<br>
<br>
For now I could just write a similar module in IPY myself, since I know .NET Threads and the Pool api I need for my script. On more general note it's a bit unfortunate that Pool isn't available in PyStdLib's threading module.<br>
<br>
Pool missing from threading might be more of a PyStdLib issue than IronPython issue, but it affects me anyway. :)<br>
<br>
<br>
><br>
> I guess I could just switch to the threading module then. But does it<br>
> support Pool e.g., so I can re-use scripts I already have written?<br>
<br>
There is a thread pool in the multiprocessing library that I believe<br>
is pure Python and should work on IronPython (perhaps with some small<br>
patches). I haven't checked to see if concurrent.futures works; it'll<br>
have to for 3.0, but I'm not sure about the 2.7 backport.<br>
<br>
Is it's in multprocessing, that isn't importable into IPY. But possibly I can just copy-patch it to be able to use Pools + threading in IPY.<br>
<br>
<br>
><br>
> Is the semantics of thread the same as in Python, being "fake" threading,<br>
> not separate processes, in the IPY version of threading?<br>
<br>
Each Python thread is 1:1 with a .NET thread, which I'm pretty sure<br>
are 1:1 with OS threads. Only one process is used (.NET, not<br>
surprisingly, takes a lot from Windows' computation model, where<br>
threads are cheap and processes are expensive).<br>
<br>
Makes sense nomenclature wise, but not semantically for Python in general; the question to ask is "would the Python scrip parallell_execution.py give the same computation result running under both CPython and IronPython?" and the answer would in general be "No" since Python Threads are running sequentially/interleaved while the IronPython execution model relies on .NET threads which is actually not interleaved (at least not in the Python GIL sense .. it might be in the windows process-thread sense). This is getting complicated :)<br>
<br>
<br>
><br>
> I guess a follow up on that is "why"? Shouldn't the semantics be as similar<br>
> as possible to the PyStdLib?<br>
<br>
Time and interest. There are other gaps in the stdlib (e.g. expat)<br>
that are more critical, and plenty of other bug fixes that will have a<br>
wider impact. If someone provides the patches to make it work I<br>
wouldn't turn it down, but so far I don't think anyone has the<br>
interest yet.<br>
<br>
I see.<br>
<br>
<br>
><br>
><br>
>> ><br>
>> > 1. Is there a web page describing exactly what python standard modules<br>
>> > are<br>
>> > available in the IronPython environment?<br>
>><br>
>> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to<br>
>> create a branch of the Python documentation that contains<br>
>> IronPython-specific information. Last time I tried I got lost yak<br>
>> shaving Sphinx and never quite finished it<br>
>> (</span><a style=" font-family:'courier new'; font-size: 9pt;" href="https://github.com/IronLanguages/ironpython-docs">https://github.com/IronLanguages/ironpython-docs</a><span style=" font-family:'Courier New'; font-size: 9pt;">). Ideally it would<br>
>> list what version of IronPython things became available (or if they're<br>
>> not) and have completely rewritten sections for extending & embedding.<br>
><br>
><br>
> I can see this being a huge task to tackle, but it is also essential for<br>
> IronPython adoption by CPython devs (as me).<br>
><br>
> It is definately need to have documentation of what is available and not<br>
> from the StdLib.<br>
<br>
I know. :( It's difficult to balance things that benefit the community<br>
(documentation) vs. things that I want to work on (Py3k features).<br>
<br>
Know that balance, it's hard!<br>
<br>
<br>
><br>
> Also, having a fork of the whole doc seems overkill - and a high maintenance<br>
> burden.<br>
<br>
Not any more work than having a fork of the standard library, which we<br>
already do. I have some git tricks up my sleeve to make it easy to do<br>
a 3-way merge which will make it easy to bring in upstream changes<br>
while preserving IronPython changes.<br>
<br>
><br>
> I'm asking for a list of modules and their status. Importable? Unit tests<br>
> passing? Sounds almost automatable to come with that list come to think of<br>
> it...<br>
><br>
> Is there any CI mechanism for IronPython in place? TravisCI or similar?<br>
<br>
</span><a style=" font-family:'courier new'; font-size: 9pt;" href="http://teamcity.codebetter.com/project.html?projectId=project110">http://teamcity.codebetter.com/project.html?projectId=project110</a><span style=" font-family:'Courier New'; font-size: 9pt;">. Not<br>
in great shape right now, mainly because the IronPython tests are<br>
extremely difficult to run. Another thing on the 3.0 TODO list is<br>
getting that straightened out.<br>
<br>
I get a login screen following that URL... <br>
<br>
<br>
<span style=" color: #888888;">- Jeff</body></html>