<br><br><div><span class="gmail_quote">On 11/3/06, <b class="gmail_sendername">Robert</b> <<a href="mailto:kxroberto@googlemail.com">kxroberto@googlemail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
repeated from c.l.p : "Feature Request: Py_NewInterpreter to create<br>separate GIL (branch)"<br><br>Daniel Dittmar wrote:<br> > robert wrote:<br> >> I'd like to use multiple CPU cores for selected time consuming Python
<br> >> computations (incl. numpy/scipy) in a frictionless manner.<br> >><br> >> Interprocess communication is tedious and out of question, so I<br> >> thought about simply using a more Python interpreter instances
<br> >> (Py_NewInterpreter) with extra GIL in the same process.<br> ><br> > If I understand Python/ceval.c, the GIL is really global, not specific<br> > to an interpreter instance:<br> > static PyThread_type_lock interpreter_lock = 0; /* This is the GIL */
<br> ><br><br>Thats the show stopper as of now.<br>There are only a handfull funcs in ceval.c to use that very global lock.<br>The rest uses that funcs around thread states.<br><br>Would it be a possibilty in next Python to have the lock separate for
<br>each Interpreter instance.<br>Thus: have *interpreter_lock separate in each PyThreadState instance and<br>only threads of same Interpreter have same GIL?<br>Separation between Interpreters seems to be enough. The Interpreter runs
<br>mainly on the stack. Possibly only very few global C-level resources<br>would require individual extra locks.</blockquote><div><br>Right, but that's the trick. For instance extension modules are shared between interpreters. Also look at the sys module and basically anything that is set by a function call is a process-level setting that would also need protection. Then you get into the fun stuff of the possibility of sharing objects created in one interpreter and then passed to another that is not necessarily known ahead of time (whether it be directly through C code or through process-level objects such as an attribute in an extension module).
<br><br>It is not as simple, unfortunately, as a few locks.<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Sooner or later Python will have to answer the multi-processor question.
<br>A per-interpreter GIL and a nice module for tunneling Python-Objects<br>directly between Interpreters inside one process might be the answer at<br>the right border-line ? Existing extension code base would remain<br>compatible, as far as there is already decent locking on module globals,
<br>which is the the usual case.</blockquote><div><br>This is not true (see above). From my viewpoint the only way for this to work would be to come up with a way to wrap all access to module objects in extension modules so that they are not trampled on because of separate locks per-interpreter, or have to force all extension modules to be coded so that they are instantiated individually per interpreter. And of course deal with all other process-level objects somehow.
<br><br>The SMP issue for Python will most likely not happen until someone cares enough to write code to do it and this take on it is no exception. There is no simple solution or else someone would have done it by now.<br>
<br>-Brett<br></div><br></div>