<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 14pt; color: #000000"><div>Hello,<br></div><div><br></div><div>My dix cents may not seem to be neither very clear, nor too argumented, but I wanted to let you know that there seem to be already interesting pythonic studies about how to release the so-called GIL and about, all, for which pragmatic use cases.<br></div><div><br></div><div>Reading the following article, I discovered that Python by itself seems not to be a language, but language specification :<br></div><div><a href="http://www.toptal.com/python/why-are-there-so-many-pythons">http://www.toptal.com/python/why-are-there-so-many-pythons</a> <br></div><div>Thus, it justifies that PyPy is a more general approach than CPython, which looks like a particular case of Python, even if the most frequently used (?)<br></div><div><br></div><div>Now, there is a specific study of PyPy aimed at removing the GIL, called "Software Transactional Memory". Here it is :<br></div><div><a href="http://doc.pypy.org/en/latest/stm.html">http://doc.pypy.org/en/latest/stm.html</a> <br></div><div><br></div><div>Hope it helps :<br></div><div>Best regards,<br></div><div> Jean-Charles.<br></div><div><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>De: </b>python-ideas-request@python.org<br><b>À: </b>python-ideas@python.org<br><b>Envoyé: </b>Dimanche 21 Juin 2015 08:41:24<br><b>Objet: </b>Python-ideas Digest, Vol 103, Issue 100<br><div><br></div>Send Python-ideas mailing list submissions to<br> python-ideas@python.org<br><div><br></div>To subscribe or unsubscribe via the World Wide Web, visit<br> https://mail.python.org/mailman/listinfo/python-ideas<br>or, via email, send a message with subject or body 'help' to<br> python-ideas-request@python.org<br><div><br></div>You can reach the person managing the list at<br> python-ideas-owner@python.org<br><div><br></div>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Python-ideas digest..."<br><div><br></div><br>Today's Topics:<br><div><br></div> 1. Re: solving multi-core Python (Nathaniel Smith)<br> 2. Re: solving multi-core Python (Nick Coghlan)<br> 3. Re: solving multi-core Python (Wes Turner)<br><div><br></div><br>----------------------------------------------------------------------<br><div><br></div>Message: 1<br>Date: Sat, 20 Jun 2015 22:25:07 -0700<br>From: Nathaniel Smith <njs@pobox.com><br>To: Eric Snow <ericsnowcurrently@gmail.com><br>Cc: python-ideas <python-ideas@python.org><br>Subject: Re: [Python-ideas] solving multi-core Python<br>Message-ID:<br> <CAPJVwBkjoK31m2-ynrGF_AmYFL0ULL3LdX6r7+d+B7RienQh7A@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><div><br></div>On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:<br>><br>><br>> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs@pobox.com> wrote:<br>> ><br>> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently@gmail.com><br>wrote:<br>> > ><br>> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,<br>> > > exposing them in Python, and adding a mechanism to safely share<br>> > > objects between them.<br>> ><br>> > This all sounds really cool if you can pull it off, and shared-nothing<br>threads do seem like the least impossible model to pull off.<br>><br>> Agreed.<br>><br>> > But "least impossible" and "possible" are different :-). From your<br>email I can't tell whether this plan is viable while preserving backcompat<br>and memory safety.<br>><br>> I agree that those issues must be clearly solved in the proposal before<br>it can be approved. I'm confident the approach I'm pursuing will afford us<br>the necessary guarantees. I'll address those specific points directly when<br>I can sit down and organize my thoughts.<br><div><br></div>I'd love to see just a hand wavy, verbal proof-of-concept walking through<br>how this might work in some simple but realistic case. To me a single<br>compelling example could make this proposal feel much more concrete and<br>achievable.<br><div><br></div>> > Suppose I have a queue between two subinterpreters, and on this queue I<br>place a list of dicts of user-defined-in-python objects, each of which<br>holds a reference to a user-defined-via-the-C-api object. What happens next?<br>><br>> You've hit upon exactly the trickiness involved and why I'm thinking the<br>best approach initially is to only allow *strictly* immutable objects to<br>pass between interpreters. Admittedly, my description of channels is very<br>vague.:) There are a number of possibilities with them that I'm still<br>exploring (CSP has particular opinions...), but immutability is a<br>characteristic that may provide the simplest *initial* approach. Going<br>that route shouldn't preclude adding some sort of support for mutable<br>objects later.<br><div><br></div>There aren't really many options for mutable objects, right? If you want<br>shared nothing semantics, then transmitting a mutable object either needs<br>to make a copy, or else be a real transfer, where the sender no longer has<br>it (cf. Rust).<br><div><br></div>I guess for the latter you'd need some new syntax for send-and-del, that<br>requires the object to be self contained (all mutable objects reachable<br>from it are only referenced by each other) and have only one reference in<br>the sending process (which is the one being sent and then destroyed).<br><div><br></div>> Keep in mind that by "immutability" I'm talking about *really* immutable,<br>perhaps going so far as treating the full memory space associated with an<br>object as frozen. For instance, we'd have to ensure that "immutable"<br>Python objects like strings, ints, and tuples do not change (i.e. via the C<br>API).<br><div><br></div>This seems like a red herring to me. It's already the case that you can't<br>legally use the c api to mutate tuples, ints, for any object that's ever<br>been, say, passed to a function. So for these objects, the subinterpreter<br>setup doesn't actually add any new constraints on user code.<br><div><br></div>C code is always going to be *able* to break memory safety so long as<br>you're using shared-memory threading at the c level to implement this<br>stuff. We just need to make it easy not to.<br><div><br></div>Refcnts and garbage collection are another matter, of course.<br><div><br></div>-n<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/c2b41fe0/attachment-0001.html><br><div><br></div>------------------------------<br><div><br></div>Message: 2<br>Date: Sun, 21 Jun 2015 16:31:33 +1000<br>From: Nick Coghlan <ncoghlan@gmail.com><br>To: Nathaniel Smith <njs@pobox.com><br>Cc: Eric Snow <ericsnowcurrently@gmail.com>, python-ideas<br> <python-ideas@python.org><br>Subject: Re: [Python-ideas] solving multi-core Python<br>Message-ID:<br> <CADiSq7cv538UBK9BE3e8eAakFB=njwHB-qnMu3m=qzLADzpsOg@mail.gmail.com><br>Content-Type: text/plain; charset=UTF-8<br><div><br></div>On 21 June 2015 at 15:25, Nathaniel Smith <njs@pobox.com> wrote:<br>> On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:<br>>><br>>><br>>> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs@pobox.com> wrote:<br>>> ><br>>> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently@gmail.com><br>>> > wrote:<br>>> > ><br>>> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,<br>>> > > exposing them in Python, and adding a mechanism to safely share<br>>> > > objects between them.<br>>> ><br>>> > This all sounds really cool if you can pull it off, and shared-nothing<br>>> > threads do seem like the least impossible model to pull off.<br>>><br>>> Agreed.<br>>><br>>> > But "least impossible" and "possible" are different :-). From your email<br>>> > I can't tell whether this plan is viable while preserving backcompat and<br>>> > memory safety.<br>>><br>>> I agree that those issues must be clearly solved in the proposal before it<br>>> can be approved. I'm confident the approach I'm pursuing will afford us the<br>>> necessary guarantees. I'll address those specific points directly when I<br>>> can sit down and organize my thoughts.<br>><br>> I'd love to see just a hand wavy, verbal proof-of-concept walking through<br>> how this might work in some simple but realistic case. To me a single<br>> compelling example could make this proposal feel much more concrete and<br>> achievable.<br><div><br></div>I was one of the folks pushing Eric in this direction, and that's<br>because it's a possibility that was conceived of a few years back, but<br>never tried due to lack of time (and inclination for those of us that<br>are using Python primarily as an orchestration tool and hence spend<br>most of our time on IO bound problems rather than CPU bound ones):<br>http://www.curiousefficiency.org/posts/2012/07/volunteer-supported-free-threaded-cross.html<br><div><br></div>As mentioned there, I've at least spent some time with Graham<br>Dumpleton over the past few years figuring out (and occasionally<br>trying to address) some of the limitations of mod_wsgi's existing<br>subinterpreter based WSGI app separation:<br>https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Python_Sub_Interpreters<br><div><br></div>The fact that mod_wsgi can run most Python web applications in a<br>subinterpreter quite happily means we already know the core mechanism<br>works fine, and there don't appear to be any insurmountable technical<br>hurdles between the status quo and getting to a point where we can<br>either switch the GIL to a read/write lock where a write lock is only<br>needed for inter-interpreter communications, or else find a way for<br>subinterpreters to release the GIL entirely by restricting them<br>appropriately.<br><div><br></div>For inter-interpreter communication, the worst case scenario is having<br>to rely on a memcpy based message passing system (which would still be<br>faster than multiprocessing's serialisation + IPC overhead), but there<br>don't appear to be any insurmountable barriers to setting up an object<br>ownership based system instead (code that accesses PyObject_HEAD<br>fields directly rather than through the relevant macros and functions<br>seems to be the most likely culprit for breaking, but I think "don't<br>do that" is a reasonable answer there).<br><div><br></div>There's plenty of prior art here (including a system I once wrote in C<br>myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with<br>Eric's "simple matter of engineering" characterisation of the problem<br>space.<br><div><br></div>The main reason that subinterpreters have never had a Python API<br>before is that they have enough rough edges that having to write a<br>custom C extension module to access the API is the least of your<br>problems if you decide you need them. At the same time, not having a<br>Python API not only makes them much harder to test, which means<br>various aspects of their operation are more likely to be broken, but<br>also makes them inherently CPython specific.<br><div><br></div>Eric's proposal essentially amounts to three things:<br><div><br></div>1. Filing off enough of the rough edges of the subinterpreter support<br>that we're comfortable giving them a public Python level API that<br>other interpreter implementations can reasonably support<br>2. Providing the primitives needed for safe and efficient message<br>passing between subinterpreters<br>3. Allowing subinterpreters to truly execute in parallel on multicore machines<br><div><br></div>All 3 of those are useful enhancements in their own right, which<br>offers the prospect of being able to make incremental progress towards<br>the ultimate goal of native Python level support for distributing<br>across multiple cores within a single process.<br><div><br></div>Cheers,<br>Nick.<br><div><br></div>-- <br>Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia<br><div><br></div><br>------------------------------<br><div><br></div>Message: 3<br>Date: Sun, 21 Jun 2015 01:41:21 -0500<br>From: Wes Turner <wes.turner@gmail.com><br>To: Eric Snow <ericsnowcurrently@gmail.com><br>Cc: Nathaniel Smith <njs@pobox.com>, Python-Ideas<br> <python-ideas@python.org><br>Subject: Re: [Python-ideas] solving multi-core Python<br>Message-ID:<br> <CACfEFw_1JVpUmZwFVkye-fbEsfH_NXVSo6WDMi1azhnXdY6PcA@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><div><br></div>Exciting!<br><div><br></div>*<br>http://zero-buffer.readthedocs.org/en/latest/api-reference/#zero_buffer.BufferView<br>* https://www.google.com/search?q=python+channels<br> * https://docs.python.org/2/library/asyncore.html#module-asyncore<br> * https://chan.readthedocs.org/en/latest/<br> * https://goless.readthedocs.org/en/latest/<br>* other approaches to the problem (with great APIs):<br> * http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords<br> * http://discodb.readthedocs.org/en/latest/<br>On Jun 20, 2015 5:55 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:<br><div><br></div>><br>> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs@pobox.com> wrote:<br>> ><br>> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently@gmail.com><br>> wrote:<br>> > ><br>> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,<br>> > > exposing them in Python, and adding a mechanism to safely share<br>> > > objects between them.<br>> ><br>> > This all sounds really cool if you can pull it off, and shared-nothing<br>> threads do seem like the least impossible model to pull off.<br>><br>> Agreed.<br>><br>> > But "least impossible" and "possible" are different :-). From your email<br>> I can't tell whether this plan is viable while preserving backcompat and<br>> memory safety.<br>><br>> I agree that those issues must be clearly solved in the proposal before it<br>> can be approved. I'm confident the approach I'm pursuing will afford us<br>> the necessary guarantees. I'll address those specific points directly when<br>> I can sit down and organize my thoughts.<br>><br>> ><br>> > Suppose I have a queue between two subinterpreters, and on this queue I<br>> place a list of dicts of user-defined-in-python objects, each of which<br>> holds a reference to a user-defined-via-the-C-api object. What happens next?<br>><br>> You've hit upon exactly the trickiness involved and why I'm thinking the<br>> best approach initially is to only allow *strictly* immutable objects to<br>> pass between interpreters. Admittedly, my description of channels is very<br>> vague.:) There are a number of possibilities with them that I'm still<br>> exploring (CSP has particular opinions...), but immutability is a<br>> characteristic that may provide the simplest *initial* approach. Going<br>> that route shouldn't preclude adding some sort of support for mutable<br>> objects later.<br>><br>> Keep in mind that by "immutability" I'm talking about *really* immutable,<br>> perhaps going so far as treating the full memory space associated with an<br>> object as frozen. For instance, we'd have to ensure that "immutable"<br>> Python objects like strings, ints, and tuples do not change (i.e. via the C<br>> API). The contents of involved tuples/containers would have to be likewise<br>> immutable. Even changing refcounts could be too much, hence the idea of<br>> moving refcounts out to a separate table.<br>><br>> This level of immutability would be something new to Python. We'll see if<br>> it's necessary. If it isn't too much work it might be a good idea<br>> regardless of the multi-core proposal.<br>><br>> Also note that Barry has a (rejected) PEP from a number of years ago about<br>> freezing objects... That idea is likely out of scope as relates to my<br>> proposal, but it certainly factors in the problem space.<br>><br>> -eric<br>><br>> _______________________________________________<br>> Python-ideas mailing list<br>> Python-ideas@python.org<br>> https://mail.python.org/mailman/listinfo/python-ideas<br>> Code of Conduct: http://python.org/psf/codeofconduct/<br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/e4aecde0/attachment.html><br><div><br></div>------------------------------<br><div><br></div>Subject: Digest Footer<br><div><br></div>_______________________________________________<br>Python-ideas mailing list<br>Python-ideas@python.org<br>https://mail.python.org/mailman/listinfo/python-ideas<br><div><br></div><br>------------------------------<br><div><br></div>End of Python-ideas Digest, Vol 103, Issue 100<br>**********************************************<br></div><div><br></div></div></body></html>