Re: [Python-ideas] Python-ideas Digest, Vol 103, Issue 100
Hello, My dix cents may not seem to be neither very clear, nor too argumented, but I wanted to let you know that there seem to be already interesting pythonic studies about how to release the so-called GIL and about, all, for which pragmatic use cases. Reading the following article, I discovered that Python by itself seems not to be a language, but language specification : http://www.toptal.com/python/why-are-there-so-many-pythons Thus, it justifies that PyPy is a more general approach than CPython, which looks like a particular case of Python, even if the most frequently used (?) Now, there is a specific study of PyPy aimed at removing the GIL, called "Software Transactional Memory". Here it is : http://doc.pypy.org/en/latest/stm.html Hope it helps : Best regards, Jean-Charles. ----- Mail original ----- De: python-ideas-request@python.org À: python-ideas@python.org Envoyé: Dimanche 21 Juin 2015 08:41:24 Objet: Python-ideas Digest, Vol 103, Issue 100 Send Python-ideas mailing list submissions to python-ideas@python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request@python.org You can reach the person managing the list at python-ideas-owner@python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. Re: solving multi-core Python (Nathaniel Smith) 2. Re: solving multi-core Python (Nick Coghlan) 3. Re: solving multi-core Python (Wes Turner) ---------------------------------------------------------------------- Message: 1 Date: Sat, 20 Jun 2015 22:25:07 -0700 From: Nathaniel Smith <njs@pobox.com> To: Eric Snow <ericsnowcurrently@gmail.com> Cc: python-ideas <python-ideas@python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CAPJVwBkjoK31m2-ynrGF_AmYFL0ULL3LdX6r7+d+B7RienQh7A@mail.gmail.com> Content-Type: text/plain; charset="utf-8" On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently@gmail.com>
wrote: threads do seem like the least impossible model to pull off.
Agreed.
But "least impossible" and "possible" are different :-). From your
email I can't tell whether this plan is viable while preserving backcompat and memory safety.
I agree that those issues must be clearly solved in the proposal before
Suppose I have a queue between two subinterpreters, and on this queue I
it can be approved. I'm confident the approach I'm pursuing will afford us the necessary guarantees. I'll address those specific points directly when I can sit down and organize my thoughts. I'd love to see just a hand wavy, verbal proof-of-concept walking through how this might work in some simple but realistic case. To me a single compelling example could make this proposal feel much more concrete and achievable. place a list of dicts of user-defined-in-python objects, each of which holds a reference to a user-defined-via-the-C-api object. What happens next?
You've hit upon exactly the trickiness involved and why I'm thinking the
Keep in mind that by "immutability" I'm talking about *really* immutable,
best approach initially is to only allow *strictly* immutable objects to pass between interpreters. Admittedly, my description of channels is very vague.:) There are a number of possibilities with them that I'm still exploring (CSP has particular opinions...), but immutability is a characteristic that may provide the simplest *initial* approach. Going that route shouldn't preclude adding some sort of support for mutable objects later. There aren't really many options for mutable objects, right? If you want shared nothing semantics, then transmitting a mutable object either needs to make a copy, or else be a real transfer, where the sender no longer has it (cf. Rust). I guess for the latter you'd need some new syntax for send-and-del, that requires the object to be self contained (all mutable objects reachable from it are only referenced by each other) and have only one reference in the sending process (which is the one being sent and then destroyed). perhaps going so far as treating the full memory space associated with an object as frozen. For instance, we'd have to ensure that "immutable" Python objects like strings, ints, and tuples do not change (i.e. via the C API). This seems like a red herring to me. It's already the case that you can't legally use the c api to mutate tuples, ints, for any object that's ever been, say, passed to a function. So for these objects, the subinterpreter setup doesn't actually add any new constraints on user code. C code is always going to be *able* to break memory safety so long as you're using shared-memory threading at the c level to implement this stuff. We just need to make it easy not to. Refcnts and garbage collection are another matter, of course. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/c2b41fe0/...> ------------------------------ Message: 2 Date: Sun, 21 Jun 2015 16:31:33 +1000 From: Nick Coghlan <ncoghlan@gmail.com> To: Nathaniel Smith <njs@pobox.com> Cc: Eric Snow <ericsnowcurrently@gmail.com>, python-ideas <python-ideas@python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CADiSq7cv538UBK9BE3e8eAakFB=njwHB-qnMu3m=qzLADzpsOg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On 21 June 2015 at 15:25, Nathaniel Smith <njs@pobox.com> wrote:
I was one of the folks pushing Eric in this direction, and that's because it's a possibility that was conceived of a few years back, but never tried due to lack of time (and inclination for those of us that are using Python primarily as an orchestration tool and hence spend most of our time on IO bound problems rather than CPU bound ones): http://www.curiousefficiency.org/posts/2012/07/volunteer-supported-free-thre... As mentioned there, I've at least spent some time with Graham Dumpleton over the past few years figuring out (and occasionally trying to address) some of the limitations of mod_wsgi's existing subinterpreter based WSGI app separation: https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Python_Sub_Inte... The fact that mod_wsgi can run most Python web applications in a subinterpreter quite happily means we already know the core mechanism works fine, and there don't appear to be any insurmountable technical hurdles between the status quo and getting to a point where we can either switch the GIL to a read/write lock where a write lock is only needed for inter-interpreter communications, or else find a way for subinterpreters to release the GIL entirely by restricting them appropriately. For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead), but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead (code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there). There's plenty of prior art here (including a system I once wrote in C myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with Eric's "simple matter of engineering" characterisation of the problem space. The main reason that subinterpreters have never had a Python API before is that they have enough rough edges that having to write a custom C extension module to access the API is the least of your problems if you decide you need them. At the same time, not having a Python API not only makes them much harder to test, which means various aspects of their operation are more likely to be broken, but also makes them inherently CPython specific. Eric's proposal essentially amounts to three things: 1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ------------------------------ Message: 3 Date: Sun, 21 Jun 2015 01:41:21 -0500 From: Wes Turner <wes.turner@gmail.com> To: Eric Snow <ericsnowcurrently@gmail.com> Cc: Nathaniel Smith <njs@pobox.com>, Python-Ideas <python-ideas@python.org> Subject: Re: [Python-ideas] solving multi-core Python Message-ID: <CACfEFw_1JVpUmZwFVkye-fbEsfH_NXVSo6WDMi1azhnXdY6PcA@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Exciting! * http://zero-buffer.readthedocs.org/en/latest/api-reference/#zero_buffer.Buff... * https://www.google.com/search?q=python+channels * https://docs.python.org/2/library/asyncore.html#module-asyncore * https://chan.readthedocs.org/en/latest/ * https://goless.readthedocs.org/en/latest/ * other approaches to the problem (with great APIs): * http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords * http://discodb.readthedocs.org/en/latest/ On Jun 20, 2015 5:55 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/e4aecde0/...> ------------------------------ Subject: Digest Footer _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas ------------------------------ End of Python-ideas Digest, Vol 103, Issue 100 **********************************************
participants (1)
-
jean-charles.douet＠laposte.net