Eric, On 2015-06-20 5:42 PM, Eric Snow wrote:

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them.

This is really great. Big +1 from me, and I'd be glad to help with the PEP/implementation. [...]

* only allow immutable objects to be shared between subinterpreters

Even if this is the only thing we have -- an efficient way for sharing immutable objects (such as bytes, strings, ints, and, stretching the definition of immutable, FDs) that will allow us to do a lot. Yury

On Jun 20, 2015 4:08 PM, "Nathaniel Smith" wrote:

On Jun 20, 2015 2:42 PM, "Eric Snow" wrote:

...

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them.

This all sounds really cool if you can pull it off, and shared-nothing

threads do seem like the least impossible model to pull off. Agreed.

But "least impossible" and "possible" are different :-). From your email I can't tell whether this plan is viable while preserving backcompat and memory safety.

I agree that those issues must be clearly solved in the proposal before it can be approved. I'm confident the approach I'm pursuing will afford us the necessary guarantees. I'll address those specific points directly when I can sit down and organize my thoughts.

Suppose I have a queue between two subinterpreters, and on this queue I

place a list of dicts of user-defined-in-python objects, each of which holds a reference to a user-defined-via-the-C-api object. What happens next? You've hit upon exactly the trickiness involved and why I'm thinking the best approach initially is to only allow *strictly* immutable objects to pass between interpreters. Admittedly, my description of channels is very vague.:) There are a number of possibilities with them that I'm still exploring (CSP has particular opinions...), but immutability is a characteristic that may provide the simplest *initial* approach. Going that route shouldn't preclude adding some sort of support for mutable objects later. Keep in mind that by "immutability" I'm talking about *really* immutable, perhaps going so far as treating the full memory space associated with an object as frozen. For instance, we'd have to ensure that "immutable" Python objects like strings, ints, and tuples do not change (i.e. via the C API). The contents of involved tuples/containers would have to be likewise immutable. Even changing refcounts could be too much, hence the idea of moving refcounts out to a separate table. This level of immutability would be something new to Python. We'll see if it's necessary. If it isn't too much work it might be a good idea regardless of the multi-core proposal. Also note that Barry has a (rejected) PEP from a number of years ago about freezing objects... That idea is likely out of scope as relates to my proposal, but it certainly factors in the problem space. -eric

On Jun 20, 2015 4:55 PM, "Devin Jeanpierre" wrote:

It's worthwhile to consider fork as an alternative. IMO we'd get a lot out of making forking safer, easier, and more efficient. (e.g. respectively: adding an atfork registration mechanism; separating out the bits of multiprocessing that use pickle from those that don't; moving the refcount to a separate page, or allowing it to be frozen prior to a fork.)

So leverage a common base of code with the multiprocessing module?

It sounds to me like this approach would use more memory than either regular threaded code or forking, so its main advantages are being cross-platform and less bug-prone. Is that right?

I would expect subinterpreters to use less memory. Furthermore creating them would be significantly faster. Passing objects between them would be much more efficient. And, yes, cross-platform.

Note: I don't count the IPC cost of forking, because at least on linux, any way to efficiently share objects between independent interpreters in separate threads can also be ported to independent interpreters in forked subprocesses,

How so? Subinterpreters are in the same process. For this proposal each would be on its own thread. Sharing objects between them through channels would be more efficient than IPC. Perhaps I've missed something?

and *should* be.

See also: multiprocessing.Value/Array. This is probably a good opportunity for that unification you mentioned. :)

I'll look.

On Sat, Jun 20, 2015 at 3:04 PM, Yury Selivanov

wrote:

...

On 2015-06-20 5:42 PM, Eric Snow wrote:

...

* only allow immutable objects to be shared between subinterpreters

Even if this is the only thing we have -- an efficient way for sharing immutable objects (such as bytes, strings, ints, and, stretching the definition of immutable, FDs) that will allow us to do a lot.

+1, this has a lot of utility, and can be extended naturally to other types and circumstances.

Agreed. -eric

On Sat, Jun 20, 2015 at 6:41 PM, Chris Angelico wrote:

On Sun, Jun 21, 2015 at 7:42 AM, Eric Snow wrote:

...

* disallow forking within subinterpreters

I love the idea as a whole (if only because the detractors can be told "Just use subinterpreters, then you get concurrency"), but this seems like a tricky restriction. That means no subprocess.Popen, no shelling out to other applications. And I don't know what of other restrictions might limit any given program.

This is just something I'm thinking about. To be honest, forking probably won't be a problem. Furthermore, if there were any restriction it would likely just be on forking Python (a la multiprocessing). However, I doubt there will be a need to pursue such a restriction. As I said, there are still a lot of open questions and subtle details to sort out.

Will it feel like subinterpreters are "write your code according to these tight restrictions and it'll work", or will it be more of "most programs will run in parallel just fine, but there are a few things to be careful of"?

I expect that will be somewhat the case no matter what. The less restrictions the better, though. :) It's a balancing act because I expect that with some initial restrictions we can land the feature sooner. Then we could look into how to relax the restrictions. I just want to be careful that we don't paint ourselves into a corner in that regard. -eric

On Sunday, June 21, 2015 at 6:12:22 AM UTC+5:30, Chris Angelico wrote:

On Sun, Jun 21, 2015 at 7:42 AM, Eric Snow javascript:> wrote:

...

* disallow forking within subinterpreters

I love the idea as a whole (if only because the detractors can be told "Just use subinterpreters, then you get concurrency"), but this seems like a tricky restriction. That means no subprocess.Popen, no shelling out to other applications. And I don't know what of other restrictions might limit any given program. Will it feel like subinterpreters are "write your code according to these tight restrictions and it'll work", or will it be more of "most programs will run in parallel just fine, but there are a few things to be careful of"?

ChrisA

Its good to get our terminology right: Are we talking parallelism or concurrency? Some references on the distinction: Bob Harper: https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurre... Rob Pike: http://concur.rspace.googlecode.com/hg/talk/concur.html#landing-slide [Or if you prefer the more famous https://www.youtube.com/watch?v=cN_DpYBzKso ]

On Jun 20, 2015 9:38 PM, "Steven D'Aprano" wrote:

On Sat, Jun 20, 2015 at 03:42:33PM -0600, Eric Snow wrote:

...

* only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables)

That doesn't sound very Pythonic to me. That's going to limit the usefulness of these subinterpreters.

It certainly would limit their usefulness. It's a tradeoff to make the project tractable. I'm certainly not opposed to dropping such restrictions, now or as a follow-up project. Also keep in mind that the restriction is only something I'm considering. It's too early to settle on many of these details.

...

* object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters

Isn't that a contradiction? If objects are read-only in all subinterpreters (except one), how can they be read/write in all subinterpreters?

True. The two statements, like the rest in the section, are summarizing different details and ideas into which I've been looking. Several of them are mutually exclusive.

All this talk about subinterpreters reminds me of an interesting blog post by Armin Ronacher:

http:// http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see

lucumr.pocoo.org http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see /2014/8/16/the-python-i-would-like-to-see http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see

Interesting. I'd read that before, but not recently. Armin has some interesting points but I can't say that I agree with his analysis or his conclusions. Regardless...

He's quite critical of a number of internal details of the CPython interpreter. But what I take from his post is that there could be significant advantages to giving the CPython interpreter its own local environment, like Lua and Javascript typically do, rather than the current model where there is a single process-wide global environment. Instead of having multiple subinterpreters all running inside the main interpreter, you could have multiple interpreters running in the same process, each with their own environment.

But that's effectively the goal! This proposal will not work if the interpreters are not isolated. I'm not clear on what Armin thinks is shared between interpreters. The only consequential shared piece is the GIL and my proposal should render the GIL irrelevant for the most part.

I may be completely misinterpreting things here, but as I understand it, this would remove the need for the GIL, allowing even plain old threads to take advantage of multiple cores. But that's a separate issue.

If we restrict each subinterpreter to a single thread and are careful with how objects are shared (and sort out exrension modules) then there will be no need for the GIL *within* each subinterpreter. However there are a couple of things that will keep the GIL around for now.

Armin writes:

I would like to see an internal interpreter design could be based on interpreters that work independent of each other, with local base types and more, similar to how JavaScript works. This would immediately open up the door again for embedding and concurrency based on message passing. CPUs won't get any faster :)

That's almost exactly what I'm aiming for. :) -eric

On 21 June 2015 at 15:25, Nathaniel Smith wrote:

On Jun 20, 2015 3:54 PM, "Eric Snow" wrote:

...

On Jun 20, 2015 4:08 PM, "Nathaniel Smith" wrote:

...

On Jun 20, 2015 2:42 PM, "Eric Snow" wrote:

...

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them.

This all sounds really cool if you can pull it off, and shared-nothing threads do seem like the least impossible model to pull off.

Agreed.

...

But "least impossible" and "possible" are different :-). From your email I can't tell whether this plan is viable while preserving backcompat and memory safety.

I agree that those issues must be clearly solved in the proposal before it can be approved. I'm confident the approach I'm pursuing will afford us the necessary guarantees. I'll address those specific points directly when I can sit down and organize my thoughts.

I'd love to see just a hand wavy, verbal proof-of-concept walking through how this might work in some simple but realistic case. To me a single compelling example could make this proposal feel much more concrete and achievable.

I was one of the folks pushing Eric in this direction, and that's because it's a possibility that was conceived of a few years back, but never tried due to lack of time (and inclination for those of us that are using Python primarily as an orchestration tool and hence spend most of our time on IO bound problems rather than CPU bound ones): http://www.curiousefficiency.org/posts/2012/07/volunteer-supported-free-thre... As mentioned there, I've at least spent some time with Graham Dumpleton over the past few years figuring out (and occasionally trying to address) some of the limitations of mod_wsgi's existing subinterpreter based WSGI app separation: https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Python_Sub_Inte... The fact that mod_wsgi can run most Python web applications in a subinterpreter quite happily means we already know the core mechanism works fine, and there don't appear to be any insurmountable technical hurdles between the status quo and getting to a point where we can either switch the GIL to a read/write lock where a write lock is only needed for inter-interpreter communications, or else find a way for subinterpreters to release the GIL entirely by restricting them appropriately. For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead), but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead (code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there). There's plenty of prior art here (including a system I once wrote in C myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with Eric's "simple matter of engineering" characterisation of the problem space. The main reason that subinterpreters have never had a Python API before is that they have enough rough edges that having to write a custom C extension module to access the API is the least of your problems if you decide you need them. At the same time, not having a Python API not only makes them much harder to test, which means various aspects of their operation are more likely to be broken, but also makes them inherently CPython specific. Eric's proposal essentially amounts to three things: 1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead)

And memcpy() updates pointer references to dependent objects magically? Surely you meant the memdeepcopy() function that's part of every standard C library! Regards Antoine.

On 21 June 2015 at 19:48, Antoine Pitrou wrote:

On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:

...

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead)

And memcpy() updates pointer references to dependent objects magically? Surely you meant the memdeepcopy() function that's part of every standard C library!

We already have the tools to do deep copies of object trees (although I'll concede I *was* actually thinking in terms of the classic C/C++ mistake of carelessly copying pointers around when I wrote that particular message). One of the options for deep copies tends to be a pickle/unpickle round trip, which will still incur the serialisation overhead, but not the IPC overhead. "Faster message passing than multiprocessing" sets the baseline pretty low, after all. However, this is also why Eric mentions the notions of object ownership or limiting channels to less than the full complement of Python objects. As an *added* feature at the Python level, it's possible to initially enforce restrictions that don't exist in the C level subinterpeter API, and then work to relax those restrictions over time. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 21 Jun 2015 20:25:47 +1000 Nick Coghlan wrote:

On 21 June 2015 at 19:48, Antoine Pitrou wrote:

...

On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:

...

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead)

And memcpy() updates pointer references to dependent objects magically? Surely you meant the memdeepcopy() function that's part of every standard C library!

We already have the tools to do deep copies of object trees [...] "Faster message passing than multiprocessing" sets the baseline pretty low, after all.

What's the goal? 10% faster? Or 10x? copy.deepcopy() uses similar internal mechanisms as pickle... Regards Antoine.

On 21 June 2015 at 20:41, Antoine Pitrou wrote:

On Sun, 21 Jun 2015 20:25:47 +1000 Nick Coghlan wrote:

...

On 21 June 2015 at 19:48, Antoine Pitrou wrote:

...

On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:

...

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead)

And memcpy() updates pointer references to dependent objects magically? Surely you meant the memdeepcopy() function that's part of every standard C library!

We already have the tools to do deep copies of object trees [...] "Faster message passing than multiprocessing" sets the baseline pretty low, after all.

What's the goal? 10% faster? Or 10x? copy.deepcopy() uses similar internal mechanisms as pickle...

I'd want us to eventually aim for zero-copy speed for at least known immutable values (int, str, float, etc), immutable containers of immutable values (tuple, frozenset), and for types that support both publishing and consuming data via the PEP 3118 buffer protocol without making a copy. For everything else I'd be fine with a starting point that was at least no slower than multiprocessing (which shouldn't be difficult, since we'll at least save the IPC overhead even if there are cases where communication between subinterpreters falls back to serialisation rather than doing something more CPU and memory efficient). As an implementation strategy, I'd actually suggest starting with *only* the latter for simplicity's sake, even though it misses out on some of the potential speed benefits of sharing an address space. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 21 Jun 2015 13:41:30 +0200 Sturla Molden wrote:

From the perspective of software design, it would be good it the CPython interpreter provided an environment instead of using global objects. It would mean that all functions in the C API would need to take the environment pointer as their first variable, which will be a major rewrite. It would also allow the "one interpreter per thread" design similar to tcl and .NET application domains.

From the point of view of API compatibility, it's unfortunately a no-no.

The reason IPC in multiprocessing is slow is due to calling pickle, it is not the IPC in itself.

No need to be pedantic :-) The "C" means communication, and pickling objects is part of the communication between Python processes.

All in all, I think we are better off finding a better way to share Python objects between processes.

Sure. This is however a complex and experimental topic (how to share a graph of garbage-collected objects between independant processes), with no guarantees of showing any results at the end.

P.S. Another thing to note is that with sub-interpreters, you can forget about using ctypes or anything else that uses the simplified GIL API (e.g. certain Cython generated extensions).

Indeed, the PyGILState API is still not subinterpreter-compatible. There's a proposal on the tracker, IIRC, but the interested parties never made any progress on it. Regards Antoine.

On Sat, Jun 20, 2015 at 11:31 PM, Nick Coghlan wrote:

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead), but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead (code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there).

The comparison is unfair -- if you can share between subinterpreters using memcpy, then you can share between processes using just a socket write, and multiprocessing becomes nearly just as fast.

Eric's proposal essentially amounts to three things:

1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines

All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process.

Why is that the goal? Whatever faults processes have, those are the problems, surely not processes in and of themselves, right? e.g. if the reason we don't like multiprocessed python is extra memory use, it's memory use we're opposed to. A solution that gives us parallel threads, but doesn't decrease memory consumption, doesn't solve anything. The solution has threads that are remarkably like processes, so I think it's really important to be careful about the differences and why this solution has the advantage. I'm not seeing that. And remember that we *do* have many examples of people using parallelized Python code in production. Are you sure you're satisfying their concerns, or whose concerns are you trying to satisfy? -- Devin

Nick Coghlan schrieb am 21.06.2015 um 03:28:

* there may be restrictions on some extension modules that limit them to "main interpreter only" (e.g. if the extension module itself isn't thread-safe, then it will need to remain fully protected by the GIL)

Just an idea, but C extensions could opt-in to this. Calling into them has to go through some kind of callable type, usually PyCFunction. We could protect all calls to extension types and C functions with a global runtime lock (per process, not per interpreter) and Extensions could set a flag on their functions and methods (or get it inherited from their extension types etc.) that says "I don't need the lock". That allows for a very fine-grained transition. Stefan

On Sun, 21 Jun 2015 13:52:36 +0200 Antoine Pitrou wrote:

...

P.S. Another thing to note is that with sub-interpreters, you can forget about using ctypes or anything else that uses the simplified GIL API (e.g. certain Cython generated extensions).

Indeed, the PyGILState API is still not subinterpreter-compatible. There's a proposal on the tracker, IIRC, but the interested parties never made any progress on it.

For reference: https://bugs.python.org/issue10915 https://bugs.python.org/issue15751 Regards Antoine.

Devin Jeanpierre wrote:

The comparison is unfair -- if you can share between subinterpreters using memcpy, then you can share between processes using just a socket write, and multiprocessing becomes nearly just as fast.

That is the main issue here. Writing to a pipe or a Unix socket is implemented with a memcpy in the kernel. So there is just a tiny constant overhead compared to using memcpy within a process. And with shared memory as IPC even this tiny overhead can be removed. The main overhead in communicating Python objects in multiprocessing is the serialization with pickle. So there is basically nothing to gain unless this part can be omitted. There is an errorneous belief among Windows programmers tht "IPC is slow". But that is because they are using out-proc DCOM server, CORBA, XMLRPC or something equally atrocious. A plain named pipe transaction is not in any way slow on Windows. Sturla

Antoine Pitrou wrote:

...

The reason IPC in multiprocessing is slow is due to calling pickle, it is not the IPC in itself.

No need to be pedantic :-) The "C" means communication, and pickling objects is part of the communication between Python processes.

Yes, currently it is. But is does not mean that it has to be. Clearly it is easier to avoid with multiple interpreters in the same process. But it does not mean it is unsolvable. Sturla

First, a minor question: instead of banning fork entirely within subinterpreters, why not just document that it is illegal to do anything between fork and exec in a subinterpreters, except for a very small (but possibly extensible) subset of Python? For example, after fork, you can no longer access any channels, and you also can't use signals, threads, fork again, imports, assignments to builtins, raising exceptions, or a whole host of other things (but of course if you exec an entirely new Python interpreter, it can do any of those things). C extension modules could just have a flag that marks whether the whole module is fork-safe or not (defaulting to not). So, this allows a subinterpreter to use subprocess (or even multiprocessing, as long as you use the forkserver or spawn mechanism), and it gives code that intentionally wants to do tricky/dangerous things a way to do them, but it avoids all of the problems with accidentally breaking a subinterpreter by forking it and then doing bad things. Second, a major question: In this proposal, are builtins and the modules map shared, or copied? If they're copied, it seems like it would be hard to do that even as efficiently as multiprocessing, much less more efficiently. Of course you could fake this with CoW, but I'm not sure how you'd do that, short of CoWing the entire heap (by using clone instead of pthreads on Linux, or by doing a bunch of explicit mmap and related calls on other POSIX systems), at which point you're pretty close to just implementing fork or vfork yourself to avoid calling fork or vfork, and unlikely to get it as efficient or as robust as what's already there. If they're shared, on the other hand, then it seems like it becomes very difficult to implement subinterpreter-safe code, because it's no longer safe to import a module, set a flag, call a registration function, etc.

On Sun, 21 Jun 2015 14:08:09 -0700 Andrew Barnert via Python-ideas wrote:

First, a minor question: instead of banning fork entirely within subinterpreters, why not just document that it is illegal to do anything between fork and exec in a subinterpreters, except for a very small (but possibly extensible) subset of Python?

It's actually already the case in POSIX that most things are illegal between fork() and exec(). However, to make fork() practical, many libraries or frameworks tend to ignore those problems deliberately. Regards Antoine.

On Mon, 22 Jun 2015 09:31:06 +1000 Nick Coghlan wrote:

Windows actually has superior native parallel execution APIs to Linux in some respects, but open source programming languages tend not to support them, presumably due to a combination of Microsoft's longstanding hostile perspective on open source licencing (which seems to finally be moderating with their new CEO), and the even longer standing POSIX mindset that "fork and file descriptors ought to be enough for anyone" (even if the workload in the child processes is wildly different from that in the main process).

Or perhaps the fact that those superiors APIs are a PITA. select() and friends may be crude performance-wise (though, strangely, we don't see providers migrating massively to Windows in order to improve I/O throughput), but they are simple to use. Regards Antoine.

On 22/06/15 01:39, Antoine Pitrou wrote:

Or perhaps the fact that those superiors APIs are a PITA.

Not all of them, no. HeapAlloc is a good example. Very easy to use, and the "one heap per thread" design often gives excellent performance compared to a single global heap. But on Linux we only have malloc et al., allocating from the global heap. How many Linux programmers have even considered using multiple heaps in combination with multi-threading? I can assure you it is not common. A good idea is to look at the Python C API. We have PyMem_Malloc, but nothing that compares to Windows' HeapAlloc. Not only does HeapAlloc remove the contention for the global heap, it can also serialize. Instead of serializing an object by traversing all references in the object tree, we just serialize the heap from which it was allocated. And as for garbage collection, why not deallocate the whole heap in one blow? Is the any reason to pair each malloc with free if one could just zap the whole heap? That is what HeapDestroy does. On Linux we would typically homebrew a memory pool to achieve the same thing. But a memory pool needs to traverse a chain of pointers and call free() multiple times, each time with contention for the spinlock protecting the global heap. And when allocating from a memory pool we also have contention for the global heap. It cannot in any way compare to the performance of the Win API HeapCreate/HeapDestroy and HeapAlloc/HeapFree. Sturla

On 21 June 2015 at 07:42, Eric Snow wrote:

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them.

This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources.

Feedback appreciated! Offers to help even more so! :)

It occurred to me in the context of another conversation that you (or someone else!) may be able to prototype some of the public API ideas for this using Jython and Vert.x: http://vertx.io/ That idea and some of the initial feedback in this thread also made me realise that it is going to be essential to keep in mind that there are key goals at two different layers here: * design a compelling implementation independent public API for CSP style programming in Python * use subinterpreters to implement that API efficiently in CPython There's a feedback loop between those two goals where limitations on what's feasible in CPython may constrain the design of the public API, and the design of the API may drive enhancements to the existing subinterpreter capability, but we shouldn't lose sight of the fact that they're *separate* goals. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 21, 2015 at 4:56 AM Devin Jeanpierre wrote:

On Sat, Jun 20, 2015 at 4:16 PM, Eric Snow wrote:

...

On Jun 20, 2015 4:55 PM, "Devin Jeanpierre"

wrote:

...

...

It's worthwhile to consider fork as an alternative. IMO we'd get a lot out of making forking safer, easier, and more efficient. (e.g. respectively: adding an atfork registration mechanism; separating out the bits of multiprocessing that use pickle from those that d, I still

disagreeon't;

...

moving the refcount to a separate page, or allowing it to be frozen prior to a fork.)

So leverage a common base of code with the multiprocessing module?

What is this question in response to? I don't understand.

...

I would expect subinterpreters to use less memory. Furthermore creating them would be significantly faster. Passing objects between them would be much more efficient. And, yes, cross-platform.

Maybe I don't understand how subinterpreters work. AIUI, the whole point of independent subinterpreters is that they share no state. So if I have a web server, each independent serving thread has to do all of the initialization (import HTTP libraries, etc.), right? Compare with forking, where the initialization is all done and then you fork, and you are immediately ready to serve, using the data structures shared with all the other workers, which is only copied when it is written to.

Unfortunately CPython subinterpreters do share some state, though it is not visible to the running code in many cases. Thus the other mentions of "wouldn't it be nice if CPython didn't assume a single global state per process" (100% agreed, but tangential to this discussion)... https://docs.python.org/3/c-api/init.html#sub-interpreter-support You are correct that some things that could make sense to share, such as imported modules, would not be shared as they are in a forked environment. This is an important oddity of subinterpreters: They have to re-import everything other than extension modules. When you've got a big process with a ton of modules (like, say, 100s of protocol buffers...), that's going to be a non-starter (pun intended) for the use of threads+subinterpreters as a fast form of concurrency if they need to import most of those from each subinterpreter. startup latency and cpu usage += lots. (possibly uses more memory as well but given our existing refcount implementation forcing needless PyObject page writes during a read causing fork to copy-on-write... impossible to guess) What this means for subinterpreters in this case is not much different from starting up multiple worker processes: You need to start them up and wait for them to be ready to serve, then reuse them as long as feasible before recycling them to start up a new one. The startup cost is high. I'm not entirely sold on this overall proposal, but I think a result of it *could* be to make our subinterpreter support better which would be a good thing. We have had to turn people away from subinterpreters in the past for use as part of their multithreaded C++ server where they wanted to occasionally run some Python code in embedded interpreters as part of serving some requests. Doing that would suddenly single thread their application (GIIIIIIL!) for all requests currently executing Python code despite multiple subinterpreters. The general advice for that: Run multiple Python processes and make RPCs to those from the C++ code. It allows for parallelism and ultimately scales better, if ever needed, as it can be easily spread across machines. Which one is more complex to maintain? Good question. -gps

Re passing objects, see below.

I do agree it's cross-platform, but right now that's the only thing I agree with.

...

...

Note: I don't count the IPC cost of forking, because at least on linux, any way to efficiently share objects between independent interpreters in separate threads can also be ported to independent interpreters in forked subprocesses,

How so? Subinterpreters are in the same process. For this proposal each would be on its own thread. Sharing objects between them through channels would be more efficient than IPC. Perhaps I've missed something?

You might be missing that memory can be shared between processes, not just threads, but I don't know.

The reason passing objects between processes is so slow is currently *nearly entirely* the cost of serialization. That is, it's the fact that you are passing an object to an entirely separate interpreter, and need to serialize the whole object graph and so on. If you can make that fast without serialization, for shared memory threads, then all the serialization becomes unnecessary, and you can either write to a pipe (fast, if it's a non-container), or used shared memory from the beginning (instantaneous). This is possible on any POSIX OS. Linux lets you go even further.

-- Devin _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

"Gregory P. Smith" wrote:

What this means for subinterpreters in this case is not much different from starting up multiple worker processes: You need to start them up and wait for them to be ready to serve, then reuse them as long as feasible before recycling them to start up a new one. The startup cost is high.

The statup cost for worker processes is high on Windows. It is very small on nearly any other OS. Sturla

On Mon, Jun 22, 2015 at 4:29 PM, Gregory P. Smith wrote:

On Mon, Jun 22, 2015 at 3:51 PM Sturla Molden wrote:

...

"Gregory P. Smith" wrote:

...

What this means for subinterpreters in this case is not much different from starting up multiple worker processes: You need to start them up and wait for them to be ready to serve, then reuse them as long as feasible before recycling them to start up a new one. The startup cost is high.

The statup cost for worker processes is high on Windows. It is very small on nearly any other OS.

While I understand that Windows adds some overhead there, startup time for Python worker processes is high on all OSes.

Python startup is slow in general. It slows down further based on the modules you must import before you can begin work.

Python does *very* little work on fork, which is what Sturla is alluding to. (Fork doesn't exist on Windows.) The only part I've found forking to be slow with is if you need to delay initialization of a thread pool and everything that depends on a thread pool until after the fork. This could hypothetically be made faster with subinterpreters if the thread pool was shared among all subinterpreters (e.g. if it was written in C.), but I would *expect* fork to be faster overall. That said, worker startup time is not actually very interesting anyway, since workers should restart rarely. I think its biggest impact is probably the time it takes to start your entire task from scratch. -- Devin

On Tue, Jun 23, 2015 at 3:03 AM, Gregory P. Smith wrote:

On Sat, Jun 20, 2015 at 5:42 PM Chris Angelico wrote:

...

On Sun, Jun 21, 2015 at 7:42 AM, Eric Snow wrote:

...

* disallow forking within subinterpreters

I love the idea as a whole (if only because the detractors can be told "Just use subinterpreters, then you get concurrency"), but this seems like a tricky restriction. That means no subprocess.Popen, no shelling out to other applications. And I don't know what of other restrictions might limit any given program. Will it feel like subinterpreters are "write your code according to these tight restrictions and it'll work", or will it be more of "most programs will run in parallel just fine, but there are a few things to be careful of"?

It wouldn't disallow use of subprocess, only os.fork(). C extension modules can alway fork. The restriction being placed in this scheme is: "if your extension module code forks from a subinterpreter, the child process MUST not return control to Python."

I'm not sure if this restriction would actually be needed or not but I agree with it regardless.

Oh! That's fine, then. Sounds good to me! ChrisA

On 23 June 2015 at 10:03, Chris Angelico wrote:

On Tue, Jun 23, 2015 at 9:59 AM, Nathaniel Smith wrote:

...

One possibility would be for subinterpreters to copy modules from the main interpreter -- I guess your average module is mostly dicts, strings, type objects, and functions; strings and functions are already immutable and could be shared without copying, and I guess copying the dicts and type objects into the subinterpreter is much cheaper than hitting the disk etc. to do a real import. (Though certainly not free.)

FWIW, functions aren't immutable, but code objects are.

Anything we come up with for optimised data sharing via channels could be applied to passing a prebuilt sys.modules dictionary through to subinterpreters. The key for me is to start from a well-defined "shared nothing" semantic model, but then look for ways to exploit the fact that we actually *are* running in the same address space to avoid copy objects. The current reference-counts-embedded-in-the-object-structs memory layout also plays havoc with the all-or-nothing page level copy-on-write semantics used by the fork() syscall at the operating system layer, so some of the ideas we've been considering (specifically, those related to moving the reference counter bookkeeping out of the object structs themselves) would potentially help with that as well (but would also have other hard to predict performance consequences). There's a reason Eric announced this as the *start* of a research project, rather than as a finished proposal - while it seems conceptually sound overall, there are a vast number of details to be considered that will no doubt hold a great many devils :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Jun 22, 2015 at 10:56 PM, wrote:

On Sun, Jun 21, 2015, at 18:41, Antoine Pitrou wrote:

...

It's actually already the case in POSIX that most things are illegal between fork() and exec(). However, to make fork() practical, many libraries or frameworks tend to ignore those problems deliberately.

I'm not _entirely_ sure that this applies to single-threaded programs, or even to multi-threaded programs that don't use constructs that will cause problems.

The text is: "A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls."

Note that it uses "may only" (which is ambiguous) rather than "shall only". It could be read that "only [stuff] until exec" is a suggestion of what the child process "may" do, under the circumstances described, to avoid the particular problems being discussed, rather than as a general prohibition.

Yeah, basically the way this works out is: (a) in practice on mainstream systems you can get away with forking and then doing whatever, so long as none of the threads in the parent process were holding any crucial locks, and the child is prepared for them to have all disappeared. (b) But, if something does break, then system builders reserve the right to laugh in your face. You can argue about things being technically ambiguous or whatever, but that's how it works. E.g. if you have a single-threaded program that does a matrix multiply, then forks, and then the child does a matrix multiply, and you run it on OS X linked to Apple's standard libraries, then the child will lock up, and if you report this to Apple they will close it as not-a-bug.

And the next paragraph is "When the application calls fork() from a signal handler and any of the fork handlers registered by pthread_atfork() calls a function that is not async-signal-safe, the behavior is undefined." suggesting that the behavior is _not_ likewise undefined when it was not called from a signal handler.

I wouldn't read anything into this. pthread_atfork registers three handlers, and two of them are run in the parent process, where normally they'd be allowed to call any functions they like. -n -- Nathaniel J. Smith -- http://vorpus.org

On 23.06.2015 13:57, Sturla Molden wrote:

On 23/06/15 01:29, Gregory P. Smith wrote:

...

While I understand that Windows adds some overhead there, startup time for Python worker processes is high on all OSes.

No it is not.

A fork() will clone the process. You don't need to run any initialization code after that. You don't need to start a new Python interpreter -- you already have one. You don't need to run module imports -- they are already imported. You don't need to pickle and build Python objects -- they are already there. Everything you had in the parent process is ready to use the child process. This magic happens so fast it is comparable to the time it takes Windows to start a thread.

To be fair, you will nevertheless get a slowdown when copy-on-write kicks in while first using whatever was cloned from the parent. This is nothing which blocks execution, but slows down execution. That is no time which can directly be measured during the fork() call, but I would still count it into start up cost. regards, jwi

On Jun 23, 2015, at 01:52 PM, Nick Coghlan wrote:

The current reference-counts-embedded-in-the-object-structs memory layout also plays havoc with the all-or-nothing page level copy-on-write semantics used by the fork() syscall at the operating system layer, so some of the ideas we've been considering (specifically, those related to moving the reference counter bookkeeping out of the object structs themselves) would potentially help with that as well (but would also have other hard to predict performance consequences).

A crazy offshoot idea would be something like Emacs' unexec, where during the build process you could preload a bunch of always-used immutable modules, then freeze the state in such a way that starting up again later would be much faster, because the imports (and probably more importantly, the searching) could be avoided. Cheers, -Barry

On Tue, Jun 23, 2015 at 9:01 AM Barry Warsaw wrote:

On Jun 23, 2015, at 01:52 PM, Nick Coghlan wrote:

...

The current reference-counts-embedded-in-the-object-structs memory layout also plays havoc with the all-or-nothing page level copy-on-write semantics used by the fork() syscall at the operating system layer, so some of the ideas we've been considering (specifically, those related to moving the reference counter bookkeeping out of the object structs themselves) would potentially help with that as well (but would also have other hard to predict performance consequences).

A crazy offshoot idea would be something like Emacs' unexec, where during the build process you could preload a bunch of always-used immutable modules, then freeze the state in such a way that starting up again later would be much faster, because the imports (and probably more importantly, the searching) could be avoided.

I actually would like something like this for Python, but I want it to work with hash randomization rather than freezing a single fixed hash seed. That means you'd need to record the location of all hash tables and cached hashes and fix them up after loading such a binary image at process start time, much like processing relocations when loading a binary executable. Non trivial. -gps

On Tue, Jun 23, 2015 at 7:55 AM, Sturla Molden wrote:

On 23/06/15 15:14, Jonas Wielicki wrote:

...

To be fair, you will nevertheless get a slowdown when copy-on-write kicks in while first using whatever was cloned from the parent. This is nothing which blocks execution, but slows down execution.

Yes, particularly because of reference counts. Unfortunately Python stores refcounts within the PyObject struct. And when a refcount is updated a copy of the entire 4 KB page is triggered. There would be fare less of this if refcounts was kept in dedicated pages.

A coworker of mine wrote a patch to Python that allows you to freeze refcounts for all existing objects before forking, if the correct compile options are set. This adds overhead to incref/decref, but dramatically changes the python+fork memory usage story. (I haven't personally played with it much, but it sounds decent.) If there's any interest I can try to upstream this change, guarded behind a compiler flag. We've also tried moving refcounts to their own pages, like you and Nick suggest, but it breaks a *lot* of third-party code. I can try to upstream it. If it's guarded by a compiler flag it is probably still useful, just any users would have to grep through their dependencies to make sure nothing directly accesses the refcount. (The stdlib can be made to work.) It sounds like it would also be useful for the main project in the topic of this thread, so I imagine there's more momentum behind it. -- Devin

On Sun, Jun 21, 2015 at 12:31 AM, Nick Coghlan wrote:

The fact that mod_wsgi can run most Python web applications in a subinterpreter quite happily means we already know the core mechanism works fine,

This is a pretty important point.

and there don't appear to be any insurmountable technical hurdles between the status quo and getting to a point where we can either switch the GIL to a read/write lock where a write lock is only needed for inter-interpreter communications, or else find a way for subinterpreters to release the GIL entirely by restricting them appropriately.

Proper multi-core operation will require at least some changes relative to the GIL. My goal is to execute the least amount of change at first. We can build on that.

For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead),

By initially focusing on immutable objects we shouldn't need to go that far. That said, a memcpy-based solution may very well be a good next step once the basic goals of the project are met.

but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead

Agreed. That's something we can experiment with once we get the core of the project working.

(code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there).

:)

There's plenty of prior art here (including a system I once wrote in C myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with Eric's "simple matter of engineering" characterisation of the problem space.

Good. :)

The main reason that subinterpreters have never had a Python API before is that they have enough rough edges that having to write a custom C extension module to access the API is the least of your problems if you decide you need them. At the same time, not having a Python API not only makes them much harder to test, which means various aspects of their operation are more likely to be broken, but also makes them inherently CPython specific.

Eric's proposal essentially amounts to three things:

1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines

All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process.

Yep. That sums it up pretty well. That decomposition should make it a bit easier to move the project forward. -eric

On Sun, Jun 21, 2015 at 3:54 AM, Antoine Pitrou wrote:

On Sat, 20 Jun 2015 23:01:20 -0600 Eric Snow wrote:

...

The only consequential shared piece is the GIL and my proposal should render the GIL irrelevant for the most part.

All singleton objects, built-in types are shared and probably a number of other things hidden in dark closets...

Yep. I expect we'll be able to sort those out under the assumption that 99% of the time they can be treated as immutable. We'll then have to find a way to keep the corner cases from breaking the subinterpreter isolation.

Not to mention the memory allocator.

This is a sticky part that I've been considering from almost day 1. It's not the #1 problem to solve, but it will be an important one if we want to have truly parallel subinterpreters.

By the way, what you're aiming to do is conceptually quite similar to Trent's PyParallel (thought Trent doesn't use subinterpreters, his main work is around trying to making object sharing safe without any GIL to trivially protect the sharing), so you may want to pair with him. Of course, you may end up with a Windows-only Python interpreter :-)

Right. I read through Trent's work on several occasions and have gleaned a couple lessons related to object sharing. I was planning on getting in touch with Trent in the near future.

I'm under the impression you're underestimating the task at hand here. Or perhaps you're not and you're just willing to present it in a positive way :-)

I'd like to think it's the latter. :) The main reason why I'm hopeful we can make a meaningful change for 3.6 is that I don't foresee any major changes to CPython's internals. Nearly all the necessary pieces are already there. <handwave/> I'm also intent on taking a minimal approach initially. We can build on it from there, easing restrictions that allowed us to roll out the initial implementation more quickly. All that said, I won't be surprised if it takes the entire 3.6 dev cycle to get it right. -eric

On Sun, Jun 21, 2015 at 4:40 AM, Stefan Behnel wrote:

If objects can make it explicit that they support sharing (and preferably are allowed to implement the exact details themselves), I'm sure we'll find ways to share NumPy arrays across subinterpreters. That feature alone tends to be a quick way to make a lot of people happy.

Are you thinking of something along the lines of a dunder method (e.g. __reduce__)? -eric

On Sun, Jun 21, 2015 at 4:57 AM, Nick Coghlan wrote:

I'd want us to eventually aim for zero-copy speed for at least known immutable values (int, str, float, etc), immutable containers of immutable values (tuple, frozenset), and for types that support both publishing and consuming data via the PEP 3118 buffer protocol without making a copy.

For everything else I'd be fine with a starting point that was at least no slower than multiprocessing (which shouldn't be difficult, since we'll at least save the IPC overhead even if there are cases where communication between subinterpreters falls back to serialisation rather than doing something more CPU and memory efficient).

Makes sense. -eric

On Sun, Jun 21, 2015 at 5:55 AM, Devin Jeanpierre wrote:

On Sat, Jun 20, 2015 at 4:16 PM, Eric Snow wrote:

...

On Jun 20, 2015 4:55 PM, "Devin Jeanpierre" wrote:

...

It's worthwhile to consider fork as an alternative. IMO we'd get a lot out of making forking safer, easier, and more efficient. (e.g. respectively: adding an atfork registration mechanism; separating out the bits of multiprocessing that use pickle from those that d, I still disagreeon't; moving the refcount to a separate page, or allowing it to be frozen prior to a fork.)

So leverage a common base of code with the multiprocessing module?

What is this question in response to? I don't understand.

It sounded like you were suggesting that we factor out a common code base that could be used by multiprocessing and the other machinery and that only multiprocessing would keep the pickle-related code.

...

I would expect subinterpreters to use less memory. Furthermore creating them would be significantly faster. Passing objects between them would be much more efficient. And, yes, cross-platform.

Maybe I don't understand how subinterpreters work. AIUI, the whole point of independent subinterpreters is that they share no state. So if I have a web server, each independent serving thread has to do all of the initialization (import HTTP libraries, etc.), right?

Yes. However, I expect that we could mitigate that cost to some extent.

Compare with forking, where the initialization is all done and then you fork, and you are immediately ready to serve, using the data structures shared with all the other workers, which is only copied when it is written to. So forking starts up faster and uses less memory (due to shared memory.)

But we are aiming for a share-nothing model with an efficient object-passing mechanism. Furthermore, subinterpreters do not have to be single-use. My proposal includes running tasks in an existing subinterpreter (e.g. executor pool), so that start-up cost is mitigated in cases where it matters. Note that ultimately my goal is to make it obvious and undeniable that Python (3.6+) has a good multi-core story. In my proposal, subinterpreters are a means to an end. If there's a better solution then great! As long as the real goal is met I'll be satisfied. :) For now I'm still confident that the subinterpreter approach is the best option for meeting the goal.

Re passing objects, see below.

I do agree it's cross-platform, but right now that's the only thing I agree with.

...

...

Note: I don't count the IPC cost of forking, because at least on linux, any way to efficiently share objects between independent interpreters in separate threads can also be ported to independent interpreters in forked subprocesses,

How so? Subinterpreters are in the same process. For this proposal each would be on its own thread. Sharing objects between them through channels would be more efficient than IPC. Perhaps I've missed something?

You might be missing that memory can be shared between processes, not just threads, but I don't know.

The reason passing objects between processes is so slow is currently *nearly entirely* the cost of serialization. That is, it's the fact that you are passing an object to an entirely separate interpreter, and need to serialize the whole object graph and so on. If you can make that fast without serialization,

That is a worthy goal!

for shared memory threads, then all the serialization becomes unnecessary, and you can either write to a pipe (fast, if it's a non-container), or used shared memory from the beginning (instantaneous). This is possible on any POSIX OS. Linux lets you go even further.

And this is faster than passing objects around within the same process? Does it play well with Python's memory model? -eric

On Sun, Jun 21, 2015 at 7:06 AM, Stefan Behnel wrote:

Nick Coghlan schrieb am 21.06.2015 um 03:28:

...

* there may be restrictions on some extension modules that limit them to "main interpreter only" (e.g. if the extension module itself isn't thread-safe, then it will need to remain fully protected by the GIL)

Just an idea, but C extensions could opt-in to this. Calling into them has to go through some kind of callable type, usually PyCFunction. We could protect all calls to extension types and C functions with a global runtime lock (per process, not per interpreter) and Extensions could set a flag on their functions and methods (or get it inherited from their extension types etc.) that says "I don't need the lock". That allows for a very fine-grained transition.

Exactly. PEP 489 helps facilitate opting in as well, right? -eric

On Sun, Jun 21, 2015 at 3:24 PM, Andrew Barnert via Python-ideas wrote:

On Jun 21, 2015, at 06:09, Nick Coghlan wrote:

...

Avoiding object serialisation is indeed the main objective. With subinterpreters, we have a lot more options for that than we do with any form of IPC, including shared references to immutable objects, and the PEP 3118 buffer API.

It seems like you could provide a way to efficiently copy and share deeper objects than integers and buffers without sharing everything, assuming the user code knows, at the time those objects are created, that they will be copied or shared. Basically, you allocate the objects into a separate arena (along with allocating their refcounts on a separate page, as already mentioned). You can't add a reference to an outside object in an arena-allocated object, although you can copy that outside object into the arena. And then you just pass or clone (possibly by using CoW memory-mapping calls, only falling back to memcpy on platforms that can't do that) entire arenas instead of individual objects (so you don't need the fictitious memdeepcpy function that someone ridiculed earlier in this thread, but you get 90% of the benefits of having one).

Yeah, I've been thinking of something along these lines. However, it's not the #1 issue to address so I haven't gotten too far into it. -eric

This has the same basic advantages of forking, but it's doable efficiently on Windows, and doable less efficiently (but still better than spawn and pass) on even weird embedded platforms, and it forces code to be explicit about what gets shared and copied without forcing it to work through less-natural queue-like APIs.

Also, it seems like you could fake this entire arena API on top of pickle/copy for a first implementation, then just replace the underlying implementation separately.

On Mon, Jun 22, 2015 at 5:59 PM, Nathaniel Smith wrote:

On Mon, Jun 22, 2015 at 10:37 AM, Gregory P. Smith wrote:

...

...

One possibility would be for subinterpreters to copy modules from the main interpreter -- I guess your average module is mostly dicts, strings, type objects, and functions; strings and functions are already immutable and could be shared without copying, and I guess copying the dicts and type objects into the subinterpreter is much cheaper than hitting the disk etc. to do a real import. (Though certainly not free.)

Yeah, I think there are a number of mechanisms we can explore to improve the efficiency of subinterpreter startup (and sharing).

This would have interesting semantic implications -- it would give similar effects to fork(), with subinterpreters starting from a snapshot of the main interpreter's global state.

...

I'm not entirely sold on this overall proposal, but I think a result of it could be to make our subinterpreter support better which would be a good thing.

We have had to turn people away from subinterpreters in the past for use as part of their multithreaded C++ server where they wanted to occasionally run some Python code in embedded interpreters as part of serving some requests. Doing that would suddenly single thread their application (GIIIIIIL!) for all requests currently executing Python code despite multiple subinterpreters.

I've also talked to HPC users who discovered this problem the hard way (e.g. http://www-atlas.lbl.gov/, folks working on the Large Hadron Collider) -- they've been using Python as an extension language in some large physics codes but are now porting those bits to C++ because of the GIL issues. (In this context startup overhead should be easily amortized, but switching to an RPC model is not going to happen.)

Would this proposal make a difference for them? -eric

On Tue, Jun 23, 2015 at 10:01 AM, Barry Warsaw wrote:

A crazy offshoot idea would be something like Emacs' unexec, where during the build process you could preload a bunch of always-used immutable modules, then freeze the state in such a way that starting up again later would be much faster, because the imports (and probably more importantly, the searching) could be avoided.

+1 -eric

On Tue, Jun 23, 2015 at 5:32 PM, Devin Jeanpierre wrote:

We've also tried moving refcounts to their own pages, like you and Nick suggest, but it breaks a *lot* of third-party code. I can try to upstream it. If it's guarded by a compiler flag it is probably still useful, just any users would have to grep through their dependencies to make sure nothing directly accesses the refcount. (The stdlib can be made to work.) It sounds like it would also be useful for the main project in the topic of this thread, so I imagine there's more momentum behind it.

Any indication of the performance impact? -eric

On 24 June 2015 at 15:33, Eric Snow wrote:

On Sun, Jun 21, 2015 at 7:06 AM, Stefan Behnel wrote:

...

Nick Coghlan schrieb am 21.06.2015 um 03:28:

...

* there may be restrictions on some extension modules that limit them to "main interpreter only" (e.g. if the extension module itself isn't thread-safe, then it will need to remain fully protected by the GIL)

Just an idea, but C extensions could opt-in to this. Calling into them has to go through some kind of callable type, usually PyCFunction. We could protect all calls to extension types and C functions with a global runtime lock (per process, not per interpreter) and Extensions could set a flag on their functions and methods (or get it inherited from their extension types etc.) that says "I don't need the lock". That allows for a very fine-grained transition.

Exactly. PEP 489 helps facilitate opting in as well, right?

Yep, as PEP 489 requires subinterpreter compatibility as a precondition for using multi-phase initialisation :) Cheers, Nick. P.S. Technically, what it actually requires is support for "multiple instances of the module existing in the same process at the same time", as it really recreates the module if you remove it from sys.modules and import it again, unlike single phase initialisation. But that's a mouthful, so "must support subinterpreters" is an easier shorthand. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 24, 2015 at 2:01 AM Eric Snow wrote:

On Sun, Jun 21, 2015 at 7:47 PM, Nick Coghlan wrote:

...

It occurred to me in the context of another conversation that you (or someone else!) may be able to prototype some of the public API ideas for this using Jython and Vert.x: http://vertx.io/

I'll take a look.

Note that Vert.x 3 was just released today, which (at least for now) drops support for Python. There is work underway to support it under version 3, but it's using CPython and Py4J, not Jython. You'd need to use Vert.x 2 to get Jython support: http://vertx.io/vertx2

On 24/06/15 07:01, Eric Snow wrote:

In return, my question is, what is the level of effort to get fork+IPC to do what we want vs. subinterpreters? Note that we need to accommodate Windows as more than an afterthought

Windows is really the problem. The absence of fork() is especially hurtful for an interpreted language like Python, in my opinion.

...

If you think avoiding IPC is clever, you are wrong. IPC is very fast, in fact programs written to use MPI tends to perform and scale better than programs written to use OpenMP in parallel computing.

I'd love to learn more about that. I'm sure there are some great lessons on efficiently and safely sharing data between isolated execution environments. That said, how does IPC compare to passing objects around within the same process?

There are two major competing standards for parallel computing in science and engineering: OpenMP and MPI. OpenMP is based on a shared memory model. MPI is based on a distributed memory model and use message passing (hence its name). The common implementations of OpenMP (GNU, Intel, Microsoft) are all implemented with threads. There are also OpenMP implementations for clusters (e.g. Intel), but from the programmer's perspective OpenMP is a shared memory model. The common implementations of MPI (MPICH, OpenMPI, Microsoft MPI) use processes instead of threads. Processes can run on the same computer or on different computers (aka "clusters"). On localhost shared memory is commonly used for message passing, on clusters MPI implementations will use networking protocols. The take-home message is that OpenMP is conceptually easier to use, but programs written to use MPI tend to be faster and scale better. This is even true when using a single computer, e.g. a laptop with one multicore CPU. Here is tl;dr explanation: As for ease of programming, it is easier to create a deadlock or livelock with MPI than OpenMP, even though programs written to use MPI tend to need fewer synchronization points. There is also less boilerplate code to type when using OpenMP, because we do not have to code object serialization, message passing, and object deserialization. For performance, programs written to use MPI seems to have a larger overhead because they require object serialization and message passing, whereas OpenMP threads can just share the same objects. The reality is actually the opposite, and is due to the internals of modern CPU, particularly hierarchichal memory, branch prediction and long pipelines. Because of hierarchichal memory, the cache used by CPUs and CPU cores must be kept in synch. Thus when using OpenMP (threads) there will be a lot of synchronization going on that the programmer does not see, but which the hardware will do behind the scenes. There will also be a lot of data passing between various cache levels on the CPU and RAM. If a core writes to a pice of memory it keeps in a cache line, a cascade of data traffic and synchronization can be triggered across all CPUs and cores. Not only will this stop the CPUs and prompt them to synchronize cache with RAM, it also invalidates their branch prediction and they must flush their pipelines and throw away work they have already done. The end result is a program that does not scale or perform very well, even though it does not seem to have any explicit synchronization points that could explain this. The term "false sharing" is often used to describe this problem. Programs written to use MPI are the opposite. There every instance of synchronization and message passing is visible. When a CPU core writes to memory kept in a cache line, it will never trigger synchronization and data traffic across all the CPUs. The scalability is as the program predicts. And even though memory and objects are not shared, there is actually much less data traffic going on. Which to use? Most people find it easier to use OpenMP, and it does not require a big runtime environment to be installed. But programs using MPI tend to be the faster and more scalable. If you need to ensure scalability on multicores, multiple processes are better than multiple threads. The scalability of MPI also applies to Python's multiprocessing. It is the isolated virtual memory of each process that allows the cores to run at full speed. Another thing to note is that Windows is not a second-class citizen when using MPI. The MPI runtime (usually an executable called mpirun or mpiexec) starts and manages a group of processes. It does not matter if they are started by fork() or CreateProcess().

Solving reference counts in this situation is a separate issue that will likely need to be resolved, regardless of which machinery we use to isolate task execution.

As long as we have a GIL, and we need the GIL to update a reference count, it does not hurt so much as it otherwise would. The GIL hides most of the scalability impact by serializing flow of execution.

IPC sounds great, but how well does it interact with Python's memory management/allocator? I haven't looked closely but I expect that multiprocessing does not use IPC anywhere.

multiprocessing does use IPC. Otherwise the processes could not communicate. One example is multiprocessing.Queue, which uses a pipe and a semaphore. Sturla

On 24/06/15 13:43, M.-A. Lemburg wrote:

That said, I still think the multiple-process is a better one (more robust, more compatible, fewer problems). We'd just need a way more efficient approach to sharing objects between the Python processes than using pickle and shared memory or pipes :-)

It is hard to get around shared memory, Unix domain sockets, or pipes. There must be some sort of IPC, regardless. One idea I have played with is to use a specialized queue instead of the current multiprocessing.Queue. In scientific computing we often need to pass arrays, so it would make sense to have a queue that could bypass pickle for NumPy arrays, scalars and dtypes, simply by using the NumPy C API to process the data. It could also have specialized code for a number of other objects -- at least str, int, float, complex, and PEP 3118 buffers, but perhaps also simple lists, tuples and dicts with these types. I think it should be possible to make a queue that would avoid the pickle issue for 99 % of scientific computing. It would be very easy to write such a queue with Cython and e.g. have it as a part of NumPy or SciPy. One thing I did some years ago was to have NumPy arrays that would store the data in shared memory. And when passed to multiprocessing.Queue they would not pickle the data buffer, only the metadata. However this did not improve on performance, because the pickle overhead was still there, and passing a lot of binary data over a pipe was not comparably expensive. So while it would save memory, it did not make programs using multiprocessing and NumPy more efficient. Sturla

On Wed, Jun 24, 2015 at 8:27 AM Sturla Molden wrote:

On 24/06/15 07:01, Eric Snow wrote:

...

In return, my question is, what is the level of effort to get fork+IPC to do what we want vs. subinterpreters? Note that we need to accommodate Windows as more than an afterthought

Windows is really the problem. The absence of fork() is especially hurtful for an interpreted language like Python, in my opinion.

You cannot assume that fork() is safe on any OS as a general solution for anything. This isn't a Windows specific problem, It simply cannot be relied upon in a general purpose library at all. It is incompatible with threads. The ways fork() can be used safely are in top level application decisions: There must be a guarantee of no threads running before all forking is done. (thus the impossibility of relying on it as a mechanism to do anything useful in a generic library - you are a library, you don't know what the whole application is doing or when you were called as part of it) A concurrency model that assumes that it is fine to fork() and let child processes continue to execute is not usable by everyone. (ie: multiprocessing until http://bugs.python.org/issue8713 was implemented). -gps

...

...

If you think avoiding IPC is clever, you are wrong. IPC is very fast, in fact programs written to use MPI tends to perform and scale better than programs written to use OpenMP in parallel computing.

I'd love to learn more about that. I'm sure there are some great lessons on efficiently and safely sharing data between isolated execution environments. That said, how does IPC compare to passing objects around within the same process?

There are two major competing standards for parallel computing in science and engineering: OpenMP and MPI. OpenMP is based on a shared memory model. MPI is based on a distributed memory model and use message passing (hence its name).

The common implementations of OpenMP (GNU, Intel, Microsoft) are all implemented with threads. There are also OpenMP implementations for clusters (e.g. Intel), but from the programmer's perspective OpenMP is a shared memory model.

The common implementations of MPI (MPICH, OpenMPI, Microsoft MPI) use processes instead of threads. Processes can run on the same computer or on different computers (aka "clusters"). On localhost shared memory is commonly used for message passing, on clusters MPI implementations will use networking protocols.

The take-home message is that OpenMP is conceptually easier to use, but programs written to use MPI tend to be faster and scale better. This is even true when using a single computer, e.g. a laptop with one multicore CPU.

Here is tl;dr explanation:

As for ease of programming, it is easier to create a deadlock or livelock with MPI than OpenMP, even though programs written to use MPI tend to need fewer synchronization points. There is also less boilerplate code to type when using OpenMP, because we do not have to code object serialization, message passing, and object deserialization.

For performance, programs written to use MPI seems to have a larger overhead because they require object serialization and message passing, whereas OpenMP threads can just share the same objects. The reality is actually the opposite, and is due to the internals of modern CPU, particularly hierarchichal memory, branch prediction and long pipelines.

Because of hierarchichal memory, the cache used by CPUs and CPU cores must be kept in synch. Thus when using OpenMP (threads) there will be a lot of synchronization going on that the programmer does not see, but which the hardware will do behind the scenes. There will also be a lot of data passing between various cache levels on the CPU and RAM. If a core writes to a pice of memory it keeps in a cache line, a cascade of data traffic and synchronization can be triggered across all CPUs and cores. Not only will this stop the CPUs and prompt them to synchronize cache with RAM, it also invalidates their branch prediction and they must flush their pipelines and throw away work they have already done. The end result is a program that does not scale or perform very well, even though it does not seem to have any explicit synchronization points that could explain this. The term "false sharing" is often used to describe this problem.

Programs written to use MPI are the opposite. There every instance of synchronization and message passing is visible. When a CPU core writes to memory kept in a cache line, it will never trigger synchronization and data traffic across all the CPUs. The scalability is as the program predicts. And even though memory and objects are not shared, there is actually much less data traffic going on.

Which to use? Most people find it easier to use OpenMP, and it does not require a big runtime environment to be installed. But programs using MPI tend to be the faster and more scalable. If you need to ensure scalability on multicores, multiple processes are better than multiple threads. The scalability of MPI also applies to Python's multiprocessing. It is the isolated virtual memory of each process that allows the cores to run at full speed.

Another thing to note is that Windows is not a second-class citizen when using MPI. The MPI runtime (usually an executable called mpirun or mpiexec) starts and manages a group of processes. It does not matter if they are started by fork() or CreateProcess().

...

Solving reference counts in this situation is a separate issue that will likely need to be resolved, regardless of which machinery we use to isolate task execution.

As long as we have a GIL, and we need the GIL to update a reference count, it does not hurt so much as it otherwise would. The GIL hides most of the scalability impact by serializing flow of execution.

...

IPC sounds great, but how well does it interact with Python's memory management/allocator? I haven't looked closely but I expect that multiprocessing does not use IPC anywhere.

multiprocessing does use IPC. Otherwise the processes could not communicate. One example is multiprocessing.Queue, which uses a pipe and a semaphore.

Sturla

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 24/06/15 22:50, M.-A. Lemburg wrote:

The tricky part is managing pointers in those data structures, e.g. a container types for other Python objects will have to store all referenced objects in the shared memory segment as well.

If a container type for Python objects contains some unknown object type we would have to use pickle as fallback.

For NumPy arrays using simple types this is a lot easier, since you don't have to deal with pointers to other objects.

The objects we deal with in scientific computing are usually arrays with a rather regular structure, not deeply nested Python objects. Even a more complex object like scipy.spatial.cKDTree is just a collection of a few contiguous arrays under the hood. So we could for most parts squash the pickle overhead that anyone will encounter by specializing a queue that has knowledge about a small set of Python types.

When saying "passing a lot of binary data over a pipe" you mean the meta-data ?

No, I mean the buffer pointed to by PyArray_DATA(obj) when using the NumPy C API. We have to send a lot of raw bytes over an IPC mechanism before this communication compares to the pickle overhead. Sturla

On Jun 24, 2015 4:49 PM, "Sturla Molden" wrote:

On 24/06/15 23:41, Sturla Molden wrote:

...

So we could for most parts squash the pickle overhead that anyone will encounter by specializing a queue that has knowledge about a small set of Python types.

But this would be very domain specific for scientific and numerical

computing, it would not be a general improvement for multiprocessing with Python. Basically C structs like Thrift or Protocol Buffers?

Sturla

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

I'm going to break mail client threading and also answer some of your other emails here. On Tue, Jun 23, 2015 at 10:26 PM, Eric Snow wrote:

It sounded like you were suggesting that we factor out a common code base that could be used by multiprocessing and the other machinery and that only multiprocessing would keep the pickle-related code.

Yes, I like that idea a lot.

...

Compare with forking, where the initialization is all done and then you fork, and you are immediately ready to serve, using the data structures shared with all the other workers, which is only copied when it is written to. So forking starts up faster and uses less memory (due to shared memory.)

But we are aiming for a share-nothing model with an efficient object-passing mechanism. Furthermore, subinterpreters do not have to be single-use. My proposal includes running tasks in an existing subinterpreter (e.g. executor pool), so that start-up cost is mitigated in cases where it matters.

Note that ultimately my goal is to make it obvious and undeniable that Python (3.6+) has a good multi-core story. In my proposal, subinterpreters are a means to an end. If there's a better solution then great! As long as the real goal is met I'll be satisfied. :) For now I'm still confident that the subinterpreter approach is the best option for meeting the goal.

Ahead of time: the following is my opinion. My opinions are my own, and bizarre, unlike the opinions of my employer and coworkers. (Who are also reading this maybe.) So there's two reasons I can think of to use threads for CPU parallelism: - My thing does a lot of parallel work, and so I want to save on memory by sharing an address space This only becomes an especially pressing concern if you start running tens of thousands or more of workers. Fork also allows this. - My thing does a lot of communication, and so I want fast communication through a shared address space This can become a pressing concern immediately, and so is a more visible issue. However, it's also a non-problem for many kinds of tasks which just take requests in and put output back out, without talking with other members of the pool (e.g. writing an RPC server or HTTP server.) I would also speculate that once you're on many machines, unless you're very specific with your design, RPC costs dominate IPC costs to the point where optimizing IPC doesn't do a lot for you. On Unix, IPC can be free or cheap due to shared memory. Threads really aren't all that important, and if we need them, we have them. When people tell me in #python that multicore in Python is bad because of the GIL, I point them at fork and at C extensions, but also at PyPy-STM and Jython. Everything has problems, but then so does this proposal, right?

And this is faster than passing objects around within the same process? Does it play well with Python's memory model?

As far as whether it plays with the memory model, multiprocessing.Value() just works, today. To make it even lower overhead (not construct an int PyObject* on the fly), you need to change things, e.g. the way refcounts work. I think it's possibly feasible. If not, at least the overhead would be negligible. Same applies to strings and other non-compound datatypes. Compound datatypes are hard even for the subinterpreter case, just because the objects you're referring to are not likely to exist on the other end, so you need a real copy. I'm sure you've thought about this. multiprocessing.Array has a solution for this, which is to unbox the contained values. It won't work with tuples.

I'd be interested in more info on both the refcount freezing and the sepatate refcounts pages.

I can describe the patches: - separate refcounts replaces refcount with a pointer to refcount, and changes incref/decref. - refcount freezing lets you walk all objects and set the reference count to a magic value. incref/decref check if the refcount is frozen before working. With freezing, unlike this approach to separate refcounts, anyone that touches the refcount manually will just dirty the page and unfreeze the refcount, rather than crashing the process. Both of them will decrease performance for non-forking python code, but for forking code it can be made up for e.g. by increased worker lifetime and decreased rate of page copying, plus the whole CPU vs memory tradeoff. I legitimately don't remember the difference in performance, which is good because I'm probably not allowed to say what it was, as it was tested on our actual app and not microbenchmarks. ;)

...

And remember that we *do* have many examples of people using parallelized Python code in production. Are you sure you're satisfying their concerns, or whose concerns are you trying to satisfy?

Another good point. What would you suggest is the best way to find out?

I don't necessarily mean that. I mean that this thread feels like you posed an answer and I'm not sure what the question is. Is it about solving a real technical problem? What is that, and who does it affect? A new question I didn't ask before: is the problem with Python as a whole, or just CPython? -- Devin

On Wed, Jun 24, 2015 at 10:28 AM, Sturla Molden wrote:

On 24/06/15 07:01, Eric Snow wrote:

...

Well, perception is 9/10ths of the law. :) If the multi-core problem is already solved in Python then why does it fail in the court of public opinion. The perception that Python lacks a good multi-core story is real, leads organizations away from Python, and will not improve without concrete changes.

I think it is a combination of FUD and the lack of fork() on Windows. There is a lot of utterly wrong information about CPython and its GIL.

Thanks for a clear summary of the common misunderstandings. While I agreed with your points, they are mostly the same things we have been communicating for many years, to no avail. They are also oriented toward larger-scale parallelism (which I don't mean to discount). That makes it easier to misunderstand. Why? Because there are enough caveats and performance downsides (see Dave Beazley's PyCon 2015 talk) that most folks stop trying to rationalize, throw their hands up, and say "Python concurrency stinks" and "you can't *really* do multicore on Python". I have personal experience with high-profile decision makers where this is exactly what happened, with adverse consequences to support for Python within the organizations. To change this perception we need to give folks a simpler, performant concurrency model that takes advantage of multiple cores. My proposal is all about doing at least *something* that makes Python's multi-core story obvious and undeniable. *That* is my entire goal with this proposal. Clearly I have opinions on the best approach to achieve that in the 3.6 timeframe. :) However, I am quite willing to investigate all the options (as I hope this thread demonstrates). So, again, thanks for the feedback and insight. You've provided me with plenty of food for thought. -eric

On 25/06/15 00:56, Eric Snow wrote:

Why? Because there are enough caveats and performance downsides (see Dave Beazley's PyCon 2015 talk) that most folks stop trying to rationalize, throw their hands up, and say "Python concurrency stinks" and "you can't *really* do multicore on Python".

Yes, that seems to be the case.

To change this perception we need to give folks a simpler, performant concurrency model that takes advantage of multiple cores. My proposal is all about doing at least *something* that makes Python's multi-core story obvious and undeniable.

I think the main issue with subinterpreters and a message-passing model is that it will be very difficult to avoid deep copies of Python objects. And in that case all we have achieved compared to multiprocessing is less scalability. Also you have not removed the GIL, so the FUD about the dreaded GIL will still be around. Clearly introducing multiprocessing in the standard library did nothing to reduce this. Sturla

On 25/06/15 00:19, Eric Snow wrote:

...

...

Solving reference counts in this situation is a separate issue that will likely need to be resolved, regardless of which machinery we use to isolate task execution.

As long as we have a GIL, and we need the GIL to update a reference count, it does not hurt so much as it otherwise would. The GIL hides most of the scalability impact by serializing flow of execution.

It does hurt in COW situations, e.g. forking. My expectation is that we'll at least need to take a serious look into the matter in the short term (i.e. Python 3.6).

Yes. It hurts performance after forking as reference counting will trigger a lot of page copies. Keeping reference counts in separate pages and replacing the field in the PyObject struct would reduce this problem by a factor of up to 512 (64 bit) or 1024 (32 bit). It does not hurt performance with multi-threading, as Python threads are serialized by the GIL. But if the GIL was removed it would result in a lot of false sharing. That is a major reason we need a tracing garbage collector instead of reference counting if we shall be able to remove the GIL. Sturla

On 25/06/15 02:09, Devin Jeanpierre wrote:

Although, threads are better for memory usage, by defaulting to sharing even on write. (Good for memory, maybe not so good for bug-freedom...)

I am not sure. Code written to use OpenMP tend to have less bugs than code written to use MPI. This suggests that shared memory is easier than message-passing, which is contrary to the common belief. My own experience with OpenMP and MPI suggests it is easier to create a deadlock with message-passing than accidentally have threads access the same address concurrently. This is also what I hear from other people who writes code for scientific computing. I see a lot of claims that message-passing is supposed to be "safer" than a shared memory model, but that is not what we see with OpenMP and MPI. With MPI, the programmer must make sure that the send and receive commands are passed in the right order at the right time, in each process. This leaves plenty of room for messing up or creating unmaintainable spaghetti code, particularly in a complex algorithm. It is easier to make sure all shared objects are protected with mutexes than to make sure a spaghetti of send and receive messages are in correct order. It might be that Python's queue method of passing messages leave less room for deadlocking than the socket-like MPI_send and MPI_recv functions. But I think message-passing are sometimes overrated as "the safe solution" to multi-core programming (cf. Go and Erlang). Sturla

Nathaniel Smith wrote:

OpenMP is an *extremely* structured and constrained subset of shared memory multithreading, and not at all comparable to pthreads/threading.py/whatever.

If you use "parallel section" it is almost as free as using pthreads directly. But if you stick to "parallel for", which most do, you have a rather constrained and more well-behaved subset. I am quite sure MPI can even be a source of more errors than pthreads used directly. Getting message passing right inside a complex algorithm is not funny. I would rather keep my mind focused on which objects to protect with a lock or when to signal a condition. Sturla

On Tue, Jun 23, 2015 at 11:01:24PM -0600, Eric Snow wrote:

On Sun, Jun 21, 2015 at 5:41 AM, Sturla Molden wrote:

...

From the perspective of software design, it would be good it the CPython interpreter provided an environment instead of using global objects. It would mean that all functions in the C API would need to take the environment pointer as their first variable, which will be a major rewrite. It would also allow the "one interpreter per thread" design similar to tcl and .NET application domains.

While perhaps a worthy goal, I don't know that it fits in well with my goals. I'm aiming for an improved multi-core story with a minimum of change in the interpreter.

This slide and the following two are particularly relevant: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=... I elicit three categories of contemporary problems where efficient use of multiple cores would be desirable: 1) Computationally-intensive work against large data sets (the traditional "parallel" HPC/science/engineering space, and lately, to today's "Big Data" space). 2a) Serving tens/hundreds of thousands of network clients with non-trivial computation required per-request (i.e. more than just buffer copying between two sockets); best example being the modern day web server, or: 2b) Serving far fewer clients, but striving for the lowest latency possible in an environment with "maximum permitted latency" restrictions (or percentile targets, 99s etc). In all three problem domains, there is a clear inflection point at which multiple cores would overtake a single core in either: 1) Reducing the overall computation time. 2a|b) Serving a greater number of clients (or being able to perform more complex computation per request) before hitting maximum permitted latency limits. For PyParallel, I focused on 2a and 2b. More specifically, a TCP/IP socket server that had the ability to dynamically adjust its behavior (low latency vs concurrency vs throughput[1]), whilst maintaining optimal usage of underlying hardware[2]. That is: given sufficient load, you should be able to saturate all I/O channels (network and disk), or all cores, or both, with *useful* work. (The next step after saturation is sustained saturation (given sufficient load), which can be even harder to achieve, as you need to factor in latencies for "upcoming I/O" ahead of time if your computation is driven by the results of a disk read (or database cursor fetch).) (Sturla commented on the "import-DDoS" that you can run into on POSIX systems, which is a good example. You're saturating your underlying hardware, sure, but you're not doing useful work -- it's important to distinguish the two.) Dynamically adjusting behavior based on low latency vs concurrency vs throughput: [1]: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploite... https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploite... Optimal hardware use: [2]: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=... So, with the focus of PyParallel established (socket server that could exploit all cores), my hypothesis was that I could find a new way of doing things that was more performant than the status quo. (In particular, I wanted to make sure I had an answer for "why not just use multiprocessing?" -- which is an important question.) https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=... So, I also made the decision to leverage threads for parallelism and not processes+IPC, which it sounds like you're leaning toward as well. Actually, other than the subinterpreter implementation aspect, everything you've described is basically on par with PyParallel, more or less. Now, going back to your original comment:

While perhaps a worthy goal, I don't know that it fits in well with my goals. I'm aiming for an improved multi-core story with a minimum of change in the interpreter.

That last sentence is very vague as multi-core means different things to different people. What is the problem domain you're going to try and initially target? Computationally-intensive parallel workloads like in 1), or the network I/O-driven socket server stuff like in 2a/2b? I'd argue it should be the latter. Reason being is that you'll rarely see the former problem tackled solely by pure Python -- e.g. Python may be gluing everything together, but the actual computation will be handled by something like NumPy/Numba/Fortran/Cython or custom C stuff, and, as Sturla's mentioned, OpenMP and MPI usually gets involved to manage the parallel aspect. For the I/O-driven socket server stuff, though, you already have this nice delineation of what would be run serially versus what would be ideal to run in parallel: import datrie import numpy as np import pyodbc import async from collections import defaultdict from async.http.server import ( router, make_routes, HttpServer, RangedRequest, ) # Tell PyParallel to invoke the tp_dealloc method explicitly # for these classes when rewinding a heap after a parallel # callback has finished. (Implementation detail: this toggles # the Py_TPFLAGS_PX_DEALLOC flag in the TypeObject's tp_flags; # when PyParallel intercepts PyObject_NEW/INIT (init_object), # classes (PyTypeObject *tp) with this flag set will be tracked # in a linked-list that is local to the parallel context being # used to service this client. When the context has its heaps # rewound back to the initial state at the time of the snapshot, # it will call tp_dealloc() explicitly against all objects of # this type that were encountered.) async.register_dealloc(pyodbc.Connection) async.register_dealloc(pyodbc.Cursor) async.register_dealloc(pyodbc.Row) # Load 29 million titles. RSS += ~9.5GB. TITLES = datrie.Trie.load('titles.trie') # Load 15 million 64-bit offsets. RSS += ~200MB. OFFSETS = np.load('offsets.npy') XML = 'enwiki-20150205-pages-articles.xml' class WikiServer(HttpServer): # All of these methods are automatically invoked in # parallel. HttpServer implements a data_received() # method which prepares the request object and then # calls the relevant method depending on the URL, e.g. # http://localhost/user/foo will call the user(request, # name='foo'). If we want to "write" to the client, # we return a bytes, bytearray or unicode object from # our callback (that is, we don't expose a socket.write() # to the user). # # Just before the PyParallel machinery invokes the # callback (via a simple PyObject_CallObject), though, # it takes a snapshot of its current state, such that # the exact state can be rolled back to (termed a socket # "rewind") when this callback is complete. If we don't # return a sendable object back, this rewind happens # immediately, and then we go straight into a read call. # If we do return something sendable, we send it. When # that send completes, *then* we do the rewind, then we # issue the next read/recv call. # # This approach is particularly well suited to parallel # callback execution because none of the objects we create # as part of the callback are needed when the callback # completes. No garbage can accumulate because nothing # can live longer than that callback. That obviates the # need for two things: reference counting against any object # in a parallel context, and garbage collection. Those # things are useful for the main thread, but not parallel # contexts. # # What if you do want to keep something around after # the callback? If it's a simple scalar type, the # following will work: # class Server: # name = None # @route # def set_name(self, request, name): # self.name = name.upper() # ^^^^^^^^^ we intercept that setattr and make # a copy of (the result of) name.upper() # using memory allocation from a different # heap that persists as long as the client # stays connnected. (There's actually # support for alternatively persisting # the entire heap that the object was # allocated from, which we could use if # we were persisting complex, external, # or container types where simply doing # a memcpy() of a *base + size_t wouldn't # be feasible. However, I haven't wired # up this logic to the socket context # logic yet.) # @route # def name(self, request): # return json_serialization(request, self.name) # ^^^^^^^^^ # This will return whatever # was set in the call above. # Once the client disconnects, # the value disappears. # # (Actually I think if you wanted to persist the object # for the lifetime of the server, you could probably # do `request.transport.parent.name = xyz`; or at least, # if that doesn't currently work, the required mechanics # definitely exist, so it would just need to be wired # up.) # # If you want to keep an object around past the lifetime of # the connected client and the server, then send it to the main # thread where it can be tracked like a normal Python object: # # USERS = async.dict() # ^^^^^^^^^^^^ shortcut for: # foo = {} # async.protect(foo) # or just: # foo = async.protect({}) # (On the backend, this instruments[3] the object such # that PyParallel can intercept setattr/setitem and # getattr/getitem calls and "do stuff"[4], depending # on the context.) [3]: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... [4]: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... # # class MyServer(HttpServer): # @route # ^^^^^^ Ignore the mechanics of this, it's just a helper # decorator I used to translate a HTTP GET for # /login/foo to a function call of `login(name='foo')`. # (see the bowls of async.http.server for details). # def login(self, request, name): # @call_from_main_thread # def _save_name(n): # USERS[n] = async.rdtsc() # return len(USERS) # count = _save_name(name) # return json_serialization(request, {'count': count}) # # The @call_from_main_thread decorator will enqueue a work # item to the main thread, and then wait on the main thread's # response. The main thread executes the callback and notifies # the parallel thread that the call has been completed and the # return value (in this case the value of `len(USERS)`). The # parallel thread resumes and finishes the client request. # Note that this will implicitly serialize execution; any number # of parallel requests can submit main thread work, but the # main thread can only call them one at a time. So, you'd # usually try and avoid this, or at least remove it from your # application's hot code path. connect_string = None all_users_sql = 'select * from user' one_user_sql = 'select * from user where login = ?' secret_key = None @route def wiki(self, request, name): # http://localhost/wiki/Python: name = Python if name not in TITLES: self.error(request, 404) # log(n) lookup against a trie with 29 million keys. offset = TITLES[name][0] # log(n) binary search against a numpy array with 15 # million int64s. ix = OFFSETS.searchsorted(offset, side='right') # OFFSETS[ix] = what's the offset after this? (start, end) = (ix-7, OFFSETS[ix]-11) # -7, +11 = adjust for the fact that all of the offsets # were calculated against the '<' of '<title>Foo</title>'. range_request = '%d-%d' % (start, end) request.range = RangedRequest(range_request) request.response.content_type = 'text/xml; charset=utf-8' return self.sendfile(request, XML) @route def users(self, request): # ODBC driver managers that implement connection pooling # behind the scenes play very nicely with our # pyodbc.connect() call here, returning a connection # from the pool (when able) without blocking. con = pyodbc.connect(self.connect_string) # The next three odbc calls would all block (in the # traditional sense), so this current thread would # not be able to serve any other requests whilst # waiting for completion -- however, this is far # less of a problem for PyParallel than single-threaded # land as other threads will keep servicing requests # in the mean time. (ODBC 3.8/SQL Server 2012/Windows 8 # did introduce async notification, such that we could # request that an event be set when the cursor/query/call # has completed, which we'd tie in to PyParallel by # submitting a threadpool wait (much like we do for async # DNS lookup[5], also added in Windows 8), however, it was # going to require a bit of modification to the pyodbc # module to support the async calling style, so, all the # calls stay synchronous for now.) [5]: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... cur = con.cursor() cur.execute(self.all_users_sql) return json_serialization(request, cur.fetchall()) @route def user(self, request, login): con = pyodbc.connect(self.connect_string) cur = con.cursor() cur.execute(self.one_user_sql, (login,)) return json_serialization(request, cur.fetchall()) @route def set_secret_key(self, request, key): # http://localhost/set_secret_key/foobar # An example of persisting a scalar for the lifetime # of the thread (that is, until it disconects or EOFs). try: self.secret_key = [ key, ] except ValueError: # This would be hit, because we've got guards in place # to assess the "clonability" of an object at this # point[6]. (Ok, after reviewing the code, we don't, # but at least we'd crash.) [6]: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... # However, this would work fine, essentially memcpy'ing # the key object at the time of assignment using a different # heap to the one that automatically gets reset at the end # of the callback. self.secret_key = key @route def secret_key(self, request): # http://localhost/secret_key -> 'foobar' return json_serialization(request, {'key': self.secret_key}) @route def stats(self, request): # Handy little json representation of various system stats; # active parallel contexts, I/O hogs, memory load, etc. stats = { 'system': dict(sys_stats()), 'server': dict(socket_stats(request.transport.parent)), 'memory': dict(memory_stats()), 'contexts': dict(context_stats()), 'elapsed': request.transport.elapsed(), 'thread': async.thread_seq_id(), } return json_serialization(request, stats) @route def debug(self, request): # Don't call print() or any of the sys.std(err|out) # methods in a parallel context. If you want to do some # poor man's debugging with print statements in lieu of not # being able to attach a pdb debugger (tracing is disabled # in parallel threads), then use async.debug(). (On # Windows, this writes the message to the debug stream, # which you'd monitor via dbgview or VS.) async.debug("received request: %s" % request.data) # Avoid repr() at the moment in parallel threads; it uses # PyThreadState_SetDictItem() to control recursion depths, # which I haven't made safe to call from a parallel context. # If you want to attach Visual Studio debugger at this point # though, you can do so via: async.debugbreak() # (That literally just generates an INT 3.) @route def shutdown(self, request): # Handy helper for server shutdown (stop listening on the # bound IP:PORT, wait for all running client callbacks to # complete, then return. Totally almost works at the # moment[7].) [7]: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... request.transport.shutdown() def main(): server = async.server('0.0.0.0', port) protocol = HttpServer protocol.connect_string = 'Driver={SQL Server}...' async.register(transport=server, protocol=protocol) ^^^^^^^^^^^^^^ this will create a special 'server' instance of the protocol, which will issue the bind() call. It then creates a configurable number (currently ncpu * 2) of parallel contexts and triggers parallel AcceptEx() invocation (you can prime "pre-accepted" sockets on Windows, which removes the serialization limits of accept() on POSIX). # If an exception occurs in a parallel thread, it is queued # to a special list the main thread has. The main thread # checks this list each time async.run_once() is called, so, # we call it here just to propagate any exceptions that # may have already occurred (like attempting to bind to an # invalid IP, or submitting a protocol that had an error). async.run_once() return server # (This also facilitates interactive console usage whilst # serving request in parallel.) if __name__ == '__main__': main() # Run forever. Returns when there are no active contexts # or ctrl-c is pressed. async.run() All of that works *today* with PyParallel. The main thread preps everything, does the importing, loads the huge data structures, establishes all the code objects and then, once async.run() is called, sits there dormant waiting for feedback from the parallel threads. It's not perfect; I haven't focused on clean shutdown yet, so you will 100% crash if you ctrl-C it currently. That's mainly an issue with interpreter finalization destroying the GIL, which clears our Py_MainThreadId, which makes all the instrumented macros like Py_INCREF/Py_DECREF think they're in a parallel context when they're not, which... well, you can probably guess what happens after that if you've got 8 threads still running at the time pointer dereferencing things that aren't what they think they are. None of the problems are showstoppers though, it's just a matter of prioritization and engineering effort. My strategic priorities to date have been: a) no changes to semantics of CPython API b) high performance c) real-world examples Now, given that this has been something I've mostly worked on in my own time, my tactical priority each development session (often started after an 8 hour work day where I'm operating at reduced brain power) is simply: a) forward progress at any cost The quickest hack I can think of that'll address the immediate problem is the one that gets implemented. That hack will last until it stops working, at which point, the quickest hack I can think of to replace it wins, and so on. At no time do I consider the maintainability, quality or portability of the hack -- as long as it moves the overall needle forward, perfect; it can be made elegant later. I think it's important to mention that, because if you're reviewing the source code, it helps explain things like how I implemented the persistence of an object within a client session (e.g. intercepting the setattr/setitem and doing the alternate heap memcpy dance alluded to above): https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... Without that bit of code, you'll leak memory, with it, you won't. I attacked pyodbc a few weeks ago -- it was also leaking memory when called from parallel callbacks because tp_dealloc wasn't being called on any of the Connection, Cursor or Row objects, so handles that were allocated (i.e. SQLAllocHandle()) were never paired with a SQLFreeHandle() (because we don't refcount in a parallel context, which means there's never a Py_DECREF that hits 0, which means Py_Dealloc() never gets called for that object (which works fine for everything that allocates via PyObject/PyMem facilities, because we intercept those and roll them back in bulk)), and thus, leak. Quickest fix I could think of at the time: async.register_dealloc(pyodbc.Connection) async.register_dealloc(pyodbc.Cursor) async.register_dealloc(pyodbc.Row) Which facilitates this during our interception of PyObject_NEW/INIT: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... Which allows us to do this for each heap... https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... ....that we encounter as part of "socket rewinding": https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33... Absolutely horrendous hack from a software engineering perspective, but is surprisingly effective at solving the problem. Regards, Trent.

On Wed, Jun 24, 2015 at 05:26:59PM +0200, Sturla Molden wrote:

On 24/06/15 07:01, Eric Snow wrote:

...

In return, my question is, what is the level of effort to get fork+IPC to do what we want vs. subinterpreters? Note that we need to accommodate Windows as more than an afterthought

Windows is really the problem. The absence of fork() is especially hurtful for an interpreted language like Python, in my opinion.

UNIX is really the problem. The absence of tiered interrupt request levels, memory descriptor lists, I/O request packets (Irps), thread agnostic I/O, non-paged kernel memory, non-overcommitted memory management, universal page/buffer cache, better device driver architecture and most importantly, a kernel architected around waitable events, not processes, is harmful for efficiently solving contemporary optimally with modern hardware. VMS got it right from day one. UNIX did not. :-) Trent.

Nathaniel Smith wrote:

Continuing my vague and uninformed impressions, I suspect that this would actually be relatively easy to fix by hooking the import system to do something more intelligent, like nominate one node as the leader and have it do the file lookups and then tell everyone else what it found (via the existing message-passing systems).

There are two known solutions. One is basically what you describe. The other, which at least works on IBM blue brain, is to import modules from a ramdisk. It seems to be sufficient to make sure whatever is serving the shared disk can deal with the 100k client DDoS. Sturla

On 25.06.2015 11:35, Sturla Molden wrote:

Nathaniel Smith wrote:

...

Continuing my vague and uninformed impressions, I suspect that this would actually be relatively easy to fix by hooking the import system to do something more intelligent, like nominate one node as the leader and have it do the file lookups and then tell everyone else what it found (via the existing message-passing systems).

There are two known solutions. One is basically what you describe. The other, which at least works on IBM blue brain, is to import modules from a ramdisk. It seems to be sufficient to make sure whatever is serving the shared disk can deal with the 100k client DDoS.

Another way to solve this problem may be to use our eGenix PyRun which embeds modules right in the binary. As a result, all reading is done from the mmap'ed binary and automatically shared between processes by the OS: http://www.egenix.com/products/python/PyRun/ I don't know whether this actually works on an IBM Blue Brain with 100k clients - we are not fortunate enough to have access to one of those machines :-) Note: Even though the data reading is shared, the resulting code and modules objects are, of course, not shared, so you still have the overhead of using up memory for this, unless you init your process cluster using fork() after you've imported all necessary modules (then you benefit from the copy-on-write provided by the OS - code objects usually don't change after they have been created). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 25 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2015-06-25: Released mxODBC 3.3.3 ... http://egenix.com/go79 2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015-07-20: EuroPython 2015, Bilbao, Spain ... 25 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 25 June 2015 at 02:28, Sturla Molden wrote:

On 24/06/15 07:01, Eric Snow wrote:

...

Well, perception is 9/10ths of the law. :) If the multi-core problem is already solved in Python then why does it fail in the court of public opinion. The perception that Python lacks a good multi-core story is real, leads organizations away from Python, and will not improve without concrete changes.

I think it is a combination of FUD and the lack of fork() on Windows. There is a lot of utterly wrong information about CPython and its GIL.

The reality is that Python is used on even the largest supercomputers. The scalability problem that is seen on those systems is not the GIL, but the module import. If we have 1000 CPython processes importing modules like NumPy simultaneously, they will do a "denial of service attack" on the file system. This happens when the module importer generates a huge number of failed open() calls while trying to locate the module files.

Slight tangent, but folks hitting this issue on 2.7 may want to investigate Eric's importlib2: https://pypi.python.org/pypi/importlib2 It switches from stat-based searching for files to the Python 3.3+ model of directory listing based searches, which can (anecdotally) lead to a couple of orders of magnitude of improvement in startup for code loading modules from NFS mounts.

And while CPython is being used for massive parallel computing to e.g. model the global climate system, there is this FUD that CPython does not even scale up on a laptop with a single multicore CPU. I don't know where it is coming from, but it is more FUD than truth.

Like a lot of things in the vast sprawling Python ecosystem, I think there are aspects of this that are a discoverabiilty problem moreso than a capability problem. When you're first experimenting with parallel execution, a lot of the time folks start with computational problems like executing multiple factorials at once. That's trivial to do across multiple cores even with a threading model like JavaScript's worker threads, but can't be done in CPython without reaching for the multiprocessing module. This is the one place where I'll concede that folks learning to program on Windows or the JVM and hence getting the idea that "creating threads is fast, creating processes is slow" causes problems: folks playing this kind of thing are far more likely to go "import threading" than they are "import multiprocessing" (and likewise for the ThreadPoolExecutor vs the ProcessPoolExecutor if using concurrent.futures), and their reaction when it doesn't work is far more likely to be "Python can't do this" than it is "I need to do this differently in Python from the way I do it in C/C++/Java/JavaScript".

The main answers to FUD about the GIL and Python in scientific computing are these:

It generally isn't scientific programmers I personally hit problems with (although we have to allow for the fact many of the scientists I know I met *because* they're Pythonistas). For that use case, there's not only HPC to point to, but a number of papers that talking about Cython and Numba in the same breath as C, C++ and FORTRAN, which is pretty spectacular company to be in when it comes to numerical computation. Being the fourth language Nvidia supported directly for CUDA doesn't hurt either. Instead, the folks that I think have a more valid complaint are the games developers, and the folks trying to use games development as an educational tool. They're not doing array based programming the way numeric programmers are (so the speed of the NumPy stack isn't any help), and they're operating on shared game state and frequently chattering back and forth between threads of control, so high overhead message passing poses a major performance problem. That does suggest to me a possible "archetypal problem" for the work Eric is looking to do here: a 2D canvas with multiple interacting circles bouncing around. We'd like each circle to have its own computational thread, but still be able to deal with the collision physics when they run into each other. We'll assume it's a teaching exercise, so "tell the GPU to do it" *isn't* the right answer (although it might be an interesting entrant in a zoo of solutions). Key performance metric: frames per second Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25/06/15 16:08, Nick Coghlan wrote:

It generally isn't scientific programmers I personally hit problems with (although we have to allow for the fact many of the scientists I know I met *because* they're Pythonistas). For that use case, there's not only HPC to point to, but a number of papers that talking about Cython and Numba in the same breath as C, C++ and FORTRAN, which is pretty spectacular company to be in when it comes to numerical computation.

Cython can sometimes give the same performance as C or Fortran, but as soon as you start to use classes in the Cython code you run into GIL issues. It is not that the GIL is a problem per se, but because Cython compiles to C, the GIL is not released until the Cython function returns. That is, unless you manually release it inside Cython. This e.g. means that the interpreter might be locked for longer durations, and if you have a GUI it becomes unresponsive. The GIL is more painful in Cython than in Python. Personally I often end up writing a mix of Cython and C or C++. Numba is impressive but still a bit immature. It is an LLVM based JIT compiler for CPython that for simple computational tasks can give performance similar to C. It can also run Python code on Nvidia GPUs. Numba is becoming what the dead swallow should have been.

Instead, the folks that I think have a more valid complaint are the games developers, and the folks trying to use games development as an educational tool.

I have not developed games myself, but for computer graphics with OpenGL there is certainly no reason to complain. NumPy arrays are great for storing vertex and texture data. OpenGL with NumPy is just as fast as OpenGL with C arrays. GLSL shaders are just plain text, Python is great for that. Cython and Numba are both great if you call glVertex* functions the old way, doing this as fast as C. Display lists are also equally fast from Python and C. But if you start to call glVertex* multiple times from a Python loop, then you're screwed.

That does suggest to me a possible "archetypal problem" for the work Eric is looking to do here: a 2D canvas with multiple interacting circles bouncing around. We'd like each circle to have its own computational thread, but still be able to deal with the collision physics when they run into each other.

There are people doing Monte Carlo simulations with thousands or millions of particles, but not with one thread per particle. :-) Sturla

Eric Snow schrieb am 24.06.2015 um 06:15:

On Sun, Jun 21, 2015 at 4:40 AM, Stefan Behnel wrote:

...

If objects can make it explicit that they support sharing (and preferably are allowed to implement the exact details themselves), I'm sure we'll find ways to share NumPy arrays across subinterpreters. That feature alone tends to be a quick way to make a lot of people happy.

Are you thinking of something along the lines of a dunder method (e.g. __reduce__)?

Sure. Should not be the first problem to tackle here, but dunder methods would be the obvious way to interact with whatever "share/move/copy between subinterpreters" protocol there will be. Stefan

On 06/25/2015 08:25 AM, Sturla Molden wrote:

On 25/06/15 16:31, Nick Coghlan wrote:

...

3. The potential for collisions between objects means it isn't an embarrassingly parallel problem where the different computational threads can entirely ignore the existence of the other threads

Well, you can have a loop that updates all particles, e.g. by calling a coroutine associated with each particle, and then this loop is an embarrassingly parallel problem. You don't need to associate each particle with its own thread.

It is bad to teach students to use one thread per particle anyway. Suddenly they write a system that have thousands of threads.

Speaking as a novice to this area, I do understand that what we learn with may not be (and usually isn't) production-ready code, I do see Nick's suggestion as being one that is easy to understand, easy to measure, and good for piquing interest. At least, I'm now interested. :) (look ma! bowling for circles!) -- ~Ethan~

On 26 Jun 2015 05:37, "Trent Nelson" wrote:

On Wed, Jun 24, 2015 at 05:26:59PM +0200, Sturla Molden wrote:

...

On 24/06/15 07:01, Eric Snow wrote:

...

In return, my question is, what is the level of effort to get fork+IPC to do what we want vs. subinterpreters? Note that we need to accommodate Windows as more than an afterthought

Windows is really the problem. The absence of fork() is especially

hurtful

...

for an interpreted language like Python, in my opinion.

UNIX is really the problem. The absence of tiered interrupt request levels, memory descriptor lists, I/O request packets (Irps), thread agnostic I/O, non-paged kernel memory, non-overcommitted memory management, universal page/buffer cache, better device driver architecture and most importantly, a kernel architected around waitable events, not processes, is harmful for efficiently solving contemporary optimally with modern hardware.

Platforms are what they are :) As a cross-platform, but still platform dependent, language runtime, we're actually in a pretty good position to help foster some productive competition between Windows and the *nix platforms. However, we'll only be able to achieve that if we approach their wildly divergent execution and development models with respect for their demonstrated success and seek to learn from their respective strengths, rather than dismissing them over their respective weaknesses :) Cheers, Nick.

Eric Snow wrote:

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them.

This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources.

I've heard very little about this since the original announcement (https://lwn.net/Articles/650521/), so I was wondering if this is still an active idea being worked on? Or has it turned out to be too difficult? Sebastian