On Aug 3, 2015, at 10:11, Sven R. Kunze wrote:

...

On 02.08.2015 02:02, Andrew Barnert wrote: Your idea of having a single global "pool manager" object, where you could submit tasks and, depending on how they're marked, they get handled differently might have merit. But that's something you could build pretty easily on top of concurrent.futures (at least for threads vs. processes; you can add in coroutines later, because they're not quite as easy to integrate), upload to PyPI,

You mean something like this?

https://pypi.python.org/pypi/xfork

Did you just write this today? Then yes, that proves my point about how easy it is to write it. Now you just have to get people using it, get some experience with it, etc. and you can come back with a proposal to put something like this in the stdlib, add syntactic support, etc. that it will be hard for anyone to disagree with. (Or to discover that it has flaws that need to be fixed, or fundamental flaws that can't be fixed, before making the proposal.) One quick comment: from my experience (mostly with other languages that are very different from Python, so I can't promise how well it applies here...), implicit futures without implicit laziness or even an explicit delay mechanism are not as useful as they look at first glance. Code that forks off 8 Fibonacci calls, but waits for each one's result before forking off the next one, might as well have just stayed sequential. And if you're going to use the result by forking off another job, then it's actually more convenient to use explicit futures like the ones in the stdlib. One slightly bigger idea: If you really want to pursue your implicit-as-possible design further, you might want to consider making the decorators replace the function with an object whose __call__ method just implicitly submits it to the pool. Then you can use normal function-calling syntax and pretend everything is magic. You can even add operator dunder methods to your future class that do the same thing (so "result * 2" just builds a new future out of "self.get() * 2", either submitted to the pool, probably better, tacked on as an add_done_callback). I think there's a limit to how far you can push this without some mechanism to mark when you need to actual value (in ML-derived languages and C++, static types make this easier: a cast, implicit or explicit, forces a wait; in Python, that doesn't work), but it might be worth exploring that limit. Or it might be better to just stop at the magic function calls and leave the futures alone.

On Aug 4, 2015, at 11:09, Sven R. Kunze wrote:

...

On 04.08.2015 05:21, Andrew Barnert wrote:

...

On Aug 3, 2015, at 10:11, Sven R. Kunze wrote:

...

On 02.08.2015 02:02, Andrew Barnert wrote: Your idea of having a single global "pool manager" object, where you could submit tasks and, depending on how they're marked, they get handled differently might have merit. But that's something you could build pretty easily on top of concurrent.futures (at least for threads vs. processes; you can add in coroutines later, because they're not quite as easy to integrate), upload to PyPI,

You mean something like this?

https://pypi.python.org/pypi/xfork

Did you just write this today? Then yes, that proves my point about how easy it is to write it. Now you just have to get people using it, get some experience with it, etc. and you can come back with a proposal to put something like this in the stdlib, add syntactic support, etc. that it will be hard for anyone to disagree with. (Or to discover that it has flaws that need to be fixed, or fundamental flaws that can't be fixed, before making the proposal.)

I presented it today. The team members already showed interest. They also noted they like its simplicity. The missing syntax support seemed like minor issue compared to what complexity is hidden.

Others admitted they knew about the existence of concurrent.futures and such but never used it due to - its complexity - AND *drum roll* the '.result()' of the future objects As it seems, it doesn't feel natural.

I don't know how to put this nicely, but I think anyone who finds the complexity of concurrent.futures too daunting to even attempt to learn it should not be working on any code that uses less explicit concurrency. I have taught concurrent.futures to rank novices in a brief personal session or a single StackOverflow answer and they responded, "Wow, I didn't realize it could be this simple". Someone who can't grasp it is almost certain to be someone who introduces races all over your code and can't even understand the problem, much less debug it.

...

One quick comment: from my experience (mostly with other languages that are very different from Python, so I can't promise how well it applies here...), implicit futures without implicit laziness or even an explicit delay mechanism are not as useful as they look at first glance. Code that forks off 8 Fibonacci calls, but waits for each one's result before forking off the next one, might as well have just stayed sequential. And if you're going to use the result by forking off another job, then it's actually more convenient to use explicit futures like the ones in the stdlib.

One slightly bigger idea: If you really want to pursue your implicit-as-possible design further, you might want to consider making the decorators replace the function with an object whose __call__ method just implicitly submits it to the pool.

I added two new decorators for this. But they don't work with the @ syntax. It seems like a well-known issue of Python:

_pickle.PicklingError: Can't pickle : it's not the same object as __main__.fib_fork

Would be great if somebody could fix that.

...

Then you can use normal function-calling syntax and pretend everything is magic. You can even add operator dunder methods to your future class that do the same thing (so "result * 2" just builds a new future out of "self.get() * 2", either submitted to the pool, probably better, tacked on as an add_done_callback). I think there's a limit to how far you can push this without some mechanism to mark when you need to actual value (in ML-derived languages and C++, static types make this easier: a cast, implicit or explicit, forces a wait; in Python, that doesn't work), but it might be worth exploring that limit. Or it might be better to just stop at the magic function calls and leave the futures alone.

I actually like the idea of contagious futures and I might outline why this is not an issue with the current Python language.

Have a look at the following small interactive Python session:

...

...

...

3+4 7 _ 7 a=9 _ 7 a+=10 _ 7 a 19 _ 19

Question: When has the add operation being executed?

Answer: Unknown from the programmer's perspective.

Not true. The language clearly defines when each step happens. The a.__add__ method is called, then the result is assigned to a, then the statement finishes. (Then, in the next statement, nothing happens--except, because this is happening in the interactive interpreter, and it's an expression statement, after the statement finishes doing nothing, the value of the expression is assigned to _ and its repr is printed out.) This ordering relationship may be very important if the variable a is shared by multiple threads, especially if more than one thread may modify it, especially if you're using non-atomic operations like += (where another thread can read, use, and assign the variable between the __add__ call and the assignment). If a references a mutable object with an __iadd__ method, the variable doesn't even need to be shared, only the value, for this to matter. The only way to safely ignore these problems is to never share any variables or any mutable values between threads. (This is why concurrency features are easier to design in pure functional languages.) Hiding this fact when you or the people you're hiding it from don't even understand the issue is exactly how you create races.

Only requirement: Exceptions are raised exactly where the operation is supposed to take place in the source code (even if the operation that raises the exception is performed later).

On Aug 4, 2015, at 14:03, Sven R. Kunze wrote:

...

On 04.08.2015 21:38, Andrew Barnert wrote: I think anyone who finds the complexity of concurrent.futures too daunting to even attempt to learn it should not be working on any code that uses less explicit concurrency.

I am sorry because I disagree here with you.

...

I have taught concurrent.futures to rank novices in a brief personal session or a single StackOverflow answer and they responded, "Wow, I didn't realize it could be this simple".

Nobody says that concurrent.futures is not an vast improvement over previous approaches. But it is still not the end of the line of simplifications.

...

Someone who can't grasp it is almost certain to be someone who introduces races all over your code and can't even understand the problem, much less debug it.

Nobody wants races, yet everybody still talks about them. Don't allow races in the first place and be done with it.

What does that even mean? How would you not allow races? If you let people throw arbitrary tasks at a thread pool, with no restriction on mutable shared state, you've allowed races.

...

Not true. The language clearly defines when each step happens. The a.__add__ method is called, then the result is assigned to a, then the statement finishes. (Then, in the next statement, nothing happens--except, because this is happening in the interactive interpreter, and it's an expression statement, after the statement finishes doing nothing, the value of the expression is assigned to _ and its repr is printed out.)

Where can find this definition in the docs?

To me, we are talking about class customization as described on reference/datamodel.html. Seems like an implementation detail, not a language detail.

No, the data model is a feature of the language, not one specific implementation. The fact that you can define classes that work the same way as builtin types like int is a fundamental feature. It's something Guido and others worked very hard on making true back in Python 2.2-2.3. It's one of the things that makes Python or C++ more pleasant to use than Tcl or Java. Any implementation that didn't do the same would not be Python, and would not run a good deal of Python code.

I am not saying, CPython doesn't do it like that, but I saying the Python language could support lazy evaluation and not disagreeing with the docs.

...

This ordering relationship may be very important if the variable a is shared by multiple threads, especially if more than one thread may modify it, especially if you're using non-atomic operations like += (where another thread can read, use, and assign the variable between the __add__ call and the assignment). If a references a mutable object with an __iadd__ method, the variable doesn't even need to be shared, only the value, for this to matter. The only way to safely ignore these problems is to never share any variables or any mutable values between threads.

Mutual variables are global variables. And these have gone out of style quite some time ago.

No. Shared values include global variables, nonlocal variables used by two closures from the same scope, attributes of objects passed to both functions, members of collections passed to both functions, etc. The existence of all of these other things is why global variables are not necessary. They have many advantage over globals, allowing you to better control how state is shared, to share it reentrantly, to make it more explicit in the code, etc. But because they have all the same benefits, they also have the exact same race problem when used to share state between threads.

Btw. this is races again and I thought we agreed on not having them because nobody really can/wants to debug them.

And how do you propose "not having them"? It's not impossible to write purely functional code that doesn't use any mutable state, in which case it doesn't matter whether your state is shared. But the fact that your example uses += proves that this isn't your intention. If you take the code from your example and run it in two threads simultaneously, you have a race. The fact that you didn't intend to create a race because you don't understand that doesn't mean the problem isn't there, it just means you have no idea you've just written buggy code and no idea how to test for it or debug it. And that's exactly the problem. What makes concurrent code with shared state hard, more than anything else, is people who don't realize what's hard about it and write code that seems to work but doesn't. Making it easier for such people to write broken code without even realizing they're doing so is not a good thing.

On Aug 11, 2015, at 06:54, Sven R. Kunze wrote:

Co-workers proposed using function scopes as the ultimate evaluation scope. That is when a function returns a ResultProxy, it gets evaluated. However, I have absolutely no idea how to do this as I couldn't find any __returned__ hook or something.

I'm not sure I completely understand what you're looking for here. If you just want a hook that gets called whenever a function returns, just write a decorator that calls the real function then does the hook thing: def hookify(func): @wraps def wrapper(*args, **kwargs): result = func(*args, **kwargs) do_hook_stuff() return result return wrapper (Or, if you want to hook both raising and returning, use a finally.) But I'm not sure what good that would do anyway. If you unwrap futures every time they're returned, they're not doing anything useful as futures in the first place; you might as well just return the values directly.

Am 05-Aug-2015 16:30:27 +0200 schrieb abarnert@yahoo.com:

What does that even mean? How would you not allow races? If you let people throw arbitrary tasks at a thread pool, with no restriction on mutable shared state, you've allowed races.

Let me answer this in a more implicit way. Why do we need to mark global variables as such? I think the answer is clear: to mark side-effects (quoting the docs). Why are all variables thread-shared by default? I don't know, maybe efficiency reasons but that hardly apply to Python in the first place.

And how do you propose "not having them"?

What would happen if all shared variables were thread-local by default and need to marked as shared if desired? I think the answer would also be very clear: to mark side-effects and to have people think about it explicitly.

And that's exactly the problem. What makes concurrent code with shared state hard, more than anything else, is people who don't realize what's hard about it and write code that seems to work but doesn't.

Precisely because 'shared state' is hard, why is it the default?

Making it easier for such people to write broken code without even realizing they're doing so is not a good thing.

That argument only applies when the broken code (using shared states) is the default. As you can see, this thought experiment assumes that there could be another way to approach that situation. How and when this can be done and if at all is a completely different matter. As usual, I leave that to the experts like you to figure out. Best, Sven ------------------------------------------------------------------------------------------------- FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSITÄT UND KOMFORT

On 12.08.2015 05:33, Andrew Barnert wrote:

On Aug 11, 2015, at 07:33, Sven R. Kunze wrote:

...

Am 05-Aug-2015 16:30:27 +0200 schrieb abarnert@yahoo.com:

...

What does that even mean? How would you not allow races? If you let people throw arbitrary tasks at a thread pool, with no restriction on mutable shared state, you've allowed races. Let me answer this in a more implicit way.

Why do we need to mark global variables as such? I think the answer is clear: to mark side-effects (quoting the docs).

Why are all variables thread-shared by default? I don't know, maybe efficiency reasons but that hardly apply to Python in the first place. First, are you suggesting that your idea doesn't make sense unless Python is first modified to not have shared variables? In that case, it doesn't seem like a very useful proposal, because it applies to some different language that isn't Python. And applying it to Python instead means you're still inviting race conditions. Pointing out that in a different language those races wouldn't exist is not really an answer to that. I don't understand. In order to propose a change, one needs to imagine how things could be if there were different. I don't see how this relates to 'if you change Python, it wouldn't be Python anymore'. You certainly agree that Python 2.7 is not Python 3.5 and Python 3.7 will also be different (if existent at all). So, what's your point here? Second, the reason for the design is that that's what threads mean, by definition: things that are like processes except that they share the same heap and other global state. Where do you get this kind of definition from? What's the point of a proposal that lets people select between threads and processes if its threads aren't actually processes? Where did I say that?

My point was: 1) processes are fine (more or less) 2) threads aren't because their are hard to manage, so let's make them easier

Finally, just making variables thread-local wouldn't help. You'd need a completely separate heap for each thread; So?

At this point, talking about internal implementation hardly seems to relevant. Not exactly sure, what you mean by heap here, but I could imagine more like a overlay approach. As long as, I only read the original variable, we are fine. But setting it would require me to store the thread-local value somewhere else. I am uncertain why you are so averse about making threading easier to handle and to maintain. If you bother about 'easier', let's call it 'code works more reliably', 'code is more readable', 'code has lesser side-effects', 'code produces lesser races'. I am not talking about 100%. I am talking about 80% less places in your code where you need to worry thread-safety. That makes 20% places where you really need to. Btw. the stdlib would also benefit from this, in order to provide thread-safe modules out of the box. Not every maintainer needs to re-implement the desired thread-safety from scratch over and over again.

otherwise, just passing a list to another thread means it can modify your values.

Just depends on what you want here. I would rather see Python assuming thread-safe behavior by default whereas the programmer can actively chose a more flexible/dangerous model if needed for some small areas.

[...implementation...] what happens when you do x[0]=y if x is local and y shared, or vice-versa?

Now, we are talking. A) As soon as a single variable (x or/and y) is shared, all expressions using/writing such variables are basically unsafe. It's dangerous, you might need some locks and so forth to get it running properly. You might need extra thought to handle some weird corner cases and so forth. B) If all variables of an expression are local, everything is fine. No additional work needed. I regard case B) as the common case where you DON'T want others to mess around with your variables and you don't can do anything about it. Case A) is more like the data communication channel where threads could communicate with each other, aggregate results in a common list, and so forth. I only can image this taking place at the end of the threading part of a program where the results needs to be propagated back to the MainThread.

Of course there's no reason that you couldn't implement something that's basically a process at the abstract level, but implemented with threads at the OS level. And that could make both explicit shared memory and IPC simpler at least under the covers, and more efficient. And it could lead to a way to eliminate the GIL. And there could be other benefits as well. That's why people are exploring things like the recent subinterpreters thread, PyParallel, PyPy+STM, etc.

Yes, transactional memory would basically the term that would cover that. A thread basically gets a snapshot of the world right from the start and after it finishes, the variables get merged back. However, I am unsure whether I would want that for all variables ("shared vs local" exists also here; and I would prefer an explicit way to declare it).

The default is to write sequential code. You have to go out of your way to use threads. And when you do, you have to intentionally choose threads over processes or some kind of microthreads.

It's only when you've chosen to use shared-memory threading as the design for your app that shared memory becomes the default. I am not sure if I can follow here. If I look at the threading API of

We are talking about threading all the way long. There is no point in going back to sequential. the Python standard lib, it is shared-memory. So, it is the default, like it or not.

But that is the default in Python, so your proposal would make it easier for such people to write broken code without even realizing they're doing so, so it's not a good thing.

I am sorry? Because shared-memory is the default in Python, my proposal would make it easier for such people to write broken code? We must be talking about different proposals. Maybe, you could give an example. Just for the record, my proposal: 1) processes are almost fine 2) threads aren't, so let's make it easier to work with them Best, Sven

On 18.08.2015 18:55, Chris Angelico wrote:

The notion of "a completely separate heap for each thread" is talking about part B - you need a completely separate pile of objects.

Right. However, only if needed. As long as, threads only reads from a common variable, there is no need to interfere. I would except the same behavior as with class attributes and instance attributes. The latter overlay the former once they are assigned. Otherwise, the former can be read without an issue.

And if you're going to do that, you may as well make them separate processes.

Consulting the table elaborated on the other thread "Concurrency Modules", that is not entirely true. I agree, behavior-wise, processes behave almost as desired (relevant data is copied over and there are no shared variables). However, regarding the cpu/memore/communication footprint for a new process (using spawn) is enormous compared to a thread. So, threading still have its merits (IMHO).

There's no way to make module globals thread-local without making the module object itself thread-local, and if you do that, you probably need to make every object it references thread-local too, etc, etc, etc.

Something wrong with that? Shouldn't matter as long as there is only a single thread.

Does that answer the questions? Apart from "heap" being perhaps a term of implementation, this isn't about the internals - it's about the script-visible behaviour.

Yep, thanks a lot, Chris. :) Best, Sven

On 18.08.2015 19:27, Chris Angelico wrote:
> On Wed, Aug 19, 2015 at 3:17 AM, Sven R. Kunze  wrote:
>> On 18.08.2015 18:55, Chris Angelico wrote:
>>> The notion of "a completely separate heap for each thread" is talking
>>> about part B - you need a completely separate pile of objects.
>>
>> Right. However, only if needed. As long as, threads only reads from a common
>> variable, there is no need to interfere.
> Sure, but as soon as you change something, you have to
> thread-local-ify it. So I suppose what you would have is three
> separate pools of objects:
>
> 1) Thread-specific objects, which are referenced only from one thread.
> These can be read and written easily and cheaply.
> 2) Global objects which have never been changed in any way since
> threading began. These can be read easily, but if written, must be
> transformed into...
> 3) Thread-local objects, which exist for all threads, but are
> different. The id() of such an object depends on which thread is
> asking.
>
> Conceptually, all three types behave the same way - changes are
> visible only within the thread that made them. But the implementation
> could have these magic "instanced" objects for only those ones which
> have actually been changed, and save a whole lot of memory for the
> others.

Indeed. I think that is sensible approach here. Speaking of an 
implementation though, I don't know where I would start when looking at 
CPython.

Thinking more about id(). Consider a complex object like an instance of 
a class. Is it really necessary to deep copy it? It seems to me that we 
actually just need to hide the atomic/immutable values (e.g. strings, 
integers etc.) of that object. The object itself can remain the same.

# first thread
class X:
     a = 0
class Y:
     x = X

#thread spawned by first thread
Y.x.a = 3  # should leave id(X) and id(Y) alone


Maybe, that example is too simple, but I cannot think of an issue here. 
As long as the current thread is the only one being able to change the 
values of its variables, all is fine.

>> I agree, behavior-wise, processes behave almost as desired (relevant data is
>> copied over and there are no shared variables).
>>
>> However, regarding the cpu/memore/communication footprint for a new process
>> (using spawn) is enormous compared to a thread. So, threading still have its
>> merits (IMHO).
> So really, you're asking for process semantics, with some
> optimizations to take advantage of the fact that most of the processes
> are going to be just reading and not writing. That may well be
> possible, using something like the above three-way-split, but I'd want
> the opinion of someone who's actually implemented something like this
> - from me, it's just "hey, wouldn't this be cool".

If you put it this way, maybe yes. I also look forward to more feedback 
on this.

To me, a process/thread or any other concurrency solution, is basically 
a function that I can call but runs in the background. Later, when I am 
ready, I can collect its result. In the meantime, the main thread 
continues. (Again) to me, that is the only sensible way to approach 
concurrency. When I recall the details of locks, semaphores etc. and 
compare it to what real-world applications really need... You can create 
huge tables of all the possible cases that might happen just in order to 
find out that you missed an important one.

Even worse, as soon as you change something about your program, you are 
doomed to redo the complete case analysis, find a dead/live-lock-free 
solution and so forth. It's a time sink; costly and dangerous from a 
company's point of view.

Best,
Sven

On 18.08.2015 23:53, Paul Moore wrote:

On 18 August 2015 at 21:32, Sven R. Kunze wrote:

...

To me, a process/thread or any other concurrency solution, is basically a function that I can call but runs in the background. Later, when I am ready, I can collect its result. In the meantime, the main thread continues. (Again) to me, that is the only sensible way to approach concurrency. It seems to me that this is accurate, but glosses over all of the issues that result in multiple solutions being needed.

I can identify 2 common patterns I label as jobs and servers. Jobs are things that get delegated out to some background process/thread/coroutine/subinterpreter. They come back when the job is done. No shared state necessary. Servers are more like while true loops running in some separate process/thread/coroutine/subinterpreter. They only return on shutdown or so. Shared state a là queues for input/output could come in handy. Maybe, there is more to discover but it's more like research that production.

Sure, all concurrency solutions provide this. But the difference lies in the environment of the function you're calling. Does it have access to the same non-local name bindings as it would if run in the foreground? To the same objects? Is it able to write to those objects safely, or must it treat them as read only. Or can it write, but only if it follows a particular protocol (semaphores, locks, etc fit here)?

I am uncertain of how to approach this in the correct way. However, my approach here would be simply by imitating sequential behavior. I always would ask: "What would happen if that is executed sequentially?" The answer would then be retrofitted to the parallel scenario.

Conversely, it *is* true that a lot of problems that benefit from concurrency can work with the minimal guarantees of a lowest common denominator solution (no shared state pure functions). Functional programming has shown us that. For those problems, any of the options are fine, and the decision gets made on other factors (most likely performance, as each solution makes different performance trade-offs in the process of providing whatever extra guarantees they make).

I hope eventually, the interpreter will take the decision burden away from the developers and make educated guesses to achieve the best performance.

I'm confused as to what your point is. "People should write concurrent code in a no shared state pure function manner" seems to be what your comment "the only sensible way" implies.

I think you are referring to that statement only. It's just the underlaying motivation for me to engage in this discussion and it's born by observing real-world code development and maintenance. If I were to ask people around the globe of what they use in production, I guess I would get answers like that (that'll be an awesome survey btw.): 30% declaration - you say what you want (cfg files, decorators, sql, css, html, etc.) 20% imperative - you say what to do (function call hierarchies) 25% object oriented - you think in "objects" (object relationship tree) 15% packages/modules - you cluster functionality - (file/directory hierarchies) 10% magic stuff - you never get it right (generators, concurrency, meta-classes, import hooks, AST, etc.) Just look at big projects. Look at different programming languages. All the same. Maybe, my observation is wrong; that could be. Maybe, the observation is true and people are just dumb, lazy and unable to appreciate a fine 10 hours live-lock bug hunting session (but I doubt that). Most professionals, I am working with, are highly intelligent people working on very complex problems and with a very few resources. Thus, they don't appreciate tools making their lives more difficult by changing the problem's domain from complex to complicated. As a result, they are not going to use it at all. Make it like the top 90% of what people are used to and they are going to use it. Point is not, people should do this or that. Point is, the tools should make it stupidly easy to get things done. People then will follow and do this or that automatically. Let's improve the tools. Best, Sven

On 20.08.2015 02:18, Steven D'Aprano wrote:

On Thu, Aug 20, 2015 at 12:01:12AM +0200, Sven R. Kunze wrote:

...

Point is, the tools should make it stupidly easy to get things done. "The problem with this is that we will have done what humans often do, which is to use technology to make things easier while missing an opportunity to make them significantly better." -- Rory Sutherland and Glen Weyl

If Python becomes everything that you want from your proposal, how will it be *better* rather than just easier?

"I am uncertain why you are so averse about making threading easier to handle and to maintain. If you bother about 'easier', let's call it 'code works more reliably', 'code is more readable', 'code has lesser side-effects', 'code produces lesser races'." -- Sven R. Kunze This quoted: what is your definition of *better*? Best, Sven PS: there is no such thing as **better* (at least from my perspective). It all comes down to a personal definition. Watching https://vimeo.com/79539317 , I can tell that there much more potential for improvements under the hoods. However, the public API for the "end developers" should be made and stay as simple as possible. Just for the sake of "getting things done".

On 19.08.2015 04:09, Andrew Barnert wrote:

On Aug 18, 2015, at 13:32, Sven R. Kunze wrote:

...

Indeed. I think that is sensible approach here. Speaking of an implementation though, I don't know where I would start when looking at CPython.

Thinking more about id(). Consider a complex object like an instance of a class. Is it really necessary to deep copy it? It seems to me that we actually just need to hide the atomic/immutable values (e.g. strings, integers etc.) of that object. Why wouldn't hiding the mutable members be just as necessary? In your example, if I can replace Y.x, isn't that even worse than replacing Y.x.a? That was a question to the experts. I don't know.

Not sure what you mean about 'worse'. Replacing Y.x is just pointer to some value. So, if I replace it with something else, it's not different/worse/better than replacing Y.x.a, right?

...

The object itself can remain the same. What does it mean for an object to be "the same" if it potentially holds different values in different threads.

I was talking about the id(...) and deep copying.

...

# first thread class X: a = 0 class Y: x = X

#thread spawned by first thread Y.x.a = 3 # should leave id(X) and id(Y) alone (*) OK, but does the second thread see 0 or 3? If the former, then these aren't shared objects at all. If the latter, then that's how things already work.

before (*) first thread sees: Y.x.a == 0 thread spawned by first thread sees: Y.x.a = 0 after (*) first thread sees: Y.x.a == 0 thread spawned by first thread sees: Y.x.a = 3 If that wasn't clear, we were talking about the preferred 'process-like' semantics. Just assume for a moment, that, by default, Python would wrap up variables (as soon as they are shared across 2 or more threads) like this (self is the variable): class ProxyObject: def __init__(self, variable) self.__original__ = variable self.__threaded__ = threading.local() def __proxy_get__(self): return getattr(self.__threaded__, 'value', self.__original) def __proxy_set__(self, value): self.__threaded__.value = value I think you get the idea; it should work like descriptors. Basically, descriptors for general access on a variable and not for classes only => proxy objects. Is there something like that Python? That would vastly simplify the implementation of xfork, btw. So, to get you some example (still assuming the behavior described above), I abuse our venerable thumbnails. Let's calculate the total sum of the thumbnail bytes created: 1: images = ['0.jpg', '1.jpg', '2.jpg', '3.jpg', '4.jpg'] 2: sizes = [] 3: for image in images: 4: fork create_thumbnails(image, sizes) 5: wait # for all forks to come back 6: shared sizes 7: print('sum:', sum(sizes)) 8: 9: @io_bound 10: def create_thumbnails(image, sizes): 11: with open(image) as image_file: 12: # and so forth 13: shared sizes 14: sizes.append(100) Here, you can see what I meant by explicitly stating that we enter dangerous space: the keyword "shared" in lines 5 and 13. It basically removes the wrapper described above and reveals the dangerous/shared stated of the object (like 'global'). So, both functions needs to agree to remove the veil and thus to be able to read/modify the shared state. shared x translates to: x = x.__original__

Have you looked into the subinterpreters project, the PyParallel project, or the PyPy-STM project, all of which, as I mentioned earlier, are possible ways of getting some of the advantages of process semantics without all of the performance costs? (Although none of them are exactly that, of course.)

Yes, I did. STM is nice as a proof of concept, waiting for HTM. However, as I mentioned it earlier I am not sure whether I would really want that within the semantics of multiple threads. Trent Nelson (PyParallel) seems to agree on this. It's kind of weird and would be followed by all sorts of workarounds in case of a transaction failure. The general intention of PyParallel seems to be interesting. It also is all in about "built-in thread-safety", which is very nice. Trent also agrees on 'never share state'.

...

To me, a process/thread or any other concurrency solution, is basically a function that I can call but runs in the background. Later, when I am ready, I can collect its result. In the meantime, the main thread continues. (Again) to me, that is the only sensible way to approach concurrency. When I recall the details of locks, semaphores etc. and compare it to what real-world applications really need... You can create huge tables of all the possible cases that might happen just in order to find out that you missed an important one. Yes, that is the problem that makes multithreading hard in the first place (except in pure functional languages). If the same value is visible in two threads, and can be changed by either of those threads, you have to start thinking either about lock discipline, or about ordering atomic operations; either way, things get very complicated very fast.

A compromise solution is to allow local mutable objects, but not allow them to be shared between threads; instead, you provide a way to (deep-)copy them between threads, and/or to (destructively) move them between threads. You can do that syntactically, as with the channel operators used by Erlang and the languages it's inspired, or you can do it purely at a semantic level, as with Python's multiprocessing library; the effect is the same: process semantics, or message-passing semantics, or whatever you want to call it gives you the advantages of immutable threading in a language with mutability.

...

Even worse, as soon as you change something about your program, you are doomed to redo the complete case analysis, find a dead/live-lock-free solution and so forth. It's a time sink; costly and dangerous from a company's point of view. This is an argument for companies to share as little mutable state as possible across threads. If you don't have any shared state at all, you don't need locks or other synchronization mechanisms at all. If you only have very limited and specific shared state, you have very limited and hopefully simple locking, which is a lot easier to keep track of.

I am glad, we agree on this. However, just saying it's hard and keeping status quo does not help, I suppose.

On 08/18/2015 01:27 PM, Chris Angelico wrote:

...

...

There's no way to make module globals thread-local without making the

...

...

module object itself thread-local, and if you do that, you probably need to make every object it references thread-local too, etc, etc, etc.

Something wrong with that? Shouldn't matter as long as there is only a single thread.

As long as there's only a single thread, there's no difference between process-wide and thread-local. Once you start a second thread, something needs to know what objects belong where. That's all.

Just a thought... What if a new name type, co_mutables, is added to functions objects, and a new keyword to go with it, "mutable" to be used like "globals" and "nonlocals". And then raise an error on an attempt to bind a mutable object to a name not in co_mutables. Also, don't include co_mutables in functions scopes. That would require mutable objects to be passed as function arguments to be seen. Then only mutable objects/names explicitly passed to a new thread need to be tracked. It might be possible to turn that on with a compiler directive at the top of a module so normal python code would work normally, and thread safe python code would be limited. Would something in this direction simplify the problem? Cheers, Ron

On Aug 19, 2015, at 09:57, Ron Adam wrote:

On 08/18/2015 01:27 PM, Chris Angelico wrote:

...

...

...

There's no way to make module globals thread-local without making the

...

...

module object itself thread-local, and if you do that, you probably need to make every object it references thread-local too, etc, etc, etc.

Something wrong with that? Shouldn't matter as long as there is only a single thread.

...

As long as there's only a single thread, there's no difference between process-wide and thread-local. Once you start a second thread, something needs to know what objects belong where. That's all.

Just a thought...

What if a new name type, co_mutables, is added to functions objects, and a new keyword to go with it, "mutable" to be used like "globals" and "nonlocals". And then raise an error on an attempt to bind a mutable object to a name not in co_mutables.

Also, don't include co_mutables in functions scopes. That would require mutable objects to be passed as function arguments to be seen.

Then only mutable objects/names explicitly passed to a new thread need to be tracked.

It might be possible to turn that on with a compiler directive at the top of a module so normal python code would work normally, and thread safe python code would be limited.

Would something in this direction simplify the problem?

Well, the problem can be solved with a strong enough static typing system, as multiple ML-derived languages that add mutability and/or unsafety prove. But what you're suggesting doesn't nearly strong enough, or static enough, to accomplish that. First, in this code, is i mutable or not: def spam(a): for i in a: eggs(i) And if eggs is imported from a module not marked "thread-safe", does is the call illegal, or assumed to do something that mutates i, or assumed to be safe? Also, whether eggs is from a "thread-safe" module or a normal one, how does the compiler know whether it's passing i to a new thread? And what happens if eggs stores i in a list or other mutable object and some other code mutates it later? Finally, if you only track mutability at the top level, how can you tell the compiler that a (mutable) queue of ints is thread-safe, but a queue of lists is not? And how can the compiler know which one it's looking at without doing a complete whole-program type inference?

On 20.08.2015 04:20, Ron Adam wrote:

In this case 'i' is immutable as it's not marked as 'mutable'. That determines what byte code is used, (or how the byte code that is used acts).

...

...

...

dis(spam) 2 0 SETUP_LOOP 24 (to 27) 3 LOAD_FAST 0 (a) 6 GET_ITER >> 7 FOR_ITER 16 (to 26) 10 STORE_FAST 1 (i)

3 13 LOAD_GLOBAL 0 (eggs) 16 LOAD_FAST 1 (i) 19 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 22 POP_TOP 23 JUMP_ABSOLUTE 7 >> 26 POP_BLOCK >> 27 LOAD_CONST 0 (None) 30 RETURN_VALUE

If a compiler directive/flag for threading was set, then the interpreter/compiler may use different LOAD_FAST and GET_ITER routines that will raise an exception if "i" or "a" is mutable. (by checking a flag on the object, or other means.)

If however...

def eggs(i): mutable i # mutable objects clearly marked. i.scramble()

def spam(a): mutable a, i for i in a: eggs(i) # eggs can't access 'a' or 'i' in it's scope. # so i needs to be passed.

I had absolutely not idea what you mean when saying: "co_mutables". Reading the byte code and the examples above though, I just can agree. Thanks a lot for this. Yesterday, I used "shared" in my posts to illustrate that. "mutable" is another word for it. Best, Sven

On Thu, Aug 20, 2015 at 12:27:55PM -0400, Ron Adam wrote:

When a bytecode to load an object is executed such as LOAD_FAST, it gets it's reference to the object from the function's list of names in it's code object.

Bytes codes are implementation, not semantics: there is no part of the Python language that promises that retrieving local variables will use a byte code LOAD_FAST. That's not how IronPython or Jython work, and there is no stability guarantee that CPython will always use LOAD_FAST.

LOAD_FAST 0, reads __code__.co_varnames[0] LOAD_FAST 1, reads __code__.co_varnames[1]

Adding a co_mutables name list to the __code__ attribute, along with new bytecodes to access them, it would create a way to keep private local names without changing how the other bytecodes work.

LOAD_MUTABLE 0, would get the first reference in __code__.co_mutables.

What difference would it make to have a different implemention for retrieving and setting the object? How will having this byte code "keep private local names"? -- Steve

On Aug 20, 2015, at 13:02, Ron Adam wrote:

...

On 08/20/2015 12:51 PM, Steven D'Aprano wrote:

...

On Thu, Aug 20, 2015 at 12:27:55PM -0400, Ron Adam wrote:

...

When a bytecode to load an object is executed such as LOAD_FAST, it gets it's reference to the object from the function's list of names in it's code object. Bytes codes are implementation, not semantics: there is no part of the Python language that promises that retrieving local variables will use a byte code LOAD_FAST. That's not how IronPython or Jython work, and there is no stability guarantee that CPython will always use LOAD_FAST.

Yes, but semantics needs a workable implementation at some point.

The semantics is to have a way to make names (for mutable objects) in outer scopes not be visible to function defined in inner scopes.

def foo(x): """ Example of hiding a mutable object from inner scopes. """ mutable items items = [1, 2, 3] def bar(y): """ can't see items here. So can't mutate it.""" return -y return [bar(y)+x for y in items]

So foo is able to protect items from being mutated by functions defined in it's scope. We could use localonly instead of mutable, but in the context of threading mutable may be more appropriate. (Way too soon to decide what color to paint this bike.)

It may seem like it isn't needed, because you have control over what a function has access too... ie... just don't do that. But when you have many programmers working on large projects, things can get messy. And this helps with that, but also helps in the case of threads.

The only case you're helping with here is the case where the race is entirely local to one function and the functions it defines--a relatively uncommon case that also gets the least messy and is the easiest to spot and debug. Also, the "dangerous" cases are already marked today: the local function has to explicitly declare the variable nonlocal or it can't assign to it.

...

...

...

LOAD_FAST 0, reads __code__.co_varnames[0] LOAD_FAST 1, reads __code__.co_varnames[1]

Adding a co_mutables name list to the __code__ attribute, along with new bytecodes to access them, it would create a way to keep private local names without changing how the other bytecodes work.

LOAD_MUTABLE 0, would get the first reference in __code__.co_mutables.

As a side note, closure variables aren't accessed by LOAD_FAST and SAVE_FAST from either side; that's what cellvars and freevars are for. So, your details don't actually work. But it's not hard to s/fast/cell/ and similar in your details and understand what you mean. But I don't see why you couldn't just implement the "mutable" keyword to mean that the variable must be in names rather than cellnames or freenames or varnames (raising a compile-time error if that's not possible) and just continue using *_FAST on them. That would be a lot simpler to implement. It's also a lot simpler to explain: declaring a variable mutable means it can't participate in closures.

As I said it's a partial solution. Shared & mutable names passed as function arguments will still need to be protected

The problem here is the same as in Sven's proposal: the problem is shared values, not shared variables, so any solution that just tries to limit shared variables is only a vanishingly tiny piece of the solution. It doesn't do anything for mutable values passed to functions, or returned or yielded, or stored on self, or stored in any other object's attributes or in any container, or even in globals. It also doesn't prevent you from mutating them by calling a method (including __setattr__ and __setitem__ and the various __i*__ methods as well as explicit method calls). And, even if you managed to solve all of those problems, it still wouldn't be useful, because it doesn't do anything for any case where you share a member or element of the object rather than the object itself--e.g., if I have a dict mapping sockets to Connection objects, marking the dict unshareable doesn't protect the Connection objects in any way.

On Aug 20, 2015, at 16:57, Ron Adam wrote:

On 08/20/2015 05:26 PM, Andrew Barnert via Python-ideas wrote:

...

...

It may seem like it isn't needed, because you have control over what a function has access too... ie... just don't do that. But when you have many programmers working on large projects, things can get messy. And this helps with that, but also helps in the case of threads. The only case you're helping with here is the case where the race is entirely local to one function and the functions it defines--a relatively uncommon case that also gets the least messy and is the easiest to spot and debug.

Also, the "dangerous" cases are already marked today: the local function has to explicitly declare the variable nonlocal or it can't assign to it.

But you can mutate it, so it's not already marked today.

But your solution does nothing to stop values from being mutated, only to stop variables from being reassigned, so you haven't fixed anything.

...

...

...

...

...

>>>> LOAD_FAST 0, reads __code__.co_varnames[0] LOAD_FAST >>>> 1, reads __code__.co_varnames[1] >>>> >>>> Adding a co_mutables name list to the __code__ >>>> attribute, along with new bytecodes to access them, >>>> it would create a way to keep private local names >>>> without changing how the other bytecodes work. >>>> >>>> LOAD_MUTABLE 0, would get the first reference in >>>> __code__.co_mutables. As a side note, closure variables aren't accessed by LOAD_FAST and SAVE_FAST from either side; that's what cellvars and freevars are for. So, your details don't actually work. But it's not hard to s/fast/cell/ and similar in your details and understand what you mean.

Yes... thats what I mean. ;-)

...

But I don't see why you couldn't just implement the "mutable" keyword to mean that the variable must be in names rather than cellnames or freenames or varnames (raising a compile-time error if that's not possible) and just continue using *_FAST on them. That would be a lot simpler to implement. It's also a lot simpler to explain: declaring a variable mutable means it can't participate in closures.

Sounds good to me.

...

...

...

As I said it's a partial solution. Shared & mutable names passed as function arguments will still need to be protected The problem here is the same as in Sven's proposal: the problem is shared values, not shared variables, so any solution that just tries to limit shared variables is only a vanishingly tiny piece of the solution.

The problem is shared mutable values.

One thing I've been wondering is what is considered a mutable object in an practical sense. For example a tuple is not mutable, but it's items may be.

Yes, that problem already exists for hashabllity (which, in Python, is strongly connected to immutability). Calling hash() on a tuple may still raise an exception of one of its elements is mutable. The reason this is rarely a problem is that it's not hard to just avoid storing mutable values in tuples that are used as dict keys. But technically, the problem is there. In your case, the same doesn't help, because it _is_ hard to avoid storing mutable values in everything in the world except specially-marked local variables.

But it seems to me it should still be considered a mutable object if it has any mutable sub items in it. So a test for mutability needs to test content. It probably needs another word or description. Completely immutable? (?) It seems like there should already be a word for an object that has no mutable sub items or parts. It also seems like there should be an easy way to test an object for that.

Only by recursing through the entire thing, as calling hash() does. In a statically-typed language, it's a different story, because if you declare, or the compiler infers, that you've got, say, a 3-tuple of nothing but ints or other 3-tuples of the same type or frozensets of the same type, then any value that it holds must be recursively immutable. (Although that can get tricky in practice, of course. Imagine writing that ADT manually, or reading it.) But that doesn't help for Python. The only obvious efficient way I can think of to solve this is in Python what I said before: having an entirely separate heap of objects, so you know that the objects in that thread can't be part of any objects in another thread. In other words, process semantics, which we already have. Maybe there are other ways to solve it, but ignoring it definitely doesn't count as solving it.

...

It doesn't do anything for mutable values passed to functions, or returned or yielded, or stored on self, or stored in any other object's attributes or in any container, or even in globals.

Returned mutable values shouldn't be a problem unless the function reuses the same object over again.

Sure it would: connections.append(connect()) with Pool() as pool: for _ in range(10): pool.apply_async(broadcast, connections, "warning") Now you've got 10 threads that all want to mutate the same object returned by connect(), even though it isn't stored in a variable.

But yes it doesn't solve those cases.

Right, it only solves an uncommon case, which also happens to be the simplest to detect and debug.

...

It also doesn't prevent you from mutating them by calling a method (including __setattr__ and __setitem__ and the various __i*__ methods as well as explicit method calls).

Nonlocal doesn't prevent that. Not being able to get to it does. For small programs that aren't nested deeply it usually not a problem. Just don't reuse names and be sure to initialize values locally.

That's the whole problem: reusing names isn't the issue, it's different parts of the program having the same values under different names (or just using them without binding them to names at all). That's where races come from in real-life programs. And most complex programs are not deeply nested (they're wide, not tall), so that isn't the problem either.

...

And, even if you managed to solve all of those problems, it still wouldn't be useful, because it doesn't do anything for any case where you share a member or element of the object rather than the object itself--e.g., if I have a dict mapping sockets to Connection objects, marking the dict unshareable doesn't protect the Connection objects in any way.

That would be the part that this won't address, you would still need to either deep copy or lock all the shared mutable locations within that dict including the items and any attributes. Not only the items passed and returned, but for all mutable values that can seen by any functions in threads. This last part was what I was thinking could be made easier.. is it possible to reduce how much is seen, and so reduce the amount of locks.

I think it was an interesting discussion and will help me think about these things in the future, but I'm not sure it would do what I was thinking at the start.

If you want to take this idea further, look at what you could do with a two-level store. For example, in Oz (slightly oversimplifying and distorting things), names are bound to variables, and lists hold variables, and so on, while variables hold values. You can also implement this in something like C++, if you only store shared_ptr<T> values rather than storing T values directly (at least whenever T is mutable). I think you could build something around declaring the sharedness of the variables, rather than the names. Would that be sufficient without transitive static typing? I'm not sure, but it might be worth thinking through. But I don't think that would lead to anything useful for Python. Another possibility is to look at ways to move rather than copy values. That doesn't solve everything, but a few years of experience with C++11 seems to show that it can solve many real-world problems. (Of course this assumes the existence of collections that you can only move, not copy, things into. Or maybe just using a move-to-queue API and not directly accessing the collection underneath it?) There might be something doable for Python there.

On 08/20/2015 10:57 PM, Steven D'Aprano wrote:

On Thu, Aug 20, 2015 at 04:02:52PM -0400, Ron Adam wrote:

...

...

The semantics is to have a way to make names (for mutable objects) in outer scopes not be visible to function defined in inner scopes. Why would you want that? This is a serious question -- there is already an easy way to ensure that two functions can't see the other's variables:*don't* nest them. The reason for nesting them is to ensure that the inner function*can* see the outer's variables. That's the whole point of nesting.

Ok, I'm convinced it's wouldn't do what I initially was thinking. It could possibly offer some benefits to catch some programming errors, but not enough, and would not help with threads. Hmmm... I think maybe I mixed up some dynamic scope behaviour with static scope in my initial thoughts. That would be quite different, but not python. (No, don't explain further, it was a mistake on my part, as I know the difference.) Oh, and thanks to you and Andrew for the feedback, even though it didn't go anywhere. Cheers, Ron