[Python-ideas] easy thread-safety [was: fork]

[Sven R. Kunze]

On 18.08.2015 23:53, Paul Moore wrote:
> On 18 August 2015 at 21:32, Sven R. Kunze <srkunze at mail.de> wrote:
>> To me, a process/thread or any other concurrency solution, is basically a
>> function that I can call but runs in the background. Later, when I am ready,
>> I can collect its result. In the meantime, the main thread continues.
>> (Again) to me, that is the only sensible way to approach concurrency.
> It seems to me that this is accurate, but glosses over all of the
> issues that result in multiple solutions being needed.

I can identify 2 common patterns I label as jobs and servers.

Jobs are things that get delegated out to some background 
process/thread/coroutine/subinterpreter. They come back when the job is 
done. No shared state necessary.
Servers are more like while true loops running in some separate 
process/thread/coroutine/subinterpreter. They only return on shutdown or 
so. Shared state a là queues for input/output could come in handy.


Maybe, there is more to discover but it's more like research that 
production.

> Sure, all
> concurrency solutions provide this. But the difference lies in the
> environment of the function you're calling. Does it have access to the
> same non-local name bindings as it would if run in the foreground? To
> the same objects? Is it able to write to those objects safely, or must
> it treat them as read only. Or can it write, but only if it follows a
> particular protocol (semaphores, locks, etc fit here)?

I am uncertain of how to approach this in the correct way.

However, my approach here would be simply by imitating sequential 
behavior. I always would ask: "What would happen if that is executed 
sequentially?" The answer would then be retrofitted to the parallel 
scenario.

> Conversely, it *is* true that a lot of problems that benefit from
> concurrency can work with the minimal guarantees of a lowest common
> denominator solution (no shared state pure functions). Functional
> programming has shown us that. For those problems, any of the options
> are fine, and the decision gets made on other factors (most likely
> performance, as each solution makes different performance trade-offs
> in the process of providing whatever extra guarantees they make).

I hope eventually, the interpreter will take the decision burden away 
from the developers and make educated guesses to achieve the best 
performance.

> I'm confused as to what your point is. "People should write concurrent
> code in a no shared state pure function manner" seems to be what your
> comment "the only sensible way" implies.

I think you are referring to that statement only. It's just the 
underlaying motivation for me to engage in this discussion and it's born 
by observing real-world code development and maintenance.

If I were to ask people around the globe of what they use in production, 
I guess I would get answers like that (that'll be an awesome survey btw.):

30% declaration - you say what you want (cfg files, decorators, sql, 
css, html, etc.)
20% imperative - you say what to do (function call hierarchies)
25% object oriented - you think in "objects" (object relationship tree)
15% packages/modules - you cluster functionality - (file/directory 
hierarchies)
10% magic stuff - you never get it right (generators, concurrency, 
meta-classes, import hooks, AST, etc.)

Just look at big projects. Look at different programming languages. All 
the same.

Maybe, my observation is wrong; that could be.
Maybe, the observation is true and people are just dumb, lazy and unable 
to appreciate a fine 10 hours live-lock bug hunting session (but I doubt 
that).

Most professionals, I am working with, are highly intelligent people 
working on very complex problems and with a very few resources. Thus, 
they don't appreciate tools making their lives more difficult by 
changing the problem's domain from complex to complicated. As a result, 
they are not going to use it at all. Make it like the top 90% of what 
people are used to and they are going to use it.


Point is not, people should do this or that.

Point is, the tools should make it stupidly easy to get things done. 
People then will follow and do this or that automatically. Let's improve 
the tools.


Best,
Sven