[Python-ideas] easy thread-safety [was: fork]

[Ron Adam]

On 08/19/2015 04:46 PM, Andrew Barnert via Python-ideas wrote:
>> >Would something in this direction simplify the problem?
> Well, the problem can be solved with a strong enough static typing system, as multiple ML-derived
 > languages that add mutability and/or unsafety prove. But what you're 
suggesting doesn't nearly
 > strong enough, or static enough, to accomplish that.

I'll try to explain what I'm thinking, and see where this goes.

The general idea is to keep mutable objects in function local only 
names.  It's not a complete solution by it self.  Some objects will 
still need to be managed as shared objects, but they will easy(er) to 
identify.


> First, in this code, is i mutable or not:
>
>      def spam(a):
>          for i in a:
>              eggs(i)

In this case 'i' is immutable as it's not marked as 'mutable'.  That 
determines what byte code is used, (or how the byte code that is used acts).

 >>> dis(spam)
   2           0 SETUP_LOOP              24 (to 27)
               3 LOAD_FAST                0 (a)
               6 GET_ITER
         >>    7 FOR_ITER                16 (to 26)
              10 STORE_FAST               1 (i)

   3          13 LOAD_GLOBAL              0 (eggs)
              16 LOAD_FAST                1 (i)
              19 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              22 POP_TOP
              23 JUMP_ABSOLUTE            7
         >>   26 POP_BLOCK
         >>   27 LOAD_CONST               0 (None)
              30 RETURN_VALUE

If a compiler directive/flag for threading was set, then the 
interpreter/compiler may use different LOAD_FAST and GET_ITER routines 
that will raise an exception if "i" or "a" is mutable.  (by checking a 
flag on the object, or other means.)

If however...

     def eggs(i):
         mutable i       # mutable objects clearly marked.
         i.scramble()

     def spam(a):
         mutable a, i
         for i in a:
             eggs(i)    # eggs can't access 'a' or 'i' in it's scope.
                        # so i needs to be passed.

Instead of LOAD_FAST and STORE_FAST, it would have LOAD_MUTABLE and 
STORE_MUTABLE.  That doesn't mean they are mutable, but that they may 
be, and are stored as co_mutable names that can't be accessed by a 
closure or non_local scope.

It's not as extreme as requiring all immutable objects.


> And if eggs is imported from a module not marked "thread-safe", does is the call illegal, or assumed
 > to do something that mutates i, or assumed to be safe?

The module could be local to only that thread.  That may be one way a 
thread can use non thread safe code safely.

> Also, whether eggs is from a "thread-safe" module or a normal one, how does the compiler know whether
 > it's passing i to a new thread?

The only objects that are both shared and mutable with this model are 
those passed through a thread API call explicitly.

    future_result = newthread(foo, ...)  # <-- only these "foo, ..."
                                         # need attention.


> And what happens if eggs stores i in a list or other mutable
 > object and some other code mutates it later?

If it's all in the same thread, no problem.  If it's in another thread, 
then it may be an issue, and those details will still need to be worked out.

    def main():
        mutable a, f
        new_thread(foo, a, b, [e, f, g])
        new_thread(bar, e, f, g)
        ...

Only a and f are both shared and mutable.

Or:

    def main():
        mutable persons, people
        people = get_people()
        places = get_places()
        new_thread(foo, persons)
        new_thread(bar, persons)
        ...

The list of persons would need all mutable items in them managed in some 
way.  places is not mutable, and so they will be visible to foo and bar, 
but need no special handling.

But we don't have to worry about the list of people as it's not passed 
or visible to new_threads.  So the problem is reduced to a smaller set 
of relatively easy to identify objects.


> Finally, if you only track mutability at the top level, how can you tell the compiler that a (mutable)
 > queue of ints is thread-safe, but a queue of lists is not?
 > And how can the compiler know which one it's looking at without doing 
a complete whole-program
 > type inference?

The problem of nested mutable objects will still be a concern.

Cheers,
    Ron