[pypy-dev] Thinking about the GIL

[hyarion at iinet.net.au]

On Fri Mar 25 21:41 , Armin Rigo  sent:

>Hi,
>
>On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au
>hyarion at iinet.net.au> wrote:
>> (...) Wrapping each bytecode in an STM
>> transaction would give you an as-if-serial execution order, again with no 
guarantees
>> about which order. You get transaction overheads instead of lock/unlock 
overheads,
>> but some STM systems can be quite efficient for short transactions that rarely
>> conflict.
>
>Yes, I also thought about this as one of the solutions that would "fit
>the model" of PyPy by not needing changes all over the place.
>However, I am unsure that the performance of STM is good enough for
>that application so far.  Maybe I'm wrong, but I fear (a priori, with
>no precise experience) that it would be really too slow to wrap *all*
>memory reads and writes with the STM machinery.

Yeah, I suppose to get the performance I'm thinking of you probably need to know 
what you're sharing (and for it not to be everything). You wouldn't need all memory 
reads and writes to be transactional, just ones to interpreter-level values that are 
mutable and shared between threads, and ones to things that represent app-level 
shareable values.

The STM system I worked on was for a declarative language with immutable data. In 
that context it turns out the system only had to be careful when retrieving 
references from STM; what the reference points to could still be used as normal non-
shared data. In a language like Python I guess almost any bit of memory could 
potentially be (or later become) a shared value that another thread could be using.

What's the "success-rate" of the JIT's malloc-removal like? You could omit the 
transactional overhead for those values. Would that get close enough for the 
important 20% of code? You could probably do some other jiggery-pokery under the 
hood to minimise the amount of data protected by STM overhead too.

If I had more time I'd love to look at this sort of thing.

-- Ben