Re: [pypy-dev] Thinking about the GIL
On Fri Mar 25 21:41 , Armin Rigo sent:
Hi,
On Fri, Mar 18, 2011 at 1:44 AM, hyarion@iinet.net.au hyarion@iinet.net.au> wrote:
(...) Wrapping each bytecode in an STM transaction would give you an as-if-serial execution order, again with no guarantees about which order. You get transaction overheads instead of lock/unlock overheads, but some STM systems can be quite efficient for short transactions that rarely conflict.
Yes, I also thought about this as one of the solutions that would "fit the model" of PyPy by not needing changes all over the place. However, I am unsure that the performance of STM is good enough for that application so far. Maybe I'm wrong, but I fear (a priori, with no precise experience) that it would be really too slow to wrap *all* memory reads and writes with the STM machinery.
Yeah, I suppose to get the performance I'm thinking of you probably need to know what you're sharing (and for it not to be everything). You wouldn't need all memory reads and writes to be transactional, just ones to interpreter-level values that are mutable and shared between threads, and ones to things that represent app-level shareable values. The STM system I worked on was for a declarative language with immutable data. In that context it turns out the system only had to be careful when retrieving references from STM; what the reference points to could still be used as normal non- shared data. In a language like Python I guess almost any bit of memory could potentially be (or later become) a shared value that another thread could be using. What's the "success-rate" of the JIT's malloc-removal like? You could omit the transactional overhead for those values. Would that get close enough for the important 20% of code? You could probably do some other jiggery-pokery under the hood to minimise the amount of data protected by STM overhead too. If I had more time I'd love to look at this sort of thing. -- Ben
On Sun, Mar 27, 2011 at 8:53 PM, hyarion@iinet.net.au <hyarion@iinet.net.au>wrote:
On Fri Mar 25 21:41 , Armin Rigo sent:
Hi,
On Fri, Mar 18, 2011 at 1:44 AM, hyarion@iinet.net.au hyarion@iinet.net.au> wrote:
(...) Wrapping each bytecode in an STM transaction would give you an as-if-serial execution order, again with no guarantees about which order. You get transaction overheads instead of lock/unlock overheads, but some STM systems can be quite efficient for short transactions that rarely conflict.
Yes, I also thought about this as one of the solutions that would "fit the model" of PyPy by not needing changes all over the place. However, I am unsure that the performance of STM is good enough for that application so far. Maybe I'm wrong, but I fear (a priori, with no precise experience) that it would be really too slow to wrap *all* memory reads and writes with the STM machinery.
Yeah, I suppose to get the performance I'm thinking of you probably need to know what you're sharing (and for it not to be everything). You wouldn't need all memory reads and writes to be transactional, just ones to interpreter-level values that are mutable and shared between threads, and ones to things that represent app-level shareable values.
The STM system I worked on was for a declarative language with immutable data. In that context it turns out the system only had to be careful when retrieving references from STM; what the reference points to could still be used as normal non- shared data. In a language like Python I guess almost any bit of memory could potentially be (or later become) a shared value that another thread could be using.
What's the "success-rate" of the JIT's malloc-removal like? You could omit the transactional overhead for those values. Would that get close enough for the important 20% of code? You could probably do some other jiggery-pokery under the hood to minimise the amount of data protected by STM overhead too.
If I had more time I'd love to look at this sort of thing.
-- Ben _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
https://bitbucket.org/pypy/extradoc/raw/63e4617062b2/talk/pepm2011/bolz-allo... info on the success rate of allocation removal. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
participants (2)
-
Alex Gaynor -
hyarion＠iinet.net.au