On Sat, Jun 26, 2010 at 10:39, John Nagle <nagle@animats.com> wrote:
The rationale behind "freezing" some of the language semantics when the program goes multi-thread comes from two sources - Adam Olsen's Safethread work, and the acceptance of the multiprocessing module. Olsen tried to retain all the dynamism of the language in a multithreaded environment, but locking all the underlying dictionaries was a boat-anchor on the whole system, and slowed things down so much that he abandoned the project. The Unladen Swallow documentation indicates that early thinking on the project was that Olsen's approach would allow getting rid of the GIL, but later notes indicate that no path to a GIL-free JIT system is currently in development.
That's not true. Refcounting was the boat-anchor, not dicts. I was unable to come up with a relatively simple replacement that scaled fully. The dicts shared as module globals and class dicts were a design issue, but more of an ideological one: concurrency mentality says you should only share immutable objects. Python prefers ad-hoc design, where you can do what you want so long as it's not particularly nasty. I was unable to find a way to have both, so I declared the python mentality the winner. The shareddict I came up with uses a read write lock, so that it's safe when you do mutate and doesn't bottleneck when you don't mutate. The only thing fancy was my method of checkpointing when doing a readlock->writelock transition, but there's a hundred other ways to accomplish that.