<div>Here are issues to think about. You wrote:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)">'locking' value [',' value]*':' suite</span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><br></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)">The list of values are the objects that can be mutated in this </span><span style="font-family:arial, sans-serif;font-size:14px;background-color:rgb(255, 255, 255)">lock. An immutable object showing up in the list of values is a </span><span style="font-family:arial, sans-serif;font-size:14px;background-color:rgb(255, 255, 255)">TypeError.</span></div>
</blockquote><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><br>
</span></div><div><font face="arial, sans-serif">However:</font></div><div><font face="arial, sans-serif"><br></font></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 14px; background-color: rgb(255, 255, 255); ">(1) Greg Ewing gave an example sort of like this:</span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><br>
</span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="arial, sans-serif"><span style="font-size:13px;background-color:rgb(255, 255, 255)">new_balance = balance + deposit<br>
lock(balance)<br>balance = new_balance<br>unlock(balance)<br></span></font></div></blockquote><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-size:13px"><br>
</span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">and pointed out that the locks don't help. He was talking about the race condition on reading balance before writing. But it's worse than that. If these are numbers, then they're immutable and locking them does nothing and this isn't any better:</span></span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><br>
</span></div><blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:40px;border-top-style:none;border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">
<div>lock(balance)</div><div><font face="arial, sans-serif"><span style="font-size:13px;background-color:rgb(255, 255, 255)">new_balance = balance + deposit<br>balance = new_balance<br>unlock(balance)<br></span></font></div>
</blockquote><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-size:13px"><br></span></span></div></span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">Consider this operating on lists:</span></span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div>locking stuff: #1</div><div> stuff += added_stuff</div><div><br></div><div>locking stuff: #2</div><div> stuff = stuff + added_stuff</div></blockquote><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br>
</span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">In #2, locking is completely useless. Sure the values are mutable but I'm not mutating them. Is it obvious that #1 works and #2 doesn't? You don't want to lock the *value* of stuff, you want to lock the *variable*, i.e., locking globals() or wherever the value of stuff is found. Locking globals() seems like a bad idea. </span></span><span style="background-color:rgb(255, 255, 255)"> Which brings me to...</span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">(2) How does </span></span>locking something like a dictionary work? Does it lock the dictionary and all the values in it, walking the entire data structure to lock it? Ypu suggest that's the case and that seems <span style="background-color:rgb(255, 255, 255)">like a performance nightmare. </span>Given</div>
<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">x = [1, 2]<div><div>d = {3: x, 4: 5}</div></div></blockquote><div><div><br></div><div>How do I lock d[3]? That is, I want to lock the dictionary slot where the value of d[3] is stored so another thread can't do d[3] = 0. If I write</div>
</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>locking d[3]: # do stuff</div></blockquote><div><br></div><div>that's going to lock the value of x. Another thread won't be able to do x.append(0) but they would be able to do x = 0 or d[3] = 0. If I have to write</div>
<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>locking d: # do stuff</div></blockquote><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br>
</span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">that hampers concurrency. If I can lock the dictionary slot d[3], can I lock list slots? After all, the compiler doesn't know they type of d. </span></span><span style="background-color:rgb(255, 255, 255)">How do I lock just an attribute without locking the whole object?</span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">(3) What happens if one thread does:</span></span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">locking d, x: # stuff</span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br>
</span></span></div></blockquote><span style="background-color:rgb(255, 255, 255)">and another does</span><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small">locking x, d: # stuff</span></span></div>
<div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><br></span></span></div></blockquote>I think I know the answer and it's "deadlock". Avoiding deadlock is an important problem and by forcing me to lock every single object before I mutate it the important locks (the ones for objects that are actually shared) will be lost among the many that are unnecessary, making it very difficult to find bad lock ordering.<div>
<br></div><div>(4) What if the lock can't be acquired right away? Maybe there's other stuff my thread can do while it's waiting for the lock. You don't consider that. Maybe I have a whole bunch of things all of which can be done in any order.</div>
<div><br></div><div><div><font face="arial, sans-serif">(5) You haven't thought about read vs. write locks. If I'm passed an iterable or a sequence in a concurrent program, I want to read lock it so no one else changes it while I'm working with it. But you prohibit locking immutable objects, so I first have to find out if it's immutable which is something else you'll have to add to the language.</font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><br></font></div><div><span style="font-family: arial, sans-serif; font-size: 13px; background-color: rgb(255, 255, 255); "><br></span></div></div><div>On Mon, Oct 31, 2011 at 10:32 PM, Mike Meyer <span dir="ltr"><<a href="mailto:mwm@mired.org" target="_blank">mwm@mired.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex">
<div class="gmail_quote">I've identified the problem I want to solve: I want to make concurrent<br>use of python objects "safe by default", ...</div></blockquote><div> </div></div><div>To me it looks like this proposal deals with a small part of the problem with the equivalent of a sledgehammer. When I said identify the problem, the above issues are more on the lines of what I was talking about, not a general statement "make concurrency better."</div>
<div><br></div><div>But as an overall goal, I think this is better: "make it as easy as possible to write error-free concurrent code." I would think that "without making it hard to write non-concurrent code" is implicit but I'll spell that out since I've heard that explicit is better than implicit.</div>
<div><br></div><div>And here are some things people writing concurrent programs want to do (in no particular order):</div><div><br></div><div>(1) lock an object (even when the object doesn't have any code to support locking)</div>
<div>(2) lock part of an object (a slot in list or dictionary, or an attribute) - consider that databases support row and rang elocking because if the only thing you can do is lock entire tables, you're going to get poor performance</div>
<div>(3) lock multiple things at the same time</div><div>(4) queue operations to be performed on an object when it can be locked, which requires ...</div><div>(5) wait until a queued operation completes, which requires ...</div>
<div>(6) avoid starvation</div><div></div><div>(7) avoid deadlock</div><div>(8) avoid other concurrency bugs</div><div>(9) avoid debugging (not avoid it when it's necessary but do things to avoid it being necessary)</div>
<div><br></div><div><br></div><div><div><span style="font-family:arial, sans-serif;font-size:13px;background-color:rgb(255, 255, 255)"><span style="font-family:arial;font-size:small"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex">
<div class="gmail_quote">... so that doing unsafe things<br>causes the programmer to have to do something explicit about making<br>things safe. I believe this can be done at the mutation points<br>(again, clojure shows that it can be done). I also want to preserve as<br>
much of Python's existing code as possible. It may be that Python's<br>existing data structures mean my believe about mutation points is<br>wrong. This may be the wrong solution. It may be that such a change is<br>
to large to be acceptable. But the only way to find out is to<br>investigate it.<br></div></blockquote></div><br><div>Clojure is very different from Python. Core data structures are immutable in Clojure so adding to a list creates a new also immutable list. Changing Python's core data structures to be immutable would be a bigger compatibility break than 2 to 3. </div>
</span></span></div><div><span style="font-size: 13px; background-color: rgb(255, 255, 255); "><span style="font-size: small; "><br clear="all">
<font face="arial, helvetica, sans-serif" style="font-family: arial; ">--- Bruce</font><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div></span></span></div>
</div>