On Sun, Oct 30, 2011 at 8:21 PM, Nick Coghlan wrote:

On Mon, Oct 31, 2011 at 1:11 PM, Mike Meyer wrote:

...

The one glaring exception is in concurrent programs. While the tools python has for dealing with such are ok, there isn't anything to warn you when you fail to use those tools and should be.

This will basically run into the same problem that free-threading-in-CPython concepts do - the fine grained checks you need to implement it will kill your single-threaded performance.

These argument seems familiar. Oh, right, it's the "lack of performance will kill you." That was given as the reason that all of the following were unacceptable: - High level languages. - Byte-compiled languages. - Structured programming. - Automatic memory management. - Dynamic typing. - Object Oriented languages. All of those are believed (at least by their proponents) to make programming easier and/or faster at the cost of performance. The performance cost was "too high" when all of them when they were introduced, but they all became broadly accepted as the combination of increasing computing power (especially CPU support for them) and increasingly efficient implementation techniques drove that cost down to the point where it wasn't a problem except for very special cases.

Since Python is a scripting language that sees heavy single-threaded use, that's not an acceptable trade-off.

Right - few languages manage to grow one of those features without a name change of some sort, much less two (with the obvious exception of LISP). Getting them usually requires moving to a new language. That's one reason I said it might never make it into CPython. But the goal is to get people to think about fixing the problems, not dismiss the suggestion because of problems that will go away if we just wait long enough. For instance, the issue of single-threaded performance can be fixed by taking threading out of a library, and giving control of it to the interpreter. This possibility is why I said "thread of execution" instead of just "thread." If the interpreter knows when an application has concurrent execution, it also knows when there aren't any, so it can support an option not to make those checks until the performance issues go away.

Software transactional memory does offer some hope for a more reasonable alternative, but that has its own problems (mainly I/O related). It will be interesting to see how PyPy's experiments in this space pan out.

Right - you can't do I/O inside a transaction. For writes, this isn't a problem. For reads, it does, since they imply binding and/or rebinding. So an STM solution may require a second mechanism designed for single statements to allow reads to happen.

On Mon, Oct 31, 2011 at 11:25 AM, Antoine Pitrou wrote:

On Mon, 31 Oct 2011 10:59:56 -0700 Mike Meyer wrote:

...

On Sun, Oct 30, 2011 at 8:21 PM, Nick Coghlan wrote:

...

On Mon, Oct 31, 2011 at 1:11 PM, Mike Meyer wrote:

...

The one glaring exception is in concurrent programs. While the tools python has for dealing with such are ok, there isn't anything to warn you when you fail to use those tools and should be.

This will basically run into the same problem that free-threading-in-CPython concepts do - the fine grained checks you need to implement it will kill your single-threaded performance.

These argument seems familiar. Oh, right, it's the "lack of performance will kill you." That was given as the reason that all of the following were unacceptable:

Agreed, but killing performance by double digits in a new release is generally considered quite ugly by users.

That may be why languages rarely adopt such features. The users won't put up with the cost hit for the development versions. Except for LISP, of course, whose users know the value of everything but the cost of nothing :-).

Also, I'm not convinced that such approaches really bring anything. My opinion is that good multi-threaded programming is achieved through careful abstraction and separation of concerns, rather than advanced language idioms.

Doesn't that cover all kinds of good programming? But advanced language features are there because they are supposed to help with either abstraction or separation of concerns. Look at the list I presented again: - High level languages. - Byte-compiled languages. - Structured programming. - Automatic memory management. - Dynamic typing. - Object Oriented languages. All help with either abstraction or separation of concerns in some way (ok, byte-compilation the concerns are external, in that it's code portability). So do the features I'd like tosee. In particular, they let you separate code that in which concurrency is a concern from code where it isn't. Another aspect of this issue (and yet another possible reason that these features show up in new languages rather than being added to old ones) is that such changes usually require changing the way you think about programming. It takes a different mindset to program with while loops than with if & goto, or OO than procedural, or .... Similarly, it takes a different mindset to program in a language where changing an object requires special consideration. This may be to much of a straight-jacket for a multi-paradigm language like Python (though Oz manages it), but making the warnings ignorable defeats the purpose.

On Mon, Oct 31, 2011 at 12:14 PM, Antoine Pitrou wrote:

On Mon, 31 Oct 2011 12:03:14 -0700 Mike Meyer wrote:

...

Doesn't that cover all kinds of good programming? But advanced language features are there because they are supposed to help with either abstraction or separation of concerns. Look at the list I presented again:

- High level languages. - Byte-compiled languages. - Structured programming. - Automatic memory management. - Dynamic typing. - Object Oriented languages.

All help with either abstraction or separation of concerns in some way (ok, byte-compilation the concerns are external, in that it's code portability). So do the features I'd like tosee.

I don't think the latter is true. Manually listing objects that are allowed to be mutated seems actually quite low-level to me.

Did you not read the next paragraph, the one where I explained how it helped separate issues? But you're right. Manually listing them isn't all that desirable, it's just better than what we have now. I initially saw a locking keyword with no list as implying that all such things should be locked automatically. I can't see how to do that with the locking implementation, so I dropped it. However, it can be done with an STM implementation. Hmm. That could be the distinguishing feature to deal with IO: If you don't have the list, you get an STM, and it does the copy/fingerprint dance when you mutate something. If you list a value, you get real locks so it won't retry and you can safely do IO.

On Mon, Oct 31, 2011 at 7:33 AM, Amaury Forgeot d'Arc wrote:

2011/10/31 Mike Meyer

...

Any attempt to mutate an object that isn't currently locked will raise an exception. Possibly ValueError, possibly a new exception class just for this purpose. This includes rebinding attributes of objects that aren't locked. PyPy offers a nice platform to play with this kind of concepts. For example, even if it's not the best implementation, it's easy to add a __setattr__ to the base W_Object class, which will check whether the object is allowed to mutate. But you certainly will open a can of worms here: even immutable objects are modified (e.g str.__hash__ is cached) and many functions that you call will need to add their own locks, is it possible to avoid deadlocks in this case?

Just what I need - another project. I'll go take a look at PyPy. In theory, things that don't change the externally visible behavior of an object don't need to be checked. The goal isn't to make the language perfectly safe, it's to make you be explicit about when you want to do something that might not be safe in a concurrent environment. This provides a perfect example of where an immutable subclass would be useful. It would add an __setattr__ that throws an exception. Then the string class (which would inheret from the immutable subclass) could cache it's hash by doing something like W_Object.__setattr__(self, "_cashed_hash", hash(self)) (possibly locked).

On Mon, Oct 31, 2011 at 1:47 PM, Jim Jewett wrote:

On Sun, Oct 30, 2011 at 11:11 PM, Mike Meyer wrote:

...

immutable types will work well in this environment exactly as is. If there isn't a way to generate this list automatically, it is the equivalent of manual memory management.

No, it isn't. The difference is in how mistakes are handled. With manual memory management, references through unassigned or freed pointers are mistakes, but may not generate an error immediately. In fact, it's possible the program will run fine and pass all your unit tests. This is the situation we have now in concurrent programming: mutating a shared object without an appropriate lock is an error that probably passes silently, and it may well pass all your tests without a problem (constructing a test to reliably trigger such a big is an interesting problem in and of itself). While you can automatically manage memory, there are other resources that still have to be managed by hand (open files spring to mind). In some cases you might be able to handle them completely automatically, in others not. In either case, Python manages things so that reading from a file that hasn't been opened is impossible, and reading from one that has been closed generates an immediate error. The goal here is to move from where we are to a place similar to where handling files is, so that failing to properly deal with the possibility of concurrent access causes an error when it happens, not at a point distant in both time and space. BTW, regarding the performance issue. I figured out how to implement this so that the run time cost is zero aside from the lock & unlock steps.

On Mon, Oct 31, 2011 at 11:00 PM, Greg Ewing wrote:

Mike Meyer wrote:

...

The goal here is to move from where we are to a place similar to where handling files is, so that failing to properly deal with the possibility of concurrent access causes an error when it happens, not at a point distant in both time and space.

I don't think what you're suggesting would achieve this, though. The locking required for correctness often involves more than one object or more than one operation on an object. Consider

new_balance = balance + deposit lock(balance) balance = new_balance unlock(balance)

This wouldn't trigger any of your alarms, but it would still be wrong.

You're right - I chose my words poorly. As stated, solving it would involve solving the halting problem. Replace the word "properly" with "at all". I.e. - if you don't think about a concurrent access and should have, it'll cause an error. If you think about it and get it wrong - well, nothing will prevent all bugs. Partially automated resource allocation doesn't prevent the programmer from writing bad code, and this is in that category.

On Mon, Oct 31, 2011 at 3:58 PM, Bruce Leban wrote:

On Sun, Oct 30, 2011 at 8:11 PM, Mike Meyer wrote:

...

Any attempt to mutate an object that isn't currently locked will raise an exception. Possibly ValueError, possibly a new exception class just for this purpose. This includes rebinding attributes of objects that aren't locked.

Do you mean that at any time attempting to mutate an unlocked object throws an exception?

Yes, that's the idea. There are some exceptions, but you have to explicitly work around that restriction.

That would mean that all of my current code is broken.

Pretty much, yes. It's like adding garbage collection and removing alloc*/free. It's going to break a *lot* of code. That's why I said "not in 3. and possibly never in cPython."

Do you mean, that inside the control of 'locking', you can't mutate an unlocked object? That still breaks lots of code that is safe. You can't use itertools.cycle anymore until that's updated in a completely unnecessary way:

def cycle(iterable): # cycle('ABCD') --> A B C D A B C D A B C D ... saved = [] for element in iterable: yield element saved.append(element) *# throws an exception when called on a locked iterable* while saved: for element in saved: yield element

According to what I wrote, yes, it does.Since the list being mutated is only visible inside the function, it doesn't need to be. It might be possible to figure out that this is the case at compile time and thus allow the code to run unmodified. But that's 1) hard, 2) will miss some cases, 3) seems like a corner case. This proposal would break enough code that not breaking this case doesn't seem to be worth the effort. That's a question that needs to be answered.

I think the semantics of this need to be tightened up.

That's why I brought it up. I'm trying to get more eyes on the issue.

Furthermore, merely *reading* an object that isn't locked can cause problems. This code is not thread-safe:

if element in dictionary: return dictionary[element]

so you have to decide how much safety you want and what cost we're willing to pay for this.

You're right - it's not thread safe. However, it also doesn't suffer from the problem I'm trying to deal with, where you mutate an object in a way that leaves things broken, but won't be detected at that point. If it breaks because someone mutates the object underneath it, it'll throw an exception at that point. I know you can construct cases where that isn't so. Maybe we need two types of locking - one that allows readers, and one that doesn't. I could live with that, as you'd still have to consider the issue where you mutate the object.

I wouldn't say it necessarily has to be a new language. All too many new languages have arisen when the problem was actually with some specific part of an implementation, or could have been fixed with a careful iteration. It might be slightly off-topic, but one of Python's strengths in my eyes has been very careful incremental improvements. Prior to 2.5 Python was of no interest to me due to various concerns that have now been fixed. Indeed the language is significantly more consistent and usable than it was in the past. The last thing I (and many others) would want to see is yet another language created, because an an existing excellent language's implementation couldn't keep up with the times, or surmount an obstacle. Innovations like PyPy, multiprocessing, and the currently debated coroutines are what give Python a chance in the future. I might be beating a dead dog, but the last thing Python should do is roll over and die because "it's too hard". Just wanted to point out that language != implementation. On Tue, Nov 1, 2011 at 12:07 PM, Bruce Leban wrote:

On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer wrote:

...

On Mon, Oct 31, 2011 at 3:58 PM, Bruce Leban wrote:

...

On Sun, Oct 30, 2011 at 8:11 PM, Mike Meyer wrote:

...

Any attempt to mutate an object that isn't currently locked will raise an exception. Possibly ValueError, possibly a new exception class just for this purpose. This includes rebinding attributes of objects that aren't locked.

Do you mean that at any time attempting to mutate an unlocked object throws an exception?

Yes, that's the idea.  There are some exceptions, but you have to explicitly work around that restriction.

...

That would mean that all of my current code is broken.

Pretty much, yes. It's like adding garbage collection and removing alloc*/free. It's going to break a *lot* of code. That's why I said "not in 3. and possibly never in cPython."

In order to make concurrent code slightly safer, you're going to break all existing programs that don't use concurrency. That seems to me like a new language, not Python. You've been on this list long enough to see the attention that's paid to backward compatibility.

...

...

Furthermore, merely *reading* an object that isn't locked can cause problems. This code is not thread-safe:     if element in dictionary: return dictionary[element] so you have to decide how much safety you want and what cost we're willing to pay for this.

You're right - it's not thread safe. However, it also doesn't suffer from the problem I'm trying to deal with, where you mutate an object in a way that leaves things broken, but won't be detected at that point. If it breaks because someone mutates the object underneath it, it'll throw an exception at that point. I know you can construct cases where that isn't so.

I think the cases where non-thread-safe code won't throw an exception are numerous, for example, the equally trivial:     if element not in dictionary: dictionary[element] = 0 heck even this is not safe:     dictionary[element] +=1 If you're going to tackle thread safety, it should address more of the problem. These bugs are in many ways worse than mutating "an object in a way that leaves things broken, but won't be detected at that point." The above bugs may *never* be detected. I've come across bugs like that that were in code for many years before I found them (and I'm sure that's happened to others on this list as well). The first thing to do is identify the problems you want to solve and make sure that the problems are well understood. Then design some solutions. Starting with a bad solution to a fraction of the problem isn't a good start. --- Bruce _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas