On 10/20/05, Phillip J. Eby <pje@telecommunity.com> wrote:

At 08:57 AM 10/20/2005 -0700, Guido van Rossum wrote:

...

Whoa, folks! Can I ask the gentlemen to curb their enthusiasm?

PEP 343 is still (back) on the drawing table, PEP 342 has barely been implemented (did it survive the AST-branch merge?), and already you are talking about adding more stuff. Please put on the brakes!

Sorry. I thought that 343 was just getting a minor tune-up.

Maybe, but the issues on the table are naming issues -- is __with__ the right name, or should it be __context__? Should the decorator be applied implicitly? Should the decorator be called @context or @contextmanager?

In the months since the discussion and approval (and implementation; Michael Hudson actually had a PEP 343 patch out there),

Which he described previously as "a hack" and apparently didn't feel comfortable checking in. At least some of it will have to be redone, (a) for the AST code, and (b) for the revised PEP.

I've been doing a lot of thinking about how they will be used in applications, and thought that it would be a good idea to promote people using task-specific variables in place of globals or thread-locals.

That's clear, yes. :-) I still find it unlikely that a lot of people will be using trampoline frameworks. You and Twisted, that's all I expect.

The conventional wisdom is that global variables are bad, but the truth is that they're very attractive because they allow you to have one less thing to pass around and think about in every line of code.

Which doesn't make them less bad -- they're still there and perhaps more likely to trip you up when you least expect it. I think there's a lot of truth in that conventional wisdom.

Without globals, you would sooner or later end up with every function taking twenty arguments to pass through states down to other code, or else trying to cram all this data into some kind of "context" object, which then won't work with code that doesn't know about *your* definition of what a context is.

Methinks you are exaggerating for effect.

Globals are thus extremely attractive for practical software development. If they weren't so useful, it wouldn't be necessary to warn people not to use them, after all. :)

The problem with globals, however, is that sometimes they need to be changed in a particular context. PEP 343 makes it safer to use globals because you can always offer a context manager that changes them temporarily, without having to hand-write a try-finally block. This will make it even *more* attractive to use globals, which is not a problem as long as the code has no multitasking of any sort.

Hm. There are different kinds of globals. Most globals don't need to be context-managed at all, because they can safely be shared between threads, tasks or coroutines. Caches usually fall in this category (e.g. the compiled regex cache). A little locking is all it takes. The globals that need to be context-managed are the pernicious kind of which you can never have too few. :-) They aren't just accumulating global state, they are implicit parameters, thereby truly invoking the reasons why globals are frowned upon.

Of course, the multithreading scenario is usually fixed by using thread-locals. All I'm proposing is that we replace thread locals with task locals, and promote the use of task-local variables for managed contexts (such as the decimal context) *that would otherwise be a global or a thread-local variable*. This doesn't seem to me like a very big deal; just an encouragement for people to make their stuff easy to use with PEP 342 and 343.

I'm all for encouraging people to make their stuff easy to use with these PEPs, and with multi-threading use. But IMO the best way to accomplish those goals is to refrain from global (or thread-local or task-local) context as much as possible, for example by passing along explicit context. The mere existence of a standard library module to make handling task-specific contexts easier sends the wrong signal; it suggests that it's a good pattern to use, which it isn't -- it's a last-resort pattern, when all other solutions fail. If it weren't for Python's operator overloading, the decimal module would have used explicit contexts (like the Java version); but since it would be really strange to have such a fundamental numeric type without the ability to use the conventional operator notation, we resorted to per-thread context. Even that doesn't always do the right thing -- handling decimal contexts is surprisingly subtle (as Nick can testify based on his experiences attempting to write a decimal context manager for the with-statement!). Yes, coroutines make it even subtler. But I haven't seen the use case yet for mixing coroutines with changes to decimal context settings; somehow it doesn't strike me as a likely use case (not that you can't construct one, so don't bother -- I can imagine it too, I just think YAGNI).

By the way, I don't know if you do much with Java these days, but a big part of the whole J2EE fiasco and the rise of the so-called "lightweight containers" in Java has all been about how to manage implicit context so that you don't get stuck with either the inflexibility of globals or the deadweight of passing tons of parameters around.

I have to trust your word on that; I'm using Tomcat and not liking it but overly long parameter lists or context management aren't on my list of gripes. I have no idea what a "lightweight container" is. It sounds (especially since you put it in scare quotes :-) like a typical Java understatement.

One of the big selling points of AspectJ is that it lets you implicitly funnel parameters from point A to point B without having to modify all the call signatures in between.

Again, I'll have to trust you on this. I've never tried AspectJ or any other aspect-oriented system. But frankly I believe the idea is overhyped -- there are a few example cases that everyone uses to show it off (persistence, thread-safety) but I'm not sure these warrant the weight of the solution.

In other words, its use is promoted for precisely the sort of thing that 'with' plus a task variable would be ideal for.

This I simply don't follow (except that you seem to agree with me that AspectJ is overkill :-). The with-statement is primarily useful for mandatory cleanup (or release) and for restoring temporary changes. Even if decimal contexts were always passed around explicitly, a with-statement around a block with temporarily increased precision or changed error handling would make sense.

As far as I can tell, 'with' plus a task variable is *much* easier to explain, use, and understand than an aspect-oriented programming tool is! (Especially from the "if the implementation is easy to explain, it may be a good idea" perspective.)

And this may be a very good thing. But I still expect that the number of people who need these is a lot smaller than you think (since clearly *you* need it :-).

...

I know that somewhere in the proto-PEP Phillip argues that the context API needs to be made a part of the standard library so that his trampoline can efficiently swap implicit contexts required by arbitrary standard and third-party library code. My response to that is that library code (whether standard or third-party) should not depend on implicit context unless it assumes it can assume complete control over the application.

I think maybe there's some confusion here, at least on my part. :) I see two ways to read your statement, one of which seems to be saying that we should get rid of the decimal context (because it doesn't have complete control over the application), and the other way of reading it doesn't seem connected to what I proposed.

I simply see decimal as the exception that proves the rule.

Anything that's a global variable is an "implicit context".

See above for my quibbles with that.

Because of that, I spent considerable time and effort in PEAK trying to utterly stamp out global variables. *Everything* in PEAK has an explicit context. But that then becomes more of a pain to *use*, because you are now stuck with managing it, even if you cram it into a Zope-style acquisition tree so there's only one "context" to deal with. Plus, it assumes that everything the developer wants to do can be supplied by *one* framework, be it PEAK, Zope, or whatever, which is rarely the case but still forces framework developers to duplicate everybody else's stuff.

Well, face it. Frameworks want to control the world. Multiple frameworks rarely cooperate until they somehow agree on a common ground. That usually doesn't happen until both frameworks are already mature, and then it's painful of course. But I don't see a solution -- that's just the nature of frameworks.

In other words, I've come to realize that the path the major Python application frameworks is not really Pythonic.

(Is there a missing work "take" after "frameworks"?)

A Pythonic framework shouldn't load you down with new management burdens and keep you from using other frameworks. It should make life easier, and make your code *more* interoperable, not less. Indeed, I've pretty much come to agreement with the part of the Python developer community that has says Frameworks Are Evil.

I would agree, yes. :-)

A primary source of this evil in the big three frameworks (PEAK, Twisted, and Zope) stem from their various approaches to dealing with this issue of context, which lack the simplicity of global (or task-local) variables.

I think that's rather an exaggeration (again for effect?). They're frameworks, they want you to do everything in a way that reflects the framework's philosophy. Python, in its design philosophy, tries hard *not* to be a framework. (This sets it apart from Java, which is hostile to non-Java code.) Python tries to be helpful when you want to solve part of your problem using a different tool. It tries to work well even if Python is only a small part of your total solution. It tries to be agnostic of platform-specific frameworks, optionally working with them (e.g. fork and pipes on Unix) but not depending or relying on them. Even threads are quite optional to Python.

So, the lesson I've taken from my attempt to make everything explicit is that what developers *really* want is to have global variables, just without the downsides of uncontrolled modifications, and inter-thread or inter-task pollution. Explicit isn't always better than implicit, because oftentimes the practicality of having implicit things is much more important than the purity of making them all explicit. Simple is better than complex, and task-local variables are *much* simpler than trying to make everything explicit.

Maybe. But this may just be a case where you simply can't have your cake and eat it too. I expect that having 100 task-local variables would probe to be just as big a pain as 100 other forms of context, implicit or explicit.

...

Also, Nick wants the name 'context' for PEP-343 style context managers. I think it's overloading too much to use the same word for per-thread or per-coroutine context.

Actually, I was the one who originally proposed the term "context manager", and it doesn't seem like a conflict to me. Indeed, I suggested in the pre-PEP that "@context.manager" might be where we could put the decorator. The overload was intentional, to suggest that when creating a new context manager, it's worth considering whether the state should be kept in a context variable, rather than a global variable. The naming choice was for propaganda purposes, in other words. :)

That may be, but I think it's confusing, since most of the popular uses of the with-statement will have nothing to do with task-locals.

Anyway, I'll withdraw the proposal for now.

Thanks.

We can always leave it out of 2.5, I can release an independent implementation, and then submit it for consideration again in the 2.6 timeframe.

That sounds like a much better plan than rushing into it now.

I just thought it would be a no-brainer to use task locals where thread locals are currently being used, and that's really all I was proposing we do as far as stdlib changes anyway. I was also hoping to get good input from Python-dev regarding some of the open issues, to try and build a consensus on them from the beginning.

If you look at the code in decimal.py, it already has three different ways to handle contexts, depending on the Python version and whether it has threads. Adding task-locals would just complicate matters. (Sorry for the long post -- there just wasn't anything you said that I felt could be left unquoted. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)