[Python-Dev] Pre-PEP: Task-local variables

[Phillip J. Eby]

At 08:57 AM 10/20/2005 -0700, Guido van Rossum wrote:
>Whoa, folks! Can I ask the gentlemen to curb their enthusiasm?
>
>PEP 343 is still (back) on the drawing table, PEP 342 has barely been
>implemented (did it survive the AST-branch merge?), and already you
>are talking about adding more stuff. Please put on the brakes!

Sorry.  I thought that 343 was just getting a minor tune-up.  In the months 
since the discussion and approval (and implementation; Michael Hudson 
actually had a PEP 343 patch out there), I've been doing a lot of thinking 
about how they will be used in applications, and thought that it would be a 
good idea to promote people using task-specific variables in place of 
globals or thread-locals.

The conventional wisdom is that global variables are bad, but the truth is 
that they're very attractive because they allow you to have one less thing 
to pass around and think about in every line of code.  Without globals, you 
would sooner or later end up with every function taking twenty arguments to 
pass through states down to other code, or else trying to cram all this 
data into some kind of "context" object, which then won't work with code 
that doesn't know about *your* definition of what a context is.

Globals are thus extremely attractive for practical software 
development.  If they weren't so useful, it wouldn't be necessary to warn 
people not to use them, after all.  :)

The problem with globals, however, is that sometimes they need to be 
changed in a particular context.  PEP 343 makes it safer to use globals 
because you can always offer a context manager that changes them 
temporarily, without having to hand-write a try-finally block.  This will 
make it even *more* attractive to use globals, which is not a problem as 
long as the code has no multitasking of any sort.

Of course, the multithreading scenario is usually fixed by using 
thread-locals.  All I'm proposing is that we replace thread locals with 
task locals, and promote the use of task-local variables for managed 
contexts (such as the decimal context) *that would otherwise be a global or 
a thread-local variable*.  This doesn't seem to me like a very big deal; 
just an encouragement for people to make their stuff easy to use with PEP 
342 and 343.

By the way, I don't know if you do much with Java these days, but a big 
part of the whole J2EE fiasco and the rise of the so-called "lightweight 
containers" in Java has all been about how to manage implicit context so 
that you don't get stuck with either the inflexibility of globals or the 
deadweight of passing tons of parameters around.  One of the big selling 
points of AspectJ is that it lets you implicitly funnel parameters from 
point A to point B without having to modify all the call signatures in 
between.  In other words, its use is promoted for precisely the sort of 
thing that 'with' plus a task variable would be ideal for.  As far as I can 
tell, 'with' plus a task variable is *much* easier to explain, use, and 
understand than an aspect-oriented programming tool is!  (Especially from 
the "if the implementation is easy to explain, it may be a good idea" 
perspective.)


>I know that somewhere in the proto-PEP Phillip argues that the context
>API needs to be made a part of the standard library so that his
>trampoline can efficiently swap implicit contexts required by
>arbitrary standard and third-party library code. My response to that
>is that library code (whether standard or third-party) should not
>depend on implicit context unless it assumes it can assume complete
>control over the application.

I think maybe there's some confusion here, at least on my part.  :)  I see 
two ways to read your statement, one of which seems to be saying that we 
should get rid of the decimal context (because it doesn't have complete 
control over the application), and the other way of reading it doesn't seem 
connected to what I proposed.

Anything that's a global variable is an "implicit context".  Because of 
that, I spent considerable time and effort in PEAK trying to utterly stamp 
out global variables.  *Everything* in PEAK has an explicit context.  But 
that then becomes more of a pain to *use*, because you are now stuck with 
managing it, even if you cram it into a Zope-style acquisition tree so 
there's only one "context" to deal with.  Plus, it assumes that everything 
the developer wants to do can be supplied by *one* framework, be it PEAK, 
Zope, or whatever, which is rarely the case but still forces framework 
developers to duplicate everybody else's stuff.

In other words, I've come to realize that the path the major Python 
application frameworks is not really Pythonic.  A Pythonic framework 
shouldn't load you down with new management burdens and keep you from using 
other frameworks.  It should make life easier, and make your code *more* 
interoperable, not less.  Indeed, I've pretty much come to agreement with 
the part of the Python developer community that has says Frameworks Are 
Evil.  A primary source of this evil in the big three frameworks (PEAK, 
Twisted, and Zope) stem from their various approaches to dealing with this 
issue of context, which lack the simplicity of global (or task-local) 
variables.

So, the lesson I've taken from my attempt to make everything explicit is 
that what developers *really* want is to have global variables, just 
without the downsides of uncontrolled modifications, and inter-thread or 
inter-task pollution.  Explicit isn't always better than implicit, because 
oftentimes the practicality of having implicit things is much more 
important than the purity of making them all explicit.  Simple is better 
than complex, and task-local variables are *much* simpler than trying to 
make everything explicit.


>Also, Nick wants the name 'context' for PEP-343 style context
>managers. I think it's overloading too much to use the same word for
>per-thread or per-coroutine context.

Actually, I was the one who originally proposed the term "context manager", 
and it doesn't seem like a conflict to me.  Indeed, I suggested in the 
pre-PEP that "@context.manager" might be where we could put the 
decorator.  The overload was intentional, to suggest that when creating a 
new context manager, it's worth considering whether the state should be 
kept in a context variable, rather than a global variable.  The naming 
choice was for propaganda purposes, in other words.  :)

Anyway, I'll withdraw the proposal for now.  We can always leave it out of 
2.5, I can release an independent implementation, and then submit it for 
consideration again in the 2.6 timeframe.  I just thought it would be a 
no-brainer to use task locals where thread locals are currently being used, 
and that's really all I was proposing we do as far as stdlib changes 
anyway.  I was also hoping to get good input from Python-dev regarding some 
of the open issues, to try and build a consensus on them from the beginning.