Mailman 3 October 2005 - Python-Dev

Questionable AST wibbles
by Neal Norwitz Oct. 21, 2005

Oct. 21, 2005

Jeremy, There are a bunch of mods from the AST branch that got integrated into head. Hopefully, by doing this on python-dev more people will get involved. I'll describe high level things first, but there will be a ton of details later on. If people don't want to see this crap on python-dev, I can take this offline. Highlevel overview of code size (rough decrease of 300 C source lines): * Python/compile.c -2733 (was 6822 now 4089) * Python/Python-ast.c +2281 (new file) * Python/asdl.c +… [View More]92 (new file) * plus other minor mods symtable.h has lots of changes to structs and APIs. Not sure what needs to be doc'ed. I was very glad to see that ./python compileall.py Lib took virtually the same time before and after AST. Yeah! Unfortunately, I can't say the same for memory usage for running compileall: Before AST: [10120 refs] After AST: [916096 refs] I believe there aren't that many true memory leaks from running valgrind. Though there are likely some ref leaks. Most of this is probably stuff that we are just hanging on to that is not required. I will continue to run valgrind to find more problems. A bunch of APIs changed and there is some additional name pollution. Since these are pretty internal APIs, I'm not sure that part is a big deal. I will try to find more name pollution and eliminate it by prefixing with Py. One API change which I think was a mistake was _Py_Mangle() losing 2 parameters (I think this was how it was a long time ago). See typeobject.c, Python.h, compile.c. pythonrun.h has a bunch of changes. I think a lot of the APIs changed, but there might be backwards compatible macros. I'm not sure. I need to review closely. symtable.h has lots of changes to structs and APIs. Not sure what needs to be doc'ed. Some #defines are history (I think they are in the enum now): TYPE_*. code.h was added, but it mostly contains stuff from compile.h. Should we remove code.h and just put everything in compile.h? This will remove lots little changes. code.h & compile.h are tightly coupled. If we keep them separate, I would like to see some other changes. This probably is not a big deal, but I was surprised by this change: +++ test_repr.py 20 Oct 2005 19:59:24 -0000 1.20 @@ -123,7 +123,7 @@ def test_lambda(self): self.failUnless(repr(lambda x: x).startswith( - "<function <lambda")) + "<function lambda")) This one may be only marginally worse (names w/parameter unpacking): test_grammar.py - verify(f4.func_code.co_varnames == ('two', '.2', 'compound', - 'argument', 'list')) + vereq(f4.func_code.co_varnames, + ('two', '.1', 'compound', 'argument', 'list')) There are still more things I need to review. These were the biggest issues I found. I don't think most are that big of a deal, just wanted to point stuff out. n [View Less]

6 5

Pre-PEP: Task-local variables
by Phillip J. Eby Oct. 21, 2005

Oct. 21, 2005

This is still rather rough, but I figured it's easier to let everybody fill in the remaining gaps by arguments than it is for me to pick a position I like and try to convince everybody else that it's right. :) Your feedback is requested and welcome. PEP: XXX Title: Task-local Variables Author: Phillip J. Eby <pje(a)telecommunity.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-Oct-2005 Python-Version: 2.5 Post-History: 19-Oct-2005 Abstract ======== … [View More]Many Python modules provide some kind of global or thread-local state, which is relatively easy to implement. With the acceptance of PEP 342, however, co-routines will become more common, and it will be desirable in many cases to treat each as its own logical thread of execution. So, many kinds of state that might now be kept as a thread-specific variable (such as the "current transaction" in ZODB or the "current database connection" in SQLObject) will not work with coroutines. This PEP proposes a simple mechanism akin to thread-local variables, but which will make it easy and efficient for co-routine schedulers to switch state between tasks. The mechanism is proposed for the standard library because its usefulness is dependent on its adoption by standard library modules, such as the ``decimal`` module. The proposed features can be implemented as pure Python code, and as such are suitable for use by other Python implementations (including older versions of Python, if desired). Motivation ========== PEP 343's new "with" statement makes it very attractive to temporarily alter some aspect of system state, and then restore it, using a context manager. Many of PEP 343's examples are of this nature, whether they are temporarily redirecting ``sys.stdout``, or temporarily altering decimal precision. But when this attractive feature is combined with PEP 342-style co-routines, a new challenge emerges. Consider this code, which may misbehave if run as a co-routine:: with opening(filename, "w") as f: with redirecting_stdout(f): print "Hello world" yield pause(5) print "Goodbye world" Problems can arise from this code in two ways. First, the redirection of output "leaks out" to other coroutines during the pause. Second, when this coroutine is finished, it resets stdout to whatever it was at the beginning of the coroutine, regardless of what another co-routine might have been using. Similar issues can be demonstrated using the decimal context, transactions, database connections, etc., which are all likely to be popular contexts for the "with" statement. However, if these new context managers are written to use global or thread-local state, coroutines will be locked out of the market, so to speak. Therefore, this PEP proposes to provide and promote a standard way of managing per-execution-context state, such that coroutine schedulers can keep each coroutine's state distinct. If this mechanism is then used by library modules (such as ``decimal``) to maintain their current state, then they will be transparently compatible with co-routines as well as threaded and threadless code. (Note that for Python 2.x versions, backward compatibility requires that we continue to allow direct reassignment to e.g. ``sys.stdout``. So, it will still of course be possible to write code that will interoperate poorly with co-routines. But for Python 3.x it seems worth considering making some of the ``sys`` module's contents into task-local variables rather than assignment targets.) Specification ============= This PEP proposes to offer a standard library module called ``context``, with the following core contents: Variable A class that allows creation of a context variable (see below). snapshot() Returns a snapshot of the current execution context. swap(ctx) Set the current context to `ctx`, returning a snapshot of the current context. The basic idea here is that a co-routine scheduler can switch between tasks by doing something like:: last_coroutine.state = context.swap(next_coroutine.state) Or perhaps more like:: # ... execute coroutine iteration last_coroutine.state = context.snapshot() # ... figure out what routine to run next context.swap(next_coroutine.state) Each ``context.Variable`` stores and retrieves its state using the current execution context, which is thread-specific. (Thus, each thread may execute any number of concurrent tasks, although most practical systems today have only one thread that executes coroutines, the other threads being reserved for operations that would otherwise block co-routine execution. Nonetheless, such other threads will often still require context variables of their own.) Context Variable Objects ------------------------ A context variable object provides the following methods: get(default=None) Return the value of the variable in the current execution context, or `default` if not set. set(value) Set the value of the variable for the current execution context. unset() Delete the value of the variable for the current execution context. __call__(*value) If called with an argument, return a context manager that sets the variable to the specified value, then restores the old value upon ``__exit__``. If called without an argument, return the value of the variable for the current execution context, or raise an error if no value is set. Thus:: with some_variable(value): foo() would be roughly equivalent to:: old = some_variable() some_variable.set(value) try: foo() finally: some_variable.set(old) Implementation Details ---------------------- The simplest possible implementation is for ``Variable`` objects to use themselves as unique keys into an execution context dictionary. The context dictionary would be stored in another dictionary, keyed by ``get_thread_ident()``. This approach would work with almost any version or implementation of Python. For efficiency's sake, however, CPython could simply store the execution context dictionary in its "thread state" structure, creating an empty dictionary at thread initialization time. This would make it somewhat easier to offer a C API for access to context variables, especially where efficiency of access is desirable. But the proposal does not depend on this. In the PEP author's experiments, a simple copy-on-write optimization to the the ``set()`` and ``unset()`` methods allows for high performance task switching. By placing a "frozen" flag in the context dictionary when a snapshot is taken, and then checking for the flag before making changes, a single snapshot can be shared by multiple callers, and thus a ``swap()`` operation is little more than two dictionary writes and a read. This leads to higher performance in the typical case, because context variables are more likely to set in outer loops, but task switches are more likely to occur in inner loops. A copy-on-write approach thus prevents copying from occurring during most task switches. Possible Enhancements --------------------- The core of this proposal is extremely minimalist, as it should be possible to do almost anything desired using combinations of ``Variable`` objects or by simply using variables whose values are mutable objects. There are, however, a variety of options for enhancement: ``manager`` decorator The ``context`` module could perhaps be the home of the PEP 343 ``contextmanager`` decorator, effectively renamed to ``context.manager``. This could be a natural fit, in that it would remind the creators of new context managers that they should consider tracking any associated state in a ``context.Variable``. Proxy class Sometimes it's useful to have an object that looks like a module global (e.g. ``sys.stdout``) but which actually delegates its behavior to a context-specific instance. Thus, you could have one ``sys.stdout``, but its actual output would be directed based on the current execution context. The simplest form of such a proxy class might look something like:: class Proxy(object): def __init__(self, initial_value): self.var = context.Variable() self.var.set(initial_value) def __call__(self,*value): return object.__getattribute__(self,'var')(*value) def __getattribute__(self, attr): var = object.__getattribute__(self,'var') return getattr(var, attr) sys.stdout = Proxy(sys.stdout) # make sys.stdout selectable with sys.stdout(somefile): # temporary redirect in current context print "hey!" The main open issues in implementing this sort of proxy are in the precise set of special methods (e.g. ``__getitem__``, ``__setattr__``, etc.) that should be supported, and what API should be supplied for changing the value, setting a default value for new threads, etc. Low-level API Currently, this PEP does not specify an API for accessing and modifying the current execution context, nor a C API for such access. It currently assumes that ``snapshot()``, ``swap()`` and ``Variable`` are the only public means of accessing context information. It may be desirable to offer finer-grained APIs for use by more advanced uses (such as creating an API for management of proxies). And it may be desirable to have a C API for use by Python extensions that wish convenient access to context variables. Rationale ========= Different libraries have different uses for maintaining a "current" state, be it global or local to a specific thread or task. There is currently no way for task-management code to find and switch all of these "current" states. And even if it could, task switching performance would degrade linearly as new libraries were added. One possible alternative approach to this proposal, would be for explicit task objects to exist, and to provide a way to give them identities, so that libraries could instead store their own state as a property of the task, rather than storing their state in a task-specific mapping. This offers similar potential performance to a copy-on-write strategy, but would use more memory than this proposal when only one task is involved. (Because each variable would have a dictionary mapping from task to the variable's value, but in this proposal there is simply a single dictionary for the task.) Some languages offer "dynamically scoped" variables that are somewhat similar in behavior to the context variables proposed by this PEP. The principal differences are that: 1. Context variables are objects used to obtain or save a value, rather than being a syntactic construct of the language. 2. PEP 343 allows for *controlled* manipulation of context variables, helping to prevent "duelling libraries" from changing state on each other. Also, a library can potentially ``snapshot()`` a desired state at startup, and use ``swap()`` to restore that state on re-entry. (And could even define a simple decorator to wrap its entry points to ensure this.) 3. The PEP author is not aware of any language that explicitly offers coroutine-scoped variables, but presumes that they can be modelled with monads or continuations in functional languages like Haskell. (And I only mention this to forestall the otherwise-inevitable response from fans of such techniques, pointing out that it's possible.) Reference Implementation ======================== The author has prototyped an implementation with somewhat fancier features than shown here, but prefers not to publish it until the basic features and choices of optional functionality have been discussed on Python-Dev. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: [View Less]

5 9

Re: [Python-Dev] Pre-PEP: Task-local variables
by Phillip J. Eby Oct. 21, 2005

Oct. 21, 2005

At 07:57 PM 10/20/2005 -0700, Guido van Rossum wrote: >(Sorry for the long post -- there just wasn't anything you said that I >felt could be left unquoted. :-) Wow. You've brought up an awful lot of stuff I want to respond to, about the nature of frameworks, AOP, Chandler, PEP 342, software deployment, etc. But I know you're busy, and the draft I was working on in reply to this has gotten simply huge and still unfinished, so I think I should just turn it all into a blog article on "… [View More]

1 0

Re: [Python-Dev] Coroutines, generators, function calling
by Gustavo J. A. M. Carneiro Oct. 20, 2005

Oct. 20, 2005

On Tue, 2005-10-18 at 09:07 -0400, Jim Jewett wrote: > Suppose now I want to move the window animation to a function, to > factorize the code: > > def animate(win, steps): > for y in steps: > win.move(0, y*20) > yield Timeout(0.1) > > def show_message(msg): > win = create_window(msg) > animate(win, xrange(10)) # slide down > yield Timeout(3) > animate(win, xrange(10, 0, -1)) # slide up > win.destroy() > > This … [View More]

4 5

A solution to the evils of static typing and interfaces?
by Simon Belak Oct. 20, 2005

Oct. 20, 2005

Hi, I was thinking why not have a separate file for all the proposed optional meta-information (in particular interfaces, static types)? Something along the lines of IDLs in CORBA (with pythonic syntax, of curse). This way most of the benefits are retained without "contaminating" the actual syntax (dare I be so pretentious to even hope making both sides happy?). For the sole purpose of illustration, let meta-files have extension .pym and linking to source-files be name based: parrot.py … [View More]

1 0

Re: [Python-Dev] Pre-PEP: Task-local variables
by Phillip J. Eby Oct. 20, 2005

Oct. 20, 2005

At 04:04 PM 10/20/2005 -0400, Jeremy Hylton wrote: >On 10/20/05, Guido van Rossum <guido(a)python.org> wrote: > > Whoa, folks! Can I ask the gentlemen to curb their enthusiasm? > > > > PEP 343 is still (back) on the drawing table, PEP 342 has barely been > > implemented (did it survive the AST-branch merge?), and already you > > are talking about adding more stuff. Please put on the brakes! > >Yes. PEP 342 survived the merge of the AST branch. I … [View More]

2 1

bool(iter([])) changed between 2.3 and 2.4
by Guido van Rossum Oct. 20, 2005

Oct. 20, 2005

I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None). In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items. I realize that this was a deliberate feature, and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2;… [View More]

6 10

list splicing
by Karl Chen Oct. 20, 2005

Oct. 20, 2005

Hi, has anybody considered adding something like this: a = [1, 2] [ 'x', *a, 'y'] as syntactic sugar for a = [1, 2] [ 'x' ] + a + [ 'y' ]. Notes: - This is a common operation - To me, the splicing form looks much better than the concatenation form - It can be implemented more efficiently than a bunch of list concatenations - This would be similar to the "apply" feature [ foo(*a) ]. The '*' operator is reminiscent of dereferencing operators in C-derived languages. … [View More]

6 6

enumerate with a start index
by Martin Blais Oct. 20, 2005

Oct. 20, 2005

Hi Just wondering, would anyone think of it as a good idea if the enumerate() builtin could accept a "start" argument? I've run across a few cases where this would have been useful. It seems generic enough too.

4 3

Early PEP draft (For Python 3000?)
by Jim Jewett Oct. 20, 2005

Oct. 20, 2005

I'll try to be more explicit; if Josiah and I are talking past each other, than the explanation was clearly not yet mature. (In http://mail.python.org/pipermail/python-dev/2005-October/057251.html) Eyal Lotem suggested: > Name: Attribute access for all namespaces ... > global x ; x = 1 > Replaced by: > module.x = 1 I responded: > Attribute access as an option would be nice, but might be slower. > Also note that one common use for a __dict__ is that you don't &… [View More]

2 1