[Python-3000] List & set comprehensions patch
g.brandl at gmx.net
Tue Mar 6 18:38:27 CET 2007
Nick Coghlan schrieb:
> Georg and I have been working on the implementation of list
> comprehensions which don't leak their iteration variables, along with
> the implementation of set comprehensions. The latest patch can be found
> as SF patch #1660500 . The file new-set-comps.diff is the combined
> patch which implements both features, and unifies handling of the
> different kinds of comprehension. In an effort to improve readability,
> the patch also converts the sets in symtable.c to be actual PySet
> objects, rather than PyDict objects with None keys and tries to reduce
> the number of different meanings assigned to the term 'scope'.
> One of the comments made on Georg's initial attempt at implementing
> these features was that it would be nice to avoid the function call
> overhead in the listcomp & setcomp case (as it appears at first glance
> that the internal scope can be temporary). I tried to do that and
> essentially failed outright - working through symtable.c and compile.c,
> I found that dealing with the scoping issues created by the possibility
> of nested genexps, lambdas and list or set comprehensions would pretty
> much require reimplementing all of the scoping rules that functions
> already provide.
I have to thank you for digging through all that code and cases, which I always
> There are also a couple of tests we had to disable - one in test_dis,
> one in test_grammar. Suggestions on how to reinstate those (or agreement
> that it is OK to get rid of them) would be appreciated.
The one in test_grammar is certainly okay, since it uses syntax which
was agreed upon to be invalid in Py3k.
> The PySet update code in symtable.c currently uses PyNumber_InplaceOr
> with a subsequent call to Py_DECREF to counter the implicit call to
> Py_INCREF. Should this be changed to use PyObject_CallMethod to invoke
> the Python level update method?
Why not add a C-level PySet_Update API?
> There are also two backwards compatibility problems which came up:
> - code which explicitly deleted the listcomp variable started
> throwing NameErrors. Several tweaks were needed in the standard library
> to fix this.
This may be hard to catch with the 2to3 tool, but perhaps a certain heuristics
(del statement after a statement involving a list comprehension) could apply.
Furthermore, I don't think that "trick" is mostly used on module-level to
avoid cluttering up the namespace, so a codebase which really uses its modules
should see early failures.
> - only the outermost iterator expression is evaluated in the scope
> containing the comprehension (just like generator expressions). This
> means that the inner expressions can no longer see class variables and
> values in explicit locals() dictionaries provided to exec & friends.
> This didn't actually cause any problems in the standard library - I only
> note it because my initial implementation mistakenly evaluated the
> outermost iterator in the new scope, which *did* cause severe problems
> along these lines.
It must be noted that this was always the case for generator expressions, so
if a LC should only be syntactic sugar for list(genexp) as written in the PEP,
this is even correct behavior.
More information about the Python-3000