[Python-3000] List & set comprehensions patch

[Georg Brandl]

Nick Coghlan schrieb:
> Georg and I have been working on the implementation of list 
> comprehensions which don't leak their iteration variables, along with 
> the implementation of set comprehensions. The latest patch can be found 
> as SF patch #1660500 [1]. The file new-set-comps.diff is the combined 
> patch which implements both features, and unifies handling of the 
> different kinds of comprehension. In an effort to improve readability, 
> the patch also converts the sets in symtable.c to be actual PySet 
> objects, rather than PyDict objects with None keys and tries to reduce 
> the number of different meanings assigned to the term 'scope'.
> 
> One of the comments made on Georg's initial attempt at implementing 
> these features was that it would be nice to avoid the function call 
> overhead in the listcomp & setcomp case (as it appears at first glance 
> that the internal scope can be temporary). I tried to do that and 
> essentially failed outright - working through symtable.c and compile.c, 
> I found that dealing with the scoping issues created by the possibility 
> of nested genexps, lambdas and list or set comprehensions would pretty 
> much require reimplementing all of the scoping rules that functions 
> already provide.

I have to thank you for digging through all that code and cases, which I always
dreaded ;)

> There are also a couple of tests we had to disable - one in test_dis, 
> one in test_grammar. Suggestions on how to reinstate those (or agreement 
> that it is OK to get rid of them) would be appreciated.

The one in test_grammar is certainly okay, since it uses syntax which
was agreed upon to be invalid in Py3k.

> The PySet update code in symtable.c currently uses PyNumber_InplaceOr 
> with a subsequent call to Py_DECREF to counter the implicit call to 
> Py_INCREF. Should this be changed to use PyObject_CallMethod to invoke 
> the Python level update method?

Why not add a C-level PySet_Update API?

> There are also two backwards compatibility problems which came up:
> 
>    - code which explicitly deleted the listcomp variable started 
> throwing NameErrors. Several tweaks were needed in the standard library 
> to fix this.

This may be hard to catch with the 2to3 tool, but perhaps a certain heuristics
(del statement after a statement involving a list comprehension) could apply.
Furthermore, I don't think that "trick" is mostly used on module-level to
avoid cluttering up the namespace, so a codebase which really uses its modules
should see early failures.

>    - only the outermost iterator expression is evaluated in the scope 
> containing the comprehension (just like generator expressions). This 
> means that the inner expressions can no longer see class variables and 
> values in explicit locals() dictionaries provided to exec & friends. 
> This didn't actually cause any problems in the standard library - I only 
> note it because my initial implementation mistakenly evaluated the 
> outermost iterator in the new scope, which *did* cause severe problems 
> along these lines.

It must be noted that this was always the case for generator expressions, so
if a LC should only be syntactic sugar for list(genexp) as written in the PEP,
this is even correct behavior.

cheers,
Georg