[Python-Dev] PEP-xxx: Unification of for statement and list-comp syntax

[Heiko Wundram]

Hi all!

The following PEP tries to make the case for a slight unification of for 
statement and list comprehension syntax.

Comments appreciated, including on the sample implementation.

===
PEP: xxx
Title: Unification of for-statement and list-comprehension syntax
Version: $Revision$
Last-Modified: $Date$
Author: Heiko Wundram <me at modelnine.org>
Status: Active
Type: Standards Track
Content-Type: text/plain
Created: 21-May-2006
Post-History: 21-May-2006 17:00 GMT+0200


Abstract

    When list comprehensions were introduced, they added the ability
    to add conditions which are tested before the expression which is
    associated with the list comprehension is evaluated. This is
    often used to create new lists which consist only of those items
    of the original list which match the specified condition(s). For
    example:

      [node for node in tree if node.haschildren()]

    will create a new list which only contains those items of the
    original list (tree) whose items match the havechildren()
    condition. Generator expressions work similarily.

    With a standard for-loop, this corresponds to adding a continue
    statement testing for the negated expression at the beginning of
    the loop body.

    As I've noticed that I find myself typing the latter quite often
    in code I write, it would only be sensible to add the corresponding
    syntax for the for statement:

      for node in tree if node.haschildren():
          <do something with node>

    as syntactic sugar for:

      for node in tree:
          if not node.haschildren():
              continue
          <do something with node>

    There are several other methods (including generator-expressions or
    list-comprehensions, the itertools module, or the builtin filter
    function) to achieve this same goal, but all of them make the code
    longer and harder to understand and/or require more memory, because
    of the generation of an intermediate list.


Implementation details

    The implementation of this feature requires changes to the Python
    grammar, to allow for a variable number of 'if'-expressions before
    the colon of a 'for'-statement:

      for_stmt: 'for' exprlist 'in' testlist_safe ('if' old_test)* ':'
                    suite ['else' ':' suite]

    This change would replace testlist with testlist_safe as the
    'in'-expression of a for statement, in line with the definition of
    list comprehensions in the Python grammar.

    Each of the 'if'-expressions is evaluated in turn (if present), until
    one is found False, in which case the 'for'-statement restarts at the
    next item from the generator of the 'in'-expression immediately
    (the tests are thus short-circuting), or until all are found to be
    True (or there are no tests), in which case the suite body is executed.
    The behaviour of the 'else'-suite is unchanged.

    The intermediate code that is generated is modelled after the
    byte-code that is generated for list comprehensions:

      def f():
          for x in range(10) if x == 1:
              print x

    would generate:

      2           0 SETUP_LOOP              42 (to 45)
                  3 LOAD_GLOBAL              0 (range)
                  6 LOAD_CONST               1 (10)
                  9 CALL_FUNCTION            1
                 12 GET_ITER
            >>   13 FOR_ITER                28 (to 44)
                 16 STORE_FAST               0 (x)
                 19 LOAD_FAST                0 (x)
                 22 LOAD_CONST               2 (1)
                 25 COMPARE_OP               2 (==)
                 28 JUMP_IF_FALSE            9 (to 40)
                 31 POP_TOP

      3          32 LOAD_FAST                0 (x)
                 35 PRINT_ITEM
                 36 PRINT_NEWLINE
                 37 JUMP_ABSOLUTE           13
            >>   40 POP_TOP
                 41 JUMP_ABSOLUTE           13
            >>   44 POP_BLOCK
            >>   45 LOAD_CONST               0 (None)
                 48 RETURN_VALUE

    where all tests are inserted immediately at the beginning of the
    loop body, and jump to a new block if found to be false which pops
    the comparision from the stack and jumps back to the beginning of
    the loop to fetch the next item.

Implementation issues

    The changes are backwards-compatible, as they don't change the
    default behaviour of the 'for'-loop. Also, as the changes that
    this PEP proposes don't change the byte-code structure of the
    interpreter, old byte-code continues to run on Python with this
    addition unchanged.


Implementation

    A sample implementation (with updates to the grammar documentation
    and a small test case) is available at:

      
http://sourceforge.net/tracker/index.php?func=detail&aid=1492509&group_id=5470&atid=305470


Copyright

    This document has been placed in the public domain.
===

--- Heiko.