[Python-ideas] Temporary variables in comprehensions

[Steven D'Aprano]

On Sun, Feb 18, 2018 at 12:23:28AM +0800, fhsxfhsx wrote:
> Thank you Paul, what you said is enlightening and I agree on most part of it.
> 
> I'll propose two candidate syntaxs.
> 1. `with ... as ...`
>     This syntax is more paralles as there would be `for` and `with` 
>     clause as well as `for` and `with` statement. However, the 
>     existing `with` statement is semantically different from this one, 
>     although similar.

I don't think they are even a little bit similar. The existing `with` 
statement is for controlling cleanup code (using a context manager). 
This proposal doesn't have anything to do with context managers or 
cleanup code. It's just a different way to spell "name = value" inside 
comprehensions.


> 2. `for ... is ...`
>     This syntax is more uniform as the existing `for` clause make an 
>     iterator which is a special kind of variable. However, I'm afraid 
>     this syntax might be confused with `for ... in ...` as they differ 
>     only on one letter.

Indeed.

And frankly, treated as English grammar, "for value is name" doesn't 
make sense and is horribly ugly to my eyes:

    result = [x for value in sequence for value+1 is x]


> And here is an example which appears quite often in my code where I 
> think a new syntax can help a lot: Suppose I have an list of goods 
> showing by their ids in database, and I need to transform the ids into 
> json including information from two tables, `Goods` and 
> `GoodsCategory`, where the first table recording `id`, `name` and 
> `category_id` indicating which category the goods belongs to, the 
> second table recording `id`, `name` and `type` to the categories.
>
> With the new syntax, I can write
> [
>   {
>     'id': goods.id,
>     'name': goods.name,
>     'category': gc.name,
>     'category_type': gc.type,
>   }
>   for goods_id in goods_id_list
>   for goods is Goods.get_by_id(goods_id)
>   for gc is GoodsCategory.get_by_id(goods.category_id)
> ]
> 
> And I cannot think of any good solutions as this one without it.

I can of a few, starting with the most simple: write a helper function. 
Not every problem needs to be solved with new syntax.

def dict_from_id(goods_id):
    goods = Goods.get_by_id(goods_id)
    gc = GoodsCategory.get_by_id(goods.category_id)
    return {'id': goods.id,
            'name': goods.name,
            'category': gc.name,
            'category_type': gc.type
            }

result = [dict_from_id(goods_id) for goods_id in goods_id_list]

That's much nicer to read, you can document and test the dict_from_id() 
function, no new systax is required, it is easy to refactor, and I very 
much doubt that adding one extra function call is going to be a 
significant slowdown compared to the cost of two calls to get_by_id() 
methods and constructing a dict.

(And if as you add more fields to the dict, the overhead of the function 
call becomes an even smaller proportion.)

Or you can make this a method of the goods object, which is arguably a 
better OO design. Let the goods object be responsible for creating the 
dict.

result = [Goods.get_by_id(goods_id).make_dict() for goods_id in goods_id_list]
# or if you prefer
result = [goods.make_dict() for goods in map(Goods.get_by_id, goods_id_list)]

Here is a third solution: use a for-loop iterating over a single-item 
tuple to get the effect of a local assignment to a temporary variable:

result = [{ 
           # dict display truncated for brevity...
           } 
           for goods_id in goods_id_list
           for goods in (Goods.get_by_id(goods_id),)
           for gc in (GoodsCategory.get_by_id(goods.category_id),)
           ]

If you don't like the look of single-item tuples (foo,) you can use 
single-item lists instead [x] but they are a tiny bit slower to create.

Serhiy has suggested that the interpreter can optimize the single-item 
loop to make it as fast as a bare assignment:

https://bugs.python.org/issue32856

I think this is a neat trick, although Yuri thinks it is an ugly hack 
and doesn't want to encourage it. Neat or ugly, I think it is better 
than "for value is name".


-- 
Steve