From ben.bob at gmail.com  Sun Aug 29 15:42:47 2010
From: ben.bob at gmail.com (Bo Peng)
Date: Sun, 29 Aug 2010 06:42:47 -0700
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
Message-ID: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>

Dear all,

This might be the wrong mailinglist to ask, but I bumped into a
strange problem when I ported my code from Python 2.7 to 3.1. More
specifically, my code involves the evaluation of expressions in a lot
of local namespaces (dictionaries). For example, the following code
evaluates expressions in a dictionary dd under Python 2.7:

Python 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>> print(eval('a[1]', {}, dd))
0.1
>>> print(eval("[x for x in a.keys()]", {}, dd))
[1, 2]
>>> print(eval("[a[x] for x in a.keys()]", dd, dd))
[0.1, 0.2]
>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
[0.1, 0.2]

However, the last statement fails under Python 3.1

Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>> print(eval('a[1]', {}, dd))
0.1
>>> print(eval("[x for x in a.keys()]", {}, dd))
[1, 2]
>>> print(eval("[a[x] for x in a.keys()]", dd, dd))
[0.1, 0.2]
>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined
>>>

Because a is in clearly accessible from the passed dictionary, I have
no idea why it has to be defined in the global dictionary. Does anyone
know what is going on here? How can I change make it work with Python
3?

Many thanks in advance.
Bo

From benjamin at python.org  Sun Aug 29 15:54:02 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 29 Aug 2010 08:54:02 -0500
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
Message-ID: <AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>

2010/8/29 Bo Peng <ben.bob at gmail.com>:
> Dear all,
>
> This might be the wrong mailinglist to ask, but I bumped into a
> strange problem when I ported my code from Python 2.7 to 3.1. More
> specifically, my code involves the evaluation of expressions in a lot
> of local namespaces (dictionaries). For example, the following code
> evaluates expressions in a dictionary dd under Python 2.7:
>
> Python 2.7 (r27:82525, Jul ?4 2010, 09:01:59) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>>> print(eval('a[1]', {}, dd))
> 0.1
>>>> print(eval("[x for x in a.keys()]", {}, dd))
> [1, 2]
>>>> print(eval("[a[x] for x in a.keys()]", dd, dd))
> [0.1, 0.2]
>>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
> [0.1, 0.2]
>
> However, the last statement fails under Python 3.1
>
> Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>>> print(eval('a[1]', {}, dd))
> 0.1
>>>> print(eval("[x for x in a.keys()]", {}, dd))
> [1, 2]
>>>> print(eval("[a[x] for x in a.keys()]", dd, dd))
> [0.1, 0.2]
>>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> ?File "<string>", line 1, in <module>
> ?File "<string>", line 1, in <listcomp>
> NameError: global name 'a' is not defined
>>>>
>
> Because a is in clearly accessible from the passed dictionary, I have
> no idea why it has to be defined in the global dictionary. Does anyone
> know what is going on here? How can I change make it work with Python
> 3?

List comprehensions are now properly scoped, so having one in the
global scope, like you do in this case, will lookup the names in the
global namespace.



-- 
Regards,
Benjamin

From g.brandl at gmx.net  Sun Aug 29 15:48:12 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 29 Aug 2010 15:48:12 +0200
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
Message-ID: <i5dokp$h7k$1@dough.gmane.org>

Am 29.08.2010 15:42, schrieb Bo Peng:

> However, the last statement fails under Python 3.1
> 
> Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>>> print(eval('a[1]', {}, dd))
> 0.1
>>>> print(eval("[x for x in a.keys()]", {}, dd))
> [1, 2]
>>>> print(eval("[a[x] for x in a.keys()]", dd, dd))
> [0.1, 0.2]
>>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<string>", line 1, in <module>
>   File "<string>", line 1, in <listcomp>
> NameError: global name 'a' is not defined
>>>>
> 
> Because a is in clearly accessible from the passed dictionary, I have
> no idea why it has to be defined in the global dictionary. Does anyone
> know what is going on here? How can I change make it work with Python
> 3?

In Python 3, all comprehensions are executed in their own function scope,
like generator expressions in Python 2.  (If you used ``list(a[x] for x
in a.keys())``, it would fail in Python 2 too.)

Therefore, and because the compilation of the expression to eval() can
not generate closures (because it does not know in advance what locals
are available), it has to look up "a" as a global.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ben.bob at gmail.com  Sun Aug 29 16:14:32 2010
From: ben.bob at gmail.com (Bo Peng)
Date: Sun, 29 Aug 2010 09:14:32 -0500
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
Message-ID: <AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>

>>>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> ?File "<string>", line 1, in <module>
>> ?File "<string>", line 1, in <listcomp>
>> NameError: global name 'a' is not defined

> List comprehensions are now properly scoped, so having one in the
> global scope, like you do in this case, will lookup the names in the
> global namespace.

Could you please provide more details (like a link to documentation)
what is going on here? From what I can see, I am evaluating an
expression in a dictionary with 'a' defined (and x is a iterator
variable), why does simuPOP have to look for it in the global
namespace? I mean, if 'a[1]', 'a[2]', '[a[1], a[2]]' are all valid
expressions, why not '[a[x] for x in [1,2]]'? Should not "[a[1],
a[2]]" always return the same result as "[a[x] for x in [1,2]]"? The
fact that

dd = {'a': {1:0.1, 2:0.2}}
dd1 = {'a': {1:1.1, 2:1.2}}
eval('a[1], a[2], list(a.keys())', dd1, dd)

returns 0.1, 0.2, [1, 2] is REALLY puzzling.

Bo

From ben.bob at gmail.com  Sun Aug 29 16:17:59 2010
From: ben.bob at gmail.com (Bo Peng)
Date: Sun, 29 Aug 2010 09:17:59 -0500
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
Message-ID: <AANLkTineq4ig0kENz5meTC5aABD3X7zY3xkogCLu6gh_@mail.gmail.com>

> dd = {'a': {1:0.1, 2:0.2}}
> dd1 = {'a': {1:1.1, 2:1.2}}
> eval('a[1], a[2], list(a.keys())', dd1, dd)

Sorry, I actually meant:

'a[1], a[2], [a[x] for x in a.keys()]'

>
> returns 0.1, 0.2, [1, 2] is REALLY puzzling.

Bo

From g.brandl at gmx.net  Sun Aug 29 16:46:07 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 29 Aug 2010 16:46:07 +0200
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
Message-ID: <i5ds1c$tro$1@dough.gmane.org>

Am 29.08.2010 16:14, schrieb Bo Peng:
>>>>>> dd = {'a': {1: 0.1, 2: 0.2}}
>>>>>> print(eval("[a[x] for x in a.keys()]", {}, dd))
>>> Traceback (most recent call last):
>>>  File "<stdin>", line 1, in <module>
>>>  File "<string>", line 1, in <module>
>>>  File "<string>", line 1, in <listcomp>
>>> NameError: global name 'a' is not defined
> 
>> List comprehensions are now properly scoped, so having one in the
>> global scope, like you do in this case, will lookup the names in the
>> global namespace.
> 
> Could you please provide more details (like a link to documentation)
> what is going on here? From what I can see, I am evaluating an
> expression in a dictionary with 'a' defined (and x is a iterator
> variable), why does simuPOP have to look for it in the global
> namespace? I mean, if 'a[1]', 'a[2]', '[a[1], a[2]]' are all valid
> expressions, why not '[a[x] for x in [1,2]]'? Should not "[a[1],
> a[2]]" always return the same result as "[a[x] for x in [1,2]]"?

No.  As I wrote in my reply, try executing

list(a[x] for x in [1,2])

in Python 2, and you will see that this exhibits the same behavior as
the list comprehension does in Python 3.

The difference between simple list literals and a comprehension is that
the comprehension can bind new names.  List comprehensions in Python 2
"leaked" these names into the surrounding scope, which was fixed by
unifying their implementation with generator expressions' in Python 3.

Note that this specific issue only appears because you're using eval().
For static code, your access to "a" outside of the comprehension's scope
will be recognized and a proper closure generated.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ben.bob at gmail.com  Sun Aug 29 17:16:35 2010
From: ben.bob at gmail.com (Bo Peng)
Date: Sun, 29 Aug 2010 10:16:35 -0500
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <i5ds1c$tro$1@dough.gmane.org>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
	<i5ds1c$tro$1@dough.gmane.org>
Message-ID: <AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>

> Note that this specific issue only appears because you're using eval().
> For static code, your access to "a" outside of the comprehension's scope
> will be recognized and a proper closure generated.

Thank you very much for your explanation and now I have a real problem
to fix for my Python extension module. More specifically, I call
"eval(expression, module_dict, user_dict)" to evaluate expressions in
user_dict. However, to evaluate expressions such as "[a[x] for x in
a]", I would need "eval(expression, user_dict, user_dict)" to make 'a'
available in globals. But then I do not have access to names in
module_dict, and simple expressions such as 'len(a)' would fail
because I am doing all these from the C/C++ level and there is no
__builtins__ in user_dict.

Happy python-porting, :-)

Bo

From g.brandl at gmx.net  Sun Aug 29 18:20:35 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 29 Aug 2010 18:20:35 +0200
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>	<i5ds1c$tro$1@dough.gmane.org>
	<AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>
Message-ID: <i5e1ig$ija$1@dough.gmane.org>

Am 29.08.2010 17:16, schrieb Bo Peng:
>> Note that this specific issue only appears because you're using eval().
>> For static code, your access to "a" outside of the comprehension's scope
>> will be recognized and a proper closure generated.
> 
> Thank you very much for your explanation and now I have a real problem
> to fix for my Python extension module. More specifically, I call
> "eval(expression, module_dict, user_dict)" to evaluate expressions in
> user_dict. However, to evaluate expressions such as "[a[x] for x in
> a]", I would need "eval(expression, user_dict, user_dict)" to make 'a'
> available in globals. But then I do not have access to names in
> module_dict, and simple expressions such as 'len(a)' would fail
> because I am doing all these from the C/C++ level and there is no
> __builtins__ in user_dict.

Surely you can do

namespace = module_dict.copy()
namespace.update(user_dict)
eval(..., namespace)

?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ben.bob at gmail.com  Sun Aug 29 18:49:25 2010
From: ben.bob at gmail.com (Bo Peng)
Date: Sun, 29 Aug 2010 11:49:25 -0500
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <i5e1ig$ija$1@dough.gmane.org>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
	<i5ds1c$tro$1@dough.gmane.org>
	<AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>
	<i5e1ig$ija$1@dough.gmane.org>
Message-ID: <AANLkTimqQuTCje6fbWH3+O3fA7jg1x-HVjocxM0mk1z3@mail.gmail.com>

> Surely you can do
>
> namespace = module_dict.copy()
> namespace.update(user_dict)
> eval(..., namespace)

I know, but this would be really slow because a lot of such
evaluations are needed. I think what I will do is to set __builtins__
to user_dict and use eval(expression, user_dict, user_dict) to
evaluate the expressions. Because my expressions rarely involve
variables in module_dict (except for __builtins__), restricting the
expressions to user_dict might be acceptable. If this solution is not
enough, I can still use

try:
    eval(expression, user_dict, user_dict)
except NameError:
    get namespace like what you suggested
    eval(expression, namespace, user_dict)

to handle these rare situations.

Thank you very much for your help,
Bo

From jhamel at cyrus-computer.de  Sun Aug 29 17:36:34 2010
From: jhamel at cyrus-computer.de (=?UTF-8?B?SsO8cmdlbg==?= Hamel)
Date: Sun, 29 Aug 2010 17:36:34 +0200
Subject: [Python-porting] Strange behavior of eval() in Python 3.1
In-Reply-To: <AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>
References: <AANLkTimG33Jc2txEtHjjJ983fZa8fTcQEpZLX2TpJV30@mail.gmail.com>
	<AANLkTimcuFBDk0gOoPLWS5WfhUEV2qBsXhoVT2ewZmVA@mail.gmail.com>
	<AANLkTi=MRpweS-pzd9rmO+PFdXhFGK8pCG0SyqqkamY5@mail.gmail.com>
	<i5ds1c$tro$1@dough.gmane.org>
	<AANLkTi=wbYvms7heSa-f33CgJRW++jeoBuC-vhXTeb9F@mail.gmail.com>
Message-ID: <20100829173634.1e5b72f2@x3>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 29 Aug 2010 10:16:35 -0500
Bo Peng <ben.bob at gmail.com> wrote:

> Happy python-porting, :-)
Hi you, 
I thought since 2 years about porting and i gave up. Do it like me: 
I DO NOT PORTING TO 3.xxx

I hope, that the next 10 years the most distris have a Python 2.7 and then it gives perhaps a Python 4.0, that compatible is to 2.7 and 3.x .

ok, really, I will wait some years, it is to much work for me to port and so I hope really, that they found a solution.

And, to all at this lists, is there an Internet site, that show me the porting status of some important python modules:
pygtk, twisted, GeoIP, aiml, gtksourceview, pil, ... and so on ?

best regards
J?rgen

- -- 
Cyrus-Computer GmbH Linux Server Support J?rgen Hamel
Cuon - Warenwirtschaft mit Linux  http://www.cuon.org
Twitter: cuonOne    Jabber: jhamel at cuonsim2.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkx6foIACgkQabQFsT6uaQCwIwCeOtgxLmqfhyDbKFCbwNEtK+XE
bkoAnRDzjGTrwnvIrIo6UMMtbnSb2CRF
=b/Ko
-----END PGP SIGNATURE-----