...

...

import ast ast.literal_eval("{ 'key': 'val' }") {'key': 'val'} ast.literal_eval("{ ('key',): 'val' }") {('key',): 'val'} ast.literal_eval("{ frozenset({'key'}): 'val' }") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.3/ast.py", line 86, in literal_eval return _convert(node_or_string) File "/usr/lib/python3.3/ast.py", line 63, in _convert in zip(node.keys, node.values)) File "/usr/lib/python3.3/ast.py", line 62, in <genexpr> return dict((_convert(k), _convert(v)) for k, v File "/usr/lib/python3.3/ast.py", line 85, in _convert raise ValueError('malformed node or string: ' + repr(node)) ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450>

On Sat, Feb 2, 2013 at 1:34 AM, Stefan Behnel wrote:

Hua Lu, 02.02.2013 07:24:

...

This is similar to another proposal: http://mail.python.org/pipermail/python-3000/2008-January/011798.html

Anyway, I was using ast.literal_eval and attempted to use frozenset({...}) as a key in a dictionary, which failed, because frozenset isn't a literal

This has nothing to do with being a literal or not. The way you created your frozenset doesn't impact its behaviour.

Could you give an example of what's not working for you? Frozensets as dict keys work just fine for me:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

frozenset({1,2,3}) frozenset({1, 2, 3}) hash(frozenset({1,2,3})) -7699079583225461316 { frozenset({1,2,3}) : 1 } {frozenset({1, 2, 3}): 1}

Stefan

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

So, why exactly are you using ast.literal_eval() to evaluate a non-literal expression?

Stefan

PS: please reply without top-posting.

Sorry, I'm quite new to mailing lists. I want to be able to use frozensets in ast.literal_eval portions (especially as semantically correct keys for certain domains where tuples have too much structure). Writing an ast filter just to accommodate a builtin type feels way too hacky to me, and there are legitimate reasons for having frozenset literals exist, some of which are discussed in the link to that 2008 discussion. Also, there are serious hacks for eval, and after seeing some of those... ay caramba. Thanks for being patient with my newbishness.

On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote:

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation. Regards Antoine.

Perhaps, but we'd have to be careful with how we introduce those symbols: http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html You could possibly not allow attribute access but permit those symbols... is there an exploit possible with that much filtering? On Sat, Feb 2, 2013 at 12:30 PM, MRAB wrote:

On 2013-02-02 14:28, Antoine Pitrou wrote:

...

On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote:

...

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation.

Does it need a special notation as such?

Wouldn't an alternative be to support "frozenset(...)" where "..." is parseable by ast.literal_eval()?

The same thing could be done with some other built-in classes.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Sat, Feb 2, 2013 at 11:07 AM, MRAB wrote:

On 2013-02-02 18:46, Hua Lu wrote:

...

On Sat, Feb 2, 2013 at 12:30 PM, MRAB wrote:

...

On 2013-02-02 14:28, Antoine Pitrou wrote:

...

On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote:

...

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation.

Does it need a special notation as such?

Wouldn't an alternative be to support "frozenset(...)" where "..." is parseable by ast.literal_eval()?

The same thing could be done with some other built-in classes.

Perhaps, but we'd have to be careful with how we introduce those symbols:

http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html

You could possibly not allow attribute access but permit those symbols... is there an exploit possible with that much filtering?

I'm talking about hard-coding "frozenset(...)" and a few other built-in classes to return the appropriate object, so, for example, "set([0, 1, 2])" would be the same as "{0, 1, 2}".

You're proposing to do this just for literal_eval(), right? But how would you implement it? It seems it would require lots of special cases. Where would you stop? dict(key1=..., key2=...)? list({...})? -- --Guido van Rossum (python.org/~guido)

On 03/02/13 05:30, MRAB wrote:

On 2013-02-02 14:28, Antoine Pitrou wrote:

...

On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote:

...

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation.

Does it need a special notation as such?

Wouldn't an alternative be to support "frozenset(...)" where "..." is parseable by ast.literal_eval()?

The same thing could be done with some other built-in classes.

I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears. However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like: 5x^3 - 2x + log(1/y) + n!/√π I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time. In any case, I think something like that belongs in a third-party module, at least at first. -- Steven

On 2013-02-03 02:18, Steven D'Aprano wrote:

On 03/02/13 05:30, MRAB wrote:

...

On 2013-02-02 14:28, Antoine Pitrou wrote:

...

On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote:

...

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation.

Does it need a special notation as such?

Wouldn't an alternative be to support "frozenset(...)" where "..." is parseable by ast.literal_eval()?

The same thing could be done with some other built-in classes.

I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears.

It needn't go beyond a currently-supported literal passed as an argument to frozenset, etc, for a very limited number of built-in classes. A literal list or a literal set passed to frozenset would be treated as a literal frozenset.

However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like:

5x^3 - 2x + log(1/y) + n!/√π

I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time.

In any case, I think something like that belongs in a third-party module, at least at first.

On Sun, 3 Feb 2013 14:50:45 +1000 Nick Coghlan wrote:

This is why we need a PEP or a PyPI module. It's certainly not clear to me that special casing in ast.literal_eval (or a new "ast.limited_eval") is a superior solution to s{} and fs{} syntax for creating the empty set and frozen sets.

Well, it's superior because it doesn't need new syntax.

(And, as Raymond notes, there are other compile-time benefits in terms of constant-caching when it comes to dedicated syntax)

Which is a rather dubious benefit when it comes to a little-used datatype. If you want to avoid the cost of instantiating a frozenset in every loop iteration, just hoist it manually out of the loop.

On the other hand, a "limited_eval" style solution might be easier to extend to other builtins like range, reversed and enum, as well as to container comprehensions and generator expressions. It also has the virtue of being possible to write as a PyPI module, and made available for *current* Python versions, rather than only being available in Python 3.4+.

Agreed. Regards Antoine.

On Sun, 3 Feb 2013 00:40:47 +1000 Nick Coghlan wrote:

On 3 Feb 2013 00:21, "Steven D'Aprano" wrote:

...

On 03/02/13 00:06, Stefan Behnel wrote:

...

Hua Lu, 02.02.2013 08:38:

...

...

...

> > import ast > ast.literal_eval("{ 'key': 'val' }")

{'key': 'val'}

...

...

> > ast.literal_eval("{ ('key',): 'val' }")

{('key',): 'val'}

...

...

> > ast.literal_eval("{ frozenset({'key'}): 'val' }")

Traceback (most recent call last): File "<stdin>", line 1, in<module> File "/usr/lib/python3.3/ast.py", line 86, in literal_eval return _convert(node_or_string) File "/usr/lib/python3.3/ast.py", line 63, in _convert in zip(node.keys, node.values)) File "/usr/lib/python3.3/ast.py", line 62, in<genexpr> return dict((_convert(k), _convert(v)) for k, v File "/usr/lib/python3.3/ast.py", line 85, in _convert raise ValueError('malformed node or string: ' + repr(node)) ValueError: malformed node or string:<_ast.Call object at

0x7f865a8c1450>

...

...

So, why exactly are you using ast.literal_eval() to evaluate a

non-literal

...

expression?

Stefan, you did ask Hua Lu to show an example of what isn't working. It's just a demonstration of what doesn't work -- you can't create a frozenset using ast.literal_eval.

I think Hua Lu's original post made it quite clear. He wishes there to be a frozenset literal, because currently there is no way to have ast.literal_eval evaluate something containing a frozenset. Because there is no frozenset literal.

I think Raymond Hettinger's proposal back in 2008:

http://mail.python.org/pipermail/python-3000/2008-January/011798.html

and the following thread is worth reading. Guido even pronounced his agreement:

http://mail.python.org/pipermail/python-3000/2008-January/011814.html

but then changed his mind (as did Raymond). So the status quo remains.

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

To clarify Guido's comment in that post, I'm fairly sure it *can* be made to work, it just won' t be the same way that string prefixes work (because the contents of dict and set displays are not opaque to the compiler the way string contents are).

The hypothetical "What if we want to allow expr{} as a general construct?" objection needs to be balanced against the immediate value of a more expressive language subset for use in ast.literal_eval().

I've just tried: ast.literal_eval() works just fine on a normal set:

...

...

ast.literal_eval("{1,2,3}") {1, 2, 3}

So I'd like to know why people find it important to build a frozenset rather than a normal set. The situation of wanting to use sets as hash keys is very rare in my experience. Regards Antoine.

On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote:

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I was going to post about not being sure what the objection is (if it's multiple tokens, let it be multiple tokens - the contents are multiple tokens anyway - and saying it would block a future syntax extension doesn't seem like a reasonable objection to a proposed syntax extension), but I had a new idea so I'll post that instead: { as frozenset { ... } } The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the thing after it is a keyword in that context (and only that context, otherwise "frozenset" remains an identifier naming an ordinary builtin) and specifies what kind of literal the following sequence is. You could also extend it to alternate forms for some other builtin types - for example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5". Or... { as set { } }

Since the _real_ issue seems to be wanting an easy way to construct an OrderedDict, I thought the earlier discussion about frozenset literals might be relevant. On Wed, Feb 13, 2013, at 17:02, random832@fastmail.us wrote:

On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote:

...

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I was going to post about not being sure what the objection is (if it's multiple tokens, let it be multiple tokens - the contents are multiple tokens anyway - and saying it would block a future syntax extension doesn't seem like a reasonable objection to a proposed syntax extension), but I had a new idea so I'll post that instead:

{ as frozenset { ... } }

The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the thing after it is a keyword in that context (and only that context, otherwise "frozenset" remains an identifier naming an ordinary builtin) and specifies what kind of literal the following sequence is. You could also extend it to alternate forms for some other builtin types - for example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5". Or... { as set { } } _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Random832

It looks like we can special-case usage of set literal as a key for a dict or a member for another set and create a frozenset constant instead.

So that adict[{1, 2, 3}] should be interpreted as adict[frozenset([1, 2, 3])] and { {'b', 'a', 'r'}, 'foo' } as { frozenset('bar'), 'foo' }

This could interfere with the behavior of types overriding the index operator. Also, it's a special case.

This will provide a minimal change to the interpreter while making it possible to use any literal-parsing with frozensets while keeping method calls out of literal_eval.

Why shouldn't we change the interpreter when it could make the language better? Honestly, I can't think of an element which will require special syntax like symbol{...}. Furthermore, it could be really confusable with current syntax. Example: for fn { bar(foo) for foo in baz }: What if bar was a higher order function and that syntax meant something like "make a generator applying each thing in the set to an original fn before introducing fn in the scope" when you really meant `for fn in {...}`? Python 3 has already made some calls more explicitly a function (e.g. print) so having that as syntactic sugar for that would also be weird. There doesn't seem to be any _use_ for syntax like symbol{...}, so I feel for example s{...}, f{...}, and d{...} would be a natural extension to Python insofar as b'...' and u'...' are in the language, _even though_ sets and maps feel more general mathematically speaking (so the need for special use syntax _may_ be inappropriate (but people write math in languages ;)).

On Sun, 3 Feb 2013 00:28:55 +1000 Nick Coghlan wrote:

FWIW, I could personally tolerate the introduction of s{} and fs{} literals. We'd just declare the "s" prefix optional for non-empty sets to match the current rules.

Encouraging the use of ast.literal_eval() over the security nightmare that is eval() would be more than enough justification for me. (As a syntax change, the idea would still need a PEP, though)

If it is enough of a justification, then why not literal support for more useful datatypes (decimal, datetime...) rather than frozenset? Regards Antoine.

On Sun, 3 Feb 2013 00:56:57 +1000 Nick Coghlan wrote:

On 3 Feb 2013 00:39, "Antoine Pitrou" wrote:

...

On Sun, 3 Feb 2013 00:28:55 +1000 Nick Coghlan wrote:

...

FWIW, I could personally tolerate the introduction of s{} and fs{} literals. We'd just declare the "s" prefix optional for non-empty sets

...

...

match the current rules.

Encouraging the use of ast.literal_eval() over the security nightmare

to that

...

...

is eval() would be more than enough justification for me. (As a syntax change, the idea would still need a PEP, though)

If it is enough of a justification, then why not literal support for more useful datatypes (decimal, datetime...) rather than frozenset?

The difference is that decimal and datetime already have safe "from string" conversion operations. The empty set and frozen sets do not.

You can call set() or frozenset() on a tuple literal, if you know you want to end up with a set of frozenset. Regards Antoine.