[docs] [issue31778] ast.literal_eval supports non-literals in Python 3
report at bugs.python.org
Sat Oct 14 02:08:21 EDT 2017
David Bieber <dbieber at google.com> added the comment:
> Rolling back previous enhancements would break existing code.
I sympathize completely with the need to maintain backward compatibility. And if this is the reason that this issue gets treated only as a documentation issue, rather than as a behavior issue, I can appreciate that.
If that is the case and literal_eval is not going to only evaluate literals, then for my use case I'll need a way to determine from a string whether it represents a literal. I can implement this myself using ast.parse and walking the resulting tree, looking for non-literal AST nodes. Would such an "is_literal" function would be more appropriate in the ast module than as a one-off function in Python Fire?
> The key promise that literal_eval makes is that it will not permit arbitrary code execution.
I disagree that this is the only key promise, and here's my reasoning. The docstring has two sentences, and each one makes a promise:
1. "Safely evaluate an expression node or a string containing a Python expression."
2. "The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None."
(1) says that evaluation is safe -- this is the key promise that you reference. (2) is also a promise though, that only certain types are allowed. While one could argue that the behavior of the function is not specified for inputs violating that criteria, I think the clear correct thing to do is to raise a ValueError if the value doesn't meet the criteria. This is what was done in Python 2, where the docstring for literal_eval is these same two sentences (modulo the inclusion of bytes and sets). It's my opinion that Python 2's behavior better matches the docstring as well as the behavior implied by the function's name.
# Additional observations
1. Python 2 _does_ support parsing complex literals, but does not support treating e.g. '1+1' as a literal.
ast.literal_eval('1+1j') # works in both Python 2 and Python 3
ast.literal_eval('1j+1') # raises a ValueError in Python 2, returns 1+1j in Python 3
ast.literal_eval('1+1') # raises a ValueError in Python 2, returns 2 in Python 3
2. Python 3 supports parsing addition and subtraction at any level of nesting.
ast.literal_eval('(1, (0, 1+1+1))') # raises a ValueError in Python 2, returns (1, (0, 3)) in Python 3
In my opinion, Python 2's behavior is correct in these situations since it matches the documentation and only evals literals as defined in the documentation.
The relevant code in Python 2.7.3 is [here](https://github.com/enthought/Python-2.7.3/blob/69fe0ffd2d85b4002cacae1f28ef2eb0f25e16ae/Lib/ast.py#L67). It explicitly allows NUM +/- COMPLEX, but not even COMPLEX +/- NUM. The corresponding code for Python 3 is [here](https://github.com/python/cpython/blob/ef611c96eab0ab667ebb43fdf429b319f6d99890/Lib/ast.py#L76). You'll notice it supports adding and subtracting arbitrary numeric types (int, float, complex).
Thanks for your replies and for hearing me out on this issue.
Python tracker <report at bugs.python.org>
More information about the docs