[New-bugs-announce] [issue31778] ast.literal_eval supports non-literals in Python 3

David Bieber report at bugs.python.org
Thu Oct 12 18:31:18 EDT 2017

New submission from David Bieber <dbieber at google.com>:

# Overview
ast.literal_eval supports some non-literals in Python 3.
The behavior of this function runs contrary to the documented behavior.

# The Issue
The [documentation](https://docs.python.org/3/library/ast.html#ast.literal_eval) says of the function "It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing."

However, literal_eval is capable of evaluating expressions with certain operators, particular the operators "+" and "-".

As has been explained previously, the reason for this is to support "complex" literals such as 2+3j. However, this has unintended consequences which I believe to be indicative of a bug. Examples of the unintended consequences include `ast.literal_eval('1+1') == 2` `ast.literal_eval('2017-10-10') == 1997`. I would expect each of these calls to literal_eval to throw a ValueError, as the input string is not a literal. Instead, literal_eval successfully evaluates the input string, in the latter case giving an unexpected result (since the intent of the string is to represent a 21st century date.)

Since issue arose as a [Python Fire issue](https://github.com/google/python-fire/issues/97), where the behavior of Python Fire was unexpected for inputs such as those described above (1+1 and 2017-10-10) only in Python 3. For context, Python Fire is a CLI library which uses literal_eval as part of its command line argument parsing procedure.

I think the resolution to this issue is having literal_eval raise a ValueError if the ast of the input represents anything other than a Python literal, as described in the documentation. That is, "The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None." Additional operations, such as the binary operations "+" and "-", unless they explicitly create a complex number, should produce a ValueError.

If that resolution is not the direction we take, I also would appreciate knowing if there is another built in approach for determining if a string or ast node represents a literal.

# Reproducing
The following code snippets produce different behaviors in Python 2 and Python 3.
import ast

import ast

# References
- The Python Fire issue is here: https://github.com/google/python-fire/issues/97
- Prior discussion of a similar issue: https://bugs.python.org/issue22525
- I think is where complex number support was originally added: https://bugs.python.org/issue4907
- In this thread, https://bugs.python.org/issue24146, one commenter explains that literal_eval's support for "2+3" is an unintentional side effect of supporting complex literals.

messages: 304294
nosy: David Bieber
priority: normal
severity: normal
status: open
title: ast.literal_eval supports non-literals in Python 3
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list