[New-bugs-announce] [issue42806] Incorrect offsets in new parser for f-string substitutions

[Anthony Sottile]

New submission from Anthony Sottile <asottile at umich.edu>:

here's a sample file.  this is distilled down from an example which broke a code formatter here: https://github.com/asottile/add-trailing-comma/issues/106

def f():
    x = (
            'PERL_MM_OPT', (
                f'INSTALL-BASE={shlex.quote(venv)} '
                f'wat'
            ),
    )


A diff from astpretty (https://github.com/asottile/astpretty) between the old and new parser:

```console
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(.tox/py39/bin/python -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:58:09.108060968 -0800
+++ /dev/fd/62	2021-01-01 15:58:09.108060968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,
```

the old parser is correct here, and the new parser is wrong

notably it thinks that the call inside the f-string substitution is positioned at column 16 which is the position of the f-string start

this bug is also present in 3.10:

```console
$ venv/bin/python3.10 --version
Python 3.10.0a3
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(venv/bin/python3.10 -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:59:54.000059968 -0800
+++ /dev/fd/62	2021-01-01 15:59:54.000059968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,
```

----------
components: Interpreter Core
messages: 384208
nosy: Anthony Sottile
priority: normal
severity: normal
status: open
title: Incorrect offsets in new parser for f-string substitutions
type: behavior
versions: Python 3.10, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42806>
_______________________________________