Issue 35212: Expressions with format specifiers in f-strings give wrong code position in AST

ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier.

Compare the trees of f'{a}' and f'{a:b}' :

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets,

Name(lineno=1, col_offset=3, id='a', ctx=Load())

and

Name(lineno=1, col_offset=1, id='a', ctx=Load())

.

Expected behavior: col_offset=3 for name "a" in both f-strings.

(In my specific use case, this breaks a semantic highlighting plugin for vim.)

There is another instance of incorrectly reported offsets in f-strings, in this case affecting multi-line strings.

Example:

(
    f'aaa{foo}bbb'
    f'ccc{bar}ddd'
    f'eee{baz}fff'
)

Relevant part of the syntax tree:

JoinedStr(
    lineno=2,
    col_offset=4,
    values=[
        Str(lineno=2, col_offset=4, s='aaa'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=10, id='foo', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='bbbccc'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=5, id='bar', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='dddcccaaaaaaaaaaaaaa'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=5, id='baz', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='ddd'),
    ],
),

The reported code position for "foo" is correct (2,10), but not for "bar" (2,5) and "baz" (2,5).


Could you give an estimate as to whether this issue will be looked at any time soon?

The same thing occurs with specifiers like {a!r}.

This was fixed by https://github.com/python/cpython/pull/27729.

On 3.10:
```console
>>> ast.parse("f'{a}'").body[0].value.values[0].value.col_offset
3
>>> ast.parse("f'{a:b}'").body[0].value.values[0].value.col_offset
3
```

The other issue also no longer reproduces.

messages: + msg415337
pull_requests: + pull_request30022

pull_requests: + pull_request18760
stage: patch review