[Python-Dev] Backward incompatible change about docstring AST

Terry Reedy tjreedy at udel.edu
Tue Feb 27 17:31:30 EST 2018


On 2/27/2018 9:32 AM, Serhiy Storchaka wrote:
> 27.02.18 15:37, INADA Naoki пише:
>> Of course, this change was backward incompatible.
>> Tools reading/writing docstring via AST will be broken by this change.
>> For example, it broke PyFlakes, and PyFlakes solved it already.
>>
>> https://github.com/PyCQA/pyflakes/pull/273
> 
> Other examples:
> 
> coveragepy: https://bitbucket.org/ned/coveragepy/commits/99176232199b
> pytest: https://github.com/pytest-dev/pytest/pull/2870
> 
> 
>> Last week, Mark Shannon reported issue about this backward 
>> incompatibility.
>> As he said, this change losted lineno and column of docstring from AST.
> 
> While losing lineno and column is a loss, it is not so large. There are 
> existing issues with docstring position.
> 
> 1. CPython and PyPy set different position for multiline strings. PyPy 
> sets the position of the start of string, but CPython sets the position 
> of the end of the string. A program that utilizes the docstring position 
> needs to handle both of these cases.
> 
> 2. Usually the position of the docstring is used for determining the 
> absolute position of some fragments in the docstring (for example 
> doctests). But since the literal string can contain \n and escaped 
> newlines, and this information is lost in AST, the position of the 
> fragment can be determined accurately.

You obviously meant 'cannot be determined accurately', because for 
reasons including the ones you gave, the relation between string 
literals in code and the resulting string objects is many-to-one, (as 
with int literals).

> This is just a best effort approximation.
> 
> 3. You can determine an approximate position of the docstring by 
> positions of preceding or following nodes.

-- 
Terry Jan Reedy




More information about the Python-Dev mailing list