[issue39810] Generic script for finding bugs in get_source_segment

[Alex Hall]

New submission from Alex Hall <alex.mojaki at gmail.com>:

Attached is a script which:

- Gets all the source code it can find from sys.modules
- Looks at every node in the parsed source
- Gets source text for that node using ast.get_source_segment
- Parses the source text again
- Compares the original node with the newly parsed node
- Points out if the nodes don't match

I ran this on Python 3.8.0, and it found several issues which have now been solved. So if there was a test like this then many bugs would have been caught earlier. I haven't tried it on a build of master, so I'm actually not sure which bugs have been fixed and what new bugs have been introduced.

The script partly relies on [asttokens](https://github.com/gristlabs/asttokens) which is another way to get the source code of a node. This helps to skip some known issues and to show what the output from get_source_segment should probably be. You don't strictly need to install asttokens to run the script but it's helpful.

----------
components: Interpreter Core
files: get_source_segment_test.py
messages: 363056
nosy: alexmojaki
priority: normal
severity: normal
status: open
title: Generic script for finding bugs in get_source_segment
type: behavior
versions: Python 3.8, Python 3.9
Added file: https://bugs.python.org/file48940/get_source_segment_test.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39810>
_______________________________________