[docs] [issue28418] Raise Deprecation warning for tokenize.generate_tokens

[Matthias Bussonnier]

New submission from Matthias Bussonnier:

the Tokenize module has the following code:

# An undocumented, backwards compatible, API for all the places in the standard
# library that expect to be able to use tokenize with strings
def generate_tokens(readline):
    return _tokenize(readline, None)

So I'm going to assume it is Deprecated... (since 3.0 AFAICT). 

If it is expected from Python developers to not use it, may I suggest:

1) Marking it as deprecated in the docstring/
2) Raise a deprecation warning. 

Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless.

----------
assignee: docs at python
components: Documentation
messages: 278500
nosy: docs at python, mbussonn
priority: normal
severity: normal
status: open
title: Raise Deprecation warning for tokenize.generate_tokens
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28418>
_______________________________________