[issue28418] Raise Deprecation warning for tokenize.generate_tokens
New submission from Matthias Bussonnier: the Tokenize module has the following code: # An undocumented, backwards compatible, API for all the places in the standard # library that expect to be able to use tokenize with strings def generate_tokens(readline): return _tokenize(readline, None) So I'm going to assume it is Deprecated... (since 3.0 AFAICT). If it is expected from Python developers to not use it, may I suggest: 1) Marking it as deprecated in the docstring/ 2) Raise a deprecation warning. Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless. ---------- assignee: docs@python components: Documentation messages: 278500 nosy: docs@python, mbussonn priority: normal severity: normal status: open title: Raise Deprecation warning for tokenize.generate_tokens versions: Python 3.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28418> _______________________________________
Martin Panter added the comment: There is related discussion in Issue 12486 about supporting unencoded text input. The current patch there actually already raises a warning and removes call sites from the Python library, though it does not add a doc string. ---------- nosy: +martin.panter _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28418> _______________________________________
Thomas Kluyver <thomas@kluyver.me.uk> added the comment: I've opened a PR moving in the other direction (making this public rather than deprecating it): https://github.com/python/cpython/pull/6957 ---------- nosy: +takluyver _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue28418> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: Thanks Thomas. I've been a long term user of tokenize.generate_tokens() and would be sad to see it go. The underlying _tokenize() functionality is used else within the module, so there the no benefit to removing the API. ---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue28418> _______________________________________
Karthikeyan Singaravelan <tir.karthi@gmail.com> added the comment: I just stumbled upon this issue while triaging. With the related issue https://bugs.python.org/issue12486 closed making generate_tokens public can this issue be closed since the original issue is not valid? Thanks ---------- nosy: +xtreak _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue28418> _______________________________________
Thomas Kluyver <thomas@kluyver.me.uk> added the comment: Yes, I think this can be closed now. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue28418> _______________________________________
Karthikeyan Singaravelan <tir.karthi@gmail.com> added the comment: Since generate_tokens was made public and documented with issue12486 I am closing this issue as won't fix as part of triaging. Feel free to reopen this issue if needed. Thanks ---------- resolution: -> wont fix stage: -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue28418> _______________________________________
participants (5)
-
Karthikeyan Singaravelan -
Martin Panter
-
Matthias Bussonnier
-
Raymond Hettinger
-
Thomas Kluyver