You should maybe move the code out of the stdlib (to tests?) if it should not be used. Otherwise, someone somehow will start to rely on it, and then complain when it breaks :-) Victor On Mon, May 30, 2022 at 6:51 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
Is on the main branch but as I mentioned is **exclusively** for internal consumption:
https://github.com/python/cpython/blob/8136606769661c103c46d142e52ecbbbb8880...
On Mon, 30 May 2022 at 17:37, Jack <jack.jjmillist@gmail.com> wrote:
Hi Pablo, could you clarify please? Is that on the main branch, or would you be willing to share the code?
On 30/05/2022 16:23, Pablo Galindo Salgado wrote:
There is no *public* one but there is a private one accesible from Python I added for testing purposes.
On Mon, 30 May 2022, 15:17 Victor Stinner, <vstinner@python.org> wrote:
On Mon, May 30, 2022 at 1:40 AM Eric V. Smith <eric@trueblade.com> wrote:
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the tokenizer, it does not call the one used for python code.
Ah right, I would be surprised that there would be a public Python API to get the tokenizer output, since there is no public C API for that :-)
I just removed <token.h> header file since it was never usable outside Python C internals: there is no public C API to just run the tokenizer and gets its output.
Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSP... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UXPSZFOK... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.