[Python-checkins] bpo-33899: Mention tokenize behavior change in What's New (GH-10073)

Gregory P. Smith webhook-mailer at python.org
Wed Oct 24 03:20:13 EDT 2018


https://github.com/python/cpython/commit/dfba1f67e7f1381ceb7cec8fbcfa37337620a9b0
commit: dfba1f67e7f1381ceb7cec8fbcfa37337620a9b0
branch: master
author: Tal Einat <taleinat+github at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-10-24T00:20:05-07:00
summary:

bpo-33899: Mention tokenize behavior change in What's New (GH-10073)

files:
M Doc/whatsnew/3.6.rst
M Doc/whatsnew/3.7.rst
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst
index fec7c620d3b8..bba16543686c 100644
--- a/Doc/whatsnew/3.6.rst
+++ b/Doc/whatsnew/3.6.rst
@@ -2414,3 +2414,11 @@ Notable changes in Python 3.6.5
 The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
 locale to the ``LC_NUMERIC`` locale in some cases.
 (Contributed by Victor Stinner in :issue:`31900`.)
+
+Notable changes in Python 3.6.7
+===============================
+
+In 3.6.7 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token
+when provided with input that does not have a trailing new line.  This behavior
+now matches what the C tokenizer does internally.
+(Contributed by Ammar Askar in :issue:`33899`.)
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index ea460b74025e..d4daf8f71f9c 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -2507,3 +2507,8 @@ calling :c:func:`Py_Initialize`.
 In 3.7.1 the C API for Context Variables
 :ref:`was updated <contextvarsobjects_pointertype_change>` to use
 :c:type:`PyObject` pointers.  See also :issue:`34762`.
+
+In 3.7.1 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token
+when provided with input that does not have a trailing new line.  This behavior
+now matches what the C tokenizer does internally.
+(Contributed by Ammar Askar in :issue:`33899`.)
diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 93dd24acaa8a..a09492137e77 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -173,6 +173,14 @@ Added :attr:`SSLContext.post_handshake_auth` to enable and
 post-handshake authentication.
 (Contributed by Christian Heimes in :issue:`34670`.)
 
+tokenize
+--------
+
+The :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token when
+provided with input that does not have a trailing new line.  This behavior
+now matches what the C tokenizer does internally.
+(Contributed by Ammar Askar in :issue:`33899`.)
+
 tkinter
 -------
 



More information about the Python-checkins mailing list