[Python-checkins] bpo-43014: Improve performance of tokenize.tokenize by 20-30%

[isidentical]

https://github.com/python/cpython/commit/15bd9efd01e44087664e78bf766865a6d2e06626
commit: 15bd9efd01e44087664e78bf766865a6d2e06626
branch: master
author: Anthony Sottile <asottile at umich.edu>
committer: isidentical <isidentical at gmail.com>
date: 2021-01-24T12:23:17+03:00
summary:

bpo-43014: Improve performance of tokenize.tokenize by 20-30%

files:
A Misc/NEWS.d/next/Library/2021-01-24-00-37-40.bpo-43014.BVPhEr.rst
M Lib/tokenize.py

diff --git a/Lib/tokenize.py b/Lib/tokenize.py
index 1aee21b5e18fa..42c1f10373de9 100644
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -27,6 +27,7 @@
 from builtins import open as _builtin_open
 from codecs import lookup, BOM_UTF8
 import collections
+import functools
 from io import TextIOWrapper
 import itertools as _itertools
 import re
@@ -95,6 +96,7 @@ def _all_string_prefixes():
                 result.add(''.join(u))
     return result
 
+ at functools.lru_cache
 def _compile(expr):
     return re.compile(expr, re.UNICODE)
 
diff --git a/Misc/NEWS.d/next/Library/2021-01-24-00-37-40.bpo-43014.BVPhEr.rst b/Misc/NEWS.d/next/Library/2021-01-24-00-37-40.bpo-43014.BVPhEr.rst
new file mode 100644
index 0000000000000..02898e4a3a42e
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2021-01-24-00-37-40.bpo-43014.BVPhEr.rst
@@ -0,0 +1 @@
+Improve performance of :mod:`tokenize` by 20-30%.  Patch by Anthony Sottile.