[Python-checkins] cpython (3.4): Issue #22599: Enhance tokenize.open() to be able to call it during Python

[victor.stinner]

https://hg.python.org/cpython/rev/cd3244416592
changeset:   93735:cd3244416592
branch:      3.4
parent:      93733:c3c060e5ba99
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri Dec 05 10:17:10 2014 +0100
summary:
  Issue #22599: Enhance tokenize.open() to be able to call it during Python
finalization.

Before the module kept a reference to the builtins module, but the module
attributes are cleared during Python finalization. Instead, keep directly a
reference to the open() function.

This enhancement is not perfect, calling tokenize.open() can still fail if
called very late during Python finalization.  Usually, the function is called
by the linecache module which is called to display a traceback or emit a
warning.

files:
  Lib/tokenize.py |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Lib/tokenize.py b/Lib/tokenize.py
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -24,7 +24,6 @@
 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
                'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
                'Michael Foord')
-import builtins
 from codecs import lookup, BOM_UTF8
 import collections
 from io import TextIOWrapper
@@ -430,11 +429,13 @@
     return default, [first, second]
 
 
+_builtin_open = open
+
 def open(filename):
     """Open a file in read only mode using the encoding detected by
     detect_encoding().
     """
-    buffer = builtins.open(filename, 'rb')
+    buffer = _builtin_open(filename, 'rb')
     encoding, lines = detect_encoding(buffer.readline)
     buffer.seek(0)
     text = TextIOWrapper(buffer, encoding, line_buffering=True)
@@ -657,7 +658,7 @@
         # Tokenize the input
         if args.filename:
             filename = args.filename
-            with builtins.open(filename, 'rb') as f:
+            with _builtin_open(filename, 'rb') as f:
                 tokens = list(tokenize(f.readline))
         else:
             filename = "<stdin>"

-- 
Repository URL: https://hg.python.org/cpython