Mailman 3 Providing fix for modulefinder.py regression on Windows - Python-Dev

11 Apr 2020


      modulefinder.py does not open source files in "rb" which
prevents compile() from applying the encoding rules.
This first showed up for me on Windows with Python 3.8.
I'd like to provide patches for 3.8 (if any more releases)
and 3.9.
What is the process I should follow?
I have signed the Contributor Agreement.
Here is my test case and the results on Windows with 3.7.
import modulefinder
with open('import_functools.py', 'w') as f:
    f.write('import functools\n')
mf = modulefinder.ModuleFinder()
mf.run_script('import_functools.py')
print('Test passed')
py -3.8-32 modulefinder_test.py
Traceback (most recent call last):
  File "modulefinder_test.py", line 7, in <module>
    mf.run_script('import_functools.py')
  File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module
    self.scan_code(co, m)
  File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "C:\Python38.win32\lib\modulefinder.py", line 378, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook
    q, tail = self.find_head_package(parent, name)
  File "C:\Python38.win32\lib\modulefinder.py", line 233, in find_head_package
    q = self.import_module(head, qname, parent)
  File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module
    m = self.load_module(fqname, fp, pathname, stuff)
  File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module
    co = compile(fp.read()+'\n', pathname, 'exec')
  File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 308: character maps to <undefined>
Barry