[New-bugs-announce] [issue40924] Recent importlib change breaks most recent certifi == 2020.4.5.2

[Ned Deily]

New submission from Ned Deily <nad at python.org>:

The very recent latest commits for Issue39791, "New `files()` api from importlib_resources", have broken the popular certifi package, a package which provides a basic set of Root Certificates for TLS secure network connection verification. Among other users of it, the python.org macOS installers encourage its users to run a provided script to install certifi.  Alas, we discovered just after v3.9.2b2 was tagged that that script is broken because certifi.where() is returning a bogus path toe the pem file, what appears to be a path in a deleted temp directory.

The culprit commits are 843c27765652e2322011fb3e5d88f4837de38c06 (master) and 9cf1be46e3692d565461afd3afa326d124d743dd (3.9). This is now a critical problem because in its most recent release, certifi 2020.4.5.2, certifi was changed to try to use importlib.resources.path(), if available, to find the path to the installed cacert.pem file. (The previous release, 2020.4.5.1, appears not to use path() and so is not affected by this bug.)

https://github.com/certifi/python-certifi/commit/3fc8fec0466b0f12f10ad3e429b8d915bc5c26fb

https://pypi.org/project/certifi/2020.4.5.2/

Without trying to debug the bug, I was able to bisect the branch and then reduce the problem seen in macOS installer testing to a fairly simple reproducible test case. The problem was reproduced on both Linux and macOS systems.

The test case:

# in a current cpython git repo, checkout the commit before the failing one
git checkout 843c27765652e2322011fb3e5d88f4837de38c06^
git log HEAD^..HEAD
git clean -fdxq
./configure --prefix=$PWD/root -q 
make -j4
./python -m ensurepip
./python -m pip install --upgrade pip # not necessary to reproduce
./python -m pip install --force --no-binary certifi certifi==2020.4.5.2
./python -c 'import certifi;print(certifi.where())'

The output tail should be something like:
[...]
Successfully installed certifi-2020.4.5.2
/home/nad/cpython/root/lib/python3.10/site-packages/certifi/cacert.pem

Now checkout the failing commit and repeat all the other steps:

git checkout 843c27765652e2322011fb3e5d88f4837de38c06
git log HEAD^..HEAD
[...]

The output tail is now incorrect:
[...]
Successfully installed certifi-2020.4.5.2
/tmp/tmpqfjnbj5bcacert.pem

The cacert.pem is installed to the expected (same) location in either case; its just the output from importlib.resources.path that is incorrect:

./python
Python 3.10.0a0 (remotes/upstream/master:0a40849eb9, Jun  9 2020, 00:35:07)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.resources, os, os.path, certifi
>>> certifi.__file__
'/home/nad/cpython/root/lib/python3.10/site-packages/certifi/__init__.py'
>>> os.listdir(os.path.dirname(certifi.__file__))
['__pycache__', 'cacert.pem', '__init__.py', 'core.py', '__main__.py']
>>> with importlib.resources.path('certifi', 'cacert.pem') as f: print(f)
...
/tmp/tmpxsrjxb8lcacert.pem
>>> with importlib.resources.path('certifi', 'core.py') as f: print(f)
...
/tmp/tmpjq8h3si5core.py

No test suite failures were noted. Perhaps there should be a test case for this?

Presumably any other downstream users of importlib.resources.path() are affected.

Łukasz as 3.9 release manager is aware there is an issue but was awaiting the tracking down of the problem before making a decision about what to do for 3.9.0b2.

cc: Donald as the author of the certifi change.

----------
assignee: jaraco
components: Library (Lib)
messages: 371073
nosy: dstufft, jaraco, lukasz.langa, ned.deily
priority: release blocker
severity: normal
stage: needs patch
status: open
title: Recent importlib change breaks most recent certifi == 2020.4.5.2
type: behavior
versions: Python 3.10, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40924>
_______________________________________