[Python-Dev] Tests failing on Windows with TESTFN

[Giampaolo Rodola']

On Thu, Jul 26, 2018 at 12:16 AM Tim Golden <mail at timgolden.me.uk> wrote:
>
> I'm just easing back into core development work by trying to get a
> stable testing environment for Python development on Windows.
>
> One problem is that certain tests use support.TESTFN (a local directory
> constructed from the pid) for output files etc. However this can cause
> issues on Windows when recreating the folders / files for multiple
> tests, especially when running in parallel.
>
> Here's an example on my laptop deliberately running 3 tests with -j0
> which I know will generate an error about one time in three:
>
> C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2
> test_importlib
>
> Running Debug|Win32 interpreter...
> Run tests in parallel using 6 child processes
> 0:00:23 [1/3/1] test_urllib2 failed
> test test_urllib2 failed -- Traceback (most recent call last):
>    File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line
> 821, in test_file
>      f = open(TESTFN, "wb")
> PermissionError: [Errno 13] Permission denied: '@test_15564_tmp'
>
> Although these errors are both intermittent and fairly easily spotted,
> the effect is that I rarely get a clean test run when I'm applying a patch.
>
> I started to address this years ago but things stalled. I'm happy to
> pick this up again and have another go, but I wanted to ask first
> whether there was any objection to my converting tests to using tempfile
> functions which should avoid the problem?
>
> TJG

>From my experience open() raising PermissionDenied on Windows often
means "file is already in use" as I think is this case. The same issue
exists for directories: https://bugs.python.org/issue33240.

Being TESTFN a global name it appears not suited for parallel testing.
Windows makes this more noticeable than POSIX as it's more strict, but
accessing the same file from 2 unit tests is technically a problem
regardless of the platform. I don't think that means we should ditch
TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup
functions (support.unlink() and support.rmtree()) may be more clever
and (important) they should always be used in setUp / tearDown. For
instance, the traceback you pasted refers to a test class which
doesn't do this.

In psutil I've had occasional Windows failures like this for years
then I got tired of it and came up with this:
https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686
...which basically aggressively retries os.unlink or shutil.rmtree for
1 sec in case of (any) error, and I haven't had this problem since
then.

I suppose test.support's unlink() and rmtree() can do something
similar, maybe just by using a better exception handling, and all unit
tests should use them in setUp / tearDown. I think this will diminish
the occasional failures on Windows, although not completely.

--
Giampaolo - http://grodola.blogspot.com