[Python-Dev] Tests failing on Windows with TESTFN

[Jeremy Kloth]

On Sat, Jul 28, 2018 at 11:20 AM Tim Golden <mail at timgolden.me.uk> wrote:
> Although things have moved on since that discussion and
> test.support.unlink has grown some extra legs, all it's done really is
> to push the bump along the carpet for a bit. I've got a newly-installed
> Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for
> locks with every other process. I've just done a full test run:
>
> python -mtest -j0 -v > test.log

I, for one, would like to see that log.  The issues you are have are
fairly unique.  Just check out the buildbot status page.  I know that
some of the workers are somewhat limited, but my worker
(https://buildbot.python.org/all/#/workers/12) is running on dedicated
hardware.  Before https://bugs.python.org/issue15496 was applied, the
errors you describe were indeed happening, but no longer.

> Here's the thing. The TESTFN approach creates a directory per process
> test_python_<pid> and some variant of @test_<pid>_tmp inside that
> directory. But the same filename in the same directory is used for every
> test in that process. Now, leaving aside the particular mechanism by
> which Windows processes might be holding locks which prevent removal or
> re-addition, that already seems like an odd choice.

Well, since every test (well, test file) is run in its own process, a
directory per process doesn't seem that odd.  You seem to be focusing
on 2 tests in particular, both of which have not been converted to use
test.support.unlink for removing.  The "trick" with
test.support.unlink now is that it waits until the file has truly
removed from the directory before continuing.  Using this, in turn,
would mean that following create *should* succeed without error.  If
test.support.unlink returns without warning, the removed file is
really gone.

> I think that was my starting point: rather than develop increasingly
> involved and still somewhat brittle mechanisms, why not do what you'd
> naturally do with a new test and use tempfile? I was expecting someone
> to come forward to highlight some particular reason why the TESTFN
> approach is superior, but apart from a reference to the possibly cost of
> creating a new temporary file per test, no-one really has.

*PLEASE*, don't use tempfile to create files/directories in tests.  It
is unfriendly to (Windows) buildbots.  The current approach of
directory-per-process ensures no test turds are left behind, whereas
the tempfile solution slowly fills up my buildbot.  Windows doesn't
natively clean out the temp directory.

Additionally, I'm working on a solution to allow the test machinery to
use a given directory for test outputs to allow for a RAM-backed
filesystem for temporary files.  With this, I've had the full test
suite (with -uall including bigmem) take ~5min.  Having some tests use
the Windows default test directory would break this.  It is simply not
feasible to have the entire Windows TEMP in RAM, but just Python's
test suite usage is small enough (<6Gb).

Another point, some tests require that the temporary filename resides
on the same drive as the source directory (discovered in developing
the above).  This means that, Windows development would be restricted
to the same drive as the user directory, if the per-directory approach
isn't used.

-- 
Jeremy Kloth