On 28/07/2018 17:27, Jeremy Kloth wrote:
On Sat, Jul 28, 2018 at 8:41 AM Tim Golden <mail@timgolden.me.uk> wrote:
1) Why are these errors occurring? ie are we dodging a root cause issue
The root cause is how Windows handles file deletions. When a file is removed, it is not immediately removed from the directory, instead, it is simply marked for deletion. Only once all open handles to the file are closed, is it removed from the directory. The most common cause of additional open handles to a file is a antivirus scanner.
Thanks, Jeremy. I'm already aware of that. (If you look back at https://bugs.python.org/issue7443 you'll find me explaining the same to MvL some years ago). [In case the tone isn't clear, there's absolutely no sarcasm implied here. I just wanted to make it clear that I'm at least partly au fait with the situation :)]. Although things have moved on since that discussion and test.support.unlink has grown some extra legs, all it's done really is to push the bump along the carpet for a bit. I've got a newly-installed Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for locks with every other process. I've just done a full test run: python -mtest -j0 -v > test.log and I've got a mixture of Permission (winerror 13) & Access errors (winerror 5). The former are usually attempting to open a TESTFN file; the latter are attempting to unlink one. Most of the Error 5 are os.unlink, but at least one is from test.support.unlink. I understand the unable-to-recreate (virus checkers with share-delete handles) but I'm not so clear on what's blocking the unlink. IOW I think we have more than one issue. Here's the thing. The TESTFN approach creates a directory per process test_python_<pid> and some variant of @test_<pid>_tmp inside that directory. But the same filename in the same directory is used for every test in that process. Now, leaving aside the particular mechanism by which Windows processes might be holding locks which prevent removal or re-addition, that already seems like an odd choice. I think that was my starting point: rather than develop increasingly involved and still somewhat brittle mechanisms, why not do what you'd naturally do with a new test and use tempfile? I was expecting someone to come forward to highlight some particular reason why the TESTFN approach is superior, but apart from a reference to the possibly cost of creating a new temporary file per test, no-one really has.
If you are not seeing the RuntimeWarnings, then something else is amiss.
I think I've seen one RuntimeWarning in the many, many times I've been running through tests today. TJG