[issue11677] make test has horrendous performance on an ecryptfs

[Barry A. Warsaw]

New submission from Barry A. Warsaw <barry at python.org>:

When your home directory is on a Linux (e.g. Ubuntu 10.10) ecryptfs, 'make test' and company can be horrendously slow.  Of course, some performance hit should be expected, but depending on which combinations of tests I've run, I can see up to 25000x (!) slower on an ecryptfs than on a normal ext4 file system.

regrtest.py changes its cwd to a TEMPDIR, but actually when you're running the tests from inside the Python build directory, this just becomes $srcdir/build so you don't get any advantage of running the tests out of e.g. a much faster tmpfs.

(Aside: I'm not sure under what cases you would *not* be normally running out of the build dir, but I guess if you 'cd /tmp; /path/to/python/configure' and such, it would put you in a normal temp directory.  OTOH, you're already in a tmpdir by then so what's the point of _make_temp_dir_for_build()?)

I'd like to at least provide the option to create regrtest temporary files elsewhere so that they can live on a fast file system.  There are several ways I can think of doing this and I'm not sure what the best way is:

* Remove the special case from _make_temp_dir_for_build() so that it always sets TESTCWD into a tmpdir.

* Add a boolean option --usetmp/-p which enables this override or the moral equivalent.

* Make TESTFN relative to mkdtemp().  A quick and dirty test of this showed that it did significantly improve test times, but there were test failures too.  

I'm open to other ideas, but I really do want to be able to ./configure && ake && make testall in an ecryptfs build dir and get reasonable test times.

You'll need an atexit handler or similar to clean up the tempdir.

This does affect older Pythons, but probably any solution would be classified as a new feature so could only go in Python 3.3.

----------
assignee: barry
messages: 132171
nosy: barry
priority: normal
severity: normal
stage: needs patch
status: open
title: make test has horrendous performance on an ecryptfs
type: performance
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11677>
_______________________________________