[issue1723] test runner filling temp directory with _testcapi modules
New submission from mattip <matti.picus@gmail.com>: Every test run is recompiling _testcapimodule.c, filling up temp directory with many subdirs for each nightly buildbot run. We should fix lib_pypy/_testcapi.py to use a hash like cffi does to build once for each version of pypy and c-files We should also add a cleanup step to buildbot? ---------- messages: 6653 nosy: mattip, pypy-issue priority: feature status: unread title: test runner filling temp directory with _testcapi modules ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
Armin Rigo <armin.rigo@gmail.com> added the comment: Looking at the files in /tmp on tannit, dated say from the night of March 27, I see indeed a large quantity of subdirectories "tmpXXXXXXX" with a _testcapi.pypy-23.so in them. Fixing that would be the first step. But there is a more stuff around, too. Is there any way we could instead convince the builds to use some other general directory than /tmp, like a local directory in the build, maybe with Linux-level hackery to redirect /tmp? ---------- nosy: +arigo status: unread -> chatting ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
mattip <matti.picus@gmail.com> added the comment: What do you think of the hash idea to build once, use many times? fwiw, we support various non-documented environmental variables to control things like where to put files (PYPY_USESSION_DIR), compiler (CC) and probably others, "someone" should document them in pypy/doc/config/index.rst ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
Armin Rigo <armin.rigo@gmail.com> added the comment: Sorry, I'm fine with the hash idea. Please do it. I was already looking for a general solution that would fix all the places that write stuff in /tmp, instead of fixing "only" the most obvious place, but that may not be very useful after all. PYPY_USESSION_DIR is not a configuration option (as pypy/doc/config/index.rst documents), it is an environment variable. (And this particular variable only controls where /tmp/usession-NUM is created, which is only one directory among the large number created in /tmp in a test run.) I don't really think they are essential either; they are just hacks to change various directories. Again, if someone like a Linux distribution wants to ensure that things stay in a controlled environment, they would be better served by chroot for example. ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
mattip <matti.picus@gmail.com> added the comment: fix commited to build testcapi once, needs confirmation after nightly builds changeset ac3fb021b26a ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
mattip <matti.picus@gmail.com> added the comment: need to set TMPDIR both on 'translation' (so we can save the last X build directories, maybe even zip and upload them?) and 'test' steps of buildbots, then add a step to clean TMPDIR once testing is complete ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
Philip Jenvey <pjenvey@underboss.org> added the comment: mattip, any idea why py3k 32 bit runs into an issue of the _testcapi so apparently clashing w/ the 64 bit one, whereas default does not? Or maybe something else is going on http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-c- app-level-linux-x86-32&build=2614&mod=lib-python.3.test.test___all__ ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue1723> ________________________________________
participants (3)
-
Armin Rigo -
mattip
-
Philip Jenvey