Speedup hudson job with ccache
I tried to speedup runtests with ccache How to run: CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py Running 'CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py --no-cpp' first time takes 490 seconds on my PC, second time is 214, speed up is 2.2x, cache size is 16MB https://github.com/vitek/cython/commit/c2fb58c780d8727f95353d3f0c28028977a86... -- vitja.
Very cool. We should get ccache on sage.math to start using it on hudson. On Wed, Jul 6, 2011 at 12:24 PM, Vitja Makarov <vitja.makarov@gmail.com> wrote:
I tried to speedup runtests with ccache
How to run: CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py
Running 'CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py --no-cpp' first time takes 490 seconds on my PC, second time is 214, speed up is 2.2x, cache size is 16MB
https://github.com/vitek/cython/commit/c2fb58c780d8727f95353d3f0c28028977a86...
-- vitja. _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
2011/7/7 Stefan Behnel <stefan_ml@behnel.de>
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used? -- vitja.
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one. Also, the test runner clears the target directory on startup. I wonder where you got your numbers from... Stefan
2011/7/7 Stefan Behnel <stefan_ml@behnel.de>
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
Also, the test runner clears the target directory on startup. I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory, how it works: 1. It runs preprocessor (cpp), then adds compiler version, compilation flags and so on. 2. Calculates checksum (md5? I don't know) 3. Check if it's already in the cache, no compilation is required http://ccache.samba.org/ So I think that would be safe to use ccache here. -- vitja.
2011/7/7 Vitja Makarov <vitja.makarov@gmail.com>
2011/7/7 Stefan Behnel <stefan_ml@behnel.de>
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
Also, the test runner clears the target directory on startup. I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory, how it works:
1. It runs preprocessor (cpp), then adds compiler version, compilation flags and so on. 2. Calculates checksum (md5? I don't know) 3. Check if it's already in the cache, no compilation is required
So I think that would be safe to use ccache here.
How it works: http://ccache.samba.org/manual.html#_how_ccache_works Ccache >=3.0 has new "direct mode" think we should disable this, I was running ccache 2.4 that allways do full preprocessing -- vitja.
On Wed, Jul 6, 2011 at 11:54 PM, Vitja Makarov <vitja.makarov@gmail.com> wrote:
2011/7/7 Stefan Behnel <stefan_ml@behnel.de>
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
I think you can set global hudson environment variables as well, which would probably be preferable.
Also, the test runner clears the target directory on startup.
This isn't an issue, if it can, it just copies the .so files from the cache (with new timestamps).
I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory, how it works: 1. It runs preprocessor (cpp), then adds compiler version, compilation flags and so on. 2. Calculates checksum (md5? I don't know) 3. Check if it's already in the cache, no compilation is required http://ccache.samba.org/ So I think that would be safe to use ccache here.
Yes, ccache should be totally safe here, unless it's fundamentally broken. - Robert
Vitja Makarov, 07.07.2011 08:54:
2011/7/7 Stefan Behnel
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
;) I actually set it globally in Jenkins. However, that doesn't automatically remove the "rm -fr *" in the job configs - I initially thought that would be a problem, but, apparently, it isn't.
Also, the test runner clears the target directory on startup. I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory
Hmm, I guess changing that directory into a job local directory would still require changing each job... Are there any concurrency issues with ccache? Anyway, I can't see a .ccache in my home directory, but if that's how it works, it should at least be properly enabled now. Stefan
Stefan Behnel, 07.07.2011 09:29:
Vitja Makarov, 07.07.2011 08:54:
2011/7/7 Stefan Behnel
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
;) I actually set it globally in Jenkins. However, that doesn't automatically remove the "rm -fr *" in the job configs - I initially thought that would be a problem, but, apparently, it isn't.
Also, the test runner clears the target directory on startup. I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory
Hmm, I guess changing that directory into a job local directory would still require changing each job... Are there any concurrency issues with ccache?
Anyway, I can't see a .ccache in my home directory, but if that's how it works, it should at least be properly enabled now.
Clearly makes a difference. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-c/b... The .ccache directory is also there now, size is ~750M. I'll move it to the local scratch disc, CCACHE_DIR should do that. Does it make sense to delete it once a week, or should that be done more often? Stefan
2011/7/7 Stefan Behnel <stefan_ml@behnel.de>:
Stefan Behnel, 07.07.2011 09:29:
Vitja Makarov, 07.07.2011 08:54:
2011/7/7 Stefan Behnel
Vitja Makarov, 07.07.2011 08:32:
2011/7/7 Stefan Behnel
Robert Bradshaw, 06.07.2011 21:32:
We should get ccache on sage.math to start using it on hudson.
> > It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now.
Is that already used?
Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one.
a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile
;) I actually set it globally in Jenkins. However, that doesn't automatically remove the "rm -fr *" in the job configs - I initially thought that would be a problem, but, apparently, it isn't.
Also, the test runner clears the target directory on startup. I wonder where you got your numbers from...
By default ccache stores object files inside ~/.ccache directory
Hmm, I guess changing that directory into a job local directory would still require changing each job... Are there any concurrency issues with ccache?
Anyway, I can't see a .ccache in my home directory, but if that's how it works, it should at least be properly enabled now.
Clearly makes a difference.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-c/b...
The .ccache directory is also there now, size is ~750M. I'll move it to the local scratch disc, CCACHE_DIR should do that. Does it make sense to delete it once a week, or should that be done more often?
Probably it's better to increase cache size, that will be shared between projects: $ ccache -s 5G (I don't know what's the right size to go)
From man ccache:
""" CACHE SIZE MANAGEMENT By default ccache has a one gigabyte limit on the cache size and no maximum number of files. You can set a different limit using the "ccache -M" and "ccache -F" options, which set the size and number of files limits. When these limits are reached ccache will reduce the cache to 20% below the numbers you specified in order to avoid doing the cache clean opera‐ tion too often. """ Btw it's a good idea to cleanup cache once a week: $ ccache --cleanup (or --clear?) -- vitja.
Vitja Makarov, 07.07.2011 11:00:
2011/7/7 Stefan Behnel:
Stefan Behnel, 07.07.2011 09:29: The .ccache directory is also there now, size is ~750M. I'll move it to the local scratch disc, CCACHE_DIR should do that. Does it make sense to delete it once a week, or should that be done more often?
Probably it's better to increase cache size, that will be shared between projects:
$ ccache -s 5G (I don't know what's the right size to go)
Done. I think 5GB is fine, it's a rather large disk.
Btw it's a good idea to cleanup cache once a week:
$ ccache --cleanup (or --clear?)
Added this to my crontab. Stefan
On Thu, Jul 7, 2011 at 2:24 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Vitja Makarov, 07.07.2011 11:00:
2011/7/7 Stefan Behnel:
Stefan Behnel, 07.07.2011 09:29: The .ccache directory is also there now, size is ~750M. I'll move it to the local scratch disc, CCACHE_DIR should do that. Does it make sense to delete it once a week, or should that be done more often?
Probably it's better to increase cache size, that will be shared between projects:
$ ccache -s 5G (I don't know what's the right size to go)
Done. I think 5GB is fine, it's a rather large disk.
This is well worth the time/load savings. I'll look at enabling this for Sage too, which should be a big win (it's been done before).
Btw it's a good idea to cleanup cache once a week:
$ ccache --cleanup (or --clear?)
Why would this be needed, doesn't the cache get recycled naturally on its own? - Robert
participants (3)
-
Robert Bradshaw -
Stefan Behnel -
Vitja Makarov