timeit improvement idea: add an option to measure a script execution time
Hello, Today I had a quite simple need, I am unsure about the best way to do it, and saw a possible improvement for the *timeit *module. I have about 30 Python scripts and I want to measure precisely their execution times, without measuring the interpreter startup time, because for most of them it is quite short (<1ms). I see several ways to do this: 0. Just measure the execution time with "time python script.py" > Not great for me, because for the fastest scripts, it is very imprecise. 1. Modify the scripts to add some "import time; start = time.perf_counter()" at the beginning of each script and "print(time.perf_counter() - start)" at the end of each script. > I would prefer to not change the scripts source code, but I agree that this works otherwise. 2. Have a way to dynamically inject those "time.perf_counter()" as an command line option. I searched and do not think this is currently possible, but 2 options looking like that would work : $ python script.py --before 'import time; start = time.perf_counter()' --after 'print(time.perf_counter() - start)' 3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py 4. Write custom code to import each script. Unless I am mistaken, I feel like solution #3 with a new option of timeit is the most appropriate. I may have missed something more obvious though. What do you think?
On Mon, Jan 11, 2021 at 4:42 AM Alex Prengère <alexprengere@gmail.com> wrote:
Hello, Today I had a quite simple need, I am unsure about the best way to do it, and saw a possible improvement for the timeit module.
I have about 30 Python scripts and I want to measure precisely their execution times, without measuring the interpreter startup time, because for most of them it is quite short (<1ms).
I see several ways to do this: 0. Just measure the execution time with "time python script.py" > Not great for me, because for the fastest scripts, it is very imprecise.
1. Modify the scripts to add some "import time; start = time.perf_counter()" at the beginning of each script and "print(time.perf_counter() - start)" at the end of each script. > I would prefer to not change the scripts source code, but I agree that this works otherwise.
2. Have a way to dynamically inject those "time.perf_counter()" as an command line option. I searched and do not think this is currently possible, but 2 options looking like that would work : $ python script.py --before 'import time; start = time.perf_counter()' --after 'print(time.perf_counter() - start)'
3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py
4. Write custom code to import each script.
Unless I am mistaken, I feel like solution #3 with a new option of timeit is the most appropriate. I may have missed something more obvious though.
What do you think?
Hmm, might run into problems wherein the script execution time isn't really repeatable in that sense. You want to NOT measure interpreter startup time, but you do want to measure the one-off execution time. Are you able to redefine the important part of the code so it's a function? If so, you can use timeit in the normal way (import your script, then time the calls to that function.) Are you trying to measure the time cost of all your top-level imports, but not all the time that the interpreter itself consumes? I think your best bet will be to time the running of the script, then subtract out the time taken for a null script. ChrisA
On 2021-01-10 at 18:38:12 +0100, Alex Prengère <alexprengere@gmail.com> wrote:
Today I had a quite simple need, I am unsure about the best way to do it, and saw a possible improvement for the *timeit *module.
I have about 30 Python scripts and I want to measure precisely their execution times, without measuring the interpreter startup time, because for most of them it is quite short (<1ms).
Are you solving the right problem? If starting up the interpreter dominates actual wall clock execution time and/or user time in your computer, then why are you concerned with the scripts' execution time? And if you're trying to determine the fastest way to build a piece of a larger process, will your measurements carry forward into that larger process? In other words, if Script One runs in X milliseconds all by itself, will it also run in X milliseconds when it's called from a program that's also doing other things?
3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py
4. Write custom code to import each script.
One of those seem like the right solution, depending on why you're measuing what you're measuring.
On 2021-01-10 at 18:38:12 +0100, Alex Prengère <alexprengere@gmail.com> wrote:
3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py
There's always py -m timeit -s "from pathlib import Path; data = Path('your_script.py').read_text()" "exec(data, globals())" Paul
On Mon, Jan 11, 2021 at 6:06 AM Paul Moore <p.f.moore@gmail.com> wrote:
On 2021-01-10 at 18:38:12 +0100, Alex Prengère <alexprengere@gmail.com> wrote:
3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py
There's always
py -m timeit -s "from pathlib import Path; data = Path('your_script.py').read_text()" "exec(data, globals())"
Depending on what's being measured, that might not be an accurate measurement. After the first execution, all imports will be resolved out of sys.modules. ChrisA
On Sun, 10 Jan 2021 at 19:24, Chris Angelico <rosuav@gmail.com> wrote:
On Mon, Jan 11, 2021 at 6:06 AM Paul Moore <p.f.moore@gmail.com> wrote:
On 2021-01-10 at 18:38:12 +0100, Alex Prengère <alexprengere@gmail.com> wrote:
3. Use timeit. The scripts have no side effects so repeating their execution the way timeit does, works for me. The only issue is that, as far as I know, timeit only allows statements as input parameters, not the whole script, like for example: $ python -m timeit --script script.py
There's always
py -m timeit -s "from pathlib import Path; data = Path('your_script.py').read_text()" "exec(data, globals())"
Depending on what's being measured, that might not be an accurate measurement. After the first execution, all imports will be resolved out of sys.modules.
Obviously, yes. The point here is mainly that there's a few ways of doing what the OP asked, and which is better depends on precisely what they want (which they didn't state in detail). As regards the suggestion of adding this functionality to the timeit module, the fact that there *are* multiple options with different trade-offs is precisely why building one particular choice in as "the way to do it" is probably a mistake. Paul
participants (4)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
Alex Prengère -
Chris Angelico -
Paul Moore