On Mon, 14 Mar 2016 at 07:27 Antoine Pitrou solipsis@pitrou.net wrote:
On Sun, 13 Mar 2016 17:44:10 +0000 Brett Cannon brett@python.org wrote:
- One iteration of all searches on full text takes 29 seconds on my
computer. Isn't this too long? In any case I want first optimize some bottlenecks in the re module.
I don't think we have established a "too long" time. We do have some benchmarks like spectral_norm that don't run unless you use rigorous mode and this could be one of them.
- Do we need one benchmark that gives an accumulated time of all
searches, or separate microbenchmarks for every pattern?
I don't care either way. Obviously it depends on whether you want to measure overall re perf and have people aim to improve that or let people target specific workload types.
This is a more general latent issue with our current benchmarking philosophy. We have built something which aims to be a general-purpose benchmark suite, but in some domains a more comprehensive set of benchmarks may be desirable. Obviously we don't want to have 10 JSON benchmarks, 10 re benchmarks, 10 I/O benchmarks, etc. in the default benchmarks run, so what do we do for such cases? Do we tell people domain-specific benchmarks should be developed independently? Do we include some facilities to create such subsuites without them being part of the default bunch?
(note a couple domain-specific benchmarks -- iobench, stringbench, etc. -- are currently maintained separately)
Good point. I personally don't have a good feel on how to handle this. Part of me would like to consolidate the benchmarks so that it's easier to discover what benchmarks there are. Another part of me doesn't want to burden folks writing there own benchmarks for development purposes too much.