[Python-Dev] Breaking bytecode only imports (was Re: __file__)
Nick Coghlan
ncoghlan at gmail.com
Tue Mar 2 14:03:19 CET 2010
Barry Warsaw wrote:
> Thanks everybody for providing great input on this aspect of the PEP. I've
> updated the open issues section to include a list of the possible resolutions
> for bytecode-only imports. Unless anybody has more ideas, it might just be
> time to get a BDFL pronouncement.
I think the benchmarking in the bytecode-only section is still too weak.
"evidence shows that the extra stats can be fairly costly to start up
time" isn't a valid justification for breaking working code. Doing 4
stat calls instead of 5 on a directory miss just doesn't excite me very
much without some genuine benchmarks across different operating systems
and filesystems showing that reducing the number of stat calls by at
best 20% will result in a measurable reduction in import times for real
modules (where we can expect the import time to be dominated by the
execution of the actual module code rather than the time needed to find
that code in the first place).
Using the sample numbers Robert Collins posted:
# Startup time for bzr (cold cache):
$ drop-caches
$ time bzr --no-plugins revno
5061
real 0m8.875s
user 0m0.210s
sys 0m0.140s
# Hot cache
$ time bzr --no-plugins revno
5061
real 0m0.307s
user 0m0.250s
sys 0m0.040s
$ strace -c bzr --no-plugins revno
5061
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
56.34 0.040000 76 527 read
28.98 0.020573 9 2273 1905 open
14.43 0.010248 14 734 625 stat
0.15 0.000107 0 533 fstat
hot cache:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
45.10 0.000368 92 4 getdents
19.49 0.000159 0 527 read
16.91 0.000138 1 163 munmap
10.05 0.000082 2 54 mprotect
8.46 0.000069 0 2273 1905 open
0.00 0.000000 0 8 write
0.00 0.000000 0 367 close
0.00 0.000000 0 734 625 stat
Assuming all those stat errors are misses from the import system, we're
looking at reducing that 625 figure down to 500: 125 fewer failed calls.
With a hot cache, the impact is too small for strace to even measure.
With a cold cache, it is 1.75 milliseconds: only 1.25% of the system
time consumed in the script's execution, and not even registering
relative to the 9 second wall clock time.
Without significant measurable performance gains, a mere aesthetic
preference isn't enough to justify inflicting subtle breakage on even a
small subset of our users.
Even aside from the issue of a lack of benchmarks to justify the
breakage, bytecode only imports *cannot* legitimately be broken without
at least one release where they generate Deprecation Warnings.
Cheers,
Nick.
P.S. I actually started this thread as a +0 to the idea of dropping
bytecode only imports. Over the course of the discussion I've shifted to
a firm -1 in the absence of some proper comparative benchmarks to
justify the change in semantics.
--
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
---------------------------------------------------------------
More information about the Python-Dev
mailing list