[Python-Dev] Embedded Python startup is slow

[Stefan Behnel]

bruce bushby, 24.03.2011 18:39:
> On Thu, Mar 24, 2011 at 5:05 PM, Stefan Behnel wrote:
>> bruce bushby, 24.03.2011 16:58:
>>   On my desktop pc, when I run the most simple "Hello World" .... 78% of the
>>> overall execution time is spent opening files....most of which don't
>>> exist.
>>
>> How did you measure that?
>
> I used "strace -c hello.py"

Ok, I tried that as well, using a self-compiled CPython 3.2. On the first 
run, "open()" was at the top, on the subsequent runs, "read()" was. So it 
looks like a non-problem to me.

First run:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000026           0       677       581 open
   0.00    0.000000           0       145           read
   0.00    0.000000           0         1           write
    ...

Second run:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000016           0       145           read
   0.00    0.000000           0         1           write
   0.00    0.000000           0       677       581 open
    ...

The exact timings are likely bogus as the numbers are way too low to be 
measured accurately. The higher number of calls is partly due to the ABI 
version specific .so files that CPython 3.2 uses.


>> I'd expect that after the first startup, most of the relevant parts of the
>> file system are cached by the OS, so a subsequent run should be quick as all
>> file existence tests will run from memory. Sure, it's a bit of overhead to
>> call into the OS, and I don't know how expensive that is on ARM-Linux, but
>> given that you also claim it to be expensive on your desktop, I must say
>> that I'm a bit surprised about your above claim.
>
> On my pc, the "Hello World" script obviously executes very quickly....so
> trying to open 160 odd files that don't exist is negligible, but it still
> happens.

That brings us to the question why your processes are so short-running that 
the startup time starts to matter. And, if they are so short-running, what 
prevents the disk cache (or whatever kind of storage system you use) from 
speeding up the open() requests. However, that's the right kind of question 
to discuss on python-list, not python-devel.

Stefan