[Python-Dev] C extension import time

[Antoine Pitrou]

Le Fri, 11 Oct 2013 14:24:29 +0200,
Stefan Krah <stefan at bytereef.org> a écrit :
> Hi,
> 
> recently there has been some talk about reducing import times. It
> seems that the current import strategy for C extensions (i.e.
> importing the extension at the bottom of the .py file) is quite slow:
> 
> 
> ====================
> 
> import sys
> 
> for i in range(10000):
>     import decimal
>     del sys.modules('decimal')
>     del sys.modules('_decimal')
> 
> ====================

Try the following:

$ ./python -m timeit -s "modname='decimal'; import sys" \
  "__import__(modname); del sys.modules[modname]"
1000 loops, best of 3: 2.21 msec per loop

$ ./python -m timeit -s "modname='_decimal'; import sys" \
  "__import__(modname); del sys.modules[modname]"
10000 loops, best of 3: 112 usec per loop


Now compare with the time taken to simply find the module loader:

$ ./python -m timeit -s "modname='_decimal'; import importlib" \
  "importlib.find_loader(modname)"
10000 loops, best of 3: 87.2 usec per loop


So find_loader() is the bigger contributor here. Note that find_loader()
can have varying performance:

$ touch empty.py
$ ./python -m timeit -s "modname='empty'; import importlib" \
  "importlib.find_loader(modname)" 
10000 loops, best of 3: 24.4 usec per loop


When trying to improve import performance, I noticed that merely
diagnosing the bottlenecks is not always easy.

Regards

Antoine.