[Numpy-discussion] "import numpy" performance
Andrew Dalke
dalke at dalkescientific.com
Mon Jul 2 17:44:12 EDT 2012
On Jul 2, 2012, at 10:34 PM, Nathaniel Smith wrote:
> I don't have any opinion on how acceptable this would be, but I also
> don't see a benchmark showing how much this would help?
The profile output was lower in that email. The relevant line is
0.038 add_newdocs (numpy.core.multiarray)
This says that 'add_newdocs', which is imported from
numpy.core.multiarray (though there may be other importers)
takes 0.038 seconds to go through __import__, including
all of its children module imports.
I have attached my import profile script. It has only minor
changes since the one I posted on this list 4 years ago. Its
output is here, showing the import dependency tree first,
and then the list of slowest modules to import.
== Tree ==
rdkit: 0.150 (None)
os: 0.000 (rdkit)
sys: 0.000 (rdkit)
exceptions: 0.000 (rdkit)
sqlite3: 0.003 (pyPgSQL)
dbapi2: 0.002 (sqlite3)
datetime: 0.001 (dbapi2)
time: 0.000 (dbapi2)
_sqlite3: 0.001 (dbapi2)
cDataStructs: 0.008 (pyPgSQL)
rdkit.Geometry: 0.003 (pyPgSQL)
rdGeometry: 0.003 (rdkit.Geometry)
PeriodicTable: 0.002 (pyPgSQL)
re: 0.000 (PeriodicTable)
rdchem: 0.116 (pyPgSQL)
numpy.core.multiarray: 0.109 (rdchem)
numpy.__config__: 0.000 (numpy.core.multiarray)
version: 0.000 (numpy.core.multiarray)
_import_tools: 0.000 (numpy.core.multiarray)
add_newdocs: 0.067 (numpy.core.multiarray)
numpy.lib: 0.061 (add_newdocs)
info: 0.000 (numpy.lib)
numpy.version: 0.000 (numpy.lib)
type_check: 0.053 (numpy.lib)
numpy.core.numeric: 0.053 (type_check)
multiarray: 0.001 (numpy.core.numeric)
umath: 0.000 (numpy.core.numeric)
_internal: 0.004 (numpy.core.numeric)
warnings: 0.000 (_internal)
numpy.compat: 0.000 (_internal)
_inspect: 0.000 (numpy.compat)
types: 0.000 (_inspect)
py3k: 0.000 (numpy.compat)
numerictypes: 0.001 (numpy.core.numeric)
__builtin__: 0.000 (numerictypes)
_sort: 0.000 (numpy.core.numeric)
numeric: 0.003 (numpy.core.numeric)
_dotblas: 0.000 (numeric)
arrayprint: 0.001 (numeric)
fromnumeric: 0.000 (arrayprint)
cPickle: 0.001 (numeric)
copy_reg: 0.000 (cPickle)
cStringIO: 0.000 (cPickle)
defchararray: 0.001 (numpy.core.numeric)
numpy: 0.000 (defchararray)
records: 0.000 (numpy.core.numeric)
memmap: 0.000 (numpy.core.numeric)
scalarmath: 0.000 (numpy.core.numeric)
numpy.core.umath: 0.000 (scalarmath)
function_base: 0.000 (numpy.core.numeric)
machar: 0.000 (numpy.core.numeric)
numpy.core.fromnumeric: 0.000 (machar)
getlimits: 0.000 (numpy.core.numeric)
shape_base: 0.000 (numpy.core.numeric)
numpy.testing: 0.041 (numpy.core.numeric)
unittest: 0.039 (numpy.testing)
result: 0.002 (unittest)
traceback: 0.000 (result)
linecache: 0.000 (traceback)
StringIO: 0.000 (result)
errno: 0.000 (StringIO)
: 0.000 (result)
functools: 0.001 (result)
_functools: 0.000 (functools)
case: 0.035 (unittest)
difflib: 0.034 (case)
heapq: 0.031 (difflib)
itertools: 0.029 (heapq)
operator: 0.001 (heapq)
bisect: 0.001 (heapq)
_bisect: 0.000 (bisect)
_heapq: 0.000 (heapq)
collections: 0.001 (difflib)
_abcoll: 0.000 (collections)
_collections: 0.000 (collections)
keyword: 0.000 (collections)
thread: 0.000 (collections)
pprint: 0.000 (case)
util: 0.000 (case)
suite: 0.000 (unittest)
loader: 0.001 (unittest)
fnmatch: 0.000 (loader)
main: 0.001 (unittest)
signals: 0.001 (main)
signal: 0.000 (signals)
weakref: 0.001 (signals)
UserDict: 0.000 (weakref)
_weakref: 0.000 (weakref)
_weakrefset: 0.000 (weakref)
runner: 0.000 (unittest)
decorators: 0.001 (numpy.testing)
numpy.testing.utils: 0.001 (decorators)
nosetester: 0.000 (numpy.testing.utils)
utils: 0.000 (numpy.testing)
numpytest: 0.000 (numpy.testing)
ufunclike: 0.000 (type_check)
index_tricks: 0.003 (numpy.lib)
numpy.core.numerictypes: 0.000 (index_tricks)
math: 0.000 (index_tricks)
numpy.core: 0.000 (index_tricks)
numpy.lib.twodim_base: 0.000 (index_tricks)
_compiled_base: 0.000 (index_tricks)
arraysetops: 0.001 (index_tricks)
numpy.lib.utils: 0.001 (arraysetops)
numpy.matrixlib: 0.001 (index_tricks)
defmatrix: 0.001 (numpy.matrixlib)
numpy.lib._compiled_base: 0.000 (index_tricks)
stride_tricks: 0.000 (numpy.lib)
twodim_base: 0.000 (numpy.lib)
scimath: 0.000 (numpy.lib)
numpy.lib.type_check: 0.000 (scimath)
polynomial: 0.001 (numpy.lib)
numpy.lib.function_base: 0.000 (polynomial)
numpy.linalg: 0.001 (polynomial)
linalg: 0.000 (numpy.linalg)
numpy.matrixlib.defmatrix: 0.000 (linalg)
npyio: 0.002 (numpy.lib)
format: 0.000 (npyio)
_datasource: 0.001 (npyio)
shutil: 0.001 (_datasource)
stat: 0.000 (shutil)
os.path: 0.000 (shutil)
pwd: 0.000 (shutil)
grp: 0.000 (shutil)
_iotools: 0.000 (npyio)
financial: 0.000 (numpy.lib)
arrayterator: 0.000 (numpy.lib)
__future__: 0.000 (arrayterator)
numpy.lib.index_tricks: 0.000 (add_newdocs)
testing: 0.000 (numpy.core.multiarray)
core: 0.000 (numpy.core.multiarray)
compat: 0.000 (numpy.core.multiarray)
lib: 0.000 (numpy.core.multiarray)
fft: 0.001 (numpy.core.multiarray)
fftpack: 0.000 (fft)
fftpack_lite: 0.000 (fftpack)
helper: 0.000 (fft)
polyutils: 0.000 (numpy.core.multiarray)
polytemplate: 0.002 (numpy.core.multiarray)
string: 0.002 (polytemplate)
strop: 0.000 (string)
chebyshev: 0.005 (numpy.core.multiarray)
legendre: 0.004 (numpy.core.multiarray)
hermite: 0.004 (numpy.core.multiarray)
hermite_e: 0.004 (numpy.core.multiarray)
laguerre: 0.004 (numpy.core.multiarray)
random: 0.007 (numpy.core.multiarray)
mtrand: 0.006 (random)
ctypeslib: 0.004 (numpy.core.multiarray)
ctypes: 0.003 (ctypeslib)
_ctypes: 0.001 (ctypes)
struct: 0.001 (ctypes)
_struct: 0.000 (struct)
ctypes._endian: 0.000 (ctypes)
numpy.core._internal: 0.000 (ctypeslib)
ma: 0.003 (numpy.core.multiarray)
extras: 0.001 (ma)
matrixlib: 0.000 (numpy.core.multiarray)
rdmolfiles: 0.005 (pyPgSQL)
rdmolops: 0.006 (pyPgSQL)
inchi: 0.000 (pyPgSQL)
== Slowest (including children) ==
0.150 rdkit (None)
0.116 rdchem (pyPgSQL)
0.109 numpy.core.multiarray (rdchem)
0.067 add_newdocs (numpy.core.multiarray)
0.061 numpy.lib (add_newdocs)
0.053 type_check (numpy.lib)
0.053 numpy.core.numeric (type_check)
0.041 numpy.testing (numpy.core.numeric)
0.039 unittest (numpy.testing)
0.035 case (unittest)
0.034 difflib (case)
0.031 heapq (difflib)
0.029 itertools (heapq)
0.008 cDataStructs (pyPgSQL)
0.007 random (numpy.core.multiarray)
0.006 mtrand (random)
0.006 rdmolops (pyPgSQL)
0.005 rdmolfiles (pyPgSQL)
0.005 chebyshev (numpy.core.multiarray)
0.004 hermite (numpy.core.multiarray)
>> With instrumentation I found that 0.083s of the 0.119s
>> is spent loading numpy.core.multiarray.
>
> Sounds like this would be useful to profile. I have no idea why
> importing this would need so much time, since the actual import just
> requires filling in a few C structs. Could be the linker, could be
> lots of things, profiling is useful.
I also have no idea why "import_array()" needs to include ~90 modules.
My hope it to prune a good number of those.
Cheers,
Andrew
dalke at dalkescientific.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_profile.py
Type: text/x-python-script
Size: 1168 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120702/f12439d2/attachment.bin>
More information about the NumPy-Discussion
mailing list