profile reveals calls to astype()
Hi all, I have a problem trying to profile a program using numarray, maybe someone with more experience can give me a hint... basically, the program I am profiling has a function like this: foo(): # some code # a call to astype() for i in xrange(N): # some other code and NO explicit call to astype() the problem is that when I print the 'callees' of foo(), astype() gets listed with an occurrence of N+1, as if it was called inside the loop. So now the first doubt I have is that astype() gets listed because called from some function called by foo(), even if this should not happen. Here is the list of numarray functions called in foo() Function called... generic.py:651(getshape)(14) 0.070 generic.py:918(reshape)(2) 0.000 generic.py:1013(where)(2) 0.050 generic.py:1069(concatenate)(2) 4.270 morphology.py:150(binary_erosion)(2) 0.070 numarraycore.py:698(__del__)(120032) 3.240 numarraycore.py:817(astype)(12002) 37.290 numarraycore.py:857(is_c_array)(36000) 10.450 numarraycore.py:878(type)(4) 0.000 numarraycore.py:964(__mul__)(12) 0.340 numarraycore.py:981(__div__)(8) 0.010 numarraycore.py:1068(__pow__)(8) 0.000 numarraycore.py:1180(__imul__)(12000) 0.930 numarraycore.py:1250(__eq__)(2) 0.080 numarraycore.py:1400(zeros)(54) 0.060 numarraycore.py:1409(ones)(8) 0.020 The second thing I can think of is that astype() is implicitly called by some conversion. Can this be? curzio
astype() is used in a bunch of places, including the C-API, so it's hard to guess how it's getting called with the information here. In general, astype() gets called to "match up types" based on a particular parameterization of a function call, i.e. the c-code underlying some function call needs a different type than was passed in so astype() is used to convert an array to a workable type. One possibility for debugging this might be to drop N to something reasonable, like say 2, and then run under pdb with a breakpoint set on astype(). Something like this is what I have in mind; it may not be exactly right but with fiddling this approach might work:
from yourmodule import newfoo # you redefined foo to accept N as a parameter import pdb pdb.run("newfoo(N=2)") (pdb) s # step along a little to get into newfoo() ... step output (pdb) import numarray.numarraycore as nc (pdb) break nc.astype (pdb) c ... breakpoint output (pdb) where ... function traceback showing where astype() got called from (pdb) c ... breakpoint output (pdb) where ... more function traceback, eventually you should find it... ...
Regards, Todd On Thu, 2005-04-07 at 12:56, Curzio Basso wrote:
Hi all,
I have a problem trying to profile a program using numarray, maybe someone with more experience can give me a hint...
basically, the program I am profiling has a function like this:
foo(): # some code # a call to astype() for i in xrange(N): # some other code and NO explicit call to astype()
the problem is that when I print the 'callees' of foo(), astype() gets listed with an occurrence of N+1, as if it was called inside the loop. So now the first doubt I have is that astype() gets listed because called from some function called by foo(), even if this should not happen. Here is the list of numarray functions called in foo()
Function called... generic.py:651(getshape)(14) 0.070 generic.py:918(reshape)(2) 0.000 generic.py:1013(where)(2) 0.050 generic.py:1069(concatenate)(2) 4.270 morphology.py:150(binary_erosion)(2) 0.070 numarraycore.py:698(__del__)(120032) 3.240 numarraycore.py:817(astype)(12002) 37.290 numarraycore.py:857(is_c_array)(36000) 10.450 numarraycore.py:878(type)(4) 0.000 numarraycore.py:964(__mul__)(12) 0.340 numarraycore.py:981(__div__)(8) 0.010 numarraycore.py:1068(__pow__)(8) 0.000 numarraycore.py:1180(__imul__)(12000) 0.930 numarraycore.py:1250(__eq__)(2) 0.080 numarraycore.py:1400(zeros)(54) 0.060 numarraycore.py:1409(ones)(8) 0.020
The second thing I can think of is that astype() is implicitly called by some conversion. Can this be?
curzio
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion --
Todd Miller wrote:
astype() is used in a bunch of places, including the C-API, so it's hard to guess how it's getting called with the information here. In
ok, so probably C functions are somehow 'transparent' to the profiler which does not report them, but reports the python functions called by the C one...
from yourmodule import newfoo # you redefined foo to accept N as a parameter import pdb pdb.run("newfoo(N=2)")
(pdb) s # step along a little to get into newfoo() ... step output (pdb) import numarray.numarraycore as nc (pdb) break nc.astype
strange, what I get now is:
(Pdb) b nc.astype *** The specified object 'nc.astype' is not a function or was not found along sys.path.
and in fact if I look at nc.__dict__ there is no 'astype' key. I'm running the whole program (rather than just the function) under ipython, starting it with
%run -d myprog.py
maybe this could mess up things? curzio
On Fri, 2005-04-08 at 09:29, Curzio Basso wrote:
Todd Miller wrote:
astype() is used in a bunch of places, including the C-API, so it's hard to guess how it's getting called with the information here. In
ok, so probably C functions are somehow 'transparent' to the profiler which does not report them, but reports the python functions called by the C one...
from yourmodule import newfoo # you redefined foo to accept N as a parameter import pdb pdb.run("newfoo(N=2)")
(pdb) s # step along a little to get into newfoo() ... step output (pdb) import numarray.numarraycore as nc (pdb) break nc.astype
strange, what I get now is:
(Pdb) b nc.astype *** The specified object 'nc.astype' is not a function or was not found along sys.path.
and in fact if I look at nc.__dict__ there is no 'astype' key. I'm running the whole program (rather than just the function) under ipython, starting it with
%run -d myprog.py
maybe this could mess up things?
No. I should have said "b nc.NumArray.astype". I just tried this out with an astype() callback from numarray.convolve's C-code and it worked OK for me. Regards, Todd
participants (2)
-
Curzio Basso -
Todd Miller