[Numpy-discussion] Memory profiling NumPy code?

[Joe Kington]

I know you're looking for something with much more fine-grained control,
(which I can't help much with) but I often find it useful to just plot the
overall memory of the program over time.

There may be an slicker way to do it, but here's the script I use, anyway...
(saved as ~/bin/quick_profile, usage quick_profile (whatever), e.g.
"quick_profile python script.py")

# /bin/sh

# Setup
datfile=$(mktemp)
echo "ElapsedTime MemUsed" > $datfile

starttime=$(date +%s.%N)

# Run the specified command in the background
$@ &

# While the last process is still going
while [ -n "`ps --no-headers $!`" ]
do
    bytes=$(ps -o rss -C $1 --no-headers | awk '{SUM += $1} END {print
SUM}')
    elapsed=$(echo $(date +%s.%N) - $starttime | bc)
    echo $elapsed $bytes >> $datfile
    sleep 0.1
done

# Plot up the results with matplotlib
cat <<EOF | python
import pylab, sys, numpy
infile = file("$datfile")
infile.readline() # skip first line
data = [[float(dat) for dat in line.strip().split()] for line in infile]
data = numpy.array(data)
time,mem = data[:,0], data[:,1]/1024
pylab.plot(time,mem)
pylab.title("Profile of: """ "\"%s\" """ % $@)
pylab.xlabel('Elapsed Time (s): Total %0.5f s' % time.max())
pylab.ylabel('Memory Used (MB): Peak %0.2f MB' % mem.max())
pylab.show()
EOF

rm $datfile

Hope that helps a bit, anyway...
-Joe

On Mon, Apr 26, 2010 at 6:16 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Mon, Apr 26, 2010 at 7:57 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
> > I'd like to profile the memory usage of my application using tools like
> > e.g. Heapy. However since NumPy arrays are allocated in C they are not
> > tracked by Python memory profiling.
> >
> > Does anybody have a workaround to share? I really just want to track a
> > few arrays in a friendly way from within Python (I am aware of the
> > existance of C-level profilers).
>
> I think heapy has some hooks so that you can add support for
> extensions. Maybe we could provide a C API in numpy to make this easy,
>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20100426/92c78177/attachment.html>