On Tue, Jun 30, 2009 at 17:16, Ariel Rokem<arokem@berkeley.edu> wrote:

Hi everyone,

I have stumbled on some interesting behavior of scipy.io.loadmat. The short of it - it looks like loadmat is gobbling up memory in some unjustified manner and releasing it under some strange circumstances.

Here's the long story. This is all happening on a Mac OS 10.5.7, running EPD 4.0.30001 (Python 2.5.2), but with a relatively new version of scipy (see below).

I start ipython -pylab in one terminal and run 'top' in another, in order to monitor the memory usage. Here's what I get initially:

PhysMem:  417M wired,  449M active,  183M inactive, 1055M used, 3041M free.

Then:

In [1]: import scipy

In [2]: scipy.__version__ Out[2]: '0.8.0.dev5606'

In [3]: import scipy.io as sio

Here's what it looks like now:

PhysMem:  418M wired,  450M active,  183M inactive, 1058M used, 3038M free.

So far, so good. I read in a large matfile with tons of data in it:

In [4]: a = sio.loadmat('/Users/arokem/Projects/SchizoSpread/Scans/SMR033109_MC/Gray/Original/TSeries/Scan1/tSeries1.mat')

PhysMem:  419M wired, 1024M active,  183M inactive, 1632M used, 2464M free.

So - about 600 MB of memory is taken up by this new variable.

Now to the wierdness:

In [5]: b --------------------------------------------------------------------------- NameError                                 Traceback (most recent call last)

/Users/arokem/<ipython console> in <module>()

NameError: name 'b' is not defined

Of course - 'b' doesn't exist!

But now, the memory usage has dramatically gone down:

PhysMem:  420M wired,  740M active,  183M inactive, 1350M used, 2746M free.

So - just invoking an error in the ipython command line has freed up 300 MB. Where did they come from? I tried different things - assigning other variables doesn't seem to free up this memory. Neither do calls to other functions. Except "plot()", which does seem to do the trick for some reason. Interestingly, when I run all this in a python interactive session (and not ipython), I get a similar memory usage initially. Calling a non-existent variable does not free up the memory, but other things do. For example, import matplotlib.pylab into the namespace did the trick. Does anyone have any idea what is going on?

Garbage collection? Try the above after a call to gc.disable(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco