Upper memory limit

[Chris Barker]

I've followed this whole thread, and I wonder if a few things have been missed:

Siegfried Gonzi wrote:

> I have a calculation (external C function included with the help of
> SWIG; external Fortran function included with the help of F2PY;

I highly suspect that your memory leak is in your C or Fortran
functions. Python DOES NOT handle memory mangement for extenal
functions. If your C or Fortran functions create Python objects and
don't DECREF them properly, you can create a memory leak there. I have
had that problem. It is pretty easy to test for. before and after
calling your functions, do a memory use check. You could do this by hand
by putting a input() statement or something in to halt the code, or you
can write a little function that makes a system call to check how much
memory Python is using. I have dsone this on Linux, I have no idea how
on Windows.

> If I start the
> calculation for a third time I have to see that Python wreaks havoc.

When you say start the third time, have you completely shut down Python
itself and started it again, rather than running a function from the
python command line for the third time, in a single Python session. If
you are starting a fresh Python, than your problem is definately the OS,
not Python, and I have never had a problem like this on Linux. By the
way, on linux, you can set a memory limit on a user with the ulimit -m
command (in BASH anyway). This can be pretty handy when you are dealing
with a runaway memory problem, because once you get to a lot of swap,
you won't crash the system, but it will be very unresponsive.

> I need some advice for my further strategy  (I am about to write my
> simulations in Fortran 90/95 and use Python only for simple plotting
> jobs):

You really do want to look into using Numeric. It takes a little getting
used to, but it is much faster and more memory efficient for large
arrays of numbers, Also, once you get used to it, it is also cleaner,
easier to understand code.

> def f(list,list2):
> >       erg = []
> >       for k in range( len(list) ):
> >               erg.append( function_on_list2( list2[j] ) )
> >       return erg

In Numeric, this would be something like:

result = function_on_array(list2(:len(list)))

Do you really think that is less clear? Besides clarity, there is less
room for error. in teh above example, you accidentally used j rather
than k in the inner loop. You would probably have caught that on
testing, but maybe not right away!

Send a real function to this list or the Numeric list, and someone will
Numericise it for you so you can have a ggod example. By the way,
Fortran90/95 also has array operations, you should be sure to use them
if you go that route.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov