[Tutor] Memory Management etc

[Philip Smith]

Hi

I use Activestate Python (2.4.3) in a Windows 32 bit environment.

I have a problem with a large programme that uses a lot of memory (numerous large arrays which are dynamically extended).  I keep getting unpredictable crashes (on a machine with 1/2 GB memory) whereas on my laptop (1 GB memory) it seems OK.

The portion of the code which crashes varies a bit but curiously is NEVER a part where any memory allocation is happening.

There are no debugger messages it just crashes (and reboots the machine more often than not).

I have so far tried:

Using the gc module:  can't seem to locate any circular references and garbage collection works OK.

Surrounding all memory allocation code with Try...Except but no MemoryError ever gets reported.

Running under various versions of Python (standard distrib from Python Org, Stackless Python etc)

This kind of crash is classically a stack/heap collision problem (and/or segmentation fault) but if I monitor memory usage during execution (with Windows Monitor) I usually have 200 or so MB free at the point of crash.

I have to say I have noticed (the programme is basically a batch-factoring programme for integers) that no matter how I tune gc I can't get it to reliably free memory in between factoring each integer.

Because this is a development programme (and for performance reasons) I use global variables some of which refer to the large arrays.

My questions are:

1)    Does the mere fact that a function cites a variable as global create a reference which prevents it being collected by gc?
2)    I notice that in Unix I would have access to a resource module which might help - is there nothing similar in Python for Windows?
3)    Is there any way to allocate a pool of available memory to my programme at the outset from which I can allocate my large arrays?
4)    Is there any module / function which would enable me to access the BYTE size of Python objects such as arrays etc or to track memory usage from within Python.
5)    Does anyone have ANY suggestions please?

I'm keen to solve this because I would like to make my programme generally available - in every other respect its near complete and massively outperforms the only other comparable pure python module (nzmath) which does the same job.

Thanks in anticipation.

Phil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/tutor/attachments/20060504/6ea1cfab/attachment.htm