[Tutor] Memory Management etc

[Philip Smith]

Hi

Thanks for response.

I'm pretty sure for one reason and another that hardware is not at fault.

I use the windows monitor to track memory usage.  I must say I hadn't 
realised that Python doesn't release memory back to the OS - someone else 
advised me to try the alpha 2.5 version which cures this problem - I have 
but regretably the same problem keeps occurring.

The programme is about 600 lines and I'd be loth to let anyone have it in 
its current form until I understand this problem.

Interestingly I have programmed this in 3 different ways - as a class, as a 
modular programme and (currently) as a single file application (using global 
variables instead of parameter passing purely because in some cases I was 
passing and returning up to 20 parameters which seemed clumsy).

I have profiled the programme to death and there is no doubt that using 
globals is faster (this is a time-critical application).  However I may 
translate the current version back into modular or class form and see if the 
problem vanishes.

Regards

Phil

----- Original Message ----- 
From: "Danny Yoo" <dyoo at hkn.eecs.berkeley.edu>
To: "Philip Smith" <philipasmith at blueyonder.co.uk>
Cc: <tutor at python.org>
Sent: Thursday, May 04, 2006 6:31 PM
Subject: Re: [Tutor] Memory Management etc


>> I have a problem with a large programme that uses a lot of memory 
>> (numerous large arrays which are dynamically extended).  I keep getting 
>> unpredictable crashes (on a machine with 1/2 GB memory) whereas on my 
>> laptop (1 GB memory) it seems OK.
>
> Hi Philip,
>
> Can you be more specific about what you mean by "crash"?  Does the whole 
> operating system freeze, or do you get an error message from Python, or 
> ...?
>
> The reason I ask is because what you see may not necessarily have to do 
> with Python --- there may be lower-level issues such as physical hardware, 
> for example.  So more information on symptoms will be very helpful.
>
>
>> There are no debugger messages it just crashes (and reboots the machine 
>> more often than not).
>
> Ok, if I understand what you're saying: are you saying that the machine 
> physically reboots without user prompting?  If so, that's almost certainly 
> NOT Python then.  System reboot means that even your operating system is 
> having difficulty keeping the machine usable.  The most likely explanation 
> in this circumstance is that the physical hardware is defective.
>
> I'd recommend running diagnostics like a RAM checker.  But try running 
> your program on another machine as another data point to support the 
> possibility that perhaps the hardware, and not the software is the issue.
>
>
>> I have to say I have noticed (the programme is basically a 
>> batch-factoring programme for integers) that no matter how I tune gc I 
>> can't get it to reliably free memory in between factoring each integer.
>
> How are you measuring this?
>
> Note that Python does not necessarily give allocated memory back to the 
> operating system: it keeps a memory pool that it reuses for performance 
> reasons.
>
> Is the amount of memory you're using at least bounded?
>
>
>> Because this is a development programme (and for performance reasons) I 
>> use global variables some of which refer to the large arrays.
>
> Global variables don't necessarily make programs fast.  I would strongly 
> discourage this kind of ad-hoc performance optimization.  Don't guess: let 
> the machine tell you where the program is slow. If you really want to make 
> your program fast, use a profiler.
>
> Also note that parameter passing from one function to another is a 
> constant-time operation: no object values are being copied.  So the cost 
> assumptions you're making about passing large arrays around functions may 
> not be correct.
>
>
>
>> 1)  Does the mere fact that a function cites a variable as global create 
>> a reference which prevents it being collected by gc?
>
> This isn't a contributing factor.  But global variable values at the 
> toplevel don't die: that's the point about global variables, because they 
> always have at least one reference to them and they're always accessible 
> to the outside.
>
>
>> 3)  Is there any way to allocate a pool of available memory to my 
>> programme at the outset from which I can allocate my large arrays?
>
> Practially everything in Python is done at runtime, not compile time.  At 
> program startup, I suppose you can preallocate some arrays and keep a pool 
> of them for your usage.
>
> But are you finding this to be a significant factor in your program, 
> though?  Again, before you go ahead with optimization, I'd strongly 
> recommend using a profiler to do a principled analysis of the hotspots of 
> your program.  Have you looked at the Python Profiler yet?
>
>
>
>> I'm keen to solve this because I would like to make my programme 
>> generally available - in every other respect its near complete and 
>> massively outperforms the only other comparable pure python module 
>> (nzmath) which does the same job.
>
> If you would like a code review, and if the program is short enough, I'm 
> sure people here would be happy to give some pointers.
>
> Good luck to you!
>