[ python-Feature Requests-540952 ] Memory Usage Reporting

[SourceForge.net]

Feature Requests item #540952, was opened at 2002-04-08 08:01
Message generated for change (Comment added) made by jimjjewett
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=540952&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Allan Crooks (amc1)
Assigned to: Nobody/Anonymous (nobody)
Summary: Memory Usage Reporting

Initial Comment:
I would personally like a way in Python to report how 
many bytes of memory that the interpreter is using 
(perhaps through the sys module)?

If this sort of mechanism is added, then it may allow 
SoftReferences (a la Java) to be introduced, which 
would definitely be useful for memory sensitive 
caches...

----------------------------------------------------------------------

Comment By: Jim Jewett (jimjjewett)
Date: 2004-03-25 16:58

Message:
Logged In: YES 
user_id=764593


For many uses, an exact count isn't required; it is enough to 
know that memory has grown by approximately X over some 
baseline.

_PyObject_DebugMallocStats has most of the desired 
information.  (Except that it is usually #ifdef ed out entirely, 
instead of being exposed as a python function.)

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2004-03-22 13:31

Message:
Logged In: NO 

What is the minimum worth including?

For instance, objmalloc.c states that it never returns an arena 
to the operating system, so just (narenas * ARENA_SIZE) is an 
upper bound on python-managed generic small object memory. 
Whether that is enough of the used memory (or a tight enough 
bound, in case of good VM), I can't yet judge.

When I have wanted it, it was to ask either "Do I have to 
shrink my cache yet?" or "Is this getting slow because I've 
taken too much memory for the system it is running on".  Both 
probably require more a accurate version.  (Walking the 
individual arenas and pools?)



----------------------------------------------------------------------

Comment By: Jean M. Brouwers (jbrouwers)
Date: 2003-07-28 19:57

Message:
Logged In: YES 
user_id=832557

It would be useful to get some figure(s) about the memory
usage of the current process.

For example, just the 'high water mark' returned by sbrk(0)
on uni* is very helpful in many cases.

I tried the resource.getrusage(resource.RUSAGE_SELF)
function but that does return all zero's in slice [2:6] on
my uni* system.  Not sure whether that is an artifact of the
O/S or Python.

----------------------------------------------------------------------

Comment By: Anthony Baxter (anthonybaxter)
Date: 2002-04-10 10:03

Message:
Logged In: YES 
user_id=29957

You _can_ do this for most O/Ss, true. The problem is that
it's something that's extremely operating system specific,
and, worse yet, varies significantly between operating 
system releases. Have a look at the source for the system
utility 'top' one day - most of it is in the 'operating 
system specific' section, and there's all sorts of horrible
horrible hoops you have to jump through for different releases.

In addition, the notion of "how much memory is being used"
isn't always that clear in the case of things like shared
libraries.

On the plus side, implementing this would move the HP/UX
threading problem to #2 on the list of "most annoying 
operating-system dependent bugs".

----------------------------------------------------------------------

Comment By: Allan Crooks (amc1)
Date: 2002-04-09 10:45

Message:
Logged In: YES 
user_id=39733

Thanks for the response.

Rather than having an internal way of figuring out how much 
memory is used, is there no external way that we can get 
the OS to return how big the process is?

I know that it is less than elegant, and would have to have 
different platform-dependent code to do this, but it is 
still an idea. I know that Jython can quiz the VM to see 
how much memory is being used, and I'll assume that you can 
do that with different OS's as well...

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-04-08 12:38

Message:
Logged In: YES 
user_id=31435

Allan, this isn't easy, so the cost/benefit ratio is high.  
For the most part, Python gets memory from the system 
malloc, and doesn't even try to keep track of it now; nor 
has it any idea how much overhead (padding, control bytes) 
the system malloc adds; nor is there a portable interface 
to C's malloc for finding out such things.

Still, I agree it would be nice to have such things <wink>.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=540952&group_id=5470