Searching for solution: Severe memory leak, HELP!
randall_young at hotmail.com
Sat Jan 18 23:55:47 CET 2003
"Steve Holden" <sholden at holdenweb.com> wrote in message news:<RGeW9.5340$bs2.4910 at news2.central.cox.net>...
> "Randy Young" <randall_young at hotmail.com> wrote in message
> news:8338c7cc.0301180618.5c67da85 at posting.google.com...
> > Sirs,
> > I have a very large Python application squarely in my lap, that's
> > leaking almost 20mb of main memory an hour!
> > I spent most of last night looking through various postings, and not
> > surprisingly, I'm wondering if I have a "circular reference" at work.
> > Specifically, something happening with the heavy use of the "exec"
> > library call.
> > Here's a code snippet, can anyone see anything grievously wrong? I
> > understand that we may need to look at WHAT is being "exec'd", but I
> > thought this might be a place to start...
> > ---- CODE BEGINS
> > #import the module, then execute the probe
> > where = 'import'
> > win32event.WaitForSingleObject(self.iSemaphore, win32event.INFINITE)
> > try:
> > exec ("import %s" % probeInfo['ImportName'])
> > except Exception, e:
> > ret = "Import error on " + str(probeInfo['ImportName']) + " " +
> > str(e)
> > self.ExceptOut(e, 'Exception importing Probe ID ' +
> > str(probeInfo['ID']))
> > win32event.ReleaseSemaphore(self.iSemaphore, 1)
> > if (ret == None):
> > where = 'exec'
> > try:
> > s = "ret = %s.%s (probeInfo)" % (probeInfo['ImportName'],
> > probeInfo['Invoke'])
> > exec (s)
> > # ... much more yada-yada ...
> > ---- CODE ENDS
> > As you can see there are "exec(s)" all over the place in the module,
> > one mistake would be magnified many times! I'm trying to understand if
> > I should be forcing a garbage collection or some other obvious thing?
> > Thanks in advance for the consideration of a reply.
> Well, firstly it might be better to use
> exec "import ..."
> Also note, by the way, that exec is a statement and not a function, so the
> parentheses are redundant.
> What are you trying to import? It's conceivable that failed imports of a
> badly-constructed extension might leak memory. Also note that if you leave
> pointers to the stack frame lying about in "except" clauses that can cause
> Just a few random thoughts.
As an early experiment I did test some modules that have C ext's by
writing a small driver program(s) in Python and calling them say
10,000 times. In fact the program(s) did not seem to lose a byte! So I
assumed the ext's were and are solid.
The whole purpose of the import and exec statement is to provide
dynamic extension of the core service at runtime.
Here for example is a simple ping probe... it's called by the
aforementioned module, the I suspect to be the culprit. That is when I
disbable running probes in the GUI, and therefore no probes run at
all, the leak stops.
---- CODE BEGINS
eStr = ''
min = 0xFFFF
max = 0
failCount = 0
successCount = 0
alltime = 0
avg = 0
# extract parameters
eStr = 'Ping probe error -- number of packets parameter missing'
numberOfPackets = int(probe['params']['NumberOfPackets'])
eStr = 'Ping probe error -- failed packet limit parameter
failedPacketLimit = int(probe['params']['FailedPacketLimit'])
eStr = 'Ping probe error -- failed timit limit parameter missing'
failedTimeLimit = int(probe['params']['PingTimeout'])
eStr = 'Ping probe error -- proximal address missing'
address = probe['ProximalAddress']
# okay, do the ping thing
print "PING " + address + ' n ' + str(numberOfPackets)
for i in range(numberOfPackets):
# do a single ping packet at a time for finer grained control
retList = pingtr.ping(address, 1, float(failedTimeLimit)/1000 )
if(retList['success'] == 1):
if(int(retList['avg']) > failedTimeLimit):
eStr = "Ping timeout exceeded!"
if(min > int(retList['min'])):
min = int(retList['min'])
if(max < int(retList['max'])):
max = int(retList['max'])
alltime = alltime + int(retList['avg'])
successCount = successCount + 1
failCount = failCount + 1
if (successCount > 0):
avg = alltime / successCount
if (min == 0xFFFF): # never got set
min = 0
print "Ping results: max=" + str(max) + "ms min=" + str(min) +
"ms avg=" + str(avg) \
+ "ms success=" + str(successCount) + " fail=" + str(failCount)
probe['results']['AvgTransitTime']['Result'] = avg
if(failCount == 0):
probe['results']['PctPacketLoss']['Result'] = 0
int(float(failCount)/float(numberOfPackets) * 100)
if (failCount == numberOfPackets):
eStr = "Device or host appears to be down!"
return (prperrors.ERROR_HOSTDOWN, eStr)
if (failCount > failedPacketLimit):
eStr = 'Packet failure threshold exceeded: failed %d packets of %d'
% (failCount, numberOfPackets)
except Exception, e:
eStr = "Exception raised during ping probe! " + str(e)
---- CODE ENDS
This module is intended to be invoked on the fly, understand what it's
supposed to do, and return an appropriate result.
I'm a little confused by the reference to the "pointer to a stack
frame in/on a exception statment". I'm not used to thinking of Python
in a "pointer sense".
Probably just my inexperience, could you give an example?
My next job on this will to run Rational's Purity on this and see what
(Never did a greener Pythoneer, have a bigger program to fix :( .)
More information about the Python-list