Searching for solution: Severe memory leak, HELP!

[Randy Young]

"Steve Holden" <sholden at holdenweb.com> wrote in message news:<RGeW9.5340$bs2.4910 at news2.central.cox.net>...
> "Randy Young" <randall_young at hotmail.com> wrote in message
> news:8338c7cc.0301180618.5c67da85 at posting.google.com...
> > Sirs,
> >
> > I have a very large Python application squarely in my lap, that's
> > leaking almost 20mb of main memory an hour!
> >
> > I spent most of last night looking through various postings, and not
> > surprisingly, I'm wondering if I have a "circular reference" at work.
> > Specifically, something happening with the heavy use of the "exec"
> > library call.
> >
> > Here's a code snippet, can anyone see anything grievously wrong? I
> > understand that we may need to look at WHAT is being "exec'd", but I
> > thought this might be a place to start...
> >
> >
> > ---- CODE BEGINS
> >
> >
> > #import the module, then execute the probe
> >   where = 'import'
> >   win32event.WaitForSingleObject(self.iSemaphore, win32event.INFINITE)
> >     try:
> >       exec ("import %s" % probeInfo['ImportName'])
> >     except Exception, e:
> >       ret = "Import error on " + str(probeInfo['ImportName']) + " " +
> > str(e)
> >       self.ExceptOut(e, 'Exception importing Probe ID ' +
> > str(probeInfo['ID']))
> >     win32event.ReleaseSemaphore(self.iSemaphore, 1)
> >
> >     if (ret == None):
> >       where = 'exec'
> >       try:
> > s = "ret = %s.%s (probeInfo)" % (probeInfo['ImportName'],
> > probeInfo['Invoke'])
> >         exec (s)
> >
> > # ... much more yada-yada ...
> >
> > ---- CODE ENDS
> >
> > As you can see there are "exec(s)" all over the place in the module,
> > one mistake would be magnified many times! I'm trying to understand if
> > I should be forcing a garbage collection or some other obvious thing?
> >
> > Thanks in advance for the consideration of a reply.
> >
> 
> Well, firstly it might be better to use
> 
>     __import__(...)
> 
> than
> 
>     exec "import ..."
> 
> Also note, by the way, that exec is a statement and not a function, so the
> parentheses are redundant.
> 
> What are you trying to import? It's conceivable that failed imports of a
> badly-constructed extension might leak memory. Also note that if you leave
> pointers to the stack frame lying about in "except" clauses that can cause
> leakage.
> 
> Just a few random thoughts.
> 
> regards

Sir,

As an early experiment I did test some modules that have C ext's by
writing a small driver program(s) in Python and calling them say
10,000 times. In fact the program(s) did not seem to lose a byte! So I
assumed the ext's were and are solid.

The whole purpose of the import and exec statement is to provide
dynamic extension of the core service at runtime.

Here for example is a simple ping probe... it's called by the
aforementioned module, the I suspect to be the culprit. That is when I
disbable running probes in the GUI, and therefore no probes run at
all, the leak stops.

---- CODE BEGINS

import win32api
import BaseClasses
import pingtr
import prperrors

def ping(probe):
  try:
    eStr = ''
    min = 0xFFFF
    max = 0
    failCount = 0
    successCount = 0
    alltime = 0
    avg = 0
		
    # extract parameters
     eStr = 'Ping probe error -- number of packets parameter missing'
     numberOfPackets = int(probe['params']['NumberOfPackets'])
     eStr = 'Ping probe error -- failed packet limit parameter
missing'
     failedPacketLimit = int(probe['params']['FailedPacketLimit'])
     eStr = 'Ping probe error -- failed timit limit parameter missing'
     failedTimeLimit = int(probe['params']['PingTimeout'])
     eStr = 'Ping probe error -- proximal address missing'
     address = probe['ProximalAddress']

		
     # okay, do the ping thing		
     print "PING " + address + ' n ' + str(numberOfPackets)

     for i in range(numberOfPackets):	
      # do a single ping packet at a time for finer grained control
      retList = pingtr.ping(address, 1, float(failedTimeLimit)/1000 )
	if(retList['success'] == 1):
	  if(int(retList['avg']) > failedTimeLimit):
	  eStr = "Ping timeout exceeded!"
	  print eStr
          return(prperrors.ERROR_TIMEOUT,eStr)
	if(min > int(retList['min'])):
	  min = int(retList['min'])
	if(max < int(retList['max'])):
	   max = int(retList['max'])
	alltime = alltime + int(retList['avg'])
	   successCount = successCount + 1
	     else:
	     failCount = failCount + 1
	     win32api.Sleep(500)
		
	if (successCount > 0):
	  avg = alltime / successCount
			
	if (min == 0xFFFF): 	# never got set
	  min = 0

        print "Ping results: max=" + str(max) + "ms min=" + str(min) +
"ms avg=" + str(avg) \
			  + "ms success=" + str(successCount) + " fail=" + str(failCount)
		
	probe['results']['AvgTransitTime']['Result'] = avg
	  if(failCount == 0):
	   probe['results']['PctPacketLoss']['Result'] = 0
	   else:
	probe['results']['PctPacketLoss']['Result'] =
int(float(failCount)/float(numberOfPackets) * 100)
			
	if (failCount == numberOfPackets):
          eStr = "Device or host appears to be down!"
	   print eStr
	   return (prperrors.ERROR_HOSTDOWN, eStr)
			
	if (failCount > failedPacketLimit):
	 eStr = 'Packet failure threshold exceeded: failed %d packets of %d'
% (failCount, numberOfPackets)
	 print eStr
	return (prperrors.ERROR_PACKET_LOSS,eStr)

	return None
	
	except Exception, e:
	 eStr = "Exception raised during ping probe! " + str(e)
	 print eStr
	 return (prperrors.ERROR_EXCEPTION,eStr)

---- CODE ENDS

This module is intended to be invoked on the fly, understand what it's
supposed to do, and return an appropriate result.

I'm a little confused by the reference to the "pointer to a stack
frame in/on a exception statment". I'm not used to thinking of Python
in a "pointer sense".

Probably just my inexperience, could you give an example? 

My next job on this will to run Rational's Purity on this and see what
it thinks.

 (Never did a greener Pythoneer, have a bigger program to fix :( .) 

Thanks again,

Regard,

Randy