multythreading app memory consumption
tier at inbox.ru
Mon Oct 23 01:31:28 CEST 2006
I've just faced with very nasty memory consumption problem.
I have a multythreaded app with 150 threads which use the only and the
same function - through urllib2 it just gets the web page's html code
and assigns it to local variable. On the next turn the variable is
overritten with another page's code. At every moment the summary of
values of the variables containig code is not more than 15Mb (I've just
invented a tricky way to measure this). But during the first 30 minutes
all the system memory (512Mb) is consumed and 'MemoryError's is arising.
Why is it so and how can I limit the memory consumption in borders, say,
400Mb? Maybe there is a memory leak there?
The test app code:
Q = Queue.Queue()
for i in rez: #rez length - 5000
url = Q.get()
opener = urllib2.urlopen(url)
data = opener.read()
sys.stderr.write('ERROR: %s\n' % traceback.format_exc())
for i in xrange(150):
new_thread = threading.Thread(target=checker)
More information about the Python-list