multythreading app memory consumption
tier at inbox.ru
Mon Oct 23 10:07:47 CEST 2006
Dennis Lee Bieber wrote:
> On Mon, 23 Oct 2006 03:31:28 +0400, Roman Petrichev <tier at inbox.ru>
> declaimed the following in comp.lang.python:
>> Hi folks.
>> I've just faced with very nasty memory consumption problem.
>> I have a multythreaded app with 150 threads which use the only and the
>> same function - through urllib2 it just gets the web page's html code
>> and assigns it to local variable. On the next turn the variable is
>> overritten with another page's code. At every moment the summary of
>> values of the variables containig code is not more than 15Mb (I've just
>> invented a tricky way to measure this). But during the first 30 minutes
>> all the system memory (512Mb) is consumed and 'MemoryError's is arising.
>> Why is it so and how can I limit the memory consumption in borders, say,
>> 400Mb? Maybe there is a memory leak there?
> How much stack space gets allocated for 150 threads?
Actually I don't know. How can I get to know this?
>> Q = Queue.Queue()
>> for i in rez: #rez length - 5000
> Can't be the "test code" as you don't show the imports or where
> "rez" is defined.
Isn't it clear that "rez" is just a list of 5000 urls? I cannot place it
here, but believe me all of them are not big - "At every moment the
summary of values of the variables containig code is not more than 15Mb"
More information about the Python-list