Memory problem with Python

soring at gmail.com soring at gmail.com
Mon Jun 18 08:06:47 CEST 2007


On Jun 17, 8:51 pm, Squzer Crawler <Squ... at gmail.com> wrote:
> i am developing distributed environment in my college using Python. I
> am using therads in client for downloading wepages. Even though i am
> reusing the thread, memory usage get increased. I don know why.? I am
> using BerkelyDB for URLQueue, BeautifulShop for Parsing the webpages.

      Isn't the increased memory resulted from storing the already
processed pages?

      Look first at all places where your code instantiates new
objects - and make sure you don't keep references to such objects that
are not needed anymore.

      Also, reusing threads has nothing to do with saving memory - but
with saving on thread creation time, if I understand your problem
description.




More information about the Python-list mailing list