getting memory usage of varaibles
Ned Batchelder
ned at nedbatchelder.com
Wed May 3 10:00:44 EDT 2017
On Wednesday, May 3, 2017 at 8:40:09 AM UTC-4, Larry.... at gmail.com wrote:
> On Wed, May 3, 2017 at 8:29 AM, Ned Batchelder <ned at nedbatchelder.com> wrote:
> > On Tuesday, May 2, 2017 at 11:49:37 PM UTC-4, Larry.... at gmail.com wrote:
> >> On Tue, May 2, 2017 at 7:01 PM, Erik <python at lucidity.plus.com> wrote:
> >> > On 02/05/17 23:28, Larry Martell wrote:
> >> >>>>
> >> >>>> Anyone have any thoughts on how I can monitor the variables' memory
> >> >>>> usage as the script runs?
> >> >>>
> >> >>>
> >> >>> This is application-specific, but sometimes it helps to look at the
> >> >>> objects' types, or even their values.
> >> >>
> >> >>
> >> >> The types are dict and list, so they are not very useful, nor were the
> >> >> values.
> >> >
> >> >
> >> > Given your description, I would hazard a guess that you have a circular
> >> > reference somewhere (such that some objects may remain referenced after you
> >> > believe they should have been discarded). You could _probably_ break that
> >> > circle with some carefully placed 'del' statements somewhere - but you'd
> >> > need to analyze your data structures (and closures) first to see if that's
> >> > the case.
> >>
> >> This script is part of a django project, but it is not part of the
> >> core web django app. It's a stand alone script that is run from the
> >> command line. I added some profiling to the script and I found that
> >> the object count was increasing repidly with each loop (and of course
> >> with that the memory usage). I originally wrote it using the django
> >> ORM but now I have rewritten it using MySQLdb and embedded SQL. That
> >> stopped the increase in the object count and drastically reduced the
> >> memory leakage, but did not stop it. But at least now I can run it
> >> without it using all of the memory and getting killed. I'd still love
> >> to know where it's leaking memory, but I may not get the time for
> >> that.
> >
> > When you were using the Django ORM, did you have settings.DEBUG=True?
> > If so, Django stores the SQL queries for debugging.
>
> Yes, I did have DEBUG=True, but I wasn't trying to see the SQL Django
> was generating. But looking at the objects I found that when the
> script entered its main loop there were around 28K objects, and each
> pass through the loop would add around 10K new objects. When I looked
> at their content most of them were lists of the column names of the
> table I was selecting from - many, many objects with the same values.
> When I switched to not using the ORM there were 28K objects throughout
> the entire run of the script (which was 100's of millions of
> iterations of its loop).
That sure sounds like you were seeing the stored SQL queries, and perhaps
their results. I'm not sure how extensive the saved information is. If
you still have the Django ORM version, try setting DEBUG=False.
--Ned.
More information about the Python-list
mailing list