[CentralOH] Django Management Command Memory Usage

[Kurtis Mullins]

Hey,

It looks like you've got the fat trimmed off this one about as much as you
can. I'd say filter out your results but even then you're still using every
result. I'd recommend modifying this to run as raw SQL and I'm sure your
memory usage will go down significantly. Pulling in this many records as
Python Objects and conversely creating new Python objects on top of that
(your Point objects) is most likely the cause of this memory usage.

On Mon, Jun 4, 2012 at 10:28 AM, <jep200404 at columbus.rr.com> wrote:

> How can I reduce the memory usage in a Django management command?
> I have some Django code like follows in a management program:
>
> class Command(BaseCommand):
> ...
>    def handle(self, *args, **options):
>        for record in Places.objects.all():
>            if record.x and record.y:
>                record.point = (
>                    Point(float(record.x)/1000.,
>                    float(record.y)/1000.))
>            else:
>                record.point = None
>            record.save()
>        django.db.connection.close()
>
> In the settings.py file I have:
>
> DEBUG = False
>
> Places has millions of rows.
> top reveals that the program is using 18.6 Gigabytes of memory.
> How can I reduce that memory usage?
> Am I neglecting to close or release something?
>
> The only dox I'm finding about memory use related to query sets
> advise to use iterators instead of converting to a list.
> I'm already following that advice, but I'm not finding
> further guidance about memory use about record modification.
>
> Since DEBUG is False, I've already heeding the following.
>
>
> https://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> I have found that the following might be nice,
> but doubt it addresses the memory issue.
>
>            record.save(update_fields=['point'])
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> http://mail.python.org/mailman/listinfo/centraloh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20120604/cf0b1871/attachment.html>