[CentralOH] Django Management Command Memory Usage

[Austin Godber]

Hi,
I've not had to solve this problem myself, yet.  But if I had to
there'd be two things I'd look into.  Model pagination and perhaps
using the query cache iterator method.

#3 here:
http://blog.davidziegler.net/post/548363214/some-common-django-orm-pitfalls

First comment on:
http://dayhacker.blogspot.com/2009/02/why-django-orm-sucks-it-takes-hell-lot.html

I'd be interested to know your results if you try either of these.

Austin


On Mon, Jun 4, 2012 at 7:28 AM,  <jep200404 at columbus.rr.com> wrote:
> How can I reduce the memory usage in a Django management command?
> I have some Django code like follows in a management program:
>
> class Command(BaseCommand):
> ...
>    def handle(self, *args, **options):
>        for record in Places.objects.all():
>            if record.x and record.y:
>                record.point = (
>                    Point(float(record.x)/1000.,
>                    float(record.y)/1000.))
>            else:
>                record.point = None
>            record.save()
>        django.db.connection.close()
>
> In the settings.py file I have:
>
> DEBUG = False
>
> Places has millions of rows.
> top reveals that the program is using 18.6 Gigabytes of memory.
> How can I reduce that memory usage?
> Am I neglecting to close or release something?
>
> The only dox I'm finding about memory use related to query sets
> advise to use iterators instead of converting to a list.
> I'm already following that advice, but I'm not finding
> further guidance about memory use about record modification.
>
> Since DEBUG is False, I've already heeding the following.
>
>   https://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> I have found that the following might be nice,
> but doubt it addresses the memory issue.
>
>            record.save(update_fields=['point'])
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> http://mail.python.org/mailman/listinfo/centraloh