Please help with MemoryError

[Steve Howell]

On Feb 11, 5:50 pm, Steven D'Aprano <st... at REMOVE-THIS-
cybersource.com.au> wrote:
> On Thu, 11 Feb 2010 15:39:09 -0800, Jeremy wrote:
> > My Python program now consumes over 2 GB of memory and then I get a
> > MemoryError.  I know I am reading lots of files into memory, but not 2GB
> > worth.

> > 2.    When do I need
> > to manually allocate/deallocate memory and when can I trust Python to
> > take care of it?
>
> You never need to manually allocate memory.
>
> You *may* need to deallocate memory if you make "reference loops", where
> one object refers to itself:
>
> l = []  # make an empty list
> l.append(l)  # add the list l to itself
>
> Python can break such simple reference loops itself, but for more
> complicated ones, you may need to break them yourself:
>
> a = []
> b = {2: a}
> c = (None, b)
> d = [1, 'z', c]
> a.append(d)  # a reference loop
>
> Python will deallocate objects when they are no longer in use. They are
> always considered in use any time you have them assigned to a name, or in
> a list or dict or other structure which is in use.
>
> You can explicitly remove a name with the del command. For example:
>
> x = ['my', 'data']
> del x
>
> After deleting the name x, the list object itself is no longer in use
> anywhere and Python will deallocate it. But consider:
>
> x = ['my', 'data']
> y = x  # y now refers to THE SAME list object
> del x
>
> Although you have deleted the name x, the list object is still bound to
> the name y, and so Python will *not* deallocate the list.
>
> Likewise:
>
> x = ['my', 'data']
> y = [None, 1, x, 'hello world']
> del x
>
> Although now the list isn't bound to a name, it is inside another list,
> and so Python will not deallocate it.
>

Another technique that comes up some time is that you have a list of
objects that you are processing:

x = [obj1, obj2, obj3, obj4]

When you are done processing obj1, you want to remove the reference to
it, but you do not necessarily want to change the list itself.  You
can break the reference by saying "x[0] = None" when you are done
handling obj1.  Of course, if you can avoid creating the list in the
first place, as some people have suggested, then you really get a
savings.

The setting-to-None technique is also occasionally useful with
objects, where you can say foo.bar = None when you are done with "bar"
but not with "foo."  Of course, the need to use such a technique often
points out a deeper code smell with Foo itself, but I've seen it come
up.

Steven's examples of a reference loop are deliberately simplified, of
course, but the chain of references in a real world program can get
quite long, and there's often great savings to be reaped if you can
break the "keystone" reference, so to speak.  In other words, breaking
just one reference often allows other references to fall down like
dominos.