Destruction of generator objects

[Graham Dumpleton]

On Aug 8, 8:28 am, Stefan Bellon <sbel... at sbellon.de> wrote:
> Hi all,
>
> I'm generating a binding from Python to C using SWIG. On the C side I
> have iterators over some data structures. On the Python side I
> currently use code like the following:
>
>     def get_data(obj):
>         result = []
>         iter = make_iter(obj)
>         while more(iter):
>             item = next(iter)
>             result.append(item)
>         destroy(iter)
>         return result
>
> Now I'd like to transform it to a generator function like the following
> in order to make it more memory and time efficient:
>
>     def get_data(obj):
>         iter = make_iter(obj)
>         while more(iter):
>             yield next(iter)
>         destroy(iter)
>
> But in the generator case, I have a problem if the generator object is
> not iterated till the StopIteration occurs, but if iteration is stopped
> earlier. In that case, the C iterator's destroy is not called, thus the
> resource is not freed.
>
> Is there a way around this? Can I add some sort of __del__() to the
> generator object so that in case of an early destruction of the
> generator object, the external resource is freed as well?
>
> I'm looking forward to hearing your hints!
>
> --
> Stefan Bellon

Perhaps read through:

  http://www.python.org/dev/peps/pep-0325/
  http://www.python.org/dev/peps/pep-0342/

The latter superseding the first.

Based on these what the WSGI specification:

  http://www.python.org/dev/peps/pep-0333/

did was mandate that the consumer of data from the generator
explicitly call the close() method on the generator no matter whether
all data was consumed or not, or whether an exception occurred.

    result = application(environ, start_response)
    try:
        for data in result:
            if data:    # don't send headers until body appears
                write(data)
        if not headers_sent:
            write('')   # send headers now if body was empty
    finally:
        if hasattr(result,'close'):
            result.close()

That the consumer called close() allowed generators to provide a
close() method to cleanup resources even though older version of
Python was being used which didn't support automatic means of close()
being called. In other words, it allowed the required cleanup and was
forward compatible with newer versions of Python.

Graham