[New-bugs-announce] [issue11299] Allow deepcopying and pickling paused generators

Ram Rachum report at bugs.python.org
Wed Feb 23 17:15:39 CET 2011

New submission from Ram Rachum <cool-rr at cool-rr.com>:

Please allow to deepcopy and to pickle paused generators, including all their state.

This is implemented in Pypy:

	Python 2.5.2 (335e875cb0fe, Dec 28 2010, 20:31:56)
	[PyPy 1.4.1] on win32
	Type "copyright", "credits" or "license()" for more information.
	DreamPie 1.1.1
	>>> import pickle, copy
	>>> def next(thing): # For compatibility
	...     return thing.next()
	>>> def g():
	...     for i in range(4):
	...         yield i
	>>> list(g())
	[0, 1, 2, 3]
	>>> live_generator = g()
	>>> next(live_generator)
	>>> next(live_generator)
Now `live_generator` holds a generator which is in the middle of its operation. It went through 0 and 1, and it still has 2 and 3 to yield.

We deepcopy it:	
	>>> live_generator_deepcopy = copy.deepcopy(live_generator)
The deepcopied generator assumes the same state of the original one. Let's exhaust it:	
	>>> list(live_generator_deepcopy)
	[2, 3]
	>>> list(live_generator_deepcopy)
Pypy also lets us pickle and unpickle the live generator:
	>>> live_generator_pickled = pickle.dumps(live_generator)
	>>> live_generator_unpickled = pickle.loads(live_generator_pickled)
	>>> list(live_generator_unpickled)
	[2, 3]
	>>> list(live_generator_unpickled)
And the original live generator was unchanged by all these operations:
	>>> list(live_generator)
	[2, 3]
	>>> list(live_generator)
All the above was demonstrated in Pypy. In Python 3.2, trying to pickle a live generator raises this exception:

	>>> pickle.dumps(live_generator)
	Traceback (most recent call last):
	  File "<stdin>", line 1, in <module>
	_pickle.PicklingError: Can't pickle <class 'generator'>: attribute lookup builtins.generator failed

And trying to deepcopy one raises this exception:
	>>> copy.deepcopy(live_generator)
	Traceback (most recent call last):
	  File "<stdin>", line 1, in <module>
	  File "c:\Python32\lib\copy.py", line 174, in deepcopy
		y = _reconstruct(x, rv, 1, memo)
	  File "c:\Python32\lib\copy.py", line 285, in _reconstruct
		y = callable(*args)
	  File "c:\Python32\lib\copyreg.py", line 88, in __newobj__
		return cls.__new__(cls, *args)
	TypeError: object.__new__(generator) is not safe, use generator.__new__()

It would be nice if Python 3.2 could pickle and deepcopy live generators.

components: Library (Lib)
messages: 129213
nosy: cool-RR
priority: normal
severity: normal
status: open
title: Allow deepcopying and pickling paused generators
type: feature request
versions: Python 3.3

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list