[issue14288] Make iterators pickleable

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson <kristjan at ccpgames.com>:

A common theme in many talks last year about cloud computing was the need to suspend execution, pickle state, and resume it on a different node.  This patch is the result of last year's stackless sprint at pycon, finally completed and submitted for review.

Python does not currently support pickling of many run-time structures, but pickling for things like iterators is trivial.
A large piece of Stackless' branch is to make sure that various run-time constructs are pickleable, including function objects.  While this patch does not do that, it does add pickling for dictiter, and the lot.
This makes it possible to have compilcated data sets, iterate through them, and pickle them in a semi-consumed state.

Please note that a slight hack is needed to pickle some iterators.  Many of these classes are namely "hidden" and there is no way to access their constructors by name.  instead, an unpickling trick is to invoke "iter" on an object of the target type instead.  Not the most elegant solution but I didn't want to complicate matters by adding iterator classes into namespaces.  Where should stringiter live for example?  Be a builtin like str?

We also didn't aim to make all iterators copy.copy-able using the __reduce__ protocol.  Some iterators actually use internal iterators themselves, and if a (non-deep) copy were to happen, we would have to shallow copy those internal objects.  Instead, we just return the internal iterator object directly from __reduce__ and allow recursive pickling to proceed.

----------
files: pickling.patch
keywords: patch
messages: 155626
nosy: krisvale, loewis, michael.foord
priority: normal
severity: normal
status: open
title: Make iterators pickleable
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file24822/pickling.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14288>
_______________________________________