[New-bugs-announce] [issue14573] json iterencode can not handle general iterators

[Aaron Staley]

New submission from Aaron Staley <usaar33 at gmail.com>:

The json library's encoder includes a function called 'iterencode'.  iterencode allows for encoding to be streamed; as tokens are produced they are yielded. This allows for the encoded object to be streamed to a file, over a socket, etc. without being placed all into memory.

Unfortunately, iterencode cannot encode general iterators.  This significantly limits the usefulness of the function.  For my use case I wish to convert a large stream (iterator) of objects into json.  Unfortunately, I currently have to:

A. Bring all the objects into memory by encasing the iterator in a list()
B. Make a hack where I subclass list and making that object's __iter__ function return my desired iterator.

The problem is that the json library explicitly checks for something being a list:

                if isinstance(value, (list, tuple)):
                    chunks = _iterencode_list(value, _current_indent_level)

It would work just as well (and be more pythonic) to see if the value supports the iterator protocol:
                if isinstance(value, collections.Iterable):
                    chunks = _iterencode_list(value, _current_indent_level)


Erroring example:

>>> import json
>>> e = json.JSONEncoder()
>>> r = xrange(20)
>>> gen = e.iterencode(r)
<generator object _iterencode at 0x14a5460>
>>> next(gen)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/json/encoder.py", line 419, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.2/json/encoder.py", line 170, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: xrange(0, 20) is not JSON serializable

----------
components: Library (Lib)
messages: 158239
nosy: Aaron.Staley
priority: normal
severity: normal
status: open
title: json iterencode can not handle general iterators
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14573>
_______________________________________