MemoryError in data conversion
Peter Otten
__peter__ at web.de
Tue Apr 15 05:59:00 EDT 2014
Gregory Ewing wrote:
> Mok-Kong Shen wrote:
>> I have yet a question out of curiosity: Why is my 2nd list structure,
>> that apparently is too complex for handling by eval and json, seemingly
>> not a problem for pickle?
>
> Pickle is intended for arbitrary data structures, so it
> is designed to be able to handle deeply-nested and/or
> recursive data. Eval only has to handle nesting to depths
> likely to be encountered in source code. Apparently the
> json parser also assumes you're not going to be using
> very deep nesting.
But pickle does have the same limitation:
>>> def check(load, dump):
... items = []
... try:
... for i in range(10**6):
... assert load(dump(items)) == items
... items = [items]
... except RuntimeError:
... return i
...
>>> check(json.loads, json.dumps)
994
>>> check(pickle.loads, pickle.dumps)
499
Mok-Kong Shen, for pickle and json you can increase the limit a bit:
>>> import sys
>>> sys.setrecursionlimit(2000)
>>> check(json.loads, json.dumps)
1994
>>> check(pickle.loads, pickle.dumps)
999
But be careful, if you choose the limit too high you'll see Python react
like any other C program:
>>> sys.setrecursionlimit(100000)
>>> items = []
>>> for i in range(100000):
... items = [items]
...
>>> s = pickle.dumps(items)
Segmentation fault
For literal_eval() the limit is unfortunately hard-coded in the C source.
Mok-Kong Shen wrote:
> What I need is to have my (complicated) list to be put into
> a bytearray, do some proceesing, transfer it to the recipient.
> The recipient reverses the processing, obtains a bytearray
> that is the same as my original one and gets from it the same
> list as mine. Using pickle I can manage to do it (in fact I
> did try and succeed with my 2nd list), but that's IMHO a
> rather unnatural/awkward work-around. (It means that I have
> to pickle out the list to a file and read in the content of
> the file in order to have it as a bytearray etc. etc.)
The limit has been increased before, see
http://bugs.python.org/issue1881
and maybe you can get the core developers to increase it again, but
generally speaking the existence of a recursion limit is the price you pay
for easy interfacing with C.
literal_eval() could allocate its stack dynamically, but that would probably
complicate the code so that return on investment is questionable.
More information about the Python-list
mailing list