[issue6594] json C serializer performance tied to structure depth on some systems

Valentin Kuznetsov report at bugs.python.org
Wed Dec 2 19:47:08 CET 2009


Valentin Kuznetsov <vkuznet at gmail.com> added the comment:

Hi,
I'm sorry for delay, I was busy. Here is a test data file:
http://www.lns.cornell.edu/~vk/files/mangled.json

Its size is 150 MB, 50MB less of original, due to scrambled values I was 
forced to do.

The tests with stock json module in python 2.6.2 is 2GB
source = open('mangled.json', 'r')
data = json.load(source)

Using simplejson 2.0.9 from PyPi I saw the same performance, please note 
_speedups.so C module was compiled.

Using cjson module, I observed 180MB of RAM utilization
source = open('mangled.json', 'r')
data = cjson.encode(source.read())

cjson is about 10 times faster!

I re-factor code which deals with XML version of the same data and I was 
able to process it using cElementTree only using 20MB (!) of RAM.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6594>
_______________________________________


More information about the Python-bugs-list mailing list