[Baypiggies] json using huge memory footprint and not releasing

Heikki Toivonen hjtoi at comcast.net
Fri Jun 15 23:29:06 CEST 2012

You probably want a json parser that does not load the whole thing into memory. Check out ijson.

David Lawrence <david at bitcasa.com> wrote:

>When I load the file into json, pythons memory usage spike to about
>and I can't seem to get that memory to be released.  I put together a
>case that's very simple:
>with open("test_file.json", 'r') as f:
>    j = json.load(f)
>I'm sorry that I can't provide a sample json file, my test file has a
>of sensitive information, but for context, I'm dealing with a file in
>order of 240MB.  After running the above 2 lines I have the
>previously mentioned 1.8GB of memory in use.  If I then do "del j"
>usage doesn't drop at all.  If I follow that with a "gc.collect()" it
>doesn't drop.  I even tried unloading the json module and running
>I'm trying to run some memory profiling but heapy has been churning
>CPU for about an hour now and has yet to produce any output.
>Does anyone have any ideas?  I've also tried the above using cjson
>than the packaged json module.  cjson used about 30% less memory but
>otherwise displayed exactly the same issues.
>I'm running Python 2.7.2 on Ubuntu server 11.10.
>I'm happy to load up any memory profiler and see if it does better then
>heapy and provide any diagnostics you might think are necessary.  I'm
>hunting around for a large test json file that I can provide for anyone
>else to give it a go.
>Baypiggies mailing list
>Baypiggies at python.org
>To change your subscription options or unsubscribe:

Sent from my Android phone with K-9. Please excuse my brevity.

More information about the Baypiggies mailing list