When I load the file into json, pythons memory usage spike to about 1.8GB and I can't seem to get that memory to be released. I put together a test case that's very simple:<div><br></div><div>with open("test_file.json", 'r') as f:</div>
<div> j = json.load(f)</div><div><br></div><div>I'm sorry that I can't provide a sample json file, my test file has a lot of sensitive information, but for context, I'm dealing with a file in the order of 240MB. After running the above 2 lines I have the previously mentioned 1.8GB of memory in use. If I then do "del j" memory usage doesn't drop at all. If I follow that with a "gc.collect()" it still doesn't drop. I even tried unloading the json module and running another gc.collect.</div>
<div><br></div><div>I'm trying to run some memory profiling but heapy has been churning 100% CPU for about an hour now and has yet to produce any output.</div><div><br></div><div>Does anyone have any ideas? I've also tried the above using cjson rather than the packaged json module. cjson used about 30% less memory but otherwise displayed exactly the same issues.</div>
<div><br></div><div>I'm running Python 2.7.2 on Ubuntu server 11.10.</div><div><br></div><div>I'm happy to load up any memory profiler and see if it does better then heapy and provide any diagnostics you might think are necessary. I'm hunting around for a large test json file that I can provide for anyone else to give it a go.</div>