Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
Greg Ewing
greg.ewing at canterbury.ac.nz
Wed Oct 2 01:27:54 EDT 2024
On 2/10/24 12:26 pm, avi.e.gross at gmail.com wrote:
> The real problem is how the JSON is set up. If you take umpteen data
> structures and wrap them all in something like a list, then it may be a tad
> hard to stream as you may not necessarily be examining the contents till the
> list finishes gigabytes later.
Yes, if you want to process the items as they come in, you might
be better off sending a series of separate JSON strings, rather than
one JSON string containing a list.
Or, use a specialised JSON parser that processes each item of the
list as soon as it's finished parsing it, instead of collecting the
whole list first.
--
Greg
More information about the Python-list
mailing list