Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Thomas Passin]

On 9/30/2024 11:30 AM, Barry via Python-list wrote:
> 
> 
>> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list at python.org> wrote:
>>
>>
>> import polars as pl
>> pl.read_json("file.json")
>>
>>
> 
> This is not going to work unless the computer has a lot more the 60GiB of RAM.
> 
> As later suggested a streaming parser is required.

There is also the json-stream library, on PyPi at

https://pypi.org/project/json-stream/