Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Thomas Passin]

On 9/30/2024 11:30 AM, Barry via Python-list wrote:
> 
> 
>> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list at python.org> wrote:
>>
>>
>> import polars as pl
>> pl.read_json("file.json")
>>
>>
> 
> This is not going to work unless the computer has a lot more the 60GiB of RAM.
> 
> As later suggested a streaming parser is required.

Streaming won't work because the file is gzipped.  You have to receive 
the whole thing before you can unzip it. Once unzipped it will be even 
larger, and all in memory.