Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Barry barry at barrys-emacs.org
Mon Sep 30 11:30:19 EDT 2024



> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list at python.org> wrote:
> 
> 
> import polars as pl
> pl.read_json("file.json")
> 
> 

This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.

Barry




More information about the Python-list mailing list