Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Greg Ewing greg.ewing at canterbury.ac.nz
Tue Oct 1 17:48:24 EDT 2024


On 1/10/24 8:34 am, Left Right wrote:
> You probably forgot that it has to be _streaming_. Suppose you parse
> the first digit: can you hand this information over to an external
> function to process the parsed data? -- No! because you don't know the
> magnitude yet.

By that definition of "streaming", no parser can ever be streaming,
because there will be some constructs that must be read in their
entirety before a suitably-structured piece of output can be
emitted.

The context of this discussion about integers is the claim that
they *could* be parsed incrementally if they were written little
endian instead of big endian, but the same argument applies either
way.

-- 
Greg


More information about the Python-list mailing list