Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
Greg Ewing
greg.ewing at canterbury.ac.nz
Tue Oct 1 17:48:24 EDT 2024
On 1/10/24 8:34 am, Left Right wrote:
> You probably forgot that it has to be _streaming_. Suppose you parse
> the first digit: can you hand this information over to an external
> function to process the parsed data? -- No! because you don't know the
> magnitude yet.
By that definition of "streaming", no parser can ever be streaming,
because there will be some constructs that must be read in their
entirety before a suitably-structured piece of output can be
emitted.
The context of this discussion about integers is the claim that
they *could* be parsed incrementally if they were written little
endian instead of big endian, but the same argument applies either
way.
--
Greg
More information about the Python-list
mailing list