On Mon, 18 Nov 2013 16:44:59 -0600 Tim Peters <tim.peters@gmail.com> wrote:
[Tim]
But it has a different kind of advantage: PREFETCH was optional. As Guido said, it's annoying to bloat the size of small pickles (which may, although individually small, occur in great numbers) by 8 bytes each. There's really no point to framing small chunks of data, right?
[Antoine]
You can't know how much space the pickle will take until the pickling ends, though, which makes it difficult to decide whether you want to emit a PREFETCH opcode or not.
Ah, of course. Presumably the outgoing pickle stream is first stored in some memory buffer, right? If pickling completes before the buffer is first flushed, then you know exactly how large the entire pickle is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH part. Else do.
Yet another possibility: keep framing but use a variable-length encoding for the frame size: - first byte: bits 7-5: N (= frame size bytes length - 1) - first byte: bits 4-0: first 5 bits of frame size - remaning N bytes: remaining bits of frame size With this scheme, very small pickles have a one byte overhead; small ones a two byte overhead; and the max frame size is 2**61 rather than 2**64, which should still be sufficient. And the frame size is read using either one or two read() calls, which is efficient. Regards Antoine.