[Alexandre Vassalotti]
Looking at the different options available to us:
1A. Mandatory framing (+) Allows the internal buffering layer of the Unpickler to rely on the presence of framing to simplify its implementation. (-) Forces all implementations of pickle to include support for framing if they want to use the new protocol. (-) Cannot be removed from future versions of the Unpickler without breaking protocols which mandates framing. 1B. Optional framing (+) Could allow optimizations to disable framing if beneficial (e.g., when pickling to and unpickling from a string).
Or to slash the size of small pickles (an 8-byte size field can be more than half the total pickle size).
2A. With explicit FRAME opcode (+) Makes optional framing simpler to implement. (+) Makes variable-length encoding of the frame size simpler to implement. (+) Makes framing visible to pickletools.
(+) Adds (a little) redundancy for sanity checking.
(-) Adds an extra byte of overhead to each frames. 2B. No opcode
3A. With fixed 8-bytes headers (+) Is simple to implement (-) Adds overhead to small pickles. 3B. With variable-length headers (-) Requires Pickler implemention to do extra data copies when pickling to strings.
4A. Framing baked-in the pickle protocol (+) Enables faster implementations 4B. Framing through a specialized I/O buffering layer (+) Could be reused by other modules
I may change my mind as I work on the implementation, but at least for now, I think the combination of 1B, 2A, 3A, 4A will be a reasonable compromise here.
At this time I'd make the same choices, so don't expect an argument from me ;-) Thank you!