On Tue, Nov 19, 2013 at 2:09 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Well, I don't think it's a big deal to add a FRAME opcode if it doesn't change the current framing logic. I'd like to defer to Alexandre on this one, anyway.
Looking at the different options available to us: 1A. Mandatory framing (+) Allows the internal buffering layer of the Unpickler to rely on the presence of framing to simplify its implementation. (-) Forces all implementations of pickle to include support for framing if they want to use the new protocol. (-) Cannot be removed from future versions of the Unpickler without breaking protocols which mandates framing. 1B. Optional framing (+) Could allow optimizations to disable framing if beneficial (e.g., when pickling to and unpickling from a string). 2A. With explicit FRAME opcode (+) Makes optional framing simpler to implement. (+) Makes variable-length encoding of the frame size simpler to implement. (+) Makes framing visible to pickletools. (-) Adds an extra byte of overhead to each frames. 2B. No opcode 3A. With fixed 8-bytes headers (+) Is simple to implement (-) Adds overhead to small pickles. 3B. With variable-length headers (-) Requires Pickler implemention to do extra data copies when pickling to strings. 4A. Framing baked-in the pickle protocol (+) Enables faster implementations 4B. Framing through a specialized I/O buffering layer (+) Could be reused by other modules I may change my mind as I work on the implementation, but at least for now, I think the combination of 1B, 2A, 3A, 4A will be a reasonable compromise here.