[Python-Dev] Accepting PEP 3154 for 3.4?

[Guido van Rossum]

Well, both fixed 8-byte framing and variable-size framing it introduces a
new way of representing numbers in the stream, which means that everyone
parsing and generating pickles must be able to support both styles. (But
fixed is easier since the XXX8 opcodes use the same format.)

I'm thinking of how you correctly read a pickle from a non-buffering pipe
with the minimum number of read() calls without ever reading beyond the end
of a valid pickle. (That's a requirement, right?)

If you know it's protocol 4:
    with fixed framing:
        read 10 bytes, that's the magic word plus first frame; then you can
start buffering
    with variable framing:
        read 3 bytes, then depending on the 3rd byte read some more to find
the frame size; then you can start buffering
    with mandatory frame opcode:
        pretty much the same
    with optional frame opcode:
        pretty much the same (the 3rd byte must be a valid opcode, even if
it isn't a frame opcode)

if you don't know the protocol number:
    read the first byte, then read the second byte (or not if it's not
explicitly versioned), then you know the protocol and can do the rest as
above


On Tue, Nov 19, 2013 at 11:11 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> On Tue, 19 Nov 2013 11:05:45 -0800
> Guido van Rossum <guido at python.org> wrote:
>
> > So using an opcode for framing is out? (Sorry, I've lost track of the
> > back-and-forth.)
>
> It doesn't seem to bring anything, and it makes the overhead worse for
> tiny pickles (since it will be two bytes at least, instead of one byte
> with the current variable length encoding proposal).
>
> If overhead doesn't matter, I'm fine with keeping a simple 8-bytes
> frame size :-)
>
> Regards
>
> Antoine.
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/fe342929/attachment.html>