[Python-ideas] Direct byte<->int conversions (was Re: bitwise operations on bytes)

[Alexandre Vassalotti]

On Sat, Aug 15, 2009 at 8:14 AM, Mark Dickinson<dickinsm at gmail.com> wrote:
> I'm not convinced that it's valuable to a have a variable-size
> version of this;  I'd make size a required argument.
>
> The problem with the variable size version is that the choice of
> byte-length for the output for a given integer input is a little bit
> arbitrary.  For a particular requirement (producing code to conform
> with some existing serialization protocol, for example) it seems
> likely that the choice Python makes will disagree with what's
> required by that protocol, so that size still has to be given explicitly.
> On the other hand, if a user just wants a quick and easy way
> to serialize ints, without caring about the exact form of the
> serialization, then there are number of solutions already
> available within Python.
>

Well, the only use-case in the standard library I found (i.e.,
simplifying encode_long() and decode_long() in pickle.py) needed the
variable-length version. However, unlike I originally thought, the
variable length version is not difficult to emulate using
`int.bit_length()`. For example, with my patch I can rewrite:

def encode_long(x):
    if x == 0:
        return b""
    return x.as_bytes(little_endian=True)

as:

def encode_long(x)
    if x == 0:
        return b""
    nbytes = (x.bit_length() >> 3) + 1
    result = x.as_bytes(nbytes, little_endian=True)
    if x < 0 and nbytes > 1:
        if result[-1] == 0xff and (result[-2] & 0x80) != 0:
            result = result[:-1]
    return result

I usually hate with passion APIs that requires you to know the length
of the result in advance. But this doesn't look bad. The only use-case
for the variable-length version I have is the encode_long() function
in pickle.py. In addition, it sounds reasonable to leave the duty of
long serialization to pickle.

So, +1 from me.

-- Alexandre