[Python-ideas] Ideas for improving the struct module

[Cameron Simpson]

On 19Jan2017 12:08, Elizabeth Myers <elizabeth at interlinked.me> wrote:
>I also didn't mention that when you are unpacking iteratively (e.g., you
>have multiple strings), the code becomes a bit more hairy:
>
>>>> test_bytes = b'\x00\x05hello\x00\x07goodbye\x00\x04test'
>>>> offset = 0
>>>> while offset < len(test_bytes):
>...     length = struct.unpack_from('!H', test_bytes, offset)[0]
>...     offset += 2
>...     string = struct.unpack_from('{}s'.format(length), test_bytes,
>offset)[0]
>...     offset += length
>
>It actually gets a lot worse when you have to unpack a set of strings in
>a context-sensitive manner. You have to be sure to update the offset
>constantly so you can always unpack strings appropriately. Yuck!

Whenever I'm doing iterative stuff like this, either variable length binary or 
lexical stuff, I always end up with a bunch of functions which can be called 
like this:

  datalen, offset = get_bs(chunk, offset=offset)

The notable thing here is just that they return the data and the new offset, 
which makes updating the offset impossible to forget, and also makes the 
calling code more succinct, like the internal call to get_bs() below:

such as this decoder for a length encoded field:

  def get_bsdata(chunk, offset=0):
    ''' Fetch a length-prefixed data chunk.
        Decodes an unsigned value from a bytes at the specified `offset`
        (default 0), and collects that many following bytes.
        Return those following bytes and the new offset.
    '''
    ##is_bytes(chunk)
    offset0 = offset
    datalen, offset = get_bs(chunk, offset=offset)
    data = chunk[offset:offset+datalen]
    ##is_bytes(data)
    if len(data) != datalen:
      raise ValueError("bsdata(chunk, offset=%d): insufficient data: expected %d 
  bytes, got %d bytes"
                       % (offset0, datalen, len(data)))
    offset += datalen
    return data, offset

Cheers,
Cameron Simpson <cs at zip.com.au>