[Python-ideas] Ideas for improving the struct module

[Elizabeth Myers]

On 19/01/17 20:54, Cameron Simpson wrote:
> On 19Jan2017 16:04, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> This is a neat idea, but this will only work for parsing framed
>> binary protocols.  For example, if you protocol prefixes all packets
>> with a length field, you can write an efficient read buffer and
>> use your proposal to decode all of message's fields in one shot.
>> Which is good.
>>
>> Not all protocols use framing though.  For instance, your proposal
>> won't help to write Thrift or Postgres protocols parsers.
> 
> Sure, but a lot of things fit the proposal. Seems a win: both simple and
> useful.
> 
>> Overall, I'm not sure that this is worth the hassle.  With proposal:
>>
>>   data, = struct.unpack('!H$', buf)
>>   buf = buf[2+len(data):]
>>
>> with the current struct module:
>>
>>   len, = struct.unpack('!H', buf)
>>   data = buf[2:2+len]
>>   buf = buf[2+len:]
>>
>> Another thing: struct.calcsize won't work with structs that use
>> variable length fields.
> 
> True, but it would be enough for it to raise an exception of some kind.
> It won't break any in play code, and it will prevent accidents for users
> of new variable sizes formats.
> 
> We've all got things we wish struct might cover (I have a few, but
> strangely the top of the list is nonsemantic: I wish it let me put
> meaningless whitespace inside the format for readability).
> 
> +1 on the proposal from me.
> 
> Oh: subject to one proviso: reading a struct will need to return how
> many bytes of input data were scanned, not merely returning the decoded
> values.

This is a little difficult without breaking backwards compatibility,
but, it is not difficult to compute the lengths yourself. That said,
calcsize could require an extra parameter if given a format string with
variable-length specifiers in it, e.g.:

  struct.calcsize("z", (b'test'))

Would return 5 (zero-length terminator), so you don't have to compute it
yourself.

Also, I filed a bug, and proposed use of Z and z.

> 
> Cheers,
> Cameron Simpson <cs at zip.com.au>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/