[Python-ideas] Ideas for improving the struct module

[MRAB]

On 2017-01-19 12:47, Elizabeth Myers wrote:
> On 19/01/17 05:58, Rhodri James wrote:
>> On 19/01/17 08:31, Mark Dickinson wrote:
>>> On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano <steve at pearwood.info>
>>> wrote:
>>>> [...] struct already supports
>>>> variable-width formats.
>>>
>>> Unfortunately, that's not really true: the Pascal strings it supports
>>> are in some sense variable length, but are stored in a fixed-width
>>> field. The internals of the struct module rely on each field starting
>>> at a fixed offset, computable directly from the format string. I don't
>>> think variable-length fields would be a good fit for the current
>>> design of the struct module.
>>>
>>> For the OPs use-case, I'd suggest a library that sits on top of the
>>> struct module, rather than an expansion to the struct module itself.
>>
>> Unfortunately as the OP explained, this makes the struct module a poor
>> fit for protocol decoding, even as a base layer for something.  It's one
>> of the things I use python for quite frequently, and I always end up
>> rolling my own and discarding struct entirely.
>>
>
> Yes, for variable-length fields the struct module is worse than useless:
> it actually reduces clarity a little. Consider:
>
>>>> test_bytes = b'\x00\x00\x00\x0chello world!'
>
> With this, you can do:
>
>>>> length = int.from_bytes(test_bytes[:4], 'big')
>>>> string = test_bytes[4:length]
>
Shouldn't that be:

string = test_bytes[4:4+length]

> or you can do:
>
>>>> length = struct.unpack_from('!I', test_bytes)[0]
>>>> string = struct.unpack_from('{}s'.format(length), test_bytes, 4)[0]
>
> Which looks more readable without consulting the docs? ;)
>
Which is more likely to be correct? :-)

> Building anything on top of the struct library like this would lead to
> worse-looking code for minimal gains in efficiency. To quote Jamie
> Zawinksi, it is like building a bookshelf out of mashed potatoes as it
> stands.
>
> If we had an extension similar to netstruct:
>
>>>> length, string = struct.unpack('!I$', test_bytes)
>
> MUCH improved readability, and also less verbose. :)
>