[Python-ideas] Ideas for improving the struct module

[Elizabeth Myers]

On 19/01/17 05:58, Rhodri James wrote:
> On 19/01/17 08:31, Mark Dickinson wrote:
>> On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>>> [...] struct already supports
>>> variable-width formats.
>>
>> Unfortunately, that's not really true: the Pascal strings it supports
>> are in some sense variable length, but are stored in a fixed-width
>> field. The internals of the struct module rely on each field starting
>> at a fixed offset, computable directly from the format string. I don't
>> think variable-length fields would be a good fit for the current
>> design of the struct module.
>>
>> For the OPs use-case, I'd suggest a library that sits on top of the
>> struct module, rather than an expansion to the struct module itself.
> 
> Unfortunately as the OP explained, this makes the struct module a poor
> fit for protocol decoding, even as a base layer for something.  It's one
> of the things I use python for quite frequently, and I always end up
> rolling my own and discarding struct entirely.
> 

Yes, for variable-length fields the struct module is worse than useless:
it actually reduces clarity a little. Consider:

>>> test_bytes = b'\x00\x00\x00\x0chello world!'

With this, you can do:

>>> length = int.from_bytes(test_bytes[:4], 'big')
>>> string = test_bytes[4:length]

or you can do:

>>> length = struct.unpack_from('!I', test_bytes)[0]
>>> string = struct.unpack_from('{}s'.format(length), test_bytes, 4)[0]

Which looks more readable without consulting the docs? ;)

Building anything on top of the struct library like this would lead to
worse-looking code for minimal gains in efficiency. To quote Jamie
Zawinksi, it is like building a bookshelf out of mashed potatoes as it
stands.

If we had an extension similar to netstruct:

>>> length, string = struct.unpack('!I$', test_bytes)

MUCH improved readability, and also less verbose. :)