This is a neat idea, but this will only work for parsing framed binary protocols. For example, if you protocol prefixes all packets with a length field, you can write an efficient read buffer and use your proposal to decode all of message's fields in one shot. Which is good. Not all protocols use framing though. For instance, your proposal won't help to write Thrift or Postgres protocols parsers. Overall, I'm not sure that this is worth the hassle. With proposal: data, = struct.unpack('!H$', buf) buf = buf[2+len(data):] with the current struct module: len, = struct.unpack('!H', buf) data = buf[2:2+len] buf = buf[2+len:] Another thing: struct.calcsize won't work with structs that use variable length fields. Yury On 2017-01-18 5:24 AM, Elizabeth Myers wrote:
Hello,
I've noticed a lot of binary protocols require variable length bytestrings (with or without a null terminator), but it is not easy to unpack these in Python without first reading the desired length, or reading bytes until a null terminator is reached.
I've noticed the netstruct library (https://github.com/stendec/netstruct) has a format specifier, $, which assumes the previous type to pack/unpack is the string's length. This is an interesting idea in of itself, but doesn't handle the null-terminated string chase. I know $ is similar to pascal strings, but sometimes you need more than 255 characters :p.
For null-terminated strings, it may be simpler to have a specifier for those. I propose 0, but this point can be bikeshedded over endlessly if desired ;) (I thought about using n/N but they're :P).
It's worth noting that (maybe one of?) Perl's equivalent to the struct module, whose name escapes me atm, has a module which can handle this case. I can't remember if it handled variable length or zero-terminated though; maybe it did both. Perl is more or less my 10th language. :p
This pain point is an annoyance imo and would greatly simplify a lot of code if implemented, or something like it. I'd be happy to take a look at implementing it if the idea is received sufficiently warmly.
-- Elizabeth _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/