Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis <martin at v.loewis.de> wrote:

>> You construct a format string for the "value" portion based on
>> the type/length header.
>
> Can you kindly provide example code on how to do this?

OK, something like this to handle received data where there is
an initial 8-bit type field that is 1 for 16-bit unsigned
integers in network byte-order, 2 for 32-bit IEEE floats in
network byte-order.  We'll further assume that the 'length'
field comes next as a 16 bit unsigned value in network order
and represents "how many" objects of the specified type follow:

  dtype = ord(rawdata[0])
  dcount = struct.unpack("!H",rawdata[1:3])
  if dtype == 1:
     fmtstr = "!" + "H"*dcount
  elif dtype == 2:
     fmtstr = "!" + "f"*dcount
  rlen = struct.calcsize(fmtstr)
  
  data = struct.unpack(fmtstr,rawdata[3:3+rlen])
  
  leftover = rawdata[3+rlen:]

>> I don't see how that can be the case.  There may not be a
>> single C struct that can represent all frames, but for every
>> frame you should be able to come up with a C struct that can
>> represent that frame.
>
> Sure. You would normally have a struct such as
>
> struct TLV{
>   char type;
>   char length;
>   char *data;
> };
>
> However, the in-memory representation of that struct is *not*
> meant to be sent over the wire. In particular, the character
> pointer has no meaning outside the address space, and is thus
> not to be sent.

Well if it's not representing the layout of the data we're
trying to deal with, then it's irrelevent.  We are talking
about how convert python objects to/from data in the
'on-the-wire' format, right?

Or isn't that what the OP is asking about?

>> Both.  For varible size/format stuff you decode the first few
>> bytes and use them to figure out what format/layout to use for
>> the next chunk of data.  It's pretty much the same thing you do
>> in other languages.
>
> In the example he gave, I would just avoid using the struct module
> entirely, as it does not provide any additional value:
>
> def encode(type, length, value):
>   return chr(type)+chr(length)+value

Like this?

>>> def encode(type,length,value):
...  return chr(type)+chr(length)+value
... 
>>> print encode('float', 1, 3.14159)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in encode
TypeError: an integer is required
>>> 

-- 
Grant