An idiom for code generation with exec

[Scott David Daniels]

bruno.desthuilliers at gmail.com wrote:
 > On 20 juin, 21:44, eliben <eli... at gmail.com> wrote:
...
 >> The generic version has to make a lot of decisions at runtime, based
 >> on the format specification.
 >> Extract the offset from the spec, extract the length.
... <example with lists of operations>...
 > Just my 2 cents. Truth is that as long as it works and is
 > maintainable, then who cares...

To chime in on "non-exec" operations alternatives.  The extraction
piece-by-piece looks like it might be slow.  You could figure out
where the majority of endian-ness is (typically it will be all one way),
and treat specially any others (making the string fields with an
operation applied).  The best optimizations come from understanding
the regularities in your specific case; seldom do they come from
generating code that hides the regularity and depending on a compiler
to deduce that regularity.

Have you tried something like this (a sketch of a solution)?:

import struct
from functools import partial
import operator


class Decoder(object):
     def __init__(self, unpack, processors, finals):
         '''Of course this might be simply take the yaml in and go'''
         self.unpack = unpack
         self.processors = processors
         self.length = struct.calcsize(unpack)
         self.finals = finals

     def packet(self, data):
         parts = list(struct.unpack(self.unpack, data))
         for n, action, result in self.processors:
             if result is None:
                 parts.append(action(parts[n]))
             else:
                 parts[n] = action(parts[n])
         return tuple(parts[n] for n in self.finals) # or NamedTuple ...


def _bits(from_bit, mask, v):
     return (v >> from_bit) & mask


Your example extended a bit:
     fmt = Decoder('<cBiBIi', [(1, partial(operator.mul, 2048), '-'),
                               (3, partial(_bits, 5, 0x3), None),
                               (3, partial(operator.iand, 0x80), None),
                               (3, partial(operator.iand, 0x1F), '-'),
                               (6, bool, '-')],
                   [0, 1, 2, 3, 6, 7, 4, 5])
     print fmt.packet(source.read(fmt.length))


--Scott David Daniels
Scott.Daniels at Acm.Org