On Jun 12, 2011, at 8:29 AM, Lukas Lueg wrote:

Hi.

We extensively use the struct module to crunch large amounts of binary data. There are basically two operations for us that only seem to the naked eye as one: Filtering (see if certain fields have certain values, throw everything away if not) and inspection (care about all the fields' values). The filtering-part is very important as most of the binary data can actually be thrown away and never have to be inspected any further. When thinking about how to increase performance, one thought was that a lot of objects are generated by the struct module that we never need: Unpacking six fields in order to look at one and then throwing everything away is inefficient concerning the five other fields. It also makes filtering and inspecting basically the same operation regarding the (slow) unpacking of values so we don't really benefit from filtering. This is a huge problem when crunching gigabytes of data and creating millions of fields.

One solution to this is using two format-strings instead of only one (e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields (e.g. '8x i 8x') and one that unpacks all the fields except the one already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution works very well and increases throughput by far. I

This is what people normally do (unpack just the values they need, when they need them).

t however also creates complexity in the code as we have to keep track and combine field-values that came from the filtering-part with the ones unpacked during inspection-part (we don't want to simply unpack twice).

I'd like to propose an enhancement to the struct module that should solve this dilemma and ask for your comments.

The function s_unpack_internal() inside _struct.c currently unpacks all values from the buffer-object passed to it and returns a tuple holding these values. Instead, the function could create a tuple-like object that holds a reference to it's own Struct-object (which holds the format) and a copy of the memory it is supposed to unpack. This object allows access to the unpacked values through the sequence protocol, basically unpacking the fields if - and only if - accessed through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] == 'ab'). The object can also unpack all fields only once (as all unpacked objects are immutable, we can hold references to them and return these instead once known). This approach is possible because there are no further error conditions inside the unpacking-functions that we would *have* to deal with at the time .unpack() is called; in other words: Unpacking can't fail if the format-string's syntax had been correct and can therefor be deferred (while packing can't).

I understand that this may seem like a single-case-optimization.

Yes, it does.

We can however assume that most people will benefit from the new behavior unknowingly while everyone else takes now harm: The object mimicking the otherwise returned tuple is immutable (therefor it's not suddenly part of GC) and the memory overhead caused by holding references to the original memory a little longer (reclaimed after the result becomes unreachable) should be comparable to the memory used by unneeded fields (reclaimed directly after creation).

I'd like to hear your thoughts and am perfectly willing to provide a patch if it has a chance of inclusion.

The problem you're trying to solve isn't unique to structs. That's why we get periodic requests for ropes-like behaviors for strings for example. Someone could also request lazy extraction of fields in regular expressions or lazy parses of json objects. I don't think there is a net win from adding complexity to the struct module. Introducing lazy behaviors creates its own overhead that would compete with code optimized using the traditional approach (unpack what you need, when you need it). Also, the new behaviors add to the cognitive load when learning and remembering how to use this module. In general, Python has opted for the most straight-forward, least magical implementations of object (why we use array based lists instead of blist for example). The are exceptions such as Python 3's version of super() but this isn't the norm. I do suggest that you publish your code as a third-party module to make the optional available and to validate whether there is any real interest in this. Raymond

On 6/12/2011 11:29 AM, Lukas Lueg wrote: This sort of speculative idea might fit the python-ideas list better. [Summary: we often need to extract a field or two from a binary record in order to decide whether to toss it or unpack it all and process.]

One solution to this is using two format-strings instead of only one (e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields (e.g. '8x i 8x') and one that unpacks all the fields except the one already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution works very well and increases throughput by far. It however also creates complexity in the code as we have to keep track and combine field-values that came from the filtering-part with the ones unpacked during inspection-part (we don't want to simply unpack twice).

With just 1 or 2 filter fields, and very many other fields, I would just unpack everything, including the filter field. I expect the extra time to do that would be comparalbe to the extra time to combine. It certainly would make your code easier. I suspect you could write a function to create the filter field only format by field number from the everything format.

I'd like to propose an enhancement to the struct module that should solve this dilemma and ask for your comments.

The function s_unpack_internal() inside _struct.c currently unpacks all values from the buffer-object passed to it and returns a tuple holding these values. Instead, the function could create a tuple-like object that holds a reference to it's own Struct-object (which holds the format) and a copy of the memory it is supposed to unpack. This object allows access to the unpacked values through the sequence protocol, basically unpacking the fields if - and only if - accessed through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] == 'ab'). The object can also unpack all fields only once (as all unpacked objects are immutable, we can hold references to them and return these instead once known). This approach is possible because there are no further error conditions inside the unpacking-functions that we would *have* to deal with at the time .unpack() is called; in other words: Unpacking can't fail if the format-string's syntax had been correct and can therefor be deferred (while packing can't).

I understand that this may seem like a single-case-optimization.

Yep.

We can however assume that most people will benefit from the new behavior unknowingly while everyone else takes now harm:

I will not assume that without code and timings. I would expect that unpacking one field at a time would take longer than all at once. To me, this is the sort of thing that should be written, listed on PyPI, and tested by multiple users on multiple systems first. -- Terry Jan Reedy