<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Jun 12, 2011, at 8:29 AM, Lukas Lueg wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi.<br><br>We extensively use the struct module to crunch large amounts of binary<br>data. There are basically two operations for us that only seem to the<br>naked eye as one: Filtering (see if certain fields have certain<br>values, throw everything away if not) and inspection (care about all<br>the fields' values). The filtering-part is very important as most of<br>the binary data can actually be thrown away and never have to be<br>inspected any further. When thinking about how to increase<br>performance, one thought was that a lot of objects are generated by<br>the struct module that we never need: Unpacking six fields in order to<br>look at one and then throwing everything away is inefficient<br>concerning the five other fields. It also makes filtering and<br>inspecting basically the same operation regarding the (slow) unpacking<br>of values so we don't really benefit from filtering. This is a huge<br>problem when crunching gigabytes of data and creating millions of<br>fields.<br><br>One solution to this is using two format-strings instead of only one<br>(e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields<br>(e.g. '8x i 8x') and one that unpacks all the fields except the one<br>already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution<br>works very well and increases throughput by far. I</div></blockquote><div><br></div><div>This is what people normally do (unpack just the values they need,</div><div>when they need them).</div><br><blockquote type="cite"><div>t however also<br>creates complexity in the code as we have to keep track and combine<br>field-values that came from the filtering-part with the ones unpacked<br>during inspection-part (we don't want to simply unpack twice).<br><br>I'd like to propose an enhancement to the struct module that should<br>solve this dilemma and ask for your comments.<br><br>The function s_unpack_internal() inside _struct.c currently unpacks<br>all values from the buffer-object passed to it and returns a tuple<br>holding these values. Instead, the function could create a tuple-like<br>object that holds a reference to it's own Struct-object (which holds<br>the format) and a copy of the memory it is supposed to unpack. This<br>object allows access to the unpacked values through the sequence<br>protocol, basically unpacking the fields if - and only if - accessed<br>through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] ==<br>'ab'). The object can also unpack all fields only once (as all<br>unpacked objects are immutable, we can hold references to them and<br>return these instead once known). This approach is possible because<br>there are no further error conditions inside the unpacking-functions<br>that we would *have* to deal with at the time .unpack() is called; in<br>other words: Unpacking can't fail if the format-string's syntax had<br>been correct and can therefor be deferred (while packing can't).<br><br>I understand that this may seem like a single-case-optimization.</div></blockquote><div><br></div><div>Yes, it does.</div><br><blockquote type="cite"><div> We<br>can however assume that most people will benefit from the new behavior<br>unknowingly while everyone else takes now harm: The object mimicking<br>the otherwise returned tuple is immutable (therefor it's not suddenly<br>part of GC) and the memory overhead caused by holding references to<br>the original memory a little longer (reclaimed after the result<br>becomes unreachable) should be comparable to the memory used by<br>unneeded fields (reclaimed directly after creation).<br><br>I'd like to hear your thoughts and am perfectly willing to provide a<br>patch if it has a chance of inclusion.<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><div><br></div><div>The problem you're trying to solve isn't unique to structs.</div><div>That's why we get periodic requests for ropes-like behaviors</div><div>for strings for example. Someone could also request lazy</div><div>extraction of fields in regular expressions or lazy parses of</div><div>json objects. </div><div><br></div><div>I don't think there is a net win from adding complexity to the struct</div><div>module. Introducing lazy behaviors creates its own overhead</div><div>that would compete with code optimized using the traditional</div><div>approach (unpack what you need, when you need it). Also,</div><div>the new behaviors add to the cognitive load when learning</div><div>and remembering how to use this module. </div><div><br></div><div>In general, Python has opted for the most straight-forward, least</div><div>magical implementations of object (why we use array based</div><div>lists instead of blist for example). The are exceptions </div><div>such as Python 3's version of super() but this isn't the norm.</div><div><br></div><div>I do suggest that you publish your code as a third-party module</div><div>to make the optional available and to validate whether there</div><div>is any real interest in this.</div><div><br></div><div><br></div><div>Raymond</div><div><br></div><div><br></div><br></div><div><br></div><div><br></div></body></html>