[Python-ideas] Ideas for improving the struct module

[Nick Coghlan]

On 21 January 2017 at 14:51, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Jan 20, 2017 at 7:39 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> [...]
>> Some of these strategies that you might find helpful (or not):
>
> Oh right, and of course just after I hit send I realized I forgot one
> of my favorites!
>
> - come up with a real chunk of code from a real project that would
> benefit from the change being proposed, and show what it looks like
> before/after the feature is added. This can be incredibly persuasive
> *but* it's *super important* that the code be as real as possible. The
> ideal is for it to solve a *concrete* *real-world* problem that can be
> described in a few sentences, and be drawn from a real code base that
> faces that problem. One of the biggest challenges for maintainers is
> figuring out how Python is actually used in the real world, because we
> all have very little visibility outside our own little bubbles, so
> people really appreciate this -- but at the same time, python-ideas is
> absolutely awash with people coming up with weird hypothetical
> situations where their pet idea would be just the ticket, so anything
> that comes across as cherry-picked like that tends to be heavily
> discounted.

In the specific case of this proposal, an interesting stress test of
any design proposal would be to describe the layout of a CPython tuple
in memory. If you trace through the struct and macro definition
details in https://hg.python.org/cpython/file/tip/Include/object.h and
https://hg.python.org/cpython/file/tip/Include/tupleobject.h you'll
find that the last two fields in PyTupleObject are:

    Py_ssize_t ob_size;
    PyObject *ob_item[1];

So this is a C struct definition *in the CPython code base* that the
struct module currently cannot describe (other PyVarObject definitions
are similar to tuples, but don't necessarily guarantee that ob_size is
the last field before the variable length section).

Similarly, PyASCIIObject and PyCompactUnicodeObject append a data
buffer to the preceding struct in order to include both a string's
metadata and its contents into the same memory allocation. In that
case, the buffer is also null-terminated in addition to having its
length specified in the string metadata.

So both of the proposals Elizabeth is making reflect ways that CPython
itself uses C structs (matching the heritage of the module's name),
even though the primary practical motivation and use case is common
over-the-wire protocols.

Cheers,
Nick.

P.S. "the reference interpreter does this" and "the standard library
does this" can be particularly compelling sources of real world
example code :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia