[Python-ideas] Ideas for improving the struct module

Fri Jan 20 22:39:15 EST 2017

On Fri, Jan 20, 2017 at 3:37 PM, Elizabeth Myers
<elizabeth at interlinked.me> wrote:
[...]
>> Some of the responses on the bug are discouraging... mostly seems to
>> boil down to people just not wanting to expand the struct module or
>> discourage its use. Everyone is a critic. I didn't know adding two
>> format specifiers was going to be this controversial. You'd think I
>> proposed adding braces or something :/.
>>
>> I'm hesitant to go forward on this until the bug has a resolution.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> Also, btw, adding 128-bit length specifiers sounds like a good idea in
> theory, but the difficulty stems from the fact there's no real native
> 128-bit type that's portable. I don't know much about how python handles
> big ints internally, either, but I could learn.

The "b128" in "uleb128" is short for "base 128"; it refers to how each
byte contains one 7-bit "digit" of the integer being encoded -- so
just like decimal needs 1 digit for 0-9, 2 digits for 10 - 99 = (10**2
- 1), etc., uleb128 uses 1 byte for 0-127, 2 bytes for 128 - 16383 =
(128**2 - 1), etc. In practice most implementations are written in C
and use some kind of native fixed width integer as the in-memory
representation, and just error out if asked to decode a uleb128 that
doesn't fit. In Python I suppose we could support encoding and
decoding arbitrary size integers if we really wanted, but I also doubt
anyone would be bothered if we were restricted to "only" handling
numbers between 0 and 2**64 :-).

> I was looking into implementing this already, and it appears it should
> be possible by teaching the module that "not all data is fixed length"
> and allowing functions to report back (via a Py_ssize_t *) how much data
> was actually unpacked/packed. But again, waiting on that bug to have a
> resolution before I do anything. I don't want to waste hours of effort
> on something the developers ultimately decide they don't want and will
> just reject.

That's not really how Python bugs work in practice. For better or
worse (and it's very often both), CPython development generally
follows a traditional open-source model in which new proposals are
only accepted if they have a champion who's willing to run the
gauntlet of first proposing them, and then keep pushing forward
through the hail of criticism and bike-shedding from random
kibbitzers. This is at least in part a test to see how
dedicated/stubborn you are about this feature. If you stop posting,
then what will happen is that everyone else stops posting too, and the
bug will just sit there unresolved indefinitely until you get (more)
frustrated and give up.

On the one hand, this does tend to guarantee that accepted proposals
are very high quality and solve some important issue (b/c the champion
didn't *really care* about the issue then they wouldn't put up with
this). On the other hand, it's often pretty hellish for the
individuals involved, and probably drives away all kinds of helpful
contributions. But maybe it helps to know it's not personal? Having
social capital definitely helps, but well-known/experienced
contributors get put through this wringer too; the main difference is
that we do it with eyes open and have strategies for navigating the
system (at least until we get burned out).

Some of these strategies that you might find helpful (or not):

- realize that it's really common for someone to be all like "this is
TERRIBLE and should definitely not be merged because of <issue> which
is a TOTAL SHOW-STOPPER", but then if you ignore the histrionics and
move forward anyway, it often turns out that all that person
*actually* wanted was to see a brief paragraph in your design summary
that acknowledges that you are aware of the existence of <issue>, and
once they see this they're happy. (See also: [1])

- speaking of which, it is often very helpful to write up a short
document to summarize and organize the different ideas proposed,
critiques raised, and what you conclude based on them! That's
basically what a "PEP" is - just an email in a somewhat standard
format that reviews all the important issues that were raised and then
says what you conclude and why, and which eventually also ends up on
the website as a record. If you decide to try this then there are some
guidelines [2][3] and a sample PEP [4] to start with. (The guidelines
make it sound much more formal and scary than it really is, though --
e.g. when they say "your submission may be AUTOMATICALLY REJECTED"
then in my experience what they actually mean is you might get a reply
back saying "hey fyi the formatter script barfed on your document
because you were missing a quote so I fixed it for you".) This
particular proposal is really on the lower boundary of needing a PEP
and you might well be able to push it through without one, but it
might be easier to go this way than not.

- sift through the responses to pick the ones that seem actually
useful to you, then close the browser tab and go off and implement
what you actually think should be implemented and come back with a
patch. This does a few things: (a) it helps get everyone on the same
page and make the discussion much more concrete, which tends to
eliminate a lot of idle criticism/bikeshedding, (b) it tends to
attract higher-quality responses because it demonstrates you're
serious and makes it look more like this is a thing that will actually
happen (see again the "trial by combat" thing above), (c) many of the
experts whose good opinion is important are attention-scattered
volunteers who are bad at time management and prioritization (I
include myself in this category!), so if you stick a patch in front of
their faces then you can trick them into switching into code review
mode instead of design critique mode. And it's much easier to respond
to "your semicolon is in the wrong place" than "but what is the struct
module really *for*, I mean, in its heart of hearts?", you know?

- join the core-mentorship list [5] and ask for help there. Actually
this should probably be the first suggestion on this list! it's a
group of folks who're specifically volunteering to help people like
you get through this process :-)

Hope that helps,
-n

[1] http://www.ftrain.com/wwic.html
[2] https://www.python.org/dev/peps/pep-0001/#submitting-a-pep
[3] https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep
[4] https://github.com/python/peps/blob/master/pep-0012.txt
[5] https://mail.python.org/mailman/listinfo/core-mentorship

-- 
Nathaniel J. Smith -- https://vorpus.org