[Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[Steven D'Aprano]

On Thu, Jul 27, 2017 at 02:05:47AM +1000, Nick Coghlan wrote:
> On 26 July 2017 at 11:05, Steven D'Aprano <steve at pearwood.info> wrote:
> > I don't see any way that this proposal can be anything by a subtle
> > source of bugs. We have two *incompatible* requirements:
> >
> > - we want to define the order of the fields according to the
> >   order we give keyword arguments;
> >
> > - we want to give keyword arguments in any order without
> >   caring about the field order.
> >
> > We can't have both, and we can't give up either without being a
> > surprising source of annoyance and bugs.
> 
> I think the second stated requirement isn't a genuine requirement, as
> that *isn't* a general expectation.
> 
> After all, one of the reasons we got ordered-by-default keyword
> arguments is because people were confused by the fact that you
> couldn't reliably do:
> 
>     mydict = collections.OrderedDict(x=1, y=2)


Indeed. But the reason we got *keyword arguments* in the first place was 
so you didn't need to care about the order of parameters. As is often 
the case, toy examples with arguments x and y don't really demonstrate 
the problem in real code. We need more realistic, non-trivial examples. 
Most folks can remember the first two arguments to open:

open(name, 'w')

but for anything more complex, we not only want to skip arguments and 
rely on their defaults, but we don't necessarily remember the order of 
definition:

open(name, 'w', newline='\r', encoding='macroman', errors='replace')

Without checking the documentation, how many people could tell you 
whether that order matches the positional order? I know I couldn't.


You say

>     ntuple(x=1, y=2) == ntuple(y=1, x=2) == tuple(1, 2)
>     ntuple(x=2, y=1) == ntuple(y=2, x=1) == tuple(2, 1)
> 
> Putting the y-coordinate first would be *weird* though

Certainly, if you're used to the usual mathematics convention that the 
horizontal coordinate x comes first. But if you are used to the curses 
convention that the vertical coordinate y comes first, putting y first 
is completely natural.

And how about ... ?

    ntuple(flavour='strange', spin='1/2', mass=95.0, charge='-1/3',
           isospin='-1/2', hypercharge='1/3')

versus:

    ntuple(flavour='strange', mass=95.0, spin='1/2', charge='-1/3',
           hypercharge='1/3', isospin='-1/2')

Which one is "weird"?

This discussion has been taking place for many days, and it is only now 
(thanks to MRAB) that we've noticed this problem. I think it is 
dangerous to assume that the average Python coder will either:

- always consistently specify the fields in the same order;

- or recognise ahead of time (during the design phase of the program) 
  that they should pre-declare a class with the fields in a particular 
  order.


Some people will, of course. But many won't. Instead, they'll happily 
start instantiating ntuples with keyword arguments in inconsistent 
order, and if they are lucky they'll get unexpected, tricky to debug 
exceptions. If they're unlucky, their program will silently do the wrong 
thing, and nobody will notice that their results are garbage.

SimpleNamespace doesn't have this problem: the fields in SimpleNamespace 
aren't ordered, and cannot be packed or unpacked by position.

namedtuple doesn't have this problem: you have to predeclare the fields 
in a certain order, after which you can instantiate them by keyword in 
any order, and unpacking the tuple will always honour that order.


> Now, though, that's fully supported and does exactly what you'd expect:
> 
>     >>> from collections import OrderedDict
>     >>> OrderedDict(x=1, y=2)
>     OrderedDict([('x', 1), ('y', 2)])
>     >>> OrderedDict(y=2, x=1)
>     OrderedDict([('y', 2), ('x', 1)])
> 
> In this case, the "order matters" expectation is informed by the
> nature of the constructor being called: it's an *ordered* dict, so the
> constructor argument order matters.

I don't think that's a great analogy. There's no real equivalent of 
packing/unpacking OrderedDicts by position to trip us up here. It is 
better to think of OrderedDicts as "order-preserving dicts" rather than 
"dicts where the order matters". Yes, it does matter, in a weak sense. 
But not in the important sense of binding values to keys:

py> from collections import OrderedDict
py> a = OrderedDict([('spam', 1), ('eggs', 2)])
py> b = OrderedDict([('eggs', -1), ('spam', 99)])
py> a.update(b)
py> a
OrderedDict([('spam', 99), ('eggs', -1)])

update() has correctly bound 99 to key 'spam', even though the keys are 
in the wrong order. The same applies to dict unpacking:

a.update(**b)

In contrast, named tuples aren't just order-preserving. The field order 
is part of their definition, and tuple unpacking honours the field 
order, not the field names. While we can't update tuples in place, we 
can and often do unpack them into variables. When we do, we need to know 
the field order:


flavour, charge, mass, spin, isospin, hypercharge = mytuple

but the risk is that the field order may not be what we expect unless we 
are scrupulously careful to *always* call ntuple(...) with the arguments 
in the same order.



-- 
Steve