<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 2, 2018 at 9:39 AM, Michael Selik <span dir="ltr"><<a href="mailto:mike@selik.org" target="_blank">mike@selik.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><div dir="ltr">On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <<a href="mailto:nicolas.rolin@tiime.fr" target="_blank">nicolas.rolin@tiime.fr</a>> wrote:<br></div></span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>For example the default could be such that grouping unpack tuples (key, value) from the iterator and do what's expected with it (group value by key). It is quite reasonable, and you have one example with (key, value) in your example, and no example with the current default.</div></div></blockquote></span></div></div></blockquote></div></div></blockquote></div></div></div></blockquote><div><br></div><div>I agree, the default should do something that has some chance of being useful on its own, and ideally, the most "common" use, if we can identify that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Jul 2, 2018 at 3:22 AM Nicolas Rolin <<a href="mailto:nicolas.rolin@tiime.fr" target="_blank">nicolas.rolin@tiime.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>My question would be: does it have to be a key function? Can't we just remove the "key" argument?<br></div></div></blockquote><div><br></div><div>In the examples or from the parameters? A key function is necessary to support a wide variety of uses.</div></div></div></blockquote><div><br></div><div>not if you have the expectation of an iterable of (key, value) pairs as the input -- then any processing required to get a different key happens before hand, allowing folks to use comprehension syntax.</div><div> </div><div>as so: :-)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Because for pretty much all the given examples, I would find my default as readable and nearly as short as the "key" syntax :</div><div><br></div><div>> grouping(words, key=len)<br></div><div>grouping((len(word), word for word in words))<br></div></div></blockquote><div><br></div></span><div>I think the fact that you misplaced a closing parenthesis demonstrates how the key-function pattern can be more clear.</div></div></div></blockquote><div><br></div><div>I think it demonstrates that you shouldn't post untested code :-) -- the missing paren is a syntax error -- it would be caught right away in real life.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The code is slightly more verbose, but it is akin to filter(iterable, function) vs (i for i in iterable if function(i)).<br></div></div></blockquote><div><br></div></span><div>Sometimes I prefer ``map`` and sometimes I prefer a list comprehension. </div></div></div></blockquote><div><br></div><div>That is a "problem" with python: there are two ways to do things like map and filter, and one way is not always the clearest choice.</div><div><br></div><div>But I wonder if map and filter would exist if they didn't pre-date comprehensions..... That aside, the comprehension approach is pretty well liked and useful. And almost always prefer it -- an expression is simple on the eyes to me :-)</div><div><br></div><div>But when it's really a win is when you don't have a handy built-in function to do what you want, even though it's simple expression. </div><div><br></div><div>With the map, filter, key approach, you have to write a bunch of little utility functions or lambdas, which can really clutter up the code.</div><div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> If so, I like to write out the comprehension to provide that extra variable name for clarity.</div><div><br></div><div>I'd write: </div><div> map(len, words)</div><div><br></div><div>But I'd also write</div><div> [len(fullname) for fullname in contacts]</div></div></div></blockquote><div><br></div><div>A
key (pun intended) issue is that passing functions around looks so neat
and clean when it's a simple built in function like "len" -- but if
not, the it gets uglier, like:</div><div><br></div><font face="monospace, monospace">map(lambda name: name.first_name, all_names)</font><br><div><br></div><div>vs</div><div><br></div><div><font face="monospace, monospace">[name.first_name for nam in all names]</font></div><div><br></div><div>I
really like the comprehension form much better when what you really
want is a simple expression like an attribute access or index or simple
calculation, or ....</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>I
appreciate that defaulting the grouping key-function to
``itemgetter(0)`` would enable a pleasant flexibility for people to make
that same choice for each use. I haven't fully come around to that,
yet, because so many other tools use the equality function as the
default.</div></div></div></blockquote><div><br></div><div>well, kinda, but maybe those aren't "pythonic" :-)</div><div><br></div><div>(and yes, itemgetter() is probably a better option than lambda in many cases -- but I always forget that exists)<br></div><div><br></div></div><div>I started out in this topic answering a question about how to do a grouping for a list of tuples, in that case the OP wanted a comprehension. I don't think there's a good way to get a direct comprehension, but I do think we can make a class of function that takes something you could build with a comprehension.</div><div><br></div><div>And I took a look at itertools.groupby, and found it very, very awkward, ultimately arriving at:</div><br><font face="monospace, monospace">student_school_list = [('Fred', 'SchoolA'),<br> ('Bob', 'SchoolB'),<br> ('Mary', 'SchoolA'),<br> ('Jane', 'SchoolB'),</font></div><div class="gmail_quote"><font face="monospace, monospace"> ('Nancy', 'SchoolC')]</font><br><br><font face="monospace, monospace">grouped = {a:[t[0] for t in b] for a,b in groupby(sorted(student_school_<wbr>list, key=lambda t: t[1]), key=lambda t: t[1])}</font></div><font face="monospace, monospace"><br>{'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'], 'SchoolC': ['Nancy']}<br><br></font>So why is that so painful?<font face="monospace, monospace"><br></font><div class="gmail_quote"> </div><div class="gmail_quote">("it" is itertools.groupby)</div><div class="gmail_quote"><br></div><div class="gmail_quote">a) it returns an iterable of tuples, so to get a dict you need to do the dict comp</div><div class="gmail_quote">b) it requires the data to be sorted -- so you ned to sort it first</div><div class="gmail_quote">c) I need to provide a key function to sort by</div><div class="gmail_quote">d) I need to provide (the same, always?) key function to group by</div><div class="gmail_quote">e) when I make the dict, I need to make the list, and use an expression to get the value I want.</div><div class="gmail_quote">f) because I need those key functions, I need to use lambda for what could be a simple expression</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">So the current proposal in the PEP makes that a lot better:</div><div class="gmail_quote"><br></div><div class="gmail_quote">a) It makes a dict, so that step is done</div><div class="gmail_quote">b) It doesn't require the data to be sorted</div><div class="gmail_quote"><br></div><div class="gmail_quote">but:</div><div class="gmail_quote"><br></div><div class="gmail_quote">d) I still need the key function to do anything useful</div><div class="gmail_quote">e) If my data aren't clean, I need to do some post-processing to get the value I want.</div><div class="gmail_quote"><br></div><div class="gmail_quote">So -- the above example with the (current version of the) PEP's function:</div><div class="gmail_quote"><br></div><font face="monospace, monospace">In [35]: grouping(student_school_list, key=lambda t: t[1])<br>Out[35]: <br>{'SchoolA': [('Fred', 'SchoolA'), ('Mary', 'SchoolA')],<br> 'SchoolB': [('Bob', 'SchoolB'), ('Jane', 'SchoolB')],<br> 'SchoolC': [('Nancy', 'SchoolC')]}</font><div class="gmail_quote"><div><br></div><div>Darn! that's not right -- I need to clean up the values, too:</div><div><br></div><div><font face="monospace, monospace">gr = { k: [t[0] for t in l] for k, l in gr.items()}<br></font></div><div><br></div><div>OK, but pretty painful really, so I guess I should clean it up first, but I can't actually see any clean way to do that! Am I missing something?<br><br></div><div>Ah -- I see it in your PEP: "Sequences of values that are already paired with their keys can be
easily transformed after grouping.<span class="gmail-">"</span> -- sorry that code is not "easily" -- having to write that kind of code makes this whole thing pretty much useless compared to using, say, <span style="font-family:monospace,monospace">setdefault()</span> in the first place.<br></div><div><br></div><div>One option would be to add a value function, to unpack the value, analogous to the key function:</div><div><br></div><font face="monospace, monospace">In [44]: gr = grouping(student_school_list, key=lambda t: t[1], value=lambda t: t[0])<br><br>Out[45]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'], 'SchoolC': ['Nancy']}<br></font><br></div><div class="gmail_quote">That's pretty slick.<font face="monospace, monospace"><br></font></div><div class="gmail_quote"><br></div><div class="gmail_quote">However, I still much prefer an API that assumes an iterator of (key,value) pairs:</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote"><font face="monospace, monospace">def grouping(iterable):</font></div><div class="gmail_quote"><font face="monospace, monospace"> groups = {}</font></div><div class="gmail_quote"><font face="monospace, monospace"> for k, g in iterable:</font></div><div class="gmail_quote"><font face="monospace, monospace"> groups.setdefault(k, []).append(value)</font></div><div class="gmail_quote"><font face="monospace, monospace"> return groups</font></div></div><div class="gmail_quote"><font face="monospace, monospace"><br>(easy to impliment :-) )</font></div><div class="gmail_quote"><font face="monospace, monospace"><br></font></div>And now you get something that "just works" for at least one case:<div class="gmail_quote"><font face="monospace, monospace"><br></font></div><div class="gmail_quote"><div class="gmail_quote"><font face="monospace, monospace">In [54]: def grouping(iterable):</font></div><div class="gmail_quote"><font face="monospace, monospace"> ...: groups = {}</font></div><div class="gmail_quote"><font face="monospace, monospace"> ...: for key, value in iterable:</font></div><div class="gmail_quote"><font face="monospace, monospace"> ...: groups.setdefault(key, []).append(value)</font></div><div class="gmail_quote"><font face="monospace, monospace"> ...: return groups</font></div><div class="gmail_quote"><font face="monospace, monospace"> ...: </font></div><div class="gmail_quote"><font face="monospace, monospace"><br></font></div><div class="gmail_quote"><font face="monospace, monospace">In [55]: school_student_list</font></div><div class="gmail_quote"><font face="monospace, monospace">Out[55]: </font></div><div class="gmail_quote"><font face="monospace, monospace">[('SchoolA', 'Fred'),</font></div><div class="gmail_quote"><font face="monospace, monospace"> ('SchoolB', 'Bob'),</font></div><div class="gmail_quote"><font face="monospace, monospace"> ('SchoolA', 'Mary'),</font></div><div class="gmail_quote"><font face="monospace, monospace"> ('SchoolB', 'Jane'),</font></div><div class="gmail_quote"><font face="monospace, monospace"> ('SchoolC', 'Nancy')]</font></div><div class="gmail_quote"><font face="monospace, monospace"><br></font></div><div class="gmail_quote"><font face="monospace, monospace">In [56]: grouping(school_student_list)</font></div><div class="gmail_quote"><font face="monospace, monospace">Out[56]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'], 'SchoolC': ['Nancy']}</font></div></div><div class="gmail_extra"><br></div><div class="gmail_quote">And if you need to massage the data you can do so with a generator expression:<br><br><font face="monospace, monospace">In [58]: grouping((reversed(t) for t in student_school_list))<br><br>Out[58]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'], 'SchoolC': ['Nancy']}<br></font><br></div><div class="gmail_quote">And here are the examples from the PEP:<font face="monospace, monospace"><br></font></div><div class="gmail_quote"><font face="monospace, monospace">(untested -- I may hav missed some brackets, etc)<br></font></div><div class="gmail_quote"><font face="monospace, monospace"><br># Group words by length:
<span></span>
<p class="gmail-m_6403292849730066649gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_6403292849730066649gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p>grouping(((len(word), word) for word in words))<div class="gmail_quote"><br></div><div class="gmail_quote"># Group names by first initial:</div><div class="gmail_quote"><br></div><div class="gmail_quote">grouping((name[0], name) for name in names))</div><div class="gmail_quote"><br></div><div class="gmail_quote"># Group people by city:</div><div class="gmail_quote"><br></div><div class="gmail_quote">grouping((contact.city, contact) for contact in contacts) </div><div class="gmail_quote"><br></div><div class="gmail_quote"># Group employees by department:</div><div class="gmail_quote"><br></div><div class="gmail_quote">grouping((employee['department'] for employee in employees)</div><div class="gmail_quote"><br></div><div class="gmail_quote"># Group files by extension:</div><div class="gmail_quote"><br></div><div class="gmail_quote">grouping((<font face="monospace, monospace">os.path.splitext(filepath)[1]</font> for filepath in os.listdir('.')))</div><div class="gmail_quote"><br># Group transactions by type:</div><div class="gmail_quote"><br></div><div class="gmail_quote">grouping(( <font face="monospace, monospace">'debit' if v > 0 else 'credit' for v in </font>transactions))</div><div class="gmail_quote"><br># Invert a dictionary, d, without discarding repeated values:</div><div class="gmail_quote"><br></div></font></div><div class="gmail_quote"><font face="monospace, monospace">grouping(((v, k) for v, k in d.items()))<br></font></div><div class="gmail_quote"><div><br></div><div>So that was an interesting exercise -- many of those are a bit clearer (or more compact) with the key function. But I also notice a pattern -- all thos examples fit very well into the key function pattern:<br><br></div><div>you want the entire item stored in your iterable.<br></div><div>you want to group by some quality of the item itself.<br><br></div><div>Perhaps those ARE the most common use cases -- I honestly don't know, but from an earlier post:<br><br>" In practice, instead of (key, value) pairs, it's usually either
individual values or n-tuple rows. In the latter case, sometimes the key
should be dropped from the row when grouping, sometimes kept in the
row, and sometimes the key must be computed from multiple values within
the row."<br><br></div><div>It seems that the comprehension style I'm suggesting would be a win for the case of n-tuple rows.<br><br></div><div>Doing this exercise has expanded my view, so I suggest that:<br><br></div><div>- keep the key function optional parameter.<br></div><div>- add a value function optional parameter. -- it really makes any case where you don't want to store the whole item a lot easier.<br><br></div><div>- Have the default key function be itemgetter(0) and the default value function be itemgetter(1) (or some similar way to implement default support for processing an iterable of (key, value) pairs.<br><br></div><div>Having no value function and an equality default for the key function may be "common", but it's also pretty much useless -- why have a useless default?<br><br></div><div>Thinking this through now I do see that having key and value default to to the pair method means that if you specify key function, you will probably have to specify a value function as well -- so maybe that's not ideal.<br><br></div><div>hmm.<br><br><br></div><div>A couple other comments:<br><br></div><div>implementation detail: Do you gain anything by using the itertools groupby? over, say:<br><br>groups = {}<br>for item in iterable:<br> groups.setdefault(key(item), []).append(item)<br><br></div><div>Final point:<br><br></div><div>I still prefer the class idea over a utility function, because:<br><br></div><div>* with a class, you can ad stuff to the grouping later:<br><br></div><div><span style="font-family:monospace,monospace">a_grouping['key'] = value<br></span><br></div><div>or maybe <span style="font-family:monospace,monospace">a_grouping.add(item)</span><br><br></div><div>* with a class, you can add utility methods -- I kinda liked that in your original PEP.<br><br></div><div>I see this in the section about a custom class: "Merging groupings is not a one-liner,<span class="gmail-">"</span> -- I'm pretty sure the update() method on my prototype was a merge operation -- so very easy :-) -- and another argument for a custom class.<br><br></div><div>final thought about custon class. If you want to be abel to do:<br><br><span style="font-family:monospace,monospace">a_grouping['key'] = value<span class=""><br><br></span></span></div>then that really reinforces the "natural" mapping of keys to values -- it's kind of another way to think about this -- rather than thinking about it as "groupby" function that creates a dict, think of it as a dict-like object, that, instead of writing over an existing key, accumulates the multiple values.<br><br></div>which means you really want the "normal" constructor to take an iterable of (key, value) pairs, to make it more dict-like.<br><br></div><div class="gmail_extra">-CHB<br><br></div><div class="gmail_extra"><div class="gmail_quote"><br><div><br><br><br></div><div><br><br><br><br></div><div><br></div><div><br><br><br></div><br></div>-- <br><div class="gmail-m_6403292849730066649gmail-m_4599128910531418084m_5885662694786674068gmail-m_-1091559232340231579gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>