<div dir="ltr"><div>Thanks for this thorough review, Raymond! Especially the user research is amazing.<br><br>Â And thanks for Antoine for writing the PEP -- you never know how an idea pans out until you've tried it.<br><br></div>--Guido<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 14, 2015 at 7:29 AM, Raymond Hettinger <span dir="ltr"><<a href="mailto:raymond.hettinger@gmail.com" target="_blank">raymond.hettinger@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Before the Python 3.5 feature freeze, I should step-up and<br>
formally reject PEP 455 for "Adding a key-transforming<br>
dictionary to collections".<br>
<br>
I had completed an involved review effort a long time ago<br>
and I apologize for the delay in making the pronouncement.<br>
<br>
What made it a interesting choice from the outset is that the<br>
idea of a "transformation" is an enticing concept that seems<br>
full of possibility. I spent a good deal of time exploring<br>
what could be done with it but found that it mostly fell short<br>
of its promise.<br>
<br>
There were many issues. Here are some that were at the top:<br>
<br>
* Most use cases don't need or want the reverse lookup feature<br>
 (what is wanted is a set of one-way canonicalization functions).<br>
 Those that do would want to have a choice of what is saved<br>
 (first stored, last stored, n most recent, a set of all inputs,<br>
 a list of all inputs, nothing, etc). In database terms, it<br>
 models a many-to-one table (the canonicalization or<br>
 transformation function) with the one being a primary key into<br>
 another possibly surjective table of two columns (the<br>
 key/value store). A surjection into another surjection isn't<br>
 inherently reversible in a useful way, nor does it seem to be a<br>
 common way to model data.<br>
<br>
* People are creative at coming up with using cases for the TD<br>
 but then find that the resulting code is less clear, slower,<br>
 less intuitive, more memory intensive, and harder to debug than<br>
 just using a plain dict with a function call before the lookup:<br>
 d[func(key)]. It was challenging to find any existing code<br>
 that would be made better by the availability of the TD.<br>
<br>
* The TD seems to be all about combining data scrubbing<br>
 (case-folding, unicode canonicalization, type-folding, object<br>
 identity, unit-conversion, or finding a canonical member of an<br>
 equivalence class) with a mapping (looking-up a value for a<br>
 given key). Those two operations are conceptually orthogonal.<br>
 The former doesn't get easier when hidden behind a mapping API<br>
 and the latter loses the flexibility of choosing your preferred<br>
 mapping (an ordereddict, a persistentdict, a chainmap, etc) and<br>
 the flexibility of establishing your own rules for whether and<br>
 how to do a reverse lookup.<br>
<br>
<br>
Raymond Hettinger<br>
<br>
<br>
P.S. Besides the core conceptual issues listed above, there<br>
are a number of smaller issues with the TD that surfaced<br>
during design review sessions. In no particular order, here<br>
are a few of the observations:<br>
<br>
* It seems to require above average skill to figure-out what<br>
 can be used as a transform function. It is more<br>
 expert-friendly than beginner friendly. It takes a little<br>
 while to get used to it. It wasn't self-evident that<br>
 transformations happen both when a key is stored and again<br>
 when it is looked-up (contrast this with key-functions for<br>
 sorting which are called at most once per key).<br>
<br>
* The name, TransformDict, suggests that it might transform the<br>
 value instead of the key or that it might transform the<br>
 dictionary into something else. The name TransformDict is so<br>
 general that it would be hard to discover when faced with a<br>
 specific problem. The name also limits perception of what<br>
 could be done with it (i.e. a function that logs accesses<br>
 but doesn't actually change the key).<br>
<br>
* The tool doesn't self describe itself well. Looking at the<br>
 help(), or the __repr__(), or the tooltips did not provide<br>
 much insight or clarity. The dir() shows many of the<br>
 _abc implementation details rather than the API itself.<br>
<br>
* The original key is stored and if you change it, the change<br>
 isn't stored. The _original dict is private (perhaps to<br>
 reduce the risk of putting the TD in an inconsistent state)<br>
 but this limits access to the stored data.<br>
<br>
* The TD is unsuitable for bijections because the API is<br>
 inherently biased with a rich group of operators and methods<br>
 for forward lookup but has only one method for reverse lookup.<br>
<br>
* The reverse feature is hard to find (getitem vs __getitem__)<br>
 and its output pair is surprising and a bit awkward to use.<br>
 It provides only one accessor method rather that the full<br>
 dict API that would be given by a second dictionary. The<br>
 API hides the fact that there are two underlying dictionaries.<br>
<br>
* It was surprising that when d[k] failed, it failed with<br>
 transformation exception rather than a KeyError, violating<br>
 the expectations of the calling code (for example, if the<br>
 transformation function is int(), the call d["12"]<br>
 transforms to d[12] and either succeeds in returning a value<br>
 or in raising a KeyError, but the call d["12.0"] fails with<br>
 a TypeError). The latter issue limits its substitutability<br>
 into existing code that expects real mappings and for<br>
 exposing to end-users as if it were a normal dictionary.<br>
<br>
* There were other issues with dict invariants as well and<br>
 these affected substitutability in a sometimes subtle way.<br>
 For example, the TD does not work with __missing__().<br>
 Also, "k in td" does not imply that "k in list(td.keys())".<br>
<br>
* The API is at odds with wanting to access the transformations.<br>
 You pay a transformation cost both when storing and when<br>
 looking up, but you can't access the transformed value itself.<br>
 For example, if the transformation is a function that scrubs<br>
 hand entered mailing addresses and puts them into a standard<br>
 format with standard abbreviations, you have no way of getting<br>
 back to the cleaned-up address.<br>
<br>
* One design reviewer summarized her thoughts like this:<br>
 "There is a learning curve to be climbed to figure out what<br>
 it does, how to use it, and what the applications [are].<br>
 But, the [working out the same] examplea with plain dicts<br>
 requires only basic knowledge." -- Patricia<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/guido%40python.org" target="_blank">https://mail.python.org/mailman/options/python-dev/guido%40python.org</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div>