[Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)]

[Alex Martelli]

On Apr 2, 2006, at 7:11 AM, Paul Moore wrote:
    ...
> On the face of it, there's nothing implicit going on. There has to be
> an explicit adaptation call. However, I agree that systems that make
> extensive use of adaptation can seem to end up in a situation where
> data seems to magically appear without anybody providing it. Black
> magic of that level seems to me to be a clear abuse, though. The

I believe that such "magically appearing" does not depend on  
adaptation, per se, but on the mix of "convenience" approaches to  
adaptation and registration that one chooses to provide alongside it.

With the simplistic scheme I illustrated (protocols denoted by unique  
strings, adaptation occurring only from an object's exact [leafmost]  
type to a specific protocol depending on registration, not even  
support for inheritance of types, much less of protocols, no  
transitivity whatsoever, ...) there is exactly zero risk of anything  
"magically appearing".

Of course, the flip side of that is the inconvenience of having to  
register everything.  I do believe that inconvenience can be reduced  
by supporting mechanisms that Python programmers are already very  
familiar to, such as inheritance of types: nobody finds anything  
strange if, after declaring 'class X(Y):...', they find that  
instances of X "magically acquire" attributes supply by Y -- after  
all, that IS roughly the whole POINT of doing inheritance;-).  I  
believe that, similarly, nothing will be found surprising if, along  
with attributes, X's instances inherit the adaptations enjoyed by Y  
(of course, inherited adaptations can be overridden, just like  
inherited attributes can).

But supplying something like the "duck adaptation" that Brett is so  
keen on might easily destroy this kind of transparency: essentially,  
that would be saying something like """forget the twaddle about  
syntax, semantics and pragmatics: we won't even check half of SYNTAX  
compatibility [[the fact that methods need to be signature- 
compatible]], just the possibly-accidental coincidence of method  
NAMES, and we'll have a hearty laugh at the expense of anybody  
foolish enough to think that "T conforms to protocol P" should mean  
"I have studied the documentation about all the constraints of P, and  
the implementation of T, and I certify that the latter meets the  
former with no need for any wrapping"...quack, quack!""".

After all, we can all plainly see that "class GraphicArtist" supplies  
a method call "draw", and since that's what protocol  
'wild.west.gunslinger' syntactically requires (net of signature- 
compatibility), we can obviously claim that all graphic artists are  
gunslingers, quack quack.

Now if THAT was part of one's adaptation/registration machinery, I  
can well see it as problematic enough for the poor artist who finds  
himself cast in a duel at the OK Corrall.

To a lesser extent, "inheritance of protocols" might sometimes cause  
surprises (though Philip Eby has extensive experience with it and  
claims otherwise, I do not know how much of his complete lack of  
surprises and confusion in the matter might depend on his personal  
skills and not be applicable to many other programmers).  That would  
be a feature like: a protocol P1 might be declared as "inheriting  
from" (or "adaptable to") another protocol P2.  Then, when type T is  
adaptable to P1, and we're looking for an adaptation of T to P2, we'd  
walk a path T -> P1 -> P2 -- possibly with two wrappers in play  
(unless protocol inheritance does not support any true wrapping, just  
identity; in the case the one and only wrapper, is any, is the one  
for the T -> P1 adaptation).  This seems attractive, but at least  
potentially it does mean there might be multiple ways to get from T  
to P2, and it's possible that not all ways would be obvious to an  
observer.  This also applies to inheritance of types, but in that  
case we're all perfectly used to Python solving such "ambiguities" by  
walking T's MRO, so I contend there would arise no surprise nor  
ambiguity; for adaptation of protocol to protocol, we don't have a  
similar intuition based on solid experience to see us through.


>> OTOH, there may be a hidden assumption among the fans of  
>> adaptation that
>> adaptation to a mutable interface should never add state to, nor  
>> copy the
>> state of, an adapted object. Any mutation made via an adaptor  
>> would be
>> reflected as a mutation of the original object. Adaptation to  
>> immutable
>> interfaces would always be fine, naturally. If that's an unwritten  
>> rule of
>> adaptation, then:
>>    1. It addresses the main evil of implicit type conversion  
>> (hidden state)
>>    2. It needs to become a *written* rule, so that anyone writing  
>> a stateful
>> adapter can be duly admonished by their peers
>
> I don't know if that's an "unwritten rule" as such - but I can barely
> imagine what you're describing as unacceptable (adaptation to a
> mutable interface which adds or copies state). It just seems like a
> stupid thing to do (or at least, not at all what adaptation is about).
> But maybe that's what you mean by a "hidden assumption".

Uh?  Consider iteration -- that's a prime example of an adaptation  
which adds "hidden state", and I don't see it as particularly  
problematic to frame as an adaptation.

When I adapt an instance L of list to 'org.python.stdlib.iterator', I  
add one bit of state -- the "current index".  I mutate that state  
each and every time I call .next() on the resulting object, and that  
state is not at all reflected on the original list, which in fact is  
totally unaware of whether there are iterators outstanding on it, and  
if so, how many, and in which states.

Why is this seen as a problem?

> Regardless, I'd have no problem with a style guide, or good practice
> document, stating that this is what adaptation is about, and stateful
> adapters are bad practice. (That's just my opinion - better check this
> with people who make heavy use of adaptation). But to me it feels like
> labouring the obvious - along the lines of explicitly prohibiting
> metaclass or decorator abuse.

Do we have a "good practice document" about what you should or  
shouldn't do with metaclasses, or decorators, or, for that matter,  
with inheritance, operator overloading, and other powerful constructs  
and tools that have been with Python a long, long time?

>> The other thing is that it makes more sense to me for there to be a
>> per-protocol type->adapter registry, rather than a global registry  
>> with tuples
>> of source type/target protocol pairs.
>
> What difference would that make in practice?

Not all that much: I see this as a mere implementation detail.  You  
can implement a mapping from (A,B) to C in two ways (and, no doubt,  
many others yet):

-- the most direct one: a dict with (A,B) as the key, C as the value
-- an indirect one: a dict with A as the key, whose value is a dict  
with B as the key and C as the value

If all you do is lookups from (A,B) to get C, the former is simpler;  
the latter may be faster when you need to look up all B's  
corresponding to an A, since it makes that operation O(1) rather than  
O(N).  So, if you add to the adapt and register primitives other  
primitives for "give me all protocols to which T can be adapted", or  
"give me all types which can adapted to P", choosing the "right"  
nested-dict implementation might make one of these lookups (not both)  
faster.  Probably worth doing only if such lookup are indeed  
frequent; since I believe their frequency of use will be tiny (as  
will the number of calls to registration) compared with the number of  
calls to adapt, I'd go all out to optimize the latter, first and  
foremost -- after which, one can pick tradeoffs between cost of  
registration and costs of other kinds of lookups (after all, one  
might also want to see "all (A,B)s for a given C", no?-).  For  
example, keep the "most direct one" dict, and add auxiliary ones to  
support other lookups -- this makes registration slower (but it's a  
very rare operation anyway) and takes up a bit more memory (but, I  
believe we're talking pennies), but can speed up all kinds of lookups.

Exactly why we're so prematurely discussing fine-tuning-level  
optimization concerns, at this stage, escapes me a bit, though.

>> Secondly, given that each framework is likely to be defining the  
>> protocols
>> that it consumes, I don't see the problem with each one defining  
>> its *own*
>> adaptation registry, rather than having one mega-registry that adapts
>> everything to everything.
> [...]
>> Then the role of an adaptation module in the standard library  
>> would be to
>> provide a standard API for per-framework registries, without also  
>> providing a
>> mega-registry for adapting everything to everything.
>
> Not an unreasonable idea, but how valuable would it be in practice?
> Alex's proposal allowed for explicitly specifying a registry, while
> still having a default "central" registry. For 99% of use, I'd suspect
> that people would not bother with a special registry. And if protocols

I don't think this other tweak would be a _big_ "bother", but neither  
would it be at all useful, just a medium-level useless bother.

Say, for example, that protocols are identified (as in my strawman  
proposal) by unique strings anyway. E.g., if I were to invent a  
protocol, I could name it 'it.aleax.myprot' -- since I own the  
aleax.it domain, nobody else could create a name conflict.  Saying  
that each framework has a separate registry is just the same as  
saying that each protocol "lives" in one specific registry, so that  
any registration or lookup regarding protocol P *must* also specify  
registryof(P).  Hopefully, rather than having to keep this  
correspondence in our human memory, we're allowed to have a registry  
of registries which remembers this correspondence for you: we can  
register_protocol(P, registry) and we can lookup the registry for a  
given protocol with function registryof.  E.g.:

_reg_of_regs = {}
def register_protocol(P, registry): _reg_of_regs[P] = registry
def registryof(P): return _reg_of_regs[P]

So now, all calls which, in my proposal, would be (e.g.) adapt(x, P),  
must instead become adapt(x, P, registryof(P)).

Not a big bother, just an amount of totally useless boilerplate  
that's just sufficient to be annoying, it seems to me.  Of course, if  
the registry of registries was somehow forbidden, then the bother  
WOULD become bigger, since in that case the poor programmer would  
have to mentally memorize, or continually look up (with grep, Google  
search, or similar means) the total equivalent of the registryof(P)  
function result.

I may be missing something here, I guess, because I just don't see  
the point.

> were defined via some "interface" approach (like zope.interfaces and
> PyProtocols do) then encapsulation is taken care of by uniqueness of
> types/interfaces. I know interfaces are outside the scope of what's
> being proposed right now, but one of their benefits is that they *do*
> solve this problem. Structured strings naming protocols
> ("org.python.std.index" or whatever) do this as well, but without
> language support.

I did mention that one issue with my "strawman proposal" was exactly  
that it performs no error checking: it entirely relies on programers  
respecting some simple and reasonable conventions, rather than piling  
up machinery to provide enforcement. Much like, oh, say, Python.

Isn't it just wonderful, how the foes of adaptation switch horses on  
you?  First they request a simple-as-dirt, bare-bones "example  
system" -- then as soon as you provide one they come back at you with  
all sort of "cruft" to be piled on top.  Ah well, having tried to  
evangelize for adaptation for years, I've grown wearily accustomed to  
this kind of response; it sometimes looks more like such foes are  
feeling defensive, and ready to pull any trick to stop adaptation  
from getting in the language, rather than interested in the various  
technica aspect of the issue.  To be fair, this isn't all that  
different from the average reaction one always gets from python-dev  
as a whole to any proposal whatsoever.  Anyway, I hope it's clearer  
now why, each and every time, I end up giving up, and deciding that  
beating my head against a wall is more productive and fun;-).

I guess I'll just adopt a signature of "Praeterea censeo adaptatio  
esse adoptanda!", for all the good that all the detailed discussions  
appear to have been doing;-).


Alex