I just attempted to post the Monkey Typing draft pre-PEP, but it bounced
due to being just barely over the size limit for the list. :)
So, I'm just posting the preamble and abstract here for now, and a link to
a Wiki page with the full text. I hope the moderator will approve the
actual posting soon so that replies can quote from the text.
=== original message ===
This is only a partial first draft, but the Motivation section nonetheless
attempts to briefly summarize huge portions of the various discussions
regarding adaptation, and to coin a hopefully more useful terminology than
some of our older working adjectives like "sticky" and "stateless" and
such. And the specification gets as far as defining a simple
decorator-based syntax for creating operational (prev. "stateless") and
extension (prev. "per-object stateful") adapters.
I stopped when I got to the API for declaring volatile (prev. per-adapter
stateful) adapters, and for enabling them to be used with type
declarations, because Clark's post on his revisions-in-progress seem to
indicate that this can probably be handled within the scope of PEP 246
itself. As such, this PEP should then be viewed more as an attempt to
formulate how "intrinsic" adapters can be defined in Python code, without
the need to manually create adapter classes for the majority of
type-compatibility and "extension" use cases. In other words, the
implementation described herein could probably become part of the front-end
for the PEP 246 adapter registry.
Feedback and corrections (e.g. if I've repeated myself somewhere, spelling,
etc.) would be greatly appreciated. This uses ReST markup heavily, so if
you'd prefer to read an HTML version, please see:
But I'd prefer that corrections/discussion quote the relevant section so I
know what parts you're talking about. Also, if you find a place where a
more concrete example would be helpful, please consider submitting one that
I can add. Thanks!
Title: "Monkey Typing" for Agile Type Declarations
Version: $Revision: X.XX $
Last-Modified: $Date: 2003/09/22 04:51:50 $
Author: Phillip J. Eby <pje(a)telecommunity.com>
Type: Standards Track
Python has always had "duck typing": a way of implicitly defining types by
the methods an object provides. The name comes from the saying, "if it walks
like a duck and quacks like a duck, it must *be* a duck". Duck typing has
enormous practical benefits for small and prototype systems. For very large
frameworks, however, or applications that comprise multiple frameworks, some
limitations of duck typing can begin to show.
This PEP proposes an extension to "duck typing" called "monkey typing", that
preserves most of the benefits of duck typing, while adding new features to
enhance inter-library and inter-framework compatibility. The name comes from
the saying, "Monkey see, monkey do", because monkey typing works by stating
how one object type may *mimic* specific behaviors of another object type.
Monkey typing can also potentially form the basis for more sophisticated type
analysis and improved program performance, as it is essentially a simplified
form of concepts that are also found in languages like Dylan and Haskell. It
is also a straightforward extension of Java casting and COM's QueryInterface,
which should make it easier to represent those type systems' behaviors within
Python as well.
[see the web page above for the remaining text]
At 11:50 PM 1/15/05 +0100, Just van Rossum wrote:
>Phillip J. Eby wrote:
> > >But it _does_ perform an implicit adaptation, via PyObject_GetIter.
> > First, that's not implicit. Second, it's not adaptation, either.
> > PyObject_GetIter invokes the '__iter__' method of its target -- a
> > method that is part of the *iterable* interface. It has to have
> > something that's *already* iterable; it can't "adapt" a non-iterable
> > into an iterable.
> > Further, if calling a method of an interface that you already have in
> > order to get another object that you don't is adaptation, then what
> > *isn't* adaptation? Is it adaptation when you call 'next()' on an
> > iterator? Are you then "adapting" the iterator to its next yielded
> > value?
>That's one (contrived) way of looking at it. Another is that
> y = iter(x)
>adapts the iterable protocol to the iterator protocol. I don't (yet) see
>why a bit of state disqualifies this from being called adaptation.
Well, if you go by the GoF "Design Patterns" book, this is actually what's
called an "Abstract Factory":
"Abstract Factory: Provide an interface for creating ... related or
dependent objects without specifying their concrete classes."
So, 'iter()' is an abstract factory that creates an iterator without
needing to specify the concrete class of iterator you want. This is a much
closer fit for what's happening than the GoF description of "Adapter":
"Adapter: Convert the interface of a class into another interface
clients expect. Adapter lets classes work together that couldn't otherwise
because of incompatible interfaces."
IMO, it's quite "contrived" to try and squeeze iteration into this concept,
compared to simply saying that 'iter()' is an abstract factory that creates
"related or dependent objects".
While it has been pointed out that the GoF book is not handed down from
heaven or anything, its terminology is certainly widely used to describe
certain patterns of programming. If you read their full description of the
adapter pattern, nothing in it is about automatically getting an adapter
based on an interface. It's just about the idea of *using* an adapter that
you already have, and it's strongly implied that you only use one adapter
for a given source and destination that need adapting, not create lots of
instances all over the place.
So really, PEP 246 'adapt()' (like 'iter()') is more about the Abstract
Factory pattern. It just happens in the case of PEP 246 that it's an
Abstract Factory that *can* create adapters, but it's not restricted to
handing out *just* adapters. It can also be used to create views,
iterators, and whatever else you like. But that's precisely what makes it
problematic for use as a type declaration mechanism, because you run the
risk of it serving up entirely new objects that aren't just interface
transformers. And of course, that's why I think that you should have to
declare that you really want to use it for type declarations, if in fact
it's allowed at all. Explicit use of 'adapt()', on the other hand, can
safely create whatever objects you want.
Oh, one other thing -- distinguishing between "adapters" and merely
"related" objects allows you to distinguish whether you should adapt the
object or what it wraps. A "related" object (like an iterator) is a
separate object, so it's safe to adapt it to other things. An actual
*adapter* is not a separate object, it's an extension of the object it
wraps. So, it should not be re-adapted when adapting again; instead the
underlying object should be adapted.
So, while I support in principle all the use cases for "adaptation"
(so-called) that have been discussed here, I think it's important to refine
our terminology to distinguish between GoF "adapters" and "things you might
want to create with an abstract factory", because they have different
requirements and support different use cases.
We have gotten a little bogged down by our comparisons of "good" and "bad"
adapters; perhaps to move forward we should distinguish between "adapters"
and "views", and say that an iterator is an example of a view: you may have
more than one view on the same thing, and although a view depends on the
thing it "views", it doesn't really "convert an interface"; it provides
distinct functionality on a per-view basis.
Currently, PEP 246 'adapt()' is used "in the field" to create both adapters
and views, because 1) it's convenient, and 2) it can. :) However, for
type declarations, I think it's important to distinguish between the two,
to avoid implicit creation of additional views.
A view needs to be managed within the scope that it applies to. By that, I
mean for example that a 'for' loop creates an iterator view and then
manages it within the scope of the loop. However, if you need the iterator
to remain valid outside the 'for' loop, you may need to first call 'iter()'
to get an explicit iterator you can hold on to.
Similarly, if you have a file that you are reading things from by calling
routines and passing in the file, you don't want to pass each of those
routines a filename and have them implicitly open the file; they won't be
reading from it sequentially then. So, again, you have to manage the view
by opening a file or creating a StringIO or whatever.
Granted that there are some scenarios where implicit view creation will do
exactly the right thing, introducing it also opens the opportunity for it
to go very badly. Today's PEP 246 implementations are as easy to use as
'iter()', so why not use them explicitly when you need a view?
At 10:09 AM 1/14/05 +0100, Just van Rossum wrote:
>Guido van Rossum wrote:
> > Are there real-life uses of stateful adapters that would be thrown out
> > by this requirement?
>Here are two interfaces we're using in a project:
> http://just.letterror.com/ltrwiki/PenProtocol (aka "SegmentPen")
>They're both abstractions for drawing glyphs (characters from a font).
>Sometimes the former is more practical and sometimes the latter. We
>really need both interfaces. Yet they can't be adapted without keeping
>some state in the adapter.
Maybe I'm missing something, but for those interfaces, isn't it okay to
keep the state in the *adapted* object here? In other words, if PointPen
just added some private attributes to store the extra data?
>Implicit adaptations may be dangerous here, but I'm not so sure I care.
>In my particular use case, it will be very rare that people want to do
But if the extra state were stored on the segmentPen rather than the
adapter, this would work correctly, wouldn't it? Whereas with it stored in
an adapter, it wouldn't.
> I don't see how [LiskovViolation could have a more descriptive name].
> Googling on Liskov immediately brings up clear and understandable
> descriptions of the principle
> Clearly, I disagree. [...]
> I had never heard the term before and consulted the Google oracle as well.
This must be one of those cases where I am mislead by my background...
I thought of Liskov substitution principle as a piece of basic CS
background that everyone learned in school (or from the net, or wherever
they learned programming). Clearly, that's not true.
> How about SubstitutabilityError?
It would be less precise and informative to ME but apparently more so
to a beginner. Obviously, we should support the beginner!
-- Michael Chermside
At 10:39 AM 1/15/05 +0100, Just van Rossum wrote:
>That sounds extremely complicated as apposed to just storing the sate
>where it most logically belongs: on the adapter.
Oh, the state will be on the adapter all right. It's just that for type
declarations, I'm saying the system should return the *same* adapter each time.
> And all that to work
>around a problem that I'm not convinced needs solving or even exists. At
>the very least *I* don't care about it in my use case.
> > Anyway, as you and I have both pointed out, sticky adaptation is an
> > important use case; when you need it, you really need it.
>Maybe I missed it, but was there an example posted of when "sticky
>adaptation" is needed?
No; but Glyph and I have independent use cases for them. Here's one of
mine: code generation from a UML or MOF model. The model classes can't
contain methods or data for doing code generation, unless you want to cram
every possible kind of code generation into them. The simple thing to do
is to adapt them to a PythonCodeGenerator or an SQLCodeGenerator or
what-have-you, and to do so stickily. (Because a code generator may need
to walk over quite a bit of the structure while keeping state for different
things being generated.)
You *could* keep state in an external dictionary, of course, but it's much
easier to use sticky adapters.
>It's not at all clear to me that "sticky" behavior is the best default
>behavior, even with implicit adoptation. Would anyone in their right
>mind expect the following to return [0, 1, 2, 3, 4, 5] instead of [0, 1,
>2, 0, 1, 2]?
> >>> from itertools import *
> >>> seq = range(10)
> >>> list(chain(islice(seq, 3), islice(seq, 3)))
> [0, 1, 2, 0, 1, 2]
I don't understand why you think it would. What does islice have to do
For a little background, I'm working on making an edit and continue
support in python a little more robust. So, in replacing references to
unmodifiable types like tuples and bound-methods (instance or class), I
iterate over gc.get_referrers.
So, I'm working on frame types, and wrote this code::
def replaceFrame(self, ref, oldValue, newValue):
for name, value in ref.f_locals.items():
if value is oldValue:
ref.f_locals[name] = newValue
assert ref.f_locals[name] is newValue
But unfortunately, the assert fires. f_locals is writable, but not
modifiable. I did a bit of searching on Google Groups, and found
references to a desire for smalltalk like "swap" functionality using a
similar approach, but no further ideas or solutions.
While I am full well expecting the smack of "don't do that", this
functionality would be very useful for debugging long-running
applications. Is this possible to implement in CPython and ports? Is
there an optimization reason to not do this?
At worst, if this isn't possible, direct references in the stack will be
wrong above the reload call, and corrected on the invocation of the
function. This is a subtle issue with reloading code, and can be
documented. And at best there is an effective way to replace it, the
system can be changed to a consistent state even in the stack, and I can
rejoice. Even if I have to wait until 2.5. ;)
Thanks for your time!
> Date: Fri, 14 Jan 2005 02:38:05 -0500
> From: "Phillip J. Eby" <pje(a)telecommunity.com>
> Subject: Re: [Python-Dev] PEP 246: lossless and stateless
> To: guido(a)python.org
> Cc: "Clark C. Evans" <cce(a)clarkevans.com>, python-dev(a)python.org
> Message-ID: <18.104.22.168.0.20050114014238.0308e850(a)mail.telecommunity.com>
> Content-Type: text/plain; charset="us-ascii"; format=flowed
> Each of these examples registers the function as an implementation of the
> "file.read" operation for the appropriate type. When you want to build an
> adapter from SomeKindOfStream or from a string iterator to the "file" type,
> you just access the 'file' type's descriptors, and look up the
> implementation registered for that descriptor for the source type
> (SomeKindOfStream or string-iter). If there is no implementation
> registered for a particular descriptor of 'file', you leave the
> corresponding attribute off of the adapter class, resulting in a class
> representing the subset of 'file' that can be obtained for the source class.
> The result is that you generate a simple adapter class whose only state is
> a read-only slot pointing to the adapted object, and descriptors that bind
> the registered implementations to that object. That is, the descriptor
> returns a bound instancemethod with an im_self of the original object, not
> the adapter. (Thus the implementation never even gets a reference to the
> adapter, unless 'self' in the method is declared of the same type as the
> adapter, which would be the case for an abstract method like 'readline()'
> being implemented in terms of 'read'.)
> Anyway, it's therefore trivially "guaranteed" to be stateless (in the same
> way that an 'int' is "guaranteed" to be immutable), and the implementation
> is also "guaranteed" to be able to always get back the "original" object.
> Defining adaptation in terms of adapting operations also solves another
> common problem with interface mechanisms for Python: the dreaded "mapping
> interface" and "file-like object" problem. Really, being able to
> *incompletely* implement an interface is often quite useful in practice, so
> this "monkey see, monkey do" typing ditches the whole concept of a complete
> interface in favor of "explicit duck typing". You're just declaring "how
> can X act 'like' a duck" -- emulating behaviors of another type rather than
> converting structure.
I get it! Your last description didn't quite sink in but this one does and
I've been thinking about this quite a bit, and I like it. I'm starting to
see how it nicely sidesteps the problems discussed in the thread so far.
Partial implementation of interfaces (read, implementing only the operations
you care about on a method by method basis instead of an entire interface)
really is very useful and feels quite pythonic to me. After all, in most
cases of substitutability in Pyhton (in my experience), it's not the *type*
you do anything with, but that type's operations. Does anyone know of any
other languages that take this "operational" aproach to solving the
There seem to be some downsides vs. interfaces (I think) the lack of "it's
documentation too" aspect, I find zope 3 interfaces.py modules the best way
to learn about it, but again the upside is, no complex interface
relationships just to define the subtle variations of "mapping" and users can
always just say help(file.read).
Another thing I see used fairly commonly are marker interfaces. While I'm not
sure of their overall usefullness I don't see how they can be done using your
operational scheme. Maybe that means they were a bad idea in the first
I also think this is easier for beginners to understand, instead of "you have
to implement this interface, look at it over here, that's the "file"
interface, now you implement that in your object and you better do it all
right" you just tell them "call your method 'read' and say its 'like
file.read' and your thing will work where any file can be read.
At 06:56 PM 1/14/05 +0100, Just van Rossum wrote:
>Phillip J. Eby wrote:
> > For example, if there were a weak reference dictionary mapping
> > objects to their (stateful) adapters, then adapt() could always
> > return the same adapter instance for a given source object, thus
> > guaranteeing a single state.
>Wouldn't that tie the lifetime of the adapter object to that of the
Well, you also need to keep the object alive if the adapter is still
hanging around. I'll get to implementation details and alternatives in the
>Possibly naive question: is using adaptation to go from iterable to
>iterator abuse? That would be a clear example of per-adapter state.
I don't know if it's abuse per se, but I do know that speciifying whether a
routine takes an iterable or can accept an iterator is often something
important to point out, and it's a requirement that back-propagates through
code, forcing explicit management of the iterator's state.
So, if you were going to do some kind of adaptation with iterators, it
would be much more useful IMO to adapt the *other* way, to turn an iterator
into a reiterable. Coincidentally, a reiterable would create per-object
In other words, if you *did* consider iterators to be adaptation, it seems
to me an example of wanting to be explicit about when the adapter gets
created, if its state is per-adapter. And the reverse scenario
(iterator->reiterable) is an example of adaptation where shared state could
solve a problem for you if it's done implicitly. (E.g. by declaring that
you take a reiterable, but allowing people to pass in iterators.)
back in Python 2.1 (and before), an object could define how copy.copy should
work simply by definining a __copy__ method. here's the relevant portion:
copierfunction = _copy_dispatch[type(x)]
copier = x.__copy__
raise error, \
"un(shallow)copyable object of type %s" % type(x)
y = copier()
I recently discovered that this feature has disappeared in 2.3 and 2.4. in-
stead of looking for an instance method, the code now looks at the object's
cls = type(x)
copier = _copy_dispatch.get(cls)
copier = getattr(cls, "__copy__", None)
(copy.deepcopy still seems to be able to use __deepcopy__ hooks, though)
is this a bug, or a feature of the revised copy/pickle design? (the code in
copy_reg/copy/pickle might be among the more convoluted pieces of python
coding that I ever seen... and what's that smiley doing in copy.py?)
and if it's a bug, does the fact that nobody reported this for 2.3 indicate that
I'm the only one using this feature? is there a better way to control copying
that I should use instead?