Overloading operators on the rigth.

[Alex Martelli]

Bengt Richter <bokr at oz.net> wrote:

> Unfortunately, I am still frustrated, because there seems no way (other
> than looking in sources) to discover a method whose name you don't know
> ahead of time. You can just determine the order of access to known-name
> methods. So I'm still looking for the magic metaclass formula that can
> reveal the search for unknown methods. Maybe we need a sys._getattrhook
> that could monitor operation-induced searching?

I assume you're talking about types, aka newstyle classes, because for
oldstyle ones __getattr__ does get into play.

Now, the problem is as follows: no searching, stricto sensu, is induced
by an operation, in most cases.  Rather, *when the type is built* some
code is responsible for filling out a C-level structure, the 'type'
struct, which mostly holds 'slots' which are pointers to C functions.
This "phase 0" is the time when some special method names may be checked
for, in the class dictionary and bases, in order to determine what to
place in the slots (a null pointer, a pointer to a C function which will
call the appropriate Python function, whatever).

What the operation induces, call it "phase 1", is one or a few checks
regarding those C pointers and sometimes after one is used for a call a
check for NonImplemented as a return value to determine whether to keep
checking.  Typically no Python names are involved during phase 1 (the
main exception being classic classes, which are singled out for special
and more complicated treatment for backwards compatibility purposes).

((There may be a "phase 2" when you modify a class object, in which some
slots in the C-level structure are altered; but I don't see any way,
even conceptually, to exploit this for "fishing for unknown names")).

By writing a custom metaclass you may alter phase 0, but being able to
alter a certain complicated processing need not mean you can easily
introspect it.  If the slot-filling process was altered to accept and
fully support any mapping in lieu of the class dict, you could thus get
the information about all the names that the slot-filling is looking for
(not necessarily how the presence of such names affects what slots, but
still you might use the bare info on names to inject artificially
constructed methods or other descriptors which later provide some
tracing information, perhaps).

Thinking about phase 1, the best architecture I can imagine for that
involves a C extension which, given a type object, instruments all of
its relevant slots, wrapping each C function it finds with a hook-caller
that is called for the specific purpose of providing tracing information
or other aspect-oriented before/after hooking.

I would assess the amount of work required for either or both projects
as non-trivial, particularly if one wants to do it in such a way as to
not impact performance (so it can stay in some future released Python,
maybe 2.5).  It also appears to inevitably involve some reasonably deep
tinkering with internals that are currently coded in C.  Of all the
separate aspects involved, the one with potentially highest return would
appear to me to be the ability to use any mapping, not just a dict --
that one might perhaps provide other benefits, once in place.  It can be
done without excessive performance penalty by acting similarly to the
implementation of the STORE_LOCAL opcode (I'm not sure how you could
ever get the 'locals dict' of the current frame to be anything but a
real dict at the time a STORE_LOCAL executes, but, if you manage that,
STORE_LOCAL's implementation *is* coded to [a] specialcase dict first
for speed, [b] fallback to more abstract setitem if the locals of the
current frame are not exactly a dict, i.e. if they're an instance of any
dict subclass or anything else whatsoever).

On a parallel thread (about "reimplementing Python"), I'm pointing out
one potential advantage of rewriting an existing project in a higher
level language: tinkering and experimentation such as this becomes way
easier.  The specific project I'm involved with, pypy, also enables the
tinkerer to know (just about) only Python, not necessitating other
programming languages (well, pyrex would still help for some of the
kinds of tinkering one might want, and so might machine language and
even, in-between, C or Common Lisp; but mucho tinkering could be done at
a purely Python level, if pypy matures and consolidates as we hope).


Alex