[Python-Dev] FW: METH_COEXIST

[Raymond Hettinger]

Long ago, I noticed that non-operator calls to dict.__contains__() ran
about 50% slower than dict.has_key().  Since then, the issue has
surfaced several times for other wrapped methods like __init__(),
__getitem__(), and __setitem__(), and next().  Also, it came up again in
the context of set.__contains__.  The latter bugged me enough to get to
the bottom of it (unlike dicts, sets do not have has_key() as an
alternative so you feel it when writing ismember=myset.__contains__).

Over 95% of the speed difference is due to PyCFunctions having optimized
calls.  Of that, much of the benefit comes from the METH_O and
METH_NOARGS methods having their arguments passed directly instead of by
assembling and disassembling an argument tuple.

If the problem were widespread and important in every context, the
solution would be to provide wrapperobjects with special cases just like
PyCFunction.

Instead, there is a minimally invasive alternative.  If I could just
write a "__contains__" method as a PyCFunction, the job would be done.
Unfortunately, if the corresponding slot has already filled-in the
method with a wrapper object, the explicitly defined method is ignored.
While that is normally fine, it is possible to make the explicitly
defined method coexist with the slot.

So, I propose to add a method flag, METH_COEXIST, which allows an entry
in the method table to overwrite the wrapperobject and coexist with the
slot:

*** typeobject.c        13 Nov 2003 22:50:00 -0000      2.253
--- typeobject.c        12 Dec 2003 13:58:42 -0000
***************
*** 2792,2799 ****

        for (; meth->ml_name != NULL; meth++) {
                PyObject *descr;
!               if (PyDict_GetItemString(dict, meth->ml_name))
!                       continue;
                if (meth->ml_flags & METH_CLASS) {
                        if (meth->ml_flags & METH_STATIC) {
                                PyErr_SetString(PyExc_ValueError,
--- 2792,2802 ----

        for (; meth->ml_name != NULL; meth++) {
                PyObject *descr;
!               if (PyDict_GetItemString(dict, meth->ml_name)) {
!                       if (!(meth->ml_flags & METH_COEXIST))
!                               continue;
!                       PyDict_DelItemString(dict, meth->ml_name);
!               }
                if (meth->ml_flags & METH_CLASS) {
                        if (meth->ml_flags & METH_STATIC) {
                                PyErr_SetString(PyExc_ValueError,


With that minor change, it becomes possible to write:

static PyMethodDef mapp_methods[] = {
      {"__contains__",	(PyCFunction)dict_has_key, METH_O |
METH_COEXIST,
       contains__doc__},


And, presto, now calls to dict.__contains__() are as fast as
dict.has_key().

The two immediate applications are dict.__contains__() and
set.__contains__(). 

Guido is basically okay with the idea but thought I should check here
first to see if anyone has any profound thoughts about unexpected
implications or perhaps a better way.

If not, then I would like to make the above changes (define the flag,
teach typeobject.c to recognize the flag, and apply it to dicts and
sets).


Raymond Hettinger