PyArg_ParseTuple O format semantics

Robin Becker robin at jessikat.fsnet.co.uk
Sat Sep 8 04:34:41 EDT 2001


In article <mailman.999911770.2065.python-list at python.org>, Tim Peters <tim.one at home.com> writes
>[Robin Becker]
>> Steve Alexander kindly forwarded me a note relating to a change in
.... see below
>
>News to me, and I see nothing about that in the CVS log.  The only 2.0.1
>patch to getargs.c plugged a memory leak in vgetargskeywords(); it was
>leaving the refcounts on names of keyword arguments too high, when iterating
>over a keyword dict.
>

I guess that might be it. I suppose if anyone had taken action to correct
the memory leak then when it was fixed --> 2.0.1 then their extra DECREF
would be wrong.

>> The documentation doesn't mention any increment so it seems to return a
>> borrowed ref.
>
>The docs are explicit:  "any Python object references which are provided to
>the caller are borrowed references; do not decrement their reference
>count!".
>

That's what I thought. I guess I hadn't noticed if any leaks were happening.

>> I'm using the O flag a bit and would like to know if I should be
>> patching to cover the change -->2.0.1?
>
>Doubt it; there doesn't appeart to be any relevant change in 2.0.1, but I
>haven't seen the note you're paraphrasing either.
>
>


This is his included Note
=================================================================
Subject: [Zope-dev] Zope (and Python) crashing bug found
Date: 06 Sep 2001 02:37:05 -0400
From: Andy Dustman <andy at dustman.net>
To: zope-dev at zope.org

There have been a couple of reports from various people (myself
included) about a bug that will cause Zope to mysteriously die and dump
core. I myself had the same problem, particularly with MySQLdb on recent
versions of Python. I decided I better hunt the bastard down sooner
rather than later...

Symptoms: Mysterious core dumps fairly early on after starting up
MySQLdb, isolated to the first query. After some investigation, I
determined that this did not occur with Python 1.5.2 or 2.0, but does
occur with 2.0.1, 2.1, and 2.1.1.

Now, when calling _mysql.connect(), one of the items that you MySQLdb
will pass in is a type conversion dictionary as a keyword argument, i.e.
apply(connect, args, kwargs). After injecting some print statements here
and there, and some getrefcount(), I found that the refcount on the
converter dictionary was going to zero, i.e. getrefcount() returns 1.
However, this only happened with Python > 2.0.

So I got out diff and found the critical difference between 2.0 (MySQLdb
not broken) and 2.0.1 (MySQLdb broken):

diff -u -r Python-2.0/Python/getargs.c Python-2.0.1/Python/getargs.c
--- Python-2.0/Python/getargs.c Mon Oct 16 17:49:29 2000
+++ Python-2.0.1/Python/getargs.c       Sat Mar 31 08:18:35 2001
@@ -1124,6 +1124,7 @@
                                 return 0;
                         }
                         converted++;
+                       Py_DECREF(item);
                 }
                 else {
                         PyErr_Clear();

The general upshot of this is: In versions of Python (numerically) prior
to 2.0.1, if you use the O format with PyArgs_ParseTupleAndKeywords()
(and possibly also PyArgs_ParseTuple()) to return a PyObject *, you get
a new reference. In 2.0.1 and later, you get a borrowed reference and
must Py_INCREF() if you want to keep it. In Python 1.5.2, however, doing
this causes a memory leak! Since I want to keep compatibility with
Python 1.5.2 for now, my solution is this:

        if (!conv)
                conv = PyDict_New();
#if PY_VERSION_HEX > 0x02000100
        else
                Py_INCREF(conv);
#endif

In this example, conv is initialized to NULL, and
PyArgs_ParseTupleAndKeywords() may or may not set it to some object
pointer if it was passed in.

Since Zope is only going to be supporting 2.1 and newer releases, this
is not necessary. However, it may be worth auditing the code to see if
the O format is being used, and if so, do Py_INCREF()s need to be added.
I suspect some might, since with Python 1.5.2, having them there is a
memory leak.

I'm going to bed now...
=================================================================
-- 
Robin Becker



More information about the Python-list mailing list