From ideasman42 at gmail.com  Fri May  1 04:29:15 2009
From: ideasman42 at gmail.com (Campbell Barton)
Date: Thu, 30 Apr 2009 19:29:15 -0700
Subject: [capi-sig] An unexpected visit_decref assertion violation (code
	works with 2.4, not with 2.5)
In-Reply-To: <49F9FC6A.1040803@ilm.com>
References: <49F9FC6A.1040803@ilm.com>
Message-ID: <7c1ab96d0904301929k4a7ccfdbl296b806f8c17cc58@mail.gmail.com>

Hi Joe,
Start off by saying I don't know the answer to your problem, but will
try give some hints.
Unless anyone else has encountered this problem you could also mail
the python dev's about this too since they are more likely to know
about changes in pythons internals.
Though with obscure changes in python I have found your usually on your own :|
You might better pinpoint the cause of the problem by building python
from svn and finding the revision that broke it.

For doing any py/C api work I find it essential to build my own python
and keep the source in the build dir so I can step through python
internals.
the configure flag for debugging didnt work well for me either so I build wth
 ./configure --prefix=/opt/python/ --with-pydebug
 make CFLAGS="-O0 -g3 -fno-inline"; make install

Maybe this is obvious too but in gdb some useful commands when
stepping through pythons internals (where ob is a PyObject).
p *ob->ob_type
 ...lets you quickly see the type which can give you a hint as to whats going on
p PyObject_Print(ob, stderr, 0)

Should probably use gdb macro's but I have this function for quicker
printing, also means you dont have to check for NULL values before
printing (which will segfault).

void PyObSpit(char *name, PyObject *var) {
	fprintf(stderr, "<%s> : ", name);
	if (var==NULL) {
		fprintf(stderr, "<NIL>");
	}
	else {
		PyObject_Print(var, stderr, 0);
	}
	fprintf(stderr, "\n");
}


Interesting converting C++ to python, out of interest are you
inheriting the C++ class as a PyObject?
for Blender3D's game engine api I recently reworked some of the
python/C++ integration based on an API from 1997!
http://www.python.org/doc/PyCPP.html
We have our own ref-counting going on, so this mixed with pythons
quite badly - now the PyObjects are member of the class which I find
more flexible.

On Thu, Apr 30, 2009 at 12:30 PM, Joe Ardent <jardent at ilm.com> wrote:
> Hello,
>
> I have a Python module here that takes a bunch of C++ classes, and turns
> them into Python classes. ?When built against and loaded into Python 2.4,
> everything is fine. ?But when built against and loaded into Python 2.5, I
> get an assert violation from Python's gc module. ?Here is the code that is
> failing:
>
> ? ?klass = PyClass_New(bases, classDict, className);
> ? ?if (klass && methods) {
> ? ? ? ?/* add methods to class */
> ? ? ? ?for (def = methods; def->ml_name != NULL; def++) {
> ? ? ? ? ? ?printf( "IlmPyClass: %d, def = %s\n", __LINE__, def->ml_name);
>
> ? ? ? ? ? ?PyObject *func = IlmPyClass_NewFunction(def);
> ? ? ? ? ? ?if (!func) {
> ? ? ? ? ? ? ? ?Py_XDECREF(klass);
> ? ? ? ? ? ? ? ?return NULL;
> ? ? ? ? ? ?}
> ? ? ? ? ? ?printf( "We get here\n" );
> ? ? ? ? ? ?func = PyMethod_New(func, NULL, klass); //this line fails
> ? ? ? ? ? ?printf( "We don't get here\n" );
> ? ? ? ? ? ?# .......
> ? ? ? ?}
> ? ?}
>
> The output of 'python2.5 -c "import mymod"' is:
>
> """
> [.... snip a bunch of "we get here, we don't get here" etc. as things fail
> to fail...]
> IlmPyClass: 197, def = __init__
> We get here
> python2: Modules/gcmodule.c:276: visit_decref: Assertion `gc->gc.gc_refs !=
> 0' failed.
>
> Abort
> """
>
> The obvious things, such as Py_INCREFing klass or func, do not work.
>
> What's extra strange, in addition to this code working fine in an
> earlier python version, is that this code works fine for most of the
> classes that are instantiated.
>
> What I really want to do, though, is see what exactly is being decref'd.
> ?Does anyone have any tips for doing this? ?I've already tried using a debug
> build of Python; it doesn't seem to provide that type of ref tracing.
> ?Thanks in advance for any tips or insights.
>
>
> --
> Joe Ardent
>
> _______________________________________________
> capi-sig mailing list
> capi-sig at python.org
> http://mail.python.org/mailman/listinfo/capi-sig
>



-- 
- Campbell

From lanyjie at yahoo.com  Fri May 15 17:49:41 2009
From: lanyjie at yahoo.com (Yingjie Lan)
Date: Fri, 15 May 2009 08:49:41 -0700 (PDT)
Subject: [capi-sig] Syntax for parsing tuples into C arrays
Message-ID: <217510.87792.qm@web54204.mail.re2.yahoo.com>


Parsing tuples into C arrays is quite painful 
with current C API. Below is is an example
when you don't know the tuple size before hand:

....
//ref: http://docs.python.org/c-api/arg.html
PyObject* pyObj;
if(!PyArg_ParseTuple(args,"O",&pyObj) 
        || !pyTuple_Check(pyObj))
   return NULL;

tupleSize = PyTuple_Size(pyObj);
array = (double*) malloc(sizeof(double)*tupleSize);
for(i=0; i < tupleSize; i++ ) {
   tupleItem = PyTuple_GetItem(pyObj, i);
   if( PyFloat_Check(tupleItem) ) {
      array[i] = PyFloat_AsDouble(tupleItem);
   }else{
      printf("Error: tuple contains a non-float value");
      exit(1);
   }
}
....

I'd like to propose a syntax to facilitate that kind 
of tasks. There is already syntax for parsing strings:

{
const char * s; int i;
PyArg_ParseTyple(args, "s#", &s, &i);
}

Consider the string as a sequence of char's.
Then note that in python, a tuple of int's,
is also a sequence of int's. This 
perspective provides a natural way to 
extend the the use of "#" symbol. 
First, to denote a tuple as a sequence 
of a given type of objects, let's use
"(<type>:)". For example, to denote a tuple
as a sequence of integers: "(i:)". 
The ":" is chosen to denote possible 
repetition of the whole pattern inside
the parentheses. As for why choose ":", 
here is a bad story, but a good example:

Once upone a time there was a temple on
a hill, and there lived an old and a
young monk. One day the old monk told the
the young a story:
Once upone a time there was a temple on
a hill, and there lived an old and a
young monk. One day the old monk told the
the young a story:
...

OK, enough...but you got the idea :)

Now consider 3 different scenarios of parsing
tuples into arrays.

(1) Unkown size. Here are some examples:

{
//parsing a tuple and its size
int i, *t; 
PyArg_ParseTuple(args, "(i:)#", &t, &i);
}

{
//parsing a matrix
int i,j,**t;
PyArg_ParseTuple(args, "((i:):)##", &t, &i, &j);
for(a=0; a<i; a++){
   for(b=0; b<j; b++)
      printf("%d, ", t[a][b]);
   printf("\n");
}

}

Here is an informal explanation of "((i:):)##":
the '#' can be seen as a peeling operator, which
peels away the outer most ':'-parentheses pair "(.:)" 
that denote a "free sequence" (i.e. sequence of 
unknown size) with some pattern ".". 

For types like 's' or its similar types, pretend 
as if there is an invisible pair of parentheses 
around it. For example, "(s:)##" matches a tuple
of strings of some fixed length. Here the first
'#' peels away the parentheses, leaving only "s#",
which is equivalent to "(<char>:)#" in concept, 
where "<char>" denote the character type. 
As the second '#' is outside the tuple,
it is shared by all strings inside, so they
must have the same length.

If there is nothing to peel for a '#', it is 
considered a dangling '#', an error.

Now consider some more interesting examples:
{
int i, *t, *v; 
PyArg_ParseTuple(args, "(ii:)#", &t, &v, &i);
//args=(1,2,3,4,5,6)
//t={1,3,5}
//v={2,4,6}
//i=3
}

In that example, a pattern is provided inside the sequence "(ii:)",
which can repeat itself multiple times. The repeated times is 
extracted by the '#' symbol. Below is another example:

{
int i,j, *t, **v; 
PyArg_ParseTuple(args, "(i(i:):)##", &t, &v, &i, &j);
//args=(1,(3,),5,(7,))
//t={1,5}
//v={{3},{7}}
//i=2, j=1
}

And another one:

{
int i,j, **t, **v; 
PyArg_ParseTuple(args, "((i:)(i:):)##", &t, &v, &i, &j);
//args=((1,),(3,),(5,),(7,))
//t={{1},{5}}
//v={{3},{7}}
//i=2, j=1
}

Note the second '#' is applied to both the free sequences inside the
outer tuple. This indicates that all the inside tuples must be of 
the same length. What if they have their own common lengths? We
provide a separate matching '#' for each inside tuple:

{
int i,j,k, **t, **v; 
PyArg_ParseTuple(args, "((i:)(i:):)#(##)", &t, &v, &i, &j,&k);
//args=((1,),(3,4),(5,),(7,8))
//t={{1},{5}}
//v={{3,4},{7,8}}
//i=2, j=1, k=2
}

The syntax can also deal with more complicated situations, 
such as an irregular 2-dimensional array, where each row
has a different row length:

{
int i, *j, **t;
PyArg_ParseTuple(args, "((i:)#:)#", &t, &j, &i);
for(a=0; a<i; a++){
   for(b=0; b<j[i]; b++)
      printf("%d, ", t[a][b]);
   printf("\n");
}
//args=((1,2), (1,2,3))
//t={{1,2},{1,2,3}}
//i=2, j={2,3}
}

Here is a mixed example: three dimensional 
array 't' has fixed length on the third 
dimention 'k', but has variable length on the 
second dimension 'j':

{
int i, *j, k, ***t;
PyArg_ParseTuple(args, "(((i:):)#:)##", &t, &j, &i, &k);
//args=(((1,2),(1,2)),((1,2),))
//t={{{1,2},{1,2}},{{1,2}}}
//i=2, k=2
//j={2,1}
}

This syntax can be used for parsing a tuple of strings:

{
int i, *j; 
const char ** s;
PyArg_ParseTuple(args, "(s#:)#", &s, &j, &i);
//args=("ab", "cde")
//s={"ab", "cde"}
//j={2, 3}
//i=2
}

(2) If you know the size of a tuple before hand, 
use this syntax: "(<type>integer)", meaning there
are exactly 'integer' number of <type> objects in
the tuple.

{
int *t;
//
PyArg_ParseTuple(args, "(i5)", &t);
//args = (1,2,3,4,5)
//t = {1,2,3,4,5}
}

A complicated case:
{
int **t;
PyArg_ParseTuple(args, "((i5)2)", &t);
//args=((1,2,3,4,5),(5,6,7,8,9))
//t={{1,2,3,4,5},{5,6,7,8,9}}
}

Another complicated case:

{
int **t, **v, i;
PyArg_ParseTuple(args, "(i3i2:)#", &t,&v,&i);
//args=(1,2,3,4,5,6,7,8,9,10)
//t={{1,2,3},{6,7,8}}
//v={{4,5},{9,10}}
//i=2
}

One more thing: the difference between 
"(ii)" and "(i2)" is that the former
yields two single integers, while the 
latter gives an array of length two.

(3) Concatenating arrays while parsing. 
Sometimes you would like to have a matrix 
parsed into a single array. Let's use a "(+" 
to start a tuple, such that the arrays 
parsed from the elements in that tuple 
should be concatenated into a single array.

{
int i,j, *t;
PyArg_ParseTuple(args, "(+(i))##", &t, &i, &j);
//args=((1,2,3),(4,5,6))
//t={1,2,3,4,5,6}
//i=2, j=3
}

Another example:
{
int i, *t;
PyArg_ParseTuple(args, "(+(i2))#", &t, &i);
}

Still another example, with strings:
{
int i, *j;
const char * t; //string
PyArg_ParseTuple(args, "(+(+s#2))#", &t, &j, &i);
//args=(("ab","c"),("d","efg"))
//t="abcdefg"
//i=2
////notice the size arrays are contatenated similarly
//j={2,1,1,3} 

const char ** sa;
PyArg_ParseTuple(args, "(+(s2))", &sa);
//sa = {"ab","c","d","efg"}

PyArg_ParseTuple(args, "((+s2))", &sa);
//sa = { "abc", "defg" }
}

And finally, an example with multi-item pattern:

{
int *t, *v, i;
PyArg_ParseTuple(args, "(+i1i2:)#", &t, &v, &i);
//args=(1,2,3,4,5,6)
//t={1,4}
//v={2,3,5,6}
//i=2
}

Epilogue:

This syntax can be applied to lists: 
just use "[.:]" instead of "(.:)".
It can also be used for building tuples
or lists out of arrays. 

That's it. Comments?

-Yingjie



      

From swapnil.st at gmail.com  Sun May 24 10:50:12 2009
From: swapnil.st at gmail.com (swapnil talekar)
Date: Sun, 24 May 2009 01:50:12 -0700 (PDT)
Subject: [capi-sig] Invitation to connect on LinkedIn
Message-ID: <54703600.1042215.1243155012870.JavaMail.app@ech3-cdn12.prod>

LinkedIn
------------



   
I'd like to add you to my professional network on LinkedIn.

- swapnil

Learn more:
https://www.linkedin.com/e/isd/594582295/TffXp9WB/

------------------------------------------

What is LinkedIn and why should you join?
http://learn.linkedin.com/what-is-linkedin

 
------
(c) 2009, LinkedIn Corporation


From ideasman42 at gmail.com  Fri May 29 11:41:40 2009
From: ideasman42 at gmail.com (Campbell Barton)
Date: Fri, 29 May 2009 02:41:40 -0700
Subject: [capi-sig] Garbage output from warnings with embedded python
In-Reply-To: <7c1ab96d0902221745ye64e97brebf0dea6c433c01a@mail.gmail.com>
References: <7c1ab96d0902191924p1811021cie9c24f6d27457879@mail.gmail.com>
	<7c1ab96d0902221745ye64e97brebf0dea6c433c01a@mail.gmail.com>
Message-ID: <7c1ab96d0905290241y2041759eg93a032ed5b80f73c@mail.gmail.com>

had another look into it and figured out whats going on, in short for
warnings python wants there to be a file on disk for the script and in
our case there is None, its not getting __file__ from the global
namespace so it gets argv[0] which is the blender3d binary in our
case.

The only fix I found so far was to set the __file__ variable in the
namespace before running the script.

commit log.
http://lists.blender.org/pipermail/bf-blender-cvs/2009-May/019681.html

On Sun, Feb 22, 2009 at 6:45 PM, Campbell Barton <ideasman42 at gmail.com> wrote:
> Still not sure why this happens.
> Did a test with a small C/Python file that just runs external scripts,
> with compiling AND running as a string I don't print gibberish.
>
> So for now I added this to our own warning function...
>
> ? ? ? ? ? ? ? // import sys; print '\t%s:%d' %
> (sys._getframe(0).f_code.co_filename, sys._getframe(0).f_lineno)
>
> ? ? ? ? ? ? ? PyObject *getframe, *frame;
> ? ? ? ? ? ? ? PyObject *f_lineno, *f_code, *co_filename;
>
> ? ? ? ? ? ? ? getframe = PySys_GetObject("_getframe"); // borrowed
> ? ? ? ? ? ? ? if (getframe) {
> ? ? ? ? ? ? ? ? ? ? ? frame = PyObject_CallObject(getframe, NULL);
> ? ? ? ? ? ? ? ? ? ? ? if (frame) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? f_lineno= PyObject_GetAttrString(frame,
> "f_lineno");
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? f_code= PyObject_GetAttrString(frame, "f_code");
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if (f_lineno && f_code) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? co_filename=
> PyObject_GetAttrString(f_code, "co_filename");
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if (co_filename) {
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? printf("\t%s:%d\n",
> PyString_AsString(co_filename),(int)PyInt_AsLong(f_lineno));
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_DECREF(f_lineno);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_DECREF(f_code);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_DECREF(co_filename);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_DECREF(frame);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return;
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? }
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? }
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_XDECREF(f_lineno);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_XDECREF(f_code);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Py_DECREF(frame);
> ? ? ? ? ? ? ? ? ? ? ? }
>
> ? ? ? ? ? ? ? }
> ? ? ? ? ? ? ? PyErr_Clear();
> ? ? ? ? ? ? ? printf("\tERROR - Could not access
> sys._getframe(0).f_lineno or sys._getframe().f_code.co_filename\n");
>



-- 
- Campbell