[pypy-dev] Yet another trace tool!

Richard Emslie rxe at ukshells.co.uk
Fri Aug 27 13:27:34 CEST 2004

Hi Armin,

Sorry I was meaning to get back to your last mail about c++ pointers.  Was 
going to flesh out gencpp.py a bit this weekend.

On Fri, 27 Aug 2004, Armin Rigo wrote:

> Hello Richard,
> On Tue, Aug 24, 2004 at 12:12:51AM +0100, Richard Emslie wrote:
>> To run go to translator/tool and run flowtrace.py ! There is a debug flag
>> to FlowTracer which will give you some idea what is actully being traced!
> It uses the earthenware package, about which I can't seem to find more info
> with Google...

Oooops.  They can be commented out.  It is an opensource package from my 
company (I'm still there).  It hasn't got full clearance to be released 
and I didn't mean to include it in the code :-(.  I'll fix it tonight. 
Also importing pygame is not happy since there is a local path in working 
directory called pygame.

Anyways it is a throw away hack bit of code.  I wasn't sure I should check 
it in, but incase the tracing is useful, and it is easy to remove from 
svn, I did. I am just playing, trying to get an understanding for the 
flow/annotation code.  I was hoping to get it to a stage where I could 
point it to an imported module and it could convert it (and then maybe try 
intergrate with traceinteractive - although I've no idea how to convert 
app_xxx() space functions - I started hacking the FlowObjSpace <bad idea>, 
but I hoping you'd share you thoughts on that one???? :-))

>> Well - back to gencpp.py then. :-)
> I am still struggling to find a reasonably clean way to have genc.py emit
> typed C code.  I was wondering about using C++'s overloaded functions:
> gencpp.py could write an operation as a call like z=op_add(x,y), which the C++
> compiler would resolve into a call to one of the predefined overloaded
> implementations of op_add().

Yes - that was the plan - although I wasn't too sure how it would pan out 
(and I was going to mention it when I wrote to your last mail - but I 
assumed you'd already thought of it :-)).

The first step was to remove the macros from gencpp.h into python and 
generate a load of overloaded functions.  I don't understand the 
annotation code so I haven't much idea what this involves yet.

Also was thinking we could create a number of overloaded functors and 
create a table of them.  We could then write the equivalent of 
flowtrace.py as a c++ entension and attempt to use c++'s dynamic dispatch 
(I'm not confident that c++ would up to the job, and it is likely to be 
pretty slow - but fun to try nevertheless.)

  The compiler would also insert automatic
> conversions with some C++ class trickery.  I'm a bit hesitant to go into that
> direction for two reasons: we could get C++ compile errors in the presence of
> ambiguities (i.e. when two overloaded functions could equally well be used for
> a call);

Previous experience has shown that if the types are c++ classes/structs 
then it does the job.  But I do agree with your point.

> and gcc generates suboptimal code when we use structs or classes with
> just one or two fields, like this one:
> class PyObjectRef {
>    PyObject* p;
>  public:
>    explicit PyObjectRef(PyObject* np) { p = np; } // constructor, consumes
>                                                   // reference
>    ~PyObjectRef() { Py_DECREF(p); }      // destructor
>    PyObjectRef(int n) { p = PyInt_FromLong(n); }   // conversion constructor
> };
> int op_add(int a, int b) { return a+b; }
> PyObjectRef op_add(PyObjectRef &a, PyObjectRef &b) {
>    return PyObjectRef(PyNumber_Add(a.p, b.p));
> }

> The conversion constructor from int to PyObjectRef is applied automatically by
> the compiler, so it is quite nice, but you get a program that handles pointers
> to PyObjectRefs (which are pointers to PyObject*, i.e. PyObject**) instead of
> just PyObject*.  There might be a way to fix that, but that starts to be
> hackish.

I've attached what I have so far with the reference counted pointer (I am 
happy with phrase hackish here :-)) .  It was hacked in the early hours on 
Sunday night - and I haven't used it in anger yet - so it might be riddled 
with bugs / flaws.

In terms of suboptimal code - well it is already suboptimal because it is 
c/c++ on top of cpython. ;-)

Another thought is that if can do a full translation of standard object 
space and then start removing some of the overloaded functions which have 
reference to CPython.  Then the c++ compiler can tell us if they have been 
annotated out or not.  Incrementally we could remove our reliance on 
cpython. (There is fair chance that doesn't make any sense)

> Maybe Boost.Python already solves all these problems?

I've no idea!

> Well, is it a good idea to -- again -- target a high-level language and then
> have to work around some of its concepts when they are not fully suited to our
> low-level needs...?

The really nice thing about the pypy translation is that these are 
complete end point in that a genxxx.py has no dependents at all.  Even 
nicer is that we can do several independent translations with various 
levels of not so smart ideas (like everything I do!) - and then combine 
the better ones at the end.

Well chances are I'll be hacking some more this weekend - and be around on 
irc.  We could try to combine forces if you are interested?

-------------- next part --------------

/***  Generic C header section                            ***/

#include <Python.h>

/* Ref counted C pointer */

class CPythonProxyObject {

  CPythonProxyObject(PyObject *instance, bool borrow = false) :
    instance(instance) {
  ~CPythonProxyObject() {
  bool decrement() {
    if (refCount == 0) {
      if (!borrowed and instance != NULL) {
      return true;
    return false;
  void increment() {
  // Logically private
  // The actual reference counter
  mutable int refCount;
  bool borrowed;

  // The actual data
  PyObject *instance;


class Handle {

  explicit Handle() :
    ptProxyObj (getNullProxyObject()) {
  // Main constructor
  explicit Handle (PyObject *instance) {
    if (instance == NULL) {
      ptProxyObj = getNullProxyObject();
    } else {
      ptProxyObj = new CPythonProxyObject(instance);
  // Copy constuctor (increments)
  Handle (const Handle &ref) :
    ptProxyObj (ref.ptProxyObj) {
  // Assignment constuctor (increments unless itself)
  Handle & operator= (const Handle &ref) {
    if (this->ptProxyObj != ref.ptProxyObj) {
      ptProxyObj = ref.ptProxyObj;
    return *this;
  ~Handle() {
  // This saves headaches 
  Handle (CPythonProxyObject *aPtrProxyObj) :
    ptProxyObj (aPtrProxyObj) {

  PyObject* get() {
    return ptProxyObj->instance;

  bool isNull() const {
    return ptProxyObj == getNullProxyObject();

  // Will delete memory.  Note this is required if we want to add extra elements to proxy object.
  void decrement() {
    if (ptProxyObj->decrement()) {
      delete ptProxyObj;
  static CPythonProxyObject *getNullProxyObject() {
    static CPythonProxyObject *nullObject = NULL;
    if (nullObject == NULL) {
      nullObject = new CPythonProxyObject ((PyObject *) NULL);
    return nullObject;
    CPythonProxyObject *ptProxyObj;


class PyException {

/* Compare Ops */

bool handleCmp(Handle x, Handle y) {
  return PyObject_RichCompareBool(x.get(), y.get() ,Py_EQ);

void opNAME(Handle x, Handle y, Handle r) { \
  PyObject *xx; \
  if (!(xx=PyObject_RichCompare(x.get(), y.get(), Py_NAME))) \
    throw PyException(); \
  r = Handle(xx); \


/* Use c++ overloading  */
void opGE(int x, 
	  int y,
	  int &r) {

  if (x > y) {
    r = 1;
  } else {
    r = 0;


void opIS_TRUE(Handle x, Handle r) {

  PyObject *xx;
  switch (PyObject_IsTrue(x.get())) {
    case  0: 
      xx = Py_False; 
    case -1: 
      throw PyException();  
      xx = Py_True; 
  r = Handle(xx);

/* Unary Ops */

void NAME(Handle x, Handle r) { \
  PyObject *xx; \
  if (!(xx=CFUNCTION(x.get()))) \
	  throw PyException(); \
  r = Handle(xx); \


/* Binary Ops */

void NAME(Handle x, Handle y, Handle r) { \
  PyObject *xx; \
  if (!(xx=CFUNCTION(x.get(), y.get()))) \
	  throw PyException(); \
  r = Handle(xx); \

CREATE_BINARY_OP(opMOD, PyNumber_Remainder);

/* Macros left over */

#define CASE(x,y) if (handleCmp(x,y))
#define ELSE(x,y) assert(handleCmp(x,y));

 /***  The rest is produced by genc.py                     ***/
-------------- next part --------------
Handle is_perfect_number__0(Handle v134)
        Handle v199;
        Handle v200;
        Handle v201;
        Handle v202;
        Handle v203;
        Handle v210;
        Handle v212;
        Handle v213;
        Handle v135;
        Handle v245;
        Handle v246;
        Handle v247;
        Handle v248;
        Handle v249;
        Handle v216;
        Handle v217;
        Handle v218;
        Handle v195;
        Handle v196;
        Handle v197;
        Handle v198;
        Handle v258;
        Handle v259;
        Handle v260;
        Handle v236;

        // Input args are v134
        // Doing basic block operations

        // Doing exits
        // Doing input args to block1
        v199 = v134;
        v200 = kint_1;
        v201 = kint_0;
        goto block1;
        // Input args are v199, v200, v201
        // Doing basic block operations
        opLT(v200, v199, v202);
        opIS_TRUE(v202, v203);

        // Doing exits
        CASE(v203, kint_0) {
                // Doing input args to block2
                v210 = v199;
                v212 = v201;
                goto block2;
        ELSE(v203, kint_1) {
                // Doing input args to block4
                v245 = v199;
                v246 = v200;
                v247 = v201;
                v248 = v202;
                v249 = v203;
                goto block4;
        // Input args are v210, v212
        // Doing basic block operations
        opEQ(v210, v212, v213);

        // Doing exits
        // Doing input args to block3
        v135 = v213;
        goto block3;
        // Input args are v135
        // Doing basic block operations

        // Doing exits
        return v135;
        // Input args are v245, v246, v247, v248, v249
        // Doing basic block operations
        opMOD(v245, v246, v216);
        opEQ(v216, kint_0, v217);
        opIS_TRUE(v217, v218);

        // Doing exits
        CASE(v218, kint_0) {
                // Doing input args to block5
                v195 = v245;
                v196 = v246;
                v197 = v247;
                goto block5;
        ELSE(v218, kint_1) {
                // Doing input args to block6
                v258 = v245;
                v259 = v246;
                v260 = v247;
                goto block6;
        // Input args are v195, v196, v197
        // Doing basic block operations
        opINPLACE_ADD(v196, kint_1, v198);

        // Doing exits
        // Doing input args to block1
        v199 = v195;
        v200 = v198;
        v201 = v197;
        goto block1;
        // Input args are v258, v259, v260
        // Doing basic block operations
        opINPLACE_ADD(v260, v259, v236);

        // Doing exits
        // Doing input args to block5
        v195 = v258;
        v196 = v259;
        v197 = v236;
        goto block5;

More information about the Pypy-dev mailing list