From kurt at quansoft.com  Fri Jun  5 06:50:03 2009
From: kurt at quansoft.com (Kurt Sutter)
Date: Fri, 5 Jun 2009 06:50:03 +0200
Subject: [capi-sig] embedding vs. multiprocessing
Message-ID: <0564E1FD-125F-4E3C-A8B2-F0C4F5F39D50@quansoft.com>

Dear list

We have a GUI-based application written basically in C. The  
application allows the user to embed Python scripts, e.g. for  
scripting some functionality of the application.

Some of our users are interested in running multiprocessing Python  
code. We have found that this does not work, bringing up various types  
of errors, depending on how the script invokes multiprocessing.

I have tried to isolate the issue, and found that the following simple  
C-code fails:

	Py_Initialize();

	const char* script =
		"from multiprocessing import Process\n"
		"import os, time\n"
		"def sleeper(name, seconds):\n"
		"    time.sleep(seconds)\n"	
		"p = Process(target=sleeper, args=('bob', 1))\n"
		"print 'starting child process'\n"
		"p.start()\n"
		"print 'joining'\n"
		"p.join()\n"
		"print 'done'\n";
	PyRun_SimpleString(script);

with the following output:

starting child process
joining
Traceback (most recent call last):
   File "<string>", line 9, in <module>
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/multiprocessing/process.py", line 119, in join
     res = self._popen.wait(timeout)
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/multiprocessing/forking.py", line 117, in wait
     return self.poll(0)
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/multiprocessing/forking.py", line 106, in poll
     pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call

The same Python script, when saved as a .py file and invoked from the  
console, works fine.

System information: MacOS 10.5.6, Pyton 2.6.2.

Thanks for any advice in advance.

Kurt Sutter
QuantumSoft



From philip at semanchuk.com  Sat Jun 13 23:36:56 2009
From: philip at semanchuk.com (Philip Semanchuk)
Date: Sat, 13 Jun 2009 17:36:56 -0400
Subject: [capi-sig] KeyboardInterrupt eats my error and then won't be caught
Message-ID: <0B8CF82D-69F9-4F15-AB96-BC1B4BA4A10A@semanchuk.com>

Hi all,
I need help understanding how Python deals with Ctrl-C.

A user has reported a bug in my posix_ipc module. When a Python app is  
waiting to acquire an IPC semaphore, a KeyboardInterrupt seems to (a)  
overwrite my error and (b) refuse to be caught by try/except. Here's a  
sample program that demonstrates the problem when run from the command  
line:

# -----------------------------------
import posix_ipc

sem = posix_ipc.Semaphore(None, posix_ipc.O_CREX)

try:
     sem.acquire()
except:
     print "********* I caught it!"

sem.close()
sem.unlink()
# -----------------------------------

I expected that code to raise a posix_ipc.Error with the text, "The  
wait was interrupted by a signal". Instead a KeyboardInterrupt error  
is propagated up to the interpreter and the process is killed as if  
the try/except wasn't even there.

By adding some debug messages to my C code I can see that sem_wait()  
behaves as advertised: it returns -1 and errno is set to EINTR. In  
response to that, my C function calls PyErr_SetString(pBaseException,  
"The wait was interrupted by a signal") and returns NULL. Somewhere  
along the line, the error I set is getting lost.

If I substitute my sysv_ipc module for posix_ipc (very similar to  
posix_ipc but using Sys V semaphores instead of POSIX), I get the same  
behavior.

I see this w/Python 2.5 under OS X and also w/Python 2.5 under Ubuntu  
8.0.4.

http://semanchuk.com/philip/posix_ipc/
http://semanchuk.com/philip/sysv_ipc/


Any suggestions would be appreciated.

Thanks
Philip





From skoranda at gravity.phys.uwm.edu  Sat Jun 20 06:08:21 2009
From: skoranda at gravity.phys.uwm.edu (Scott Koranda)
Date: Fri, 19 Jun 2009 23:08:21 -0500
Subject: [capi-sig] wrapping C library with asynchronous callbacks
Message-ID: <20090620040821.GA29189@gravity.phys.uwm.edu>

Hello,

I am wrapping a C library that uses an asynchronous
programming model. 

Operations in the C library are invoked by calling a function
and including as one of the arguments a callback function that
is to be called when the operation is completed. The function
call to start the operation returns immediately and the work
is done by a separate thread(s). When the operation is
complete the separate thread(s) calls the callback function
provided as input.

Below is a skeleton code sample that demonstrates how I am
attempting to wrap the C library.

I would be grateful if the experts could verify that I am
using  PyGILState_Ensure()/PyGILState_Release() correctly, or
if not please help me understand what I am doing wrong.

Also, is the call to PyEval_InitThreads() before the C library
creates any new threads necessary and sufficient?

Some Python code that I wrote that uses my wrapping module
(and a large number of other modules) is occasionally
experiencing deadlock. I am attempting to debug that and would
like to rule out errors in how I have wrapped the C library.

I appreciate any input you have.

Also, right now my need is to support using this wrapper
module with Python 2.4.x and Python 2.5.x.

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <ctype.h>

#include "mylibrary.h"

#include "Python.h"

// used to store pointers to the Python objects that should
// be used during a callback for completion of operation 01
typedef struct
{
    PyObject * pyfunction; // Python object for the Python function to call as callback
    PyObject * pyarg;      // Python object for the Python argument to pass in to the callback
} operation01_callback_bucket_t;

// callback for the completion of operation 01
static void operation01_callback(void * user_data) 
{
    PyObject * func;
    PyObject * arglist; 
    PyObject * result; 
    PyObject * arg;

    operation01_callback_bucket_t  * callbackBucket;

    // cast the user_data that the libraries are passing in to the
    // callback structure where we previously stored the Python function and
    // arguments to call
    callbackBucket = (operation01_callback_bucket_t *) user_data;
    
    // we need to obtain the Python GIL before this thread can manipulate any Python object
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    // pick off the function and argument pointers we want to pass back into Python
    func = callbackBucket -> pyfunction;
    arg = callbackBucket -> pyarg;

    // prepare the arg list to pass into the Python callback function
    arglist = Py_BuildValue("(O)", arg);

    // now call the Python callback function
    result = PyEval_CallObject(func, arglist);

    if (result == NULL) {
        // something went wrong so print to stderr
        PyErr_Print();
    }

    // take care of reference handling
    Py_DECREF(arglist);
    Py_XDECREF(result);

    // release the Python GIL from this thread
    PyGILState_Release(gstate);

    // free the space the callback bucket was holding
    free(callbackBucket);

    return;
}


// initialize operation 01
PyObject * operation01(PyObject *self, PyObject *args)
{
    PyObject * completeCallbackFunctionObj;
    PyObject * completeCallbackArgObj;

    operation01_callback_bucket_t * callbackBucket = NULL;

    int result;

    // get Python arguments
    if (!PyArg_ParseTuple(args, "OO", 
            &completeCallbackFunctionObj,
            &completeCallbackArgObj
            )){
        PyErr_SetString(PyExc_RuntimeError, "unable to parse arguments");
        return NULL;
    }
 
    // create a third party callback struct to hold the callback information
    callbackBucket = (operation01_callback_bucket_t *) malloc(sizeof(operation01_callback_bucket_t));
    callbackBucket -> pyfunction = completeCallbackFunctionObj;
    callbackBucket -> pyarg = completeCallbackArgObj;

    // since we are holding pointers to these objects we need to increase
    // the reference count for each
    Py_XINCREF(callbackBucket -> pyfunction);
    Py_XINCREF(callbackBucket -> pyarg);

    // kick off operation01

    Py_BEGIN_ALLOW_THREADS

    result = operation01(
                        operation01_complete_callback,
                        (void *) callbackBucket
                        );

    Py_END_ALLOW_THREADS

    if (result != 0){
        sprintf(msg, "rc = %d: unable to start operation01", result);
        PyErr_SetString(PyExc_RuntimeError, msg);
        return NULL;
    }
    
    // return None to indicate success
    Py_RETURN_NONE;

}

// method table mapping names to wrappers
static PyMethodDef mywrappermethods[] = {
    {"operation01", operation01, METH_VARARGS},
    {NULL, NULL}
};


// module initialization function
void initmywrapper(){
    PyObject * module;
    PyObject * moduleDict;

    // this is called before the C library creates any threads
    PyEval_InitThreads();

    // get handle to the module dictionary
    module = Py_InitModule("mywrapper", mywrappermethods);
    moduleDict = PyModule_GetDict(module);

}



From jesse_lehrman at animationlab.com  Mon Jun 22 08:52:35 2009
From: jesse_lehrman at animationlab.com (Jesse Lehrman)
Date: Mon, 22 Jun 2009 09:52:35 +0300
Subject: [capi-sig] Differences between 32-bit and 64-bit python25.lib
Message-ID: <1FF132D810C1AC409FD8A47BDC96E3DB01CFBB7C2D@alabex01>

I'm trying to compile SIP/PyQT 4.5 for Windows Python 64-bit. I'm running into an error when attempting to link the SIP/PyQT object files against python25.lib. The errors are "can't find reference..." to symbols that should be in python25.lib. I tried using the 32-bit version of python25.lib and it worked. This led me to look at python25.lib using dumpbin.exe. I'm using the install binaries from www.python.org.

It seems that most of the exports in the 32-bit version have a preceding underscore while in the 64-bit version they don't. For example:

python25.lib 32-bit:
                  _PyFloat_AsDouble
                  _PyFloat_AsReprString
                  _PyFloat_AsString
                  _PyFloat_Fini
                  _PyFloat_FromDouble
                  _PyFloat_FromString
                  _PyFloat_Type

python25.lib 64-bit:
                  PyFloat_AsDouble
                  PyFloat_AsReprString
                  PyFloat_AsString
                  PyFloat_Fini
                  PyFloat_FromDouble
                  PyFloat_FromString
                  PyFloat_Type

Does anybody have an idea of why this is done and how I can get around it?

Thank you,

Jesse


________________________________
Please consider the environment before printing this email.

From doriankrause at web.de  Mon Jun 22 19:43:28 2009
From: doriankrause at web.de (Dorian Krause)
Date: Mon, 22 Jun 2009 19:43:28 +0200
Subject: [capi-sig] Differences between 32-bit and 64-bit python25.lib
In-Reply-To: <1FF132D810C1AC409FD8A47BDC96E3DB01CFBB7C2D@alabex01>
References: <1FF132D810C1AC409FD8A47BDC96E3DB01CFBB7C2D@alabex01>
Message-ID: <4A3FC2C0.5000700@web.de>

Jesse Lehrman wrote:
> I'm trying to compile SIP/PyQT 4.5 for Windows Python 64-bit. I'm running into an error when attempting to link the SIP/PyQT object files against python25.lib. The errors are "can't find reference..." to symbols that should be in python25.lib. I tried using the 32-bit version of python25.lib and it worked. This led me to look at python25.lib using dumpbin.exe. I'm using the install binaries from www.python.org.
>
> It seems that most of the exports in the 32-bit version have a preceding underscore while in the 64-bit version they don't. For example:
>
> python25.lib 32-bit:
>                   _PyFloat_AsDouble
>                   _PyFloat_AsReprString
>                   _PyFloat_AsString
>                   _PyFloat_Fini
>                   _PyFloat_FromDouble
>                   _PyFloat_FromString
>                   _PyFloat_Type
>
> python25.lib 64-bit:
>                   PyFloat_AsDouble
>                   PyFloat_AsReprString
>                   PyFloat_AsString
>                   PyFloat_Fini
>                   PyFloat_FromDouble
>                   PyFloat_FromString
>                   PyFloat_Type
>
> Does anybody have an idea of why this is done and how I can get around it?
>
> Thank you,
>
> Jesse
>
>
> ________________________________
> Please consider the environment before printing this email.
> _______________________________________________
> capi-sig mailing list
> capi-sig at python.org
> http://mail.python.org/mailman/listinfo/capi-sig
>
>   
(forgot to reply to the mailinglist)

This might be related to the calling convention?! 32 bit compiled, 
_cdecl functions are get a prepended underscore, see 
http://support.microsoft.com/kb/100832. As this thread ( 
http://www.dotnetmonster.com/Uwe/Forum.aspx/vs/6817/Missing-Underscore-in-Name-From-64-bit-Compiler 
) there is only one 64 bit calling convention and therefor no underscore

Hope this is correct ;) ,
Dorian

From jesse_lehrman at animationlab.com  Tue Jun 23 09:46:47 2009
From: jesse_lehrman at animationlab.com (Jesse Lehrman)
Date: Tue, 23 Jun 2009 10:46:47 +0300
Subject: [capi-sig] Differences between 32-bit and 64-bit python25.lib
In-Reply-To: <4A3FC2C0.5000700@web.de>
References: <1FF132D810C1AC409FD8A47BDC96E3DB01CFBB7C2D@alabex01>
	<4A3FC2C0.5000700@web.de>
Message-ID: <1FF132D810C1AC409FD8A47BDC96E3DB01CFBB7DF9@alabex01>

Thanks - that's what I've started to figure out. That thread also supports my theory that I'll have to recompile Python using MiniGW.

Jesse

-----Original Message-----
From: capi-sig-bounces+jesse_lehrman=animationlab.com at python.org [mailto:capi-sig-bounces+jesse_lehrman=animationlab.com at python.org] On Behalf Of Dorian Krause
Sent: Monday, June 22, 2009 8:43 PM
Cc: capi-sig at python.org; cplusplus-sig at python.org
Subject: Re: [capi-sig] Differences between 32-bit and 64-bit python25.lib

Jesse Lehrman wrote:
> I'm trying to compile SIP/PyQT 4.5 for Windows Python 64-bit. I'm running into an error when attempting to link the SIP/PyQT object files against python25.lib. The errors are "can't find reference..." to symbols that should be in python25.lib. I tried using the 32-bit version of python25.lib and it worked. This led me to look at python25.lib using dumpbin.exe. I'm using the install binaries from www.python.org.
>
> It seems that most of the exports in the 32-bit version have a preceding underscore while in the 64-bit version they don't. For example:
>
> python25.lib 32-bit:
>                   _PyFloat_AsDouble
>                   _PyFloat_AsReprString
>                   _PyFloat_AsString
>                   _PyFloat_Fini
>                   _PyFloat_FromDouble
>                   _PyFloat_FromString
>                   _PyFloat_Type
>
> python25.lib 64-bit:
>                   PyFloat_AsDouble
>                   PyFloat_AsReprString
>                   PyFloat_AsString
>                   PyFloat_Fini
>                   PyFloat_FromDouble
>                   PyFloat_FromString
>                   PyFloat_Type
>
> Does anybody have an idea of why this is done and how I can get around it?
>
> Thank you,
>
> Jesse
>
>
> ________________________________
> Please consider the environment before printing this email.
> _______________________________________________
> capi-sig mailing list
> capi-sig at python.org
> http://mail.python.org/mailman/listinfo/capi-sig
>
>   
(forgot to reply to the mailinglist)

This might be related to the calling convention?! 32 bit compiled, 
_cdecl functions are get a prepended underscore, see 
http://support.microsoft.com/kb/100832. As this thread ( 
http://www.dotnetmonster.com/Uwe/Forum.aspx/vs/6817/Missing-Underscore-in-Name-From-64-bit-Compiler 
) there is only one 64 bit calling convention and therefor no underscore

Hope this is correct ;) ,
Dorian
_______________________________________________
capi-sig mailing list
capi-sig at python.org
http://mail.python.org/mailman/listinfo/capi-sig

That does seem to be the case. Any ideas of how to get around it