Easy way to Return Value from 1 line of Embedded Python?
In a C program I need to unpickle a Python pickle which is generated by someone's Python program. A Python newbie, I was able to do this, but only by writing a little python script file and loading it as a module. But reading from a script file is fragile, has lots of possible errors to handle as you can see below, and is silly since there's only one line of Python.
I'd therefore like to create the Python module from a string constant, as with PyRun_SimpleString(), instead of reading a file. But I can't find a function like that which returns a value. Or I just don't know how.
Certainly there must be a simpler way to do this...
Thank you,
Jerry Krinock
***** somePython.py *****************************
#!/usr/bin/env python
from pickle import loads
def unpickle(x): return loads(x)
***** My C "Proof of Concept" Function **********
#include <Python/Python.h> // Running on Mac OS X
char* PythonUnpickle(char* pickle) { PyObject *pName, *pModule, *pFunc; PyObject *pArgs, *pValue; char* unpickledString = NULL ;
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\"/Users/jk/Documents/Programming/Python/\")");
pName = PyString_FromString("somePython");
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL) {
pFunc = PyObject_GetAttrString(pModule, "unpickle");
/* pFunc is a new reference */
if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(1);
// Create the argument to unpickle()
pValue = PyString_FromString(pickle);
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
return NULL ;
}
// Set the argument into the pArgs tuple
PyTuple_SetItem(pArgs, 0, pValue);
// Run the python unpickle() function, get result pValue
pValue = PyObject_CallObject(pFunc, pArgs) ;
Py_DECREF(pArgs) ;
// Convert the Python string pValue to a C string
if (pValue != NULL) {
unpickledString = PyString_AsString(pValue) ;
Py_DECREF(pValue);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return NULL ;
}
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find 'unpickle' function\n") ;
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load python script file\n");
}
Py_Finalize() ;
return unpickledString ;}
Jerry Krinock wrote:
In a C program I need to unpickle a Python pickle which is generated by someone's Python program. A Python newbie, I was able to do this, but only by writing a little python script file and loading it as a module. But reading from a script file is fragile, has lots of possible errors to handle as you can see below, and is silly since there's only one line of Python.
I'd therefore like to create the Python module from a string constant, as with PyRun_SimpleString(), instead of reading a file. But I can't find a function like that which returns a value. Or I just don't know how.
Certainly there must be a simpler way to do this...
You can compile the code into code object using Py_CompileString() and then pass this to PyEval_EvalCode() for execution. This will evaluate the code and return the resulting Python object.
In your case, it's probably easier to just load the pickle module from C and call the loads() function directly rather than going through an extra layer of Python code.
Thank you,
Jerry Krinock
***** somePython.py *****************************
#!/usr/bin/env python
from pickle import loads
def unpickle(x): return loads(x)
***** My C "Proof of Concept" Function **********
#include <Python/Python.h> // Running on Mac OS X
char* PythonUnpickle(char* pickle) { PyObject *pName, *pModule, *pFunc; PyObject *pArgs, *pValue; char* unpickledString = NULL ;
Py_Initialize(); PyRun_SimpleString("import sys"); PyRun_SimpleString("sys.path.append(\"/Users/jk/Documents/Programming/Python/\")"); pName = PyString_FromString("somePython"); pModule = PyImport_Import(pName); Py_DECREF(pName); if (pModule != NULL) { pFunc = PyObject_GetAttrString(pModule, "unpickle"); /* pFunc is a new reference */ if (pFunc && PyCallable_Check(pFunc)) { pArgs = PyTuple_New(1); // Create the argument to unpickle() pValue = PyString_FromString(pickle); if (!pValue) { Py_DECREF(pArgs); Py_DECREF(pModule); fprintf(stderr, "Cannot convert argument\n"); return NULL ; } // Set the argument into the pArgs tuple PyTuple_SetItem(pArgs, 0, pValue); // Run the python unpickle() function, get result pValue pValue = PyObject_CallObject(pFunc, pArgs) ; Py_DECREF(pArgs) ; // Convert the Python string pValue to a C string if (pValue != NULL) { unpickledString = PyString_AsString(pValue) ; Py_DECREF(pValue); } else { Py_DECREF(pFunc); Py_DECREF(pModule); PyErr_Print(); fprintf(stderr,"Call failed\n"); return NULL ; } } else { if (PyErr_Occurred()) PyErr_Print(); fprintf(stderr, "Cannot find 'unpickle' function\n") ; } Py_XDECREF(pFunc); Py_DECREF(pModule); } else { PyErr_Print(); fprintf(stderr, "Failed to load python script file\n"); } Py_Finalize() ; return unpickledString ;}
The string object you get from the Python function will only be allocated while the interpreter is initialized.
In your example the unpickledString will point to unallocated memory when the function returns. It is not even guaranteed to still have the correct string data.
To correct this, you will have to get the pointer to the string data, copy it to a buffer you allocate in your app and only then finalize the interpreter.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, May 28 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-07-19: EuroPython 2010, Birmingham, UK 51 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 2010 May 28, at 01:54, M.-A. Lemburg wrote:
You can compile the code into code object using Py_CompileString() and then pass this to PyEval_EvalCode() for execution. This will evaluate the code and return the resulting Python object.
Thank you, Mark-Andre. I had trouble getting that to work. Let's move on to your preferred answer.
In your case, it's probably easier to just load the pickle module from C and call the loads() function directly rather than going through an extra layer of Python code.
I'm not sure if you meant to use PyFunction_New() or PyRun_String(). I got it to work using the latter. See code below.
The string object you get from the Python function will only be allocated while the interpreter is initialized. ... To correct this, you will have to get the pointer to the string data, copy it to a buffer you allocate in your app and only then finalize the interpreter.
Thanks. I've corrected that now by creating and returning a Cocoa object.
So, here is the working code using PyRun_String(). The NSString is the Cocoa object used in Mac OS X or iPhone OS. Otherwise, use your favorite string object.
#import <Cocoa/Cocoa.h> #include <Python/Python.h>
NSString* PythonStringCodeUnpickle(char* pickle) { NSString* answer = nil ; char* myErrorDesc = NULL ;
Py_Initialize() ;
// Create Python Namespace (dictionary of variables)
PyObject* pythonStringArg = PyString_FromString(pickle);
if (!pythonStringArg) {
myErrorDesc = "Cannot convert string arg to Python\n" ;
goto end ;
}
PyObject* pythonVarsDic = PyDict_New();
PyDict_SetItemString(
pythonVarsDic,
"__builtins__",
PyEval_GetBuiltins());
PyDict_SetItemString(
pythonVarsDic,
"x",
pythonStringArg) ;
// Create "hard" source code string to unpickle
// (For some strange reason, they call it loads()
// The "s" in "loads" stands for "string".)
char* pythonSource =
"from pickle import loads\n\n"
"y = loads(x)\n" ;
// Run the python source code
PyRun_String(pythonSource,
Py_file_input,
pythonVarsDic,
pythonVarsDic) ;
PyObject* unpickledPythonString = PyDict_GetItemString(pythonVarsDic, "y") ;
if (!unpickledPythonString) {
myErrorDesc = "Unpickling returned NULL\n" ;
goto end ;
}
// Convert the unpickled Python string to a C string
char* unpickledString = PyString_AsString(unpickledPythonString) ;
Py_DECREF(unpickledPythonString);
if (!unpickledString) {
myErrorDesc = "Failed converting unpickled string to C string\n" ;
goto end ;
}
// Convert the C string into a string object
answer = [NSString stringWithUTF8String:unpickledString] ;end: if (myErrorDesc) { printf("%s\n", myErrorDesc) ; }
if (PyErr_Occurred()) {
PyErr_Print();
}
Py_Finalize() ;
return answer ;}
int main(int argc, char *argv[]) { NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init] ;
// The following string is an actual Python pickle.
// Of course, one would never hard code such a thing in a real program.
// This is just a test :)
char* s = "V/Users/jk/Dropbox2/Dropbox\np1\n.\n" ;
printf("Unpickling using Python code from string:\n") ;
printf(" Unpickled result: %s\n\n",
[PythonStringCodeUnpickle(s) UTF8String]) ;
// The correct answer for Unpickled result is:
// "/Users/jk/Dropbox2/Dropbox"
[pool release] ;
return 0 ;}
Jerry Krinock wrote:
On 2010 May 28, at 01:54, M.-A. Lemburg wrote:
You can compile the code into code object using Py_CompileString() and then pass this to PyEval_EvalCode() for execution. This will evaluate the code and return the resulting Python object.
Thank you, Mark-Andre. I had trouble getting that to work. Let's move on to your preferred answer.
In your case, it's probably easier to just load the pickle module from C and call the loads() function directly rather than going through an extra layer of Python code.
I'm not sure if you meant to use PyFunction_New() or PyRun_String(). I got it to work using the latter. See code below.
I was thinking of PyImport_ImportModule() to import the pickle module, PyModule_GetDict() and PyDict_GetItemString() to get the function object and finally PyEval_CallFunction() to call the function.
You can also use a short-cut directly from the module object to the function call by using PyEval_CallMethod(pickle_module, "loads", ...) - top-level symbols in a module are available as attributes of the module object, e.g.
pickle_module = PyImport_ImportModule("pickle"); result = PyEval_CallMethod(pickle_module, "loads", "s#", ...);
This avoids having to compile any Python code just to unpickle a string.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, May 30 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-07-19: EuroPython 2010, Birmingham, UK 49 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Jerry Krinock, 27.05.2010 16:26:
In a C program I need to unpickle a Python pickle which is generated by someone's Python program. A Python newbie, I was able to do this, but only by writing a little python script file and loading it as a module. But reading from a script file is fragile, has lots of possible errors to handle as you can see below, and is silly since there's only one line of Python.
I'd therefore like to create the Python module from a string constant, as with PyRun_SimpleString(), instead of reading a file. But I can't find a function like that which returns a value. Or I just don't know how.
Certainly there must be a simpler way to do this...
Thank you,
Jerry Krinock
***** somePython.py *****************************
#!/usr/bin/env python
from pickle import loads
def unpickle(x): return loads(x)
Try compiling this with Cython, the generated C code will show you what you need to do.
Stefan
participants (3)
-
Jerry Krinock -
M.-A. Lemburg -
Stefan Behnel