Passing opaque data to C++ callback from Python
I'm embedding Python (using boost::python) into an application plugin that uses callbacks. Essentially, I want to do something like: In Python (say test.py): def do_something(): ... register_callback(do_something) And on the C++ side I register the register_callback() function: void register_callback(boost::python::object& o) { // Access some persistent state information } BOOST_PYTHON_MODULE(foo) { boost::python::def("register_callback", register_callback); } void library_entry() { PyImport_AppendInittab("foo", initfoo); PyInitialize(); // Create some persistent state information boost::python::exec_file("test.py", ...); } The issue here is that I need to create the Python context and store things away in an opaque pointer I return to the application. And when Python calls back into the C++, I need to recover that opaque value. For example: Application -- (calls) --> my C++ library -- (runs) --> Python script -- (calls) --> my C++ library For that last step, I need to recover some opaque data. Further, I need that opaque data to be persistent. The calls into my C++ library are callbacks from the application. The issue I'm having is trying to figure out how to pass that state information to the C++ register_callback() function. I've tried something like: namespace bp = boost::python; class state_info_t { }; void register_callback(std::shared_ptr<state_info_t>& state, bp::object& o); { // Access some persistent state information } // Create some persistent state information std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>(); PyImport_AppendInittab("foo", initfoo); Py_Initialize(); std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo"))); main_namespace["foo"] = *foo_module; bp::scope foo_scope(*foo_module); // Both of these fail with _a lot_ of errors, most related to "no matching function call to 'get_signature' bp::def("register_callback", [&](bp::object& o) { register_callback(state, o); }, bp::arg("func")); bp::def("register_callback", std::bind(register_callback, state, std::placeholders::_1), bp::arg("func")); The other thought I had was to store the persistent data in the module's dictionary. But I don't know how to recover it in the callback. For example: // Create some persistent state information std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>(); PyImport_AppendInittab("foo", initfoo); Py_Initialize(); std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo"))); bp::object foo_namespace = main_module->attr("__dict__"); main_namespace["foo"] = *foo_module; foo_namespace["state"] = bp::handle<>(state); // Whatever the appropriate wrapper is Then in C++ side of register_callback: void register_callback(bp::object& o) { // How do I extract "state" from the context? } I'm not keen on this last one, since it exposes the state information to the script. I don't want to make the state information global since there may be multiple running Python instances. And regardless, with multiple Python instances I still need a way to determine which Python instance is running to select the appropriate state information. Any suggestions?
On 16/12/14 11:13 PM, Peter LaDow wrote:
I'm embedding Python (using boost::python) into an application plugin that uses callbacks. Essentially, I want to do something like:
In Python (say test.py):
def do_something(): ...
register_callback(do_something)
And on the C++ side I register the register_callback() function:
void register_callback(boost::python::object& o) { // Access some persistent state information }
You defined 'do_something' as a callback, and registered it so it could be called from C++, yes ? Then, in your implementation of 'register_callback', 'o' is actually a reference to your callback function. Do you want to access the persistent state while registering the callback, or from within the callback ? Or am I misreading the code above ? Stefan
BOOST_PYTHON_MODULE(foo) { boost::python::def("register_callback", register_callback); }
void library_entry() { PyImport_AppendInittab("foo", initfoo);
PyInitialize();
// Create some persistent state information
boost::python::exec_file("test.py", ...); }
The issue here is that I need to create the Python context and store things away in an opaque pointer I return to the application. And when Python calls back into the C++, I need to recover that opaque value. For example:
Application -- (calls) --> my C++ library -- (runs) --> Python script -- (calls) --> my C++ library
For that last step, I need to recover some opaque data. Further, I need that opaque data to be persistent. The calls into my C++ library are callbacks from the application. The issue I'm having is trying to figure out how to pass that state information to the C++ register_callback() function. I've tried something like:
namespace bp = boost::python;
class state_info_t { };
void register_callback(std::shared_ptr<state_info_t>& state, bp::object& o); { // Access some persistent state information }
// Create some persistent state information std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>();
PyImport_AppendInittab("foo", initfoo);
Py_Initialize();
std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo")));
main_namespace["foo"] = *foo_module;
bp::scope foo_scope(*foo_module);
// Both of these fail with _a lot_ of errors, most related to "no matching function call to 'get_signature' bp::def("register_callback", [&](bp::object& o) { register_callback(state, o); }, bp::arg("func"));
bp::def("register_callback", std::bind(register_callback, state, std::placeholders::_1), bp::arg("func"));
The other thought I had was to store the persistent data in the module's dictionary. But I don't know how to recover it in the callback. For example:
// Create some persistent state information std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>();
PyImport_AppendInittab("foo", initfoo);
Py_Initialize();
std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo"))); bp::object foo_namespace = main_module->attr("__dict__");
main_namespace["foo"] = *foo_module; foo_namespace["state"] = bp::handle<>(state); // Whatever the appropriate wrapper is
Then in C++ side of register_callback:
void register_callback(bp::object& o) { // How do I extract "state" from the context? }
I'm not keen on this last one, since it exposes the state information to the script.
I don't want to make the state information global since there may be multiple running Python instances. And regardless, with multiple Python instances I still need a way to determine which Python instance is running to select the appropriate state information.
Any suggestions? _______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig
-- ...ich hab' noch einen Koffer in Berlin...
Thanks for the reply! On Thu, Dec 18, 2014 at 9:16 AM, Stefan Seefeld <stefan@seefeld.name> wrote:
You defined 'do_something' as a callback, and registered it so it could be called from C++, yes ? Then, in your implementation of
Yes. At some pointer in the future, the Python function 'do_something' will be called from C++.
'register_callback', 'o' is actually a reference to your callback
Right. Using 'o' I can maintain a reference to the Python callback.
function. Do you want to access the persistent state while registering the callback, or from within the callback ? Or am I misreading the code above ?
I need access to the persistent state while registering the callback. In fact, that persistent state is the context in which the callback from C++ into Python will be done. That is, I need to register 'o' (which references the Python 'do_something') in that persistent state. I think I figured out a mechanism. I was trying to make 'register_callback' a module function. I get that I could do something like: class persistent_state_t { public: ... void register_callback(boost::python::object& cb); }; Then I could register persistent_state_t::register_callback, then create an instance of persistent_state_t, and store that object. Then calls from Python to C++ would always be within the context of the particular instance of persistent_state_t. The issue was figuring out where to store an instance of persistent_state_t, make it visible to the user, and to prevent issues such as copies, deletion, etc. I wanted it to look like a module. The solution that I'm working with now is to store the instance of persistent_state_t in sys.modules with the name of the module. It is a class instance, rather than a module. But calls are within the context of the state. And since sys.modules (retrieved using PyImport_GetModuleDict) is a per-interpreter variable, I'm not polluting other interpreters running. Pete
On 18/12/14 10:43 PM, Peter LaDow wrote:
Thanks for the reply!
On Thu, Dec 18, 2014 at 9:16 AM, Stefan Seefeld <stefan@seefeld.name> wrote:
You defined 'do_something' as a callback, and registered it so it could be called from C++, yes ? Then, in your implementation of Yes. At some pointer in the future, the Python function 'do_something' will be called from C++.
'register_callback', 'o' is actually a reference to your callback Right. Using 'o' I can maintain a reference to the Python callback.
function. Do you want to access the persistent state while registering the callback, or from within the callback ? Or am I misreading the code above ? I need access to the persistent state while registering the callback. In fact, that persistent state is the context in which the callback from C++ into Python will be done.
That sounds a bit confusing to me. :-) Typically, if you associate state with a callback (a "closure"), you think of it as an object whose member function is the callback. But here, the callback function lives in Python, while you want to associate state from the C++ runtime...
That is, I need to register 'o' (which references the Python 'do_something') in that persistent state.
I think I figured out a mechanism. I was trying to make 'register_callback' a module function. I get that I could do something like:
class persistent_state_t { public: ...
void register_callback(boost::python::object& cb); };
Then I could register persistent_state_t::register_callback, then create an instance of persistent_state_t, and store that object. Then calls from Python to C++ would always be within the context of the particular instance of persistent_state_t. The issue was figuring out where to store an instance of persistent_state_t, make it visible to the user, and to prevent issues such as copies, deletion, etc. I wanted it to look like a module.
The solution that I'm working with now is to store the instance of persistent_state_t in sys.modules with the name of the module. It is a class instance, rather than a module. But calls are within the context of the state. And since sys.modules (retrieved using PyImport_GetModuleDict) is a per-interpreter variable, I'm not polluting other interpreters running.
OK, sounds like you have found what you need; great ! :-) Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Thu, Dec 18, 2014 at 8:10 PM, Stefan Seefeld <stefan@seefeld.name> wrote:
That sounds a bit confusing to me. :-) Typically, if you associate state with a callback (a "closure"), you think of it as an object whose member function is the callback. But here, the callback function lives in Python, while you want to associate state from the C++ runtime...
Exactly! There may be a better way to do what I'm trying to do. The relationship is one-to-many (i.e. a single Python object can be associated with many C++ objects). Because of this relationship, I need some context in which the Python object is called in order to make sure each of the C++ objects is aware that the callback occurred. That is, I don't want numerous C++ objects triggering a callback into Python. That shared information (whether or not the Python callback occurred) must be stored somewhere. Since I have no control over the order those C++ objects are invoked (they are passed to a 3rd party library), so when the first one is invoked, it needs to notify the others not to do the Python callback. Anyway, as you noted, I figured something out. I'm sure there's a more clever way to do this. But I'm an EE converted to the software world and I'm probably not thinking like a software engineer. Thanks, Pete
participants (2)
-
Peter LaDow -
Stefan Seefeld