Mailman 3 December 2002 - Cplusplus-sig

Redesign for to_python/from_python converters
by David Abrahams May 21, 2003

May 21, 2003

Some background for this thread is in the following messages: http://article.gmane.org/gmane.comp.python.c++/638 describes the global Boost.Python to/from-python converter registry which associates conversion code with the C++ type(s) being converted. On re-reading this message, I realize it's confusingly worded. Don't be discouraged if it doesn't all sink in. The point is that when a function accepting a C++ argument of type Foo is wrapped, it needs a procedure to extract a Foo from the Python object. The registry associates the code for that procedure with the type Foo. http://article.gmane.org/gmane.comp.python.c++/2044 describes a proposed scheme for dealing with the fact that currently, any given C++ type can have only one to-python conversion in the global registry. This is a problem because one module may use class_<vector<string> >, while another one registers a custom conversion from vector<string> to a Python tuple of strings. http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 describes the general procedure for defining an rvalue from_python converter, and shows why I haven't yet documented how to make user-defined conversions: it's just too hairy a procedure in v2 at the moment. The purpose of this thread is to discuss the design of a new interface for users to interact with to/from-python converters. First, I'd like to discuss some things which I think are desirable. It would be worth knowing which of these are important to the community: 1. Users should have the option to say that certain to/from-python converters shall only have a local effect in a single extension module. 2. It should be possible to optimize away the cost of registry lookups in some cases where it is known that the conversion is defined in the local extension module. This is a completely separate issue from #1. We can have either one without the other. 3. It should be easy for users to explicitly define new conversions. Let's deal with #3 first. You might ask, "what was wrong with the old Boost.Python v1 approach? It sure was simple!" PyObject* to_python(SomeUDT const& x) { ... } SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } It was simple, but it was basically not legal C++. It worked in so many places because it exploited a very common (and somewhat subtle) bug in C++ compilers, but when if you tried to use more conforming compilers (e.g. CodeWarrior >= 8 or recent EDGs), it would fail to compile. The reasons have to do with the rules for looking up these functions from within templates. Such a scheme can be made to work legally, but you have to resort to weird tricks like asking users to define all their converters before #including Boost.Python headers or adding dummy arguments to the functions for the sake of argument-dependent lookup. There is another problem with the v1 scheme. In particular, it was an important design goal of Boost.Python v2 to eliminate the use of C++ exceptions as part of the process of resolving overloaded C++ functions (in v1, we would throw a special exception to indicate a Python argument couldn't be converted to the corresponding C++ argument type). That means that from_python conversion has to be a 2-phase process: first, determine whether a conversion is possible, then if all arguments can be converted, do all the conversions and call the C++ function. Any user-defined from_python conversion needs to be able to report convertibility separately from actually doing the conversion. There were also some ways in which the v1 interface was hard to use: the 2nd argument to the from_python converter had to exactly match the argument type to any C++ functions being wrapped, so you might need: SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } SomeUDT& from_python(PyObject* p, boost::type<SomeUDT&>) { ... } SomeUDT const& from_python(PyObject* p, boost::type<SomeUDT const&>) { ... } SomeUDT* from_python(PyObject* p, boost::type<SomeUDT*>) { ... } SomeUDT const* from_python(PyObject* p, boost::type<SomeUDT const*>) { ... } and a few others. So what should the interface look like? Let's first examine the constraints that the C++ language imposes. User-defined converters can be viewed as behavioral customizations of templates in the Boost.Python library. We basically have two approaches available to us for customizing behaviors: 1. Runtime dispatching through virtual functions or function pointers. This is the approach currently taken in Boost.Python v2 for most converters. The converter registry contains pointer to functions which implement the conversions. Some runtime dispatching is always needed for cross-module conversion support, unless you want to repeat the conversion code in every module which needs it (and nobody wants that). There are other reasons to do this having to do with the way dynamic_cast and RTTI are implemented in most compilers. 2. Compile-time customization. This was the default approach in Boost.Python v1: the compiler would look up the appropriate to_/from_python function and insert a call in the function wrapper. It could even be completely inlined. The disadvantage of this approach is that it is heaviliy dependent on code visibility: the customizations have to be visible in every place that wants to take advantage of them. Boost.Python v2 uses compile-time customization only for to-python conversions of "builtin" C++ types for which there can be only one reasonable Python interpretation. For example, const char* and std::string both are converted to Python strings. Ideally, the user could select compile-time customization for to_python conversion of selected types, and additionally choose to export their converters (e.g. to the global registry). I'm really unsure about the value of compile-time customization of from_python converters. The biggest problem with it is that it limits the whole extension module to *one* conversion method for a given C++ type. Normally, a from_python converter is used to convert one Python type to a given C++ type. If another extension module has exported conversions for other Python types to the same C++ type, do you want to be able use them? There are basically two viable techniques for compile-time customization in C++: defining functions to be found by argument-dependent lookup, and template specialization. For various reasons I won't bore you with, I feel that template specialization is the only recourse for Boost.Python. I'll begin discussion of some possible interfaces in a follow-on message. -- David Abrahams dave(a)boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution

2 15

passing Numeric arrays as arguments
by Greg Landrum May 19, 2003

May 19, 2003

[Boost 1.29, python 2.2.2, g++ 3.2, RH8.0] I've been trying to get a Numeric array object passed in as an argument to a function I've defined in a BPLv2 module, without much success. At the moment I'm happy to get the object in as a PyArrayObject *, because I've got a bunch of code that is expecting those pointers. Since I can't figure out how to extract a PyArrayObject * from a python::numeric::array (if that's even possible), I've attempted to define the following simple function: PyObject *MatFunc(PyArrayObject *arg){ PyObject *res = PyTuple_New(2); return res; } (eventually this will do more, I'm starting small) but whenever I call this Python with a Numeric array as an argument, I get the dreaded catch-all "TypeError: bad argument type for built-in operation". Am I wrong to be even attempting this? Should I just give up and do a normal extension module? -greg ---- greg Landrum (greglandrum(a)earthlink.net) Software Carpenter/Computational Chemist

11 27

Re: [C++-sig] VC7 + auto_ptr + transfer of owership
by Nicolas Lelong March 17, 2003

March 17, 2003

>auto_ptr support is only really complete in the current CVS >version. There are some problems with it in 1.29.0. Huhf, I'm afraid that I checkouted the yesterday's version, and that's the one that causes the problem ! Any idea ? Thanks, Nicolas _________________________________________________________________ Protect your PC - get McAfee.com VirusScan Online http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963

3 6

Presenting a Python embedding tutorial for Boost.Python
by Dirk Gerrits Jan. 9, 2003

Jan. 9, 2003

I've finally finished the first draft of the embedding tutorial I was writing for Boost.Python. I guess I underestimated the task, because it took several days longer than I anticipated, but here it is: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost-sandbox/boo… Note that I am neither an experienced writer, nor an experienced Pythoneer. Any and all proof reading will be greatly appreciated. Regards, Dirk Gerrits

5 15

static boost_python lib?
by Bjorn Pettersen Jan. 7, 2003

Jan. 7, 2003

Is there a way to build boost_python as a static library on msvc6? Now that we can do embedding as a statically linked library, we want to not rely on any non-system .dlls at all.. (I know, give us an inch etc. etc. <wink>) -- bjorn

3 9

MSVC project file
by Brett Calcott Jan. 7, 2003

Jan. 7, 2003

Hi all, For those interested here is a MS project file that builds boost_python. Create a directory under boost/libs/python/build I called mine ./msvc-project. Drop the file into the directory and then load it in DevStudio and build. It will create a ./bin underneath it with boost_python.dll and boost_python_d.dll. David, (if you are listening) given the work in Boost.Build I'm not sure if you are interested in this at all - though I would hazard a guess that there will be more requests. In any case - adding some extra warning switches to the detail/config.hpp makes the build a little easier to watch both for bjam and DevStudio. --- config_old.hpp Sat Nov 09 19:47:06 2002 +++ config.hpp Sat Nov 09 16:19:26 2002 @@ -32,7 +32,11 @@ # define BOOST_MSVC6_OR_EARLIER 1 # endif -# pragma warning (disable : 4786) +# pragma warning (disable : 4786) // disable truncated debug symbols +# pragma warning (disable : 4251) // disable exported dll function +# pragma warning (disable : 4800) //'int' : forcing value to bool 'true' or 'false' +# pragma warning (disable : 4275) // non dll-interface class + # elif defined(__ICL) && __ICL < 600 // Intel C++ 5

11 45

1-D numeric array <-> std::vector<double>
by John Hunter Jan. 4, 2003

Jan. 4, 2003

I have been experimenting with boost::python v2, and just got tuple <-> std::vector<double> working in my extension code using the container_conversions in scitbx. Quite impressive! Now, I am wondering if someone has implemented a similar functionality for transparent passing of Numeric 1D arrays to and from C++ functions which expect std::vector<double>. If this can be done, could someone point me to some example code or documentation. Thanks, John Hunter

3 8

bjam help for embedding
by wtrenker＠hotmail.com Dec. 29, 2002

Dec. 29, 2002

I've been experimenting with Dirk's embedding.cpp and have created my own stand-alone test program but don't know how to build it. I'm getting lost figuring out how to make a Jamfile that builds an executable rather than an extension. A simple example would be greatly appreciated. Thanks, Bill

3 2

BOOST_PYTHON_MODULE across several files
by William Trenker Dec. 26, 2002

Dec. 26, 2002

Is there a way to spread a BOOST_PYTHON_MODULE definition across multiple files? Fltk, like any large library, has it's API spread across many header files. Typically each file represents the declaration of one class. And some of fltk's classes have a few dozen public members each. Ideally I'd like to split up the class_<> definitions into separate files corresponding with the organization of fltk's header files. I'm hoping to develop a unit-testing architecture so each API wrapper file can be testing independently. Another reason is to avoid re-compiling the entire source every time I make changes. Of course these are conventional goals / requirements when a project gets to be any bigger than a single, relatively small source file. I can easily construct a small Python module to cut-and-paste the BOOST_PYTHON_MODULE entries from each file into a single file for the purposes of building the entire extension, but that would still represent a lot of compiling for incremental changes. Is there some way to compile multiple BOOST_PYTHON_MODULE definitions into individual objects files and then let the linker bring it all together under one Python module name? Thanks, Bill

3 3

Marshalling Callbacks
by William Trenker Dec. 25, 2002

Dec. 25, 2002

(This turned out to be a bit lengthy, but I have tried several approaches without success. It is obvious that I am missing a key BPL concept (or two, or three). I need to get this right because there are many places in FLTK that use this callback mechanism.) What I'm trying to figure out is the recommended / design-intended method for marshalling callbacks between the python layer and C++? 3-rd party GUI libraries, like FLTK, handle external events (such as mouse clicks) with a callback mechanism that is something like this very sketchy pseudo-code; -------------------3rd Party---------------------- class Widget { //low-level GUI object void callback(void* func, void* data) { //save callback address and data pointer cbfunc = func; cbdata = data; }; void run() { //wait for mouse clicks when (mouseClicked) (*cbaddr)(cbdata); private: void (*cbfunc)(void*); void* cbdata; //opaque structure from GUI's perspective } Class Button : public Widget { public Widget(); //NB: no callback() or run() defined here } --------------------------------------------------- An application, in C++, does something like this: Class A { public: A() { b = new Button(); }; void onClick(void* appInfo) { //gets called by Widget::run cout << "Mouse was clicked!"; }; void execute(info) { b->callback(&onClick,&info); //(1) b->run(); //(2) } private: Button* b; whatever info; } At (1) the applicaton registers the callback member function so that, at (2), Widget::run knows what to call when the mouse is clicked. --------------------------------------------------- The python equivalent might be: import ext class A: def __init__(self): self.b = ext.Button() def onClick(self,appInfo): print "Mouse Clicked!" def execute(self,info): self.b.callback(self.onClick,info) self.b.run() --------------------------------------------------- When I initially set out,in blissful ignorance, to wrap the above API for Python, the following seemed to be the logical implementation: class_<Widget, boost::noncopyable>("Widget"); class_<Button, bases<Widget>, boost::noncopyable>("Button") .def("callback", &Widget::callback) .def("run", &Widget::run); But, of course, the .def("callback"...) is not as simple as wrapping an int or a char*. It needs some additional options or features to marshall the callback mechanism so that a Python method (or function) gets saved and then eventually invoked by the 3rd-party C/C++ software. Where I'm stuck is what that marshalling mechanism looks like. I'm aware of call<> and call_method<>; I've tinkered with handle<>'s and object's; I've read the docs and studied the examples. Any advice or assistance is very greatly appreciated. Thanks for helping, Bill _________________________________________________________________ The new MSN 8: smart spam protection and 3 months FREE*. http://join.msn.com/?page=features/junkmail&xAPID=42&PS=47575&PI=7324&DI=74… http://www.hotmail.msn.com/cgi-bin/getmsg&HL=1216hotmailtaglines_smartspamp…

3 2