Mailman 3 January 2003 - Cplusplus-sig

Redesign for to_python/from_python converters
by David Abrahams May 21, 2003

May 21, 2003

Some background for this thread is in the following messages: http://article.gmane.org/gmane.comp.python.c++/638 describes the global Boost.Python to/from-python converter registry which associates conversion code with the C++ type(s) being converted. On re-reading this message, I realize it's confusingly worded. Don't be discouraged if it doesn't all sink in. The point is that when a function accepting a C++ argument of type Foo is wrapped, it needs a procedure to extract a Foo from the Python object. The registry associates the code for that procedure with the type Foo. http://article.gmane.org/gmane.comp.python.c++/2044 describes a proposed scheme for dealing with the fact that currently, any given C++ type can have only one to-python conversion in the global registry. This is a problem because one module may use class_<vector<string> >, while another one registers a custom conversion from vector<string> to a Python tuple of strings. http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 describes the general procedure for defining an rvalue from_python converter, and shows why I haven't yet documented how to make user-defined conversions: it's just too hairy a procedure in v2 at the moment. The purpose of this thread is to discuss the design of a new interface for users to interact with to/from-python converters. First, I'd like to discuss some things which I think are desirable. It would be worth knowing which of these are important to the community: 1. Users should have the option to say that certain to/from-python converters shall only have a local effect in a single extension module. 2. It should be possible to optimize away the cost of registry lookups in some cases where it is known that the conversion is defined in the local extension module. This is a completely separate issue from #1. We can have either one without the other. 3. It should be easy for users to explicitly define new conversions. Let's deal with #3 first. You might ask, "what was wrong with the old Boost.Python v1 approach? It sure was simple!" PyObject* to_python(SomeUDT const& x) { ... } SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } It was simple, but it was basically not legal C++. It worked in so many places because it exploited a very common (and somewhat subtle) bug in C++ compilers, but when if you tried to use more conforming compilers (e.g. CodeWarrior >= 8 or recent EDGs), it would fail to compile. The reasons have to do with the rules for looking up these functions from within templates. Such a scheme can be made to work legally, but you have to resort to weird tricks like asking users to define all their converters before #including Boost.Python headers or adding dummy arguments to the functions for the sake of argument-dependent lookup. There is another problem with the v1 scheme. In particular, it was an important design goal of Boost.Python v2 to eliminate the use of C++ exceptions as part of the process of resolving overloaded C++ functions (in v1, we would throw a special exception to indicate a Python argument couldn't be converted to the corresponding C++ argument type). That means that from_python conversion has to be a 2-phase process: first, determine whether a conversion is possible, then if all arguments can be converted, do all the conversions and call the C++ function. Any user-defined from_python conversion needs to be able to report convertibility separately from actually doing the conversion. There were also some ways in which the v1 interface was hard to use: the 2nd argument to the from_python converter had to exactly match the argument type to any C++ functions being wrapped, so you might need: SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } SomeUDT& from_python(PyObject* p, boost::type<SomeUDT&>) { ... } SomeUDT const& from_python(PyObject* p, boost::type<SomeUDT const&>) { ... } SomeUDT* from_python(PyObject* p, boost::type<SomeUDT*>) { ... } SomeUDT const* from_python(PyObject* p, boost::type<SomeUDT const*>) { ... } and a few others. So what should the interface look like? Let's first examine the constraints that the C++ language imposes. User-defined converters can be viewed as behavioral customizations of templates in the Boost.Python library. We basically have two approaches available to us for customizing behaviors: 1. Runtime dispatching through virtual functions or function pointers. This is the approach currently taken in Boost.Python v2 for most converters. The converter registry contains pointer to functions which implement the conversions. Some runtime dispatching is always needed for cross-module conversion support, unless you want to repeat the conversion code in every module which needs it (and nobody wants that). There are other reasons to do this having to do with the way dynamic_cast and RTTI are implemented in most compilers. 2. Compile-time customization. This was the default approach in Boost.Python v1: the compiler would look up the appropriate to_/from_python function and insert a call in the function wrapper. It could even be completely inlined. The disadvantage of this approach is that it is heaviliy dependent on code visibility: the customizations have to be visible in every place that wants to take advantage of them. Boost.Python v2 uses compile-time customization only for to-python conversions of "builtin" C++ types for which there can be only one reasonable Python interpretation. For example, const char* and std::string both are converted to Python strings. Ideally, the user could select compile-time customization for to_python conversion of selected types, and additionally choose to export their converters (e.g. to the global registry). I'm really unsure about the value of compile-time customization of from_python converters. The biggest problem with it is that it limits the whole extension module to *one* conversion method for a given C++ type. Normally, a from_python converter is used to convert one Python type to a given C++ type. If another extension module has exported conversions for other Python types to the same C++ type, do you want to be able use them? There are basically two viable techniques for compile-time customization in C++: defining functions to be found by argument-dependent lookup, and template specialization. For various reasons I won't bore you with, I feel that template specialization is the only recourse for Boost.Python. I'll begin discussion of some possible interfaces in a follow-on message. -- David Abrahams dave(a)boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution

2 15

passing Numeric arrays as arguments
by Greg Landrum May 19, 2003

May 19, 2003

[Boost 1.29, python 2.2.2, g++ 3.2, RH8.0] I've been trying to get a Numeric array object passed in as an argument to a function I've defined in a BPLv2 module, without much success. At the moment I'm happy to get the object in as a PyArrayObject *, because I've got a bunch of code that is expecting those pointers. Since I can't figure out how to extract a PyArrayObject * from a python::numeric::array (if that's even possible), I've attempted to define the following simple function: PyObject *MatFunc(PyArrayObject *arg){ PyObject *res = PyTuple_New(2); return res; } (eventually this will do more, I'm starting small) but whenever I call this Python with a Numeric array as an argument, I get the dreaded catch-all "TypeError: bad argument type for built-in operation". Am I wrong to be even attempting this? Should I just give up and do a normal extension module? -greg ---- greg Landrum (greglandrum(a)earthlink.net) Software Carpenter/Computational Chemist

11 27

Re: [C++-sig] VC7 + auto_ptr + transfer of owership
by Nicolas Lelong March 18, 2003

March 18, 2003

>auto_ptr support is only really complete in the current CVS >version. There are some problems with it in 1.29.0. Huhf, I'm afraid that I checkouted the yesterday's version, and that's the one that causes the problem ! Any idea ? Thanks, Nicolas _________________________________________________________________ Protect your PC - get McAfee.com VirusScan Online http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963

3 6

FW: Converting from SWIG to Boost.Python
by BHELLA,OPINDERJIT (HP-Vancouver,ex1) Feb. 4, 2003

Feb. 4, 2003

In the example below, every function is wrapped with in a Py_BEGIN_ALLOW_THREADS/ Py_END_ALLOW_THREADS macro. This macro allows each function to be able to run in multiple threads in Python. Because Py_END_ALLOW_THREADS must be called to acquire the interpreter lock when the function is completed, the PyEval_RestoreThread(_save); is also called before each catch, if an excepting occurs and we return NULL. Thanks for your help David! "BHELLA,OPINDERJIT (HP-Vancouver,ex1)" < opinderjit.bhella(a)hp.com <mailto:opinderjit.bhella@hp.com>> writes: I have read through the tutorial and was unable > to figure out how to reproduce the > following SWIG logic that uses the Py_BEGIN_ALLOW_THREADS and > Py_END_ALLOW_THREADS macros. > <snip> Can you explain what it does? -- David Abrahams dave(a)boost-consulting.com <mailto:dave@boost-consulting.com> * <http://www.boost-consulting.com> Boost support, enhancements, training, and commercial distribution -----Original Message----- > From: BHELLA,OPINDERJIT (HP-Vancouver,ex1) > Sent: Thursday, January 30, 2003 11:52 AM > To: 'c++-sig(a)python.org' > Subject: Converting from SWIG to Boost.Python > > > Hi, > > I have been using SWIG 1.5p for about a year with good success. However, I > was interested in the newer functionality > that Boost.Python provides. I have read through the tutorial and was > unable to figure out how to reproduce the > following SWIG logic that uses the Py_BEGIN_ALLOW_THREADS and > Py_END_ALLOW_THREADS macros. > > Thanks. > > SWIG example: > %except(python) > { > Py_BEGIN_ALLOW_THREADS > try > { > $function > } > catch(BlackstoneErrorObject &e) > { > PyEval_RestoreThread(_save); > switch(e.getCode()) > { > case SUCCESS: ... break; > case GENERAL_ERROR: break; > default: ...break; > } > return NULL; > } > catch (...) > { > PyEval_RestoreThread(_save); > PyErr_SetString(BlackstoneError,"..."); > return NULL; > } > Py_END_ALLOW_THREADS > }

2 1

Converting from SWIG to Boost.Python
by BHELLA,OPINDERJIT (HP-Vancouver,ex1) Jan. 30, 2003

Jan. 30, 2003

Hi, I have been using SWIG 1.5p for about a year with good success. However, I was interested in the newer functionality that Boost.Python provides. I have read through the tutorial and was unable to figure out how to reproduce the following SWIG logic that uses the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. Thanks. SWIG example: %except(python) { Py_BEGIN_ALLOW_THREADS try { $function } catch(BlackstoneErrorObject &e) { PyEval_RestoreThread(_save); switch(e.getCode()) { case SUCCESS: ... break; case GENERAL_ERROR: break; default: ...break; } return NULL; } catch (...) { PyEval_RestoreThread(_save); PyErr_SetString(BlackstoneError,"..."); return NULL; } Py_END_ALLOW_THREADS }

2 1

VC++ Warnings
by Pere Mato Vila Jan. 30, 2003

Jan. 30, 2003

Hi All, On compiling with VC++ 6.0 our extension module based on Boost.Python 2.0, we are getting the following warnings. Is it something to be worried about? C:/ExtSoft/Boost/boost_1_29_0\boost/python/class.hpp(431) : warning C4700: local variable 'x' used without having been initialized C:/ExtSoft/Boost/boost_1_29_0\boost/python/class.hpp(159) : warning C4700: local variable 'x' used without having been initialized C:/ExtSoft/Boost/boost_1_29_0\boost/python/class.hpp(159) : warning C4700: local variable 'x' used without having been initialized C:/ExtSoft/Boost/boost_1_29_0\boost/python/class.hpp(420) : warning C4700: local variable 'x' used without having been initialized C:/ExtSoft/Boost/boost_1_29_0\boost/python/class.hpp(420) : warning C4700: local variable 'x' used without having been initialized C:/ExtSoft/Boost/boost_1_29_0\boost/mpl/for_each.hpp(73) : warning C4700: local variable 'x' used without having been initialized ------------------------------------------------------------ Pere Mato CERN, EP Division, CH 1211 Geneva 23, Switzerland e-mail: Pere.Mato(a)cern.ch tel: +41 22 76 78696 fax: +41 22 76 79425 gsm: +41 79 20 10855

2 1

New implementation for opaque_pointer_converter
by Gottfried.Ganssauge＠HAUFE.DE Jan. 30, 2003

Jan. 30, 2003

After the last failure with gcc I enhanced the implementation of opaque_pointer_converter<>, added a return_opaque_pointer-ResultConverterGenerator and wrote some documentation. This implementation fails to compile with gcc-2.95 but works on msvc and on gcc-3.2. Please see the attached patch. Cheers, Gottfried

2 1

Pycaml
by M. Evans Jan. 27, 2003

Jan. 27, 2003

OCaml is an awesome, very high-performance, RAD language. This linkage to Python may interest those doing C/C++ for performance reasons. http://pycaml.sourceforge.net/ Performance comparisons at http://www.bagley.org/~doug/shootout/craps.shtml Mark

1 0

AW: AW: [C++-sig] functions whose return type is a pointer
by Gottfried.Ganssauge＠HAUFE.DE Jan. 27, 2003

Jan. 27, 2003

> -----Ursprüngliche Nachricht----- > Von: David Abrahams [mailto:dave@boost-consulting.com] > Gesendet: Freitag, 24. Januar 2003 21:05 > An: c++-sig(a)python.org > Betreff: Re: AW: [C++-sig] functions whose return type is a pointer > > > Gottfried Ganßauge <ganssauge(a)gmx.de> writes: > > >> > I appreciate your efforts, but please don't dump reams > of unreadable > >> > error messages on the other list participants! > > You are right of course, please accept my excuse, everyone. > > > >> > This means serious trouble for your approach, and > probably also that > >> > an architectural change in Boost.Python is called for. > Basically, > >> > typeid(X) is illegal when X is a pointer type. > >> Uhm, sorry, I meant to say "when X is an INCOMPLETE type". > >> > > >> > We'll have to re-think this :( > >> > > > Obviously you already solved this for msvc... > > By exploiting its bug which allows typeid(T) where T is incomplete? I > wouldn't call that "solved" in any useful sense. true, just couldn't resist ... I tried a workaround for the problem by specializing the type_id function template: namespace boost { namespace python { template<> inline type_info type_id(boost::type<opaque_>*) { return type_info (typeid (opaque_ *)); } template<> inline type_info type_id(boost::type<const volatile opaque_ &>*) { return type_info (typeid (opaque_ *)); } }} This works quite well for gcc-3.2 but unfortunately I have no solution for gcc-2.95. That compiler chokes on a missing argument to type_id<opaque_>(). Do you have an idea how to make it work on the older gcc as well. Cheers, Gottfried

1 0

"unknown dependent target" error
by Mike Rovner Jan. 26, 2003

Jan. 26, 2003

Hi, I'm trying to build python extension outside boost tree with MSVC7. Command line: bjam -s"TOOLS=vc7" -n -a gives me an error: unknown dependent target <s:!boost!libs!python!build>boost_python.dll boost_python.dll is there in s:/boost/libs/python/build. What is my problem? I have following config files: >>> boost-build.jam file: ---------------------------------------------------------------------------- ------ boost-build s:/boost/tools/build ; >>> Jamfile file: ---------------------------------------------------------------------------- ------ project-root ; # Include definitions needed for Python modules SEARCH on python.jam = $(BOOST_BUILD_PATH) ; include python.jam ; # Declare a Python extension extension ext : # sources ext.cpp # dependencies <dll>s:/boost/libs/python/build/boost_python ; # Declare a test for the extension module boost-python-runtest test_ext : # Python test driver t_ext.py # extension modules to use <pyd>ext ; >>> Jamrules file (copied from boost root, otherwise bjam complains): ---------------------------------------------------------------------------- ------ # Boost.Build top level Jamrules # See tools/build/index.html for documentation. # Set some important global variables if they haven't already been set by the # user on the command-line or in the environment # Establish this as the root of the boost installation. Most targets will want # $(BOOST_ROOT) in their #include path. $(gTOP) is the name of the variable # containing the path from the invocation directory to the project root. path-global BOOST_ROOT : $($(gTOP)) ; # The current version of Boost. BOOST_VERSION ?= 1.30.0 ; Thank in advance, Mike

2 8