Mailman 3 July 2003 - Cplusplus-sig

Pyste: your opinion about some changes
by Nicodemus July 13, 2003

July 13, 2003

Hi everyone! Prabhu and I have engaged in some discussions on irc about Pyste, and came up with some ideas on how to fix The Order Bug, and the workings of Pyste in general, and would like to know your opinion on this. The Order problem is this: class hierarchies must be exported in order, from the base class to the most-derived class. The naive approach, parse all header files, look up all the bases in the header files and order the classes, occupies too much memory, making it prohibitive with too much classes. The first idea was based on the suggestion by David of using pickle/shelve to hold the declarations in a form that could be easily swaped in/out of disk as needed, solving then the memory problem. We implemented this, it works correctly, but it is too slow, sometimes prohibitive. Plus, Prabhu noted a flaw in Pyste that was always there: you always generate the *entire* wrapper code, there's no support for generating only the wrapper code for a single pyste file. So, he would change something in a Pyste file, like excluding a function, and *all* the wrapper code would be generated again, taking another 10 minutes in his machine. Of course, a more incremental approach is needed. We have thought that a viable solution is that we could lend the responsability of ordering the classes to the users. Ideally, one could do this in a Pyste file: Class('Derived', ...) Class('Base', ...) And Pyste would first instantiate Base, and then Derived. While this is a nice feature, it generates some of complications: - Given Derived, we must know what are its Bases, and we can only do that by calling gccxml. - Given Derived, we must know if Base was exported, so we can put "bases<Base>" inside the class_. If the user just exported Derived, "bases<Base>" should not be generated. We decided to drop this feature, since while nice, it is not totally necessary, since while exporting hierarchies it is natural to export the bases first. So, the user must write: Class('Base', ...) Class('Derived', ...) as he would if he were writing the wrapper by hand. What do you people feel about this? For Pyste to generate the correct code for a given class, it must know all the other classes that are also being exported, as explained above. So we must pass all pyste files to Pyste somehow before being able to produce code for any class. pyste --module=foo foo1.pyste foo2.pyste This generates a file named foo.cpp, with all the wrapper code on it. While convenient, this is impratical, since compile times can get very high for large libraries. That is why --multiple was created: pyste --multiple --module=foo foo1.pyste foo2.pyste That generates 3 files, _main.cpp, _foo1.cpp and _foo2.cpp. Compiling and linking them together gives the same results as without the --multiple flag. And now, to solve the incrementing generation problem, we came up with 2 approaches, and would like to know the opinions of everyone in the list that are interested. Both of them aim to improve --multiple, and should be faster then the current system. 1. One approach is to add an option like --wrap-only <pyste file>, that would generate just the code related to the pyste file, and --main-only, that would generate just the _main.cpp file: pyste --wrap-only foo1.pyste --multiple --module=foo foo1.pyste foo2.pyste Would generate just _foo1.cpp. Notice that we still pass all pyste files to the command line, because as explained before, Pyste must know all classes that are being exported to be able to generate correct code. pyste --main-only --multiple --module=foo foo1.pyste foo2.pyste Would generate just _main.cpp. The advantage with this approach is that it basically extends the current workings of Pyste, ie, users won't have to change anything to keep using it. The disadvantage of this method is that it looks weird, since you have to pass all pyste files even thought you may be interested in generating code for just one. 2. Another approach is to make the dependencies explicit in the pyste file by using another function, Import. This would make clear that for a given pyste file to be exported, the Imported pyste file would have to be taken in account. Going back to the Base/Derived example, we would have either: all.pyste: Class('Base', ...) Class('Derived', ...) or: base.pyste: Class('Base', ...) derived.pyste: Import('base.pyste') Class('Derived', ...) That way, the dependencies between the files is explicit, and the user is no longer required to pass all the files in the command line: pyste --multiple --module=foo derived.pyste would generate _derived.cpp, and: pyste --multiple --module=foo base.pyste would generate _base.cpp. To generate _main.cpp, the user would have to call: pyste --only-main --multiple --module=foo base.pyste derived.pyste The advantages of this method is that the dependencies between the files are explicit in the pyste files themselves, plus it feels more natural than passing all pyste files in the command line in order to generate code for only one of them. The disadvantages is that it complicates the pyste files a little, and changes the way that Pyste currently works. Whew, that's all. Opinions, anyone? Regards, Nicodemus.

3 3

Re: Pyste: your opinion about some changes
by Vincenzo Innocente July 13, 2003

July 13, 2003

Hi Nicodemus, in our project we badly need support for multiple files, multiple packages and inheritance from base classes defined in a different package. We have already implemented some work-arounds and at the moment the support of multiple files IS a show-stopper. I definetively vote for an "Import" statement in Pyste. I consider the explicit dependency through the import statement more clear and exportable than the implicit one through the list of files on the command line. I hope you will be able to provide an implementation (whatever solution you decide to adopt) soon in cvs. We do not mind to break backward compatibility... cheers, Vincenzo

1 0

Re: todo.txt "Implement type_info streaming for GCC" (David Abrahams)
by Jonathan Brandmeyer July 11, 2003

July 11, 2003

On Fri, Dave Abrahams wrote: > Of course, we still have the GPL problem if we link to that :( No more so than any other part of GNU libstdc++. The function is mandated the "multi vendor ABI standard." This facility's documentation states that it is licensed under the same terms as the rest of GNU libstdc++. It seems to me that you should have no issues with using these functions in Boost.Python. HTH, -Jonathan >From cxxabi.h (leading //es removed for plain text email): As a special exception, you may use this file as part of a free software library without restriction. Specifically, if other files instantiate templates or use macros or inline functions from this file, or you compile this file and link it with other files to produce an executable, this file does not by itself cause the resulting executable to be covered by the GNU General Public License. This exception does not however invalidate any other reasons why the executable file might be covered by the GNU General Public License. From http://gcc.gnu.org/onlinedocs/libstdc++/libstdc++-html-USERS-3.3/namespacea…: (Brief description of abi::__cxa_demangle()) New ABI-mandated entry point in the C++ runtime library for demangling. ( A note at the bottom) The same demangling functionality is available via libiberty (<libiberty/demangle.h> and libiberty.a) in GCC 3.1 and later, but that requires explicit installation (--enable-install-libiberty) and uses a different API, although the ABI is unchanged. >From the draft IA64 ABI docs: http://www.codesourcery.com/cxx-abi/abi.html#demangler

1 0

Re: todo.txt "Implement type_info streaming for GCC"
by Raoul Gough July 11, 2003

July 11, 2003

A while ago, David Abrahams wrote: >There are a few places where error reporting in Boost.Python depends >on being able to show users a reasonable representation of a C++ type >name. Currently, we use typeid(x).name() to get the name. Most >compilers produce a good result, however g++ does not. I just ran a >small experiment to see if the type encoding scheme used by G++ could >easily be deciphered, and it appears that it can. If we want to >improve error reporting with G++, it should be relatively easy. I seem to recall that the g++ typeid().name() gives you the type's mangled name, in which case the supplied c++filt will decode it for you (maybe modulo one leading underscore?). If so, I think a simple system() call would probably suffice for what David is describing. Or am I missing some more info on the difficulties involved? The source to the demangler is GPL'd, of course (in gcc/libiberty/cp-demangle.c) and is also including in gdb, AFAIK. It's currently >3000 lines of code, so I wouldn't recommend trying to reimplement it for Boost.Python! -- Raoul Gough "Let there be one measure for wine throughout our kingdom, and one measure for ale, and one measure for corn" - Magna Carta

5 12

Call policies for unreferencing iterator ...
by Pierre Barbier de Reuille July 11, 2003

July 11, 2003

Hello, I'm writing some templates to automate export of stl stuffs (at least some of the stuffs). I hope I'll soon be able to share my work to know what you'll think of it. But before that I have a question about the iterators. I want to export the STL iterators using a STL-like interface (I've already done some work about exporting interators with python interface). My concern is : I defined the __call__ method as the operator* of the iterator. But I want to return a reference to the object (to be able to modify it if the iterator if mutable). The problem is with the call policies ! It's not an internal reference, because the reference will still be valid after the iterator is destroyed, and it's a bad idea to create it as an "existing object" for it will crash if the user try to use the pointer after the container is deleted. My problem is I don't know how to tie the lifetime of the object with the container ... and I don't even think it's possible to do that. What would you do ? I hope I'm clear enough ... Thanks, -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68

2 1

method injector and pickling
by Ralf W. Grosse-Kunstleve July 10, 2003

July 10, 2003

I ran into pickle problems when converting some of the classes in our application to make use of David's method injector (as explained in tutorial/doc/quickstart.txt). The problems disappeared when I added the if statement below to disable injection of __module__ and __file__. Proactively I added __init__ and __del__ to the list. Is this a useful modification? Should we update quickstart.txt accordingly? Ralf from scitbx.array_family import flex # see boost/libs/python/doc/tutorial/doc/quickstart.txt boost_python_metaclass = flex.double.__class__ class injector(object): class __metaclass__(boost_python_metaclass): def __init__(self, name, bases, dict): for b in bases: if type(b) not in (self, type): for k,v in dict.items(): if (k in ("__init__", "__del__", "__module__", "__file__")): continue setattr(b,k,v) return type.__init__(self, name, bases, dict) __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com

1 0

RE: [C++-sig] Re: Wrapper for exception translation
by Gottfried.Ganssauge＠HAUFE.DE July 10, 2003

July 10, 2003

... > > > Currently, in order to wrap user defined exceptions you need to > > > > - allocate a python exception using PyErr_NewException > > Well, no. I don't see any call to PyErr_NewException in > libs/python/test/exception_translator.cpp Right, you don't need to if you are content with RuntimeError exceptions. My objective was another one: I wanted to create module exceptions that are derived from the python "Exception" class. > > > This function needs to be parametrized using a specific naming > > convention > > It's not clear to me what you're referring to. Are you sure you mean > "naming convention"? Yes. >From the documentation to PyErr_NewException: ... The __module__ attribute of the new class is set to the first part (up to the last dot) of the name argument, and the class name is set to the last part (after the last dot). > > > you also need to find a place where this exception instance lives > > and is accessible to the exception translator > > I don't know what you mean here. For one thing, first "this" is a > function and now "this" is an exception instance. "this function" meant PyErr_NewException which creates a new exception instance. The exception instance (from now on called "e") created by PyErr_NewException needs to live somewhere because in order to signal a python exception you must hand e to PyErr_SetString/PyErr_SetObject. > > > - define an exception translator which signals the exception defined > > above > > What do you mean by "define an exception translator"? A translation function in the sense of register_exception_translator. > What do you mean by "signals the exception"? Calling PyErr_SetString/PyErr_SetObject. > > > - register the exception translator with the library > > This I understand to mean something like: > > register_exception_translator<error>(&translate); > > > My facility allows to do this with only one function call > > "exception_<Exception type> (name)" within the module init function. > > It will allocate a python exception > > Do you mean it will create a Python subclass of Exception? Or is it > something else? I guess the docs for PyErr_NewException are a little > unclear on this point, but I think it's what I said. Yes, that's what I mean. > > > and store it within an instance of the > > translated_exception_handler<> class. The name for that exception is > > taken from the name parameter and the current scope according to the > > naming convention documented in the manual page for > > PyErr_NewException. > > Hmm, I didn't know about PyErr_NewException, but it looks useful. > > > A reference to the translated_exception_handler is stored within the > > class dictionary of the newly allocated python exception in order to > > keep the translator alive. The translated_exception_handler > > provides the exception translator to be registered with the > > library. > > OK, OK, wait. I was beginning to like where this was going, (I like > what you're accomplishing) but this all sounds *way* too complex for > that purpose. Especially the "creation of an anonymous class" that > you do in the code here looks like a messy hack likely to break at > some point. Why is such a complicated mechanism needed? You're probably right. What I wanted to achieve was the following: Create an instance of a translated_exception_handler<> t which will be registered with the library by calling register_exception_translator<>(). Normally t would be destructed by the end of exception_<>() thereby leaving a registration to a non-existing translator - very bad karma. Because I didn't like to have an additional container for all translated_exception_handler<> instances, I chose to store t within e's class dictionary. At that moment it seemed to be the easiest way to create a wrapper class using class_<> instead of making a true python class by hand (and it even works!). You are right, it is way too complex, I will change it to the mechanism I used in opaque_pointer_converter. > > > It uses a generic method to obtain an exception description > > by calling a function get_exception_description<>() which has a > > default implementation in the detail namespace. Finally the python > > exception itself is made available in the module's namespace under > > the name given. > > > > get_exception_description<>() currently returns a const char * which > > is passed to PyErr_SetString(). It might be worthwile to make it > > return an object which then could be passed to > > PyErr_SetObject(). This modification would make it possible to pass > > an instance of a class which wraps the exception caught, which again > > would make the exception instance itself available to python > > programs. > > > > Another enhancement could be the generation of derived python > > exceptions, an area in which I have no experience yet. This could be > > by an additional template parameter to the exception_<>() function > > which defaults to the allocation with PyErr_NewException(). > > > > What do you think? > > I think I understand this much: > > your code provides the ability to easily define new Python > exception types corresponding to particular C++ exception types. > > Bravo to that; I'd like to get this capability into the library. > > I do think the implementation is needlessly convoluted. Just by > inspection it looks to me like you make heavy use of a dictionary > > // class dictionary for the new exception class > dict d; > > which later simply goes out of scope. What's the point of that? > Have I missed something? It doesn't simply go out of scope. Before going out of scope it is passed to PyErr_NewException as the new exception's class dictionary. I was under the impression that it wouldn't have one otherwise. Ok then, I'll make a redesign ... Thnx for your feedback. > > -- > Dave Abrahams > Boost Consulting > www.boost-consulting.com Cheers, Gottfried

2 1

Patches for pyste regarding exception specifications
by Gottfried.Ganssauge＠HAUFE.DE July 10, 2003

July 10, 2003

Hi everyone, I made a set of patches for pyste which allows pyste to keep exception specifications in automatically generated wrappers for classes with virtual functions. This obviously needs support from gccxml so this only works with the CVS version of gccxml. Older versions of gccxml don't generate the necessary information for this to work but in this case the code just isn't used. Furthermore this introduces a new Wrapper-Function: Exception(), which generates a call to my exception handling facility posted yesterday. Finally two minor coding errors in the wrapping of free functions are fixed which affected my project. Please see the attachment. The patches are relative to the pyste subdirectory within boost/libs/python and cover only the src subdirectory. Cheers, Gottfried

2 1

RE: [C++-sig] Re: Wrapper for exception translation
by Gottfried.Ganssauge＠HAUFE.DE July 10, 2003

July 10, 2003

... > > This looks like a question about integration with Boost.Python > itself to me. Isn't it? You're right, of course What I meant to say was: - I would be delighted to have this facility integrated with the library - I would be even more delighted if it was usable through some pyste enhancement > > Could you explain how your facility improves on what's already in the > library? I'm sorry to be dense, but the answer's not jumping out at > me. Currently, in order to wrap user defined exceptions you need to - allocate a python exception using PyErr_NewException This function needs to be parametrized using a specific naming convention you also need to find a place where this exception instance lives and is accessible to the exception translator - define an exception translator which signals the exception defined above - register the exception translator with the library My facility allows to do this with only one function call "exception_<Exception type> (name)" within the module init function. It will allocate a python exception and store it within an instance of the translated_exception_handler<> class. The name for that exception is taken from the name parameter and the current scope according to the naming convention documented in the manual page for PyErr_NewException. A reference to the translated_exception_handler is stored within the class dictionary of the newly allocated python exception in order to keep the translator alive. The translated_exception_handler provides the exception translator to be registered with the library. It uses a generic method to obtain an exception description by calling a function get_exception_description<>() which has a default implementation in the detail namespace. Finally the python exception itself is made available in the module's namespace under the name given. get_exception_description<>() currently returns a const char * which is passed to PyErr_SetString(). It might be worthwile to make it return an object which then could be passed to PyErr_SetObject(). This modification would make it possible to pass an instance of a class which wraps the exception caught, which again would make the exception instance itself available to python programs. Another enhancement could be the generation of derived python exceptions, an area in which I have no experience yet. This could be by an additional template parameter to the exception_<>() function which defaults to the allocation with PyErr_NewException(). What do you think? > > -- > Dave Abrahams > Boost Consulting > www.boost-consulting.com Cheers, Gottfried

2 1

Wrapper for exception translation
by Gottfried.Ganssauge＠HAUFE.DE July 10, 2003

July 10, 2003

Hi everyone, for my current project I was in need for a simple method to make user defined exceptions within my module. The basic framework in the form of exception translation is there so I only needed to make it a bit more accessible. I made a template function exception_<Exception> (const char *name) which creates a python exception and an instance of templated translation handler. The translation handler instance is held in the python translation's class dictionary as the __handler__ attribute. Then the handler is registered as an exception translator for the Exception using register_exception_translator(). The python exception object is finally registered in the current scope's dictionary using the "name" argument to exception_(). To register an exception translator for exception e you simply call exception_<e> ("e") within your module init function. This implementation is quite usable for me but I would be totally glad if it could somehow be integrated with pyste. I tested it with gcc-3.2, gcc-3.3 and gcc-2.95 Cheers, Gottfried

2 1