Mailman 3 June 2003 - Cplusplus-sig

RE: [C++-sig] Converting CString to/from std::string
by Kirsebom Nikolai Sept. 25, 2003

Sept. 25, 2003

-----Original Message----- From: Ralf W. Grosse-Kunstleve [mailto:rwgk@yahoo.com] Sent: 27. mai 2003 08:46 To: c++-sig(a)python.org Subject: Re: ÆC++-sigÅ Converting CString to/from std::string --- Nikolai Kirsebom <nikolai(a)micon.no> wrote: > I have a rather large set of C++ classes I want to make accessible > (initially only parts of some of the classes) from Python. I cannot > change the classes as they are defined and used throughout a large > application. Some of the methods take CString as parameters and others > return CString values. > > Does anyone have suggestion on how I can convert these parameters so > that I can use Boost.Python to generate the wrapper code. What do you want your CStrings to be in Python? Regular Python strings? If you think "yes" remind yourself that Python strings are immutable. I.e. if you have function signatures involving non-const references or pointers to CString as arguments or return values the mapping from C++ CStrings to Python strings will not work very well. More depending on your response. Ralf Some time ago I made a 'socket based' system (using BISON/FLEX for parsing) where I expose C++ objects to python. In this system CStrings are regular Python strings. So as an example I can write: --------------------------------------------- C++: class IF { CString MenuText; CString GetHelpText(); int SetHelpText(CString s); }; In Python: o = IF() # Behind the 'curtains' create a 'link' to a C++ object print o.MenuText s = o.GetHelpText(o.MenuText + "some text") ---------------------------------------------- So I would like something similar. I'm very new to the Boost system and do not really have experience in C++ Template development so I find the Reference Manual somewhat difficult to understand. This may explain if I request 'dead easy' or 'impossible' things. Thanks for any help Nikolai

4 6

First pass at exception translator template
by Gavin Doughtie July 31, 2003

July 31, 2003

So, this is probably gross or non thread-safe in some manner that I'm not clued into yet. I'd appreciate any elegant refactorings, but this seems to work and will allow me to just specify a bunch of "REGISTER_EXCEPTION" lines in my module. --------8<------------------------------- template <typename ExceptionType> struct ExceptionTranslator { typedef ExceptionTranslator<ExceptionType> type; static PyObject *pyException; static void RegisterExceptionTranslator(scope & scope, const char* moduleName, const char* name) { // Add the exception to the module scope std::strstream exName; exName << moduleName << "." << name << '\0'; pyException = PyErr_NewException(exName.str(), NULL, NULL); handle<> instanceException(pyException); scope.attr(name) = object(instanceException); // Register a translator for the type register_exception_translator< ExceptionType > ( &ExceptionTranslator<ExceptionType>::translateException ); } static void translateException(const ExceptionType& ex) { PyErr_SetString(pyException, ex.getMessage().ptr()); } }; template<typename ExceptionType> PyObject* ExceptionTranslator<ExceptionType>::pyException; // Convenience macro #define REGISTER_EXCEPTION(scopeRef, moduleName, className) \ ExceptionTranslator<className>::RegisterExceptionTranslator(scopeRef, moduleName, #className) // Module ====================================================================== BOOST_PYTHON_MODULE(my_module) { scope moduleScope; REGISTER_EXCEPTION(moduleScope, "my_module", InstanceException); ..... } -- Gavin Doughtie DreamWorks SKG

2 2

return_self_policy
by Nikolay Mladenov July 19, 2003

July 19, 2003

Posting return_self policy implementation Nikolay ''' >>> from return_self import * >>> l1=Label() >>> l1 is l1.label("bar") 1 >>> l1 is l1.label("bar").sensitive(0) 1 >>> l1.label("foo").sensitive(0) is l1.sensitive(1).label("bar") 1 ''' def run(args = None): import sys import doctest if args is not None: sys.argv = args return doctest.testmod(sys.modules.get(__name__)) if __name__ == '__main__': print "running..." import sys sys.exit(run()[0])

3 25

Re: [C++-sig] Re: Interest in luabind
by Daniel Wallin July 5, 2003

July 5, 2003

> "Daniel Wallin" <dalwan01(a)student.umu.se> writes: > > >> "Daniel Wallin" <dalwan01(a)student.umu.se> writes: > > > > The type of the type-identifier needs to be known to the > > library though, so my point is that it can't be a > template > > which the user just specializes. You also need to set > the > > type for the compiled part of the library. Or am I > missing > > something here? > > Probably not; I think you have a good point, at least, if > we > need runtime polymorphism among type id objects or if they > need to be manipulated in the compiled part of the > library. > > typedefs still work, though. Yeah, a typedef would have been a much better solution. I think the reason we use macro's is because we used const type_info* directly, instead of wrapping it in a class. So we couldn't simply use < and == when comparing the types. > > >> >> >> > This of course causes some problems if we want > to > >> >> >> > downcast > >> >> >> > >> >> >> And if you want to support CBD (component based > >> >> >> development, i.e. cross-module conversions), > unless > >> >> >> you can somehow get all authors to agree on how > to > >> >> >> identify types. Well, I guess they could use > >> >> >> strings and manually supply what > >> >> >> std::type_info::name() does. > >> >> > > >> >> > It doesn't seem wrong that the authors need to use > the > >> >> > same typeid. If you aren't developing a closed > system, > >> >> > don't change typeid system. > >> >> > >> >> I was thinking more of id collisions among > extensions > >> >> which don't intend to share types. It becomes less > of an > >> >> issue if people use string ids. > >> > > >> > But like I said, if you intend to use your module > with other > >> > modules; don't change the typeid system. > >> > >> I'm not talking about changing systems, just the need > to > >> ensure unique type IDs for different types across > modules. > > > > Yeah I know, what I'm saying is that you will only get > > problems like that if you change your type_info > > represantation to something like 'int'. In which case > you > > have changed type id system. This isn't a problem as > long > > as you stick to typeid() and type_info? > > Assuming a single compiler and the use of distinct > namespaces, no. Right. > > >> >> > LUABIND_DONT_COPY_STRINGS > >> >> > >> >> What kind of string copying gets disabled? > >> > > >> > It causes names to be held by const char* instead of > >> > std::string. (class names, function names etc). > >> > >> Why not always do that? > > > > Hold const char*? Because you can't control their > > lifetime? :) > > You mean that some people want to compute these things > dynamically instead of using string literals? Nobody has > ever asked me for that. I bet this is one area of > configurability that could be dropped. We actually have one user that uses this feature, I don't know if he actually needs to though. So yeah, it could probably be dropped. > >> > Need to think about this some more. It's no special > >> > case in BPL though, it's just another type of > >> > instance_holder, correct? > >> > >> Not only that. It's an issue of where the instance > holder > >> gets constructed. In this case it is constructed > directly > >> in the storage for the Python object. > > > > Right, that's pretty cool. > > Yeah, but I doubt it's saving much, and it leads to much > complication (c.f. instance_new and instance_dealloc in > libs/python/src/object/class.cpp). If objects are getting > created and destroyed a *lot* it could reduce > fragmentation > and increase locality... but as I said I have big doubts. Yeah, fragmentation issues can always be solved with pool allocation. Is the complexity only due to value-holders, or just different kind of holders? Seems to me (without having looked at instance_new/instance_dealloc yet..) like you would need the same allocation routines for pointer-holders as well. > >> The to-python converter that gets registered for > >> auto_ptr<T> by using it in a class<T, std::auto_ptr<T>, > >> ... > or by using > >> register_pointer_to_python<std::auto_ptr<T> > knows > what > >> to do. > > > > Ah, right. It's pretty nice to be able to hold the > instances > > in different kind of holders. > > Yeah. > > Note also that shared_ptr is "magic", in that any wrapped > T > can be converted to shared_ptr<T> regardless of holder, > and > the resulting shared_ptr<T> can be converted back to the > original Python object (not just a new one sharing the > same > C++ object). Ok. I noticed that while browsing the code. The custom deleter is a powerful tool. :) > >> > and even the > >> > ones that doesn't can still be used in different > contexts. > >> > ('result' when doing C++ -> lua, and '_N' when doing > lua -> > >> > C++). > >> > >> How is that *not* a case of bidirectionality? > > > > :) It is, my fault. I meant it could be used in > different > > contexts with the same direction, it should read > something > > like: 'result' when calling a C++ function, '_N' when > > calling a lua function. > > > >> That rule makes me nervous, because the terms are > >> backwards when calling lua/python from C++. Do people > >> get confused? > > > > I don't think they do, I don't think anyone has reported > any > > problems with it yet. It's backwards for a reason > though, > > and it's pretty clear when to use which placeholder. > (except > > in some special cases, where documentation helps). > > OK. Maybe we could allow a default argument which would > "just work" for these cases. Another option is "_", which > is used in MPL for something similar. Sure, that would be nice. You can also introduce additional placeholder aliases for the cases where '_N' and 'result' doesn't fit very well. -- Daniel Wallin

6 21

Refactoring, Compilation Speed, Pyste, Lua/Ruby/JavaScript...
by David Abrahams July 3, 2003

July 3, 2003

A number of you (who have my sympathy!) have begun to dig in to Boost.Python's implementation to some degree, usually so you can accomplish something that isn't exposed in the documented interface. I'm sure most of you have noticed that it's not always easy to navigate the internal structure of the library. I'd like to address some of that and at the same time give us the tools to solve some other problems, which I'll discuss at the end. I propose to divide the library's implementation into several namespace layers with corresponding subdirectories of boost/python. These are just rough divisions and I would welcome suggestions for finer-grained layering, or better names, or... These layers are generally ordered from dependencies to dependents. core (for lack of a better name) - This is a Python-independent layer which contains the framework of the type-converter registry, inheritance.hpp which manages base<->derived class conversions, the exception translator framework, boost/python/type_id.hpp, and possibly a few components from the current boost/python/detail. converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities function - Wrapping of (member) (function) pointers into Python callable objects. Maybe this should be called "callable". callback - Invocation of Python callable objects from C++, e.g. call_method<...>, call<...> api - various namespace-scope functions such as del(), getattr(), etc. objects - object, str, dict, tuple, list, long_ ... classes - specific support for class wrapping, including instance_holders, support for smart pointers, etc. I see great potential in this reorganization. One thing I'd like to see happen in the near term is that Pyste might be altered to use some lower-level components of Boost.Python directly to improve compile times. For example, instead of relying on Boost.Python to generate wrappers for functions and member functions, Pyste might instead generate function wrappers directly using Boost.Python's type converters. I have no doubt that Python executes faster than the template engine in most C++ implementations! A longer-term goal is to move enough of the code into the core that we could easily re-use it to support other interpreted languages besides Python. Comments? -- Dave Abrahams Boost Consulting www.boost-consulting.com

7 35

passing a dynamic PyObject * from C++ to Python
by F. Oliver Gathmann July 1, 2003

July 1, 2003

I have a function that reads a file and returns (depending on the content of the file) either a Foo or a Bar wrapped in a PyObject* (both Foo and Bar objects are expensive to copy and have to be passed via a smart pointer). With Boost.Python v1 I was able to do PyObject * readFile(char const * fileName) { PyObject * obj; FileInfo info(fileName); if (info.isFoo()) { std::auto_ptr<Foo> foo(new Foo(info.data())); obj = to_python(foo); } else { std::auto_ptr<Bar> bar(new Bar(info.data())); obj = to_python(bar); } return ref(obj); } and wrap this like so: BOOST_PYTHON_MODULE_INIT(vigracore) { module_builder this_module("reader"); this_module.def(&readFile, "readFile"); } With Boost.Python v2, I have not been able to find an equivalent solution; I tried various approaches gathered from the Boost.Python FAQ and documentation, e.g. template <class T> T identity(T x) { return x; } template <class T> object get_pointer_reference(T x) { object f = make_function(&identity<T>, return_value_policy<manage_new_object>() ); return f(x); } object readFile(char const * fileName) { FileInfo info(fileName); if (info.isFoo()) { Foo * fooPtr = new Foo(info.data()); return get_pointer_reference(fooPtr); } else { Bar * barPtr = new Bar(info.data()); return get_pointer_reference(barPtr); } } BOOST_PYTHON_MODULE(reader) { def("readFile", &readFile); } which compiles fine but crashes at runtime (using gcc 3.2.2 on Linux with a recent CVS version of Boost). Any hints that might help me solving this mystery would be much appreciated! Oliver -------------------------------------------------------------------- F. Oliver Gathmann, Ph.D. Director IT Unit Cenix BioScience GmbH Pfotenhauer Strasse 108 phone: +49 (351) 210-2735 D-01307 Dresden, Germany fax: +49 (351) 210-1309 fingerprint: 8E0E 9A64 A07E 0D1A D302 34C2 421A AE9F 4E13 A009 public key: http://www.cenix-bioscience.com/public_keys/gathmann.gpg --------------------------------------------------------------------

6 9

Virus
by Nikolay Mladenov June 30, 2003

June 30, 2003

I got two e-mails from two members of this list containing virus This link http://www.ku.edu/acs/virus/viruses/sobigE.shtml has description of the virus. For me this means, that probably a third member of the list has infected machine. nikolay

1 0

Cross-module support?
by Hugo van der Merwe June 29, 2003

June 29, 2003

I am trying to wrap a library that has classes derived from a library that has already been wrapped - is this currently possible? If not, I am considering using some C++ code to do the parts of the work that requires access to both libraries, in this case I believe each library will need some "global registry" of sorts, so that the C++ code can change data in each library with having to get/return data from/to python. Any other alternatives? Thanks, Hugo van der Merwe

2 1

Re: [C++-sig] Re: Interest in luabind
by Daniel Wallin June 28, 2003

June 28, 2003

> "Daniel Wallin" <dalwan01(a)student.umu.se> writes: > >> Users could just specialize or overload > >> boost::luapython::type_id, which, in the no-RTTI case, > >> has no no default definition. > > > > Note that we need to store the type_info object, and > > that's why we have macros for this. So that the user can > > change the type of that object. This wouldn't really > work > > with specialization. > > template <class T> struct type_info > #ifndef NO_RTTI > { > typedef std::type_info type; > static type get(); > } > #endif > ; > > template <int n> > struct int_type_info > { > typedef int type; > static int get() { return n; } > }; > > template <> > struct type_info<Foo> > : int_type_info<42> > {}; > > Of course I have no objection to using macros to generate > specializations. The type of the type-identifier needs to be known to the library though, so my point is that it can't be a template which the user just specializes. You also need to set the type for the compiled part of the library. Or am I missing something here? > > >> >> > This of course causes some problems if we want to > >> >> > downcast > >> >> > >> >> And if you want to support CBD (component based > >> >> development, i.e. cross-module conversions), unless > you > >> >> can somehow get all authors to agree on how to > identify > >> >> types. Well, I guess they could use strings and > manually > >> >> supply what std::type_info::name() does. > >> > > >> > It doesn't seem wrong that the authors need to use > the > >> > same typeid. If you aren't developing a closed > system, > >> > don't change typeid system. > >> > >> I was thinking more of id collisions among extensions > >> which don't intend to share types. It becomes less of > an > >> issue if people use string ids. > > > > But like I said, if you intend to use your module with > other > > modules; don't change the typeid system. > > I'm not talking about changing systems, just the need to > ensure unique type IDs for different types across modules. Yeah I know, what I'm saying is that you will only get problems like that if you change your type_info represantation to something like 'int'. In which case you have changed type id system. This isn't a problem as long as you stick to typeid() and type_info? > >> > LUABIND_DONT_COPY_STRINGS > >> > >> What kind of string copying gets disabled? > > > > It causes names to be held by const char* instead of > > std::string. (class names, function names etc). > > Why not always do that? Hold const char*? Because you can't control their lifetime? :) > > >> > . and the typeid macros. > >> > > >> > Most are results of user requests. Massive > configuration > >> > is quite important to our users, since lua is used > alot on > >> > embedded systems. > >> > >> Have your users come back after getting these features > and > >> given you any feedback about how much performance > they've > >> gained or space they've saved? > > > > No. But some of them is a must-have for alot of > > developers. In particular the ability to turn off > > exception handling. > > Oh sure, I believe that one, especially because some shops > (advisedly or no) have a policy against EH and RTTI. You > don't want to just leave them out. I am just leery of > configurability in general and would tend to resist making > any new macros part of the official release until users > had > told me it made a big difference to them in alpha/beta > stages. In particular, "massive" configurability is not > neccessarily desirable. It creates "massive" maintenance > and testing headaches. I agree. I can ask around on our mailing list later about what kind of configuration people think is interesting, and how much of what we have now they are using. > > >> >> [I assume this means that all of your C++ objects > are held > >> >> within their lua wrappers by pointer. I went to > >> >> considerable lengths to allow them to be held > by-value, > >> >> though I'm not sure the efficiency gain is worth the > cost > >> >> in flexibility.] > >> > > >> > Correct. We hold all C++ objects by pointer. I can > see why > >> > it could be interesting to be able to hold objects by > value > >> > though, especially with small objects. I'm not sure > it would > >> > be worth it though, since the user can just provide a > custom > >> > pool allocator if allocation is an issue. > >> > >> Still costs an extra 4 bytes (gasp!) for the pointer. > >> Yeah, it was in one of my contract specs, so I had to > >> implement it. Besides, it seemed like fun, but I bet > >> nobody notices and I'd be very happy to rip it out and > >> follow your lead on this. > > > > Need to think about this some more. It's no special case > > in BPL though, it's just another type of > instance_holder, > > correct? > > Not only that. It's an issue of where the instance holder > gets constructed. In this case it is constructed directly > in the storage for the Python object. Right, that's pretty cool. > > >> > Wouldn't both the examples you provided require A to > be held > >> > by auto_ptr<A> in python? > >> > >> Only the A created by call_function for this callback. > >> You can have As held any number of ways in the same > >> program. > > > > Ah, right. You just instantiate different > instance_holder's. > > Does this mean there's special handling of auto_ptr's? > > What do you mean by "special handling"? I think the > answer > is no, though I also think there should be special > handling > for convenience and efficiency. I meant like an overload to extract the type held by the auto_ptr. But you already answered that. > > > Otherwise, how can you tell which type is being held by > > the pointer? (In the call_function example). > > The to-python converter that gets registered for > auto_ptr<T> > by using it in a class<T, std::auto_ptr<T>, ... > or by > using register_pointer_to_python<std::auto_ptr<T> > knows > what to do. Ah, right. It's pretty nice to be able to hold the instances in different kind of holders. > > >> >> I can begin to see the syntactic convenience of your > way, > >> >> but I worry about the parameterizability. In the > first > >> >> case "result" is the only possible appropriate arg > and in > >> >> the 2nd case it's "_1". > >> > > >> > What exactly do you worry about? That someone would > use > >> > the wrong placeholder? I think the intended use is > quite > >> > clear. > >> > >> I guess I'm just a worrywart. Do the other policies > have > >> similar broad applicability? If so, I'm inclined to > >> accept > >> that you have the right idea. > > > > Most conversion policies have double direction > > Cool, I accept your scheme. Great. > > > and even the > > ones that doesn't can still be used in different > contexts. > > ('result' when doing C++ -> lua, and '_N' when doing lua > -> > > C++). > > How is that *not* a case of bidirectionality? :) It is, my fault. I meant it could be used in different contexts with the same direction, it should read something like: 'result' when calling a C++ function, '_N' when calling a lua function. > That rule makes me nervous, because the terms are > backwards > when calling lua/python from C++. Do people get confused? I don't think they do, I don't think anyone has reported any problems with it yet. It's backwards for a reason though, and it's pretty clear when to use which placeholder. (except in some special cases, where documentation helps). > > >> > void f(B*, B*, B*, python::list) > >> > void f(D*, D*, D*, std::vector<int>) > >> > > >> > f(D(), D(), D(), [1, 2, 3]) > >> > >> Yeah, I guess that looks right, but I don't know. At > some > >> point things are just ambiguous. > > > > Exactly. Perhaps introducing weights on conversions > might > > increase the ambiguity for the user, since it's hard to > tell > > at which point the overload system will turn around and > > choose another overload. Does this make any sense? > > Yes. But as I meant to say to Andrei, none of the > "classic" > multimethod systems account for coercion, so if there's > some > reason they can't use your simple rule it has to be > something you can understand in terms of simple > inheritance > hierarchies. Right. I'll try to read up on this a bit more, I missed those Andrei mails you cc:ed yesterday, gonna read them today. -- Daniel Wallin

2 1

Embedding Python, threading and scalability
by Qiu, Wenning June 27, 2003

June 27, 2003

I am researching issues related to emdedding Python in C++ for a project. My project will be running on an SMP box and requires scalability. However, my test shows that Python threading has very poor performance in terms of scaling. In fact it doesn't scale at all. I wrote a simple test program to complete given number of iterations of a simple loop. The total number of iterations can be divided evenly among a number of threads. My test shows that as the number of threads grows, the CPU usage grows and the response time gets longer. For example, to complete the same amount of work, one thread takes 10 seconds, 2 threads take 20 seconds and 3 threads take 30 seconds. The fundamental reason for lacking scalability is that Python uses a global interpreter lock for thread safety. That global lock must be held by a thread before it can safely access Python objects. I thought I might be able to make embedded Python scalable by embedding multiple interpreters and have them run independently in different threads. However "Python/C API Reference Manual" chapter 8 says that "The global interpreter lock is also shared by all threads, regardless of to which interpreter they belong". Therefore with current implementation, even multiple interpreters do not provide scalability. Has anyone on this list run into the same problem that I have, or does anyone know of any plan of totally insulating multiple embedded Python interpreters? I've attached my test script to this message. I am using Python 2.2.3. <<mytest.py>>

2 1