Mailman 3 June 2004 - Cplusplus-sig

importing embedded dynamic lib
by Vio June 10, 2004

June 10, 2004

Perhaps I didn't explain myself correctly. My question would come down to: *what API call* does the python interpreter use when I type at the python prompt: >>> import mysharedlib where "mysharedlib" is file "mysharedlib.so" somewhere on the PYTHONPATH. What I wish is to use *that* call, but change arguments as to substitute to the [my guess] file pointer for a memory buffer pointer. The memory buffer having the exact contents of file "mysharedlib.so", thus avoiding a disk access. While the result here is not very sexy, this exercise becomes interesting when the app receives "mysharedlib.so" over the network. Once received over network, easy solution is to write "mysharedlib.so" to local disk, then either do something like: PyRun_SimpleString("import mysharedlib") or PyImport_ImportModule("mysharedlib") I simply want to skip that. In still other words, I guess I'm looking for something like PyImport_ExecCodeModule(char *name, PyObject *co) only I'd have to construct a PyObject out of my buffer, since that's what PyImport_ExecCodeModule() wants as 2nd argument. But API docs say this 2nd argument should be python bytecode data, so I don't know if it won't choke on compiled ELF binaries instead, hence I'm not too sure about this particular hack. Anyway, my gut guess is that I must find what calls python uses to import shared libs, and somehow use that and substitute that file pointer to my buffer pointer. Just a wild guess. If someone has some ideas on this, I'd appreciate it. Vio

2 1

importing embedded dynamic lib
by Vio June 10, 2004

June 10, 2004

Many thanks for your solution Graeme. Unfortunately, I do need to use ".so" files as I want to "extend dynamically" an embedded python interpreter, skipping recompile each time. Yes, I've tried both Boost and SWIG to generate wrappers, but right now I like to do things "manually" to understand what's going on "under the hood". More details: Imagine a situation where the ".so" files are delivered through a network; the easy solution would be to save the downloaded binary stream as a ".so" file to disk, then simply run the python command "import <modulename>" and I'm done. But I simply wish to avoid the "save to disk" step which seems superfluous in this case, and feed the binary lib directly to the embedded python interpreter. Exploring the API docs, it would seem that something like PyRun_AnyFile(FILE *fp, char *filename) could do what I want. But I don't know if/how I can give it a memory buffer instead of a FILE pointer, and also I'm too much of a c/c++ newbie right now to figure out how to handle huge binary data (as I write this, I'm trying to make sense of Stroustrup's C++ bible "the c++ prog. lang, ed. 2000"), so a simple task like putting contents of a ".so" file into a memory buffer is a big challenge for me right now. Anyway, thanks for your help, Vio > I've done this myself; let me try and explain what I did. First, write >the extension module as normal. I used Boost.Python, but what's >important is having access to the 'extern "C" void >init<modulenamehere>()' function pointer. >Second, instead of compiling your module into its own shared library >(.so file), simply link it in with your executable. Using the >Boost.Build system, you can just add the .cpp files that make up your >module to the dependency list of the executable. >Third, before you create the embedded interpreter, make a call to >PyImport_AppendInittab(). For example, if my module is called funcs, >the init function is declared as 'extern "C" void initfuncs();', so I >call: >PyImport_AppendInittab("funcs", initfuncs); >then I call Py_Initialize(). >Fourth, finally I can actually import the module into my embedded >interpreter: >PyRun_SimpleString("import funcs"); > > So, if you follow my steps, you have only one binary file, the >executable. This file contains code for the module, as well as the >regular C++ stuff of your program. You embed the Python interpreter, >and import your module into its Python namespace. > The embedding part of the Boost.Python tutorial is a bit lacking, and >some of the code snippets won't compile, but give it a try anyway, as >there's no other option ;) > Hope this helps > >>/ I have a basic dynamic lib "noddy.so" which I want to >/>/ 1- embed inside my executable >/>/ 2- import by embedded python interpreter using an API call. >/>/ >/>/ I have two questions: >/>/ >/>/ 1- what would be the recommended technique to embed a binary file >/>/ inside >/>/ an executable? >/>/ My guess would be to add the binary "noddy.so" to the linking command, >/>/ but then >/>/ how do I get access to that chunk of code in my source file? My goal >/>/ would be to achieve something like >/>/ >/>/ int* noddy_so = <binary here> ... but then how do I include the >/>/ binary code, or make the source code aware of it? >/>/ >/>/ Using "#include" doesn't feel right, as this is a binary file, not >/>/ ASCII. And loading the file at run-time isn't right either, I need to >/>/ load it at compile time. And is "int*" the appropriate data type for >/>/ binary data? >/>/ >/>/ 2- Let's imagine that I managed to load "noddy.so" into my source code >/>/ somehow, and now have a pointer to it. >/>/ Any hints as to the correct python API call to import a dynamic lib >/>/ from a memory buffer? Could someone >/>/ tell me for example if the following makes sense for my purpose? >/>/ >/>/ ------------------------------ >/>/ char *modulename="noddy", *filename="no_real_file"; >/>/ PyObject *tmp; >/>/ >/>/ /*Here try to translate the binary lib into a PyObject*/ >/>/ tmp = Py_CompileString(noddy_so, filename, Py_file_input); >/>/ if (tmp == NULL) >/>/ error("Can't compile module [tmp]"); >/>/ >/>/ /*Import the "noddy" module*/ >/>/ pmod = PyImport_ExecCodeModule(modulename, tmp); >/>/ if (pmod == NULL) >/>/ error("Can't load module [pmod]"); >/>/ ------------------------------ >/ >

3 2

importing embedded dynamic lib
by Vio June 10, 2004

June 10, 2004

followup... I think what I'm looking for is "PyCObject_Import()", which I found in wxPython.h ------------------------ // This needs to be called in modules that make calls to any of the functions // exported by the wxPython API. It sets a static pointer to a structure of // function pointers located in the wx._core extension module. static void wxPyCoreAPI_IMPORT() { wxPyCoreAPIPtr = (wxPyCoreAPI*)PyCObject_Import("wx._core", "_wxPyCoreAPI"); if (! wxPyCoreAPIPtr) wxPyCoreAPIPtr = (wxPyCoreAPI*)PyCObject_Import("_core", "_wxPyCoreAPI"); } ------------------------ wx._core referred here is file "_core.so". Unfortunately, PyCObject_Import() does not seem to be documented in the Python API docs. More digging required... Vio

1 0

RE: [C++-sig] warning: "_POSIX_C_SOURCE" redefined
by Pere Mato Vila June 9, 2004

June 9, 2004

My test program to show the problem contains only one line: #include "boost/python.hpp" and therefore I can not move it before anything. Pere > -----Original Message----- > From: c++-sig-bounces(a)python.org > [mailto:c++-sig-bounces@python.org] On Behalf Of Ralf W. > Grosse-Kunstleve > Sent: 08 June 2004 12:26 > To: Development of Python/C++ integration > Subject: Re: [C++-sig] warning: "_POSIX_C_SOURCE" redefined > > > --- Pere Mato Vila <Pere.Mato(a)cern.ch> wrote: > > I am having the following compilation warnings when including > > "boost/python.hpp". I am using Boost 1.31.0 Python 2.3.3 > and gcc 3.2. > > Is anything I am doing wrong? > > Due to a Python requirement the boost/python.hpp header has > to be included before any system header. Please move the > python.hpp include to the very top of your file. Let us know > if it doesn't help. Ralf > > > > > > __________________________________ > Do you Yahoo!? > Friends. Fun. Try the all-new Yahoo! Messenger. > http://messenger.yahoo.com/ > > _______________________________________________ > C++-sig mailing list > C++-sig(a)python.org > http://mail.python.org/mailman/listinfo/c++-sig >

2 1

Re: [C++-sig] importing embedded dynamic lib
by Graeme Lufkin June 9, 2004

June 9, 2004

I've done this myself; let me try and explain what I did. First, write the extension module as normal. I used Boost.Python, but what's important is having access to the 'extern "C" void init<modulenamehere>()' function pointer. Second, instead of compiling your module into its own shared library (.so file), simply link it in with your executable. Using the Boost.Build system, you can just add the .cpp files that make up your module to the dependency list of the executable. Third, before you create the embedded interpreter, make a call to PyImport_AppendInittab(). For example, if my module is called funcs, the init function is declared as 'extern "C" void initfuncs();', so I call: PyImport_AppendInittab("funcs", initfuncs); then I call Py_Initialize(). Fourth, finally I can actually import the module into my embedded interpreter: PyRun_SimpleString("import funcs"); So, if you follow my steps, you have only one binary file, the executable. This file contains code for the module, as well as the regular C++ stuff of your program. You embed the Python interpreter, and import your module into its Python namespace. The embedding part of the Boost.Python tutorial is a bit lacking, and some of the code snippets won't compile, but give it a try anyway, as there's no other option ;) Hope this helps > I have a basic dynamic lib "noddy.so" which I want to > 1- embed inside my executable > 2- import by embedded python interpreter using an API call. > > I have two questions: > > 1- what would be the recommended technique to embed a binary file > inside > an executable? > My guess would be to add the binary "noddy.so" to the linking command, > but then > how do I get access to that chunk of code in my source file? My goal > would be to achieve something like > > int* noddy_so = <binary here> ... but then how do I include the > binary code, or make the source code aware of it? > > Using "#include" doesn't feel right, as this is a binary file, not > ASCII. And loading the file at run-time isn't right either, I need to > load it at compile time. And is "int*" the appropriate data type for > binary data? > > 2- Let's imagine that I managed to load "noddy.so" into my source code > somehow, and now have a pointer to it. > Any hints as to the correct python API call to import a dynamic lib > from a memory buffer? Could someone > tell me for example if the following makes sense for my purpose? > > ------------------------------ > char *modulename="noddy", *filename="no_real_file"; > PyObject *tmp; > > /*Here try to translate the binary lib into a PyObject*/ > tmp = Py_CompileString(noddy_so, filename, Py_file_input); > if (tmp == NULL) > error("Can't compile module [tmp]"); > > /*Import the "noddy" module*/ > pmod = PyImport_ExecCodeModule(modulename, tmp); > if (pmod == NULL) > error("Can't load module [pmod]"); > ------------------------------ > > Regards, > Vio -- - Graeme gwl(a)u.washington.edu "This sentence contains exactly threee erors."

1 0

Survey of IDE users and source code editors
by aherrman＠mi.fu-berlin.de June 9, 2004

June 9, 2004

As part of a University study of Integrated Development Environment (IDE) users, we are conducting a survey to appraise the relative utility of different features of IDE source code editors, such as those included with Visual Studio and Eclipse as well as stand-alone editors like Emacs that support programming tasks. Our survey is designed to assess how often users employ editor features like syntax highlighting and code completion, and how satisfied users are with these features in their particular environments. If you or your project group have active experience using an IDE, we would like you to participate in our study. You can find the survey at http://physik.fu-berlin.de/~zabel/ide_editor_survey/ The survey should take no more than 20 minutes to complete. No personal information is required of participants. The survey period is two weeks (until June 22nd, 2004). Participants who opt to enter their e-mail addresses on the survey page will receive a copy of the findings of our study after the survey period. We look forward to hearing from you. Philipp Zabel, Minor Gordon and Antoinette Herrmann Freie Universitaet Berlin Institut fuer Informatik Takustr. 9 14195 Berlin

1 0

importing embedded dynamic lib
by Vio June 9, 2004

June 9, 2004

I have a basic dynamic lib "noddy.so" which I want to 1- embed inside my executable 2- import by embedded python interpreter using an API call. I have two questions: 1- what would be the recommended technique to embed a binary file inside an executable? My guess would be to add the binary "noddy.so" to the linking command, but then how do I get access to that chunk of code in my source file? My goal would be to achieve something like int* noddy_so = <binary here> ... but then how do I include the binary code, or make the source code aware of it? Using "#include" doesn't feel right, as this is a binary file, not ASCII. And loading the file at run-time isn't right either, I need to load it at compile time. And is "int*" the appropriate data type for binary data? 2- Let's imagine that I managed to load "noddy.so" into my source code somehow, and now have a pointer to it. Any hints as to the correct python API call to import a dynamic lib from a memory buffer? Could someone tell me for example if the following makes sense for my purpose? ------------------------------ char *modulename="noddy", *filename="no_real_file"; PyObject *tmp; /*Here try to translate the binary lib into a PyObject*/ tmp = Py_CompileString(noddy_so, filename, Py_file_input); if (tmp == NULL) error("Can't compile module [tmp]"); /*Import the "noddy" module*/ pmod = PyImport_ExecCodeModule(modulename, tmp); if (pmod == NULL) error("Can't load module [pmod]"); ------------------------------ Regards, Vio

1 0

Pyste exception handlers
by paul.bridger June 9, 2004

June 9, 2004

Nicodemus, How would you recommend exception handler registration be implemented in Pyste? A problem here would be what to do in the differing cases of: - building a single pyste file into a module : in this case specifying exception handlers in the pyste file is sufficient - building many pyste files into a single module : in this case you probably want exception handlers to be specified in a module-level file, or perhaps you could put them in an arbitrary pyste file I think Pyste has some deficiencies in organising modules from multiple files. It is wonderful that this can be done, but I don't like that so much has to be specified on the command-line (and that the meaning of command-line options changes depending on --multiple or not). I think a better organisation would involve a module-level file, which could then contain such things as exception handlers. This would also make building modules out of several pyste files cleaner. (A disadvantage is that it expands the scope of the tool into something of a build tool). What do others think? Paul Bridger

2 2

RE: [C++-sig] release compilation with bjam
by Shukla, Nitin (Export) June 8, 2004

June 8, 2004

You should first build the Boost library in the release mode. To do that you should specify this command line argument "-sBUILD=release" while build Boost. Again when building you own project specify this command line argument to bjam and it would build the release version of your project. Make sure that in the jamfile you specify the <library-path> to the release version of Boost.Python dll. HTH Nitin -----Original Message----- From: Olivier Ravard [mailto:olivier.ravard@novagrid.com] Sent: Tuesday, June 08, 2004 8:26 AM To: Development of Python/C++ integration Subject: [C++-sig] release compilation with bjam Hi, I compiled boost.python under windows with mingw. I test some projects and it works fine. I build my own project with boost.build and it works but the compilation used only the debug release (linking). How should I do to compile with optimization and make a release in a non debug mode ? Thanks. O.R.

3 2

Forward References
by Jeff Holle June 8, 2004

June 8, 2004

I want to express this issue to pyste users. With a class that uses forward references to express things like bi-directional assocition, it is necessary to add line in the associated pyste configure file like: declaration_code('#include "MyClass.h"\n') If this isn't done, the compiler errors, with at least gcc, aren't very helpful.

1 0