Mailman 3 April 1998 - Cplusplus-sig

Re: [C++-SIG] Re: [PSA MEMBERS] Fw: the dynamic load mystery...
by John Skaller April 22, 1998

April 22, 1998

See Also, From News: ------- I have problems to get Python 1.5 to load a shared library under HP-UX. I have installed Python exactly as described in the HPUX-NOTES. The shared lib was made with SWIG. The problem is: Python tells me the library would not export the symbol initlibc, but nm tells me that is not true. ------- ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [C++-SIG] Re: [PSA MEMBERS] Fw: the dynamic load mystery...
by John Skaller April 21, 1998

April 21, 1998

At 16:23 17/04/98 +0200, Konrad Hinsen wrote: >No, but it's my problem. I am trying to convince my colleagues to use >Python (and my libraries), but among the people I know, more than 50% >have given up due to installation problems related to NumPy. That was >supposed to be fixed by the new version... Keep trying. Hard task. >BTW, what was the problem that caused you to ask for a change in the >behaviour under HP/UX? Not HP/UX particularly: I'm running Linux. >I understand that it's nicer to get error >messages immediately for testing, but was there any other problem? If >the 1.4 behaviour is fine for production use, maybe it is more >reasonable to use the 1.5 behaviour only in debugging mode. From memory there are _two_ switches: either immediate load (RTLD_NOW) or defered load (RTLD_LAZY) (that's _one_ switch :-), and also a flag to 'share' symbols, RTLD_GLOBAL. The sharing flag allows one dynamically loaded library to make its symbols available to another (I think). I think it was this flag that needed to be added. Not sure. :-( So you could be right, RTLD_LAZY could still be used. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [C++-SIG] Re: [PSA MEMBERS] Fw: the dynamic load mystery...
by John Skaller April 17, 1998

April 17, 1998

At 16:28 16/04/98 +0200, Konrad Hinsen wrote: >So why is umath not linked with _numpy? First of all, that doesn't >seem to work with some systems. I needs to work 'by definition', at least where 'link' is interpreted 'liberally' :-) >Second, everytging might be linked >statically with the Python interpreter, and in that case there >wouldn't be any dynamic library to link to. Third, the names of shared >libraries vary between systems. Yes. But I think that 'the fact that linkage is not portable' means that ... >The scheme I used has the enormous >advantage of allowing client modules to use NumPy in a perfectly >portable way, i.e. a single Setup file works for all systems and >all ways of installing NumPy. Well, almost :-( ... it isn't portable. :-) But it still has to be done. Somehow :-( >> 2) in general you must ensure _intentionally_ unresolved >> externals will be satisfied at dynamic load time >> by linking all dynamic libraries against dependent >> libraries proving these symbols > >That's the big problem - I don't see any way to do this in a portable >way. That's not a problem. You're probably right. So do it in a non-portable way :-) With lots of cases, and people emailing you what worked for them. :-(( Ugly. But it isn't your fault. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [C++-SIG] Re: [PSA MEMBERS] Fw: the dynamic load mystery...
by John Skaller April 16, 1998

April 16, 1998

At 12:45 2/04/98 +0200, Konrad Hinsen wrote: >> static application. In the static case, everything is fine. In the dynamic >> case, python 1.5 complains that the shared library does not not contain the >> module's init routine, named initexample. But, as you'll see below, it does. > >Sounds familiar. I have exactly that problem with plain C extensions, >e.g. NumPy, both with HP/UX 10 and HP/UX 9. And it works fine with Python 1.4. >But it doesn't occur with all extensions, and to make things worse >it depends on what other dynamic modules have been imported before. I didn't realise this problem was so widespread so I will repeat some of the private email I sent Paul. I hope it helps. I have had an _apparently_ similar problem which I have solved. I have a C extension '_pytcl' which sometimes failed to import during development. [I had a LOT of problems because pytcl itself loads Tcl and somewhat indirectly causes Tk and X libraries to be loaded too. (Tcl and Tk 8.0 are 'incorrectly' built in the distribution).] What happened to me was that I declared and called a function with external linkage but did NOT define it. When that happened under Python 1.4, I would eventually get an 'unresolved symbol' diagnostic. Under Pythnon 1.5, at my suggestion, dlopen now uses a flag supported by linux and some other Unices which causes immediate failure if a library contains an unresolved external reference (after attempted dynamic linkage). This failure causes the actual dlopen call to fail, 'as if' the module or it's entry point could not be found. Now, this problem is probably closely related to the problem Konrad reports -- a dependency on the order of 'import'. This is because Unix dynamic loading is by and large an utter hack, and is badly implemented and defined. (The Windows mechanism is much saner). What is happening is probably as follows: in Python 1.4, the dlopen flags are different to Python 1.5. in 1.5 they are RTLD_NOW and RTLD_GLOBAL. What that means is that (1) RTLD_GLOBAL this dl library will provide symbols for others (I do NOT understand the details, try as I might, it doesn't work at all sanely, and is, of course, in Unix tradition, not documented properly) (2) RTLD_NOW means 'give an error if the library cannot be fully bound' (all external references in it satisfied) What that means is that you can NOT simply use incremental linkage, loading a library depending on another, not yet loaded, and then load that other library. Instead you MUST link the library against the dependent library so it is loaded automatically by the loader (NOT by Python import which calls dlopen). For example: cc libB.c -o libB.so -rdynamic .... will FAIL where cc libB.c -lA -o libB.so ... ^^^ will work, where 'libA.so' is required by 'libB.so'. That is because libB.so will be loaded when libA.so is loaded, by the loader, a subsequent 'dlopen' on libA.so (caused by Python import) will simply bind to an already loaded library. As an example of this problem: Tcl and Tk are INCORRECTLY built in the standard distribution. in particular, Tk is NOT linked against libtcl.so or against libX. This works just fine for 'wish' because 'wish' IS linked against tcl and X, but just try tcl tcl>package require Tk and you will get an error, (hundreds of them -- one for each X windows function) because tcl (or the tcl shell tclsh) of course is NOT linked against X. [As I understand it this error in the distribution should NOT happen if RTLD_GLOBAL actually worked properly, which it doesn't seem to] The Tk distribution must be patched to explicitly link the Tk _library_ against tcl and X (not just that one application, 'wish'). A special note for C++ developers: you will have fun with template instantiation and dynamic linkage! In summary: 1) Unix dynamic loading is brain dead on many systems including Linux, the 'compile time linker' will not detect an unresolved external reference 2) in general you must ensure _intentionally_ unresolved externals will be satisfied at dynamic load time by linking all dynamic libraries against dependent libraries proving these symbols 3) EXCEPT that symbols provided by the main application are supplied automatically provided it is linked with the right flags (-rdynamic or -Xlinker, -export-dynamic) The 'dynamic export' mechanism does NOT work correctly (whatever that means) across three level dynamic loads, at least under Linux. Thus (2) -- explicitly link a library against dependent libraries. I would love to hear from someone who really understands how the dynamic linker tries to resolve external references incrementally. (Since it doesn't work :-( ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [C++-SIG] Fw: the dynamic load mystery...
by Paul F. Dubois April 7, 1998

April 7, 1998

I really appreciate your taking the time to help me. Indeed, we discovered the secret was to use KCC to load. I replaced the definition for LDSHARED and it worked. This is consistent with your explanation, too. Well, "worked" means "ran some and then croaked", but at least I got past the import, and at least it might even now be my own fault. Thank you to everyone who tried to help. -----Original Message----- From: John Skaller <skaller(a)maxtal.com.au> To: dubois1(a)llnl.gov <dubois1(a)llnl.gov> Date: Tuesday, April 07, 1998 5:53 AM Subject: Re: [C++-SIG] Fw: the dynamic load mystery... >At 15:44 1/04/98 -0800, Paul F. Dubois wrote: >>I've been trying to get my recent C++ work to work on an HPUX-10 system. It >>compiles and makes a dynamic library just fine; or you can build it into a >>static application. In the static case, everything is fine. In the dynamic >>case, python 1.5 complains that the shared library does not not contain the >>module's init routine, named initexample. But, as you'll see below, it does. >> >>In an effort to diagnose this, we took the spammodule.c example, changed >>spammodule.c to spammodule.cxx, adding the >>external "C" statement for initspam, and made it on an HP using KCC -x and >>cc -Ae. Works fine: import spam succeeds. >> >>Then we changed the name spaminit everywhere to exampleinit, and renamed the >>file example.cxx. Works fine: import example succeeds. >> >>Copied this example.cxx into the CXX/Demo directory and rebuilt there. >>Works fine. >> >>Added one line to example.cxx: #include "CXX_Objects.h". >>Now we get the failing behavior. nm reveals >> >>laura[63] nm *.sl | grep initexample >>00032d08 T __sti____Demo_example_cxx_initexample >>00032cf0 T __sti____Demo_example_cxx_initexample >>000316f4 T initexample >>00031694 T initexample >> >>Demo/example.cxx is the file name in which initexample resides. I wonder if >>this extra initexample inside a name is confusing some part of the dynamic >>load. >> >>Does any of this ring a bell with anyone? > >I have no idea if this helps .. but I had a problem which I >_thought_ was like your description. but it turned out to be >something quite different: an unsatisfied external reference >in the dynamic library. > > >[I'm pisssed at the STUPID way Unices/compilers/linkers/loaders handle this, >it's an utter hack. Windows has saner system. :-[ > >Since you are using templates, it is possible the dynamic link library >indeed contains an external reference to an uninstantiated template >instance. > >Check the output of 'nm' carefully for unexpected undefined references. > >My current C extension to Python (PyTcl) exhibits this problem: >if I declare and call, but forget to define, a function (all my >interface functions have external linkage), the resultant library >is built fine but cannot be loaded. Because the load uses a flag >forcing immediate resolution of undefined references in the >library, failure is immediate (rather than at the point of call). > >Thus, the python 'import' fails with a WRONG message 'can't find >init_pytcl' -- the _actual_ error is different. > >With slightly different options on the build of the library >and/or Python (forget which) I _used_ to get the actual error, >namely 'can't resolve symbol XXXX'. >------------------------------------------------------- >John Skaller email: skaller(a)maxtal.com.au > http://www.maxtal.com.au/~skaller > phone: 61-2-96600850 > snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia > > >

1 0

CXX-a4 available at ftp-icf.llnl.gov/pub/python
by Paul F. Dubois April 3, 1998

April 3, 1998

I have started coordinating with Geoff Furnish, who is hard at work on the trampoline. Geoff and I have made a number of naming changes as follows: Former macros Py_Null and Nothing are now inline functions Null() and Nothing. This eliminates all macros so that the namespaces can do their job properly. Name change edits: CXX -> Py PyException_ -> // delete PyException -> Exception Py_Null -> Null () Nothing -> Nothing () Signature change: Exception::clear() is now not const. Technically, it doesn't alter the Exception object, but morally, it does. Thus to catch an exception you may wish to clear, catch it as catch (PyException& e) { ...; e.clear(); ...} For one you don't intend to clear, catch it as catch (const PyException&) {...}

1 0

CXX-a3
by Paul F. Dubois April 2, 1998

April 2, 1998

CXX-a3.zip and .exe are at ftp-icf.llnl.gov/pub/python. Changed namespace name to Py to match work being done by Furnish. Changed PythonExtension<T>::methods() to api() and added a new methods() to hold a method table for additional, non-api methods. Added new methods to the Demo/ "r" object and R class, and excercised the new facilities in rtest.cxx. Misc. small improvements.

1 0

Re: [C++-SIG] Fw: the dynamic load mystery...
by Neal Becker April 2, 1998

April 2, 1998

Paul> laura[63] nm *.sl | grep initexample Paul> 00032d08 T __sti____Demo_example_cxx_initexample Paul> 00032cf0 T __sti____Demo_example_cxx_initexample Paul> 000316f4 T initexample Paul> 00031694 T initexample I very much doubt it. Look at any hpux .sl file. They all look like that.

1 0

Fw: the dynamic load mystery...
by Paul F. Dubois April 2, 1998

April 2, 1998

I've been trying to get my recent C++ work to work on an HPUX-10 system. It compiles and makes a dynamic library just fine; or you can build it into a static application. In the static case, everything is fine. In the dynamic case, python 1.5 complains that the shared library does not not contain the module's init routine, named initexample. But, as you'll see below, it does. In an effort to diagnose this, we took the spammodule.c example, changed spammodule.c to spammodule.cxx, adding the external "C" statement for initspam, and made it on an HP using KCC -x and cc -Ae. Works fine: import spam succeeds. Then we changed the name spaminit everywhere to exampleinit, and renamed the file example.cxx. Works fine: import example succeeds. Copied this example.cxx into the CXX/Demo directory and rebuilt there. Works fine. Added one line to example.cxx: #include "CXX_Objects.h". Now we get the failing behavior. nm reveals laura[63] nm *.sl | grep initexample 00032d08 T __sti____Demo_example_cxx_initexample 00032cf0 T __sti____Demo_example_cxx_initexample 000316f4 T initexample 00031694 T initexample Demo/example.cxx is the file name in which initexample resides. I wonder if this extra initexample inside a name is confusing some part of the dynamic load. Does any of this ring a bell with anyone?

2 1