Multiple typeinfos causing failure of exception catches
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've been having major problems with getting exceptions to traverse shared object boundaries with Boost.Python using GCC and Linux. There is a core SO called libTnFOX-0.80.so which is linked against by TnFOX.so, the BPL extension and these are linked into TestEmbedPython which is an executable. Now when python has an exception, I have it translate it into a C++ exception which it throws. In theory, TnFOX.so should throw a FXPythonException instance which in theory gets caught in TestEmbedPython. All is good under Win32, but on GCC & Linux you get "uncaught exception" and a terminate(). I have examined the dynamic symbol table in each binary and a typeinfo for FXPythonException appears in TnFOX.so and TestEmbedPython. I did some searching and found: http://gcc.gnu.org/ml/gcc/2002-05/msg00995.html This problem reported by Dave in 2002 is due to a typeinfo for the exception type being emitted in each object it is used within. As they are marked weak, they get coalesced into a single typeinfo *per* *shared* *object* but these happily clash when multiple shared objects are loaded. This then causes typeinfo comparisons to fail not because the mangled symbol is different but because the addresses of each are different. Dave suggested a typeinfo comparison by string and AFAICS it was refused on grounds of speed (despite that comparing hashes is the same speed as comparing addresses). Dave - was this problem ever fixed? If not, I will submit a patch implementing typeinfo comparison by string comparison and we'll swing round the roundabout once again (maybe two years of experience has changed opinions). BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQO3srsEcvDLFGKbPEQK7agCgsh+QzdJcz3DfHmnl9GatHlfkKY4AoMop ymYKrY1os+JdgM5h7EG7BE/a =N651 -----END PGP SIGNATURE-----
--- Niall Douglas <s_sourceforge@nedprod.com> wrote:
BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen.
Hi Niall, I don't use embedding, so I am not sure this is relevant, but ever since I started importing all extension modules with RTLD_GLOBAL under Linux I've not observed exception handling problems anymore. I.e. I am doing the exact opposite of what I understand you are suggesting above. Here is my trick, the "import_ext" function: http://cvs.sourceforge.net/viewcvs.py/cctbx/boost_adaptbx/boost/python.py?vi... Does it make sense to you that this approach seems to fix the EH problems? (Or am I just lucky? Hope not.) Ralf __________________________________ Do you Yahoo!? Yahoo! Mail - Helps protect you from nasty viruses. http://promotions.yahoo.com/new_mail
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 8 Jul 2004 at 21:24, Ralf W. Grosse-Kunstleve wrote:
--- Niall Douglas <s_sourceforge@nedprod.com> wrote:
BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen.
Hi Niall,
I don't use embedding, so I am not sure this is relevant, but ever since I started importing all extension modules with RTLD_GLOBAL under Linux I've not observed exception handling problems anymore. I.e. I am doing the exact opposite of what I understand you are suggesting above. Here is my trick, the "import_ext" function:
Just to clarify, Dave in his original post said that Python needed to use RTLD_LOCAL to prevent symbols in multiple extension modules from clashing. This however comes with many downsides, including breaking static variables and loads of other bad things. I was suggesting that if you use -fvisibility=hidden then no symbols are exported from any extension modules at all (apart from the minimum which you will ensure are unique) so Python can use RTLD_GLOBAL again which solves many other problems.
http://cvs.sourceforge.net/viewcvs.py/cctbx/boost_adaptbx/boost/python .py?view=markup
Does it make sense to you that this approach seems to fix the EH problems? (Or am I just lucky? Hope not.)
I assume 0x100|0x2 means RTLD_GLOBAL. Well, since python extension modules have no reason to throw a C++ exception into python you're probably fairly safe. The issues begin when ld.so loads a so due to dependency requirements of a previous so which then causes the merging of symbols to come out with different addresses so typeinfo comparisons magically fail. With a bit of work, the shared library system on Linux can be made to play nice with C++. I'm kinda surprised it wasn't done already as I'd have thought people would be clamouring for it but if they didn't before, now GCC 3.4 is so standards compliant they surely will in the future. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQO7iQsEcvDLFGKbPEQIElwCfcZJ9Q25bwj7kt79P/oxoWo0MBTwAoLZA VOW8RoeTT1AWhMOgxB909H5A =BX0l -----END PGP SIGNATURE-----
--- Niall Douglas <s_sourceforge@nedprod.com> wrote:
http://cvs.sourceforge.net/viewcvs.py/cctbx/boost_adaptbx/boost/python .py?view=markup
Does it make sense to you that this approach seems to fix the EH problems? (Or am I just lucky? Hope not.)
I assume 0x100|0x2 means RTLD_GLOBAL.
I think it is RTLD_GLOBAL | RTLD_NOW. Ralf __________________________________ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail
"Niall Douglas" <s_sourceforge@nedprod.com> writes:
I've been having major problems with getting exceptions to traverse shared object boundaries with Boost.Python using GCC and Linux. There is a core SO called libTnFOX-0.80.so which is linked against by TnFOX.so, the BPL extension and these are linked into TestEmbedPython which is an executable.
Now when python has an exception, I have it translate it into a C++ exception which it throws. In theory, TnFOX.so should throw a FXPythonException instance which in theory gets caught in TestEmbedPython. All is good under Win32, but on GCC & Linux you get "uncaught exception" and a terminate(). I have examined the dynamic symbol table in each binary and a typeinfo for FXPythonException appears in TnFOX.so and TestEmbedPython.
I did some searching and found:
http://gcc.gnu.org/ml/gcc/2002-05/msg00995.html
This problem reported by Dave in 2002 is due to a typeinfo for the exception type being emitted in each object it is used within. As they are marked weak, they get coalesced into a single typeinfo *per* *shared* *object* but these happily clash when multiple shared objects are loaded.
This then causes typeinfo comparisons to fail not because the mangled symbol is different but because the addresses of each are different. Dave suggested a typeinfo comparison by string and AFAICS it was refused on grounds of speed (despite that comparing hashes is the same speed as comparing addresses).
Dave - was this problem ever fixed?
Not AFAIK.
If not, I will submit a patch implementing typeinfo comparison by string comparison and we'll swing round the roundabout once again (maybe two years of experience has changed opinions).
There's a better fix that involves dealing differently with the way the ELF loader resolves weak symbols. You might ask Jason Merrill (cc:'d here) about the idea - it was his.
BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen.
I don't think so; you can still have symbol clashes if two extension modules explicitly export the same names. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9 Jul 2004 at 8:36, David Abrahams wrote:
There's a better fix that involves dealing differently with the way the ELF loader resolves weak symbols. You might ask Jason Merrill (cc:'d here) about the idea - it was his.
I'd imagine he'll get back to me on Monday. Was it to do with getting the Linux ld.so to behave more like the Solaris one?
BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen.
I don't think so; you can still have symbol clashes if two extension modules explicitly export the same names.
True, but then why would they need exporting in a python extension? AFAICS most extensions export a table to python of what they provide plus a few init and deinit functions which I would have thought are looked up via dlsym() and thus unaffected by clashes? I suppose if an extension was written using BPL and the extension could also be dual used as an embedded library then you might have issues. But you're still way ahead of how things stand now. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQO7jhcEcvDLFGKbPEQJJzgCcDgN7UtRFkU+hGKtfaCtrRE3weRIAn2yI ICCNejU7m1Q/OgEb0zrmL4rV =/gu4 -----END PGP SIGNATURE-----
"Niall Douglas" <s_sourceforge@nedprod.com> writes:
On 9 Jul 2004 at 8:36, David Abrahams wrote:
There's a better fix that involves dealing differently with the way the ELF loader resolves weak symbols. You might ask Jason Merrill (cc:'d here) about the idea - it was his.
I'd imagine he'll get back to me on Monday. Was it to do with getting the Linux ld.so to behave more like the Solaris one?
I know the solaris loader came up in the conversation, but I think Jason had a very specific idea that isn't implemented anywhere yet.
BTW, your problem with python having to load extension modules with RTDL_LOCAL rather than RTDL_GLOBAL goes away if you use my - fvisibility=hidden patch as no symbol clashes can happen.
I don't think so; you can still have symbol clashes if two extension modules explicitly export the same names.
True, but then why would they need exporting in a python extension?
Because they're not self-contained; perhaps they need to interact with other shared libs.
AFAICS most extensions export a table to python of what they provide plus a few init and deinit functions which I would have thought are looked up via dlsym() and thus unaffected by clashes?
Right.
I suppose if an extension was written using BPL and the extension could also be dual used as an embedded library then you might have issues. But you're still way ahead of how things stand now.
Agreed. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 A little more on this: [ned@katey2 lib]$ ./TestEmbedPython TnFOX Python test: - -=-=-=-=-=-=-=-=-Working with Python 2.3 (#8, Jul 10 2004, 11:28:33) [GCC 3.4.0] If you can read this, python prints! Typeinfo of FXException in DLL is 0x804cba0,0x41f65004 (N2FX11FXExceptionE) Typeinfo of FXPythonException in DLL is 0x804a9d8,0x804a9e4 (N2FX17FXPythonExceptionE) Throwing a python exception ... Typeinfo of caught exception in DLL is 0x804a9d8,0x804a9e4 (N2FX17FXPythonExceptionE) C++ exception caught: exceptions.IOError,A parameter to a python exception at line 2 in <string> (code 0xFFDE) Typeinfo of pointer to be returned from DLL is 0x804a9d8,0x804a9e4 (N2FX17FXPythonExceptionE) Typeinfo of pointer returned from DLL is 0x804a9d8,0x804a9e4 (N2FX17FXPythonExceptionE) Typeinfo of FXException in EXE is 0x804cba0,0x41f65004 (N2FX11FXExceptionE) Typeinfo of FXPythonException in EXE is 0x804a9d8,0x804a9e4 (N2FX17FXPythonExceptionE) Throwing a python exception ... terminate called after throwing an instance of 'FX::FXPythonException' Deleted secure heap Unhandled Signal 0x6 (Abnormal termination) at address 0x5979 - immediate exit! What I've done here is copy the throw/catch code into the bindings shared object. As we can see, the first throw/catch works fine and the second dies with terminate(). Both sets of code are identical - just in the second the throw is in the bindings DLL and the catch is in the EXE. I have it printing the address of the typeinfo, the typeinfo.name() address and then the string. I was surprised to see that in both the DLL and EXE all addresses are the same, so therefore typeinfo comparison by address should be working. Therefore it would seem that this bug is not the same as Dave's original one. I'll investigate GCC's source code next, maybe get it printing what the hell it's up to when it's catching exceptions. I've already looked through the assembler output from GCC for each of the source files and I'm fairly confident it's not that - also, a mockup of the same DLL & EXE configuration works fine so I think it's the linker, perhaps because it's linking 120Mb of non-debug object files with nearly 160,000 symbols. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQO/bb8EcvDLFGKbPEQLXpACgqwP/hgNoh0+eXgmDVBYpuVt4sugAniGe CWzSIkAX1gcgqXTG0jlH4Mun =wGDg -----END PGP SIGNATURE-----
participants (3)
-
David Abrahams -
Niall Douglas -
Ralf W. Grosse-Kunstleve