I think bundling the sources is fine, but it should *ONLY* be a fallback if you do not find the Expat library installed on the system. *ALWAYS* link against a system-installed library first.
I'm not so sure about that. There are many problems doing that: - if there is no shared libexpat, a static one is only usable if it is compiled either as PIC, or if pyexpat is statically linked into the Python interpreter. These conditions are impossible to test. - if the installed version of expat is older than the one we ship, I think Python should still use the included one. I think the problem you had with Apache does not apply to Python:
Apache 1.3 would get loaded and export the Expat symbols to the rest of the process.
In Python, symbols of one extension module are not exported to anything else. We would not build a libexpat.so (not sure whether Apache did), and nobody would link with pyexpat.so. Furthermore, dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden by the application (and you really have to know what you're doing when changing the dlopen flags).
Now, some *other* module is loaded and dynamically links against /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and crashes are going to start happening.
No, that can't happen. pyexpat.so does not export its symbols (due to RTLD_LOCAL), so the other module will still use libexpat.so. Regards, Martin
On Mon, Oct 08, 2001 at 10:03:54AM +0200, Martin v. Loewis wrote:
I think bundling the sources is fine, but it should *ONLY* be a fallback if you do not find the Expat library installed on the system. *ALWAYS* link against a system-installed library first.
I'm not so sure about that. There are many problems doing that:
- if there is no shared libexpat, a static one is only usable if it is compiled either as PIC, or if pyexpat is statically linked into the Python interpreter. These conditions are impossible to test.
What? I've never seen a .a installed on a system that isn't usable. What makes Python modules so special that they could not use a library? I find your statement rather surprising, and I'm quite doubtful of the issue. I've linked .a files into .so modules often. Never a problem.
- if the installed version of expat is older than the one we ship, I think Python should still use the included one.
Every time you use the included one, you run into the possibility of symbol conflict problems. If you refuse to use the one installed in the system, then you increase the chance of problems. Granted: there isn't much you can do if pyexpat *requires* something newer than what is on the system (does it require the latest? or can pyexpat work with all versions of expat?).
I think the problem you had with Apache does not apply to Python:
Apache 1.3 would get loaded and export the Expat symbols to the rest of the process.
In Python, symbols of one extension module are not exported to anything else.
Eh? Again: what makes Python so special? How can you guarantee this rule? On *all* platforms?
We would not build a libexpat.so (not sure whether Apache did),
We never built a .so. The library was linked right into the Apache httpd executable.
and nobody would link with pyexpat.so. Furthermore, dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden by the application (and you really have to know what you're doing when changing the dlopen flags).
This argument doesn't hold. dlopen() is not used on every platform. Thus, you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee that symbols in a library will remain private.
Now, some *other* module is loaded and dynamically links against /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and crashes are going to start happening.
No, that can't happen. pyexpat.so does not export its symbols (due to RTLD_LOCAL), so the other module will still use libexpat.so.
See above. Incorporating Expat directly into Python will definitely be "nice", but I've been through this already. It isn't pretty. If that shared library exists, then use the darned thing, or you're gonna get burned. Cheers, -g -- Greg Stein, http://www.lyra.org/
What? I've never seen a .a installed on a system that isn't usable. What makes Python modules so special that they could not use a library? I find your statement rather surprising, and I'm quite doubtful of the issue.
I've linked .a files into .so modules often. Never a problem.
It happens on Solaris with gcc as the compiler and /usr/ccs/bin/ld as the linker. Watch this pandora loewis 39 ( ~/tmp ) > cat a.c int i; int f(){return i;} pandora loewis 40 ( ~/tmp ) > gcc -c a.c pandora loewis 41 ( ~/tmp ) > gcc -o a.so -shared a.o Text relocation remains referenced against symbol offset in file i 0x4 a.o i 0x8 a.o ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status The problem here is that gcc passes -z text to the linker, unless -mimpure-text is specified. ld(1) then complains if the code was not compiled with -fPIC/-KPIC, i.e. that relocations remain. Python is not special at all in this respect; that happens every time when you integrate a non-PIC static library (or object file) into a shared library, and - the system is Solaris (or probably any SysV-derived system) - the linker is the system linker - the compiler passes -z text to the linker (i.e. it is gcc) If the linker is binutils ld, it won't happen since the linker ignores -z text. If the compiler is SUNWspro, it won't happen because cc does not pass -z text by default. On a Linux system, it won't happen because relocations against the text segment are not considered a problem in the first place - COW will take care of that. Of course, relocations against the text segment *are* a problem: they cause the shared library not to be shared at run-time. I don't know whether the problem is specific to Python; perhaps we deal with more varying configurations than anybody else. Please see the bug reports #232783, #404322, and #405329 for prior reports of this issue.
- if the installed version of expat is older than the one we ship, I think Python should still use the included one.
Every time you use the included one, you run into the possibility of symbol conflict problems. If you refuse to use the one installed in the system, then you increase the chance of problems.
How would such a conflict occur using the current mechanisms for dynamic loading in Python?
I think the problem you had with Apache does not apply to Python:
Apache 1.3 would get loaded and export the Expat symbols to the rest of the process.
In Python, symbols of one extension module are not exported to anything else.
Eh? Again: what makes Python so special? How can you guarantee this rule? On *all* platforms?
Unlike everybody else, Python does *not* use RTLD_GLOBAL in the flags passed to dlopen. This alone is a guarantee that shared libraries won't interfere with each other: Optionally, RTLD_GLOBAL may be or'ed with flag, in which case the external symbols defined in the library will be made available to subsequently loaded libraries. [from the Linux dlopen man page] It will work on all systems using dynload_shlib, plus it will work on Windows (where you have to explicitly specify the symbols to export and the libraries you import symbols from). Please also note the Misc/HISTORY item - Revert a new feature in Unix dynamic loading: for one or two revisions, modules were loaded using the RTLD_GLOBAL flag. It turned out to be a bad idea. The specific problem was that somebody was using a library that exposed an initsocket function, which conflicted with the init function of the socket module. So we cannot do much for systems that cannot protect independent shared libraries from each other.
This argument doesn't hold. dlopen() is not used on every platform. Thus, you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee that symbols in a library will remain private.
Systems that do not support local symbols in shared libraries have bigger problems than competing libexpat implementations. BTW, which of the systems in today's use do not have dlopen, and aren't Windows? [How does MacOS X operate in this area?]
Incorporating Expat directly into Python will definitely be "nice", but I've been through this already. It isn't pretty. If that shared library exists, then use the darned thing, or you're gonna get burned.
OTOH, I'm not looking forward to bug reports which turn out problems with obsolete libexpat installations. You have to judge the risks. Any user who is concerned about using the installed libexpat can still modify Modules/Setup to use that instead of the one coming with Python. Regards, Martin
participants (2)
-
Greg Stein -
Martin v. Loewis