[Python-Dev] Integrating Expat

[Martin v. Loewis]

> What? I've never seen a .a installed on a system that isn't usable. What
> makes Python modules so special that they could not use a library? I find
> your statement rather surprising, and I'm quite doubtful of the issue.
> 
> I've linked .a files into .so modules often. Never a problem.

It happens on Solaris with gcc as the compiler and /usr/ccs/bin/ld as
the linker. Watch this

pandora loewis 39 ( ~/tmp ) > cat a.c
int i;
int f(){return i;}
pandora loewis 40 ( ~/tmp ) > gcc -c a.c
pandora loewis 41 ( ~/tmp ) > gcc -o a.so -shared a.o
Text relocation remains                         referenced
    against symbol                  offset      in file
i                                   0x4         a.o
i                                   0x8         a.o
ld: fatal: relocations remain against allocatable but non-writable sections
collect2: ld returned 1 exit status

The problem here is that gcc passes -z text to the linker, unless
-mimpure-text is specified. ld(1) then complains if the code was not
compiled with -fPIC/-KPIC, i.e. that relocations remain.

Python is not special at all in this respect; that happens every time
when you integrate a non-PIC static library (or object file) into a
shared library, and
- the system is Solaris (or probably any SysV-derived system)
- the linker is the system linker
- the compiler passes -z text to the linker (i.e. it is gcc)

If the linker is binutils ld, it won't happen since the linker ignores
-z text. If the compiler is SUNWspro, it won't happen because cc does
not pass -z text by default. On a Linux system, it won't happen
because relocations against the text segment are not considered a
problem in the first place - COW will take care of that.

Of course, relocations against the text segment *are* a problem: they
cause the shared library not to be shared at run-time.

I don't know whether the problem is specific to Python; perhaps we
deal with more varying configurations than anybody else. Please see
the bug reports #232783, #404322, and #405329 for prior reports of
this issue.


> > - if the installed version of expat is older than the one we ship,
> >   I think Python should still use the included one.
> 
> Every time you use the included one, you run into the possibility of
> symbol conflict problems. If you refuse to use the one installed in the
> system, then you increase the chance of problems.

How would such a conflict occur using the current mechanisms for
dynamic loading in Python?

> > I think the problem you had with Apache does not apply to Python:
> > 
> > > Apache 1.3 would get loaded and export the Expat symbols to the rest
> > > of the process.
> > 
> > In Python, symbols of one extension module are not exported to
> > anything else.
>
> Eh? Again: what makes Python so special? How can you guarantee this
> rule? On *all* platforms?

Unlike everybody else, Python does *not* use RTLD_GLOBAL in the flags
passed to dlopen. This alone is a guarantee that shared libraries
won't interfere with each other:

  Optionally, RTLD_GLOBAL may be or'ed with flag, in which case the
  external symbols defined in the library will be made available to
  subsequently loaded libraries. [from the Linux dlopen man page]

It will work on all systems using dynload_shlib, plus it will work on
Windows (where you have to explicitly specify the symbols to export
and the libraries you import symbols from). Please also note the
Misc/HISTORY item

- Revert a new feature in Unix dynamic loading: for one or two
revisions, modules were loaded using the RTLD_GLOBAL flag.  It turned
out to be a bad idea.

The specific problem was that somebody was using a library that
exposed an initsocket function, which conflicted with the init
function of the socket module. So we cannot do much for systems that
cannot protect independent shared libraries from each other.


> This argument doesn't hold. dlopen() is not used on every platform. Thus,
> you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee
> that symbols in a library will remain private.

Systems that do not support local symbols in shared libraries have
bigger problems than competing libexpat implementations. BTW, which of
the systems in today's use do not have dlopen, and aren't Windows?
[How does MacOS X operate in this area?]

> Incorporating Expat directly into Python will definitely be "nice",
> but I've been through this already. It isn't pretty. If that shared
> library exists, then use the darned thing, or you're gonna get
> burned.

OTOH, I'm not looking forward to bug reports which turn out problems
with obsolete libexpat installations. You have to judge the risks.
Any user who is concerned about using the installed libexpat can still
modify Modules/Setup to use that instead of the one coming with
Python.

Regards,
Martin