[Python-Dev] Handling linker scripts reached when dynamically loading a module

[Nadeem Vawda]

I can confirm that libpthread.so (/usr/lib/x86_64-linux-gnu/libpthread.so)
is a linker script on my Ubuntu 11.04 install. This hasn't ever caused me
any problems, though.

As for why distributions do this, here are the contents of the script:

    /* GNU ld script
       Use the shared library, but some functions are only in
       the static library, so try that secondarily.  */
    OUTPUT_FORMAT(elf64-x86-64)
    GROUP ( /lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread_nonshared.a )

Cheers,
Nadeem

On Sun, Sep 11, 2011 at 12:24 AM, Guido van Rossum <guido at python.org> wrote:
> Odd. Let's see what other core devs say.
>
> On Sat, Sep 10, 2011 at 2:50 PM, Howard B. Golden
> <howard_b_golden at yahoo.com> wrote:
>> I don't know why, but some Linux distributions place scripts into .so
>> files instead of the actual binaries. This takes advantage of a feature
>> of GNU ld that it will process the script (which points to the actual
>> binary) when it links the .so file.
>>
>> This feature works fine when you are linking a binary, but it doesn't
>> take into account that binaries can be loaded dynamically by
>> interpreters (e.g., Python or GHCi). If dlopen finds a linker script, it
>> doesn't know what to do with it. It simply diagnoses the file as either
>> an invalid ELF header or too short.
>>
>> On Gentoo Linux, some common libraries that are represented as linker
>> scripts include libm.so, libpthread.so and libpcre.so. I know this also
>> affects Ubuntu.
>>
>> Howard
>>
>> On Sat, 2011-09-10 at 14:39 -0700, Guido van Rossum wrote:
>>> Excuse me for asking a newbie question, but what are linker scripts
>>> and why are they important? I don't recall anyone ever having
>>> requested this feature before.
>>>
>>> --Guido
>>>
>>> On Wed, Sep 7, 2011 at 12:33 PM, Howard B. Golden
>>> <howard_b_golden at yahoo.com> wrote:
>>> > Hi,
>>> >
>>> > In Haskell I experienced a situation where dynamically loaded modules
>>> > were experiencing "invalid ELF header" errors. This was caused by the
>>> > module names actually referring to linker scripts rather than ELF
>>> > binaries. I patched the GHC runtime system to deal with these scripts.
>>> >
>>> > I noticed that this same patch has been ported to Ruby and Node.js, so I
>>> > suggested to the libc developers that they might wish to incorporate the
>>> > patch into their library, making it available to all languages. They
>>> > rejected this suggestion, so I am making the suggestion to the Python
>>> > devs in case it is of interest to you.
>>> >
>>> > Basically, when a linker script is loaded by dlopen, an "invalid ELF
>>> > header" error occurs. The patch checks to see if the file is a linker
>>> > script. If so, it finds the name of the real ELF binary with a regular
>>> > expression and tries to dlopen it. If successful, processing proceeds.
>>> > Otherwise, the original "invalid ELF error" message is returned.
>>> >
>>> > If you want to add this code to Python, you can look at my original
>>> > patch (http://hackage.haskell.org/trac/ghc/ticket/2615) or the Ruby
>>> > version (https://github.com/ffi/ffi/pull/117) or the Node.js version
>>> > (https://github.com/rbranson/node-ffi/pull/5) to help port it.
>>> >
>>> > Note that the GHC version in GHC 7.2.1 has been enhanced to also handle
>>> > another possible error when the linker script is too short, so you might
>>> > also want to add this enhancement also (see
>>> > https://github.com/ghc/blob/master/rts/Linker.c line 1191 for the
>>> > revised regular expression):
>>> >
>>> > "(([^ \t()])+\\.so([^ \t:()])*):([ \t])*(invalid ELF header|file too
>>> > short)"
>>> >
>>> > At this point, I don't have the free time to write the Python patch
>>> > myself, so I apologize in advance for not providing it to you.
>>> >
>>> > HTH,
>>> >
>>> > Howard B. Golden
>>> > Northridge, California, USA
>>> >
>>> > _______________________________________________
>>> > Python-Dev mailing list
>>> > Python-Dev at python.org
>>> > http://mail.python.org/mailman/listinfo/python-dev
>>> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>> >
>>>
>>>
>>>
>>
>>
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com
>