RPython Darwin platform description so_prefixes empty
Hi all, I have been doing some coding using RFFI to load function pointers from shared libraries (`llexternal` function). However what bothers me is that apparently `Darwin_x86_64` platform class (defined in translator/platform/darwin.py) does not include ‘lib’ in `so_prefixes`. This is causing me some problems with not being able to find my shared libraries on macOS X. I’m wondering if that’s intentional, or just a mistake? It should be a simple fix if it’s a mistake. Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang@anu.edu.au<mailto:john.zhang@anu.edu.au>
Hi John, On 30 November 2016 at 04:14, John Zhang <John.Zhang@anu.edu.au> wrote:
I don't know, and that's a good question. But the first question is: how does it work for PyPy on OS/X? All (or most) tests pass there. Can you pinpoint the difference? If not, can you point us to your version of the code, so that we can have a guess at what the difference might be? A bientôt, Armin.
Hi Armin, The test `rpython.rtyper.lltypesystem.test.TestPlatform.test_prefix` illustrates the problem quite well. Currently this test is skipped on non-linux platforms. When that is turned off, the test fails on macOS X. With `Darwin_x86.so_prefix == (‘’, )`, line ll2ctypes:1218 will fail, because it’s trying to load `c_file.dylib` rather than `libc_file.dylib`. Shared libraries on macOS X should also have ‘lib’ prefix just as on linux. According to man page of `ld`: -lx This option tells the linker to search for libx.dylib or libx.a in the library search path. If string x is of the form y.o, then that file is searched for in the same places, but without prepending `lib' or appending `.a' or `.dylib' to the filename. And on Apple developer website (https://developer.apple.com/library/content/documentation/DeveloperTools/Con...): The filename of a dynamic library normally contains the library’s name with the lib prefix and the .dylib extension. For example, a library called Dynamo would have the filename libDynamo.dylib. So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms. The attached is a suggested solution patch. Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang@anu.edu.au<mailto:john.zhang@anu.edu.au> On 1 Dec. 2016, at 04:56, Armin Rigo <armin.rigo@gmail.com<mailto:armin.rigo@gmail.com>> wrote: Hi John, On 30 November 2016 at 04:14, John Zhang <John.Zhang@anu.edu.au<mailto:John.Zhang@anu.edu.au>> wrote: `so_prefixes`. This is causing me some problems with not being able to find my shared libraries on macOS X. I’m wondering if that’s intentional, or just a mistake? It should be a simple fix if it’s a mistake. I don't know, and that's a good question. But the first question is: how does it work for PyPy on OS/X? All (or most) tests pass there. Can you pinpoint the difference? If not, can you point us to your version of the code, so that we can have a guess at what the difference might be? A bientôt, Armin.
Hi John, On 6 December 2016 at 03:06, John Zhang <John.Zhang@anu.edu.au> wrote:
So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms.
I think we can include 'lib' in so_prefix of all Posix platforms, is that right? It's already in linux, bsd and cygwin. I still have no clue why all our tests still pass on OS/X (apart from the skipped one you found) but I guess that's a fine fix anyway. A bientôt, Armin.
Hi Armin, I think we can include 'lib' in so_prefix of all Posix platforms, is that right? I can’t seem to find a definitive answer for all POSIX platforms. But for all the ones that PyPy currently supports, I think the answer is true. So if you want to make the changes then I think it’s a reasonable assumption and a good idea. I still have no clue why all our tests still pass on OS/X I think it’s because `rpython.translator.platform.Platform._finish_linking` function doesn’t include `so_prefixes` on the platform in the first place, so for tests like `test_lib_on_libpaths` will generate a shared library called `c_file.dylib`. Then in `get_ctypes_callable` function it prepends the library name with the currently empty prefix on Darwin platform, it can find the shared library file with the constructed absolute path. So the proposed fix will still work because it includes both ‘lib’ and empty prefix. On the other hand, I wonder if it’s also reasonable to modify the `_finish_linking` function to include `platform.so_prefix[0]` to the generated shared library name (see the attached patch). Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang@anu.edu.au<mailto:john.zhang@anu.edu.au> On 6 Dec. 2016, at 20:48, Armin Rigo <armin.rigo@gmail.com<mailto:armin.rigo@gmail.com>> wrote: Hi John, On 6 December 2016 at 03:06, John Zhang <John.Zhang@anu.edu.au<mailto:John.Zhang@anu.edu.au>> wrote: So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms. I think we can include 'lib' in so_prefix of all Posix platforms, is that right? It's already in linux, bsd and cygwin. I still have no clue why all our tests still pass on OS/X (apart from the skipped one you found) but I guess that's a fine fix anyway. A bientôt, Armin.
Hi John, On 7 December 2016 at 02:06, John Zhang <John.Zhang@anu.edu.au> wrote:
Checked in 4b1508ade877.
It might or might not break stuff. ``.so`` files without the ``lib`` prefix exist and are generally useful. As far as it works this way, I'd be wary of changes. A bientôt, Armin.
Hi John, On 30 November 2016 at 04:14, John Zhang <John.Zhang@anu.edu.au> wrote:
I don't know, and that's a good question. But the first question is: how does it work for PyPy on OS/X? All (or most) tests pass there. Can you pinpoint the difference? If not, can you point us to your version of the code, so that we can have a guess at what the difference might be? A bientôt, Armin.
Hi Armin, The test `rpython.rtyper.lltypesystem.test.TestPlatform.test_prefix` illustrates the problem quite well. Currently this test is skipped on non-linux platforms. When that is turned off, the test fails on macOS X. With `Darwin_x86.so_prefix == (‘’, )`, line ll2ctypes:1218 will fail, because it’s trying to load `c_file.dylib` rather than `libc_file.dylib`. Shared libraries on macOS X should also have ‘lib’ prefix just as on linux. According to man page of `ld`: -lx This option tells the linker to search for libx.dylib or libx.a in the library search path. If string x is of the form y.o, then that file is searched for in the same places, but without prepending `lib' or appending `.a' or `.dylib' to the filename. And on Apple developer website (https://developer.apple.com/library/content/documentation/DeveloperTools/Con...): The filename of a dynamic library normally contains the library’s name with the lib prefix and the .dylib extension. For example, a library called Dynamo would have the filename libDynamo.dylib. So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms. The attached is a suggested solution patch. Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang@anu.edu.au<mailto:john.zhang@anu.edu.au> On 1 Dec. 2016, at 04:56, Armin Rigo <armin.rigo@gmail.com<mailto:armin.rigo@gmail.com>> wrote: Hi John, On 30 November 2016 at 04:14, John Zhang <John.Zhang@anu.edu.au<mailto:John.Zhang@anu.edu.au>> wrote: `so_prefixes`. This is causing me some problems with not being able to find my shared libraries on macOS X. I’m wondering if that’s intentional, or just a mistake? It should be a simple fix if it’s a mistake. I don't know, and that's a good question. But the first question is: how does it work for PyPy on OS/X? All (or most) tests pass there. Can you pinpoint the difference? If not, can you point us to your version of the code, so that we can have a guess at what the difference might be? A bientôt, Armin.
Hi John, On 6 December 2016 at 03:06, John Zhang <John.Zhang@anu.edu.au> wrote:
So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms.
I think we can include 'lib' in so_prefix of all Posix platforms, is that right? It's already in linux, bsd and cygwin. I still have no clue why all our tests still pass on OS/X (apart from the skipped one you found) but I guess that's a fine fix anyway. A bientôt, Armin.
Hi Armin, I think we can include 'lib' in so_prefix of all Posix platforms, is that right? I can’t seem to find a definitive answer for all POSIX platforms. But for all the ones that PyPy currently supports, I think the answer is true. So if you want to make the changes then I think it’s a reasonable assumption and a good idea. I still have no clue why all our tests still pass on OS/X I think it’s because `rpython.translator.platform.Platform._finish_linking` function doesn’t include `so_prefixes` on the platform in the first place, so for tests like `test_lib_on_libpaths` will generate a shared library called `c_file.dylib`. Then in `get_ctypes_callable` function it prepends the library name with the currently empty prefix on Darwin platform, it can find the shared library file with the constructed absolute path. So the proposed fix will still work because it includes both ‘lib’ and empty prefix. On the other hand, I wonder if it’s also reasonable to modify the `_finish_linking` function to include `platform.so_prefix[0]` to the generated shared library name (see the attached patch). Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang@anu.edu.au<mailto:john.zhang@anu.edu.au> On 6 Dec. 2016, at 20:48, Armin Rigo <armin.rigo@gmail.com<mailto:armin.rigo@gmail.com>> wrote: Hi John, On 6 December 2016 at 03:06, John Zhang <John.Zhang@anu.edu.au<mailto:John.Zhang@anu.edu.au>> wrote: So I think it’s right to make a simple fix as suggested in the patch to include ‘lib’ in the so_prefix on Darwin platforms. I think we can include 'lib' in so_prefix of all Posix platforms, is that right? It's already in linux, bsd and cygwin. I still have no clue why all our tests still pass on OS/X (apart from the skipped one you found) but I guess that's a fine fix anyway. A bientôt, Armin.
Hi John, On 7 December 2016 at 02:06, John Zhang <John.Zhang@anu.edu.au> wrote:
Checked in 4b1508ade877.
It might or might not break stuff. ``.so`` files without the ``lib`` prefix exist and are generally useful. As far as it works this way, I'd be wary of changes. A bientôt, Armin.
participants (2)
-
Armin Rigo -
John Zhang