Change how a pypy executable finds its lib-python/lib_pypy directories?
Hi all, Right now the "pypy" executable looks for its lib-python/lib_pypy directories by starting from the path of the "pypy" executable and going up. That path is derived from the C-level argv[0], on which we try once to call readlink() if it is a symlink. That logic is in pypy.module.sys.initpath. When using cffi's embedding mode, however, the logic starts differently: we start at the path of the "libpypy-c.so" library instead, and then we look around in the same way. This is because, of course, there is no "pypy" executable in this case. This difference breaks some packaged pypy's: it is the only thing that prevents e.g. the Arch Linux pypy from working in embedded mode. The reason is that they install /opt/pypy/* like we recommend, and set a symlink /usr/bin/pypy -> /opt/pypy/bin/pypy like we also recommend, but they really move libpypy-c.so in /usr/lib/libpypy-c.so. It works fine in the regular case, but fails with the embedding mode. I tried to make a symlink /usr/lib/libpypy-c.so -> /opt/pypy/bin/libpypy-c.so and embedding mode then works fine. As an attempt to fix this more generally, we should make the two cases more similar. Does it make sense to only look around from the "libpypy-c.so" location? Finding the path of "libpypy-c.so" requires a little bit more hackery than just reading "argv[0]", but on the other hand it is not likely to break even if "pypy" is called strangely (because argv[0] can in theory be any random string). It is code that is so far in pypy.module._cffi_backend.embedding, which would probably move to pypy.module.sys.initpath. This change would require minor fixes for pypy packagers---e.g. arch linux would simply make a symlink for /usr/lib/libpypy-c.so instead of for /usr/bin/pypy. Makes sense? A bientôt, Armin.
Hi all, Now PyPy looks for lib-python/lib_pypy starting from the (real) location of its ``libpypy-c.so``, no longer from the ``pypy`` executable. Linux distribution packagers, take note! At a minimum, the ``libpypy-c.so`` must really be inside the path containing ``lib-python`` and ``lib_pypy``. Of course, you can (and likely should) also put a symlink to it from somewhere else. It is what you had previously to do with the ``pypy`` executable. Now it must be done with ``libpypy-c.so`` instead (and you no longer have to use a symlink for the ``pypy`` executable, as long as it finds its ``libpypy-c.so``). If, instead, you edit PyPy to change the way it is installed, then please take a moment to check that *embedding* works correctly. Test case: https://bitbucket.org/cffi/cffi/raw/default/demo/embedding.py https://bitbucket.org/cffi/cffi/raw/default/demo/embedding_test.c A bientôt, Armin.
Does this have any effect on virtualenvs? virtualenv symlinks libpypy-c.so and _pypy_init_home returns its location realpath()’d.. -- Philip Jenvey
On Sep 24, 2016, at 1:52 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi all,
Now PyPy looks for lib-python/lib_pypy starting from the (real) location of its ``libpypy-c.so``, no longer from the ``pypy`` executable.
Linux distribution packagers, take note! At a minimum, the ``libpypy-c.so`` must really be inside the path containing ``lib-python`` and ``lib_pypy``. Of course, you can (and likely should) also put a symlink to it from somewhere else.
It is what you had previously to do with the ``pypy`` executable. Now it must be done with ``libpypy-c.so`` instead (and you no longer have to use a symlink for the ``pypy`` executable, as long as it finds its ``libpypy-c.so``).
If, instead, you edit PyPy to change the way it is installed, then please take a moment to check that *embedding* works correctly. Test case:
https://bitbucket.org/cffi/cffi/raw/default/demo/embedding.py https://bitbucket.org/cffi/cffi/raw/default/demo/embedding_test.c
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi again, On 25 September 2016 at 09:13, Armin Rigo <arigo@tunes.org> wrote:
On 24 September 2016 at 23:30, Philip Jenvey <pjenvey@underboss.org> wrote:
Does this have any effect on virtualenvs?
virtualenv symlinks libpypy-c.so and _pypy_init_home returns its location realpath()’d..
How about looking first from the place of the symlink itself, and if not found, we dereference the symlink and look again? A bientôt, Armin.
Hi, On 25 September 2016 at 10:16, Armin Rigo <arigo@tunes.org> wrote:
How about looking first from the place of the symlink itself, and if not found, we dereference the symlink and look again?
That can't work, because there is no way to know the path of the symlink to libpypy-c.so. We can only learn the dereferenced path, using dladdr(). I've partly reverted my check-in: now we look first based on the pypy executable's real location, like before; and only if that fails, we look based on the libpypy-c.so's real location. Like before, we never look based on where a symlink is. That means it should be fully backward compatible, but in addition it also works if the symlink is on the libpypy-c.so instead of the pypy executable. We should still try to convince a few distributions to use this new approach, in order to fix embedding there. A bientôt, Armin.
participants (2)
-
Armin Rigo -
Philip Jenvey