[PyPy-issue] [issue614] Packaging PyPy: installation path for PyPy's libraries

Dave Malcolm pypy-dev-issue at codespeak.net
Thu Dec 23 02:54:02 CET 2010

New submission from Dave Malcolm <dmalcolm at redhat.com>:

Short version:

I'm working on packaging PyPy in RPM form for Fedora; see:
and ran into an issue with how the pypy binary searches for its libraries.

I'm attaching a patch I've written that fixes things, though it's not good enough 
yet to be applied to hg.

Long version:

As I understand things, PyPy's prebuilt tarballs have a directory structure 
generated by pypy/tool/release/package.py.

Looking at e.g.
everything is below:
  pypy-1.4.1-linux64/lib-python/2.5.2/*.py (and subdirs)
  pypy-1.4.1-linux64/lib-python/modified-2.5.2/*.py (and subdirs)
  pypy-1.4.1-linux64/bin/pypy (the JIT binary)
  pypy-1.4.1-linux64/include/ (some header files)

(See also pypy/tool/release/test/test_package.py, which verifies this structure)

This works for a standalone distribution of PyPy.

However, eventually Linux distribution will want to ship "system" builds of PyPy, 
deployed as .rpmor .deb packages.  Those packages ideally ought to follow the layout 
recommendations of the Filesystem Hierarchy Standard:

In particular this suggests separating out programs (to /usr/bin) from libraries (to 
/usr/lib) and data (/usr/share).

Exactly where the pypy libraries should go is debatable; they're all architecture-
independent for now, but I hope we'll soon have a rich collection of extension 
modules that are stable enough to package.

So I'm attempting to split the libraries out to /usr/lib (and /usr/lib64 on 64-bit, 
which is a Fedora-ism, AIUI).

My naive attempt to do this leads to a /usr/bin/pypy which emits this message at 
$ pypy-jit
debug: WARNING: library path not found, using compiled-in sys.path
'import site' failed
Python 2.5.2 (78826, Dec 15 2010, 01:41:24)
[PyPy 1.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
debug: OperationError:
debug:  operror-type: ImportError
debug:  operror-value: No module named _pypy_interact

and has this sys.path:
$ pypy -c 'import sys; print sys.path'
debug: WARNING: library path not found, using compiled-in sys.path
'import site' failed
['', '/builddir/build/BUILD/pypy-1.4/lib_pypy',

pypy/translator/goal/app_main.py's startup code calls into sys's state.py, which 
  def getinitialpath(prefix):
which searches for the dirs, raising OSError if they're not found.  If they're all 
their, it returns them in a list.

app_main wraps this as sys.pypy_initial_path, turning OSError into a return of None.

I'm attaching a proof-of-concept patch which adds an initial search for the 
directory structure, this time within LIBRARY_INSTALLATION_PATH.

Ideally this would be an optional, configuration-supplied constant, but for now, 
this is a non-existant variable, which I pre-process within the build, using sed, 
  sed -i \
    -e 's|LIBRARY_INSTALLATION_PATH|"%{pypyprefix}"|' \

where %{pypyprefix} is expanded by another tool, giving the string literal 

I wanted to prove the concept, and hour-long builds make for tedious experimentation 

Presumably this should be something like a:
that gets passed to the translate.py

Building with this patch (with the above caveat), I obtain a binary that, when 
copied to /usr/bin/pypy, successfully finds its libraries within our downstream 
distribution's directory structure (under /usr/lib64/pypy-1.4.1 on this build):

$ pypy
Python 2.5.2 (, Dec 22 2010, 23:14:02)
[PyPy 1.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``PyPy is an exciting technology
that lets you to write fast, portable, multi-platform interpreters with less
>>>> import sys
>>>> sys.path
['', '/usr/lib64/pypy-1.4.1/lib_pypy', '/usr/lib64/pypy-1.4.1/lib-python/modified-
2.5.2', '/usr/lib64/pypy-1.4.1/lib-python/2.5.2', '/usr/lib64/pypy-1.4.1/lib-
python/2.5.2/plat-linux2', '/usr/lib64/pypy-1.4.1/site-packages']

and also successfully finds its libraries within the build tree, when run from there 
in the usual "goal" subdirectory.
>>>> import collections
>>>> collections.__file__

effort: ???
files: pypy-1.4.1-add-LIBRARY_INSTALLATION_PATH.patch
messages: 2031
nosy: dmalcolm, pypy-issue
priority: feature
release: ???
status: unread
title: Packaging PyPy: installation path for PyPy's libraries

PyPy development tracker <pypy-dev-issue at codespeak.net>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-1.4.1-add-LIBRARY_INSTALLATION_PATH.patch
Type: text/x-patch
Size: 542 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-issue/attachments/20101223/f1dda35b/attachment.bin>

More information about the Pypy-issue mailing list