Translation with pypy picking up wrong libraries
Hi, I'm experimenting with using non-default library directories by tweeking pypy.tool.lib_pypy however I came up with the following error when translating with pypy: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 274, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 810, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 228, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 236, in get_entry_point [translation:ERROR] rebuild = import_from_lib_pypy('ctypes_config_cache/rebuild') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/tool/lib_pypy.py", line 12, in import_from_lib_pypy [translation:ERROR] return modname.pyimport() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/py/_path/local.py", line 531, in pyimport [translation:ERROR] mod = __import__(modname, None, None, ['__doc__']) [translation:ERROR] File "/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/rebuild.py", line 11, in <module> [translation:ERROR] execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py)) [translation:ERROR] IOError: [Errno 2] No such file or directory: '/usr/local/pypy-1.9/pypy/tool/autopath.py' This error does not happen when translating with python. Upon investigation I found that pypy has: # pypy -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' True but python has: # python -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' False also: # pypy -c 'import sys; import pypy.tool.lib_pypy; print sys.modules["ctypes_config_cache"]' <module 'ctypes_config_cache' from '/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/__init__.pyc'> note the above always happens regardless of pypy.tool.lib_pypy being modified or not. It appears to me that there are two bugs: 1) pypy ends up importing ctypes_config_cache when it shouldn't??? 2) py/_path/local.py doesn't ensure import from correct location And attached is a patch for fixing (2), which works around (1). Regards
Hi David, On Fri, Jun 22, 2012 at 1:05 PM, David Naylor <naylor.b.david@gmail.com> wrote:
And attached is a patch for fixing (2), which works around (1).
Your patch modifies the py lib, more precisely py/_path. The py lib is nowadays a separate project. You should try to discuss it and submit it to the py lib mailing list. Armin
Hi David, your change to pylib introduces a worse bug thats not acceptable it really should error out in that case, since it would enforce a partial code reload of the old code or even a wrong code reload for subpackages -- Ronny On 06/22/2012 01:05 PM, David Naylor wrote:
Hi,
I'm experimenting with using non-default library directories by tweeking pypy.tool.lib_pypy however I came up with the following error when translating with pypy:
[translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 274, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 810, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 228, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 236, in get_entry_point [translation:ERROR] rebuild = import_from_lib_pypy('ctypes_config_cache/rebuild') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/tool/lib_pypy.py", line 12, in import_from_lib_pypy [translation:ERROR] return modname.pyimport() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/py/_path/local.py", line 531, in pyimport [translation:ERROR] mod = __import__(modname, None, None, ['__doc__']) [translation:ERROR] File "/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/rebuild.py", line 11, in<module> [translation:ERROR] execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py)) [translation:ERROR] IOError: [Errno 2] No such file or directory: '/usr/local/pypy-1.9/pypy/tool/autopath.py'
This error does not happen when translating with python.
Upon investigation I found that pypy has: # pypy -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' True
but python has: # python -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' False
also: # pypy -c 'import sys; import pypy.tool.lib_pypy; print sys.modules["ctypes_config_cache"]' <module 'ctypes_config_cache' from '/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/__init__.pyc'> note the above always happens regardless of pypy.tool.lib_pypy being modified or not.
It appears to me that there are two bugs: 1) pypy ends up importing ctypes_config_cache when it shouldn't??? 2) py/_path/local.py doesn't ensure import from correct location
And attached is a patch for fixing (2), which works around (1).
Regards
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Hi David, i created the kill-import_from_lib_pypy branch, which should fix the issue - its still in testing, please take a look if any issue comes up -- Ronny On 06/24/2012 10:31 AM, Ronny Pfannschmidt wrote:
Hi David,
your change to pylib introduces a worse bug thats not acceptable
it really should error out in that case, since it would enforce a partial code reload of the old code or even a wrong code reload for subpackages
-- Ronny
On 06/22/2012 01:05 PM, David Naylor wrote:
Hi,
I'm experimenting with using non-default library directories by tweeking pypy.tool.lib_pypy however I came up with the following error when translating with pypy:
[translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 274, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 810, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 228, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 236, in get_entry_point [translation:ERROR] rebuild = import_from_lib_pypy('ctypes_config_cache/rebuild') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/tool/lib_pypy.py", line 12, in import_from_lib_pypy [translation:ERROR] return modname.pyimport() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/py/_path/local.py", line 531, in pyimport [translation:ERROR] mod = __import__(modname, None, None, ['__doc__']) [translation:ERROR] File "/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/rebuild.py", line 11, in<module> [translation:ERROR] execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py)) [translation:ERROR] IOError: [Errno 2] No such file or directory: '/usr/local/pypy-1.9/pypy/tool/autopath.py'
This error does not happen when translating with python.
Upon investigation I found that pypy has: # pypy -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' True
but python has: # python -c 'import sys; import pypy.tool.lib_pypy; print "ctypes_config_cache" in sys.modules.keys()' False
also: # pypy -c 'import sys; import pypy.tool.lib_pypy; print sys.modules["ctypes_config_cache"]' <module 'ctypes_config_cache' from '/usr/local/pypy-1.9/lib_pypy/ctypes_config_cache/__init__.pyc'> note the above always happens regardless of pypy.tool.lib_pypy being modified or not.
It appears to me that there are two bugs: 1) pypy ends up importing ctypes_config_cache when it shouldn't??? 2) py/_path/local.py doesn't ensure import from correct location
And attached is a patch for fixing (2), which works around (1).
Regards
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
On Monday, 25 June 2012 23:33:12 Ronny Pfannschmidt wrote:
Hi David,
i created the kill-import_from_lib_pypy branch, which should fix the issue - its still in testing, please take a look if any issue comes up
Alas, the changes you made to pypy/tool/compat.py and pypy/translator/goal/targetpypystandalone.py didn't work in my setup... [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 274, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 810, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 228, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 235, in get_entry_point [translation:ERROR] from lib_pypy.ctypes_config_cache.rebuild import try_rebuild [translation:ERROR] File "/usr/local/lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py", line 11, in <module> [translation:ERROR] execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py)) [translation:ERROR] IOError: [Errno 2] No such file or directory: '/usr/local/pypy/tool/autopath.py' I have the following changes: # cd pypy # mkdir lib # mv lib-python/2.7 lib/pypy1.9 # mv lib_pypy lib/pypy1.9/lib_pypy # patch << _EOF --- lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py~ 2012-06-22 11:42:55.000000000 +0200 +++ lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py 2012-06-22 11:43:12.000000000 +0200 @@ -6,7 +6,7 @@ # get the correct path import os.path this_dir = os.path.dirname(__file__) -autopath_py = os.path.join(this_dir, '../../pypy/tool/autopath.py') +autopath_py = os.path.join(this_dir, '../../../../pypy/tool/autopath.py') autopath_py = os.path.abspath(autopath_py) execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py)) --- pypy/module/sys/state.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/module/sys/state.py 2012-06-22 14:45:23.000000000 +0200 @@ -36,14 +36,12 @@ platform = sys.platform def getinitialpath(state, prefix): - from pypy.module.sys.version import CPYTHON_VERSION - dirname = '%d.%d' % (CPYTHON_VERSION[0], - CPYTHON_VERSION[1]) - lib_python = os.path.join(prefix, 'lib-python') - python_std_lib = os.path.join(lib_python, dirname) + from pypy.module.sys.version import PYPY_VERSION + libpath = os.path.join(prefix, 'lib') + python_std_lib = os.path.join(libpath, 'pypy%d.%d' % PYPY_VERSION[:2]) checkdir(python_std_lib) - lib_pypy = os.path.join(prefix, 'lib_pypy') + lib_pypy = os.path.join(python_std_lib, 'lib_pypy') checkdir(lib_pypy) importlist = [] --- pypy/tool/lib_pypy.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/tool/lib_pypy.py 2012-06-22 14:46:42.000000000 +0200 @@ -1,12 +1,11 @@ import py import pypy import pypy.module -from pypy.module.sys.version import CPYTHON_VERSION +from pypy.module.sys.version import PYPY_VERSION LIB_ROOT = py.path.local(pypy.__path__[0]).dirpath() -LIB_PYPY = LIB_ROOT.join('lib_pypy') -LIB_PYTHON = LIB_ROOT.join('lib-python', '%d.%d' % CPYTHON_VERSION[:2]) - +LIB_PYTHON = LIB_ROOT.join('lib', 'pypy%d.%d' % PYPY_VERSION[:2]) +LIB_PYPY = LIB_PYTHON.join('lib_pypy') def import_from_lib_pypy(modname): modname = LIB_PYPY.join(modname+'.py') _EOF and with the following patch the translate works successfully for me: # patch << _EOF --- py/_path/local.py~ 2012-06-22 12:20:36.000000000 +0200 +++ py/_path/local.py 2012-06-22 12:21:45.000000000 +0200 @@ -516,6 +516,9 @@ pkgpath = self.pypkgpath() if pkgpath is not None: if ensuresyspath: + import sys + if pkgpath.basename in sys.modules: + del sys.modules[pkgpath.basename] self._prependsyspath(pkgpath.dirpath()) pkg = __import__(pkgpath.basename, None, None, []) names = self.new(ext='').relto(pkgpath.dirpath()) _EOF Sorry, I should have mentioned this is my earlier post...
On Wednesday, 27 June 2012 20:47:28 David Naylor wrote:
On Monday, 25 June 2012 23:33:12 Ronny Pfannschmidt wrote:
Hi David,
i created the kill-import_from_lib_pypy branch, which should fix the issue - its still in testing, please take a look if any issue comes up
With the changes below your patch works (also, unpatched the translation works, whereas before it did not).
Alas, the changes you made to pypy/tool/compat.py and pypy/translator/goal/targetpypystandalone.py didn't work in my setup...
I believe that, due to my relocation of lib_pypy, I didn't catch all the cases (specifically autopath.py), and due to my error these errors have been occuring... Sorry!
I have the following changes:
And these additions to get a "properly" working environment
# cd pypy # mkdir lib # mv lib-python/2.7 lib/pypy1.9 # mv lib_pypy lib/pypy1.9/lib_pypy
# ln lib/pypy1.9/lib_pypy lib_pypy
# patch << _EOF --- lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py~ 2012-06-22 11:42:55.000000000 +0200 +++ lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py 2012-06-22 11:43:12.000000000 +0200 @@ -6,7 +6,7 @@ # get the correct path import os.path this_dir = os.path.dirname(__file__) -autopath_py = os.path.join(this_dir, '../../pypy/tool/autopath.py') +autopath_py = os.path.join(this_dir, '../../../../pypy/tool/autopath.py') autopath_py = os.path.abspath(autopath_py) execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py))
Using your patches the above change is not needed.
--- pypy/module/sys/state.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/module/sys/state.py 2012-06-22 14:45:23.000000000 +0200 @@ -36,14 +36,12 @@ platform = sys.platform
def getinitialpath(state, prefix): - from pypy.module.sys.version import CPYTHON_VERSION - dirname = '%d.%d' % (CPYTHON_VERSION[0], - CPYTHON_VERSION[1]) - lib_python = os.path.join(prefix, 'lib-python') - python_std_lib = os.path.join(lib_python, dirname) + from pypy.module.sys.version import PYPY_VERSION + libpath = os.path.join(prefix, 'lib') + python_std_lib = os.path.join(libpath, 'pypy%d.%d' % PYPY_VERSION[:2]) checkdir(python_std_lib)
- lib_pypy = os.path.join(prefix, 'lib_pypy') + lib_pypy = os.path.join(python_std_lib, 'lib_pypy') checkdir(lib_pypy)
importlist = [] --- pypy/tool/lib_pypy.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/tool/lib_pypy.py 2012-06-22 14:46:42.000000000 +0200 @@ -1,12 +1,11 @@ import py import pypy import pypy.module -from pypy.module.sys.version import CPYTHON_VERSION +from pypy.module.sys.version import PYPY_VERSION
LIB_ROOT = py.path.local(pypy.__path__[0]).dirpath() -LIB_PYPY = LIB_ROOT.join('lib_pypy') -LIB_PYTHON = LIB_ROOT.join('lib-python', '%d.%d' % CPYTHON_VERSION[:2]) - +LIB_PYTHON = LIB_ROOT.join('lib', 'pypy%d.%d' % PYPY_VERSION[:2]) +LIB_PYPY = LIB_PYTHON.join('lib_pypy')
def import_from_lib_pypy(modname): modname = LIB_PYPY.join(modname+'.py') _EOF
And with just the above translation works successfully for me. Regards
Hi David, i proceeded with refining my changes, moving all code related to writing these dumps out of lib_pypy i'd like to know why you are relocating lib_pypy/lib-python -- Ronny On 06/27/2012 09:19 PM, David Naylor wrote:
On Wednesday, 27 June 2012 20:47:28 David Naylor wrote:
On Monday, 25 June 2012 23:33:12 Ronny Pfannschmidt wrote:
Hi David,
i created the kill-import_from_lib_pypy branch, which should fix the issue - its still in testing, please take a look if any issue comes up
With the changes below your patch works (also, unpatched the translation works, whereas before it did not).
Alas, the changes you made to pypy/tool/compat.py and pypy/translator/goal/targetpypystandalone.py didn't work in my setup...
I believe that, due to my relocation of lib_pypy, I didn't catch all the cases (specifically autopath.py), and due to my error these errors have been occuring... Sorry!
I have the following changes:
And these additions to get a "properly" working environment
# cd pypy # mkdir lib # mv lib-python/2.7 lib/pypy1.9 # mv lib_pypy lib/pypy1.9/lib_pypy
# ln lib/pypy1.9/lib_pypy lib_pypy
# patch<< _EOF --- lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py~ 2012-06-22 11:42:55.000000000 +0200 +++ lib/pypy1.9/lib_pypy/ctypes_config_cache/rebuild.py 2012-06-22 11:43:12.000000000 +0200 @@ -6,7 +6,7 @@ # get the correct path import os.path this_dir = os.path.dirname(__file__) -autopath_py = os.path.join(this_dir, '../../pypy/tool/autopath.py') +autopath_py = os.path.join(this_dir, '../../../../pypy/tool/autopath.py') autopath_py = os.path.abspath(autopath_py) execfile(autopath_py, dict(__name__='autopath', __file__=autopath_py))
Using your patches the above change is not needed.
--- pypy/module/sys/state.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/module/sys/state.py 2012-06-22 14:45:23.000000000 +0200 @@ -36,14 +36,12 @@ platform = sys.platform
def getinitialpath(state, prefix): - from pypy.module.sys.version import CPYTHON_VERSION - dirname = '%d.%d' % (CPYTHON_VERSION[0], - CPYTHON_VERSION[1]) - lib_python = os.path.join(prefix, 'lib-python') - python_std_lib = os.path.join(lib_python, dirname) + from pypy.module.sys.version import PYPY_VERSION + libpath = os.path.join(prefix, 'lib') + python_std_lib = os.path.join(libpath, 'pypy%d.%d' % PYPY_VERSION[:2]) checkdir(python_std_lib)
- lib_pypy = os.path.join(prefix, 'lib_pypy') + lib_pypy = os.path.join(python_std_lib, 'lib_pypy') checkdir(lib_pypy)
importlist = [] --- pypy/tool/lib_pypy.py.orig 2012-06-07 14:24:48.000000000 +0200 +++ pypy/tool/lib_pypy.py 2012-06-22 14:46:42.000000000 +0200 @@ -1,12 +1,11 @@ import py import pypy import pypy.module -from pypy.module.sys.version import CPYTHON_VERSION +from pypy.module.sys.version import PYPY_VERSION
LIB_ROOT = py.path.local(pypy.__path__[0]).dirpath() -LIB_PYPY = LIB_ROOT.join('lib_pypy') -LIB_PYTHON = LIB_ROOT.join('lib-python', '%d.%d' % CPYTHON_VERSION[:2]) - +LIB_PYTHON = LIB_ROOT.join('lib', 'pypy%d.%d' % PYPY_VERSION[:2]) +LIB_PYPY = LIB_PYTHON.join('lib_pypy')
def import_from_lib_pypy(modname): modname = LIB_PYPY.join(modname+'.py') _EOF
And with just the above translation works successfully for me.
Regards
On Friday, 29 June 2012 21:46:58 Ronny Pfannschmidt wrote:
Hi David,
Hi
i proceeded with refining my changes, moving all code related to writing these dumps out of lib_pypy
Thanks
i'd like to know why you are relocating lib_pypy/lib-python
I am slowly integrating pypy into FreeBSD Port's Collection, with the eventual goal of installing third party python based ports (i.e. setup.py and such). FreeBSD follows a standard (I think) posix directory layout. As such I wanted the pypy binary to live in $PREFIX/bin and the libraries in $PREFIX/lib. The convention for cPython is to have the libraries as $PREFIX/lib/python2.7, so for pypy I wanted to have $PREFIX/lib/pypy1.9 (although I did consider $PREFIX/lib/pypython2.7), which leaves the lib_pypy libraries. These cannot be renamed to anything such as pypy1.9 so I stuck them within the standard libraries directory. A condition for this is the libraries cannot conflict with cPython. Previously I had an explicit directory $PREFIX/pypy-1.9 where I had all the files (including the binary) with sym-links for the binaries to $PREFIX/bin. This, however, I considered less clean. Regards
participants (3)
-
Armin Rigo -
David Naylor -
Ronny Pfannschmidt