Mailman 3 January 2016 - pypy-dev

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

Looking for clues to consistent Seg Fault in PyPy 2.6.1
by Jeff Doran 17 Feb '16

17 Feb '16

Hi, I've run out of options trying to find a Seg Fault which happens when running lxml under PyPy 2.6.1. This problem only occurs under PyPy as the rest of the code works fine under CPython 2.7. I've been in contact with the lxml dev team and they confirmed my problem, but could not determine where the cause of the Seg Fault lies. They suggested reaching out to you folks for ideas or a possible work around as the stack trace is always in libpypy-c.so and seems to be triggered by eventual garbage collection. Below is the gist of the email I sent to the lxml team earlier. It describes my ability to reproduce this Seg Fault using their supplied test suite. As of this writing, lxml is at version 3.5.0b1. I'd appreciate any feedback you might supply on how to deal with this error. Seg Faults are notoriously hard to find since they inevitably happen upstream for the crash. We are anxious to perform a complete evaluation of transitioning to PyPy as our Python platform of choice and this appears to be the last major blockage. Thanks for your help. - Jeff Doran ---------------- I'm running the lxml test suite under PyPy which I installed from here,https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-2.6.1-linux… . I've downloaded the current lxml src from https://github.com/lxml/lxml. I built lxml using 'python setup.py build --with-cython' *From ..lxml/src/lxml I run: * nosetests -v --pdb --nocapture Comparing with ElementTree 1.3.0 Comparing with cElementTree 1.3.0 TESTED VERSION: 3.5.0.beta1 Python: (major=2, minor=7, micro=10, releaselevel='final', serial=42) lxml.etree: (3, 5, 0, -99) libxml used: (2, 8, 0) libxml compiled: (2, 8, 0) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 26) lxml.html.tests.test_autolink.test_suite ... ok lxml.html.tests.test_basic.test_suite ... ok test_allow_tags (lxml.html.tests.test_clean.CleanerTest) ... ok test_clean_invalid_root_tag (lxml.html.tests.test_clean.CleanerTest) ... ok test_safe_attrs_excluded (lxml.html.tests.test_clean.CleanerTest) ... ok test_safe_attrs_included (lxml.html.tests.test_clean.CleanerTest) ... ok lxml.html.tests.test_clean.test_suite ... ok lxml.html.tests.test_diff.test_suite ... ok test_body (lxml.html.tests.test_elementsoup.SoupParserTestCase) ... ok test_broken_attribute (lxml.html.tests.test_elementsoup.SoupParserTestCase) ... fish: Job 1, “nosetests -v --pdb --nocapture” terminated by signal SIGSEGV (Address boundary error) ``` Running these tests under gdb reveals the following partial stack trace: #0 0x00007ffff57fa08a in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #1 0x00007ffff57fa110 in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #2 0x00007ffff5776be3 in PyPyWeakref_LockObject () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #3 0x00007ffff0337399 in __pyx_f_4lxml_5etree__isProxyAliveInPypy.isra.265 () from /home/jeff/lxml/src/lxml/etree.pypy-26.so #4 0x00007ffff033fbfb in __pyx_f_4lxml_5etree_attemptDeallocation () from /home/jeff/lxml/src/lxml/etree.pypy-26.so #5 0x00007ffff033fd70 in __pyx_tp_dealloc_4lxml_5etree__Element () from /home/jeff/lxml/src/lxml/etree.pypy-26.so #6 0x00007ffff62186f9 in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #7 0x00007ffff57aec90 in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #8 0x00007ffff57fa670 in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #9 0x00007ffff5fa0da8 in ?? () from /home/jeff/debian_pypy_env/bin/libpypy-c.so #10 0x00007ffff5f89a31 in ?? ()

6 17

windows and python27.lib
by Matti Picus 29 Jan '16

29 Jan '16

When linking to a dll with MSVC, one needs to provide not the dll itself rather an import library. Also, the linker must be able to resolve all function definitions in order to run to completion, so the linker must be able to find the import library. On cpython, the import library used to link to python27.dll is called Python27.lib, it is located in a 'libs' directory at the top level in a cpython distribution together with things like pyexpat.lib, _socket.lib etc. distutils (and setuptools) do not need to know about the import library, since a #pragma is used to issue a link directive for the import library whenever pyconfig.h is included in a c file. A debug build will create and use python27_d.lib that corresponds to python27_d.dll What do we do in PyPy? At translation we build a libpypy-c.dll, and an export library libpypy-c.lib. These are copied together with pypy-c.exe to the directory where translation occurred (on the build bots this is in pypy/goal) Then the package script copies the libpypy-c.lib to pypy's include directory as python27.lib. A debug build will use these same names. The same pragma is used as in cpython to force linking with the import library whenever pyconfig.h is included So what you ask? I think the exe should be created as pypy.exe, the dll should be called pypy27.dll, and the import library should be consistently named pypy27.lib. There should be no renaming in package.py. This has implications in the following places: - the exe_name in targetpypystandalone should drop the -%(backend) modifier - pyconfig.h and the package script should be modified to use pypy27 consistently - probably some tests will fail, they should be fixed - cffi/api.py needs tweaking in _apply_embedding_fix - package.py should not rename (what do we do on linux about pypy-c -> pypy?) what did I forget? We should also handle a debug build of pypy, we should be creating a pypy27_d.lib, pypy27_d.dll, today the usual names are reused. Are there compelling reasons _not_ to make the naming consistent with cpython? As a fallback, we could just rename the import library to pypy27.lib My current motivation to do this is that _apply_embedding_fix does not work for win32 pypy Matti

1 0

Stack limit in the jit backends
by Richard Plangger 28 Jan '16

28 Jan '16

Hi, the file rpython/translator/c/src/stack.h defines MAX_STACK_SIZE. PPC has a bigger limit than e.g. x86. I experienced that on s390x there is as well a higher consumption of memory for stack frames (they are variable sized with a pretty high minimum limit (160 bytes) by the ABI). I have two questions: 1) The OS (i.e. linux) defines a stack limit (ulimit -s), does pypy overwrite this value with MAX_STACK_SIZE? 2) How would I determine which size is best for s390x? Or how did we come up with 768kb for x86, and 2.8mb for ppc? Cheers, Richard

2 1

partition in numpypy
by Sergey Matyunin 23 Jan '16

23 Jan '16

Hello. Need a little help with numpypy. I want to implement partition method for numpy array. Let's say I can compile npy_partition.h.src, import it through CFFI. Therefore I can write a python function my_partition(numpy_array, other_arguments...) that performs partitioning for a give numpy array. Now I want to create partition method for ndarray. As far as I understand methods of ndarray are defined in pypy/module/micronumpy/ndarray.py in a special way and ndarray.partition = my_partition doesn't work. Is it possible to add method to ndarray inside numpypy, not inside pypy's micronumpy module? -- Sergey

2 1

TypeError: expected string, got NoneType object with setuptools and co with pypy 4
by Nagy, Attila 21 Jan '16

21 Jan '16

Hi, After installing pypy 4.0.1 on FreeBSD (from ports), I get the above exception when trying to install setuptools, or when installing anything with pip (after pypy -m ensurepip). This all worked with the previous version (2.6). I could find this problem elsewhere: http://stackoverflow.com/questions/34566676/failed-to-install-pip-for-pypy-… so it doesn't seem to be related to FreeBSD. An example full trace: # pypy -m pip install --upgrade pip You are using pip version 6.1.1, however version 8.0.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting pip Exception: Traceback (most recent call last): File "/usr/local/pypy-4.0/site-packages/pip/basecommand.py", line 246, in main status = self.run(options, args) File "/usr/local/pypy-4.0/site-packages/pip/commands/install.py", line 342, in run requirement_set.prepare_files(finder) File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 345, in prepare_files functools.partial(self._prepare_file, finder)) File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 290, in _walk_req_to_install more_reqs = handler(req_to_install) File "/usr/local/pypy-4.0/lib_pypy/_functools.py", line 42, in __call__ return self._func(*(self._args + fargs), **fkeywords) File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 487, in _prepare_file download_dir, do_download, session=self.session, File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 827, in unpack_url session, File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 673, in unpack_http_url from_path, content_type = _download_http_url(link, session, temp_dir) File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 887, in _download_http_url with open(file_path, 'wb') as content_file: TypeError: expected string, got NoneType object And the one for setuptools: # pypy setup.py install running install Checking .pth file support in /usr/local/pypy-4.0/site-packages/ /usr/local/bin/pypy -E -c pass TEST PASSED: /usr/local/pypy-4.0/site-packages/ appears to support .pth files running bdist_egg running egg_info writing setuptools.egg-info/PKG-INFO writing dependency_links to setuptools.egg-info/dependency_links.txt writing entry points to setuptools.egg-info/entry_points.txt writing requirements to setuptools.egg-info/requires.txt writing top-level names to setuptools.egg-info/top_level.txt Traceback (most recent call last): File "setup.py", line 169, in <module> dist = setuptools.setup(**setup_params) File "/usr/local/pypy-4.0/lib-python/2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/setuptools-19.4/setuptools/command/install.py", line 67, in run self.do_egg_install() File "/tmp/setuptools-19.4/setuptools/command/install.py", line 109, in do_egg_install self.run_command('bdist_egg') File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/setuptools-19.4/setuptools/command/bdist_egg.py", line 152, in run self.run_command("egg_info") File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 186, in run self.find_sources() File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 209, in find_sources mm.run() File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 293, in run self.add_defaults() File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 322, in add_defaults sdist.add_defaults(self) File "/tmp/setuptools-19.4/setuptools/command/sdist.py", line 100, in add_defaults self.filelist.append(fn) File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 236, in append if self._safe_path(path): File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 256, in _safe_path u_path = unicode_utils.filesys_decode(path) File "/tmp/setuptools-19.4/setuptools/unicode_utils.py", line 31, in filesys_decode return path.decode(enc) TypeError: expected string, got NoneType object In this case the context is: def filesys_decode(path): """ Ensure that the given path is decoded, NONE when no expected encoding works """ fs_enc = sys.getfilesystemencoding() if isinstance(path, six.text_type): return path for enc in (fs_enc, "utf-8"): try: return path.decode(enc) except UnicodeDecodeError: continue On python 2.7, fs_enc here is: Python 2.7.11 (default, Dec 20 2015, 01:15:21) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> repr(sys.getfilesystemencoding()) "'US-ASCII'" While on pypy: Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Jan 16 2016, 01:16:36) [PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> repr(sys.getfilesystemencoding()) 'None' Returning None here is fine according to the docs, but the above code snippet doesn't handle the TypeError, which it gets when doing path.decode(None). How this could work? On pypy 2.6 on the same machine: Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jun 12 2015, 19:25:58) [PyPy 2.6.0] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>>> 'test'.decode(None) u'test' No exception! On pypy 4.0.1: Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Dec 10 2015, 01:17:03) [PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>>> 'test'.decode(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: expected string, got NoneType object Unhandled TypeError exception! python 2.7 works the same way: >>> 'test'.decode(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: decode() argument 1 must be string, not None Which installs setuptools just fine (although it doesn't get to the above snippet, because it doesn't return None to sys.getfilesystemencoding()). If I set fs_enc to None in the above snippet, even python 2.7 fails. So the key here (if I'm not completely lost) seems to be not returning None to the getfilesystemencoding to fix broken(?) software. What do you think about this? Thanks,

1 1

using python-ldap under pypy
by Elmir Jagudin 20 Jan '16

20 Jan '16

Hi I'm trying to run an application under pypy that authenticates user with LDAP. It is using python-ldap module and it fails to lookup the users. The problem is in python-ldap's c extension code. When it converts the LDAP search query from python format to C, parts of the query are corrupted. Is python-ldap supposed to work under pypy? How compatible is the python C API between cpython and pypy? Right now I can't figure out if this is a bug in python-ldap code or an compatibility with Pypy C API. Regards, Elmir Jagudin

3 7

s390x the last failing tests
by Richard Plangger 20 Jan '16

20 Jan '16

hi, I wanted to give a quick update on the state of the implementation. Good news! I think there is not that much left to be done! I'm currently waiting for a bigger VM (already wrote an email to linux1(a)us.ibm.com, 2 days ago? They are maybe on holiday?) to translate the full project. There are approx. 20 Failing tests that are left (own-linux-s390x). All other pass on my virtual machine. They are mostly related to big endian issues. Here are some questions: 1) Generally I got the impression that there are some tests that do not consider endianess (e.g. micronumpy). I guess it is time to change them to handle this? What about PPC? Did those not come up there? 2) It seems that the gcc on the build bot is quite old? It can for instance not assemble the instruction LAY (load address), but the VM I got (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4)) is able to. As soon as I can get my hands on a Debian machine that is configured similarly I can say more (end of Jan?). Cheers, Richard

4 9

Need Assistance
by Siddhartha Gairola 17 Jan '16

17 Jan '16

Dear Developers, I am new to this community and would like to get started. I have forked the pypy repository on bitbucket and have cloned it on my local machine. Would appreciate some guidance. Thank You. Regards, Sid

1 0

More Push towards Python 3.x?
by Phyo Arkar 15 Jan '16

15 Jan '16

Pypy, Python 3.x is lagging behind a lot. Python 3.5 is a version we should be Evolving to so Is there any planned timeline for Pypy Python 3.x ?

6 8