On Tue, May 24, 2011 at 3:46 PM, <wlavrijsen@lbl.gov> wrote:

Hi Maciej,

On Tue, May 24, 2011 at 7:09 AM, <wlavrijsen@lbl.gov> wrote:

...

...

running into the following traceback when translating. Any ideas? My machine too old to have certain features? Thanks!

Should be fixed now, sorry

cool, now it translates again!

Btw., the reason to move to head of default was b/c my translated pypy-c was crashing on the spot. The binary download of 1.5 and older locally translated pypy's didn't do that.

Turns out there is something with my history file: the crash happens when it's being loaded by readline:

Python 2.7.1 (9113640a83ac+8d950d4e5c98+, May 24 2011, 20:01:17) [PyPy 1.5.0-alpha0 with GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. using my private settings ...

Program received signal SIGSEGV, Segmentation fault. 0x08aae11c in pypy_g_OptHeap_emitting_operation ()

By playing a little, I find that the history file is problematic if it contains 1000 lines or more. Having 999 lines or less is fine either way.

Doesn't matter for me now that I know, but it's rather odd.

Best regards, Wim

-- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev

Given that 1000 is the default cutoff for the JIT, and the gdb backtrace you have, it's pretty clear that there's a crash during compilation, and that readline loops over the lines in the history file, and hitting that 1000th line triggers the compilation. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero

Hi Wim, On 25/05/11 01:16, wlavrijsen@lbl.gov wrote:

Okay, then I can't live with it. :}

Not sure to understand: do you get a crash even now that you merged default? If this is the case, do you get it even with a trunk version of pypy? You don't have to translate it by yourself, you can just download one of the nightly builds: http://buildbot.pypy.org/nightly/trunk/

Any idea how to start debugging this?

since you get the crash during the optimieopt phase, you can try to selectively enable/disable the various optimization passes, but using --jit enable_opts, e.g.: ./pypy-c --jit enable_opts=intbounds:rewrite:virtualize:string:heap:ffi:unroll the line above is equivalent of a plain pypy-c, because it enables all optimizations. Try to remove some of them and see if you still get the crash. ciao, Anto

Anto,

./pypy-c --jit enable_opts=intbounds:rewrite:virtualize:string:heap:ffi:unroll

the line above is equivalent of a plain pypy-c, because it enables all optimizations. Try to remove some of them and see if you still get the crash.

there are several combinations that work with the pre-built pypy but not with my local version (except one: if only intbounds is selected, neither works). The largest set of combinations that does work is: intbounds:rewrite:virtualize:ffi Several others don't crash, but raise (and again, not in the pre-built): RPython traceback: File "implement_18.c", line 205492, in optimize_loop File "implement_19.c", line 86345, in _optimize_loop File "implement_19.c", line 202071, in UnrollOptimizer_propagate_all_forward File "implement_19.c", line 248573, in Optimizer_reconstruct_for_next_iteration File "implement_19.c", line 247560, in Optimization_reconstruct_for_next_iteration Fatal RPython error: NotImplementedError Does any of that ring a bell? Changing gcc didn't work. For now, I'll live with it with the lower set of options. It may after all still be something specific to my box which I intend to replace sometime soon. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Maciej,

This is relatively known and yet not fixed :) some operations are unacceptable by backend and removed by optimizations, which in turn lead to crashes if those optimizations are disabled.

ah. :) Thanks. I'll live with it. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Armin,

As a summary, can you tell us if there is something wrong with the trunk version of PyPy, as compiled by your GCC?

I've tried different versions of GCC (4.4 and 4.5): in both cases do I get a pypy-c that crashes once the jit is invoked. It was just a guess, and it turns out not to be related to the version of GCC (or associated tools).

In this case it's a bug in the trunk version, which we have to fix, because we don't want to support only half the GCC configurations out there. Or does the bug only ever occur in your own branch?

I did a fresh hg clone from bitbucket to make sure it's nothing local. I find that the trunk on my machine does NOT crash, but the branch does from that same fresh clone does. There are only a few changes that are in trunk but not yet in the branch (I did a recent merge). Also, the runtime of the branch is somewhat different, as it is linked to one more library (libReflex.so). I'll update the branch once more; easy to test if that makes a difference. Also, I'll try once more on the branch with cppyy disabled. That should be identical to trunk. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi, [replying to myself]

okay, something has been fixed over the past couple of days: the readline now works! :) I'll move up to that then.

too fast ... it sometimes works, and when it doesn't, it sometimes does under gdb or valgrind. Sometimes if I start a new shell, it works, then stops working again for another. (Although the downloaded pypy-c's have never crashed, mind.) This would make me think that there's an uninitialized variable somewhere. Is it possible to create such a thing with a buggy RPython construct? That said, the results with gdb or valgrind are always the same, when they do show the crash: Valgrind: ==16631== Process terminating with default action of signal 11 (SIGSEGV) ==16631== Access not within mapped region at address 0x4 ==16631== at 0x8AADB0C: pypy_g_OptHeap_emitting_operation (in /home/wlav/pypydev/pypy/pypy/translator/goal/pypy-c) gdb: 0x8aadb0c <pypy_g_OptHeap_emitting_operation+604>: cmp 0x4(%edi),%eax 0x8aadb0f <pypy_g_OptHeap_emitting_operation+607>: jl 0x8aadb37 <pypy_g_OptHeap_emitting_operation+647> 0x8aadb11 <pypy_g_OptHeap_emitting_operation+609>: jmp 0x8aadbbf <pypy_g_OptHeap_emitting_operation+783> Whereas I'd expect more variation with an uninitialized variable. Valgrind does not report any uninitialized variables either (other than PyObject_Free, which I'll presume is there for the same reason as for CPython (?)): When I select only the jit options that will not give me a segfault and then run under valgrind, this is the only complaint I get (other than PyObject_Free): ==16681== Address 0x4b18010 is 3,352 bytes inside a block of size 4,100 free'd ==16681== at 0x40268A6: free (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==16681== by 0x811282C: pypy_g__ll_dict_setitem_lookup_done__DICTPtr_Address_Ad (in /home/wlav/pypydev/pypy/pypy/translator/goal/pypy-c) Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Armin,

It definitely crashes during optimization of the trace produced by the JIT. I imagine that we could get somewhere by carefully inspecting the unoptimized operations that lead to the crash. Try enabling the dump with "PYPYLOG=jit-log-unopt:log", and then reproducing the crash. This should ideally leave behind a file called "log" with all the unoptimized loops seen so far, and the faulty one is the last one (i.e. the last section between "{jit-log-unopt" and "jit-log-unopt}").

thanks; but to get something useful, I had to use jit-log-noopt: (i.e. output to stderr, since buffers were not flushed; and slightly different filter). Since readline in PyPy is a .py, I was able to reduce the loop that exhibits a segfault to this simple script: for i in range(1001): unicode("aap") although the trace is still far from simple, at least it's a lot shorter than what I got before. :) I've attached the trace.log file. And again, the problem optimization was "heap." For comparison purposes, I grabbed the latest "green" nightly (44610) and made a trace with that one, too (the prebuild editions don't crash). That file is trace_nightly.log (also attached). It is quite a bit shorter. Anything in there that rings a bell? Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

On Tue, May 24, 2011 at 3:46 PM, <wlavrijsen@lbl.gov> wrote:

Hi Maciej,

On Tue, May 24, 2011 at 7:09 AM, <wlavrijsen@lbl.gov> wrote:

...

...

running into the following traceback when translating. Any ideas? My machine too old to have certain features? Thanks!

Should be fixed now, sorry

cool, now it translates again!

Btw., the reason to move to head of default was b/c my translated pypy-c was crashing on the spot. The binary download of 1.5 and older locally translated pypy's didn't do that.

Turns out there is something with my history file: the crash happens when it's being loaded by readline:

Python 2.7.1 (9113640a83ac+8d950d4e5c98+, May 24 2011, 20:01:17) [PyPy 1.5.0-alpha0 with GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. using my private settings ...

Program received signal SIGSEGV, Segmentation fault. 0x08aae11c in pypy_g_OptHeap_emitting_operation ()

By playing a little, I find that the history file is problematic if it contains 1000 lines or more. Having 999 lines or less is fine either way.

Doesn't matter for me now that I know, but it's rather odd.

Best regards, Wim

-- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev

Given that 1000 is the default cutoff for the JIT, and the gdb backtrace you have, it's pretty clear that there's a crash during compilation, and that readline loops over the lines in the history file, and hitting that 1000th line triggers the compilation. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero

Hi Wim, On 25/05/11 01:16, wlavrijsen@lbl.gov wrote:

Okay, then I can't live with it. :}

Not sure to understand: do you get a crash even now that you merged default? If this is the case, do you get it even with a trunk version of pypy? You don't have to translate it by yourself, you can just download one of the nightly builds: http://buildbot.pypy.org/nightly/trunk/

Any idea how to start debugging this?

since you get the crash during the optimieopt phase, you can try to selectively enable/disable the various optimization passes, but using --jit enable_opts, e.g.: ./pypy-c --jit enable_opts=intbounds:rewrite:virtualize:string:heap:ffi:unroll the line above is equivalent of a plain pypy-c, because it enables all optimizations. Try to remove some of them and see if you still get the crash. ciao, Anto

Anto,

./pypy-c --jit enable_opts=intbounds:rewrite:virtualize:string:heap:ffi:unroll

the line above is equivalent of a plain pypy-c, because it enables all optimizations. Try to remove some of them and see if you still get the crash.

there are several combinations that work with the pre-built pypy but not with my local version (except one: if only intbounds is selected, neither works). The largest set of combinations that does work is: intbounds:rewrite:virtualize:ffi Several others don't crash, but raise (and again, not in the pre-built): RPython traceback: File "implement_18.c", line 205492, in optimize_loop File "implement_19.c", line 86345, in _optimize_loop File "implement_19.c", line 202071, in UnrollOptimizer_propagate_all_forward File "implement_19.c", line 248573, in Optimizer_reconstruct_for_next_iteration File "implement_19.c", line 247560, in Optimization_reconstruct_for_next_iteration Fatal RPython error: NotImplementedError Does any of that ring a bell? Changing gcc didn't work. For now, I'll live with it with the lower set of options. It may after all still be something specific to my box which I intend to replace sometime soon. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Maciej,

This is relatively known and yet not fixed :) some operations are unacceptable by backend and removed by optimizations, which in turn lead to crashes if those optimizations are disabled.

ah. :) Thanks. I'll live with it. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Armin,

As a summary, can you tell us if there is something wrong with the trunk version of PyPy, as compiled by your GCC?

I've tried different versions of GCC (4.4 and 4.5): in both cases do I get a pypy-c that crashes once the jit is invoked. It was just a guess, and it turns out not to be related to the version of GCC (or associated tools).

In this case it's a bug in the trunk version, which we have to fix, because we don't want to support only half the GCC configurations out there. Or does the bug only ever occur in your own branch?

I did a fresh hg clone from bitbucket to make sure it's nothing local. I find that the trunk on my machine does NOT crash, but the branch does from that same fresh clone does. There are only a few changes that are in trunk but not yet in the branch (I did a recent merge). Also, the runtime of the branch is somewhat different, as it is linked to one more library (libReflex.so). I'll update the branch once more; easy to test if that makes a difference. Also, I'll try once more on the branch with cppyy disabled. That should be identical to trunk. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi, [replying to myself]

okay, something has been fixed over the past couple of days: the readline now works! :) I'll move up to that then.

too fast ... it sometimes works, and when it doesn't, it sometimes does under gdb or valgrind. Sometimes if I start a new shell, it works, then stops working again for another. (Although the downloaded pypy-c's have never crashed, mind.) This would make me think that there's an uninitialized variable somewhere. Is it possible to create such a thing with a buggy RPython construct? That said, the results with gdb or valgrind are always the same, when they do show the crash: Valgrind: ==16631== Process terminating with default action of signal 11 (SIGSEGV) ==16631== Access not within mapped region at address 0x4 ==16631== at 0x8AADB0C: pypy_g_OptHeap_emitting_operation (in /home/wlav/pypydev/pypy/pypy/translator/goal/pypy-c) gdb: 0x8aadb0c <pypy_g_OptHeap_emitting_operation+604>: cmp 0x4(%edi),%eax 0x8aadb0f <pypy_g_OptHeap_emitting_operation+607>: jl 0x8aadb37 <pypy_g_OptHeap_emitting_operation+647> 0x8aadb11 <pypy_g_OptHeap_emitting_operation+609>: jmp 0x8aadbbf <pypy_g_OptHeap_emitting_operation+783> Whereas I'd expect more variation with an uninitialized variable. Valgrind does not report any uninitialized variables either (other than PyObject_Free, which I'll presume is there for the same reason as for CPython (?)): When I select only the jit options that will not give me a segfault and then run under valgrind, this is the only complaint I get (other than PyObject_Free): ==16681== Address 0x4b18010 is 3,352 bytes inside a block of size 4,100 free'd ==16681== at 0x40268A6: free (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==16681== by 0x811282C: pypy_g__ll_dict_setitem_lookup_done__DICTPtr_Address_Ad (in /home/wlav/pypydev/pypy/pypy/translator/goal/pypy-c) Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Hi Armin,

It definitely crashes during optimization of the trace produced by the JIT. I imagine that we could get somewhere by carefully inspecting the unoptimized operations that lead to the crash. Try enabling the dump with "PYPYLOG=jit-log-unopt:log", and then reproducing the crash. This should ideally leave behind a file called "log" with all the unoptimized loops seen so far, and the faulty one is the last one (i.e. the last section between "{jit-log-unopt" and "jit-log-unopt}").

thanks; but to get something useful, I had to use jit-log-noopt: (i.e. output to stderr, since buffers were not flushed; and slightly different filter). Since readline in PyPy is a .py, I was able to reduce the loop that exhibits a segfault to this simple script: for i in range(1001): unicode("aap") although the trace is still far from simple, at least it's a lot shorter than what I got before. :) I've attached the trace.log file. And again, the problem optimization was "heap." For comparison purposes, I grabbed the latest "green" nightly (44610) and made a trace with that one, too (the prebuild editions don't crash). That file is trace_nightly.log (also attached). It is quite a bit shorter. Anything in there that rings a bell? Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net