Segmentation fault at rpyvmprof_f_pypy_rrr ()
Hi, We observed the program was cored dump. And examing the core file, we found the call stack: #0 0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from /usr/local/pypy/libpypy-c.so #1 0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from /usr/local/pypy/libpypy-c.so #2 0x00007f26487a1ad4 in pypy_g_appexec___src__glob___________________import_sys () from /usr/local/pypy/libpypy-c.so #3 0x00007f26486f980b in pypy_g.pypy_execute_source () from /usr/local/pypy/libpypy-c.so #4 0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #5 0x00007f26484e6f42 in pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #6 0x00007f26484e6d82 in pypy_execute_source () from /usr/local/pypy/libpypy-c.so ... And debugging the core file, the char* buffer passed to pypy_execute_source() was: (gdb) print pyExBuffer $1 = "import sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"... The code was to add a directory to sys.path(). The error was not reproduceable by extracting the piece of code from the whole programme, and I've no ideas why the segmentation fault happened. Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And are there any methods to debug and found out the cause?
Can you please file a bug on either pypy or vmprof-python? On Mon, Aug 31, 2015 at 11:00 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
Hi,
We observed the program was cored dump. And examing the core file, we found the call stack:
#0 0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from /usr/local/pypy/libpypy-c.so #1 0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from /usr/local/pypy/libpypy-c.so #2 0x00007f26487a1ad4 in pypy_g_appexec___src__glob___________________import_sys () from /usr/local/pypy/libpypy-c.so #3 0x00007f26486f980b in pypy_g.pypy_execute_source () from /usr/local/pypy/libpypy-c.so #4 0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #5 0x00007f26484e6f42 in pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #6 0x00007f26484e6d82 in pypy_execute_source () from /usr/local/pypy/libpypy-c.so ...
And debugging the core file, the char* buffer passed to pypy_execute_source() was: (gdb) print pyExBuffer $1 = "import sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...
The code was to add a directory to sys.path(). The error was not reproduceable by extracting the piece of code from the whole programme, and I've no ideas why the segmentation fault happened.
Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And are there any methods to debug and found out the cause?
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
I am not sure whether it is a bug of pypy. I tried extracting the code from the whole programme, but it worked correctly. rpython_startup_code(); int res = pypy_setup_home("/usr/local/pypy/libpypy-c.so", 1); char *b = "import sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\00010260"; pypy_execute_source(b); I will do some further experiments. And if there are any methods to debug, I could follow. Thanks! On Mon, Aug 31, 2015 at 5:15 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Can you please file a bug on either pypy or vmprof-python?
On Mon, Aug 31, 2015 at 11:00 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
Hi,
We observed the program was cored dump. And examing the core file, we found the call stack:
#0 0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from /usr/local/pypy/libpypy-c.so #1 0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from /usr/local/pypy/libpypy-c.so #2 0x00007f26487a1ad4 in pypy_g_appexec___src__glob___________________import_sys () from /usr/local/pypy/libpypy-c.so #3 0x00007f26486f980b in pypy_g.pypy_execute_source () from /usr/local/pypy/libpypy-c.so #4 0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #5 0x00007f26484e6f42 in pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #6 0x00007f26484e6d82 in pypy_execute_source () from /usr/local/pypy/libpypy-c.so ...
And debugging the core file, the char* buffer passed to pypy_execute_source() was: (gdb) print pyExBuffer $1 = "import
sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...
The code was to add a directory to sys.path(). The error was not reproduceable by extracting the piece of code from the whole programme, and I've no ideas why the segmentation fault happened.
Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And
are
there any methods to debug and found out the cause?
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Unfortunatelly, I still could not make a simple case to reproduce the issue. However, we observed the same issue happen several times and core dump generated in the application these days. I used gdb to show the disassemble: (gdb) disassemble /rm Dump of assembler code for function rpyvmprof_f_pypy_rrr: 0x00007f27ff816b40 <+0>: 41 57 push %r15 0x00007f27ff816b42 <+2>: 41 56 push %r14 0x00007f27ff816b44 <+4>: 41 55 push %r13 0x00007f27ff816b46 <+6>: 41 54 push %r12 0x00007f27ff816b48 <+8>: 49 89 d5 mov %rdx,%r13 0x00007f27ff816b4b <+11>: 55 push %rbp 0x00007f27ff816b4c <+12>: 53 push %rbx 0x00007f27ff816b4d <+13>: 49 89 f4 mov %rsi,%r12 0x00007f27ff816b50 <+16>: 48 89 fb mov %rdi,%rbx 0x00007f27ff816b53 <+19>: 48 83 ec 18 sub $0x18,%rsp 0x00007f27ff816b57 <+23>: e8 74 ee b6 00 callq 0x7f28003859d0 <pypy_g_stack_check___> 0x00007f27ff816b5c <+28>: 48 83 3d 1c 66 94 02 00 cmpq $0x0,0x294661c(%rip) # 0x7f280215d180 <pypy_g_ExcData> 0x00007f27ff816b64 <+36>: 0f 85 36 01 00 00 jne 0x7f27ff816ca0 <rpyvmprof_f_pypy_rrr+352> 0x00007f27ff816b6a <+42>: 66 48 8d 3d 8e b3 2f 01 data16 lea 0x12fb38e(%rip),%rdi # 0x7f2800b11f00 0x00007f27ff816b72 <+50>: 66 66 48 e8 6e ec cf ff data16 data16 callq 0x7f27ff5157e8 <__tls_get_addr@plt> 0x00007f27ff816b7a <+58>: f6 43 04 01 testb $0x1,0x4(%rbx) 0x00007f27ff816b7e <+62>: 48 8b 68 30 mov 0x30(%rax),%rbp => 0x00007f27ff816b82 <+66>: 4c 8b 75 48 mov 0x48(%rbp),%r14 0x00007f27ff816b86 <+70>: 0f 85 4c 01 00 00 jne 0x7f27ff816cd8 <rpyvmprof_f_pypy_rrr+408> 0x00007f27ff816b8c <+76>: f6 45 04 01 testb $0x1,0x4(%rbp) 0x00007f27ff816b90 <+80>: 4c 89 73 18 mov %r14,0x18(%rbx) 0x00007f27ff816b94 <+84>: 0f 85 2e 01 00 00 jne 0x7f27ff816cc8 <rpyvmprof_f_pypy_rrr+392> 0x00007f27ff816b9a <+90>: 48 89 5d 48 mov %rbx,0x48(%rbp) 0x00007f27ff816b9e <+94>: 48 89 de mov %rbx,%rsi 0x00007f27ff816ba1 <+97>: 48 89 ef mov %rbp,%rdi 0x00007f27ff816ba4 <+100>: e8 f7 ae fd ff callq 0x7f27ff7f1aa0 <pypy_g_ExecutionContext_call_trace> 0x00007f27ff816ba9 <+105>: 4c 8b 3d d0 65 94 02 mov 0x29465d0(%rip),%r15 # 0x7f280215d180 <pypy_g_ExcData> 0x00007f27ff816bb0 <+112>: 4c 89 ea mov %r13,%rdx 0x00007f27ff816bb3 <+115>: 4d 85 ff test %r15,%r15 0x00007f27ff816bb6 <+118>: 49 89 dd mov %rbx,%r13 0x00007f27ff816bb9 <+121>: 49 89 ee mov %rbp,%r14 0x00007f27ff816bbc <+124>: 0f 84 5e 03 00 00 je 0x7f27ff816f20 <rpyvmprof_f_pypy_rrr+992> 0x00007f27ff816bc2 <+130>: 48 63 15 63 0e 95 02 movslq 0x2950e63(%rip),%rdx # 0x7f2802167a2c <pypydtcount> And the content of regesters: (gdb) i r rax 0x468917e8 1183389672 rbx 0x4df8bd0 81759184 rcx 0x7000000000000a60 8070450532247931488 rdx 0x14 20 rsi 0x0 0 rdi 0x7f2800b11f00 139809787027200 rbp 0x0 0x0 rsp 0x4688cc40 0x4688cc40 r8 0x7f2800f57d80 139809791507840 r9 0x100000000 4294967296 r10 0x4df8b80 81759104 r11 0x3472a7aeee 225261891310 r12 0x0 0 r13 0x0 0 r14 0x0 0 r15 0x4688ce50 1183370832 rip 0x7f27ff816b82 0x7f27ff816b82 <rpyvmprof_f_pypy_rrr+66> eflags 0x10246 [ PF ZF IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x63 99 gs 0x0 0 I searched for "rpyvmprof_f" and found the matched py file in pypy "rpython/rlib/rvmprof/cintf.py". Is this the matched code where the issue happened? And what does these code do? On Mon, Aug 31, 2015 at 5:41 PM, Yicong Huang <hengha.mao@gmail.com> wrote:
I am not sure whether it is a bug of pypy. I tried extracting the code from the whole programme, but it worked correctly.
rpython_startup_code(); int res = pypy_setup_home("/usr/local/pypy/libpypy-c.so", 1); char *b = "import sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\00010260"; pypy_execute_source(b);
I will do some further experiments. And if there are any methods to debug, I could follow.
Thanks!
On Mon, Aug 31, 2015 at 5:15 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Can you please file a bug on either pypy or vmprof-python?
On Mon, Aug 31, 2015 at 11:00 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
Hi,
We observed the program was cored dump. And examing the core file, we found the call stack:
#0 0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from /usr/local/pypy/libpypy-c.so #1 0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from /usr/local/pypy/libpypy-c.so #2 0x00007f26487a1ad4 in pypy_g_appexec___src__glob___________________import_sys () from /usr/local/pypy/libpypy-c.so #3 0x00007f26486f980b in pypy_g.pypy_execute_source () from /usr/local/pypy/libpypy-c.so #4 0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #5 0x00007f26484e6f42 in pypy_execute_source_ptr () from /usr/local/pypy/libpypy-c.so #6 0x00007f26484e6d82 in pypy_execute_source () from /usr/local/pypy/libpypy-c.so ...
And debugging the core file, the char* buffer passed to pypy_execute_source() was: (gdb) print pyExBuffer $1 = "import
sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...
The code was to add a directory to sys.path(). The error was not reproduceable by extracting the piece of code from the whole programme, and I've no ideas why the segmentation fault happened.
Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And
are
there any methods to debug and found out the cause?
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Yicong, On Tue, Sep 8, 2015 at 5:07 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
0x00007f27ff816b6a <+42>: 66 48 8d 3d 8e b3 2f 01 data16 lea 0x12fb38e(%rip),%rdi # 0x7f2800b11f00 0x00007f27ff816b72 <+50>: 66 66 48 e8 6e ec cf ff data16 data16 callq 0x7f27ff5157e8 <__tls_get_addr@plt> 0x00007f27ff816b7e <+62>: 48 8b 68 30 mov 0x30(%rax),%rbp => 0x00007f27ff816b82 <+66>: 4c 8b 75 48 mov 0x48(%rbp),%r14
The first two instructions are reading the TLS (thread-local storage); then the 3rd is mostly probably reading the current executioncontext; then the 4th is trying to read something out of the executioncontext---which is NULL at this point. I may be wrong, but I think that this crash can only occur in these two cases: if rpython_startup_code() was not called yet (which should not be the case here), or if the crash occurs in a different thread and pypy_thread_attach() was not called yet in this thread. You should be able to check if the program was running multiple threads by doing "info threads" in gdb at the point of the crash. If it is, did you call pypy_init_threads() once from the main thread and then pypy_thread_attach() in that other thread? A bientôt, Armin.
Here are the output of gdb "info threads". We didn't call "pypy_init_threads()" or "pypy_thread_attach()" in the code. (gdb) info threads Id Target Id Frame 132 Thread 0x414ea940 (LWP 32385) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 131 Thread 0x413e9940 (LWP 32384) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 130 Thread 0x4025c940 (LWP 32382) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 129 Thread 0x45f88940 (LWP 32455) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 128 Thread 0x43760940 (LWP 32412) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 127 Thread 0x43861940 (LWP 32413) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 126 Thread 0x4648d940 (LWP 32460) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 125 Thread 0x4365f940 (LWP 32409) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 124 Thread 0x46089940 (LWP 32456) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 123 Thread 0x46f98940 (LWP 32716) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 122 Thread 0x45c85940 (LWP 32452) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 121 Thread 0x41157940 (LWP 32381) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 120 Thread 0x4517a940 (LWP 32440) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 119 Thread 0x47099940 (LWP 32717) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 118 Thread 0x4234c940 (LWP 32390) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 117 Thread 0x4035d940 (LWP 32386) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 116 Thread 0x42a53940 (LWP 32397) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 115 Thread 0x41b6b940 (LWP 32380) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 114 Thread 0x44f78940 (LWP 32438) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 113 Thread 0x4719a940 (LWP 32718) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 112 Thread 0x4224b940 (LWP 32388) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 111 Thread 0x4254e940 (LWP 32392) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 110 Thread 0x4658e940 (LWP 32461) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 109 Thread 0x42b54940 (LWP 32398) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 108 Thread 0x45b84940 (LWP 32451) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 107 Thread 0x4729b940 (LWP 32719) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 106 Thread 0x41e56940 (LWP 32389) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 105 Thread 0x42f58940 (LWP 32402) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 104 Thread 0x4264f940 (LWP 32393) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 103 Thread 0x4668f940 (LWP 32462) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 102 Thread 0x45a83940 (LWP 32449) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 101 Thread 0x4739c940 (LWP 32720) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 100 Thread 0x4315a940 (LWP 32404) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- 99 Thread 0x42851940 (LWP 32395) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 98 Thread 0x45881940 (LWP 32447) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 97 Thread 0x4749d940 (LWP 32721) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 96 Thread 0x46b94940 (LWP 32712) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 95 Thread 0x4345d940 (LWP 32407) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 94 Thread 0x46790940 (LWP 2815) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 93 Thread 0x44e77940 (LWP 32437) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 92 Thread 0x4355e940 (LWP 32408) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 91 Thread 0x46c95940 (LWP 32713) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 90 Thread 0x4759e940 (LWP 32722) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 89 Thread 0x44d76940 (LWP 32436) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 88 Thread 0x46d96940 (LWP 32714) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 87 Thread 0x4769f940 (LWP 32723) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 86 Thread 0x44a73940 (LWP 32433) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 85 Thread 0x4214a940 (LWP 32377) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 84 Thread 0x477a0940 (LWP 32724) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 83 Thread 0x46e97940 (LWP 32715) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 82 Thread 0x44972940 (LWP 32432) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 81 Thread 0x478a1940 (LWP 32725) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 80 Thread 0x44871940 (LWP 32431) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 79 Thread 0x479a2940 (LWP 32726) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 78 Thread 0x47aa3940 (LWP 32727) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 77 Thread 0x47ba4940 (LWP 32728) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 76 Thread 0x47ca5940 (LWP 32729) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 75 Thread 0x47fa8940 (LWP 2188) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 74 Thread 0x47da6940 (LWP 32730) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 73 Thread 0x480a9940 (LWP 2189) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 72 Thread 0x4527b940 (LWP 32441) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 71 Thread 0x485ae940 (LWP 2195) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 70 Thread 0x4537c940 (LWP 32442) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 69 Thread 0x4618a940 (LWP 32457) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 68 Thread 0x45982940 (LWP 32448) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 67 Thread 0x484ad940 (LWP 2194) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 66 Thread 0x486af940 (LWP 2196) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- 65 Thread 0x4325b940 (LWP 32405) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 64 Thread 0x40ced940 (LWP 32375) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 63 Thread 0x41056940 (LWP 32370) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 62 Thread 0x41ced940 (LWP 32450) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 61 Thread 0x46992940 (LWP 32710) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 60 Thread 0x40772940 (LWP 3069) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 59 Thread 0x483ac940 (LWP 2193) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 58 Thread 0x4628b940 (LWP 32458) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 57 Thread 0x42049940 (LWP 32376) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 56 Thread 0x4048e940 (LWP 32373) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 55 Thread 0x488b1940 (LWP 3071) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 54 Thread 0x46a93940 (LWP 32711) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 53 Thread 0x44770940 (LWP 32430) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 52 Thread 0x42e57940 (LWP 32401) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 51 Thread 0x481aa940 (LWP 2190) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 50 Thread 0x4446d940 (LWP 32425) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 49 Thread 0x43f68940 (LWP 32420) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 48 Thread 0x40dee940 (LWP 32378) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 47 Thread 0x4567f940 (LWP 32445) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 46 Thread 0x43a63940 (LWP 32415) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 45 Thread 0x45e87940 (LWP 32454) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 44 Thread 0x487b0940 (LWP 2197) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 43 Thread 0x41969940 (LWP 32369) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 42 Thread 0x40eef940 (LWP 32379) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 41 Thread 0x41868940 (LWP 32351) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 40 Thread 0x45079940 (LWP 32439) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 39 Thread 0x4638c940 (LWP 32459) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 38 Thread 0x4466f940 (LWP 32429) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 37 Thread 0x4436c940 (LWP 32424) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 36 Thread 0x47ea7940 (LWP 2187) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 35 Thread 0x43059940 (LWP 32403) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 34 Thread 0x4416a940 (LWP 32422) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 33 Thread 0x482ab940 (LWP 2191) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 32 Thread 0x43d66940 (LWP 32418) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- 31 Thread 0x42c55940 (LWP 32399) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 30 Thread 0x45d86940 (LWP 32453) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 29 Thread 0x45780940 (LWP 32446) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 28 Thread 0x4547d940 (LWP 32443) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 27 Thread 0x44b74940 (LWP 32434) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 26 Thread 0x4557e940 (LWP 32444) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 25 Thread 0x40a40940 (LWP 32352) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 24 Thread 0x412e8940 (LWP 32350) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23 Thread 0x44c75940 (LWP 32435) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 22 Thread 0x7f27f911d8c0 (LWP 31988) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 21 Thread 0x40b41940 (LWP 32368) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 20 Thread 0x41c6c940 (LWP 32383) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 19 Thread 0x4456e940 (LWP 32426) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 18 Thread 0x4426b940 (LWP 32423) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 17 Thread 0x43e67940 (LWP 32419) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 16 Thread 0x44069940 (LWP 32421) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 15 Thread 0x43c65940 (LWP 32417) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 14 Thread 0x4244d940 (LWP 32391) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 13 Thread 0x43b64940 (LWP 32416) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 12 Thread 0x43962940 (LWP 32414) 0x000000347320af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 11 Thread 0x42d56940 (LWP 32400) 0x000000347320cd91 in sem_wait () from /lib64/libpthread.so.0 10 Thread 0x4335c940 (LWP 32406) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 9 Thread 0x42952940 (LWP 32396) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 8 Thread 0x42750940 (LWP 32394) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 7 Thread 0x415eb940 (LWP 32387) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 6 Thread 0x41a6a940 (LWP 32374) 0x0000003472ad4d98 in epoll_wait () from /lib64/libc.so.6 5 Thread 0x4093f940 (LWP 3068) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x41767940 (LWP 32346) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x40671940 (LWP 32345) 0x0000003472a9a901 in nanosleep () from /lib64/libc.so.6 2 Thread 0x4015b940 (LWP 3070) 0x000000347320b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x46891940 (LWP 3067) 0x00007f27ff816b82 in rpyvmprof_f_pypy_rrr () from /usr/local/pypy/libpypy-c.so On Tue, Sep 8, 2015 at 2:33 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong,
On Tue, Sep 8, 2015 at 5:07 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
0x00007f27ff816b6a <+42>: 66 48 8d 3d 8e b3 2f 01 data16 lea 0x12fb38e(%rip),%rdi # 0x7f2800b11f00 0x00007f27ff816b72 <+50>: 66 66 48 e8 6e ec cf ff data16 data16 callq 0x7f27ff5157e8 <__tls_get_addr@plt> 0x00007f27ff816b7e <+62>: 48 8b 68 30 mov 0x30(%rax),%rbp => 0x00007f27ff816b82 <+66>: 4c 8b 75 48 mov 0x48(%rbp),%r14
The first two instructions are reading the TLS (thread-local storage); then the 3rd is mostly probably reading the current executioncontext; then the 4th is trying to read something out of the executioncontext---which is NULL at this point.
I may be wrong, but I think that this crash can only occur in these two cases: if rpython_startup_code() was not called yet (which should not be the case here), or if the crash occurs in a different thread and pypy_thread_attach() was not called yet in this thread.
You should be able to check if the program was running multiple threads by doing "info threads" in gdb at the point of the crash. If it is, did you call pypy_init_threads() once from the main thread and then pypy_thread_attach() in that other thread?
A bientôt,
Armin.
Hi, On Tue, Sep 8, 2015 at 9:20 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
Here are the output of gdb "info threads". We didn't call "pypy_init_threads()" or "pypy_thread_attach()" in the code.
Well, then. Read again the documentation for these two functions: you need to call them if your program is going to call Python code in more than just one thread. Armin
Great thanks Armin! I found out we did call python code from different theads. Here are my understanding, please correct me if I am wrong: In main thread: rpython_startup_code(); pypy_init_threads(); res = pypy_setup_home(..); And in other worker thread, after main thread do the initilization: pypy_thread_attach(); Will there any problems if main thread exit before other worker thread do the initilization "pypy_thread_attach()"? On Tue, Sep 8, 2015 at 4:12 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Tue, Sep 8, 2015 at 9:20 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
Here are the output of gdb "info threads". We didn't call "pypy_init_threads()" or "pypy_thread_attach()" in the code.
Well, then. Read again the documentation for these two functions: you need to call them if your program is going to call Python code in more than just one thread.
Armin
Hi, On Tue, Sep 8, 2015 at 11:44 AM, Yicong Huang <hengha.mao@gmail.com> wrote:
In main thread: rpython_startup_code(); pypy_init_threads(); res = pypy_setup_home(..);
And in other worker thread, after main thread do the initilization: pypy_thread_attach();
Yes, that's how it should be.
Will there any problems if main thread exit before other worker thread do the initilization "pypy_thread_attach()"?
No, it should not be a problem. Note that every pypy_thread_attach() call leaks a bit of memory right now. If your program has a lot of threads that exit and new threads that replace them, then you'll really have a leak. I hope it is not the case for you. Otherwise, we need to think about adding some function like "pypy_thread_detach()"... A bientôt, Armin.
Yes. And if rpython_startup_code() and pypy_setup_home() was not called before, the core dump function would be at: pypy_g_IncrementalMiniMarkGC_collect_and_reserve () On Mon, Aug 31, 2015 at 5:19 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong,
On 31 August 2015 at 11:00, Yicong Huang <hengha.mao@gmail.com> wrote:
pypy_execute_source()
Did you call rpython_startup_code() and pypy_setup_home() before?
Armin
participants (3)
-
Armin Rigo -
Maciej Fijalkowski -
Yicong Huang