PyPy 1.6 not working on Windows XP
Hey guys, I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on both system when running the GarlicSim test suite. It shows a Windows error dialog saying "pypy.exe has encountered a problem and needs to close. We are sorry for the inconvenience." and giving this data: AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll ModVer: 5.1.2600.5512 Offset: 00040b0d I can also open a dialog with a lot of data on the error (don't know how useful it is) but Windows won't let me Ctrl-C it. What can I do? Thanks, Ram.
1. Switch to linux. It helps. 2. To get a meaningful error log try to run the pypy from terminal. To do copy-paste you will need 3-rd party terminal, i.e. power shell. Then you'll be able to copy the error messages. Without them it is pretty much impossible to identify the problem. 3. Another option is to use debugger to run the suite step-by-step and see what happens. Choose your poison. Alex. On Wednesday 05 October 2011 13:44:14 Ram Rachum wrote:
On Wed, Oct 5, 2011 at 1:49 PM, Alex Pyattaev <alex.pyattaev@gmail.com>wrote:
1. Switch to linux. It helps.
Not funny. 2. To get a meaningful error log try to run the pypy from terminal. To do
I am running PyPy from terminal, bash provided by msys. The error still comes up in a dialog and the shell contains only the output from `nose` up to the failure, with no word on the failure. 3. Another option is to use debugger to run the suite step-by-step and see
what happens.
I'll give that a try.
On Wed, Oct 5, 2011 at 2:07 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
I have hundreds of tests, and PyPy fails before a single one begins. It seems that PyPy crashes somewhere in nose's initialization. Isn't there a way to find the last Python line run before the crash without stepping with a finer granularity every time? Ram.
On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
How do I run the Nose test suite on Pypy with a debugger? I usually use Wing IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag which drops you into the debugger after an error, but it doesn't work here because this crash seems to be happening at a lower level. So I don't know how to start this in a debugger. Ram.
On Wed, Oct 5, 2011 at 2:22 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
I don't know how to use that... I don't program C at all, only Python. If I use a Python debugger, can't I just step forward line by line, see where I get the crash, and then isolate the offending line? Ram.
On 5 October 2011 13:50, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Well, in recent versions it gained the faulthandler module... :-) http://pypi.python.org/pypi/faulthandler/ Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
2011/10/5 Michael Foord <fuzzyman@gmail.com>:
Well, in recent versions it gained the faulthandler module... :-)
Yes, and this was a *great* improvement! I hope we will be able to adapt it to pypy. -- Amaury Forgeot d'Arc
On Wed, Oct 5, 2011 at 08:22, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
It may be early to start bringing this up, but I recently created an extension to catch crashes in C code and write minidumps: https://bitbucket.org/briancurtin/minidumper. It's 3.x only at the moment, although I'll be backporting within the next few days. I'll have a look at how/if it works on PyPy once I get there. (minidumper will likely be absorbed into faulthandler once it becomes more complete, which is why I mention it)
2011/10/5 Brian Curtin <brian.curtin@gmail.com>:
The code is quite simple, and should probably compile as is with pypy (except for the PyModuleDef of course) but it would be as simple to rewrite it in RPython. btw there is a suspect thing in your code: "app_name = pyname" will store the char* buffer into a static variable, but nothing guarantees that the PyObject passed will stay alive! -- Amaury Forgeot d'Arc
Generally, any binary-level debugger such as gdb or MSVC should work with pypy. At the very least you will find which operation crashed. If it is something really specific, for example sin/log/sign, it might be quite easy to map it back to python code. If it is not, it will be nearly impossible to find the original source line (at least I've failed when I tried). Another option is to edit the sources of the test suite adding print statements incrementally until you spot the place where it crashes. It is a slow, but very reliable way. That is of course if it is a particular segment of python code that crashes it. Also, could you send your exact environtment specs? I'll try to replicate it on a VM and see if it crashes for me too. I have an XP VM somewhere. PS: Sorry for my stupid joke about switching to linux. It was meant to cheer you up a bit. Alex. On Wednesday 05 October 2011 14:18:08 Ram Rachum wrote:
On Wed, Oct 5, 2011 at 2:23 PM, Alex Pyattaev <alex.pyattaev@gmail.com>wrote:
Generally, any binary-level debugger such as gdb or MSVC should work with pypy. At the very least you will find which operation crashed.
As I said to Amaury, I don't know how to use those... Python is the only language I program in.
I'll try, thanks.
What specs do you mean? It's just the recent PyPy 1.6 on a Windows XP SP3 32 bit machine with minimal packages installed. (distribute, pip, nosetests.) Let me know if you need any more data.
Forgiven :)
On Wed, Oct 5, 2011 at 4:11 PM, Ram Rachum <ram@rachum.com> wrote:
Okay, I've spent a few hours print-debugging, and I think I've almost got it. The crash happens on a line: st = os.stat(s) inside `os.path.isdir`, where `s` is a string 'C:\\Documents and Settings\\User\\My Documents\\Python Projects\\GarlicSim\\garlicsim\\src' This is a directory that happens not to exist, but of course this is not a good reason to crash. I have tried running `os.stat(s)` in the PyPy shell with that same `s`, but didn't get a crash there. I don't know why it's crashing in Nose but not in the shell. Does anyone have a clue? Ram.
2011/10/5 Ram Rachum <ram@rachum.com>:
it's possible that it's a RPython-level exception, or a bad handle because too many files wait for the garbage collector to close them. Can you give more information about the crash itself? - What are the last lines printed in the console? Try to disable "stdout capture" in Nose, by passing the -s option. - after the pypy process has exited, type "echo %ERRORLEVEL%" in the same console, to print the exit code of the last process. Which number is it? -- Amaury Forgeot d'Arc
On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
This is the entire output: Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011, 11:41:11) [PyPy 1.6.0 with MSC v.1500 32 bit] Running tests directly from GarlicSim repo. Pypy doesn't have wxPython, not loading `garlicsim_wx` tests. nose.config: INFO: Set working dir to C:\Documents and Settings\User\My Documents\Python Projects\GarlicSim nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.plugins.cover: INFO: Coverage report will include only packages: ['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx'] - after the pypy process has exited, type "echo %ERRORLEVEL%" in the
same console, to print the exit code of the last process. Which number is it?
-1073741819
-- Amaury Forgeot d'Arc
I've had a very similar stuff, as in something crashing only when run many times when I had a bug in a container type implemented in C. Basically, I had wrong refcount for the objets, which caused them to be freed by garbage collectore while they have been still used. Maybe something similar happens in the code that wraps windows API that handles file opening. That would explain why the bug never happens on linux. A good candidate would be incorrect refcount for the return value if the file does not exist. Try something like this: s="some_file" rets=[] for i in range(1000): rets.append(os.stat(s)) gc.collect() #Do something that uses lots of RAM (but a random amount, preferably in small blocks) print rets if it crashes then you have exactly that issue. 1000 might be not enough to toggle the crash though, as you need the OS to actually allocate different segments of memory for this to work. The more RAM you have the more cycles you need to toggle the crash. At least this approach helped me to debug extension modules written in C. BTW, for me on VM the test case does not crash. But I have SP2 windows there. On Wednesday 05 October 2011 19:37:07 Ram Rachum wrote:
Hello, Microsofts free WinDbg (http://en.wikipedia.org/wiki/WinDbg) can also attach to running processes. Regards, Martin "Ram Rachum" <ram@rachum.com> schrieb: Hey guys, I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on both system when running the GarlicSim test suite. It shows a Windows error dialog saying "pypy.exe has encountered a problem and needs to close. We are sorry for the inconvenience." and giving this data: AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll ModVer: 5.1.2600.5512 Offset: 00040b0d I can also open a dialog with a lot of data on the error (don't know how useful it is) but Windows won't let me Ctrl-C it. What can I do? Thanks, Ram.
Hi, 2011/10/5 Martin Kretschmar <kretschmar-martin@t-online.de>:
Microsofts free WinDbg (http://en.wikipedia.org/wiki/WinDbg) can also attach to running processes.
A release build of pypy contains no debug information Believe me, a debugger won't show anything useful. It's already difficult enough in a debug build (because of the generated C code) and almost impossible when JIT code is involved, except maybe for Armin :-) A reproducible (short) test case would be really appreciated. -- Amaury Forgeot d'Arc
On Thu, Oct 6, 2011 at 10:13 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
Hey guys, I've managed to produce a VM that shows the bug. You can download it here: http://dl.dropbox.com/u/1927707/VM%20for%20debugging%20PyPy.rar It's a VMWare Virtual Machine, and it weighs 2 GB compressed. Once you fire up the VM, there are short instructions in a text file on the desktop to making PyPy crash. Will anyone have time to try that? Thanks, Ram.
Hi, On Sun, Oct 9, 2011 at 12:15, Ram Rachum <ram@rachum.com> wrote:
Did anyone give that a try?
Sorry, it may take a while, because Windows is not our primary platform. In the meantime, could you try with this latest version of pypy? Thanks! http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-win32.zip A bientôt, Armin.
Hi, On Mon, Oct 10, 2011 at 13:11, Ram Rachum <ram@rachum.com> wrote:
Bah, I don't know why the recent zip files are all empty. Here is the latest non-empty one: http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-47320-6b92b3aa1cbb-win32.z... It's a binary release. A bientôt, Armin.
1. Switch to linux. It helps. 2. To get a meaningful error log try to run the pypy from terminal. To do copy-paste you will need 3-rd party terminal, i.e. power shell. Then you'll be able to copy the error messages. Without them it is pretty much impossible to identify the problem. 3. Another option is to use debugger to run the suite step-by-step and see what happens. Choose your poison. Alex. On Wednesday 05 October 2011 13:44:14 Ram Rachum wrote:
On Wed, Oct 5, 2011 at 1:49 PM, Alex Pyattaev <alex.pyattaev@gmail.com>wrote:
1. Switch to linux. It helps.
Not funny. 2. To get a meaningful error log try to run the pypy from terminal. To do
I am running PyPy from terminal, bash provided by msys. The error still comes up in a dialog and the shell contains only the output from `nose` up to the failure, with no word on the failure. 3. Another option is to use debugger to run the suite step-by-step and see
what happens.
I'll give that a try.
On Wed, Oct 5, 2011 at 2:07 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
I have hundreds of tests, and PyPy fails before a single one begins. It seems that PyPy crashes somewhere in nose's initialization. Isn't there a way to find the last Python line run before the crash without stepping with a finer granularity every time? Ram.
On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
How do I run the Nose test suite on Pypy with a debugger? I usually use Wing IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag which drops you into the debugger after an error, but it doesn't work here because this crash seems to be happening at a lower level. So I don't know how to start this in a debugger. Ram.
On Wed, Oct 5, 2011 at 2:22 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
I don't know how to use that... I don't program C at all, only Python. If I use a Python debugger, can't I just step forward line by line, see where I get the crash, and then isolate the offending line? Ram.
On 5 October 2011 13:50, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Well, in recent versions it gained the faulthandler module... :-) http://pypi.python.org/pypi/faulthandler/ Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
2011/10/5 Michael Foord <fuzzyman@gmail.com>:
Well, in recent versions it gained the faulthandler module... :-)
Yes, and this was a *great* improvement! I hope we will be able to adapt it to pypy. -- Amaury Forgeot d'Arc
On Wed, Oct 5, 2011 at 08:22, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
It may be early to start bringing this up, but I recently created an extension to catch crashes in C code and write minidumps: https://bitbucket.org/briancurtin/minidumper. It's 3.x only at the moment, although I'll be backporting within the next few days. I'll have a look at how/if it works on PyPy once I get there. (minidumper will likely be absorbed into faulthandler once it becomes more complete, which is why I mention it)
2011/10/5 Brian Curtin <brian.curtin@gmail.com>:
The code is quite simple, and should probably compile as is with pypy (except for the PyModuleDef of course) but it would be as simple to rewrite it in RPython. btw there is a suspect thing in your code: "app_name = pyname" will store the char* buffer into a static variable, but nothing guarantees that the PyObject passed will stay alive! -- Amaury Forgeot d'Arc
Generally, any binary-level debugger such as gdb or MSVC should work with pypy. At the very least you will find which operation crashed. If it is something really specific, for example sin/log/sign, it might be quite easy to map it back to python code. If it is not, it will be nearly impossible to find the original source line (at least I've failed when I tried). Another option is to edit the sources of the test suite adding print statements incrementally until you spot the place where it crashes. It is a slow, but very reliable way. That is of course if it is a particular segment of python code that crashes it. Also, could you send your exact environtment specs? I'll try to replicate it on a VM and see if it crashes for me too. I have an XP VM somewhere. PS: Sorry for my stupid joke about switching to linux. It was meant to cheer you up a bit. Alex. On Wednesday 05 October 2011 14:18:08 Ram Rachum wrote:
On Wed, Oct 5, 2011 at 2:23 PM, Alex Pyattaev <alex.pyattaev@gmail.com>wrote:
Generally, any binary-level debugger such as gdb or MSVC should work with pypy. At the very least you will find which operation crashed.
As I said to Amaury, I don't know how to use those... Python is the only language I program in.
I'll try, thanks.
What specs do you mean? It's just the recent PyPy 1.6 on a Windows XP SP3 32 bit machine with minimal packages installed. (distribute, pip, nosetests.) Let me know if you need any more data.
Forgiven :)
On Wed, Oct 5, 2011 at 4:11 PM, Ram Rachum <ram@rachum.com> wrote:
Okay, I've spent a few hours print-debugging, and I think I've almost got it. The crash happens on a line: st = os.stat(s) inside `os.path.isdir`, where `s` is a string 'C:\\Documents and Settings\\User\\My Documents\\Python Projects\\GarlicSim\\garlicsim\\src' This is a directory that happens not to exist, but of course this is not a good reason to crash. I have tried running `os.stat(s)` in the PyPy shell with that same `s`, but didn't get a crash there. I don't know why it's crashing in Nose but not in the shell. Does anyone have a clue? Ram.
2011/10/5 Ram Rachum <ram@rachum.com>:
it's possible that it's a RPython-level exception, or a bad handle because too many files wait for the garbage collector to close them. Can you give more information about the crash itself? - What are the last lines printed in the console? Try to disable "stdout capture" in Nose, by passing the -s option. - after the pypy process has exited, type "echo %ERRORLEVEL%" in the same console, to print the exit code of the last process. Which number is it? -- Amaury Forgeot d'Arc
On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
This is the entire output: Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011, 11:41:11) [PyPy 1.6.0 with MSC v.1500 32 bit] Running tests directly from GarlicSim repo. Pypy doesn't have wxPython, not loading `garlicsim_wx` tests. nose.config: INFO: Set working dir to C:\Documents and Settings\User\My Documents\Python Projects\GarlicSim nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.plugins.cover: INFO: Coverage report will include only packages: ['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx'] - after the pypy process has exited, type "echo %ERRORLEVEL%" in the
same console, to print the exit code of the last process. Which number is it?
-1073741819
-- Amaury Forgeot d'Arc
I've had a very similar stuff, as in something crashing only when run many times when I had a bug in a container type implemented in C. Basically, I had wrong refcount for the objets, which caused them to be freed by garbage collectore while they have been still used. Maybe something similar happens in the code that wraps windows API that handles file opening. That would explain why the bug never happens on linux. A good candidate would be incorrect refcount for the return value if the file does not exist. Try something like this: s="some_file" rets=[] for i in range(1000): rets.append(os.stat(s)) gc.collect() #Do something that uses lots of RAM (but a random amount, preferably in small blocks) print rets if it crashes then you have exactly that issue. 1000 might be not enough to toggle the crash though, as you need the OS to actually allocate different segments of memory for this to work. The more RAM you have the more cycles you need to toggle the crash. At least this approach helped me to debug extension modules written in C. BTW, for me on VM the test case does not crash. But I have SP2 windows there. On Wednesday 05 October 2011 19:37:07 Ram Rachum wrote:
Hello, Microsofts free WinDbg (http://en.wikipedia.org/wiki/WinDbg) can also attach to running processes. Regards, Martin "Ram Rachum" <ram@rachum.com> schrieb: Hey guys, I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on both system when running the GarlicSim test suite. It shows a Windows error dialog saying "pypy.exe has encountered a problem and needs to close. We are sorry for the inconvenience." and giving this data: AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll ModVer: 5.1.2600.5512 Offset: 00040b0d I can also open a dialog with a lot of data on the error (don't know how useful it is) but Windows won't let me Ctrl-C it. What can I do? Thanks, Ram.
Hi, 2011/10/5 Martin Kretschmar <kretschmar-martin@t-online.de>:
Microsofts free WinDbg (http://en.wikipedia.org/wiki/WinDbg) can also attach to running processes.
A release build of pypy contains no debug information Believe me, a debugger won't show anything useful. It's already difficult enough in a debug build (because of the generated C code) and almost impossible when JIT code is involved, except maybe for Armin :-) A reproducible (short) test case would be really appreciated. -- Amaury Forgeot d'Arc
On Thu, Oct 6, 2011 at 10:13 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
Hey guys, I've managed to produce a VM that shows the bug. You can download it here: http://dl.dropbox.com/u/1927707/VM%20for%20debugging%20PyPy.rar It's a VMWare Virtual Machine, and it weighs 2 GB compressed. Once you fire up the VM, there are short instructions in a text file on the desktop to making PyPy crash. Will anyone have time to try that? Thanks, Ram.
Hi, On Sun, Oct 9, 2011 at 12:15, Ram Rachum <ram@rachum.com> wrote:
Did anyone give that a try?
Sorry, it may take a while, because Windows is not our primary platform. In the meantime, could you try with this latest version of pypy? Thanks! http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-win32.zip A bientôt, Armin.
Hi, On Mon, Oct 10, 2011 at 13:11, Ram Rachum <ram@rachum.com> wrote:
Bah, I don't know why the recent zip files are all empty. Here is the latest non-empty one: http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-47320-6b92b3aa1cbb-win32.z... It's a binary release. A bientôt, Armin.
participants (8)
-
Alex Pyattaev -
Amaury Forgeot d'Arc -
Armin Rigo -
Brian Curtin -
kretschmar-martin＠t-online.de -
Maciej Fijalkowski -
Michael Foord -
Ram Rachum