[Twisted-Python] exceptions.SystemError: unknown opcode
Hello list,
I have a twisted application in production for some time now, running under Fedora Core 6, Python 2.4.4 and twisted 2.4. I know, a little outdated. The process was running fine for some weeks without any problems, although I got this exception in my logs and I really have no idea what it means. The process did not die and is still running ok.
2008/05/13 16:03 BRT [-] Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/application/app.py", line 278, in run runApp(config) File "/usr/lib/python2.4/site-packages/twisted/scripts/twistd.py", line 197, in runApp app.runReactorWithLogging(config, oldstdout, oldstderr) File "/usr/lib/python2.4/site-packages/twisted/application/app.py", line 131, in runReactorWithLogging reactor.run() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 218, in run self.mainLoop() --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 229, in mainLoop self.doIteration(t) exceptions.SystemError: unknown opcode
Regards,
Miguel
On 2008.05.14 10:22:48 -0300, Miguel Filho wrote:
I have a twisted application in production for some time now, running under Fedora Core 6, Python 2.4.4 and twisted 2.4. I know, a little outdated. The process was running fine for some weeks without any problems, although I got this exception in my logs and I really have no idea what it means. The process did not die and is still running ok.
exceptions.SystemError: unknown opcode
This means that Python found an illegal virtual machine instruction.
You can cause this by running or importing a .pyc file from a different version of Python, without having a matching .py file to fall back on.
On Wed, May 14, 2008 at 11:39 AM, David Ripton dripton@ripton.net wrote:
This means that Python found an illegal virtual machine instruction.
You can cause this by running or importing a .pyc file from a different version of Python, without having a matching .py file to fall back on.
Hmm, that stills look strange, because I didn't change anything.
Miguel
On Wed, May 14, 2008 at 11:39 AM, Miguel Filho miguel.filho@gmail.com wrote:
On Wed, May 14, 2008 at 11:39 AM, David Ripton dripton@ripton.net wrote:
This means that Python found an illegal virtual machine instruction.
You can cause this by running or importing a .pyc file from a different version of Python, without having a matching .py file to fall back on.
Hmm, that stills look strange, because I didn't change anything.
Miguel
I'd run memtest on that machine if I were you.
On Wed, 14 May 2008 12:39:11 -0300, Miguel Filho miguel.filho@gmail.com wrote:
On Wed, May 14, 2008 at 11:39 AM, David Ripton dripton@ripton.net wrote:
This means that Python found an illegal virtual machine instruction.
You can cause this by running or importing a .pyc file from a different version of Python, without having a matching .py file to fall back on.
Hmm, that stills look strange, because I didn't change anything.
Encountering this after the process has been running for a long time suggests a memory corruption issue to me. The fact that it happened only once while the process is still running and correctly servicing events further suggests that the corruption was somehow transient, a somewhat unusual occurance to say the least. :)
Before the SystemError was raised, the process probably wrote a tiny amount of debug information to stderr. Is there any chance that got logged somewhere useful? In the normal deamonization mode of twistd it probably would have been discarded since as part of daemonization stdin, stdout, and stderr are all closed. The information is barely useful, though, so it isn't a huge loss, but at least it would point out which code object had the bad opcode. The traceback suggests it was twisted.internet.selectreactor.SelectReactor.doSelect (at least, if you're using select reactor). You could attach gdb and check the the opcodes in the code object for that function, but I imagine they are correct now, since the process is still running.
So, I have no idea what happened, but it sounds interesting. :)
Jean-Paul
On Wed, May 14, 2008 at 1:01 PM, Jean-Paul Calderone exarkun@divmod.com wrote:
Encountering this after the process has been running for a long time suggests a memory corruption issue to me. The fact that it happened only once while the process is still running and correctly servicing events further suggests that the corruption was somehow transient, a somewhat unusual occurance to say the least. :)
Just to bring a conclusion to this topic, the problem is actually bad memory.
On the same machine a Postfix daemon has just died, checking the logs: May 28 08:58:31 friends postfix/master[2933]: ess May 28 08:58:31 friends postfix/master[2933]: May 28 08:58:31 friends postfix/smtpd[6301]: connect from unknown[59.95.173.59] May 28 08:58:31 friends postfix/master[2933]: ess May 28 08:58:31 friends postfix/master[2933]: terminating on signal 11
From 'man 7 signal':
SIGSEGV 11 Core Invalid memory reference
A Dovecot daemon used to suddenly die on the same box too.
Regards,
Miguel
participants (4)
-
Christopher Armstrong -
David Ripton -
Jean-Paul Calderone -
Miguel Filho