Compiler crash in RemoveUnreachableCode
The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'
Hi, thanks for the report, I can reproduce this. Andriy Kornatskyy, 19.01.2014 21:00:
The cython compiler crash report below. Steps to reproduce:
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http
The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.
Yes, that's rather surprising.
File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ...
""" """
__version__ = '0.1.129' ^ ------------------------------------------------------------
src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode
ModuleNode.body = StatListNode(__init__.py:5:14)
Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'
I only get this when Cython is compiled. When I run it from the source tree, it works. Debugging into Visitor.py's CythonTransform shows that this check fails: if isinstance(node, ModuleNode.ModuleNode): self.current_directives = node.directives "node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode. My guess is that there's something wrong with the import mechanism that leads to the module being imported twice. Relative versus absolute imports, most likely. I also see a couple of weird modules in sys.modules, e.g. "Cython.Compiler.re", so there are some more bits going generally wrong here. I also tried it in Py3, but that even gives me a hard crash. Not really better. I might be able to take a deeper look this weekend, but wouldn't mind if others beat me to it. Stefan
Stefan, Thank you for the comments back. May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools. Here is another bug (details below): 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install lxml wheezy.core It seems to have an issue while trying to install 2 or more libs at once. Andriy Kornatskyy ... Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg Processing dependencies for lxml Finished processing dependencies for lxml Searching for wheezy.core Reading https://pypi.python.org/simple/wheezy.core/ Best match: wheezy.core 0.1.129 Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.ta... Processing wheezy.core-0.1.129.tar.gz Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 Traceback (most recent call last): ... File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation self.generate_c_code(env, options, result) File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697) File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ stream = StringIO() TypeError: 'NoneType' object is not callable On Jan 20, 2014, at 10:19 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Hi,
thanks for the report, I can reproduce this.
Andriy Kornatskyy, 19.01.2014 21:00:
The cython compiler crash report below. Steps to reproduce:
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http
The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.
Yes, that's rather surprising.
File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ...
""" """
__version__ = '0.1.129' ^ ------------------------------------------------------------
src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode
ModuleNode.body = StatListNode(__init__.py:5:14)
Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'
I only get this when Cython is compiled. When I run it from the source tree, it works.
Debugging into Visitor.py's CythonTransform shows that this check fails:
if isinstance(node, ModuleNode.ModuleNode): self.current_directives = node.directives
"node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode.
My guess is that there's something wrong with the import mechanism that leads to the module being imported twice. Relative versus absolute imports, most likely. I also see a couple of weird modules in sys.modules, e.g. "Cython.Compiler.re", so there are some more bits going generally wrong here.
I also tried it in Py3, but that even gives me a hard crash. Not really better.
I might be able to take a deeper look this weekend, but wouldn't mind if others beat me to it.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Andriy Kornatskyy, 20.01.2014 22:16:
May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.
No, I think it's a problem with importing in Cython compiled modules. ISTM that some extension modules got "reloaded", i.e. their module dict got cleared and they got reinitialised, thus recreating all global objects that other modules had already imported (and still keep a reference to).
Here is another bug (details below):
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install lxml wheezy.core
It seems to have an issue while trying to install 2 or more libs at once. ... Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg Processing dependencies for lxml Finished processing dependencies for lxml Searching for wheezy.core Reading https://pypi.python.org/simple/wheezy.core/ Best match: wheezy.core 0.1.129 Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.ta... Processing wheezy.core-0.1.129.tar.gz Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 Traceback (most recent call last): ... File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation self.generate_c_code(env, options, result) File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ stream = StringIO() TypeError: 'NoneType' object is not callable
Looks like the same thing. Stefan
Stefan, Any update to the issue reported in this message thread? If you can provide a link to the issue in cython bug tracker, so I can see progress, I would greatly appreciate that. Thanks. Andriy Kornatskyy On Jan 21, 2014, at 10:16 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Andriy Kornatskyy, 20.01.2014 22:16:
May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.
No, I think it's a problem with importing in Cython compiled modules. ISTM that some extension modules got "reloaded", i.e. their module dict got cleared and they got reinitialised, thus recreating all global objects that other modules had already imported (and still keep a reference to).
Here is another bug (details below):
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install lxml wheezy.core
It seems to have an issue while trying to install 2 or more libs at once. ... Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg Processing dependencies for lxml Finished processing dependencies for lxml Searching for wheezy.core Reading https://pypi.python.org/simple/wheezy.core/ Best match: wheezy.core 0.1.129 Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.ta... Processing wheezy.core-0.1.129.tar.gz Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 Traceback (most recent call last): ... File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation self.generate_c_code(env, options, result) File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ stream = StringIO() TypeError: 'NoneType' object is not callable
Looks like the same thing.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Andriy Kornatskyy, 22.02.2014 10:35:
On Jan 21, 2014, at 10:16 AM, Stefan Behnel wrote:
I think it's a problem with importing in Cython compiled modules. ISTM that some extension modules got "reloaded", i.e. their module dict got cleared and they got reinitialised, thus recreating all global objects that other modules had already imported (and still keep a reference to).
Here is another bug (details below):
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install lxml wheezy.core
It seems to have an issue while trying to install 2 or more libs at once. ... Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg Processing dependencies for lxml Finished processing dependencies for lxml Searching for wheezy.core Reading https://pypi.python.org/simple/wheezy.core/ Best match: wheezy.core 0.1.129 Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.ta... Processing wheezy.core-0.1.129.tar.gz Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 Traceback (most recent call last): ... File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation self.generate_c_code(env, options, result) File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ stream = StringIO() TypeError: 'NoneType' object is not callable
Looks like the same thing.
Any update to the issue reported in this message thread?
No, not really. The current setup is just way too vast to debug. I don't even know how easy_install runs the installation of a single package, nor how it's possible that two packages interfere in such a way. I'd expect it to run subprocesses, so why any interference at all? It might be that the main process loads the setup.py scripts in some way to read the meta data, thus executes their imports, and then tries to unload the modules before it loads the next setup.py script. That would be rather bad for any extension modules imported by both. So bad that's I'd consider it a bug in setuptools. Let's assume that reloading really is what's happening here. Given that this basically doesn't work for extension modules, definitely not in Py2.x and barely in Py3.x, we might be able to apply a barrier to the module init function that prevents it from running a second time, and just return the already created module somehow. But that's blank guessing and I really can't tell for sure. It would help if someone could invest some time into properly analysing this further. Stefan
Andriy Kornatskyy schrieb am 19.01.2014 um 21:00:
The cython compiler crash report below. Steps to reproduce:
1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http
The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.
I tried "pip install wheezy.http" instead of "easy_install wheezy.http" and it works as expected. This sounds like a good work-around to me. Stefan
participants (2)
-
Andriy Kornatskyy -
Stefan Behnel