Mailman 3 November 2015 - cython-devel

cygdb: no debug files were found
by Nathan Goldbaum Sept. 14, 2016

Sept. 14, 2016

Hi all, I'm following the directions in the link below to debug some cython code using cygdb: http://docs.cython.org/src/userguide/debugging.html Unfortunately, when I try to actually debug my program using cygdb, I get a "no debug files were found" error. It looks like the error is getting raised here: https://github.com/cython/cython/blob/master/Cython/Debugger/Cygdb.py#L33 So for some reason the "cython_debug" folder isn't being generated at build time. I tried to follow the directions closely, you can see the notes I put together during the installation process here: https://bpaste.net/show/6b11f815d449 Unfortunately the library I'm trying to debug (yt: bitbucket.org/yt_analysis/yt) uses a somewhat nonstandard build setup based on numpy distutils, and I suspect that is breaking some of the assumptions that went into the cygdb instructions in the cython docs. Does anyone have any idea what went wrong here? For now I'm going to fall back to regular gdb, and debug the autogenerated cython sources. Thanks for your help, Nathan Goldbaum

2 1

Re: [Cython] Store list of imported modules in .so file
by Kevin Norris Nov. 28, 2015

Nov. 28, 2015

On Tue, Nov 24, 2015 at 12:50 PM, Hartmut Goebel <h.goebel(a)crazy-compilers.com> wrote: > Hi, > > I'm a core-developer for PyInstaller. We would like to enhance cython > support in PyInstaller and resolve dependencies of cython-compiled > modules automatically. This would ease freezing cython modules with > PyInstaller a lot and thus help users including cythonized-modules into > their shipped application. > > For us, the most elegant way to get the dependencies would be to have > the list of imported modules in the .so-file. E.g. some simple string > marked by some cookie which is easily grepped out of the binary. > > Is there any chance to get suche a list into the .so/.dll-file? Import statements are "just code." They can appear inside functions and more generally can occur at runtime. In the absolute worst case, you could have something like (Python 3.x): importlib.import_module(input('What should I import today?')) This is not specific to Cython, but I imagine it's a lot harder to pull out via static analysis when you don't have Python-like source (see also: pkgutil.get_data(), any use of __file__, etc.). OTOH, if you're dealing with importlib et al., you're already in a bad place for static analysis since people can do things like: import importlib as foo; bar = foo.import_module; baz = bar('qux') (which is roughly equivalent to import qux as baz, but good luck spotting it automatically) Anyway, I think it might help to clarify what *exactly* you mean by "dependency." If you mean "any module that might possibly get imported by module X," you're going to have a hard time with this. Setuptools takes a fully manual approach to this problem, requiring developers to explicitly list all dependencies. That may not work for your needs, but it does seem to be more flexible since it lets you specify PEP 440 version constraints.

3 2

Store list of imported modules in .so file
by Hartmut Goebel Nov. 26, 2015

Nov. 26, 2015

Hi, I'm a core-developer for PyInstaller. We would like to enhance cython support in PyInstaller and resolve dependencies of cython-compiled modules automatically. This would ease freezing cython modules with PyInstaller a lot and thus help users including cythonized-modules into their shipped application. For us, the most elegant way to get the dependencies would be to have the list of imported modules in the .so-file. E.g. some simple string marked by some cookie which is easily grepped out of the binary. Is there any chance to get suche a list into the .so/.dll-file? If you like, you can comment directly on our related ticket: https://github.com/pyinstaller/pyinstaller/issues/1674 -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel(a)crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

2 1

Re: [Cython] bug in doc Tutorials/Working with NumPy
by Stepan Kasal Nov. 10, 2015

Nov. 10, 2015

Hello, On Mon, Nov 09, 2015 at 09:44:15PM -0800, Robert Bradshaw wrote: > I heartily agree that using unsigned types is the source of much > surprising pain, we shouldn't be recommending it: great, thanks a lot for updating the dev doc instantly. > There's two directives: wraparound and boundscheck. The wraparound > directive can be used to disable the check for negative values. And thanks for catching this; I misunderstood this. Stepan

1 0

bug in doc Tutorials/Working with NumPy
by Stepan Kasal Nov. 10, 2015

Nov. 10, 2015

Hello, I was learning Cython from this tutorial when I noticed a glitch: Near the end, after boundscheck False, some variables are changed to unsigned int. But this has unexpected consequences: variables t_from and such are not determined correctly, as one of the arguments is negative, but gets cast to unsigned for comaprison. (Yes, this is a weird feature of C.) Actually, the use of unsigned does not seem to have any impact on performance, so I suggest to scratch that part of tutorial. Just explain that with boundscheck off, negaitive indices no longer work. (But there might perhaps be one small improvent in cython itself: with boundscheck off, the index might be cast to unsigned before it is used: this way we probably crash on negative values, instead of silently accessing the negative offset of the array data.) Thank you for nice piece of software, Stepan Kasal

2 1

[Bug] Don't compile lambda with tuple argument.
by yi huang Nov. 10, 2015

Nov. 10, 2015

λ $ cat /tmp/test.pyx f = lambda (name,v):True λ $ cython /tmp/test.pyx Error compiling Cython file: ------------------------------------------------------------ ... f = lambda (name,v):True ^ ------------------------------------------------------------ /tmp/test.pyx:1:19: Empty declarator Tested with both 0.21 and 0.23

2 2

[PATCH] Add a pass transforming Python-style properties in cdef class into Cython-style properties.
by Emmanuel Gil Peyrot Nov. 7, 2015

Nov. 7, 2015

This makes properties work at all, in cdef classes, and gives them almost the same features as the “property something:” blocks. The only missing feature being the ability to assignate a docstring to the property itself, not only to the getter, setter and deleter. Fixes T264. --- Cython/Compiler/ParseTreeTransforms.py | 72 ++++++++++++++++++++++++ Cython/Compiler/Pipeline.py | 3 +- tests/run/cdef_class_property_decorator_T264.pyx | 51 +++++++++++++++++ 3 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 tests/run/cdef_class_property_decorator_T264.pyx diff --git a/Cython/Compiler/ParseTreeTransforms.py b/Cython/Compiler/ParseTreeTransforms.py index b34f44b..67507ce 100644 --- a/Cython/Compiler/ParseTreeTransforms.py +++ b/Cython/Compiler/ParseTreeTransforms.py @@ -1270,6 +1270,78 @@ class WithTransform(CythonTransform, SkipDeclarations): return node +class PropertyTransform(ScopeTrackingTransform): + """Originally, this was the only place where decorators were + transformed into the corresponding calling code. Now, this is + done directly in DefNode and PyClassDefNode to avoid reassignments + to the function/class name - except for cdef class methods. For + those, the reassignment is required as methods are originally + defined in the PyMethodDef struct. + + The IndirectionNode allows DefNode to override the decorator + """ + + properties = {} + + def visit_DefNode(self, func_node): + func_node = self.visit_FuncDefNode(func_node) + if self.scope_type != 'cclass' or not func_node.decorators: + return func_node + return self.handle_decorators(func_node, func_node.decorators, + func_node.name) + + def handle_decorators(self, node, decorators, name): + properties = self.properties.setdefault(self.scope_node, {}) + for decorator in decorators[::-1]: + decorator_node = decorator.decorator + if (isinstance(decorator_node, ExprNodes.NameNode) and + decorator_node.name == 'property'): + node.name = '__get__' + node.decorators.remove(decorator) + stat_list = [node] + if name in properties: + property = properties[name] + property.body.stats = stat_list + return [] + else: + property = Nodes.PropertyNode(node.pos, name=name) + property.doc = EncodedString(u'') + property.body = Nodes.StatListNode(node.pos, stats=stat_list) + properties[name] = property + return [property] + elif (isinstance(decorator_node, ExprNodes.AttributeNode) and + decorator_node.attribute == 'setter' and + decorator_node.obj.name in properties): + assert decorator_node.obj.name == name + node.name = '__set__' + node.decorators.remove(decorator) + property = properties[name] + stats = property.body.stats + for i, stat in enumerate(stats): + if stat.name == '__set__': + stats[i] = node + break + else: + stats.append(node) + return [] + elif (isinstance(decorator_node, ExprNodes.AttributeNode) and + decorator_node.attribute == 'deleter' and + decorator_node.obj.name in properties): + assert decorator_node.obj.name == name + node.name = '__del__' + node.decorators.remove(decorator) + property = properties[name] + stats = property.body.stats + for i, stat in enumerate(stats): + if stat.name == '__del__': + stats[i] = node + break + else: + stats.append(node) + return [] + return node + + class DecoratorTransform(ScopeTrackingTransform, SkipDeclarations): """Originally, this was the only place where decorators were transformed into the corresponding calling code. Now, this is diff --git a/Cython/Compiler/Pipeline.py b/Cython/Compiler/Pipeline.py index 833e22c..0506aff 100644 --- a/Cython/Compiler/Pipeline.py +++ b/Cython/Compiler/Pipeline.py @@ -171,7 +171,7 @@ def create_pipeline(context, mode, exclude_classes=()): from .ParseTreeTransforms import CreateClosureClasses, MarkClosureVisitor, DecoratorTransform from .ParseTreeTransforms import InterpretCompilerDirectives, TransformBuiltinMethods from .ParseTreeTransforms import ExpandInplaceOperators, ParallelRangeTransform - from .ParseTreeTransforms import CalculateQualifiedNamesTransform + from .ParseTreeTransforms import CalculateQualifiedNamesTransform, PropertyTransform from .TypeInference import MarkParallelAssignments, MarkOverflowingArithmetic from .ParseTreeTransforms import AdjustDefByDirectives, AlignFunctionDefinitions from .ParseTreeTransforms import RemoveUnreachableCode, GilCheck @@ -216,6 +216,7 @@ def create_pipeline(context, mode, exclude_classes=()): RemoveUnreachableCode(context), ConstantFolding(), FlattenInListTransform(), + PropertyTransform(context), DecoratorTransform(context), ForwardDeclareTypes(context), AnalyseDeclarationsTransform(context), diff --git a/tests/run/cdef_class_property_decorator_T264.pyx b/tests/run/cdef_class_property_decorator_T264.pyx new file mode 100644 index 0000000..94b862c --- /dev/null +++ b/tests/run/cdef_class_property_decorator_T264.pyx @@ -0,0 +1,51 @@ +# mode: run +# ticket: 264 +# tag: property, decorator + +cdef class Prop(object): + """ + >>> p = Prop() + >>> p.prop + GETTING 'None' + >>> p.prop = 1 + SETTING '1' (previously: 'None') + >>> p.prop + GETTING '1' + 1 + >>> p.prop = 2 + SETTING '2' (previously: '1') + >>> p.prop + GETTING '2' + 2 + >>> del p.prop + DELETING '2' + >>> p.prop + GETTING 'None' + """ + cdef _value + def __init__(self): + self._value = None + + @property + def prop(self): + print("FAIL") + return 0 + + @prop.deleter + def prop(self): + print("FAIL") + + @property + def prop(self): + print("GETTING '%s'" % self._value) + return self._value + + @prop.setter + def prop(self, value): + print("SETTING '%s' (previously: '%s')" % (value, self._value)) + self._value = value + + @prop.deleter + def prop(self): + print("DELETING '%s'" % self._value) + self._value = None -- 2.6.2

2 1

[Bug] Generator already executing
by Carlos Pita Nov. 7, 2015

Nov. 7, 2015

Say you define: def f(*ts): ts = (t for t in ts) list(ts) def g(*ts): list(t for t in ts) def h(*ts): list((t for t in ts)) Then f(1,2,3) will fail with "Generator already executing" while g(1,2,3) and h(1,2, 3) won't. All versions should work the same. Cheers -- Carlos

2 1

AttributeError: 'LetNode' object has no attribute 'generate_rhs_evaluation_code'
by Pastafarianist Nov. 3, 2015

Nov. 3, 2015

I'm running Python 3.4.3 and Cython 0.23.4 under Gentoo Linux. I think I've found a bug in Cython. Steps to reproduce: ## test.py ## import pyximport pyximport.install() import mymodule print(mymodule.get(1, 2)) ## mymodule.pyx ## cdef int[2][3] arr = ( (0, 1, 2), (3, 4, 5) ) cpdef int get(int i, int j): return arr[i][j] #### `python test.py` fails with the following traceback: Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 209, in load_module inplace=build_inplace, language_level=language_level) File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 186, in build_module reload_support=pyxargs.reload_support) File "/usr/lib64/python3.4/site-packages/pyximport/pyxbuild.py", line 100, in pyx_to_dll dist.run_commands() File "/usr/lib64/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 164, in run _build_ext.build_ext.run(self) File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 320, in cython_sources full_module_name=module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 677, in compile return compile_single(source, options, full_module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 630, in compile_single return run_pipeline(source, options, full_module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 487, in run_pipeline err, enddata = Pipeline.run_pipeline(pipeline, source) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Pipeline.py", line 328, in run_pipeline data = phase(data) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage module_node.process_implementation(options, result) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 118, in process_implementation self.generate_c_code(env, options, result) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 348, in generate_c_code self.generate_module_init_func(modules[:-1], env, globalstate['init_module']) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 2173, in generate_module_init_func self.body.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 442, in generate_execution_code stat.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/UtilNodes.py", line 334, in generate_execution_code self.body.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 5223, in generate_execution_code stat.generate_rhs_evaluation_code(code) AttributeError: 'LetNode' object has no attribute 'generate_rhs_evaluation_code' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pastafarianist/ownCloud/dev/sandbox/cython/test.py", line 4, in <module> import mymodule File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 435, in load_module language_level=self.language_level) File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 225, in load_module raise exc.with_traceback(tb) File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 209, in load_module inplace=build_inplace, language_level=language_level) File "/usr/lib64/python3.4/site-packages/pyximport/pyximport.py", line 186, in build_module reload_support=pyxargs.reload_support) File "/usr/lib64/python3.4/site-packages/pyximport/pyxbuild.py", line 100, in pyx_to_dll dist.run_commands() File "/usr/lib64/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 164, in run _build_ext.build_ext.run(self) File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/usr/lib64/python3.4/site-packages/Cython/Distutils/build_ext.py", line 320, in cython_sources full_module_name=module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 677, in compile return compile_single(source, options, full_module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 630, in compile_single return run_pipeline(source, options, full_module_name) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Main.py", line 487, in run_pipeline err, enddata = Pipeline.run_pipeline(pipeline, source) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Pipeline.py", line 328, in run_pipeline data = phase(data) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage module_node.process_implementation(options, result) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 118, in process_implementation self.generate_c_code(env, options, result) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 348, in generate_c_code self.generate_module_init_func(modules[:-1], env, globalstate['init_module']) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/ModuleNode.py", line 2173, in generate_module_init_func self.body.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 442, in generate_execution_code stat.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/UtilNodes.py", line 334, in generate_execution_code self.body.generate_execution_code(code) File "/usr/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 5223, in generate_execution_code stat.generate_rhs_evaluation_code(code) ImportError: Building module mymodule failed: ["AttributeError: 'LetNode' object has no attribute 'generate_rhs_evaluation_code'\n"]

1 0

BUG: Invalid code generated from prange()
by Thouis (Ray) Jones Nov. 3, 2015

Nov. 3, 2015

Cythoning and then trying to compile the attached test causes gcc-4.9 to error out with: physics.c:1840:165: error: invalid entry to OpenMP structured block if (unlikely(!__pyx_v_XY.memview)) { __Pyx_RaiseUnboundMemoryviewSliceNogil("XY"); {__pyx_filename = __pyx_f[0]; __pyx_lineno = 21; __pyx_clineno = __LINE__; goto __pyx_L5_error;} } clang version 3.5 (with openmp) gives the less helpful "use of undeclared label '__pyx_L5_error'". Ray Jones

1 0