Mailman 3 July 2013 - cython-devel

Difference between Cython / Python isinstance() vs. NumPy scalars, whose bug is this?
by Yury V. Zaytsev July 16, 2013

July 16, 2013

Hi, I've debugged my problem with passing NumPy scalars via Cython into C++ down to the fact that inside Cython code `isinstance(x, float)` returns True on floating point NumPy scalars, but `isinstance(x, (int, long))` returns False on integer NumPy scalars. However, when I run the same code from within Python, both checks work just fine. So is this a genuine bug in Cython, or, rather, NumPy is doing some black magic behind the scenes, that prevents Cython-compiled modules from working correctly? Here is a minimal reproducer for your convenience (tested with Cython 0.19.1, Python 2.7.1 and NumPy 1.7.1): # bt.pyx def foo(x): if isinstance(x, (int, long)): print("I'm an integer number") elif isinstance(x, float): print("I'm a floating point number") else: print("I don't know who I am") # IPython transcript In [1]: import numpy as np In [2]: import bt In [3]: bt.foo(np.array( (1., 2., 3.), dtype=np.float )[2]) I'm a floating point number In [4]: bt.foo(np.array( (1., 2., 3.), dtype=np.int )[2]) I don't know who I am In [5]: %paste def bar(x): if isinstance(x, (int, long)): print("I'm an integer number") elif isinstance(x, float): print("I'm a floating point number") else: print("I don't know who I am") ## -- End pasted text -- In [6]: bar(np.array( (1., 2., 3.), dtype=np.float )[2]) I'm a floating point number In [7]: bar(np.array( (1., 2., 3.), dtype=np.int )[2]) I'm an integer number Thanks in advance for pointing me in the right direction! -- Sincerely yours, Yury V. Zaytsev

1 2

getting rid of redundancy in the docs
by Stefan Behnel July 16, 2013

July 16, 2013

Hi all, I recently added a couple of things to the documentation and constantly failed to find the right place to add it at first (or sometimes even second) guess. The problem is that the documentation is highly redundant. We discussed this before, but it's really getting in the way now. What I would like to see eventually (and let's discuss this openly on the cython-users list), is just a tutorial part to explain the major features, and a reference part that explains all the rest in detail. The user guide is highly redundant with both. However, since the current user guide carries a whole load of text, I think it's easier to make it the new reference part and merge the (few) current reference pages back into the user guide. And then extract the sections that actually belong into the tutorial part, i.e. delete the Overview, move the Basic Tutorial over, and most likely some other sections, too. Does everyone agree about that approach? Is anyone able to help with this? Stefan

3 2

next releases
by Stefan Behnel July 14, 2013

July 14, 2013

Hi, I'm preparing another bug-fix point release and very conservatively merged over a few change sets into 0.19.x that seemed to be safe fixes. Please take a look and also propose or pick more changes if you feel that I left something out. Regarding the next master release, the only larger new feature I can see is Vitja's updated type inference engine. I think it should go in, but it needs some testing against "real" code (both Python and Cython, I guess). https://github.com/cython/cython/pull/233 The exttype noclear and nogc options should also go in, preferably with some more enhancements that make Cython smarter internally. Reducing the GC overhead by avoiding unnecessary support functions sounds valuable. Is there anything else that you have pending? I would also like to clean up the documentation for the next major release, see my e-mail to cython-users. Stefan

1 0

Re: [Cython] Py3 annotation syntax for static typing
by Stefan Behnel July 13, 2013

July 13, 2013

Jukka Lehtosalo, 12.07.2013 14:03: >> On Thu, Jul 11, 2013 at 6:52 PM, Stefan Behnel wrote: > http://mypy-lang.blogspot.de/2013/07/mypy-switches-to-python-compatible.html > >> It says that you are starting to adopt the Py3 function annotation syntax >> for mypy. I think we should try to keep that in sync for both mypy and >> Cython. It would be bad to have two ways to do it. > > Agreed. > >> When we discussed this for Cython (see the trac ticket [1] and Wiki entry >> [2] below), also on the mailing list (back in 2008 I guess), the main >> problem was that it's not immediately clear what should be considered type >> annotations by the compiler and what's arbitrary "other stuff" that users >> put into declarations. There's also no obvious way to allow for multiple >> independent annotations. Should they be in a tuple? In a dict? What would >> be dict key be? How can we prevent collisions and conflicts when multiple >> tools base their features on annotations? > > I think that this general problem can approached from several > directions: > > (1) Allow different annotation styles to be used in different modules, > but expect a uniform annotation style within a module. Each tool > should be able to recognize modules that use its annotation > syntax and ignore annotations in other modules. That would still conflict with the case of more than one kind of annotation, e.g. a docstring and (one or potentially more) static types. > (2) Support a fallback annotation syntax in each tool that doesn't use > Python 3 annotations, such as function decorators. This would let > users continue to use arbitrary annotations, at the cost of an > uglier type annotation syntax. Also, the syntax must support > stacking multiple annotations. Function decorators are nice in > this respect, but annotations in comments may be more troublesome. Cython has that in its Pure Python Mode syntax. I think it's ok to use tool specific decorators approach here, although it would be nice if we could keep at least the names similar for both Cython and MyPy. So, please take a look through the Pure Mode page if you want to go this route, and let's discuss anything that doesn't fit one of the tools for some reason. > (3) Support mixing multiple independent annotation styles within a > single function, using the Python 3 annotation syntax (as you > mentioned, if I understood your point correctly). Yes, I think there should be a way to do that. I haven't really seen any tools out in the wild that use the annotation syntax for any of their features, so this still seems to be uncovered ground. I think using a dict would be a good idea. Or a multi-level approach, where the annotation value can be a) a single value, which one tool would recognise and other tools would reject (or ignore? sounds fragile to ignore unknown values...). Maybe just ignore plain (doc-)strings and reject all other values that a tool cannot recognise. b) a 2-tuple containing one (doc-)string and one tool specific annotation value, as in a). Maybe require the docstring to appear after the annotation value, as the docstring is likely to be longer, and the annotation value might also be in a string. So, a 2-tuple (type, docstring). c) a dict that maps tool specific keys to annotation values. Each tool would only care about its own keys and ignore all others. I think this also provides for an easy upgrade path: start with a single value, optionally add a docstring to it in a tuple. If you need more tools, switch to an explicit dict. People will usually know which tool the original annotations belong to (and if not, they can just rip them out, right?), so the transition towards a more verbose syntax should be quite straight forward, and the multi-level approach helps in keeping things simple in the simple case. The dict keys should then be either an identifier that multiple tools understand (e.g. "type": list), or use a tool specific prefix, e.g. "cy:", "my:", "jy:", etc. Note that "type": int might be ambiguous, as Cython would consider it a C int. But I guess "ctype": int would solve that, and still be usable by multiple tools. An alternative would be to allow for (or require?) prefixed string values as simple/2-tuple annotation values, e.g. "ctype:int" or "cy:type:int". That would also allow to distinguish simple docstrings from simple annotation values in strings. Or, well, rely on module namespaces and imports to distinguish between annotation values. I see that mypy has a "typing" module which you can import. http://mypy-lang.org/tutorial.html#typing In Cython, there's the special "cython" module namespace. Personally, I find "typing" a bit broad as a top-level module name. There are also the ABCs in Py3, spread over the modules "abc", "collections.abc" (in Py3.3, previously in "collections") and "types" in the standard library. Some of your annotations seem to overlap with those. If the goal is to avoid duplication, it would be good to integrate what's there anyway. > (4) Agree on a single common type annotation style that can be used by > multiple projects. > > I think (1) is essential, (2) is important, and I'm undecied about > (3). I'm not sure whether (4) is practically feasible, as different > projects have different goals and features they want to support. See above. I think there can and should be a mix, as long as different projects keep an eye on each other. Keep common what we can, diverge where we it makes sense. >> It's mainly a namespace problem. The annotation namespace is essentially >> flat, and I think that only practical usage can eventually establish >> suitable conventions. > >> I'd like to discuss this, maybe we can come up with a suitable Best >> Practice. What's your opinion on the issue so far? > > I think we should first focus on (1) above, as it should be easy to > solve. As for (3), we should come up with some real-world use cases > or examples. For example, here are a few potential scenarios: > > (A) A programmer uses Python 3 annotations for internal purposes, > using an ad-hoc syntax. The programmer would like to speed up the > code by using Cython, but she wants the code to remain > Python-compatible, and she wants to continue using Python 3 > annotations for internal purposes. > > (B) A programmer uses mypy for type checking his program. He'd like to > speed up a critical function using Cython, but he also wants to > retain mypy type checking. I think both cases are covered by the approach above. Although users might end up having (or wanting?) to use both or all three syntaxes in the same module, in order to cover all cases. This might have an impact on code readability. My guess is that users who start using the dict based annotation style in a module at some point are best advised to convert the entire module at that point, in order to keep it simpler to read. An automated conversion tool based on 2to3 could then do the trick. >> There's also our current Pure Python syntax mode for everything that >> cannot be expressed with function annotations: > >> http://docs.cython.org/src/tutorial/pure.html > > I'm already somewhat familiar with the Pure Python syntax, though I > haven't tried it in practice. > >> I would guess that mypy will eventually need something similar. > > Mypy already (only) has a pure Python syntax, as all mypy constructs > are now syntactically valid Python, and valid mypy programs are > basically also runnable Python programs. In fact, currently a Python > 3.2 or later VM is the only supported way of running mypy programs. Ok, so you basically have a way to statically type variables during assignments. http://mypy-lang.org/tutorial.html#collectiontypes Cython has a slightly different approach here in that it requires either an external decorator (@cython.locals()) or an explicit function call during assignments (cython.declare()). I must say, I find neither of the three approaches really pretty, but I guess there simply is no perfect way to do these things, once you accept that you have to put static type annotations *somewhere* in your code. Stefan

1 0

Py3 annotation syntax for static typing
by Stefan Behnel July 11, 2013

July 11, 2013

[CC-ing cython-dev] Hi Jukka, I stumbled over your blog post at http://mypy-lang.blogspot.de/2013/07/mypy-switches-to-python-compatible.html It says that you are starting to adopt the Py3 function annotation syntax for mypy. I think we should try to keep that in sync for both mypy and Cython. It would be bad to have two ways to do it. When we discussed this for Cython (see the trac ticket [1] and Wiki entry [2] below), also on the mailing list (back in 2008 I guess), the main problem was that it's not immediately clear what should be considered type annotations by the compiler and what's arbitrary "other stuff" that users put into declarations. There's also no obvious way to allow for multiple independent annotations. Should they be in a tuple? In a dict? What would be dict key be? How can we prevent collisions and conflicts when multiple tools base their features on annotations? It's mainly a namespace problem. The annotation namespace is essentially flat, and I think that only practical usage can eventually establish suitable conventions. I'd like to discuss this, maybe we can come up with a suitable Best Practice. What's your opinion on the issue so far? There's also our current Pure Python syntax mode for everything that cannot be expressed with function annotations: http://docs.cython.org/src/tutorial/pure.html I would guess that mypy will eventually need something similar. Stefan [1] http://trac.cython.org/cython_trac/ticket/752 [2] http://wiki.cython.org/enhancements/pure

1 0

Bug report: compiler traceback instead of error message
by John Benediktsson July 9, 2013

July 9, 2013

Hi, I had some trouble debugging a traceback that I got from Cython, which ended up being a simple typo in a .pyx file. Below is a reduced example of this problem. If you have a foo.pxd file (specifying a variable "bar"): $ cat foo.pxd cdef class Tree: cpdef build(self, int bar=*) And a foo.pyx file (accidentally calling the variable "baz"): $ cat foo.pyx cdef class Tree: cpdef build(self, int baz=None): print baz You get this nasty traceback when you try to cython it, which makes you think there is a bug in Cython instead of a bug in your code: $ cython foo.pyx Traceback (most recent call last): File "/usr/local/bin/cython", line 9, in <module> load_entry_point('Cython==0.19.1', 'console_scripts', 'cython')() File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Main.py", line 618, in setuptools_main return main(command_line = 1) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Main.py", line 635, in main result = compile(sources, options) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Main.py", line 610, in compile return compile_multiple(source, options) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Main.py", line 579, in compile_multiple result = run_pipeline(source, options, context=context) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Main.py", line 425, in run_pipeline err, enddata = Pipeline.run_pipeline(pipeline, source) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Pipeline.py", line 322, in run_pipeline data = phase(data) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Pipeline.py", line 51, in generate_pyx_code_stage module_node.process_implementation(options, result) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/ModuleNode.py", line 111, in process_implementation self.generate_c_code(env, options, result) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/ModuleNode.py", line 342, in generate_c_code self.body.generate_function_definitions(env, code) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Nodes.py", line 393, in generate_function_definitions stat.generate_function_definitions(env, code) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Nodes.py", line 4267, in generate_function_definitions self.body.generate_function_definitions(self.scope, code) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Nodes.py", line 393, in generate_function_definitions stat.generate_function_definitions(env, code) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Nodes.py", line 1704, in generate_function_definitions self.generate_argument_parsing_code(env, code) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/Nodes.py", line 2297, in generate_argument_parsing_code self.type.opt_arg_cname(declarator.name))) File "/Library/Python/2.7/site-packages/Cython -0.19.1-py2.7-macosx-10.8-intel.egg/Cython/Compiler/PyrexTypes.py", line 2684, in opt_arg_cname return self.op_arg_struct.base_type.scope.lookup(arg_name).cname AttributeError: 'NoneType' object has no attribute 'cname' Thanks, John.

2 1

Update xdress link
by Anthony Scopatz July 6, 2013

July 6, 2013

Hello All, I was wondering if it would be possible to update the xdress link on the AutoPxd wiki page [1] to the new url: http://xdress.org/ Thanks in advance! Be Well Anthony 1. http://wiki.cython.org/AutoPxd

2 2

Cython and Py2.6/Py2.7 format() differences: '{}' vs. '{0}'
by Yury V. Zaytsev July 6, 2013

July 6, 2013

Hi, I have just discovered that if one uses Python 2.7 constructs in the code like print("Object: {}".format(obj)) the resulting C++ module compiles just fine with both Python 2.6 and Python 2.7 headers, however, in the former case, ValueError is raised at runtime: ValueError: zero length field name in format Is this an intended behavior or Cython should have abstracted this difference for me, and it can be considered a bug? Thanks! -- Sincerely yours, Yury V. Zaytsev

2 3

Sage build broken on Jenkins
by Stefan Behnel July 5, 2013

July 5, 2013

Hi, the Sage build has been broken for a while now. Might be due to changes in Sage, or due to one of the (minor?) cythonize() changes in master. https://sage.math.washington.edu:8091/hudson/job/sage-build/ Build error message is: """ IOError: could not find dependency sage/calculus/numpy.pxd included in sage/calculus/riemann.pyx. """ https://sage.math.washington.edu:8091/hudson/job/sage-build/1819/console Stefan

1 0