Mailman 3 September 2018 - cython-devel

Cython 0.29 beta 1 released
by Stefan Behnel Sept. 29, 2018

Sept. 29, 2018

Hi all, after half a year of development, many community pull requests and a lot of feedback and good ideas in online discussions and at conferences, I'm proud to release the first beta of Cython 0.29. This is a major feature release that comes with many great improvements and several important bug fixes. See the long list of changes below. Please give it some testing to help us quickly advance to the final release. Download: https://github.com/cython/cython/releases/tag/0.29b1 Changelog: https://github.com/cython/cython/blob/0dcb5d1930e573caa8494fe838c4c2cd4e204… Foresight: Given that Cython has been in critical production use all over the world for several years, but never found the perfect time for a 1.0 version bump, we designate this to become the last 0.x release of the project and decided to skip the 1.0 release which the 0.x series has long represented anyway. Planning has already started [1] for the next major release, titled "3.0". It will finally switch the default Cython language level from Py2 to Py3, to match what users expect from a Python compiler these days without additional options or configuration. Cython 2 code will continue to be supported as before with the directive "language_level=2", although there are ideas how to help with the modernisation. We're happy to hear your feedback. [1] https://github.com/cython/cython/milestone/58 Have fun, Stefan 0.29 beta 1 (2018-09-16) ======================== Features added -------------- * PEP-489 multi-phase module initialisation has been enabled again. Module reloads raise an exception to prevent corruption of the static module state. * A set of ``mypy`` compatible PEP-484 declarations were added for Cython's C data types to integrate with static analysers in typed Python code. They are available in the ``Cython/Shadow.pyi`` module and describe the types in the special ``cython`` module that can be used for typing in Python code. Original patch by Julian Gethmann. (Github issue #1965) * Memoryviews are supported in PEP-489 style type declarations. (Github issue #2529) * Raising exceptions from nogil code will automatically acquire the GIL, instead of requiring an explicit ``with gil`` block. * ``(a)cython.nogil`` is supported as a C-function decorator in Python code. (Github issue #2557) * ``cython.inline()`` supports a direct ``language_level`` keyword argument that was previously only available via a directive. * In CPython 3.6 and later, looking up globals in the module dict is almost as fast as looking up C globals. (Github issue #2313) * For a Python subclass of an extension type, repeated method calls to non-overridden cpdef methods can avoid the attribute lookup in Py3.6+, which makes them 4x faster. (Github issue #2313) * (In-)equality comparisons of objects to integer literals are faster. (Github issue #2188) * Some internal and 1-argument method calls are faster. * Modules that cimport many external extension types from other Cython modules execute less import requests during module initialisation. * The coverage plugin considers more C file extensions such as ``.cc`` and ``.cxx``. (Github issue #2266) * The ``cythonize`` command accepts compile time variable values (as set by ``DEF``) through the new ``-E`` option. Patch by Jerome Kieffer. (Github issue #2315) * ``pyximport`` can import from namespace packages. Patch by Prakhar Goel. (Github issue #2294) * Some missing numpy and CPython C-API declarations were added. Patch by John Kirkham. (Github issues #2523, #2520, #2537) * Declarations for the ``pylifecycle`` C-API functions were added in a new .pxd file ``cpython.pylifecycle``. * The Pythran support was updated to work with the latest Pythran 0.8.7. Original patch by Adrien Guinet. (Github issue #2600) * ``%a`` is included in the string formatting types that are optimised into f-strings. In this case, it is also automatically mapped to ``%r`` in Python 2.x. * New C macro ``CYTHON_HEX_VERSION`` to access Cython's version in the same style as ``PY_HEX_VERSION``. Bugs fixed ---------- * The exception handling in generators and coroutines under CPython 3.7 was adapted to the newly introduced exception stack. Users of Cython 0.28 who want to support Python 3.7 are encouraged to upgrade to 0.29 to avoid potentially incorrect error reporting and tracebacks. * Crash when importing a module under Stackless Python that was built for CPython. Patch by Anselm Kruis. (Github issue #2534) * 2-value slicing of typed sequences failed if the start or stop index was None. Patch by Christian Gibson. (Github issue #2508) * Multiplied string literals lost their factor when they are part of another constant expression (e.g. 'x' * 10 + 'y' => 'xy'). * String formatting with the '%' operator didn't call the special ``__rmod__()`` method if the right side is a string subclass that implements it. (Python issue 28598) * The directive ``language_level=3`` did not apply to the first token in the source file. (Github issue #2230) * Overriding cpdef methods did not work in Python subclasses with slots. Note that this can have a performance impact on calls from Cython code. (Github issue #1771) * Fix declarations of builtin or C types using strings in pure python mode. (Github issue #2046) * Several internal function signatures were fixed that lead to warnings in gcc-8. (Github issue #2363) * The numpy helper functions ``set_array_base()`` and ``get_array_base()`` were adapted to the current numpy C-API recommendations. Patch by Matti Picus. (Github issue #2528) * Some NumPy related code was updated to avoid deprecated API usage. Original patch by jbrockmendel. (Github issue #2559) * Several C++ STL declarations were extended and corrected. Patch by Valentin Valls. (Github issue #2207) * C lines of the module init function were unconditionally not reported in exception stack traces. Patch by Jeroen Demeyer. (Github issue #2492) * When PEP-489 support is enabled, reloading the module overwrote any static module state. It now raises an exception instead, given that reloading is not actually supported. Other changes ------------- * Cython now emits a warning when no ``language_level`` (2 or 3) is set explicitly, neither as a ``cythonize()`` option nor as a compiler directive. This is meant to prepare the transition of the default language level from currently Py2 to Py3, since that is what most new users will expect these days. The next major release is intended to make that change, so that it will parse all code that does not request a specific language level as Python 3 code. The language level 2 will continue to be supported for an indefinite time. * The documentation was restructured, cleaned up and examples are now tested. The NumPy tutorial was also rewritten to simplify the running example. Contributed by Gabriel de Marmiesse. (Github issue #2245) * Cython compiles less of its own modules at build time to reduce the installed package size to about half of its previous size. This makes the compiler slightly slower, by about 5-7%.

1 2

Re: [Cython] Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
by Stefan Behnel Sept. 29, 2018

Sept. 29, 2018

Jeroen Demeyer schrieb am 20.09.2018 um 10:59: > On 2018-09-20 10:15, Jeroen Demeyer wrote: >> On 2018-09-20 07:55, Stefan Behnel wrote: >>> I can see that this would be helpful. It's unfortunate, though, that this >>> would introduce a temporary option that complicates the current integer >>> "language_level". >> >> Does it really have to be an integer? We can make it a string >> internally, but allow it to be set as integer too. That's just a matter >> of putting "language_level = str(language_level)" in the appropriate places. That's pretty much what I meant with "complicates". :) It's very easy to compare "language_level < 3" now. With strings, not so much. In fact, maybe that's not even the way to go. There are two parts of information here, so maybe we should actually split them internally (in "Main.Context.set_language_level() ?) and keep the language_level = 3 but just avoid the "unicode_literals" part. >> Once it's a string, we can add an additional language level "3str". I'm >> willing to make a PR if you agree with this strategy. I think a PR that takes care of splitting the two parts would be worth looking at. Then, make sure we do the right checks for either of them in the right places. The language level isn't always the right indication for specific behaviour. > Further brainstorming: you could also re-purpose language_level as a set of > flags describing various features, like the __future__ flags. I faintly remember proposals for a "from __past__ import …". One more use case right here. But the problem is that there are really different things to support. One is setting the behaviour globally, e.g. via setup.py. The other is setting it on a file-by-file basis. I expect projects to settle on one way to write code, so a global setting is definitely required, and that's the language level. But then, there are probably cases where this is worth overriding for a single file or two. And the language level doesn't seem right for that. Explicitly opting out of "unicode_literals" would be cleaner. So, maybe just allow "cython: unicode_literals=False" ? Stefan

1 1

Enhancing "ctyepdef class numpy.ndarray" with getter properties
by Matti Picus Sept. 28, 2018

Sept. 28, 2018

To solve issue #2498, I did some experiments https://github.com/cython/cython/issues/2498#issuecomment-414543549 with hiding direct field access in an external extension type (documented here https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#…). The idea is to write `a.ndims` in cython (in plain python code), and in C magically get the attribute lookup converted into a `PyArray_NDIMS(a)` getter, which could be a macro or a c-function. The experiments proved fruitful, and garnered some positive feedback so I am pushing forward. I would like to get some feedback on syntax before I progress too far. Should the syntax be extended to support |ctypedef class numpy.ndarray [object PyArrayObject]: cdef: # Convert python __getattr__ access to c functions. int ndims PyArray_NDIMS | or perhaps a decorator, like Python |ctypedef class numpy.ndarray [object PyArrayObject]: cdef: # Convert python __getattr__ access to c functions. @property cdef int ndims(self): return PyArray_NDIMS(self) or something else? The second seems more wordy but more explicit. I don't know which would be easier to implement or require more effort to test and maintain. Matti |

2 6

Re: [Cython] Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
by Stefan Behnel Sept. 25, 2018

Sept. 25, 2018

Stefan Behnel schrieb am 21.09.2018 um 17:46: > Robert Bradshaw schrieb am 21.09.2018 um 17:30: >> I agree that this doesn't really feel like a language-level thing. >> >> There seem three desired behaviors here: >> >> language_level=2 where currently "abc" is always a bytes object >> langauge_level=3 where currently "abc" is always a unicode object >> >> and a third option, where "abc" is a str object (depending on the >> runtime). We should support all three of these modes. > > Correction: with "language_level=2", unprefixed string literals are "str", > i.e. "bytes" in Py2 and "unicode" in Py3. "language_level=3" always makes > them unicode strings, and that's what Jeroen was referring to. > > It's really difficult to guess what the most common use case is here. The > only reason why the type changing unprefixed "str" literals are relevant is > that Py2 cannot handle Unicode in some cases (and Unicode strings are > memory hogs in Py2), but Py3 requires Unicode in most cases (and handles it > efficiently). So, the problem here is really Py2, and the problem will go > away as soon as we dump support for it. Until then, however, we're trying > to find a solution to make the language level switch bearable and easy the > transition. I added a new directive "str_is_str=True" which can be combined with "language_level=3" to get the desired behaviour. It keeps the 'str' builtin type as it is (it would otherwise become 'unicode' with level 3) and keeps unprefixed string literals as type 'str' in Py2 and Py3. Everything else should depend solely on the language_level switch. https://github.com/cython/cython/commit/cea42915c5e9ea1da9187aa3c55f3f16d04… I think we're now set for the release. I'll prepare a candidate. Stefan

3 4

Safer exception handling
by Robert Bradshaw Sept. 20, 2018

Sept. 20, 2018

One of the pain points in Cython is that one must explicitly annotate non-object returning functions with except clauses. Would it be worth trying to change the default here, making exception-suppressing opt-in rather than opt-out? There are a couple of open questions, e.g. * What would the syntax be? * What about extern functions? * What would the performance impact be? Could it be mitigated?

2 3

Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
by Stefan Behnel Sept. 20, 2018

Sept. 20, 2018

Jeroen Demeyer schrieb am 18.09.2018 um 10:23: > On 2018-09-16 17:48, Stefan Behnel wrote: >> * Cython now emits a warning when no ``language_level`` (2 or 3) is set >> explicitly > > Currently, language_level=3 breaks a lot of code due to unicode_literals. > As I mentioned a few times, it would be good to have a way to specify > "language_level=3 except for unicode_literals". Ideally, this option should > exist before 0.29 is released. I can see that this would be helpful. It's unfortunate, though, that this would introduce a temporary option that complicates the current integer "language_level". Meaning, we'd introduce something now that we then can't get rid of for…ever. And it seems to me like a separate directive wouldn't solve this either because it's not really orthogonal to the language level. Any idea? Stefan

1 0

Re: [Cython] Safer exception handling
by Jeroen Demeyer Sept. 19, 2018

Sept. 19, 2018

On 2018-09-17 15:44, Robert Bradshaw wrote: > One of the pain points in Cython is that one must explicitly annotate > non-object returning functions with except clauses. Would it be worth > trying to change the default here, making exception-suppressing opt-in > rather than opt-out? Just to clarify things: are you proposing that the default would be "except *"? An alternative would be to give *warnings* for functions where exceptions could occur but could not be propagated. For example, this function is totally fine: cdef int foo(int x): return x but this function would give a warning: cdef int foo(x): return x # implicit conversion Python -> int and this should probably be a compile-time error: cdef int foo(x): raise NotImplementedError

2 1

Re: [Cython] Safer exception handling
by Jeroen Demeyer Sept. 18, 2018

Sept. 18, 2018

On 2018-09-17 15:44, Robert Bradshaw wrote: > One of the pain points in Cython is that one must explicitly annotate > non-object returning functions with except clauses. Would it be worth > trying to change the default here, making exception-suppressing opt-in > rather than opt-out? > > There are a couple of open questions, e.g. > * What would the syntax be? > * What about extern functions? > * What would the performance impact be? Could it be mitigated? Fourth pain point: function pointers. Typically, those are used for pure C code where you don't want exception checking. And since this is opened for discussion, allow me to repeat a related request that I made some years ago but which was rejected: have a way to say "this function does not return anything, but it technically returns an int just for exception handling. 0 is returned on success and -1 on error". There are quite a few Python/C API functions like that and I also commonly write such functions myself. Of course you can do cdef int foo(...) except -1 but this is awkward since the "int" is misleading. It is also wrong for cpdef functions where the function should return None but it returns 0 instead. The syntax I suggested at the time was something like cdef nint foo(...) where nint is like int except that it converts to/from None, just like bint converts to/from bool. Jeroen.

1 0

Re: [Cython] Cython 0.29 – or 29.0 ?
by Jeroen Demeyer Sept. 17, 2018

Sept. 17, 2018

On 2018-08-19 08:26, Stefan Behnel wrote: > Should we make that a new directive rather than a language level? Like "py2_str=str"? That would allow its use together with language_level=3 already in the next release. With a new new directive, you also run into compatibility problems. What should the default be? py2_str=str or py2_str=unicode? The former breaks code assuming that it's unicode and the latter doesn't really solve anything: stuff will still break when language_level=3 becomes the default. My proposal is a new setting language_level=3str (meaning: everything that language_level=3 does, except unicode_literals) and make that the default. That way, you keep full compatibility with code already setting the language_level. You also have reasonably good chances that code that currently uses the implicit language_level=2 will continue to work with language_level=3str.

3 2

Re: [Cython] Hooking tp_clear()
by Jeroen Demeyer Sept. 7, 2018

Sept. 7, 2018

On 2018-09-07 06:35, Stefan Behnel wrote: > Maybe you actually want "tp_finalize"? > > https://www.python.org/dev/peps/pep-0442/ > > Cython moves "__del__" methods there in Py3.4+. First of all, are you sure? I tried to cythonize cdef class X: def __del__(self): pass but the generated C code has static PyTypeObject __pyx_type_6cytest_X = { /* ... */ #if PY_VERSION_HEX >= 0x030400a1 0, /*tp_finalize*/ #endif }; In any case, I want something which works with Python 2.7 too. Could we somehow call __del__ from tp_clear() in Python 2.7? Jeroen.

2 1