[Cython] Cython 0.23 released
Antoine Martin
antoine at nagafix.co.uk
Mon Aug 17 06:09:09 CEST 2015
On 09/08/15 13:54, Antoine Martin wrote:
> Hi,
>
> I've just hit a regression with 0.23: a recursion loop generating code
> for this particular pyx:
> http://xpra.org/trac/browser/xpra/trunk/src/xpra/codecs/nvenc4/encoder.pyx
>
> I don't think we have such loops in our setup.py data structures or in
> that pyx, but that's possible. (I will try to check later)
> The same code builds fine on all platforms with Cython 0.22.1.
I have narrowed it down to the problematic finally block via trial and
error:
http://xpra.org/trac/browser/xpra/trunk/src/xpra/codecs/nvenc4/encoder.pyx#L2024
Any of those statements, including the simple logging ones, cause
Cython's "fresh_finally_clause" code to go into an infinite loop via
deepcopy.
This "fixes" it by moving the statements out of the finally block
(obviously not a practical solution):
--- xpra/codecs/nvenc4/encoder.pyx (revision 10311)
+++ xpra/codecs/nvenc4/encoder.pyx (working copy)
@@ -2022,16 +2022,17 @@
self.bytes_out += size
data = (<char *> lockOutputBuffer.bitstreamBufferPtr)[:size]
finally:
- if DEBUG_API:
- log("nvEncUnlockBitstream(%#x)", <unsigned long>
self.bitstreamBuffer)
- with nogil:
- r =
self.functionList.nvEncUnlockBitstream(self.context, self.bitstreamBuffer)
- raiseNVENC(r, "unlocking output buffer")
- if DEBUG_API:
- log("nvEncUnmapInputResource(%#x)", <unsigned long>
self.bitstreamBuffer)
- with nogil:
- r =
self.functionList.nvEncUnmapInputResource(self.context,
mapInputResource.mappedResource)
- raiseNVENC(r, "unmapping input resource")
+ pass
+ if DEBUG_API:
+ log("nvEncUnlockBitstream(%#x)", <unsigned long>
self.bitstreamBuffer)
+ with nogil:
+ r = self.functionList.nvEncUnlockBitstream(self.context,
self.bitstreamBuffer)
+ raiseNVENC(r, "unlocking output buffer")
+ if DEBUG_API:
+ log("nvEncUnmapInputResource(%#x)", <unsigned long>
self.bitstreamBuffer)
+ with nogil:
+ r = self.functionList.nvEncUnmapInputResource(self.context,
mapInputResource.mappedResource)
+ raiseNVENC(r, "unmapping input resource")
download_end = time.time()
log("compress_image(..) download took %.1f ms",
(download_end-encode_end)*1000.0)
What am I doing wrong here?
Those statements look fine and they used to compile just fine too.
Thanks
Antoine
>
> Full stack trace attached to this email. Loop excerpt:
> File "Cython/Compiler/FlowControl.py", line 336, in
> Cython.Compiler.FlowControl.NameAssignment.__deepcopy__
> (/usr/src/rpmbuild/BUILD/Cython-0.23/Cython/Compiler/FlowControl.c:10989)
> File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib64/python2.7/copy.py", line 328, in _reconstruct
> args = deepcopy(args, memo)
> File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib64/python2.7/copy.py", line 237, in _deepcopy_tuple
> y.append(deepcopy(a, memo))
> File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib64/python2.7/copy.py", line 230, in _deepcopy_list
> y.append(deepcopy(a, memo))
> File "/usr/lib64/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "Cython/Compiler/FlowControl.py", line 418, in
> Cython.Compiler.FlowControl.NameReference.__deepcopy__
> (/usr/src/rpmbuild/BUILD/Cython-0.23/Cython/Compiler/FlowControl.c:13685)
> File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib64/python2.7/copy.py", line 334, in _reconstruct
> state = deepcopy(state, memo)
> File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib64/python2.7/copy.py", line 351, in _reconstruct
> item = deepcopy(item, memo)
> File "/usr/lib64/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "Cython/Compiler/FlowControl.py", line 336, in
> Cython.Compiler.FlowControl.NameAssignment.__deepcopy__
> (/usr/src/rpmbuild/BUILD/Cython-0.23/Cython/Compiler/FlowControl.c:10989)
> ....
> Sorry for not testing the 0.23 beta, which might have caught this sooner.
>
> Cheers
> Antoine
>
>
> On 09/08/15 01:47, Stefan Behnel wrote:
>> Hi everyone,
>>
>> given the positive feedback on the previous beta releases, I'm happy to
>> announce the final release of Cython 0.23. There were only minor changes
>> and fixes since the second beta.
>>
>> https://pypi.python.org/pypi/Cython/0.23
>>
>> This is a major feature release. Highlights include, in addition to several
>> important bug fixes:
>>
>> * PEP 492 (async/await) and PEP 448 (star-unpacking generalizations)
>> * inlined generator expressions
>> * support for the C++ assignment and bool() operators
>> * support for coverage analysis
>>
>> Complete changelog follows below. Upgrading is generally recommended.
>>
>>
>> You can get the signed release from here:
>>
>> http://cython.org/
>>
>> http://cython.org/release/Cython-0.23.tar.gz
>>
>> http://cython.org/release/Cython-0.23.zip
>>
>>
>> SHA1 sums:
>> 5a49123b39d89a20fb88bdc7886210c39cdff695 Cython-0.23.tar.gz
>> 9bec6c6df861be49ed608a0ab688b67bba543abe Cython-0.23.zip
>>
>> Have fun,
>>
>> Stefan
>>
>>
>>
>> 0.23 (2015-08-08)
>> =================
>>
>> Features added
>> --------------
>>
>> * PEP 492 (async/await) was implemented.
>> See https://www.python.org/dev/peps/pep-0492/
>>
>> * PEP 448 (Additional Unpacking Generalizations) was implemented.
>> See https://www.python.org/dev/peps/pep-0448/
>>
>> * Support for coverage.py 4.0+ can be enabled by adding the plugin
>> "Cython.Coverage" to the ".coveragerc" config file.
>>
>> * Annotated HTML source pages can integrate (XML) coverage reports.
>>
>> * Tracing is supported in ``nogil`` functions/sections and module init
>> code.
>>
>> * When generators are used in a Cython module and the module imports the
>> modules "inspect" and/or "asyncio", Cython enables interoperability by
>> patching these modules during the import to recognise Cython's internal
>> generator and coroutine types. This can be disabled by C compiling the
>> module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0"
>>
>> * When generators or coroutines are used in a Cython module, their types
>> are registered with the ``Generator`` and ``Coroutine`` ABCs in the
>> ``collections`` or ``collections.abc`` stdlib module at import time to
>> enable interoperability with code that needs to detect and process Python
>> generators/coroutines. These ABCs were added in CPython 3.5 and are
>> available for older Python versions through the ``backports_abc`` module
>> on PyPI. See https://bugs.python.org/issue24018
>>
>> * Adding/subtracting/dividing/modulus and equality comparisons with
>> constant Python floats and small integers are faster.
>>
>> * Binary and/or/xor/rshift operations with small constant Python integers
>> are faster.
>>
>> * When called on generator expressions, the builtins ``all()``, ``any()``,
>> ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()``
>> avoid the generator iteration overhead by inlining a part of their
>> functionality into the for-loop.
>>
>> * Keyword argument dicts are no longer copied on function entry when they
>> are not being used or only passed through to other function calls (e.g.
>> in wrapper functions).
>>
>> * The ``PyTypeObject`` declaration in ``cpython.object`` was extended.
>>
>> * The builtin ``type`` type is now declared as PyTypeObject in source,
>> allowing for extern functions taking type parameters to have the correct
>> C signatures. Note that this might break code that uses ``type`` just
>> for passing around Python types in typed variables. Removing the type
>> declaration provides a backwards compatible fix.
>>
>> * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure
>> Python mode.
>>
>> * Const iterators were added to the provided C++ STL declarations.
>>
>> * Smart pointers were added to the provided C++ STL declarations.
>> Patch by Daniel Filonik.
>>
>> * ``NULL`` is allowed as default argument when embedding signatures.
>> This fixes ticket 843.
>>
>> * When compiling with ``--embed``, the internal module name is changed to
>> ``__main__`` to allow arbitrary program names, including those that would
>> be invalid for modules. Note that this prevents reuse of the generated
>> C code as an importable module.
>>
>> * External C++ classes that overload the assignment operator can be used.
>> Patch by Ian Henriksen.
>>
>> * Support operator bool() for C++ classes so they can be used in
>> if-statements.
>>
>> Bugs fixed
>> ----------
>>
>> * Calling "yield from" from Python on a Cython generator that returned a
>> value triggered a crash in CPython. This is now being worked around.
>> See https://bugs.python.org/issue23996
>>
>> * Language level 3 did not enable true division (a.k.a. float division)
>> for integer operands.
>>
>> * Functions with fused argument types that included a generic 'object'
>> fallback could end up using that fallback also for other explicitly
>> listed object types.
>>
>> * Relative cimports could accidentally fall back to trying an absolute
>> cimport on failure.
>>
>> * The result of calling a C struct constructor no longer requires an
>> intermediate assignment when coercing to a Python dict.
>>
>> * C++ exception declarations with mapping functions could fail to compile
>> when pre-declared in .pxd files.
>>
>> * ``cpdef void`` methods are now permitted.
>>
>> * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code
>> in C++. Patch by David Vierra, original patch by Michael Enßlin.
>>
>> * Buffer index calculations using index variables with small C integer
>> types could overflow for large buffer sizes.
>> Original patch by David Vierra.
>>
>> * C unions use a saner way to coerce from and to Python dicts.
>>
>> * When compiling a module ``foo.pyx``, the directories in ``sys.path``
>> are no longer searched when looking for ``foo.pxd``.
>> Patch by Jeroen Demeyer.
>>
>> * Memory leaks in the embedding main function were fixed.
>> Original patch by Michael Enßlin.
>>
>> * Some complex Python expressions could fail to compile inside of finally
>> clauses.
>>
>> * Unprefixed 'str' literals were not supported as C varargs arguments.
>>
>> * Fixed type errors in conversion enum types to/from Python. Note that
>> this imposes stricter correctness requirements on enum declarations.
>>
>> Other changes
>> -------------
>>
>> * Changed mangling scheme in header files generated by ``cdef api``
>> declarations.
>>
>> * Installation under CPython 3.3+ no longer requires a pass of the
>> 2to3 tool. This also makes it possible to run Cython in Python
>> 3.3+ from a source checkout without installing it first.
>> Patch by Petr Viktorin.
>>
>> * ``jedi-typer.py`` (in ``Tools/``) was extended and renamed to
>> ``jedityper.py`` (to make it importable) and now works with and
>> requires Jedi 0.9. Patch by Tzer-jen Wei.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
More information about the cython-devel
mailing list