Mailman 3 October 2018 - cython-devel

Re: [Cython] longintrepr.h issues with MinGW
by Jeroen Demeyer Oct. 24, 2018

Oct. 24, 2018

On 2018-10-24 07:58, Jeroen Demeyer wrote: > If you care about MinGW, there is also this CPython issue which makes it > basically impossible to compile any Cython extension on MinGW: > > https://bugs.python.org/issue11566 I forgot to say that this makes it impossible to compile any *C++* Cython extension out of the box on MinGW.

2 1

Re: [Cython] longintrepr.h issues with MinGW
by Jeroen Demeyer Oct. 24, 2018

Oct. 24, 2018

If you care about MinGW, there is also this CPython issue which makes it basically impossible to compile any Cython extension on MinGW: https://bugs.python.org/issue11566

1 0

longintrepr.h issues with MinGW
by Robert Bradshaw Oct. 24, 2018

Oct. 24, 2018

Given that https://bugs.python.org/issue4709 results in extension modules that seem to work, but silently produce completely incorrect answers, I'm thinking we should either disable our long-unpacking code on these platforms, or at the very least give a runtime error if we detect issues like sizeof(void*) != SIZEOF_VOID_P (or both). I'm not sure what has made this issue pop up more lately (Python 3), but perhaps we should even consider releasing bugfixes for previous Cython versions. - Robert

2 3

Re: [Cython] Failing test reimport_from_subinterpreter
by Jeroen Demeyer Oct. 22, 2018

Oct. 22, 2018

OK, I finished analyzing the problem. It's a combination of: 1. the Sage wrapper around the Cython installer setting PYTHONPATH 2. the reimport_from_subinterpreter test failing if PYTHONPATH is set I'll open a PR quick.

1 0

Re: [Cython] Failing test reimport_from_subinterpreter
by Jeroen Demeyer Oct. 22, 2018

Oct. 22, 2018

n 2018-10-22 16:44, Jeroen Demeyer wrote: > The entries marked with # <---- are different but I don't know why. There is a difference in $PYTHONPATH. When it works, PYTHONPATH equals '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/support/lib.linux-ppc64le-2.7:/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src:' Note the trailing colon, which is what makes everything work since that gets translated to the current working directory! When it fails, PYTHONPATH equals '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/support/lib.linux-ppc64le-2.7:/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src:/home/jdemeyer/sage-test/local' ...which has sys.prefix added for some reason.

1 0

Re: [Cython] Failing test reimport_from_subinterpreter
by Jeroen Demeyer Oct. 22, 2018

Oct. 22, 2018

On 2018-10-21 15:44, Stefan Behnel wrote: > Maybe print "sys.path" and "sys.meta_path" in a couple of places? So I did and the problem is indeed related to sys.path. When running *only* the reimport_from_subinterpreter tests, sys.path starts with the following entries: '' '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/support/lib.linux-ppc64le-2.7' '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src' '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter' # <---- '/home/jdemeyer/sage-test/local/lib/python27.zip' '/home/jdemeyer/sage-test/local/lib/python2.7' When running all tests, sys.path starts with the following entries: '' '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/support/lib.linux-ppc64le-2.7' '/home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src' '/home/jdemeyer/sage-test/local' # <---- '/home/jdemeyer/sage-test/local/lib/python27.zip' '/home/jdemeyer/sage-test/local/lib/python2.7' The entries marked with # <---- are different but I don't know why. Further suggestions for debugging are more than welcome... In both cases, the initial entry ('') is dropped in the sub-interpreter. I assume that this is a feature.

1 0

Re: [Cython] Failing test reimport_from_subinterpreter
by Jeroen Demeyer Oct. 21, 2018

Oct. 21, 2018

On 2018-10-16 20:30, Stefan Behnel wrote: > Difficult to say why this would fail to find the package. Could it be an > import path problem? Current directory missing from the PYTHONPATH or > something like that? The problem is that the error depends on the proverbial "phase of the moon". When running the full testsuite, I always get the error. But any attempt at debugging it makes the error go away. So it must be something subtle, like a race condition or bad interaction with another test.

2 1

Failing test reimport_from_subinterpreter
by Jeroen Demeyer Oct. 16, 2018

Oct. 16, 2018

Hello, I'm getting various reports of the test reimport_from_subinterpreter failing (within Sage, Cython 0.29, Python 2.7.15). Annoyingly, the problem can only be reproduced when running the full Cython testsuite, not when running the test in isolation. I'll continue to investigate, but I'm already posting it here in case somebody has an idea. The error is not very enlightening (is there a way to get the exception from the subinterpreter?): End-to-end reimport_from_subinterpreter ... /home/jdemeyer/sage-test/local/bin/python2 setup.py build_ext --inplace Compiling package/subtest.pyx because it changed. Compiling subtest.pyx because it changed. [1/2] Cythonizing package/subtest.pyx [2/2] Cythonizing subtest.pyx running build_ext building 'package.subtest' extension creating build creating build/temp.linux-ppc64le-2.7 creating build/temp.linux-ppc64le-2.7/package gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/jdemeyer/sage-test/local/include/python2.7 -c package/subtest.c -o build/temp.linux-ppc64le-2.7/package/subtest.o gcc -pthread -shared -L/home/jdemeyer/sage-test/local/lib -Wl,-rpath,/home/jdemeyer/sage-test/local/lib -L/home/jdemeyer/sage-test/local/lib -Wl,-rpath,/home/jdemeyer/sage-test/local/lib build/temp.linux-ppc64le-2.7/package/subtest.o -L/home/jdemeyer/sage-test/local/lib -lpython2.7 -o /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/package/subtest.so building 'subtest' extension gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/jdemeyer/sage-test/local/include/python2.7 -c subtest.c -o build/temp.linux-ppc64le-2.7/subtest.o gcc -pthread -shared -L/home/jdemeyer/sage-test/local/lib -Wl,-rpath,/home/jdemeyer/sage-test/local/lib -L/home/jdemeyer/sage-test/local/lib -Wl,-rpath,/home/jdemeyer/sage-test/local/lib build/temp.linux-ppc64le-2.7/subtest.o -L/home/jdemeyer/sage-test/local/lib -lpython2.7 -o /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/subtest.so /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/package/subtest.pyx tree = Parsing.p_module(s, pxd, full_module_name) /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/subtest.pyx tree = Parsing.p_module(s, pxd, full_module_name) /home/jdemeyer/sage-test/local/bin/python2 -c "import subtest; subtest.run_main()" Module loaded: package.subtest /home/jdemeyer/sage-test/local/bin/python2 -c "import subtest; subtest.run_sub()" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named package Traceback (most recent call last): File "<string>", line 1, in <module> File "subtest.pyx", line 44, in subtest.run_sub assert 0 == run_in_subinterpreter(b'import package') AssertionError FAIL

2 1

Cython 0.29 final released
by Stefan Behnel Oct. 14, 2018

Oct. 14, 2018

Dear Cythonistas, after half a year of development, many community pull requests and a lot of feedback and good ideas in online discussions and at conferences, I'm proud to release Cython 0.29. This is a major feature release that comes with many great improvements and several important bug fixes. See the long list of changes below. Download: https://pypi.org/project/Cython/0.29/ https://github.com/cython/cython/releases/tag/0.29 Changelog: https://github.com/cython/cython/blob/0.29/CHANGES.rst Foresight: Given that Cython has been in critical production use all over the world for several years, but never found the perfect time for a 1.0 version bump, we designate this to become the last 0.x release of the project and decided to skip the 1.0 release which the 0.x series has long represented anyway. Planning has already started [1] for the next major release, titled "3.0". It will finally switch the default Cython language level from Py2 to Py3, to match what users expect from a Python compiler these days without additional options or configuration. Cython 2 code will continue to be supported as before with the directive "language_level=2", although the future default (called "3str", Py3 with str literals) is already available in the 0.29 release, so users might prefer switching to that. We're happy to hear your feedback on the proposed changes. [1] https://github.com/cython/cython/milestone/58 Have fun, Stefan 0.29 (2018-10-14) ================= Features added -------------- * PEP-489 multi-phase module initialisation has been enabled again. Module reloads in other subinterpreters raise an exception to prevent corruption of the static module state. * A set of ``mypy`` compatible PEP-484 declarations were added for Cython's C data types to integrate with static analysers in typed Python code. They are available in the ``Cython/Shadow.pyi`` module and describe the types in the special ``cython`` module that can be used for typing in Python code. Original patch by Julian Gethmann. (Github issue #1965) * Memoryviews are supported in PEP-484/526 style type declarations. (Github issue #2529) * ``(a)cython.nogil`` is supported as a C-function decorator in Python code. (Github issue #2557) * Raising exceptions from nogil code will automatically acquire the GIL, instead of requiring an explicit ``with gil`` block. * C++ functions can now be declared as potentially raising both C++ and Python exceptions, so that Cython can handle both correctly. (Github issue #2615) * ``cython.inline()`` supports a direct ``language_level`` keyword argument that was previously only available via a directive. * A new language level name ``3str`` was added that mostly corresponds to language level 3, but keeps unprefixed string literals as type 'str' in both Py2 and Py3, and the builtin 'str' type unchanged. This will become the default in the next Cython release and is meant to help user code a) transition more easily to this new default and b) migrate to Python 3 source code semantics without making support for Python 2.x difficult. * In CPython 3.6 and later, looking up globals in the module dict is almost as fast as looking up C globals. (Github issue #2313) * For a Python subclass of an extension type, repeated method calls to non-overridden cpdef methods can avoid the attribute lookup in Py3.6+, which makes them 4x faster. (Github issue #2313) * (In-)equality comparisons of objects to integer literals are faster. (Github issue #2188) * Some internal and 1-argument method calls are faster. * Modules that cimport many external extension types from other Cython modules execute less import requests during module initialisation. * Constant tuples and slices are deduplicated and only created once per module. (Github issue #2292) * The coverage plugin considers more C file extensions such as ``.cc`` and ``.cxx``. (Github issue #2266) * The ``cythonize`` command accepts compile time variable values (as set by ``DEF``) through the new ``-E`` option. Patch by Jerome Kieffer. (Github issue #2315) * ``pyximport`` can import from namespace packages. Patch by Prakhar Goel. (Github issue #2294) * Some missing numpy and CPython C-API declarations were added. Patch by John Kirkham. (Github issues #2523, #2520, #2537) * Declarations for the ``pylifecycle`` C-API functions were added in a new .pxd file ``cpython.pylifecycle``. * The Pythran support was updated to work with the latest Pythran 0.8.7. Original patch by Adrien Guinet. (Github issue #2600) * ``%a`` is included in the string formatting types that are optimised into f-strings. In this case, it is also automatically mapped to ``%r`` in Python 2.x. * New C macro ``CYTHON_HEX_VERSION`` to access Cython's version in the same style as ``PY_HEX_VERSION``. * Constants in ``libc.math`` are now declared as ``const`` to simplify their handling. * An additional ``check_size`` clause was added to the ``ctypedef class`` name specification to allow suppressing warnings when importing modules with backwards-compatible ``PyTypeObject`` size changes. Patch by Matti Picus. (Github issue #2627) Bugs fixed ---------- * The exception handling in generators and coroutines under CPython 3.7 was adapted to the newly introduced exception stack. Users of Cython 0.28 who want to support Python 3.7 are encouraged to upgrade to 0.29 to avoid potentially incorrect error reporting and tracebacks. (Github issue #1958) * Crash when importing a module under Stackless Python that was built for CPython. Patch by Anselm Kruis. (Github issue #2534) * 2-value slicing of typed sequences failed if the start or stop index was None. Patch by Christian Gibson. (Github issue #2508) * Multiplied string literals lost their factor when they are part of another constant expression (e.g. 'x' * 10 + 'y' => 'xy'). * String formatting with the '%' operator didn't call the special ``__rmod__()`` method if the right side is a string subclass that implements it. (Python issue 28598) * The directive ``language_level=3`` did not apply to the first token in the source file. (Github issue #2230) * Overriding cpdef methods did not work in Python subclasses with slots. Note that this can have a performance impact on calls from Cython code. (Github issue #1771) * Fix declarations of builtin or C types using strings in pure python mode. (Github issue #2046) * Generator expressions and lambdas failed to compile in ``@cfunc`` functions. (Github issue #459) * Global names with ``const`` types were not excluded from star-import assignments which could lead to invalid C code. (Github issue #2621) * Several internal function signatures were fixed that lead to warnings in gcc-8. (Github issue #2363) * The numpy helper functions ``set_array_base()`` and ``get_array_base()`` were adapted to the current numpy C-API recommendations. Patch by Matti Picus. (Github issue #2528) * Some NumPy related code was updated to avoid deprecated API usage. Original patch by jbrockmendel. (Github issue #2559) * Several C++ STL declarations were extended and corrected. Patch by Valentin Valls. (Github issue #2207) * C lines of the module init function were unconditionally not reported in exception stack traces. Patch by Jeroen Demeyer. (Github issue #2492) * When PEP-489 support is enabled, reloading the module overwrote any static module state. It now raises an exception instead, given that reloading is not actually supported. * Object-returning, C++ exception throwing functions were not checking that the return value was non-null. Original patch by Matt Wozniski (Github Issue #2603) * The source file encoding detection could get confused if the ``c_string_encoding`` directive appeared within the first two lines. (Github issue #2632) * Cython generated modules no longer emit a warning during import when the size of the NumPy array type is larger than what was found at compile. Instead, this is assumed to be a backwards compatible change on NumPy side. Other changes ------------- * Cython now emits a warning when no ``language_level`` (2, 3 or '3str') is set explicitly, neither as a ``cythonize()`` option nor as a compiler directive. This is meant to prepare the transition of the default language level from currently Py2 to Py3, since that is what most new users will expect these days. The future default will, however, not enforce unicode literals, because this has proven a major obstacle in the support for both Python 2.x and 3.x. The next major release is intended to make this change, so that it will parse all code that does not request a specific language level as Python 3 code, but with ``str`` literals. The language level 2 will continue to be supported for an indefinite time. * The documentation was restructured, cleaned up and examples are now tested. The NumPy tutorial was also rewritten to simplify the running example. Contributed by Gabriel de Marmiesse. (Github issue #2245) * Cython compiles less of its own modules at build time to reduce the installed package size to about half of its previous size. This makes the compiler slightly slower, by about 5-7%.

1 0