Mailman 3 - cython-devel

Shared utility Option vs. Compilation Directive
by matus valo March 3, 2025

March 3, 2025

Hi All, Based on the comment https://github.com/cython/cython/pull/6531#discussion_r1975502026 I would like to start a discussion about using Option vs Compilation Directive. Basically, we have 3 options: 1. Use PR https://github.com/cython/cython/pull/6531 as is. I will proceed with the rest of the comments. 2. Use `Option` directly with no further changes in the Cython codebase. This is the initial implementation. I created a branch with a version of the PR with rollback of all changes related to the Compilation Directive adjustments: https://github.com/cython/cython/compare/master...matusvalo:shared_library_… 3. Use code changes/cleanup in https://github.com/cython/cython/pull/6646 but Shared Utility will be configured using `Option` instead of Compilation directive (Option 2.). I am not sure it this point makes sense, only in case that we want to keep changes by @da-woods <dw-git(a)d-woods.co.uk> improves the code internals. I would like to have an agreement on this topic before moving forward with other comments in the PR. Thanks, Matus

2 2

CLI for using Shared utility module
by matus valo Feb. 17, 2025

Feb. 17, 2025

Hi All, I am writing to you to not get lost in https://github.com/cython/cython/pull/6531. I would like to get an agreement on how we would like to proceed with a CLI. The PR has two parts: 1. Utility for generating C file of shared utility. Here, I propose to pass the full path to the file e.g.: cython_shared_module scipy/_lib/cyshared.c The reason why not to use the import path is because the target package can be hidden inside a directory (e.g. src). This does not work well with the import path. Additional questions: * what we will call the utility for creation of the module? * should it be executable or we will use python -m '...' approach? * should utility create directories or to fail if directory does not exist? 2. cython parameter to instruct cython to compile module with shared utility Here, I propose to use import path and compilation directive: cython -X use_shared_utility=scipy._lib.cyshared my_module.pyx The reason to use the import path is because the file must be internally imported in any case. The import path is a full path including the shared module. Could you help me answer the mentioned questions? I think this is last piece of puzzle and after the PR should be finally complete (there are still PR review needed and rough edges to be polished). Thanks, Matus

4 9

How to see why distutils-powered tests failed?
by Miro Hrončok June 20, 2024

June 20, 2024

Hi. Apparently, I was not properly subscribed to the list when I sent this, trying again: -------- Forwarded Message -------- Subject: How to see why distutils-powered tests failed? Date: Mon, 13 May 2024 12:10:20 +0200 From: Miro Hrončok <mhroncok(a)redhat.com> To: cython-devel(a)python.org Hello, I've recently encountered the following test failure when I wanted to run the upstream Cython testusite on Fedora CI. ====================================================================== ERROR: runTest (__main__.CythonRunTestCase.runTest) [-1] compiling (cpp/cy2/pythran) and running numpy_pythran ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 266, in link self.spawn(linker + ld_args) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn spawn(cmd, dry_run=self.dry_run, **kwargs) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn raise DistutilsExecError( distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1550, in run ext_so_path = self.runCompileTest() ^^^^^^^^^^^^^^^^^^^^^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1163, in runCompileTest return self.compile( ^^^^^^^^^^^^^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1482, in compile so_path = self.run_distutils(test_directory, module, workdir, incdir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1378, in run_distutils build_extension.run() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run self.build_extensions() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions self._build_extensions_serial() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial self.build_extension(ext) File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 637, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 572, in build_extension self.compiler.link_shared_object( File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 752, in link_shared_object self.link( File "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 268, in link raise LinkError(msg) distutils.errors.LinkError: command '/usr/bin/g++' failed with exit code 1 There were 4 failures like this (all pythran_numpy related). I was clueless as to why is this failing. I was trying to make the tests tell me more. After long digging I ended up modifying /usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py in a test container directly to always make this conditional False: https://github.com/pypa/setuptools/blob/v69.1.0/setuptools/_distutils/spawn… This turned: distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1 Into: distutils.errors.DistutilsExecError: command ['/usr/bin/g++', ...] failed with exit code 1 Then, I was able to rerun the failed g++ command manually to see the actual linker error (missing flexiblas). However, the experience was far from ideal. Would it be possible (and desired) to modify the tests to actually print out the compiler/linker errors when they happen? Or perhaps this already is possible, but I am not doing it correctly? (I tried --no-capture, but no dice.) Thanks, -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 1

Contributing to the Cython Project as a beginner
by So Min An April 16, 2024

April 16, 2024

Dear Cython Core Development Team, I hope this email finds you well. My name is Somin An and I am a junior studying at the University of Michigan. As part of my final project for a course, I was hoping to work on one or two beginner issues for the Cython project. This is my very first time attempting to contribute to an open-source project and would love to receive some guidance. I would appreciate any information on how I might be able to contribute. I am currently reading through the CONTRIBUTING.rst and the HackerGuide. If you could let me know at your earliest convenience, I would greatly appreciate it. Sincerely, Somin An

2 1

Ready for Cython 3.1 ?
by Stefan Behnel Nov. 7, 2023

Nov. 7, 2023

Hi all, it looks like Cython 3.0.6 is going to be a "most things fixed" kind of release for the 3.0.x release series. Given the work that lies ahead of us for Cython 3.1, I think we're at a point to get started on that, making the future 3.0.x releases stable and "boring". As a reminder, Cython 3.1 will remove support for Python 2.7 and Python 3.[567], i.e. all Python versions that are now EOL. Python 3.8 will continue to receive security fixes for another year. Python 3.7 is EOL but still up for debate since it's probably not hard to support and still maintained in some Linux distributions for another couple of years. But I'm fine with considering it legacy. We'll probably notice if it gets in the way while preparing Cython 3.0, and can leave support in until there's a reason to remove it. https://github.com/cython/cython/issues/2800 I'd like to ease our feature development by using more modern Python features in our code base and by targeting less Python versions in Cython 3.1 compared to the "all things supported" Cython 3.0. I also consider Cython 3.1 a prime target for better Limited API support. Users probably won't care both for that and for outdated Python versions at the same time. Or, they can use Cython 3.0.x for continued legacy support. Since Cython 3.1 is mostly about ripping out old code, we can try to keep the development cycle short, so that new features don't have to wait that long. Certainly not as long as for Cython 3.0… Is everyone and everything ready to start working on Cython 3.1? Stefan

4 6

Should we start using the internal CPython APIs?
by Stefan Behnel Nov. 5, 2023

Nov. 5, 2023

Hi all, given the latest blow against exposing implementation details of CPython in their C-API (see https://github.com/cython/cython/pull/5767 for the endless story), I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal This would give us greater freedom in accessing all the implementation details, so that we could directly integrate with those. We'd obviously still need one or more fallback implementations for "stable CPython", Limited API, PyPy and friends. There's a risk, clearly, that these internals change even during point releases. Maybe not a big risk, but not impossible either. We'd have to deal with that and so would our users. OTOH, having a single macro switch would make it easy for users to adapt if something breaks on their side, and also easy to benchmark if it makes a difference for their code. We could also leave it off by default and simply allow users with high performance needs to enable it manually. Or start by leaving it off until a new CPython X.Y release has stabilised and its (used-by-us) internals have proven not to change, and then switch it on for that release series. In any case, having a single switch for this feels like it could be easy to handle. What do you think? Stefan

4 5

Can we remove the FastGIL implementation?
by Stefan Behnel Sept. 20, 2023

Sept. 20, 2023

Hi, I've seen reports that Cython's "FastGIL" implementation (which basically keeps the GIL state in a thread-local variable) is no longer faster than CPython's plain GIL implementation in recent Python 3.x versions. Potentially even slower. See the report in https://github.com/cython/cython/issues/5703 It would be helpful to get user feedback on this. If you have GIL-heavy Cython code, especially with nested with-nogil/with-gil sections across functions, and a benchmark that exercises it, could you please run the benchmark with and without the feature enabled and report the results? You can add "-DCYTHON_FAST_GIL=0" to your CFLAGS to disabled it (and "=1" to enable it explicitly). It's enabled by default in CPython 3.6-3.11 (but disabled in Cython 0.29.x on Python 3.11). Thanks, Stefan

3 5

Cython 3.0.2 released
by Stefan Behnel Aug. 27, 2023

Aug. 27, 2023

Hi all, Cython 3.0.2 is released. It fixes two major regressions in 3.0.1, so please upgrade if that failed for you. https://cython.readthedocs.io/en/latest/src/changes.html Have fun, Stefan

1 0

uvloop issue in Cython3
by matus valo Aug. 14, 2023

Aug. 14, 2023

Hi All, I was recently migrating uvloop to cython3 [1]. Unfortunately, after switching to cython3, uvloop testsuite fails [2]. I am puzzled mainly about following error: ====================================================================== ERROR: test_process_delayed_stdio__paused__no_stdin (test_process.Test_UV_Process_Delayed) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line 938, in test_process_delayed_stdio__paused__no_stdin __uvloop_sleep_after_fork=True)) File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete return future.result() File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line 911, in run_sub **kwargs) File "uvloop/loop.pyx", line 2808, in subprocess_shell return await self.__subprocess_run(protocol_factory, args, shell=True, File "uvloop/loop.pyx", line 2734, in uvloop.loop.Loop._Loop__subprocess_run @cython.iterable_coroutine TypeError: _Loop__subprocess_run() got an unexpected keyword argument '__uvloop_sleep_after_fork' ====================================================================== ERROR: test_process_delayed_stdio__paused__stdin_pipe (test_process.Test_UV_Process_Delayed) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line 919, in test_process_delayed_stdio__paused__stdin_pipe __uvloop_sleep_after_fork=True)) File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete return future.result() File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line 911, in run_sub **kwargs) File "uvloop/loop.pyx", line 2808, in subprocess_shell return await self.__subprocess_run(protocol_factory, args, shell=True, File "uvloop/loop.pyx", line 2734, in uvloop.loop.Loop._Loop__subprocess_run @cython.iterable_coroutine TypeError: _Loop__subprocess_run() got an unexpected keyword argument '__uvloop_sleep_after_fork' The issue is regarding calling this method: https://github.com/MagicStack/uvloop/blob/1dd40f17f3b0d37e3779b6ad5041bab33… When the same code base is compiled in Cython 0.29.X and the test suite is run, there is no such error. I tried to create a simple reproducer but failed. What do you think? Is it regression in Cython 3.0? Do you have any hint where the root cause can be? If everything fails, I will try to bisect the issue. Matus [1] https://github.com/MagicStack/uvloop/pull/534 [2] https://github.com/MagicStack/uvloop/pull/534/checks

2 3

Cython 3.0 final released
by Stefan Behnel July 17, 2023

July 17, 2023

Hi all, after close to five long years, I'm proud to announce the release of Cython 3.0. It's done. It's out. Finally! The full list of improvements compared to the 0.29.x release series is entirely incredible. https://cython.readthedocs.io/en/latest/src/changes.html Cython 3.0 is better than any other Cython release before, in all aspects. It's much more Python, integrates better with C and C++, supports more Python implementations and configurations, provides many great new language features – it's faster, safer and easier to use. It's simply better. New language features include: - Python 3 syntax and semantics by default - Cython type annotations in plain Python code - automatic NumPy ufunc generation - fast @dataclass and @total_ordering extension types - safe exception propagation in C functions by default - Unicode identifiers in Cython code All of this wouldn't have been possible without the help of the many, many people who contributed code and documentation, tested features, found and described bugs, helped debugging problems. Those who started using Cython in new environments, new build systems, new use cases, and helped to get it working there. Who proposed new features or found mismatches and gaps in the existing set of features. Thank you all, you helped making Cython 3.0 an awesome language! Along the way, we added two people to the list of Cython developers. * David Woods has contributed a tremendous list of features and fixes to this release. It would honestly not have been possible without his efforts. * Matúš Valo has put a lot of work into the documentation and the pure Python mode. He found many issues that make Cython now easier and more consistent to use from Python code. Thank you both for your contributions. I'm happy to work together with you. Everyone, have fun using Cython 3.0, and whatever good comes after it. Best, Stefan

1 0