Antoine Pitrou schrieb am 31.07.2018 um 09:45:

On Tue, 31 Jul 2018 09:27:03 +0200 Jeroen Demeyer J.Demeyer@UGent.be wrote:

...

On 2018-07-31 08:58, Antoine Pitrou wrote:

...

I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience).

I know this probably isn't the correct place to bring it up, but I'm sure that CPython itself could benefit from using Cython. For example, most of the C extensions in Modules/ could be written in Cython.

We don't depend on any third-party Python modules. Adding a Cython dependency for CPython development would be a tough sell.

I don't really want to get into that discussion (it's more about processes than arguments), but let me note that the CPython development already has a couple of dependencies, such as github and its bots, or tools like argument clinic (admittedly included), make and a C compiler (not included), and a text editor. It's not like it's free of tools that help in writing and maintaining the code. That's pretty much the level at which I also see Cython. It's more complex than argument clinic, but it otherwise serves a similar need.

Also, a C extension can be built-in (linked statically into the interpreter), which I think would be hard to do with Cython.

Someone recently contributed a feature of hiding the pyinit function for the embedding case, so people do these things already. This could use the normal inittab mechanism, for example. What I think you might be referring to is that Cython modules require the CPython runtime to be initialised to a certain extent, so you couldn't implement "sys" in Cython, for example. But Jeroen is right, Cython should be a viable option for (most of?) the extension modules in the stdlib. Whether the CPython core devs would accept it in their workflow or not is a totally different question.

Stefan

I don’t think there would be any trouble linking in the generated C files. The built in modules like this just have their *_init() functions called at startup, so possibly nothing would even change.

Top-posted from my Windows 10 phone

From: Brett Cannon Sent: Wednesday, 1 August 2018 17:20 To: Stefan Behnel Cc: python-dev@python.org Subject: [Python-Dev] Using Cython for the stdlib (was: Let's change to CAPI!)

[new thread as this no longer has anything to do with Victor's proposal]

On Tue, 31 Jul 2018 at 13:42 Stefan Behnel stefan_ml@behnel.de wrote: Antoine Pitrou schrieb am 31.07.2018 um 09:45:

On Tue, 31 Jul 2018 09:27:03 +0200 Jeroen Demeyer J.Demeyer@UGent.be wrote:

...

On 2018-07-31 08:58, Antoine Pitrou wrote:

...

I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience).

I know this probably isn't the correct place to bring it up, but I'm sure that CPython itself could benefit from using Cython. For example, most of the C extensions in Modules/ could be written in Cython.

We don't depend on any third-party Python modules.  Adding a Cython dependency for CPython development would be a tough sell.

I don't really want to get into that discussion (it's more about processes than arguments), but let me note that the CPython development already has a couple of dependencies, such as github and its bots, or tools like argument clinic (admittedly included), make and a C compiler (not included), and a text editor. It's not like it's free of tools that help in writing and maintaining the code. That's pretty much the level at which I also see Cython. It's more complex than argument clinic, but it otherwise serves a similar need.

You're right that since Cython generates C code that we could check in like we do all of our other generated code we could view Cython as another external tool we rely on for maintaining the code base.

Also, a C extension can be built-in (linked statically into the interpreter), which I think would be hard to do with Cython.

Someone recently contributed a feature of hiding the pyinit function for the embedding case, so people do these things already. This could use the normal inittab mechanism, for example. What I think you might be referring to is that Cython modules require the CPython runtime to be initialised to a certain extent, so you couldn't implement "sys" in Cython, for example.

I think the key thing is that on Windows all extension modules are built-in modules, so that use-case would need to be supported (I don't know Cython well enough to know whether this would be doable if we converted as much as possible to Cython itself).   But Jeroen is right, Cython should be a viable option for (most of?) the extension modules in the stdlib. Whether the CPython core devs would accept it in their workflow or not is a totally different question.

I think it's definitely something to consider, although I think Victor's proposal to clean up the C API so we have a clearly delineated private API, a version-specific API, and a stable ABI is a separate idea and so this should be its own thread (plus Victor's is easier to accomplish in the short-term ;) .

On 31 July 2018 at 17:45, Antoine Pitrou solipsis@pitrou.net wrote:

On Tue, 31 Jul 2018 09:27:03 +0200 Jeroen Demeyer J.Demeyer@UGent.be wrote:

...

On 2018-07-31 08:58, Antoine Pitrou wrote:

...

I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience).

I know this probably isn't the correct place to bring it up, but I'm sure that CPython itself could benefit from using Cython. For example, most of the C extensions in Modules/ could be written in Cython.

We don't depend on any third-party Python modules. Adding a Cython dependency for CPython development would be a tough sell.

Also, a C extension can be built-in (linked statically into the interpreter), which I think would be hard to do with Cython.

It'd be *really* nice to at least be able to write some of the C API tests directly in Cython rather than having to fiddle about with splitting the test between the regrtest parts that actually define the test case and the extension module parts that expose the interfaces that we want to test.

So just as we have a split between the core interpreter components needed to freeze importlib and the full Python interpreter, it could be very interesting to have a split between the builtin and extension modules that are required to bootstrap Cython, and those that can instead *rely* on Cython as part of their build process.

While actually doing that would likely mean introducing a venv dependency into the later stages of the build process, we already have such a dependency for the docs build, and that seems to work OK.

Cheers, Nick.

P.S. Note that even though static linking Cython generated modules with CPython doesn't work right just now, experiments like https://mdqinc.com/blog/2011/08/statically-linking-python-with-cython-genera... suggest that it shouldn't take too many adjustments to the build process to get it to work by default.

Antoine Pitrou schrieb am 04.08.2018 um 15:57:

Le 04/08/2018 à 15:13, Nick Coghlan a écrit :

...

It'd be *really* nice to at least be able to write some of the C API tests directly in Cython rather than having to fiddle about with splitting the test between the regrtest parts that actually define the test case and the extension module parts that expose the interfaces that we want to test.

Actually, I think testing the C API is precisely the kind of area where you don't want to involve a third-party, especially not a moving target (Cython is actively maintained and generated code will vary after each new Cython release). Besides, Cython itself calls the C API, which means you might end up involuntarily testing the C API against itself.

If anything, testing the C API using ctypes or cffi would probably be more reasonable... assuming we get ctypes / cffi to compile everywhere, which currently isn't the case.

I agree that you would rather not want to let Cython (or another tool) generate the specific code that tests a specific C-API call, but you could still use Cython to get around writing the setup, validation and unittest boilerplate code in C. Basically, a test could then look something like this (probably works, although I didn't test it):

from cpython.object cimport PyObject from cpython.list cimport PyList_Append

def test_PyList_Append_on_empty_list(): # setup code l = [] assert len(l) == 0 value = "abc" pyobj_value = <PyObject*> value refcount_before = pyobj_value.ob_refcnt

# conservative test call, translates to the expected C code, # although with exception propagation if it returns -1: errcode = PyList_Append(l, value)

# validation assert errcode == 0 assert len(l) == 1 assert l[0] is value assert pyobj_value.ob_refcnt == refcount_before + 1

If you don't want the exception handling, you can define your own declaration of PyList_Append() that does not have it. But personally, I'd rather use try-except in my test code than manually taking care of cleaning up (unexpected) exceptions.

What we do in Cython, BTW, is write doctests in compiled ".pyx" files. That allows us to execute certain parts of a test in Python (the doctest code) and other parts in Cython (the compiled functions/classes that have the doctests), and thus to do a direct comparison between Python and Cython. An example that you could find in a test ".pyx" file:

def test_times2(x): """ doctest that gets executed by Python:

>>> test_times2(3) == 3 * 2 True """ # Cython compiled code in a compiled function that gets tested: return x * 2

Given that CPython's current "_testcapimodule.c" is only a good 5000 lines long (divide that by the number of public C-API functions and macros!), I'm sure the above could help in improving the unit test coverage of the C-API quite quickly.

Stefan

On 5 Aug 2018, at 03:15, Nick Coghlan ncoghlan@gmail.com wrote:

On 5 August 2018 at 00:46, Stefan Behnel stefan_ml@behnel.de wrote:

...

Antoine Pitrou schrieb am 04.08.2018 um 15:57:

...

Actually, I think testing the C API is precisely the kind of area where you don't want to involve a third-party, especially not a moving target (Cython is actively maintained and generated code will vary after each new Cython release). Besides, Cython itself calls the C API, which means you might end up involuntarily testing the C API against itself.

If anything, testing the C API using ctypes or cffi would probably be more reasonable... assuming we get ctypes / cffi to compile everywhere, which currently isn't the case.

I agree that you would rather not want to let Cython (or another tool) generate the specific code that tests a specific C-API call, but you could still use Cython to get around writing the setup, validation and unittest boilerplate code in C. Basically, a test could then look something like this (probably works, although I didn't test it):

from cpython.object cimport PyObject from cpython.list cimport PyList_Append

def test_PyList_Append_on_empty_list(): # setup code l = [] assert len(l) == 0 value = "abc" pyobj_value = <PyObject*> value refcount_before = pyobj_value.ob_refcnt

   # conservative test call, translates to the expected C code,
   # although with exception propagation if it returns -1:
   errcode = PyList_Append(l, value)

   # validation
   assert errcode == 0
   assert len(l) == 1
   assert l[0] is value
   assert pyobj_value.ob_refcnt == refcount_before + 1

If you don't want the exception handling, you can define your own declaration of PyList_Append() that does not have it. But personally, I'd rather use try-except in my test code than manually taking care of cleaning up (unexpected) exceptions.

Exactly, that's the kind of thing I had in mind. At the moment, writing a new dedicated C API test requires designing 4 things:

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test

values in to the code from 2 4. The C->Python interface to get state of interest from 2 back to the test case from 1

If we were able to use Cython to handle 3 & 4 rather than having to hand craft it for every test, then I believe it would significantly lower the barrier to testing the C API directly rather than only testing it indirectly through the CPython implementation.

Having such a test suite available would then hopefully make it easier for other implementations to provide robust emulations of the public C API.

ctypes & cffi likely wouldn't help as much in the case, since they don't eliminate the need to come up with custom code for parts 3 & 4, they just let you write that logic in Python rather than C.

I’m not sure if I understand this, ctypes and cffi are used to access C APIs without writing C code including the CPython API (see for example <https://github.com/abarnert/superhackyinternals/blob/master/internals.py https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).

The code code below should be mostly equivalent to the Cython example posted earlier:

import unittest import ctypes from ctypes import pythonapi

class PyObject(ctypes.Structure): _fields_ = ( ('ob_refcnt', ctypes.c_ssize_t), )

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v): return PyObject.from_address(id(v)).ob_refcnt

def test_PyList_Append_on_empty_list(): # setup code l = [] assert len(l) == 0 value = "abc"

refcount_before = refcount(value)

errcode = pythonapi.PyList_Append(l, value)

assert errcode == 0 assert len(l) == 1 assert l[0] is value assert refcount(value) == refcount_before + 1

I write “mostly” because I rarely use ctypes and am not 100% sure that I use the API correctly.

A problem with using ctypes is that this tests the ABI and to the API, which for example means you cannot test C macros this way.

Ronald

On 5 Aug 2018, at 18:14, Nick Coghlan ncoghlan@gmail.com wrote:

On 5 August 2018 at 18:06, Ronald Oussoren ronaldoussoren@mac.com wrote:

...

I’m not sure if I understand this, ctypes and cffi are used to access C APIs without writing C code including the CPython API (see for example https://github.com/abarnert/superhackyinternals/blob/master/internals.py).

The code code below should be mostly equivalent to the Cython example posted earlier:

import unittest import ctypes from ctypes import pythonapi

class PyObject(ctypes.Structure): _fields_ = ( ('ob_refcnt', ctypes.c_ssize_t), )

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v): return PyObject.from_address(id(v)).ob_refcnt

The quoted code is what I was referring to in:

ctypes & cffi likely wouldn't help as much in the case, since they don't eliminate the need to come up with custom code for parts 3 & 4, they just let you write that logic in Python rather than C. ====

And earlier Nick wrote:

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test

values in to the code from 2 4. The C->Python interface to get state of interest from 2 back to the test case from 1

For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python. This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase.

BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.

Ronald

On 6 Aug 2018, at 17:13, Stefan Behnel stefan_ml@behnel.de wrote:

Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:

...

...

On 5 Aug 2018, at 18:14, Nick Coghlan wrote: On 5 August 2018 at 18:06, Ronald Oussoren wrote:

...

I’m not sure if I understand this, ctypes and cffi are used to access C APIs without writing C code including the CPython API (see for example https://github.com/abarnert/superhackyinternals/blob/master/internals.py).

The code code below should be mostly equivalent to the Cython example posted earlier:

import unittest import ctypes from ctypes import pythonapi

class PyObject(ctypes.Structure): _fields_ = ( ('ob_refcnt', ctypes.c_ssize_t), )

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v): return PyObject.from_address(id(v)).ob_refcnt

The quoted code is what I was referring to in:

ctypes & cffi likely wouldn't help as much in the case, since they don't eliminate the need to come up with custom code for parts 3 & 4, they just let you write that logic in Python rather than C. ====

And earlier Nick wrote:

...

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test

values in to the code from 2 4. The C->Python interface to get state of interest from 2 back to the test case from 1

For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python.

Which then means that you have a mix of Python and C in many cases.

Not really, for many cases the only C code required is the actual CPython API.

I guess that's what you meant with your next sentence:

...

This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase.

Not really. What I was trying to say is that when you use C code to make the API call and collect information you have complete control of which APIs are used (in the PyList_Append test case you can be 100% sure that the test code only calls that API), with ctypes/cffi/Cython you have less control over the exact API calls made which could affect the test. Making assertions about refcounts is one example, my test case with ctypes works but you have to think about what the code does before you understand why the test case works and tests what you want to test. In general that’s not a good quality for test code, that should be as obviously correct as possible.

Cython is probably the best option in this regard (from what I’ve seen so far) because you can in the end basically write C code with Python syntax where needed.

Not sure I understand this correctly, but I think we're on the same page here: writing test code in C is cumbersome, writing test code in a mix of C and Python across different files is aweful. And making it difficult to write or even just review test code simply means that people will either waste their precious contribution time on it, or try to get around it.

Awful is a strong word. The separation of the test case in two different languages in two different source files is definitely less than ideal.

...

BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.

I totally agree with that. For less trivial tests, people will often want to stear the test case at the C level, because some things are really difficult to do from Python. Good luck making assertions about reference counts when you're orchestrating the C-API through ctypes.

I agree, see above.

And this is where Cython shines – your code *always* ends up running in C, regardless of how much of it is plain Python. But at any point, you can do pretty arbitrary C things, all in the same function. And unittest can execute that function directly for you, without having to write a Python wrapper or separate test runner.

And for the really hard cases, you can resort to writing a literal C code snippet in your Cython source file (as a string) and let Cython drop it into the file it generates, e.g. to quickly define a macro, a little C function, or an interface wrapper around a C macro that would otherwise be difficult to test. That little feature removes the last reason for resorting to a separate C file.

Another reason for liking the translation to C is that you can use and test the macros in the CPython API, which is something you definitely cannot do with ctypes.

I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency. I do think that adding a “Cython-lite” tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython.

Ronald

On 2018-08-07, 17:34 GMT, Yury Selivanov wrote:

Speaking of which, Dropbox is working on a new compiler they call "mypyc".

How does it compare with Nuitka?

Matěj

Yury,

given that people are starting to quote enthusiastically the comments you made below, let me set a couple of things straight.

Yury Selivanov schrieb am 07.08.2018 um 19:34:

On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:

...

I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency. I do think that adding a “Cython-lite” tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython.

Speaking of which, Dropbox is working on a new compiler they call "mypyc".

mypyc will compile type-annotated Python code to an optimized C.

That's their plan. Saying that "it will" is a bit premature at this point. The list of failed attempts at writing static Python compilers is rather long, even if you only count those that compile the usual "easy subset" of Python.

I wish them the best of luck and endurance, but they have a long way to go.

The first goal is to compile mypy with it to make it faster, so I hope that the project will be completed.

That's not "the first goal". It's the /only/ goal. The only intention of mypyc is to be able to compile and optimise enough of Python to speed up the kind or style of code that mypy uses.

Essentially, mypyc will be similar to Cython, but mypyc is a *subset of Python*, not a superset.

Which is bad, right? It means that there will be many things that simply don't work, and that you need to change your code in order to make it compile at all. Cython is way beyond that point by now. Even RPython will probably continue to be way better than mypyc for quite a while, maybe forever, who knows.

Interfacing with C libraries can be easily achieved with cffi.

Except that it will be fairly slow. cffi is not designed for static analysis but for runtime operations. You can obviously also use cffi from Cython – but then, why would you, if you can get much faster code much more easily without using cffi?

That being said, if someone wants to write a static cffi optimiser for Cython, why not, I'd be happy to help with my advice. The cool thing is that this can be improved gradually, because compiling the cffi code probably already works out of the box. It's just not (much) faster than when interpreted.

Being a strict subset of Python means that mypyc code will execute just fine in PyPy.

So does normal (non-subset) Python code. You can run it in PyPy, have CPython interpret it, or compile it with Cython if you want it to run faster in CPython, all without having to limit yourself to a subset of Python. Seriously, you make this sound like requiring users to rewrite their code to make it compilable with mypyc was a good thing.

They can even apply some optimizations to it eventually, as it has a strict and static type system.

In case "they" refers to PyPy here, then I remember the PyPy project stating very clearly that they are not interested in PEP-484 typing because it is completely irrelevant for their JIT. It's really best for them to ignore it.

That's similar for Cython, simply because PEP-484 typing isn't designed for optimisation at all, definitely not for C-level optimisation. Still, Cython can make some use of PEP-484 typing, if you use it to define specific C types. That allows normal execution in CPython, static analysis with PEP-484 analyser tools (e.g. PyCharm or mypy), and efficient optimisation by Cython. The best of all worlds. See the docs on how to do that, it's been supported for about a year now (and has been around in a similar, non-PEP-484 form for years before that PEP even existed).

I'd be more willing to start using mypyc+cffi in CPython stdlib *eventually*, than Cython now. Cython is a relatively complex and still poorly documented language.

You are free to improve the documentation or otherwise help us find and discuss concrete problems with it. Calling Cython a "poorly documented language" could easily feel offensive towards those who have put a lot of work into the documentation, wiki, tutorials, trainings and what not that help people use the language. Even stack overflow is getting better and better in documenting Cython these days, even though responses over there that describe work-arounds tend to get outdated fairly quickly.

Besides, don't forget that it's Python, so consider reading the Python documentation first if something is unclear. And maybe some documentation of C data types as well. (.5 wink)

I'm speaking from experience after writing thousands of lines of Cython in uvloop & asyncpg. In skillful hands Cython is amazing, but I'd be cautious to advertise and use it in CPython.

Why not? You didn't actually give any reasons for that.

I'm also -1 on using Cython to test C API. While writing C tests is annoying (I wrote a fair share myself), their very purpose is to make third-party tools/extensions more stable. Using a third-party tool to test C API to track regressions that break third-party tools feels wrong.

I don't understand that argument. What's wrong about using a tool that helps you get around writing boiler plate code? The actual testing does not need to be done by Cython at all, you can write it any way you like.

Stefan

Yury Selivanov schrieb am 04.09.2018 um 18:19:

On Sat, Sep 1, 2018 at 6:12 PM Stefan Behnel wrote:

...

Yury Selivanov schrieb am 07.08.2018 um 19:34:

...

The first goal is to compile mypy with it to make it faster, so I hope that the project will be completed.

That's not "the first goal". It's the /only/ goal. The only intention of mypyc is to be able to compile and optimise enough of Python to speed up the kind or style of code that mypy uses.

...

Essentially, mypyc will be similar to Cython, but mypyc is a *subset of Python*, not a superset.

Which is bad, right? It means that there will be many things that simply don't work, and that you need to change your code in order to make it compile at all. Cython is way beyond that point by now. Even RPython will probably continue to be way better than mypyc for quite a while, maybe forever, who knows.

To be clear I'm not involved with mypyc, but my understanding is that the entire Python syntax will be supported, except some dynamic features like patching `globals()`, `locals()`, or classes, or __class__.

No, that's not the goal, at least from what I understood from my discussions with Jukka. The goal is to make it compile mypy, be it by supporting Python features in mypyc or by avoiding Python features in mypy. I'm sure they will take any shortcut they can in order to avoid having to make mypyc too capable, because mypyc is not more than a means to an end. For example, they may easily get away without supporting generators and closures, which are quite difficult to implement in C. But finding a non-trivial piece of Python code out there that uses neither of the two is probably not easy.

I'm also sure they will avoid Python semantics wherever they can, because implementing them in the same way as CPython and Cython would mean that certain constructs cannot safely be statically reasoned about, and thus cannot be optimised. Avoiding (full) Python semantics relieves you from these restrictions, and if you control both sides, the compiler and the code that it compiles, then it becomes much easier to apply arbitrary optimisations at will.

IMHO, what they are implementing is much closer to ShedSkin than to Cython.

...

...

Interfacing with C libraries can be easily achieved with cffi.

Except that it will be fairly slow. cffi is not designed for static analysis but for runtime operations.

Could you please clarify this point? My current understanding is that you can build a static compiler with a knowledge about cffi so that it can compile calls like `ffi.new("something_t[]", 80)` to pure C.

I'm sure there is a relatively large subset of cffi's API that could be compiled statically, as long as the declartions and their usage are kept simple and fully visible to the compiler. What that subset is remains to be seen once someone actually tries to do it.

Yeah, statically compiling cffi-enabled code is probably the way to go for mypyc and Cython.

I doubt it, given the expected restrictions and verbosity. But debating this is useless as long as no-one attempts to actually write a static compiler for cffi(-like) code.

Using Cython/C types usually means that you need to use pxd/pyx files which means that the code isn't Python anymore.

I'm aware that this is a very common misconception that is difficult to get out of people's heads. You probably got this idea from wrapping a native library, in which case the only choice you have in order to declare an external C-API is really to use Cython's special syntax. However, this would not apply to most use cases in the CPython project context, and it also does not necessarily apply to most of the code in a Cython module even if it uses external libraries.

Cython has four ways to provide type declarations: cdef statements in Cython code, external .pxd files for Python or Cython files, special decorators and declaration functions, and PEP-484/526 type annotations.

All four have their use cases (e.g. syntax support in different Python versions, efficiency of expression, readability for people with different backgrounds, etc.), and all but the first allow users to keep their module code in Python syntax. As long as you do not call into external native code, it's your choice which of these you prefer for your code base, project context and developer background. You can even mix them at will, if you feel like it.

I know that Cython has a mode to use decorators in pure Python code to annotate types, but they are less intuitive than using typing annotations in 3.6+.

You can use PEP-484/526 type annotations to declare Cython types in Python code that you intend to compile. It's entirely up to you, and it's an entirely subjective measure which "is better". Many people prefer Cython's non-Python syntax because it allows them to apply their existing C knowledge. For them, PEP-484 annotations may easily be non-intuitive in comparison.

For CPython it means that we'd have Python, C, and Cython to learn to understand code written in Cython. There's a very popular assumption that you have to be proficient in C in order to become a CPython core dev and people are genuinely surprised when I tell them that it's not a requirement. At the three conferences I've been this summer at least 5 people complained to me that they didn't even consider contributing to CPython because they don't know C. Adding yet another language would simply raise this bar even higher, IMHO.

Adding the right language would lower the bar, IMHO. Cython is Python. It allows users with a Python background to implement C things without having to thoroughly learn C /and/ the CPython C-API first. So, the way I see it, rather than /adding/ a "third" language to the mix, it substantially lowers the entry level from the current two and a half languages (Python + C + C-API) to one and a half (Python + Cython).

I'd be +0.5 on using Cython (optionally?) to compile some pure Python code to make it 30-50% faster. asyncio, for instance, would certainly benefit from that.

Since most of this (stdlib) Python code doesn't need to stay syntax compatible with Python < 3.6 (actually 3.8) anymore, you can probably get much higher speedups than that by statically typing some variables and functions here and there. I recently tried that with difflib, makes a big difference.

Stefan

On Tue, Sep 4, 2018 at 2:58 PM Stefan Behnel stefan_ml@behnel.de wrote: [..]

Cython has four ways to provide type declarations: cdef statements in Cython code, external .pxd files for Python or Cython files, special decorators and declaration functions, and PEP-484/526 type annotations.

Great to hear that PEP 484 type annotations are supported. Here's a link to the docs: https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#static-typing

[..]

...

I know that Cython has a mode to use decorators in pure Python code to annotate types, but they are less intuitive than using typing annotations in 3.6+.

You can use PEP-484/526 type annotations to declare Cython types in Python code that you intend to compile. It's entirely up to you, and it's an entirely subjective measure which "is better". Many people prefer Cython's non-Python syntax because it allows them to apply their existing C knowledge. For them, PEP-484 annotations may easily be non-intuitive in comparison.

Yeah, but if we decide to use Cython in CPython we probably need to come up with something like PEP 7 to recommend one particular style and have an overall guideline. Using PEP 484 annotations means that we have pure Python code that PyPy and other interpreters can still run.

[..]

...

I'd be +0.5 on using Cython (optionally?) to compile some pure Python code to make it 30-50% faster. asyncio, for instance, would certainly benefit from that.

Since most of this (stdlib) Python code doesn't need to stay syntax compatible with Python < 3.6 (actually 3.8) anymore, you can probably get much higher speedups than that by statically typing some variables and functions here and there. I recently tried that with difflib, makes a big difference.

I'd be willing to try this in asyncio if we start using Cython.

Yury

On Wed, 08 Aug 2018 12:44:50 +0200 Matěj Cepl mcepl@cepl.eu wrote:

On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:

...

Not sure I understand this correctly, but I think we're on the same page here: writing test code in C is cumbersome, writing test code in a mix of C and Python across different files is aweful. And making it difficult to write or even just review test code simply means that people will either waste their precious contribution time on it, or try to get around it.

I was thinking about the same when porting M2Crypto to py3k (M2Crypto is currently swig-based mix of C-code and Python). Is it even possible to make a mix of Cython, swig-based C, and Python? In the end I rather stayed with plain C, because the combination seems unimaginably awful.

I'm not sure why anyone would want to use swig nowadays.

(Also, is Cython the best of all of them? What about cffi or Nuitka?)

This sounds like comparing apples to oranges.

Regards

Antoine.

Hi,

this is getting a bit out of scope for this list. I propose to move further questions about general Cython usage to he cython-users mailing list.

Matěj Cepl schrieb am 08.08.2018 um 12:44:

On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:

...

Not sure I understand this correctly, but I think we're on the same page here: writing test code in C is cumbersome, writing test code in a mix of C and Python across different files is aweful. And making it difficult to write or even just review test code simply means that people will either waste their precious contribution time on it, or try to get around it.

I was thinking about the same when porting M2Crypto to py3k (M2Crypto is currently swig-based mix of C-code and Python). Is it even possible to make a mix of Cython, swig-based C, and Python?

As long as you take the decision at a per-module basis, sure. If you want to mix them inside of a single module, then it's either Cython+C or Swig+C, not all three. But as Antoine suggested, unless you really want an identical mapper for whole range of different languages, Swig is likely not what you should use these days.

In the end I rather stayed with plain C, because the combination seems unimaginably awful.

Probably worth expanding your imagination. :)

Stefan

On 2018-07-31, Victor Stinner wrote:

I would be nice to be able to use something to "generate" C extensions, maybe even from pure Python code. But someone has to work on a full solution to implement that.

Perhaps a "argument clinic on steroids" would be the proper approach. So, extensions would mostly be written in C. However, we would have a pre-processor that does some "magic" to make using the Python API cleaner. Defining new types using static structures, for example, is not the way to build a good API. The other approach would be something like mypyc + cffi. I agree with others that Cython is not the right tool.

Regards,

Neil

On Thu, 23 Aug 2018 13:34:02 +1200 Greg Ewing greg.ewing@canterbury.ac.nz wrote:

Neil Schemenauer wrote:

...

Perhaps a "argument clinic on steroids" would be the proper approach. So, extensions would mostly be written in C. However, we would have a pre-processor that does some "magic" to make using the Python API cleaner.

You seem to have started on the train of thought that led me to create Pyrex (the precursor to Cython).

It went like this: I was getting frustrated with SWIG because it wasn't powerful enough to do what I wanted. I thought about something that would let me write functions with Python headers and C bodies.

Then I realised that to really make the C API bearable it would need to take care of refcounting and exception handling, so the body couldn't just be plain C, it would need some Python semantics as well. Given that, it seemed more sensible to base all of the syntax on Python.

Also, I wanted to avoid the problem you get with all preprocessors, that when something goes wrong you get error messages in terms of the expanded code rather than the original source. To fix that I would need to do a full parsing and type analysis job -- essentially it would need to be a full-blown compiler in its own right.

So, "argument clinic on steroids" is actually a pretty good description of Pyrex.

Agreed, at least abstractly, Cython is the best existing solutiong to the problem.

The issues to solve for Cython to be usable for the CPython stdlib are IMHO the following:

- the bootstrap problem (Cython self-compiles with CPython) - the dependency / versioning problem (Cython is a large quick-evolving third-party package that we can't decently vendor) - the maintenance problem (how do ensure we can change small things in the C API, especially semi-private ones, without having to submit PRs to Cython as well) - the debugging problem (Cython's generated C code is unreadable, unlike Argument Clinic's, which can make debugging annoying)

Regards

Antoine.