[capi-sig]Adding an official (minimal) Cython-like tool.
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
Stupid question. Why not using Cython instead of a Cython-like which would have limitations and might be incompatible with Cython?
Do you mean add something to the CPython standard library?
Victor
2018-07-31 17:59 GMT+02:00 Eric Snow <ericsnowcurrently@gmail.com>:
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
One issue I have with the Cython model is that it is very difficult to reason about the computation that is being performed. It is almost impossible to tell when an object will be boxed/unboxed and control for that. Also, I am not sure which operations will be invoking a lot of Python code or which will be using specialized functions. The behavior also changes from version to version in Cython. When I add an extension I have already decided that the code is performance sensitive* so I want to be able to understand the runtime behavior. The auto boxing/unboxing is especially bad because often Cython/C extensions are used when an algorithm is naturally expressed as a loop, and many small object allocations can dramatically affect the performance. I think that for some people, not worrying that much about C type or Python type is a core feature of Cython; however, this makes it harder to use for performance sensitive problems.
- if I am just wrapping a C library without a big performance concern I can use cffi
On Tue, Jul 31, 2018 at 12:12 PM, Victor Stinner <vstinner@redhat.com> wrote:
Stupid question. Why not using Cython instead of a Cython-like which would have limitations and might be incompatible with Cython?
Do you mean add something to the CPython standard library?
Victor
2018-07-31 17:59 GMT+02:00 Eric Snow <ericsnowcurrently@gmail.com>:
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
Joe Jevnik via capi-sig schrieb am 31.07.2018 um 18:26:
One issue I have with the Cython model is that it is very difficult to reason about the computation that is being performed. It is almost impossible to tell when an object will be boxed/unboxed and control for that.
I agree that that's not obvious, in the same way that a C compiler may generate non-obvious code that changes with new versions. Yes, there is a tool chain between you and the CPU instructions.
People commonly use "cython -a" to reason about what Cython generates.
Also, I am not sure which operations will be invoking a lot of Python code or which will be using specialized functions. The behavior also changes from version to version in Cython.
That's called "enhancement" and "optimisation". There is no guarantee that a compilers will never change its capabilities. ;)
When I add an extension I have already decided that the code is performance sensitive* so I want to be able to understand the runtime behavior.
Most people I've seen use Cython actually care more about benchmarks and profiling results. You'd be surprised how often people ask things on mailing lists, stackoverflow, etc., that is obvious from looking at the generated C code.
One of the reasons for that is probably that Cython allows people to write correct and fast C extensions who do not have a C background at all. (Very similar to the way Python allows people to solve their problems with programming who do not have a programming background.)
The auto boxing/unboxing is especially bad because often Cython/C extensions are used when an algorithm is naturally expressed as a loop, and many small object allocations can dramatically affect the performance. I think that for some people, not worrying that much about C type or Python type is a core feature of Cython; however, this makes it harder to use for performance sensitive problems.
These cases will show up in bright yellow in Cython's code annotation view.
Stefan
On Tue, 31 Jul 2018 at 09:13 Victor Stinner <vstinner@redhat.com> wrote:
Stupid question. Why not using Cython instead of a Cython-like which would have limitations and might be incompatible with Cython?
Do you mean add something to the CPython standard library?
Yes, what Eric is suggesting is a baseline tool like Cython be added to Python itself so it becomes the minimum, common tool that we point all extension authors to. Then Cython could build on top of that -- or the C API that the tool used -- so either tool can be used for the best situation. But the key point is that if we want as slick as Julia's FFI ( https://docs.julialang.org/en/stable/manual/calling-c-and-fortran-code/index...), then we will have to provide an FFI compiler for people to use in at least the simple cases.
-Brett
Victor
2018-07-31 17:59 GMT+02:00 Eric Snow <ericsnowcurrently@gmail.com>:
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
Eric Snow schrieb am 31.07.2018 um 17:59:
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
Cython is certainly not a trivial tool, but it is what it is because the problem it solves is what it is.
I don't quite see why an FFI solution must be part of CPython. We have at least three great tools that people widely use these days: Cython, CFFI and pybind11 (and then some, but less common/hyped ones). These three tools serve three different sides of the need to extend (C)Python: Cython is Python with native C/C++ data types, pybind11 is modern C++ with Python integration, and CFFI is Python with a dynamic runtime interface to native code.
Depending on what background people come from and what needs they have for a given problem, either of the three is the perfect solution to their problem. But they are solutions because they are what they are (including their dependencies), and because they are maintained and get improved over time. I wouldn't want an incomplete tool in the stdlib that people would just best replace with one of these three tools. After all, this was tried with ctypes before, and the net result is that people recommend using CFFI instead when the specific need arises.
Stefan
On Tue, Jul 31, 2018 at 2:02 PM Stefan Behnel <python_capi@behnel.de> wrote:
Cython is certainly not a trivial tool, but it is what it is because the problem it solves is what it is.
I don't quite see why an FFI solution must be part of CPython. We have at least three great tools that people widely use these days: Cython, CFFI and pybind11 (and then some, but less common/hyped ones). These three tools serve three different sides of the need to extend (C)Python: Cython is Python with native C/C++ data types, pybind11 is modern C++ with Python integration, and CFFI is Python with a dynamic runtime interface to native code.
Depending on what background people come from and what needs they have for a given problem, either of the three is the perfect solution to their problem. But they are solutions because they are what they are (including their dependencies), and because they are maintained and get improved over time. I wouldn't want an incomplete tool in the stdlib that people would just best replace with one of these three tools. After all, this was tried with ctypes before, and the net result is that people recommend using CFFI instead when the specific need arises.
Thanks for that explanation, Stefan. That's a good summary of where Cython fits in. Ultimately, the question is how we are the Python core team can take advantage of these tools to move extension authors away from the C-API.
-eric
Hi,
I never used Cython, so I'm not the good candidate to explain when Cython is appropriate: for which kind of use cases. Is there any volunteer to fill the https://pythoncapi.readthedocs.io/cython.html page?
Maybe Stefan Behnel? :-)
Victor Le mar. 31 juil. 2018 à 18:01, Eric Snow <ericsnowcurrently@gmail.com> a écrit :
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
I think this discussion is a bit misleading. Cython allows you to wrap existing C code, but if you want to use Python objects in C or details which are not exposed at the Python level, you still need the Python C API to work with those.
Please don't forget that the Python C API was key in making it so popular for integration. It's generally a very nice API to work with as C programmer.
There are also plenty other tools for wrapping C or C++ or Java code out there. Some work as compilers or code generators, others using an interfacing layer and dynamic introspection. Each fills a niche and is good for specific use cases.
Whether to use Cython or not depends on your use case. It's not always the obvious choice (otherwise, those other tools would not exist).
I'm not saying that the Python C API doesn't have warts or issues, but limiting its usefulness or ditching it completely is not the answer. Of course, it has to evolve just like Python itself does.
On 07.09.2018 10:25, Victor Stinner wrote:
Hi,
I never used Cython, so I'm not the good candidate to explain when Cython is appropriate: for which kind of use cases. Is there any volunteer to fill the https://pythoncapi.readthedocs.io/cython.html page?
Maybe Stefan Behnel? :-)
Victor Le mar. 31 juil. 2018 à 18:01, Eric Snow <ericsnowcurrently@gmail.com> a écrit :
Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted).
-eric
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Le ven. 7 sept. 2018 à 11:02, M.-A. Lemburg <mal@egenix.com> a écrit :
I think this discussion is a bit misleading.
This is why I'm asking for help to clarify it :-)
Cython allows you to wrap existing C code, but if you want to use Python objects in C or details which are not exposed at the Python level, you still need the Python C API to work with those.
I'm trying to remove functions from the C API which allow to do things which are not possible at the Python level: https://pythoncapi.readthedocs.io/bad_api.html#no-public-c-functions-if-it-c...
It's a request coming from PyPy developers.
Please don't forget that the Python C API was key in making it so popular for integration. It's generally a very nice API to work with as C programmer.
*I* am well aware of that and I tried to repeat it in my document :-) Since some people like to repeat "Let's just remove the C A API!", I added a new page to explain why it's not possible today: https://pythoncapi.readthedocs.io/remove_c_api.html
Victor
On 07.09.2018 11:46, Victor Stinner wrote:
Le ven. 7 sept. 2018 à 11:02, M.-A. Lemburg <mal@egenix.com> a écrit :
I think this discussion is a bit misleading.
This is why I'm asking for help to clarify it :-)
Cython allows you to wrap existing C code, but if you want to use Python objects in C or details which are not exposed at the Python level, you still need the Python C API to work with those.
I'm trying to remove functions from the C API which allow to do things which are not possible at the Python level: https://pythoncapi.readthedocs.io/bad_api.html#no-public-c-functions-if-it-c...
It's a request coming from PyPy developers.
There are always things which you have to be able to do at the C level but cannot do at the Python level and that's intentional, since you're working at the C level and want to have direct access to the data, avoiding copying things around all the time and creating intermediate temporary objects or buffers for this.
You also want callbacks to work from both worlds (Python into C and C into Python), or embed the interpreter, or provide new ways of working with the existing objects, or work with code in a way which bypasses the exception machinery (in C you very often raise exceptions and catch them without the exception object itself every being created).
Overall, the C API provides a more low level access to the Python world. This has to be acknowledged. It's not just another way to run Python.
I know I'm stating the obvious, but the Python C API is meant for direct interfacing from Python to C and vice-versa. Since C is a different language with different data types, of course, there are things which you can do with this API and which you cannot do in Python, so the statement "There should not be C APIs that do something that you can’t do in Python." is too broad.
Please don't forget that the Python C API was key in making it so popular for integration. It's generally a very nice API to work with as C programmer.
*I* am well aware of that and I tried to repeat it in my document :-) Since some people like to repeat "Let's just remove the C A API!", I added a new page to explain why it's not possible today: https://pythoncapi.readthedocs.io/remove_c_api.html
The Python C API has grown some warts, definitely, and it could use an overhaul in parts where APIs were added too quickly, without staying within the original design patterns, etc.
That's certainly something which can happen without adding a Cython layer, forcing extension writers to rewrite their extensions.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
M.-A. Lemburg schrieb am 07.09.2018 um 11:02:
Cython allows you to wrap existing C code, but if you want to use Python objects in C or details which are not exposed at the Python level, you still need the Python C API to work with those.
Well, you *can* use the C-API, because Cython allows you to call arbitrary C code. But that doesn't mean you should. In fact, the code that Cython generates is usually faster than the "obvious" C-API functions, so *not* using the C-API explicitly in Cython has many advantages, such as not relying on CPython version details, simplifying your code, and speeding it up.
And Cython is not just a tool to "wrap existing C code". That's only one of its major use cases. Speeding up Python code, and writing C code without writing C code, are just as important.
Stefan
Victor Stinner schrieb am 07.09.2018 um 10:25:
I never used Cython, so I'm not the good candidate to explain when Cython is appropriate: for which kind of use cases. Is there any volunteer to fill the https://pythoncapi.readthedocs.io/cython.html page?
I find it easier to answer concrete questions than to randomly repeat parts of our documentation and hope for someone to be interested. ;-)
I think the homepage actually explains the general use cases quite well:
Additionally, I wrote a blog post that lists a couple of use cases of Cython that I see in CPython development.
http://blog.behnel.de/posts/what-cpython-could-use-cython-for.html
Stefan
participants (6)
-
Brett Cannon -
Eric Snow -
Joe Jevnik -
M.-A. Lemburg -
Stefan Behnel -
Victor Stinner