Hi Maciej, Armin, et al! Together with co-authors I've CC'd, I'm getting close to finishing the 4th edition of "Python in a Nutshell", and in particular I'm in charge of the "extending and embedding" chapter ("chap 24" based on the 3rd Ed's number), which in the first three editions focused almost exclusively on Python's C API. Working on the 4th edition I dug deeper into CFFI and I'm getting close to being convinced that it's THE way to extend (dunno about embed?) Python -- CPython as well as of course PyPy -- putting ctypes for sure, the API likely, and maybe even Cython in the shade. HOWEVER -- I now face the task of subtly convincing people who've been mostly using (especially!) ctypes, to switch to CFFI (esp. now that they can easily download free C compilers for all platforms, inc. Windows). Alas, I don't have enough real-life CFFI experience myself (and it's hard for me to get any since I'm still working 60%-time for Google, where CLIF is THE approved way to extend Python). I wonder if any of you would like to take a look and offer suggestions about the chapter, *especially* after the way-too-miniscule coverage of CFFI. I need (for evangelism purposes!) credible examples, esp. one setting CFFI head-to-head against ctypes (but comparisons with cython and the API would be fine too -- IF I could figure out how to define completely new Python types in CFFI, which so far escapes me). If asking for your collaboration is too much, is there at least a way to get your OK about reusing, with small adaptations, some of the excellent examples you give on CFFI's docs? That would probably work better than me trying to work things out from scratch, but, without your explicit permission, it would likely break copyright or something. BTW, since we're trying to slim down the page count of the Nutshell, I suspect that ch.24 will live mostly online, accessible 100% freely, with the paper book devoting just very few pages to summarizing the options for extending and embedding and linking to the online materials. I know my co-authors (CC'd) are fine with that but I still need an official OK from O'Reilly (I think I'll get it). Just in case this makes a difference to your attitude about cooperating with us in crafting this chapter!-) Thanks, Alex
Hi Alex! On Thu, 21 Jul 2022 at 02:08, Alex Martelli via pypy-dev <pypy-dev@python.org> wrote:
credible examples, esp. one setting CFFI head-to-head against ctypes (but comparisons with cython and the API would be fine too -- IF I could figure out how to define completely new Python types in CFFI, which so far escapes me).
CFFI is different from ctypes, and also different from the CPython C API, and also different from Cython. You can't always write a straightforward line-by-line comparison for any two of these four things. The main purpose of CFFI is to expose an existing C API to Python, a bit like ctypes but focusing on the level of C instead of on the details of the system's ABI. You can also write some extra C code and expose that to Python as well. That's it: you can't do things like define new Python types in C, or even write a C function that accepts an arbitrary Python object. Instead, you typically write a pythonic layer and this layer internally uses CFFI but doesn't expose it further to the rest of the program. This layer may do things like define its own Python classes to give a more object-oriented feeling to the existing C API; or it can wrap and unwrap arbitrary Python objects inside "handles" which can be passed to C and back as black boxes. You can also write regular Python functions and expose them for the C code to call. (This is generally useful, but is also the basis for how embedding works with CFFI.) CFFI is not really suited for *all* use cases. For example, writing a custom data structure that can be used to store a large number of arbitrary Python objects: while this is in theory possible, in practice you get large overheads. Or if all you want is a very compact Python class that can store two fixed-size integers.
If asking for your collaboration is too much, is there at least a way to get your OK about reusing, with small adaptations, some of the excellent examples you give on CFFI's docs?
Sure. A bientôt, Armin
How about HPY : https://github.com/hpyproject/hpy I think it is the best route currently? On Thu, Jul 21, 2022 at 6:39 AM Alex Martelli via pypy-dev < pypy-dev@python.org> wrote:
On Thu, Jul 21, 2022 at 11:01 AM Phyo Arkar Lwin <phyoakl@hexcode.tech> wrote:
How about HPY : https://github.com/hpyproject/hpy I think it is the best route currently?
HPy is hopefully the future replacement for the C API itself, but it's not something one can use in production *right now* so it's probably not what Alex would want to recommend to readers in this edition. Perhaps it can be recommended in a future edition. HPy is also a bit orthogonal to Cython and CFFI, in the sense that one day users of these two tools may simply be able to optionally compile to an HPy extension if they wish and otherwise not be particularly affected. Regards, Simon
I definitely must *mention* HPy, but yes, too early to *recommend* it for production purposes. As for Cython/CFFI/ctypes, I keep having the intuition that CFFI can (and likely should) replace all uses of ctypes (I've never seen anything else in the stdlib cause as many Python hard crashes as ctypes!), as well as many uses of the API (until HPy's mature) [probably not uses of Cython, which, if understand right, will just change to use HPy fruitfully "under the covers"] -- but I guess I'll have to work on proving that myself:-). *Thanks* for the permission to quote and/or tweak examples of CFFI use you published -- should make it easier for me (they're *neat* examples -- I still wish I had one of how to use CFFI to define a new *type*, but I guess I'll have to work on that!). Thanks, Alex On Thu, Jul 21, 2022 at 3:24 PM Simon Cross <hodgestar@gmail.com> wrote:
I definitely must mention HPy, but yes, too early to recommend it for production purposes.
:D
As for Cython/CFFI/ctypes, I keep having the intuition that CFFI can (and likely should) replace all uses of ctypes (I've never seen anything else in the stdlib cause as many Python hard crashes as ctypes!)
Some (perhaps many and potentially all) uses of ctypes potentially segfault because they rely on Python code determining the correct type used by an installed C library, when doing so correctly means passing the correct header to the compiler and using its output. As far as I know, such issues were one of the reasons for CFFI being developed in the first place.
as well as many uses of the API (until HPy's mature) [probably not uses of Cython, which, if understand right, will just change to use HPy fruitfully "under the covers"] -- but I guess I'll have to work on proving that myself:-).
Yes, the goal is to add a Cython backend that generates HPy C code, hopefully without the Cython user having to change any code. Regards, Simon
Hi Alex! On Thu, 21 Jul 2022 at 02:08, Alex Martelli via pypy-dev <pypy-dev@python.org> wrote:
credible examples, esp. one setting CFFI head-to-head against ctypes (but comparisons with cython and the API would be fine too -- IF I could figure out how to define completely new Python types in CFFI, which so far escapes me).
CFFI is different from ctypes, and also different from the CPython C API, and also different from Cython. You can't always write a straightforward line-by-line comparison for any two of these four things. The main purpose of CFFI is to expose an existing C API to Python, a bit like ctypes but focusing on the level of C instead of on the details of the system's ABI. You can also write some extra C code and expose that to Python as well. That's it: you can't do things like define new Python types in C, or even write a C function that accepts an arbitrary Python object. Instead, you typically write a pythonic layer and this layer internally uses CFFI but doesn't expose it further to the rest of the program. This layer may do things like define its own Python classes to give a more object-oriented feeling to the existing C API; or it can wrap and unwrap arbitrary Python objects inside "handles" which can be passed to C and back as black boxes. You can also write regular Python functions and expose them for the C code to call. (This is generally useful, but is also the basis for how embedding works with CFFI.) CFFI is not really suited for *all* use cases. For example, writing a custom data structure that can be used to store a large number of arbitrary Python objects: while this is in theory possible, in practice you get large overheads. Or if all you want is a very compact Python class that can store two fixed-size integers.
If asking for your collaboration is too much, is there at least a way to get your OK about reusing, with small adaptations, some of the excellent examples you give on CFFI's docs?
Sure. A bientôt, Armin
How about HPY : https://github.com/hpyproject/hpy I think it is the best route currently? On Thu, Jul 21, 2022 at 6:39 AM Alex Martelli via pypy-dev < pypy-dev@python.org> wrote:
On Thu, Jul 21, 2022 at 11:01 AM Phyo Arkar Lwin <phyoakl@hexcode.tech> wrote:
How about HPY : https://github.com/hpyproject/hpy I think it is the best route currently?
HPy is hopefully the future replacement for the C API itself, but it's not something one can use in production *right now* so it's probably not what Alex would want to recommend to readers in this edition. Perhaps it can be recommended in a future edition. HPy is also a bit orthogonal to Cython and CFFI, in the sense that one day users of these two tools may simply be able to optionally compile to an HPy extension if they wish and otherwise not be particularly affected. Regards, Simon
I definitely must *mention* HPy, but yes, too early to *recommend* it for production purposes. As for Cython/CFFI/ctypes, I keep having the intuition that CFFI can (and likely should) replace all uses of ctypes (I've never seen anything else in the stdlib cause as many Python hard crashes as ctypes!), as well as many uses of the API (until HPy's mature) [probably not uses of Cython, which, if understand right, will just change to use HPy fruitfully "under the covers"] -- but I guess I'll have to work on proving that myself:-). *Thanks* for the permission to quote and/or tweak examples of CFFI use you published -- should make it easier for me (they're *neat* examples -- I still wish I had one of how to use CFFI to define a new *type*, but I guess I'll have to work on that!). Thanks, Alex On Thu, Jul 21, 2022 at 3:24 PM Simon Cross <hodgestar@gmail.com> wrote:
I definitely must mention HPy, but yes, too early to recommend it for production purposes.
:D
As for Cython/CFFI/ctypes, I keep having the intuition that CFFI can (and likely should) replace all uses of ctypes (I've never seen anything else in the stdlib cause as many Python hard crashes as ctypes!)
Some (perhaps many and potentially all) uses of ctypes potentially segfault because they rely on Python code determining the correct type used by an installed C library, when doing so correctly means passing the correct header to the compiler and using its output. As far as I know, such issues were one of the reasons for CFFI being developed in the first place.
as well as many uses of the API (until HPy's mature) [probably not uses of Cython, which, if understand right, will just change to use HPy fruitfully "under the covers"] -- but I guess I'll have to work on proving that myself:-).
Yes, the goal is to add a Cython backend that generates HPy C code, hopefully without the Cython user having to change any code. Regards, Simon
participants (4)
-
Alex Martelli -
Armin Rigo -
Phyo Arkar Lwin -
Simon Cross