Mailman 3 discouraging direct use of the C-API - Python-ideas

discouraging direct use of the C-API

Eric Snow

6 May 2015 6 May '15

9:23 a.m.

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython. Thoughts? -eric

Show replies by date

Stefan Behnel

6 May 6 May

9:36 a.m.

Eric Snow schrieb am 06.05.2015 um 18:23:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

I've been advocating that for years now: leave the low-level stuff to the experts. (There's a reason why Cython code is usually faster than C-API code.) Not sure how big, fat and red the warning needs to be, but a big +1 from me. Stefan

Petr Viktorin

9:45 a.m.

On Wed, May 6, 2015 at 6:36 PM, Stefan Behnel wrote:

...

Eric Snow schrieb am 06.05.2015 um 18:23:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

I've been advocating that for years now: leave the low-level stuff to the experts. (There's a reason why Cython code is usually faster than C-API code.)

Not sure how big, fat and red the warning needs to be, but a big +1 from me.

Probably not too big. Cython and CFFI are easier to use, so people who know about them, and can afford the extra dependency*, should use them. I a pointer would be enough, perhaps like in https://docs.python.org/3/library/urllib.request.html * Possibly build-time dependency. Or a complete rewrite, in case of existing code.

Guido van Rossum

9:41 a.m.

On Wed, May 6, 2015 at 9:23 AM, Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Thoughts?

I think Cython is already used by those people who benefit from it. As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that that would actually break a lot of code because the latest version is so different from the version we currently include?). -- --Guido van Rossum (python.org/~guido)

Paul Moore

11:56 a.m.

On 6 May 2015 at 17:41, Guido van Rossum wrote:

...

As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that that would actually break a lot of code because the latest version is so different from the version we currently include?).

I think you are referring to libffi (used by ctypes) here rather than cffi. Libffi isn't really relevant to the original topic, but the big issue there is that ctypes on Windows uses a patched copy of a pretty old version of libffi. The reason it does is to trap stack usage errors so that it can give a ValueError if you call a C function with the wrong number of arguments. libffi doesn't offer a way to do this, so migrating to the latest libffi would mean hacking out that code, and a loss of the stack checking (which is currently tested for, so although it's not exactly an API guarantee, it would be a compatibility break). Otherwise, it's mostly a matter of getting the build steps to work. Zach Ware got 32-bit builds going, and his approach (use git bash to run configure and keep a copy of the results) should in principle be fine for 64-bit, but I stalled because I've no way of testing a libffi build short of building the whole of Python and running the ctypes tests, which is both heavy handed and likely to obscure the root cause of any actual bugs found that way :-( The big problem is that ctypes with the current embedded libffi is "good enough" on Windows, which is where the bulk of ctypes usage occurs. So there's no compelling reason to put in the work to upgrade it. And (as usual) few people with the necessary expertise (in this case, Windows, C, Unix-style build processes, and assembler-level calling conventions - a pretty impressive mix!). Paul

Guido van Rossum

12:28 p.m.

On Wed, May 6, 2015 at 11:56 AM, Paul Moore wrote:

...

...
As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that

On 6 May 2015 at 17:41, Guido van Rossum wrote: that

...
would actually break a lot of code because the latest version is so different from the version we currently include?).

I think you are referring to libffi (used by ctypes) here rather than cffi.

Oh dear. I think you're right. :-( Forget I said anything. Naming is hard. I'm still not sure how realistic it is to try and deprecate the C API. -- --Guido van Rossum (python.org/~guido)

Florian Bruhin

12:46 p.m.

* Guido van Rossum [2015-05-06 12:28:35 -0700]:

...

On Wed, May 6, 2015 at 11:56 AM, Paul Moore wrote:

...
...
As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that

On 6 May 2015 at 17:41, Guido van Rossum wrote: that

...
would actually break a lot of code because the latest version is so different from the version we currently include?).

I think you are referring to libffi (used by ctypes) here rather than cffi.

Oh dear. I think you're right. :-(

Forget I said anything. Naming is hard.

I'm still not sure how realistic it is to try and deprecate the C API.

I don't think it should be *deprecated*, but I agree the documentation should point out the (probably better) alternatives. From time to time, in the #python IRC channel there are people who want to use C libraries with Python and try doing so with the C API because they aren't aware of the alternatives. Florian -- http://www.the-compiler.org | me@the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/

Joao S. O. Bueno

12:49 p.m.

On 6 May 2015 at 16:28, Guido van Rossum wrote:

...

On Wed, May 6, 2015 at 11:56 AM, Paul Moore wrote:

...
On 6 May 2015 at 17:41, Guido van Rossum wrote:

...
As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that that would actually break a lot of code because the latest version is so different from the version we currently include?).

I think you are referring to libffi (used by ctypes) here rather than cffi.

Oh dear. I think you're right. :-(

Forget I said anything. Naming is hard.

I'm still not sure how realistic it is to try and deprecate the C API.

I am also not sure, but I feel like it would be a __huge__ step forward for other implementations and Python as a language instead of the particular cPython Software Product. Today, many people still see using the C API as "the way" to extend Python, which implies in most extensions created being invalid for all other implementations of the language. (Ok, I actually don't know if cython modules could be called from, say Pypy or Jython, but even if not, I suppose a "jcython" and "pycython" could be made available in the future) (fist-google-entry says cython has at least partial support for pypy already) js -><-

Andrew Barnert

12:57 p.m.

On May 6, 2015, at 12:49, Joao S. O. Bueno wrote:

...

...
On 6 May 2015 at 16:28, Guido van Rossum wrote:

...
On Wed, May 6, 2015 at 11:56 AM, Paul Moore wrote:

...
On 6 May 2015 at 17:41, Guido van Rossum wrote: As for CFFI, is the ownership/maintenance issue solved yet? IIRC we have some really outdated versions in the CPython tree and nobody wants to step in and upgrade these to the latest CFFI, for some reason (such as that that would actually break a lot of code because the latest version is so different from the version we currently include?).

I think you are referring to libffi (used by ctypes) here rather than cffi.

Oh dear. I think you're right. :-(

Forget I said anything. Naming is hard.

I'm still not sure how realistic it is to try and deprecate the C API.

I am also not sure, but I feel like it would be a __huge__ step forward for other implementations and Python as a language instead of the particular cPython Software Product.

Today, many people still see using the C API as "the way" to extend Python, which implies in most extensions created being invalid for all other implementations of the language.

(Ok, I actually don't know if cython modules could be called from, say Pypy or Jython, but even if not, I suppose a "jcython" and "pycython" could be made available in the future) (fist-google-entry says cython has at least partial support for pypy already)

Yes, but PyPy also has pretty good support for C API extensions, too. For Jython and IronPython, I'm not sure what the answer could be. Could Cython automatically build JNI thingies and wrap them up in Java thingies to expose them to Jython even in theory? Even if that worked, what about something like Skulpt? Compile to C code and jsctypes wrappers? So I think there's a limit to what you can expect out of "extending Python the language" in any generic way.

...

js -><- _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Mark Lawrence

1:38 p.m.

On 06/05/2015 19:56, Paul Moore wrote:

...

Otherwise, it's mostly a matter of getting the build steps to work. Zach Ware got 32-bit builds going, and his approach (use git bash to run configure and keep a copy of the results) should in principle be fine for 64-bit, but I stalled because I've no way of testing a libffi build short of building the whole of Python and running the ctypes tests, which is both heavy handed and likely to obscure the root cause of any actual bugs found that way :-(

Paul

Feel free to throw this or any other Windows issues my way. I've all the time in the world to try stuff like this. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

Chris Barker

12:24 p.m.

On Wed, May 6, 2015 at 9:41 AM, Guido van Rossum wrote:

...

I think Cython is already used by those people who benefit from it.

I wish that where the case, but I don't think so -- there is a LOT of weight behind the idea of something being "built-in" and/or "official". So folks do still right extensions using the raw C API. Some note recommending Cython in the core docs about the C API would be great. And we don't use Cython in the standard library, do we? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Antoine Pitrou

9:57 a.m.

On Wed, 6 May 2015 10:23:09 -0600 Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

CFFI is only useful for a small subset of stuff people use the C API for (mainly, thin wrappers around external libraries). Cython is a more reasonable suggestion in this context. I would advocate against red warning boxes. Warnings are for potentially dangerous constructs, we use them mainly for security issues. Adding a note and some pointers at the start of the C API docs may be enough. Regards Antoine.

Donald Stufft

10:13 a.m.

...

On May 6, 2015, at 12:57 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 10:23:09 -0600 Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

CFFI is only useful for a small subset of stuff people use the C API for (mainly, thin wrappers around external libraries). Cython is a more reasonable suggestion in this context.

You can write stuff in C itself for cffi too, it’s not just for C bindings, an example would be the .c’s and .h’s for padding and constant time compare in the cryptography project [1]. [1] https://github.com/pyca/cryptography/tree/master/src/cryptography/hazmat/pri... --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Antoine Pitrou

3:16 p.m.

On Wed, 6 May 2015 13:13:57 -0400 Donald Stufft wrote:

...

...
On May 6, 2015, at 12:57 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 10:23:09 -0600 Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

CFFI is only useful for a small subset of stuff people use the C API for (mainly, thin wrappers around external libraries). Cython is a more reasonable suggestion in this context.

You can write stuff in C itself for cffi too, it’s not just for C bindings, an example would be the .c’s and .h’s for padding and constant time compare in the cryptography project [1].

That really doesn't change what I said. CFFI is not appropriate to write e.g. actual extension classes. Besides, we have ctypes in the standard library, it would be stupid to recommend CFFI and not ctypes. Regards Antoine.

Donald Stufft

3:27 p.m.

...

On May 6, 2015, at 6:16 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 13:13:57 -0400 Donald Stufft wrote:

...
...
On May 6, 2015, at 12:57 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 10:23:09 -0600 Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

CFFI is only useful for a small subset of stuff people use the C API for (mainly, thin wrappers around external libraries). Cython is a more reasonable suggestion in this context.

You can write stuff in C itself for cffi too, it’s not just for C bindings, an example would be the .c’s and .h’s for padding and constant time compare in the cryptography project [1].

That really doesn't change what I said. CFFI is not appropriate to write e.g. actual extension classes.

What is an “actual extension class”?

...

Besides, we have ctypes in the standard library, it would be stupid to recommend CFFI and not ctypes.

Besides the fact that ctypes can only work at the ABI level which flat out doesn’t work for a lot of C projects, but even if you’re working at the ABI level ctypes isn’t nearly as nice to use as CFFI is. With ctypes you have to repeat the C declarations using ctypes special snowflake API but with cffi you just re-use the C declarations (for the most part), in most scenarios you can simply copy/paste from the .h files or man pages or what have you. Here’s a decent read: http://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Antoine Pitrou

3:34 p.m.

On Wed, 6 May 2015 18:27:20 -0400 Donald Stufft wrote:

...

...
On May 6, 2015, at 6:16 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 13:13:57 -0400 Donald Stufft wrote:

...
...
On May 6, 2015, at 12:57 PM, Antoine Pitrou wrote:

On Wed, 6 May 2015 10:23:09 -0600 Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

CFFI is only useful for a small subset of stuff people use the C API for (mainly, thin wrappers around external libraries). Cython is a more reasonable suggestion in this context.

You can write stuff in C itself for cffi too, it’s not just for C bindings, an example would be the .c’s and .h’s for padding and constant time compare in the cryptography project [1].

That really doesn't change what I said. CFFI is not appropriate to write e.g. actual extension classes.

What is an “actual extension class”?

Uh... Please take a look at the C API manual. Regards Antoine.

Andrew Barnert

12:59 p.m.

On May 6, 2015, at 09:23, Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Does this mean you also want to discourage boost::python, SIP, SWIG, etc., which as far as I know come down to automatically building C API extensions, and would need to be completely rewritten if you wanted to make them work a different way?

...

Thoughts?

-eric _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Eric Snow

3 p.m.

On Wed, May 6, 2015 at 1:59 PM, Andrew Barnert wrote:

...

On May 6, 2015, at 09:23, Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Does this mean you also want to discourage boost::python, SIP, SWIG, etc., which as far as I know come down to automatically building C API extensions, and would need to be completely rewritten if you wanted to make them work a different way?

Not really. I mentioned CFFI and Cython specifically because they are the two that kept coming up in previous discussions related to discouraging use of the C-API. If C extensions were always generated using tools, then only tools would have to adapt to (drastic) changes in the C-API. That would be a much better situation than the status quo since it drastically reduces the impact of changes. -eric

Andrew Barnert

11:58 p.m.

On May 6, 2015, at 15:00, Eric Snow wrote:

...

...
On Wed, May 6, 2015 at 1:59 PM, Andrew Barnert wrote:

...
On May 6, 2015, at 09:23, Eric Snow wrote:

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Does this mean you also want to discourage boost::python, SIP, SWIG, etc., which as far as I know come down to automatically building C API extensions, and would need to be completely rewritten if you wanted to make them work a different way?

Not really. I mentioned CFFI and Cython specifically because they are the two that kept coming up in previous discussions related to discouraging use of the C-API. If C extensions were always generated using tools, then only tools would have to adapt to (drastic) changes in the C-API. That would be a much better situation than the status quo since it drastically reduces the impact of changes.

OK, that makes sense to me. Even if there are a dozen wrappers and wrapper generators (and I think it's more like 4 or 5...), and we had to get buy-in from all of them (or get buy-in from most of them and reluctantly decide to screw over the last one), that's still orders of magnitude easier than getting buy-in from (or screw over) the 69105 people who are currently maintaining or building a C API extension, so it's still a huge win. I'm not sure it would do nearly enough, at least not for a long time (how many of the current top 100 projects on PyPI use C API extensions and would be non-trivial to rewrite?), but obviously you can make the point that if we don't do anything, we'll _never_ get there.

M.-A. Lemburg

2:41 p.m.

On 06.05.2015 18:23, Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Thoughts?

Python without the C extensions would hardly have had the success it has. It is widely known as perfect language to glue together different systems and provide integration. Deprecating the C API would mean that you deprecate all those existing C extensions together with the C API. This can hardly be in the interest of Python's quest for world domination :-) BTW: What can be more drastic than deprecating the Python C API ? There are certainly better ways to evolve an API than getting rid of it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Donald Stufft

2:54 p.m.

...

On May 6, 2015, at 5:41 PM, M.-A. Lemburg wrote:

On 06.05.2015 18:23, Eric Snow wrote:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Thoughts?

Python without the C extensions would hardly have had the success it has. It is widely known as perfect language to glue together different systems and provide integration.

Deprecating the C API would mean that you deprecate all those existing C extensions together with the C API.

This can hardly be in the interest of Python's quest for world domination :-)

BTW: What can be more drastic than deprecating the Python C API ? There are certainly better ways to evolve an API than getting rid of it.

I think “deprecate” might be a bad word for it rather than telling people they shoul use CFFI (or Python) instead of the C-API. Similar to having the urllib.request direct people towards the requests project for accessing the internet. CFFI still makes it easy to act as a glue between different systems, it just does so in a way that isn’t tied to one particular implementation’s API and which is generally much easier to work with on top of that. The biggest problems with CFFI currently are the problems in distributing a CFFI module because of some early decisions, but the CFFI 1.0 work is fixing all of that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Eric Snow

3:19 p.m.

On Wed, May 6, 2015 at 3:41 PM, M.-A. Lemburg wrote:

...

Python without the C extensions would hardly have had the success it has. It is widely known as perfect language to glue together different systems and provide integration.

Deprecating the C API would mean that you deprecate all those existing C extensions together with the C API.

As Donald noted, I'm not suggesting that the C-API be deprecated. I was careful in calling it "discouraging direct use of the C-API". :)

...

This can hardly be in the interest of Python's quest for world domination :-)

BTW: What can be more drastic than deprecating the Python C API ? There are certainly better ways to evolve an API than getting rid of it.

I'd like to hear more on alternatives. Lately all I've heard is how much better off we'd be if folks used CFFI or tools like Cython to write their extension modules. Regardless of what it is, we should try to find *some* solution that puts us in a position that we can accomplish certain architectural changes, such as moving away from ref-counting. Larry talked about it at the language summit. -eric

M.-A. Lemburg

7 May 7 May

12:56 a.m.

On 07.05.2015 00:19, Eric Snow wrote:

...

On Wed, May 6, 2015 at 3:41 PM, M.-A. Lemburg wrote:

...
Python without the C extensions would hardly have had the success it has. It is widely known as perfect language to glue together different systems and provide integration.

Deprecating the C API would mean that you deprecate all those existing C extensions together with the C API.

As Donald noted, I'm not suggesting that the C-API be deprecated. I was careful in calling it "discouraging direct use of the C-API". :)

Looks like that didn't work out when I read your suggestion :-) I'd expect a big red warning on all C API pages to have a similar effect on others.

...

...
This can hardly be in the interest of Python's quest for world domination :-)

BTW: What can be more drastic than deprecating the Python C API ? There are certainly better ways to evolve an API than getting rid of it.

I'd like to hear more on alternatives. Lately all I've heard is how much better off we'd be if folks used CFFI or tools like Cython to write their extension modules. Regardless of what it is, we should try to find *some* solution that puts us in a position that we can accomplish certain architectural changes, such as moving away from ref-counting. Larry talked about it at the language summit.

C is pretty flexible when it comes to changing APIs gradually, e.g. you can have macros adjusting signatures for you or small wrapper functions fixing semantics, providing additional arguments, etc. I think it would be better to first investigate possible changes to the C API before recommending putting a layer between Python's C API and its C extensions. Those layers are useful for people who don't want to dive into the C API, but don't work well for those who know the C API and how to use it to give them the best possible performance or best possible integration with Python. I haven't seen Larry's talk, just read a short summary of things he mentioned in that talk. Those looked like a good starting point for discussions. Perhaps we could have GSoC students investigate some of these alternatives ?! Removing the GIL and reference counting will break things, but if there is a way we can reduce this breakage, I think we should definitely go for that approach before saying "oh, no, please don't use our C API". Aside: The fact that we have so many nice C extensions out there is proof that we have a good C API. Even though it is not visible to most Python programmers, it forms a significant part of Python's success. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 07 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Florian Bruhin

1:18 a.m.

* M.-A. Lemburg [2015-05-07 09:56:15 +0200]:

...

Aside: The fact that we have so many nice C extensions out there is proof that we have a good C API. Even though it is not visible to most Python programmers, it forms a significant part of Python's success.

Are many of those using the C API directly rather than using some bindings generator? Most projects I'm aware of use Cython/cffi/SWIG/... and not the raw C API, which is kind of the whole point here :) Florian -- http://www.the-compiler.org | me@the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/

Paul Moore

1:47 a.m.

On 7 May 2015 at 08:56, M.-A. Lemburg wrote:

...

Aside: The fact that we have so many nice C extensions out there is proof that we have a good C API. Even though it is not visible to most Python programmers, it forms a significant part of Python's success.

Agreed. Maybe a useful exercise for someone thinking about this issue would be to survey some of the major projects using the C API out there, and working out what would be involved in switching them to use cffi or Cython. That would give a good idea of the scale of the issue, as well as providing some practical help to projects that would be affected by this sort of recommendation. Good ones to look at would be: - lxml - pywin32 (I refrained from adding scipy and numpy to that list, as that would make this post seem like a troll attempt, which it isn't, but has anyone thought of the implications of a recommendation like this on those projects? OK, they'd probably just ignore it as they have a genuine need for direct use of the C API, but we would be sending pretty mixed messages). I prefer Nick's suggestion of adding better documentation to the packaging user guide. Maybe even to the extent of having a worked example. The article at https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_wit... is quite a nice overview, although it's heavily numpy-focused and doesn't include cffi. Paul

Stefan Behnel

10:23 a.m.

Paul Moore schrieb am 07.05.2015 um 10:47:

...

On 7 May 2015 at 08:56, M.-A. Lemburg wrote:

...
Aside: The fact that we have so many nice C extensions out there is proof that we have a good C API. Even though it is not visible to most Python programmers, it forms a significant part of Python's success.

Oh, totally. But that doesn't mean people have to manually write code against it, in the same way that you can benefit from excellent processors without writing assembly.

...

Maybe a useful exercise for someone thinking about this issue would be to survey some of the major projects using the C API out there, and working out what would be involved in switching them to use cffi or Cython. That would give a good idea of the scale of the issue, as well as providing some practical help to projects that would be affected by this sort of recommendation.

My general answer is that "Python is way easier to write than C", and therefore "rewriting C code in Cython" is a rather fast thing to do (P's and C's set as intended). Often enough, the rewrite also leads to immediate functional improvements because stuff can easily be done in a more general way in Python syntax than in plain C(-API) code. And it's not uncommon that several ref-counting and/or error handling bugs get fixed on the way. When I rewrite C-API code in Cython, the bulk of the time is spent reverse engineering the intended Python semantics from the verbose (and sometimes cryptic) C code. After that, writing them down in Python syntax is quite easy. Once you get used to it, the plain transformation can be done at more than a hundred lines of C code per hour, if it's not overly complex or dense (the usual 5%). If you have a good test suite, debugging the rewritten code should be quite straight forward afterwards. So, if you have a project with 10000 lines of C code, 30% of which uses the C-API, you should be able to rip out the direct usage of the C-API in just a couple of days by rewriting it in Cython. The code size usually drops by a factor of 2-5 that way. That also makes it a reasonable migration path for porting Py2.x C-API code to Py3, for example. I can't speak for cffi, but my guess is that if you know its API well, the fact that it's also Python should keep the rewriting speed in the same ball park as for Cython. So, for code that isn't performance critical, it's certainly a reasonable alternative, with the added benefit of having excellent support in PyPy.

...

Good ones to look at would be: - lxml

lxml has been written in Cython even before Cython existed (it used to be a patched Pyrex at the time). In fact, writing it in C would have been entirely impossible. Even if the necessary developer resources had been available, writing C code is so difficult in comparison that many of the non-trivial features would never have been implemented.

...

(I refrained from adding scipy and numpy to that list, as that would make this post seem like a troll attempt, which it isn't, but has anyone thought of the implications of a recommendation like this on those projects? OK, they'd probably just ignore it as they have a genuine need for direct use of the C API, but we would be sending pretty mixed messages).

Much of scipy and its surrounding tools and libraries are actually written in Cython. At least much of their parts that interact with Python, and often a lot more than just the interface layer. New code in the scientific computing community is commonly written in Cython these days, or uses other tools for JIT or AOT compilation (Numba, numexpr, ...), many of which were themselves partly written in Cython. Stefan

Nathaniel Smith

12:19 p.m.

On May 7, 2015 10:24 AM, "Stefan Behnel" wrote:

...

Paul Moore schrieb am 07.05.2015 um 10:47:

...
(I refrained from adding scipy and numpy to that list, as that would make this post seem like a troll attempt, which it isn't, but has anyone thought of the implications of a recommendation like this on those projects? OK, they'd probably just ignore it as they have a genuine need for direct use of the C API, but we would be sending pretty mixed messages).

Much of scipy and its surrounding tools and libraries are actually written in Cython. At least much of their parts that interact with Python, and often a lot more than just the interface layer. New code in the scientific computing community is commonly written in Cython these days, or uses

other

...

tools for JIT or AOT compilation (Numba, numexpr, ...), many of which were themselves partly written in Cython.

Yeah, I think if anyone talks to the developers of those libraries they will get a very *un*mixed message saying, don't do what we did :-). One of scipy's GSoC projects this year is even porting a c extension to Cython, and I've been actively investigating the possibility of porting numpy into Cython as well. Mostly for the immediate benefits, but certainly it has occurred to me that in the long run this could potentially provide an escape hatch from CPython. (Numerical people are *very* interested in JITs... and something like Cython provides the unique possibility that if a project like PyPy or pyston added direct support for the language, then one could write a single source file that was fast on cpython b/c it compiled to C, and was even faster on other interpreters because the same source got jitted.) The main obstacle to porting numpy, btw, is that Cython currently assumes that each source file will generate one python extension, and any communication between source files will be via python-level imports. NumPy, of course, has 100,000 lines of C across lots of files that are all built into one extension module, and which happily communicate via direct C function calls. So incrementally porting is impossible without teaching Cython to handle this case a bit better. NumPy is an extreme outlier in this regard though. In particular this is absolutely not a reason to steer *new* projects away from Cython. -n

Nick Coghlan

6 May 6 May

9:22 p.m.

On 7 May 2015 at 02:23, Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

Thoughts?

Rather than embedding these recommendations directly in the version specific CPython docs, I'd prefer to see contributions to fill in the incomplete sections in https://packaging.python.org/en/latest/extensions.html with links back to the relevant parts of the C API documentation and docs for other projects (I was able to write the current overview section on that page in a few hours, as I didn't need to do much research for that, but filling in the other sections properly involves significantly more work). That page is already linked from the landing page for the extending & embedding documentation as part of a recommendation to consider the use of third party tools rather than handcrafting your own extension modules: https://docs.python.org/3/extending/index.html#recommended-third-party-tools The landing page for the C API docs links back to the extending & embedding guide, but the link is embedded in the header paragraph rather than being a See Also link: https://docs.python.org/3/c-api/index.html Cheers, Nick.

...

-eric _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Stefan Krah

7 May 7 May

3:15 a.m.

Nick Coghlan writes:

...

Rather than embedding these recommendations directly in the version specific CPython docs, I'd prefer to see contributions to fill in the incomplete sections in https://packaging.python.org/en/latest/extensions.html with links back to the relevant parts of the C API documentation and docs for other projects (I was able to write the current overview section on that page in a few hours, as I didn't need to do much research for that, but filling in the other sections properly involves significantly more work).

Hmm. I'm getting a twilio.com advertisement on that page. I miss the old python.org... Stefan Krah

Steven D'Aprano

4:30 a.m.

On Thu, May 07, 2015 at 10:15:04AM +0000, Stefan Krah wrote:

...

Nick Coghlan writes:

...
Rather than embedding these recommendations directly in the version specific CPython docs, I'd prefer to see contributions to fill in the incomplete sections in https://packaging.python.org/en/latest/extensions.html with links back to the relevant parts of the C API documentation and docs for other projects (I was able to write the current overview section on that page in a few hours, as I didn't need to do much research for that, but filling in the other sections properly involves significantly more work).

Hmm. I'm getting a twilio.com advertisement on that page. I miss the old python.org...

I see it too. Why is python.org displaying advertisments? -- Steve

Donald Stufft

4:44 a.m.

...

On May 7, 2015, at 7:30 AM, Steven D'Aprano wrote:

On Thu, May 07, 2015 at 10:15:04AM +0000, Stefan Krah wrote:

...
Nick Coghlan writes:

...
Rather than embedding these recommendations directly in the version specific CPython docs, I'd prefer to see contributions to fill in the incomplete sections in https://packaging.python.org/en/latest/extensions.html with links back to the relevant parts of the C API documentation and docs for other projects (I was able to write the current overview section on that page in a few hours, as I didn't need to do much research for that, but filling in the other sections properly involves significantly more work).

Hmm. I'm getting a twilio.com advertisement on that page. I miss the old python.org...

I see it too. Why is python.org displaying advertisments?

packaging.python.org is hosted on RTD, I guess that RTD added ads to it’s free service. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Terry Reedy

8:58 a.m.

On 5/7/2015 7:44 AM, Donald Stufft wrote:

...

...
...
...
https://packaging.python.org/en/latest/extensions.html

...

...
...
Hmm. I'm getting a twilio.com advertisement on that page. I miss the old python.org...

...

...
I see it too. Why is python.org displaying advertisments?

...

packaging.python.org is hosted on RTD, I guess that RTD added ads to it’s free service.

I don't see any ad (using Verizon FIOS). I have noscript running, but allowing first readthedocs.com and then grokthedocs.com did not produce an ad. -- Terry Jan Reedy

Stefan Krah

9:54 a.m.

Terry Reedy writes:

...

...
...
...
...
https://packaging.python.org/en/latest/extensions.html I don't see any ad (using Verizon FIOS). I have noscript running, but allowing first readthedocs.com and then grokthedocs.com did not produce an ad.

It's gone now. It was there when I posted earlier (I even clicked through). Stefan Krah

Stefan Krah

1:54 a.m.

Eric Snow writes:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

-1. CFFI is much slower than using the C-API directly. Python is a great language by itself, but its excellent C-API is one of the major selling points. As for garbage collection vs. refcounting: I've tried OCaml's C-API and found it 20% slower than Python's. Note that OCaml has a fantastic native code compiler (and the culture is C-friendly), so it seems to be a hard problem. Stefan Krah

Caleb Hattingh

2:01 a.m.

...

On 7 May 2015, at 6:54 pm, Stefan Krah wrote:

Eric Snow writes:

...
A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

-1. CFFI is much slower than using the C-API directly.

I am quite interested in this; do you happen have a link to a case study/gist/repo where this has been measured? Even if you can remember people’s names involved or something similar, I could google it myself. Kind regards Caleb

Stefan Krah

2:11 a.m.

Caleb Hattingh writes:

...

...
-1. CFFI is much slower than using the C-API directly.

I am quite interested in this; do you happen have a link to a case study/gist/repo where this has been measured? Even if you can remember people’s names involved or something similar, I could google it myself.

I've measured it here: https://mail.python.org/pipermail/python-dev/2013-December/130772.html CFFI is very nice (superb API), but not for high performance use cases. Stefan Krah

Donald Stufft

2:14 a.m.

...

On May 7, 2015, at 5:11 AM, Stefan Krah wrote:

Caleb Hattingh writes:

...
...
-1. CFFI is much slower than using the C-API directly.

I am quite interested in this; do you happen have a link to a case study/gist/repo where this has been measured? Even if you can remember people’s names involved or something similar, I could google it myself.

I've measured it here:

https://mail.python.org/pipermail/python-dev/2013-December/130772.html

CFFI is very nice (superb API), but not for high performance use cases.

Is the source code for this benchmark available anywhere? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Paul Moore

4:50 a.m.

On 7 May 2015 at 10:11, Stefan Krah wrote:

...

Caleb Hattingh writes:

...
...
-1. CFFI is much slower than using the C-API directly.

I am quite interested in this; do you happen have a link to a case study/gist/repo where this has been measured? Even if you can remember people’s names involved or something similar, I could google it myself.

I've measured it here:

https://mail.python.org/pipermail/python-dev/2013-December/130772.html

CFFI is very nice (superb API), but not for high performance use cases.

I'm guessing that benchmark used cffi in the "ABI level" dynamic form that matches ctypes. Did you try the cffi "API level" form that creates a C extension? I'd be curious as to where that falls in performance. Paul

Caleb Hattingh

4:58 a.m.

...

On 7 May 2015, at 9:50 pm, Paul Moore wrote:

On 7 May 2015 at 10:11, Stefan Krah wrote:

...
Caleb Hattingh writes:

...
...
-1. CFFI is much slower than using the C-API directly.

I am quite interested in this; do you happen have a link to a case study/gist/repo where this has been measured? Even if you can remember people’s names involved or something similar, I could google it myself.

I've measured it here:

https://mail.python.org/pipermail/python-dev/2013-December/130772.html

CFFI is very nice (superb API), but not for high performance use cases.

I'm guessing that benchmark used cffi in the "ABI level" dynamic form that matches ctypes. Did you try the cffi "API level" form that creates a C extension? I'd be curious as to where that falls in performance.

I had a quick look around, @eevee made this comparison some time ago: === • CPython 2.7 + Cython: 2.0s • CPython 2.7 + CFFI: 2.7s • PyPy 2.1 + CFFI: 4.3s That’s the time it takes, from a warm start, to run the test suite. === from http://eev.ee/blog/2013/09/13/cython-versus-cffi/ Kind regards Caleb

Stefan Krah

8 May 8 May

2:09 a.m.

Paul Moore writes:

...

...
https://mail.python.org/pipermail/python-dev/2013-December/130772.html

CFFI is very nice (superb API), but not for high performance use cases.

I'm guessing that benchmark used cffi in the "ABI level" dynamic form that matches ctypes. Did you try the cffi "API level" form that creates a C extension? I'd be curious as to where that falls in performance.

ffi.verify() is only about 10% faster both in pypy and cpython, so it doesn't change much in the posted figures. Stefan Krah

David Wilson

7 May 7 May

5:32 a.m.

On Wed, May 06, 2015 at 10:23:09AM -0600, Eric Snow wrote:

...

A big blocker to making certain sweeping changes to CPython (e.g. ref-counting) is compatibility with the vast body of C extension modules out there that use the C-API. While there are certainly drastic long-term solutions to that problem, there is one thing we can do in the short-term that would at least get the ball rolling. We can put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

One of CPython's traditional strongholds is its use as an embedded language. I've worked on a bunch of commercial projects using it in this way, often specifically for improved performance/access to interpreter internals, and this is not to mention the numerous free software projects doing similar: gdb, uwsgi, mod_python, Freeswitch, and so on. It might be better to discuss specifics of what should change in the API besides refcounting, and hammer out concrete steps to make those changes happen, since I doubt the C API is ever going to go away, as even if all extension modules were rewritten today its use for embedding would still prevent sweeping changes without upsetting a huge number of users and mature products. David

Stefan Behnel

8 May 8 May

3:17 a.m.

David Wilson schrieb am 07.05.2015 um 14:32:

...

On Wed, May 06, 2015 at 10:23:09AM -0600, Eric Snow wrote:

...
put a big red note at the top of every page of the C-API docs that encourages folks to either use CFFI or Cython.

One of CPython's traditional strongholds is its use as an embedded language. I've worked on a bunch of commercial projects using it in this way, often specifically for improved performance/access to interpreter internals, and this is not to mention the numerous free software projects doing similar: gdb, uwsgi, mod_python, Freeswitch, and so on.

Ah, yes, there is a big wall in the CPython docs between "extending" and "embedding" that gives users the impression that they are really different concepts. But that's just marketing. They are not. The only difference is that in one case, it's the CPython interpreter that starts up and then calls into native user code, and in the other case, it's user code that starts up and then launches a CPython interpreter. From the moment on where both the user code and the CPython interpreter are running, there is exactly zero difference between the two, and you can use the same tools for interfacing native code with Python code in both cases. What this means is that even in an embedding scenario, user code will typically only need to call a tiny set of C-API functions to start up and shut down the interpreter, and then leave all the rest, all the interesting stuff, to tools that do it better. Stefan

3273

Age (days ago)

3275

Last active (days ago)

List overview

Download

41 comments

20 participants

participants (20)

Andrew Barnert
Antoine Pitrou
Caleb Hattingh
Chris Barker
David Wilson
Donald Stufft
Eric Snow
Florian Bruhin
Guido van Rossum
Joao S. O. Bueno
M.-A. Lemburg
Mark Lawrence
Nathaniel Smith
Nick Coghlan
Paul Moore
Petr Viktorin
Stefan Behnel
Stefan Krah
Steven D'Aprano
Terry Reedy

discouraging direct use of the C-API

Mark Lawrence

tags

participants (20)