Hello. I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release. Notable projects using it: * pypycore - gevent main loop ported to cffi * pgsql2cffi * sdl-cffi bindings * tls-cffi bindings * lxml-cffi port * pyzmq * cairo-cffi * a bunch of others So relatively a lot given that the project is not even a year old (it got 0.1 release in June). As per documentation, the advantages over ctypes: * The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn. * Keep all the Python-related logic in Python so that you don’t need to write much C code (unlike CPython native C extensions). * Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.) * We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions). * We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython. * Note that this project is not about embedding executable C code in Python, unlike Weave. This is about calling existing C libraries from Python. so among other things, making a cffi extension gives you the same level of security as writing C (and unlike ctypes) and brings quite a bit more flexibility (API vs ABI issue) that let's you wrap arbitrary libraries, even those full of macros. Cheers, fijal .. [1]: http://cffi.readthedocs.org/en/release-0.5/
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to) If I'm reading the dependencies correctly, we would also need to bring Eli Bendersky's pycparser into the stdlib, correct? (not an objection, just an observation, although we'd obviously need Eli's explicit endorsement for that). There would also be the matter of converting the cffi test suite from py.test to stdlib unittest. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
If I'm reading the dependencies correctly, we would also need to bring Eli Bendersky's pycparser into the stdlib, correct? (not an objection, just an observation, although we'd obviously need Eli's explicit endorsement for that).
Yes, although, depending on the circumstances it's possible to hide it somewhere in cffi and not make it an official stdlib API (then several rules does not apply). It also pulls in some version of ply (or a generated parser).
There would also be the matter of converting the cffi test suite from py.test to stdlib unittest.
I volunteer to do that, if necessary.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 26, 2013 at 8:26 AM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
If I'm reading the dependencies correctly, we would also need to bring Eli Bendersky's pycparser into the stdlib, correct? (not an objection, just an observation, although we'd obviously need Eli's explicit endorsement for that).
Yes, although, depending on the circumstances it's possible to hide it somewhere in cffi and not make it an official stdlib API (then several rules does not apply). It also pulls in some version of ply (or a generated parser).
I'll be the first one to admit that pycparser is almost certainly not generally useful enough to be exposed in the stdlib. So just using it as an implementation detail is absolutely fine. PLY is a more interesting question, however, since PLY is somewhat more generally useful. That said, I see all this as implementation details that shouldn't distract us from the main point of whether cffi should be added. Eli
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
If I'm reading the dependencies correctly, we would also need to bring Eli Bendersky's pycparser into the stdlib, correct? (not an objection, just an observation, although we'd obviously need Eli's explicit endorsement for that).
Of course I have no problem with this. I obviously signed the contributor agreement by this point ;-) pycparser depends on David Beazley's PLY though, so that will have to be pulled in as well. Eli
On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields.
On Wed, Feb 27, 2013 at 2:39 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields.
For me, it's mostly the fact that ctypes is *less* typesafe than C itself, because you can't easily consume the header files directly, which leads directly to the segfaults caused by wrong declarations. While at the time of inclusion there was a solid "practicality beats purity" case for adding ctypes (and it's taken us quite a long way), being less typesafe than C is still quite an achievement :) As an experienced C programmer, not having to learn a new hybrid language for declarations is also a plus, but the "significantly less likely to segfault" aspect is the big one. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com>
wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields.
Yep, I can think of some reasons too. But just mentioning it so you remember explicitly listing the advantages when it comes to writing a PEP or some other sort of formal proposal. An FWIW, I think there's already enough positive feedback to at least start drafting a PEP. It can always be rejected later ;-) Eli
On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields.
Yep, I can think of some reasons too. But just mentioning it so you remember explicitly listing the advantages when it comes to writing a PEP or some other sort of formal proposal. An FWIW, I think there's already enough positive feedback to at least start drafting a PEP. It can always be rejected later ;-)
Eli
It's copy-pasted in the original mail and it's in the docs. Anyway, this is just to familizarize people with the library. We're going to discuss it on the language summit at pycon.
+1 On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben@gmail.com>
wrote:
On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <
fijall@gmail.com>
wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to)
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields.
Yep, I can think of some reasons too. But just mentioning it so you remember explicitly listing the advantages when it comes to writing a PEP or some other sort of formal proposal. An FWIW, I think there's already enough positive feedback to at least start drafting a PEP. It can always be rejected later ;-)
Eli
It's copy-pasted in the original mail and it's in the docs. Anyway, this is just to familizarize people with the library. We're going to discuss it on the language summit at pycon. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
On 26 February 2013 16:34, Eli Bendersky <eliben@gmail.com> wrote:
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
I'm probably the same, mainly because I've successfully used ctypes in the past, but I've never used cffi. That's something I need to rectify. One point which I *think* is correct, but which I don't see noted anywhere. Am I right that cffi needs a C compiler involved in the process, at least somewhere? If that's the case, then it is not a suitable option for at least one use case that I have, writing quick hacks involving the Windows API on a machine that doesn't have a C compiler installed. Another possible case would be writing zip-safe code - if cffi involves a compiled C extension, it won't work when loaded from a zipfile. In general, a proper, unbiased comparison between cffi and ctypes would be useful. BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes? Paul
On Tue, Feb 26, 2013 at 8:14 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 26 February 2013 16:34, Eli Bendersky <eliben@gmail.com> wrote:
I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior.
I'm probably the same, mainly because I've successfully used ctypes in the past, but I've never used cffi. That's something I need to rectify.
One point which I *think* is correct, but which I don't see noted anywhere. Am I right that cffi needs a C compiler involved in the process, at least somewhere? If that's the case, then it is not a suitable option for at least one use case that I have, writing quick hacks involving the Windows API on a machine that doesn't have a C compiler installed. Another possible case would be writing zip-safe code - if cffi involves a compiled C extension, it won't work when loaded from a zipfile.
cffi does require a C compiler during either runtime or installation, if you use the API. You can still use the ABI level, which is like ctypes, only if you need the extra stuff, you need a C compiler.
In general, a proper, unbiased comparison between cffi and ctypes would be useful.
that obviously can't come from me.
On 26 February 2013 18:34, Maciej Fijalkowski <fijall@gmail.com> wrote:
One point which I *think* is correct, but which I don't see noted anywhere. Am I right that cffi needs a C compiler involved in the process, at least somewhere? If that's the case, then it is not a suitable option for at least one use case that I have, writing quick hacks involving the Windows API on a machine that doesn't have a C compiler installed. Another possible case would be writing zip-safe code - if cffi involves a compiled C extension, it won't work when loaded from a zipfile.
cffi does require a C compiler during either runtime or installation, if you use the API. You can still use the ABI level, which is like ctypes, only if you need the extra stuff, you need a C compiler.
Cool, thanks. So if I use it like ctypes, cffi doesn't need a C compiler? That's nice, I never knew that. I definitely need to try it out. (So many projects, so little time... :-)) Paul
On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore@gmail.com> wrote:
BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes?
I would expect that ctypes would be deprecated eventually simply because there aren't very many people interested in maintaining it, and probably fewer if cffi is accepted. That said, I would not expect it to leave the stdlib until either the bit rot was so bad it wouldn't be worth shipping it, or (more likely) we reach Python4 and decide at that time that it is time for it to go. Of course, this is just me talking, we only have a *very* vague "sense of the house" for what Python4 means at this point :) --David
Generally speaking, deferring something to Python 4 means "never". On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray@bitdance.com> wrote:
On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore@gmail.com> wrote:
BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes?
I would expect that ctypes would be deprecated eventually simply because there aren't very many people interested in maintaining it, and probably fewer if cffi is accepted. That said, I would not expect it to leave the stdlib until either the bit rot was so bad it wouldn't be worth shipping it, or (more likely) we reach Python4 and decide at that time that it is time for it to go.
Of course, this is just me talking, we only have a *very* vague "sense of the house" for what Python4 means at this point :)
--David _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido@python.org> wrote:
Generally speaking, deferring something to Python 4 means "never".
Does that mean your aversion to double digit version numbers (i.e. 3.10) is gone or you expect to freeze Python in carbonite by then? -Brett
On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray@bitdance.com> wrote:
On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore@gmail.com> wrote:
BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes?
I would expect that ctypes would be deprecated eventually simply because there aren't very many people interested in maintaining it, and probably fewer if cffi is accepted. That said, I would not expect it to leave the stdlib until either the bit rot was so bad it wouldn't be worth shipping it, or (more likely) we reach Python4 and decide at that time that it is time for it to go.
Of course, this is just me talking, we only have a *very* vague "sense of the house" for what Python4 means at this point :)
--David _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
On Tue, 26 Feb 2013 14:21:03 -0500 Brett Cannon <brett@python.org> wrote:
On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido@python.org> wrote:
Generally speaking, deferring something to Python 4 means "never".
Does that mean your aversion to double digit version numbers (i.e. 3.10) is gone or you expect to freeze Python in carbonite by then?
Or we'll go straight to 5. (or switch to date-based numbering :-)) Regards Antoine.
Le Wed, 27 Feb 2013 12:15:05 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Antoine Pitrou wrote:
Or we'll go straight to 5. (or switch to date-based numbering :-))
We could go the Apple route and start naming them after species of snake.
We have to find sufficiently silly species of snakes, though. Regards Antoine.
On 2013-02-27, at 14:31 , Antoine Pitrou wrote:
Le Wed, 27 Feb 2013 12:15:05 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Antoine Pitrou wrote:
Or we'll go straight to 5. (or switch to date-based numbering :-))
We could go the Apple route and start naming them after species of snake.
We have to find sufficiently silly species of snakes, though.
With about 3000 extant snake species, that should be manageable.
On 27 February 2013 10:31, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Wed, 27 Feb 2013 12:15:05 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Antoine Pitrou wrote:
Or we'll go straight to 5. (or switch to date-based numbering :-))
We could go the Apple route and start naming them after species of snake.
We have to find sufficiently silly species of snakes, though.
Monty Python feature movies. There are less than snake species, but imagine porting Django to be able to run on Life of Brian!
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
Antoine Pitrou wrote:
We have to find sufficiently silly species of snakes, though.
Glancing through http://en.wikipedia.org/wiki/List_of_snakes, we have: Wart snakes Java wart snakes Pipe snakes Stiletto snakes Rubber boas Dog-faced water snakes Shovel-nosed snakes Hook-nosed snakes Leaf-nosed snakes Lyre snakes Cat-eyed snakes Worm snakes North American hog-nosed snakes Moccasins Humpnose vipers Puff adders Carpet vipers False horned vipers Sunbeam snakes And then there's this: http://www.nairaland.com/1009227/new-species-snake-discovered-brazil -- Greg
Le Thu, 28 Feb 2013 10:57:50 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Antoine Pitrou wrote:
We have to find sufficiently silly species of snakes, though.
Glancing through http://en.wikipedia.org/wiki/List_of_snakes, we have:
Wart snakes Java wart snakes Pipe snakes Stiletto snakes Rubber boas Dog-faced water snakes Shovel-nosed snakes
I'm looking forward to our shovel-nosed release plugging the XML security holes, then. Regards Antoine.
With 1.5 years per release, it'd be 10 years before we'd hit 3.10.
From a software engineering perspective, 10 years is indistinguishable from infinity, so I don't care what happens 10 years from now -- as long as you don't blame me. :-)
On Tue, Feb 26, 2013 at 11:21 AM, Brett Cannon <brett@python.org> wrote:
On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido@python.org> wrote:
Generally speaking, deferring something to Python 4 means "never".
Does that mean your aversion to double digit version numbers (i.e. 3.10) is gone or you expect to freeze Python in carbonite by then?
-Brett
On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray@bitdance.com> wrote:
On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore@gmail.com> wrote:
BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes?
I would expect that ctypes would be deprecated eventually simply because there aren't very many people interested in maintaining it, and probably fewer if cffi is accepted. That said, I would not expect it to leave the stdlib until either the bit rot was so bad it wouldn't be worth shipping it, or (more likely) we reach Python4 and decide at that time that it is time for it to go.
Of course, this is just me talking, we only have a *very* vague "sense of the house" for what Python4 means at this point :)
--David _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
-- --Guido van Rossum (python.org/~guido)
Hi Guido, On Tue, Feb 26, 2013 at 8:24 PM, Guido van Rossum <guido@python.org> wrote:
From a software engineering perspective, 10 years is indistinguishable from infinity, so I don't care what happens 10 years from now -- as long as you don't blame me. :-)
I can't resist: around today it is the 10th anniversary of PyPy. :-) A bientôt, Armin.
A big +1 from me for cffi in the stdlib it's a great library. I just recently started using it to make bindings to a C library. I looked at the ctypes library, but haven't actually used it, because the docs confused me but with cffi I was able to get somewhere just by a liberal use of copy/paste from the C library's home page. The only C I know I figured out from working with cffi and fwiw even with that minimal amount of gained knowledge I still am confused by the Ctypes docs and where to even really start to try to use it. On Tuesday, February 26, 2013 at 10:13 AM, Maciej Fijalkowski wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release. Notable projects using it:
* pypycore - gevent main loop ported to cffi * pgsql2cffi * sdl-cffi bindings * tls-cffi bindings * lxml-cffi port * pyzmq * cairo-cffi * a bunch of others
So relatively a lot given that the project is not even a year old (it got 0.1 release in June). As per documentation, the advantages over ctypes:
* The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn.
* Keep all the Python-related logic in Python so that you don’t need to write much C code (unlike CPython native C extensions).
* Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.)
* We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions).
* We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython.
* Note that this project is not about embedding executable C code in Python, unlike Weave. This is about calling existing C libraries from Python.
so among other things, making a cffi extension gives you the same level of security as writing C (and unlike ctypes) and brings quite a bit more flexibility (API vs ABI issue) that let's you wrap arbitrary libraries, even those full of macros.
Cheers, fijal
.. [1]: http://cffi.readthedocs.org/en/release-0.5/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org (mailto:Python-Dev@python.org) http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
On Tue, Feb 26, 2013 at 4:13 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release.
I think this would be great to have in the stdlib. I think it's useful to note that PyPy is planning to include this in their stdlib for the next release, too, right? Cheers, Dirkjan
Le Tue, 26 Feb 2013 17:13:44 +0200, Maciej Fijalkowski <fijall@gmail.com> a écrit :
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release.
Do you intend to actually maintain it inside the CPython repository? Regards Antoine.
On Tue, Feb 26, 2013 at 6:32 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Tue, 26 Feb 2013 17:13:44 +0200, Maciej Fijalkowski <fijall@gmail.com> a écrit :
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release.
Do you intend to actually maintain it inside the CPython repository?
Regards
Antoine.
Once we put it in, yes, of course. Me Armin and Alex.
On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
How does it compare in terms of speed. One reason ctypes has not replaces hand-tuned swig is that it apparently is much slower. I know that someone, for instance, once did a pygame fork using ctypes to wrap SDL, and got it to run, but gave it up for comparative slowness reasons. -- Terry Jan Reedy
On Wed, Feb 27, 2013 at 3:39 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
How does it compare in terms of speed. One reason ctypes has not replaces hand-tuned swig is that it apparently is much slower. I know that someone, for instance, once did a pygame fork using ctypes to wrap SDL, and got it to run, but gave it up for comparative slowness reasons.
I never did the comparison with hand-tuned swig. It's faster than ctypes for calls and manipulation, C extension beat it to some degree. It's uber fast on PyPy (a bit over the native call, but we'll get it to the native call level).
* Work either at the level of the ABI (Application Binary Interface)
or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.)
I read the cffi docs once again and went through some of the examples. I want to divide this to two topics. One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs. Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me: 1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC? 2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed? 3. The partial type specifications in C with ellipsis. What is the point? You have the C declarations somewhere anyhow, so why introduce this? The "ABI level" boasts having just C and Python to write, but those partial ellipsis-ridden declarations are hardly C. Please see this as constructive criticism, and I fully admit that I may not have went deep enough to see the big benefits of the "API" level that would justify all that complexity. Eli
On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky <eliben@gmail.com> wrote:
* Work either at the level of the ABI (Application Binary Interface)
or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.)
I read the cffi docs once again and went through some of the examples. I want to divide this to two topics.
One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs.
Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me:
1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC?
Yes. Precedent in the stdlib is really the C API. All the same rules apply (including build and ship a dll).
2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed?
We welcome a better opinion of name (indeed verify is not that great). This elevates ABI to API so either invokes the C compiler or reads stuff from the cache.
3. The partial type specifications in C with ellipsis. What is the point? You have the C declarations somewhere anyhow, so why introduce this? The "ABI level" boasts having just C and Python to write, but those partial ellipsis-ridden declarations are hardly C.
No, you don't. Some libraries contain macros for example (like OpenSSL) where you just can't use ABI because it makes no sense. It's less common on windows where binary compatibility is important, however looking on linux, multiple stdlib declaration would use ellipsis in the man page. I can't seem to find one right now, but it's something like: struct X { int public_field; ... } which is impossible to do correctly with ctypes without exposing some sort of platform dependency that might change without warning. Another usages are #define SQLITE_OK ... which you don't know at the time of writing (people assume those won't change and the do change).
Please see this as constructive criticism, and I fully admit that I may not have went deep enough to see the big benefits of the "API" level that would justify all that complexity.
Sure. It might sound strange, but on linux (or posix in general) binary compatibility is not a thing, so you need to do stuff to account for that.
On Wed, Feb 27, 2013 at 7:00 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky <eliben@gmail.com> wrote:
2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed?
We welcome a better opinion of name (indeed verify is not that great). This elevates ABI to API so either invokes the C compiler or reads stuff from the cache.
Have you considered the name "create_api"? After all, you're essentially creating a Python API given a C header declaration and a specified extension module to cache the result. The details on extension module caching, and the fact that it won't even look for a C compiler if the extension module is already available should also be in the main section of the docs on the verification step rather than only being down in the internal documentation for cffi.verifier.Verifier Cheers, Nick. P.S. I created both of those as tracker issues: - https://bitbucket.org/cffi/cffi/issue/59/missing-content-in-cffiverify - https://bitbucket.org/cffi/cffi/issue/60/proposed-rename-for-verify-and-veri... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I read the cffi docs once again and went through some of the examples. I want to divide this to two topics.
One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs.
Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me:
1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC?
Yes. Precedent in the stdlib is really the C API. All the same rules apply (including build and ship a dll).
So would you say that the main use of the API level is provide an alternative for writing C API code to interface to C libraries. IOW, it's in competition with Swig?
2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed?
We welcome a better opinion of name (indeed verify is not that great). This elevates ABI to API so either invokes the C compiler or reads stuff from the cache.
Can you elaborate on what "elevates ABI to API" means here?
3. The partial type specifications in C with ellipsis. What is the point? You have the C declarations somewhere anyhow, so why introduce this? The "ABI level" boasts having just C and Python to write, but those partial ellipsis-ridden declarations are hardly C.
No, you don't. Some libraries contain macros for example (like OpenSSL) where you just can't use ABI because it makes no sense. It's less common on windows where binary compatibility is important, however looking on linux, multiple stdlib declaration would use ellipsis in the man page.
It would be useful to find an actual example and discuss it concretely.
I can't seem to find one right now, but it's something like:
struct X { int public_field; ... }
which is impossible to do correctly with ctypes without exposing some sort of platform dependency that might change without warning.
Another usages are #define SQLITE_OK ... which you don't know at the time of writing (people assume those won't change and the do change).
Do you mean that the value of SQLITE_OK changed over time (now it's 0, but used to be different?) If so, then in a realistic use case, how would the API level help solve this? Eli
On Wed, Feb 27, 2013 at 5:57 PM, Eli Bendersky <eliben@gmail.com> wrote:
I read the cffi docs once again and went through some of the examples. I want to divide this to two topics.
One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs.
Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me:
1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC?
Yes. Precedent in the stdlib is really the C API. All the same rules apply (including build and ship a dll).
So would you say that the main use of the API level is provide an alternative for writing C API code to interface to C libraries. IOW, it's in competition with Swig?
the general goal is to provide alternative for writing C API by hand (so SWIG, ctypes, cython, whatever).
2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed?
We welcome a better opinion of name (indeed verify is not that great). This elevates ABI to API so either invokes the C compiler or reads stuff from the cache.
Can you elaborate on what "elevates ABI to API" means here?
3. The partial type specifications in C with ellipsis. What is the point? You have the C declarations somewhere anyhow, so why introduce this? The "ABI level" boasts having just C and Python to write, but those partial ellipsis-ridden declarations are hardly C.
No, you don't. Some libraries contain macros for example (like OpenSSL) where you just can't use ABI because it makes no sense. It's less common on windows where binary compatibility is important, however looking on linux, multiple stdlib declaration would use ellipsis in the man page.
It would be useful to find an actual example and discuss it concretely.
#define Py_INCREF(x) (x -> refcnt++) the API here is Py_INCREF, which is a macro and does not exist in the DLL. copy-pasting the implementation is not using the API, it's hoping that stuff won't change in the future (and say makes it not working when PyPy implements Py_INCREF as a function). Some POSIX things are macros in one version of glibc and functions in other and there is no change in API so your code will silently stop working.
I can't seem to find one right now, but it's something like:
struct X { int public_field; ... }
which is impossible to do correctly with ctypes without exposing some sort of platform dependency that might change without warning.
Another usages are #define SQLITE_OK ... which you don't know at the time of writing (people assume those won't change and the do change).
Do you mean that the value of SQLITE_OK changed over time (now it's 0, but used to be different?)
If so, then in a realistic use case, how would the API level help solve this?
Eli
no the API is SQLITE_OK. The actual numeric value of 0 is *not* part of the API. it might be 0 it might be not, you never know. In cffi you write: #define SQLITE_OK ... and let the compiler figure out. in ctypes you just hardcode 0. I ended up once with a scary mess of ifs because BSD Linux and OS X were declaring the same numeric constants with different values.
On Wed, Feb 27, 2013 at 7:57 AM, Eli Bendersky <eliben@gmail.com> wrote:
I read the cffi docs once again and went through some of the examples. I
want to divide this to two topics.
One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs.
Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me:
1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC?
Yes. Precedent in the stdlib is really the C API. All the same rules apply (including build and ship a dll).
So would you say that the main use of the API level is provide an alternative for writing C API code to interface to C libraries. IOW, it's in competition with Swig?
I'd hardly call it competition. The primary language I interface with is C++ and cffi appears not see that giant elephant in the room (it'd need to use clang for parsing if it were going to do that)... The goal is good, but being C only isn't super exciting to me. Would there be a goal of using cffi to replace C extension module code in the standard library with cffi based versions instead of hand written CPython C API code? If not, why not? (and what does that say about its limitations or practical it is?) -0.5 from me. My inclination is not to add this to the standard library. But even if it were to be added, it sounds like others are coming up with questions and reasons why it isn't yet ready (always the first important step to seriously considering inclusion). -gps
Hi Gregory, On Sat, Mar 2, 2013 at 8:40 AM, Gregory P. Smith <greg@krypto.org> wrote:
On Wed, Feb 27, 2013 at 7:57 AM, Eli Bendersky <eliben@gmail.com> wrote:
So would you say that the main use of the API level is provide an alternative for writing C API code to interface to C libraries. IOW, it's in competition with Swig?
I'd hardly call it competition. The primary language I interface with is C++ and cffi appears not see that giant elephant in the room
I don't think it's in competition with Swig, which does C++. There are certain workloads in which C++ is the elephant in the room; we don't address such workloads. If you want some more motivation, the initial goal was to access the large number of standard Linux/Posix libraries that are C (or have a C interface), but are too hard to access for ctypes (macros, partially-documented structure types, #define for constants, etc.). For this goal, it works great.
(it'd need to use clang for parsing if it were going to do that)...
I fear parsing is merely the tip of the iceberg when we talk about interfacing with C++. A bientôt, Armin.
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package) 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary? Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using: ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);") But that results in an error when calling verify: TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type". Ronald
On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
Because ctypes API is a mess and magic. We needed a cleaner (and much smaller) model.
2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)
Yes. pycparser and ply.
3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
It seems to work.
4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?
no idea.
Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using:
ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);")
But that results in an error when calling verify:
TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size
From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type".
Ronald
typedef ... is assumed to be a struct. Copy pasting from documentation, we might want to have a better support for that, but what you do right now is to do: ffi.cdef("const int mysize;") lib = ffi.verify("const int mysize = sizeof(THE_TYPE);") print lib.mysize which will tell you the size (and then you can pick up which float you want). I agree that some generic support for that would be cool. Cheers, fijal
On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
Because ctypes API is a mess and magic. We needed a cleaner (and much smaller) model.
The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.
2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)
Yes. pycparser and ply.
Which aren't part of the stdlib right now.
3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
It seems to work.
That's not a confidency inspiring comment :-). That said, I use a hacked up fork of pycparser to parse Apple's Cocoa headers for PyObjC and it appears to work fine for that.
4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?
no idea.
That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers. Ronald
On Wed, Feb 27, 2013 at 11:18 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
Because ctypes API is a mess and magic. We needed a cleaner (and much smaller) model.
The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.
I implemented ctypes. It is.
4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?
no idea.
That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.
Ronald
if cpython supports it and you can load it using dlopen, it does work then (it really is just building a C extension on the API level).
On 27/02/2013 9:21am, Maciej Fijalkowski wrote:
That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.
Ronald if cpython supports it and you can load it using dlopen, it does work then (it really is just building a C extension on the API level).
On Windows ctypes has CDLL for libraries using the cdecl calling convention and WINDLL for libraries with the stdcall calling convention. I can't see any similar distinction in cffi's documentation. Can cffi magically work out which calling convention to use, or are you restricted to the cdecl calling convention? -- Richard
On Wed, Feb 27, 2013 at 1:45 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
On 27/02/2013 9:21am, Maciej Fijalkowski wrote:
That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.
Ronald
if cpython supports it and you can load it using dlopen, it does work then (it really is just building a C extension on the API level).
On Windows ctypes has CDLL for libraries using the cdecl calling convention and WINDLL for libraries with the stdcall calling convention.
I can't see any similar distinction in cffi's documentation. Can cffi magically work out which calling convention to use, or are you restricted to the cdecl calling convention?
copy pasting from docs: Windows: you can’t yet specify the calling convention of callbacks. (For regular calls, the correct calling convention should be automatically inferred by the C backend.) Use an indirection, like in the example just above. I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though)
-- Richard
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
On 27 February 2013 11:53, Maciej Fijalkowski <fijall@gmail.com> wrote:
I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though)
That would be a deal-breaker for my use case of quick scripts working with the Windows API on a machine with no C compiler. I'd have to use ctypes in that case until cffi had this feature. Paul.
27 February 2013 18:24, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2013 11:53, Maciej Fijalkowski <fijall@gmail.com> wrote:
I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though)
That would be a deal-breaker for my use case of quick scripts working with the Windows API on a machine with no C compiler. I'd have to use ctypes in that case until cffi had this feature.
One other use case for the ABI level over the API level - the ABI level (no C extension) can be used across Python versions, where the API level needs a separate compiled extension per Python version. This can be a big deal on Windows at least, where users (not developers) with no compilers on their systems are common. I'm not trying to argue against cffi, just pointing out some cases that should be considered. Paul
On Wed, Feb 27, 2013 at 1:28 PM, Paul Moore <p.f.moore@gmail.com> wrote:
27 February 2013 18:24, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2013 11:53, Maciej Fijalkowski <fijall@gmail.com> wrote:
I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though)
That would be a deal-breaker for my use case of quick scripts working with the Windows API on a machine with no C compiler. I'd have to use ctypes in that case until cffi had this feature.
One other use case for the ABI level over the API level - the ABI level (no C extension) can be used across Python versions, where the API level needs a separate compiled extension per Python version. This can be a big deal on Windows at least, where users (not developers) with no compilers on their systems are common.
Is that still true for Windows even with the stable ABI and the dropping of narrow/wide Unicode builds?
On 27 February 2013 18:50, Brett Cannon <brett@python.org> wrote:
One other use case for the ABI level over the API level - the ABI level (no C extension) can be used across Python versions, where the API level needs a separate compiled extension per Python version. This can be a big deal on Windows at least, where users (not developers) with no compilers on their systems are common.
Is that still true for Windows even with the stable ABI and the dropping of narrow/wide Unicode builds?
Probably not, but I've never actually seen the stable ABI used in practice (and I don't know if cffi restricts itself to the stable ABI). I'm not sure that any toolchain (such as bdist_wininst or wheel) really supports it (in the sense that they tend to make the assumption that if there is a C extension, the code is version-specific). None of these are insurmountable problems, though. Paul.
Hi Paul, On Wed, Feb 27, 2013 at 7:24 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2013 11:53, Maciej Fijalkowski <fijall@gmail.com> wrote:
I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though)
That would be a deal-breaker for my use case of quick scripts working with the Windows API on a machine with no C compiler. I'd have to use ctypes in that case until cffi had this feature.
That's not correct: you can't indeed give the calling convention, but it is not needed for the common case. What is not supported is only Python-defined callbacks using the Windows-specific convention --- as documented, there is a workaround for that case. And, in case you wonder, this automatic detection comes from ctypes. I copied the hacked-up version of libffi for Windows from ctypes to cffi, and the logic to detect the actual calling convention and support both is there. The difference is only that in ctypes, after it did the call (always successfully), it checks the convention that was found to be used by the C function, and if it differs from the one specified by the user, then it complains. I basically just removed the "complaining" part. A bientôt, Armin.
On 27 February 2013 19:08, Armin Rigo <arigo@tunes.org> wrote:
That's not correct: you can't indeed give the calling convention, but it is not needed for the common case. What is not supported is only Python-defined callbacks using the Windows-specific convention --- as documented, there is a workaround for that case.
OK, that's cool. As I said, I really need to actually experiment with cffi - this thread has certainly made me want to do so. Paul
On 27 February 2013 19:26, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2013 19:08, Armin Rigo <arigo@tunes.org> wrote:
That's not correct: you can't indeed give the calling convention, but it is not needed for the common case. What is not supported is only Python-defined callbacks using the Windows-specific convention --- as documented, there is a workaround for that case.
OK, that's cool. As I said, I really need to actually experiment with cffi - this thread has certainly made me want to do so.
Actually, looking at the cffi documentation, I can't see how I'd translate the following very simple example of something I do quite a lot in ctypes: from ctypes import windll MessageBox = windll.User32.MessageBoxW MessageBox(0, "Hello, world!", "Title", 0) Note - I wrote this from memory, I honestly don't know without looking it up the precise argument types for MessageBoxW - and even if I did, I suspect they would all be macros from windows.h. I don't want to invoke the C compiler by using verify, and I don't see in the docs how I'd get the macro definitions any other way. If anyone could show me a cffi equivalent, I'd be very grateful. Paul
Hi Paul, On Wed, Feb 27, 2013 at 9:31 PM, Paul Moore <p.f.moore@gmail.com> wrote:
from ctypes import windll MessageBox = windll.User32.MessageBoxW MessageBox(0, "Hello, world!", "Title", 0)
You are right that it's a bit cumbersome in cffi up to and including 0.5, but in the cffi trunk all standard Windows types are included. So the general answer to your question is: we google MessageBox and copy that line from the microsoft site, and manually remove the unnecessary WINAPI and _In_opt declarations: from cffi import FFI ffi = FFI() ffi.cdef(""" int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); """) lib = ffi.dlopen("USER32.DLL") lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0) That's a slightly unfair example, because in this case it happens to work with ctypes without specifying the argtypes and the restype. I would argue that this feature of ctypes is not a good thing: it's mostly the same as saying "you only need to declare argtypes and restype if you get nonsense results or segfaults". Note for completeness that the version with verify() simply replaces "lib = ffi.dlopen("USER32.DLL")" with "lib = ffi.verify("")" (no #include necessary here). Then you cannot misdeclare anything without getting clear compile-time errors at the verify(). The compilation occurs only once (it's cached) and by writing two lines in your setup.py you can distribute binary installations, just like you do with hand-written C extension modules; so the extra burden of accessing the API level is in my opinion very small compared to its segfault-avoiding gain. But I know that either level of access can make sense in different situations. Typically, on Windows, the ABI-level works fine; as argued elsewhere, on other platforms and/or with some libraries, the API-level is definitely more suited. A bientôt, Armin.
Armin Rigo:
So the general answer to your question is: we google MessageBox and copy that line from the microsoft site, and manually remove the unnecessary WINAPI and _In_opt declarations:
Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call? Neil
Hi Neil, On Thu, Feb 28, 2013 at 12:34 AM, Neil Hodgson <nyamatongwe@me.com> wrote:
Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call?
Maybe. Feel like adding an issue to https://bitbucket.org/cffi/cffi/issues, with references? This looks like a Windows-specific extension, which means that I don't automatically know about it. A bientôt, Armin.
Armin Rigo:
Maybe. Feel like adding an issue to https://bitbucket.org/cffi/cffi/issues, with references?
OK, issue #62 added.
This looks like a Windows-specific extension, which means that I don't automatically know about it.
While SAL is Windows-specific, gcc supports some similar attributes including nonnull and sentinel. Neil
On 27 February 2013 23:18, Armin Rigo <arigo@tunes.org> wrote:
from cffi import FFI ffi = FFI() ffi.cdef(""" int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); """) lib = ffi.dlopen("USER32.DLL") lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0)
Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's not a correctness issue). The auto-conversion of strings is a huge improvement.
That's a slightly unfair example, because in this case it happens to work with ctypes without specifying the argtypes and the restype. I would argue that this feature of ctypes is not a good thing: it's mostly the same as saying "you only need to declare argtypes and restype if you get nonsense results or segfaults".
That's a bit unfair. I'd say "you only need to declare argtypes if you're dealing with things more complex than integers, strings and null pointers". Which means you're fine for a huge proportion of the Windows API. On the other hand, going to the API level and being able to use all the Windows constants without having to look them up (their values aren't documented, so googling doesn't help :-() is a huge plus. Paul.
Hi Paul, On Thu, Feb 28, 2013 at 9:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's not a correctness issue).
Indeed. I created https://bitbucket.org/cffi/cffi/issue/61/convert-0-to-a-null-pointer. In C, NULL is typically "#define NULL (void *)0", not just 0, which means that there are not 100% equivalent. But it's true that in C you can pass the constant 0 to mean "a NULL pointer argument". The fact that it's not working this way in CFFI is definitely a bug.
That's a slightly unfair example, because in this case it happens to work with ctypes without specifying the argtypes and the restype. I would argue that this feature of ctypes is not a good thing: it's mostly the same as saying "you only need to declare argtypes and restype if you get nonsense results or segfaults".
That's a bit unfair. I'd say "you only need to declare argtypes if you're dealing with things more complex than integers, strings and null pointers". Which means you're fine for a huge proportion of the Windows API.
Yes, you're right, and the 32-bit Windows platform is still important. However, it only works on 32-bit. On typical 64-bit Posix environments, if you don't declare argtypes/restype, you usually end up very quickly with confusion between "int" and "long". And I think that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, because it will pass a 32-bit 0. (It may be that it doesn't actually make a difference and works anyway, but I'm not sure.) Similarly, a function that returns a pointer (e.g. a handle on Windows) will not work without an explicit restype on any 64-bit platform I know of. "Explicit is better than implicit"... A bientôt, Armin.
On 28 February 2013 09:06, Armin Rigo <arigo@tunes.org> wrote:
And I think that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, because it will pass a 32-bit 0. (It may be that it doesn't actually make a difference and works anyway, but I'm not sure.) Similarly, a function that returns a pointer (e.g. a handle on Windows) will not work without an explicit restype on any 64-bit platform I know of. "Explicit is better than implicit"...
Ha, interesting. My test case was run on 64-bit Windows with 64-bit Python, and worked fine. But you're right, explicit is better, and I may have been luckier than I realised. And anyway my original point about whether cffi was as concise as ctypes for simple ABI-level Windows calls has been answered - yes it is, at least the next version will be. Thanks, Paul.
Le Thu, 28 Feb 2013 10:06:00 +0100, Armin Rigo <arigo@tunes.org> a écrit :
Yes, you're right, and the 32-bit Windows platform is still important. However, it only works on 32-bit. On typical 64-bit Posix environments, if you don't declare argtypes/restype, you usually end up very quickly with confusion between "int" and "long". And I think that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, because it will pass a 32-bit 0. (It may be that it doesn't actually make a difference and works anyway, but I'm not sure.) Similarly, a function that returns a pointer (e.g. a handle on Windows) will not work without an explicit restype on any 64-bit platform I know of.
Indeed, argument width is quite important :-) Also, on 64-bit Windows, long is 32-bit while size_t and pointers are 64-bit. Regards Antoine.
On Thu, Feb 28, 2013 at 10:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2013 23:18, Armin Rigo <arigo@tunes.org> wrote:
from cffi import FFI ffi = FFI() ffi.cdef(""" int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); """) lib = ffi.dlopen("USER32.DLL") lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0)
Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's not a correctness issue). The auto-conversion of strings is a huge improvement.
That's a slightly unfair example, because in this case it happens to work with ctypes without specifying the argtypes and the restype. I would argue that this feature of ctypes is not a good thing: it's mostly the same as saying "you only need to declare argtypes and restype if you get nonsense results or segfaults".
That's a bit unfair. I'd say "you only need to declare argtypes if you're dealing with things more complex than integers, strings and null pointers". Which means you're fine for a huge proportion of the Windows API.
No, if the int is of the wrong size or you pass int instead of float (but you meant float) you get nonsense or segfaults.
On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
Because ctypes API is a mess and magic. We needed a cleaner (and much smaller) model.
The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.
cffi actually does have a ctypes backend in addition to the ffi and "compile a CPython extension" backends. But the ctypes backend is guaranteed to be slow and messy because it is ctypes.
On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
It seems to work.
C itself shouldn't be *too* bad - it's trying to parse C++ that is truly insane (which is why Dave Beazley explicitly warned Maciej and Armin away from that based on his experience with SWIG). Looking at pycparsing more closely though, there may be a problem with the C preprocessing step on Windows. While on Linux it uses the system "cpp", for Windows it currently bundles a copy of the LCC preprocessor, which may pose licensing challenges (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this concern may not be applicable to cffi: given cffi's requirement for a full external C compiler to actually *create* the C extension modules during development, and the ability to retrieve everything necessary directly from the created C extension when no C compiler is available, it seems there would be no reason to ever need the bundled C preprocessor. So we simply don't ship it, and the LCC license becomes irrelevant. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Feb 27, 2013 at 2:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
It seems to work.
C itself shouldn't be *too* bad - it's trying to parse C++ that is truly insane (which is why Dave Beazley explicitly warned Maciej and Armin away from that based on his experience with SWIG).
Looking at pycparsing more closely though, there may be a problem with the C preprocessing step on Windows. While on Linux it uses the system "cpp", for Windows it currently bundles a copy of the LCC preprocessor, which may pose licensing challenges (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this concern may not be applicable to cffi: given cffi's requirement for a full external C compiler to actually *create* the C extension modules during development, and the ability to retrieve everything necessary directly from the created C extension when no C compiler is available, it seems there would be no reason to ever need the bundled C preprocessor. So we simply don't ship it, and the LCC license becomes irrelevant.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
cffi does not use preprocessor
On Tue, Feb 26, 2013 at 11:29 PM, Ronald Oussoren <ronaldoussoren@mac.com>wrote:
On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hello.
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib.
The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)
3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
pycparser has been around for more than 4 years and is pretty stable. I know that it's been used to parse Windows headers, Linux system headers and other "scary" things for a C parser to handle. It's only known problem has to do with over-creative abuses of C's context sensitivity: http://eli.thegreenplace.net/2011/05/02/the-context-sensitivity-of-c%E2%80%9... In practice it doesn't come up often, and almost never in declarations, which is the subset cffi uses pycparser for. Eli
Hi, looks like no-one's taken over the role of the Advocatus Diaboli yet. =) Maciej Fijalkowski, 26.02.2013 16:13:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release.
You say that "the API is fairly stable". What about the implementation? Will users want to install a new version next to the stdlib one in a couple of months, just because there was a bug in the parser in Python 3.4 that you still need to support because there's code that depends on it, or because there is this new feature that is required to make it work with library X, or ... ? What's the upgrade path in that case? How will you support this? What long-term guarantees do you give to users of the stdlib package? Or, in other words, will the normal fallback import for cffi look like this: try: import stdlib_cffi except ImportError: import external_cffi or will the majority of users end up prefering this order: try: import external_cffi except ImportError: import stdlib_cffi
* Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.)
Ok, so there are cases where you need a C compiler installed in order to support the API. Which means that it will be a very complicated thing for users to get working under Windows, for example, which then means that users are actually best off not using the API-support feature if they want portable code. Wouldn't it be simpler to target Windows with a binary than with dynamically compiled C code? Is there a way to translate an API description into a static ABI description for a known platform ahead of time, or do I have to implement this myself in a separate ABI code path by figuring out a suitable ABI description myself? In which cases would users choose to use the C API support? And, is this dependency explicit or can I accidentally run into the dependency on a C compiler for my code without noticing?
* We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions).
Ok, so the current status actually is that it's *not* complete, and that future versions will have to catch up in terms of C compatibility. So, why do you think it's a good time to get it into the stdlib *now*?
* We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython.
You mentioned that it's fast under PyPy and slow under CPython, though. What would be the reason to use it under CPython then? Some of the projects that are using it (you named a couple) also have equivalent (or maybe more or less so) native implementations for CPython already. Do you have any benchmarks available that compare those to their cffi versions under CPython? Is the slowdown within any reasonable bounds? Others have already mentioned the lack of C++ support. It's ok to say that you deliberately only want to support C, but it's also true that that's a substantial restriction. Stefan
Hi Stefan, On Sat, Mar 2, 2013 at 10:10 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
You say that "the API is fairly stable". What about the implementation? Will users want to install a new version next to the stdlib one in a couple of months,
I think that the implementation is fairly stable as well. The only place I can foresee some potential changes is in details like the location of temporary files, for example, which needs to be discussed (probably with people from python-dev too) as some point.
just because there was a bug in the parser in Python 3.4 that you still need to support because there's code that depends on it, or because there is this new feature that is required to make it work with library X, or ... ? What's the upgrade path in that case? How will you support this? What long-term guarantees do you give to users of the stdlib package?
I think these are general questions for any package that ends up in the stdlib. In the case of CFFI, it is now approaching a stability point. This is also because we are going to integrate it with the stdlib of PyPy soon. Bugs in the parser have not been found so far, but if there is any, we will treat it like we treat any other bug in the stdlib. For that matter, there is actually no obvious solution for the user either: he generally has to wait for the next micro release to have the bug fixed.
Or, in other words, will the normal fallback import for cffi look like this:
try: import stdlib_cffi except ImportError: import external_cffi
or will the majority of users end up prefering this order:
try: import external_cffi except ImportError: import stdlib_cffi
I would rather drop the external CFFI entirely, or keep it only to provide backports to older Python versions. I personally see no objection to call the stdlib one "cffi" too (but any other name is fine as well).
... Wouldn't it be simpler to target Windows with a binary than with dynamically compiled C code? Is there a way to translate an API description into a static ABI description for a known platform ahead of time, or do I have to implement this myself in a separate ABI code path by figuring out a suitable ABI description myself?
No, I believe that you missed this point: when you make "binary" distributions of a package with setup.py, it precompiles a library for CFFI too. So yes, you need a C compiler on machines where you develop the program, but not on machines where you install it. It's the same needs as when writing custom C extension modules by hand.
In which cases would users choose to use the C API support? And, is this dependency explicit or can I accidentally run into the dependency on a C compiler for my code without noticing?
A C compiler is clearly required: this is if and only if you call the function verify(), and pass it arguments that are not the same ones as the previous time (i.e. it's not in the cache).
* We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions).
Ok, so the current status actually is that it's *not* complete, and that future versions will have to catch up in terms of C compatibility. So, why do you think it's a good time to get it into the stdlib *now*?
To be honest I don't have a precise list of C99 constructs missing. I used to know of a few of them, but these were eventually supported. It is unlikely to get completed, or if it is, a fairly slow process should be fine --- just like a substantial portion of the stdlib, which gets occasional updates from one Python version to the next.
You mentioned that it's fast under PyPy and slow under CPython, though. What would be the reason to use it under CPython then?
The reason is just ease of use. I pretend that it takes less effort (and little C knowledge), and is less prone to bugs and leaks, to write a perfectly working prototype of a module to access a random C library. I do not pretend that you'll get the top-most performance. For a lot of cases performance doesn't matter; and when it does, on CPython, you can really write a C extension module by hand (as long as you make sure to keep around the CFFI version for use by PyPy). This is how I see it, anyway. The fact that we are busy rewriting existing native well-tested CPython extensions with CFFI --- this is really only of use for PyPy.
Others have already mentioned the lack of C++ support. It's ok to say that you deliberately only want to support C, but it's also true that that's a substantial restriction.
I agree that it's a restriction, or rather a possible extension that is not done. I don't have plans to do it myself. Please also keep in mind that we pitch CFFI as a better ctypes, not as the ultimate tool to access any foreign language. A bientôt, Armin.
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use. Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes: +-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+ I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too. +1 for cffi in the stdlib. Stefan Krah
Maybe someone from PyPy should bring this up as an official topic at the language summit to figure out the blockers (again). Or it can join regex on the list of "module discussed for addition at the language summit but never quite pushed to commitment". =) On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah <stefan@bytereef.org> wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too.
+1 for cffi in the stdlib.
Stefan Krah
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Tue, Dec 17, 2013 at 7:21 PM, Brett Cannon <brett@python.org> wrote:
Maybe someone from PyPy should bring this up as an official topic at the language summit to figure out the blockers (again). Or it can join regex on the list of "module discussed for addition at the language summit but never quite pushed to commitment". =)
we're still working on resolving discussed issues before officially proposing it for inclusion.
On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah <stefan@bytereef.org> wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too.
+1 for cffi in the stdlib.
Stefan Krah
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
On 18 Dec 2013 06:21, "Maciej Fijalkowski" <fijall@gmail.com> wrote:
On Tue, Dec 17, 2013 at 7:21 PM, Brett Cannon <brett@python.org> wrote:
Maybe someone from PyPy should bring this up as an official topic at the language summit to figure out the blockers (again). Or it can join
regex on
the list of "module discussed for addition at the language summit but never quite pushed to commitment". =)
we're still working on resolving discussed issues before officially proposing it for inclusion.
Note that there's also now a link chain from the CPython extension creation docs to cffi (and Cython) - the cross version Python Packaging User Guide now has a section on binary extensions that covers several of the alternatives to writing them by hand, while the stdlib extension writing guide has a note at the beginning pointing to that resource. Cheers, Nick.
On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah <stefan@bytereef.org>
wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth
noting
that Ubuntu does something in their Python build that we should do, too.
+1 for cffi in the stdlib.
Stefan Krah
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah <stefan@bytereef.org> wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too.
Ubuntu compiles their Python with FDO (feedback directed optimization / profile guided optimization) enabled. All distros should do this if they don't already. It's generally 20% interpreter speedup. Our makefile already supports it but it isn't the default build as it takes a long time given that it needs to compile everything twice and do a profiled benchmark run between compilations. -gps
+1 for cffi in the stdlib.
Stefan Krah
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
On Thu, Dec 19, 2013 at 3:17 AM, Gregory P. Smith <greg@krypto.org> wrote:
On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah <stefan@bytereef.org> wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too.
Ubuntu compiles their Python with FDO (feedback directed optimization / profile guided optimization) enabled. All distros should do this if they don't already. It's generally 20% interpreter speedup. Our makefile already supports it but it isn't the default build as it takes a long time given that it needs to compile everything twice and do a profiled benchmark run between compilations.
-gps
Hey Greg. We found out that this only speedups benchmarks that you tried during profiling and not others, so we disabled it for the default pypy build. Can you provide me with some more detailed study on how it speeds up interpreters in general and CPython in particular? Cheers, fijal
On Thu, Dec 19, 2013 at 2:07 AM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Thu, Dec 19, 2013 at 3:17 AM, Gregory P. Smith <greg@krypto.org> wrote:
On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah <stefan@bytereef.org>
wrote:
Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers.
I've tried cffi (admittedly only in a toy script) and find it very nice to use.
Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes:
+-------------------------------+----------+----------+---------+ | | _decimal | ctypes | cffi | +===============================+==========+==========+=========+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +-------------------------------+----------+----------+---------+ | cpython-2.7 (with-system-ffi) | n/a | 4.46s | 5.18s | +-------------------------------+----------+----------+---------+ | Ubuntu-cpython-2.7 | n/a | 3.63s | - | +-------------------------------+----------+----------+---------+ | pypy-2.2.1-linux64 | n/a | 125.9s | 0.94s | +-------------------------------+----------+----------+---------+ | pypy3-2.1-beta1-linux64 | n/a | 264.9s | 2.93s | +-------------------------------+----------+----------+---------+
I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth
noting
that Ubuntu does something in their Python build that we should do, too.
Ubuntu compiles their Python with FDO (feedback directed optimization / profile guided optimization) enabled. All distros should do this if they don't already. It's generally 20% interpreter speedup. Our makefile already supports it but it isn't the default build as it takes a long time given that it needs to compile everything twice and do a profiled benchmark run between compilations.
-gps
Hey Greg.
We found out that this only speedups benchmarks that you tried during profiling and not others, so we disabled it for the default pypy build. Can you provide me with some more detailed study on how it speeds up interpreters in general and CPython in particular?
That's a common concern for profile based builds but it turns out not to matter a whole lot which workloads you choose for the CPython interpreter to collect profiles for a FDO build. I believe ubuntu's packages just use the test suite. In our own tests at work this produced good results. Interpreter loops and other common code paths in the interpreter have a *lot* of low hanging fruit in terms of more optimal code generation. Link time optimization adds additional benefits IF you can get it working (not always easy or reliable right now as Matthias mentions in issue17781). -gps
Gregory P. Smith <greg@krypto.org> wrote:
Ubuntu compiles their Python with FDO (feedback directed optimization / profile guided optimization) enabled. All distros should do this if they don't already. It's generally 20% interpreter speedup. Our makefile already supports it but it isn't the default build as it takes a long time given that it needs to compile everything twice and do a profiled benchmark run between compilations.
Yes, I didn't know we already had `make profile-opt`. With that option the self-compiled results are nearly the same as with the Ubuntu version, the remaining difference might be due to Ubuntu's use of -flto, as Matthias suggests in http://bugs.python.org/issue17781 . Stefan Krah
participants (22)
-
Antoine Pitrou -
Armin Rigo -
Brett Cannon -
Daniel Holth -
Dirkjan Ochtman -
Donald Stufft -
Eli Bendersky -
Greg Ewing -
Gregory P. Smith -
Guido van Rossum -
Joao S. O. Bueno -
Maciej Fijalkowski -
Neil Hodgson -
Nick Coghlan -
Paul Moore -
R. David Murray -
Richard Oudkerk -
Ronald Oussoren -
Stefan Behnel -
Stefan Krah -
Terry Reedy -
Xavier Morel