libffi embedded in CPython
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython? Cheers, fijal
Maciej Fijalkowski wrote:
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython?
We use it for ctypes, so there's certainly still a need. Are you asking whether we need a fork of it as opposed to treating it like an external (like OpenSSL)?
Cheers, fijal
On Thu, Dec 18, 2014 at 9:17 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Maciej Fijalkowski wrote:
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython?
We use it for ctypes, so there's certainly still a need. Are you asking whether we need a fork of it as opposed to treating it like an external (like OpenSSL)?
yes (why is there a copy of libffi in the cpython source). And I'm asking not why it landed there, but why it is still there
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython?
It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer.
well, the problem is essentially that libffi gets patched (e.g. for ARM) and it does not make it's way to CPython quickly. This is unlikely to be a security issue (for a variety of reasons, including ctypes), but it's still an issue I think. Segfaults related to e.g. stack alignment are hard to debug On Thu, Dec 18, 2014 at 9:30 PM, Benjamin Peterson <benjamin@python.org> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
After reading this http://bugs.python.org/issue23085 and remembering struggling having our own patches into cpython's libffi (but not into libffi itself), I wonder, is there any reason any more for libffi being included in CPython?
It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer.
On Thu, Dec 18, 2014, at 14:50, Maciej Fijalkowski wrote:
well, the problem is essentially that libffi gets patched (e.g. for ARM) and it does not make it's way to CPython quickly. This is unlikely to be a security issue (for a variety of reasons, including ctypes), but it's still an issue I think. Segfaults related to e.g. stack alignment are hard to debug
Certainly it's a suboptimal situation, but resolving it requires someone to figure out whether we still need/want whatever patches are in there.
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
[And why a fork, instead of just treating it as an external dependency] Benjamin Peterson responded:
It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer.
Are any of the following false? (1) Ideally, we would treat it as an external dependency. (2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC. (3) Upstream libffi maintenance has picked back up. (4) Alas, that means the switch merge would not be trivial. (5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is "safe", so long as our integration doesn't break?] (6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes. (7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins. (8) The need for future support makes this a bad candidate for "patches wanted"/"bug bounty"/GSoC. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
On Thu, Dec 18, 2014, at 15:36, Jim J. Jewett wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
[And why a fork, instead of just treating it as an external dependency]
Benjamin Peterson responded:
It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer.
Are any of the following false?
(1) Ideally, we would treat it as an external dependency.
(2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC.
(3) Upstream libffi maintenance has picked back up.
(4) Alas, that means the switch merge would not be trivial.
(5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is "safe", so long as our integration doesn't break?]
(6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes.
(7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins.
(8) The need for future support makes this a bad candidate for "patches wanted"/"bug bounty"/GSoC.
Sounds about right.
On Thu, Dec 18, 2014 at 10:36 PM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
[And why a fork, instead of just treating it as an external dependency]
Benjamin Peterson responded:
It has some sort of Windows related patches. No one seems to know whether they're still needed for newer libffi. Unfortunately, ctypes doesn't currently have a maintainer.
Are any of the following false?
(1) Ideally, we would treat it as an external dependency.
(2) At one point, it was intentionally forked to get in needed patches, including at least some for 64 bit windows with MSVC.
(3) Upstream libffi maintenance has picked back up.
(4) Alas, that means the switch merge would not be trivial.
(5) In theory, we could now switch to the external version. [In particular, does libffi have a release policy such that we could assume the newest released version is "safe", so long as our integration doesn't break?]
(6) By its very nature, libffi changes are risky and undertested. At the moment, that is also true of its primary user, ctypes.
(7) So a switch is OK in theory, but someone has to do the non-trivial testing and merging, and agree to support both libffi and and ctypes in the future. Otherwise, stable wins.
(8) The need for future support makes this a bad candidate for "patches wanted"/"bug bounty"/GSoC.
-jJ
I would like to add that "not doing anything" is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython).
On 19 December 2014 at 08:26, Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to add that "not doing anything" is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython).
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible. Paul
On 19.12.2014 10:52, Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible.
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff . The file *should* be up to date. Christian
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ...
Christian Heimes wrote:
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff
That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835. Besides http://bugs.python.org/issue23085, there is at least http://bugs.python.org/issue22733 http://bugs.python.org/issue20160 http://bugs.python.org/issue11835 which sort of drives home the point that making sure we have a good merge isn't trivial, and this isn't an area where we should just assume that tests will catch everything. I don't think it is just a quicky waiting on permission. I've no doubt that upstream libffi is better in many ways, but those are ways people have already learned to live with. That said, I haven't seen any objections in principle, except perhaps from Steve Dower in the issues. (I *think* he was just saying "not worth the time to me", but it was ambiguous.) I do believe that Christian or Maciej *could* sort things out well enough; I have no insight into whether they have (or someone else has) the time to actually do so. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ...
Christian Heimes wrote:
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ ctypes/libffi.diff
That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835.
Besides http://bugs.python.org/issue23085, there is at least http://bugs.python.org/issue22733 http://bugs.python.org/issue20160 http://bugs.python.org/issue11835
which sort of drives home the point that making sure we have a good merge isn't trivial, and this isn't an area where we should just assume that tests will catch everything. I don't think it is just a quicky waiting on permission.
I've no doubt that upstream libffi is better in many ways, but those are ways people have already learned to live with.
That said, I haven't seen any objections in principle, except perhaps from Steve Dower in the issues. (I *think* he was just saying "not worth the time to me", but it was ambiguous.)
I do believe that Christian or Maciej *could* sort things out well enough; I have no insight into whether they have (or someone else has) the time to actually do so.
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. And I'm not saying we need to remove it in Python 3.6 or something. But I think it would be wise to deprecate the module to promote people to use third-party solutions to interfacing with C code and to eventually remove the maintenance burden at some point when we clear out all of our deprecated cruft (I call that Python 4, you can call it "some day way in the future" if you prefer).
On Wed, 11 Mar 2015 17:27:58 +0000 Brett Cannon <brett@python.org> wrote:
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous.
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module.
This is gratuitous.
We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with.
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes.
I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Regards Antoine.
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 11 Mar 2015 17:27:58 +0000 Brett Cannon <brett@python.org> wrote:
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous.
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned)
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module.
This is gratuitous.
I'm +1 on deprecating ctypes
We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with.
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
You mean like Fortran? Or what precisely?
I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations.
Our main motivation for "safer" is "comes with less magic and less gotchas", which also means "does less stuff". It's also smaller.
On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned)
Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation.
We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with.
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
You mean like Fortran? Or what precisely?
Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi "simple" while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine.
On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
You mean like Fortran? Or what precisely?
Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction.
For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration: https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 Regards Antoine.
On Wed, Mar 11, 2015 at 8:17 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
You mean like Fortran? Or what precisely?
Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction.
For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration:
https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937
Regards
Antoine.
It might be a matter of taste, but I don't find declaring C functions any more awkward than using strange interface that ctypes comes with. the equivalent in cffi would be ffi.cast("double (*)()", x)
Hi
It might be a matter of taste, but I don't find declaring C functions any more awkward than using strange interface that ctypes comes with. the equivalent in cffi would be ffi.cast("double (*)()", x)
- Could you please elaborate on "using strange interface"? - Is there an easy way to convert the current code using ctypes to cffi? Thanks in advance! francis
On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned)
Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation.
My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler)
We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with.
But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations.
You mean like Fortran? Or what precisely?
Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction.
Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears.
As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it.
And calling cffi "simple" while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation.
My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler)
So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. Regards Antoine.
On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation.
My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler)
So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is.
So, really, I don't see how switching from ctypes to cffi solves any of this.
You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem.
On 11 March 2015 at 22:33, Maciej Fijalkowski <fijall@gmail.com> wrote:
You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem.
OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a "system" libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. I don't see how anyone can say "yes, we'll do it" without seeing evidence that it'll work. But equally, I don't think there's any reason to say it absolutely wouldn't be accepted regardless of evidence. So why not prepare a patch for 3.6 (I assume it's too late to make such a big change for 3.5) and we'll see what happens. Or even better, prepare a test build of 3.5 or even 3.4 that switches libffi - people can replace an existing install (I'd be willing to) and test it in live situations. But unless someone provides a patch, the status quo will win. At least until an actual bug that affects live code forces the issue. Paul
On Wed, 11 Mar 2015 22:55:33 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 22:33, Maciej Fijalkowski <fijall@gmail.com> wrote:
You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem.
OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes.
I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a "system" libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered.
Agreed. We have enough tests and enough buildbots that it can be attempted. (besides, the only more-or-less officially supported platforms are Linux, Windows, OS X; we also try to not to be too broken on the BSDs, but that's it) Regards Antoine.
On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" <fijall@gmail.com> wrote:
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis@pitrou.net>
wrote:
On Wed, 11 Mar 2015 17:27:58 +0000 Brett Cannon <brett@python.org> wrote:
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that
isn't
being properly maintained seems to me to make it even more dangerous.
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned)
Are there tests for this?
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module.
This is gratuitous.
I'm +1 on deprecating ctypes
-1. These docs are helpful for understanding the pros and cons of different CPython <-> C interfaces. https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_wit... (https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses documenting CFFI here as well)
On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" <fijall@gmail.com> wrote:
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 11 Mar 2015 17:27:58 +0000 Brett Cannon <brett@python.org> wrote:
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous.
Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change.
And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned)
Are there tests for this?
What do you mean? The usual failure mode is "will segfault every now and again if the moon is in the right position" (e.g. the stack alignment thing only happens if the underlaying function uses certain SSE instructions that compilers emit these days in certain circumstances)
On 11 March 2015 at 17:27, Brett Cannon <brett@python.org> wrote:
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes.
-1. On Windows a huge amount of code uses ctypes to interface with the Windows API. Many of those projects transitioned from pywin32 because that is a big dependency, and a binary build. Not having in-core access to the Windows API would be a huge problem for many projects. Who would rewrite projects like pyreadline or colorama, for example? How would pip (which has to vendor everything and can't include binary dependencies because of its nature) locate the correct windows config file folders without ctypes in core? Paul
Le 11 mars 2015 18:29, "Brett Cannon" <brett@python.org> a écrit :
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module.
In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? If you want to move forward, you should help to integrate cffi into the stdlib. What's the status? Victor
On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Le 11 mars 2015 18:29, "Brett Cannon" <brett@python.org> a écrit :
I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module.
In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-)
Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib.
Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler.
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( Paul [1] There may be Unix users of ctypes, but AFAIK, the vast majority of use cases are on Windows.
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-(
You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around.
On Wed, 11 Mar 2015 22:20:58 +0000 Brett Cannon <brett@python.org> wrote:
You're ignoring that it's not maintained,
According to which metric/criterion? changeset: 94932:86c9ef950288 user: Steve Dower <steve.dower@microsoft.com> date: Tue Mar 10 09:56:38 2015 -0700 summary: Issue #23606: Disable ctypes.util.find_library("c") on Windows so tests are skipped while we figure out how best to approach the CRT change changeset: 94756:7c6e3358221a user: Serhiy Storchaka <storchaka@gmail.com> date: Thu Feb 26 15:27:57 2015 +0200 summary: Silenced minor GCC warnings. changeset: 94653:a84ae2ccd220 user: Serhiy Storchaka <storchaka@gmail.com> date: Mon Feb 16 20:52:17 2015 +0200 summary: Issue #23450: Fixed possible integer overflows. [etc.]
Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out.
Well, who knows? How about you enlighten us about that? And why do you think cffi, which *also* requires libffi, would be better served in that regard? Regards Antoine.
On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon <brett@python.org> wrote:
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-(
You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around.
It's a bit worse than that. Each time someone wants to touch the code (e.g. push back the upstream libffi fixes), there is "we need to review it, but there is noone to do it", "noone knows how it works, don't touch it" kind of feedback, which leads to disincentives to potential maintainers. I would be likely willing to rip off the libffi from CPython as it is for example (and just use the upstream one)
Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem. On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett@python.org> wrote:
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-(
You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Wed, Mar 11, 2015 at 8:20 PM Guido van Rossum <guido@python.org> wrote:
Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem.
I mostly proposed deprecating ctypes because we were not keeping up with libffi upstream. If we solve the latter I'm not bothered enough to personally pursue the former. -Brett
On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett@python.org> wrote:
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-(
You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Thu, Mar 12, 2015 at 1:20 AM, Guido van Rossum <guido@python.org> wrote:
Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem.
ctypes and cffi do actually solve the same problem, just not in the same way.
On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett@python.org> wrote:
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall@gmail.com> wrote:
Is it possible to use cffi without a C compiler/headers as easily than ctypes?
yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler
So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-(
You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/thomas%40python.org
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On 03/11/2015 06:27 PM, Brett Cannon wrote:
On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ...
Christian Heimes wrote:
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ ctypes/libffi.diff
That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835.
Besides http://bugs.python.org/issue23085, there is at least http://bugs.python.org/issue22733 http://bugs.python.org/issue20160 http://bugs.python.org/issue11835
which sort of drives home the point that making sure we have a good merge isn't trivial, and this isn't an area where we should just assume that tests will catch everything. I don't think it is just a quicky waiting on permission.
I've no doubt that upstream libffi is better in many ways, but those are ways people have already learned to live with.
That said, I haven't seen any objections in principle, except perhaps from Steve Dower in the issues. (I *think* he was just saying "not worth the time to me", but it was ambiguous.)
I do believe that Christian or Maciej *could* sort things out well enough; I have no insight into whether they have (or someone else has) the time to actually do so.
Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi.
my plan was to merge the next libffi release, something along the lines which is released with GCC 5. But this release isn't yet done.
I started this message about 3 months ago; at this point I'm just getting it posted so it stops rotting in my Drafts folder. On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ...
Christian Heimes wrote:
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff
That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835.
I took a little bit of time to look at this a while ago. Something that quickly became clear that I'm not sure everybody is on the same page about (I, for one, had no idea previously) is that libffi for Windows and libffi for everything else are entirely separate beasts: the Windows build pulls all of its libffi source from Modules/_ctypes/libffi_msvc and doesn't even touch Modules/_ctypes/libffi (which is libffi for nearly everything else). Most of the files in libffi_msvc haven't been touched in about 4 years, except for a couple fixes by Steve Dower recently. README.ctypes in that folder states that the files were taken from the libffi CVS tree in 2004 and there have been some custom modifications since then, so there is really very little resemblance between libffi_msvc and current libffi. There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly amazed if Python itself built on any version of Windows CE; personally I would be for ripping out any supposed support that we have for it (though if somebody is actually using it and it works, it's not hurting anything and should stay). That's a completely different issue, though. I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. As for the lightly-patched version of libffi that we keep around for Linux et. al., looking at the diff file I don't see that there's a whole lot of reason to keep it around. -- Zach
On Thu, Mar 12, 2015 at 10:39 AM Zachary Ware <zachary.ware+pydev@gmail.com> wrote:
I started this message about 3 months ago; at this point I'm just getting it posted so it stops rotting in my Drafts folder.
Thanks for looking into this!
On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
... http://bugs.python.org/issue23085 ... is there any reason any more for libffi being included in CPython?
Paul Moore wrote:
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version ...
Christian Heimes wrote:
That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ ctypes/libffi.diff
That (200+ lines) doesn't seem to have all the C changes, such as the win64 sizeof changes from issue 11835.
I took a little bit of time to look at this a while ago. Something that quickly became clear that I'm not sure everybody is on the same page about (I, for one, had no idea previously) is that libffi for Windows and libffi for everything else are entirely separate beasts: the Windows build pulls all of its libffi source from Modules/_ctypes/libffi_msvc and doesn't even touch Modules/_ctypes/libffi (which is libffi for nearly everything else). Most of the files in libffi_msvc haven't been touched in about 4 years, except for a couple fixes by Steve Dower recently. README.ctypes in that folder states that the files were taken from the libffi CVS tree in 2004 and there have been some custom modifications since then, so there is really very little resemblance between libffi_msvc and current libffi.
That's what I was afraid of.
There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly amazed if Python itself built on any version of Windows CE; personally I would be for ripping out any supposed support that we have for it (though if somebody is actually using it and it works, it's not hurting anything and should stay). That's a completely different issue, though.
I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken.
So it seems possible to use upstream libffi but will require some work.
As for the lightly-patched version of libffi that we keep around for Linux et. al., looking at the diff file I don't see that there's a whole lot of reason to keep it around.
For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows?
On 12 March 2015 at 17:26, Brett Cannon <brett@python.org> wrote:
I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken.
So it seems possible to use upstream libffi but will require some work.
I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help. Paul.
On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help.
I'll send you everything I've got next chance I have, but I don't know when that will be. -- Zach
On 12 March 2015 at 18:54, Zachary Ware <zachary.ware+pydev@gmail.com> wrote:
On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help.
I'll send you everything I've got next chance I have, but I don't know when that will be.
Thanks. No rush :-) Paul
On 12 March 2015 at 17:44, Paul Moore <p.f.moore@gmail.com> wrote:
On 12 March 2015 at 17:26, Brett Cannon <brett@python.org> wrote:
I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken.
So it seems possible to use upstream libffi but will require some work.
I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help.
One thing that seems to be an issue. On Windows, ctypes detects if the FFI call used the wrong number of arguments off the stack, and raises a ValueError if it does. The tests rely on that behaviour. But it's based on ffi_call() returning a value, which upstream libffi doesn't do. As far as I can tell (not that the libffi docs are exactly comprehensive...) there's no way of getting that information from upstream libffi. What does Unix ctypes do when faced with a call being made with the wrong number of arguments? On Windows, using upstream libffi and omitting the existing check, it seems to crash the Python process, which obviously isn't good. But the test that fails is Windows-specific, and short of going through all the tests looking for one that checks passing the wrong number of arguments and isn't platform-specific, I don't know how Unix handles this. Can anyone on Unix tell me if a ctypes call with the wrong number of arguments returns ValueError on Unix? Something like strcmp() (with no args) should do as a test, I guess... If there's a way Unix handles this, I can see about replicating it on Windows. But if there isn't, I fear we could always need a patched libffi to maintain the interface we currently have... Thanks, Paul
On Tue, Mar 24, 2015 at 11:31 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 12 March 2015 at 17:44, Paul Moore <p.f.moore@gmail.com> wrote:
On 12 March 2015 at 17:26, Brett Cannon <brett@python.org> wrote:
I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken.
So it seems possible to use upstream libffi but will require some work.
I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help.
One thing that seems to be an issue. On Windows, ctypes detects if the FFI call used the wrong number of arguments off the stack, and raises a ValueError if it does. The tests rely on that behaviour. But it's based on ffi_call() returning a value, which upstream libffi doesn't do. As far as I can tell (not that the libffi docs are exactly comprehensive...) there's no way of getting that information from upstream libffi.
What does Unix ctypes do when faced with a call being made with the wrong number of arguments? On Windows, using upstream libffi and omitting the existing check, it seems to crash the Python process, which obviously isn't good. But the test that fails is Windows-specific, and short of going through all the tests looking for one that checks passing the wrong number of arguments and isn't platform-specific, I don't know how Unix handles this.
Can anyone on Unix tell me if a ctypes call with the wrong number of arguments returns ValueError on Unix? Something like strcmp() (with no args) should do as a test, I guess...
If there's a way Unix handles this, I can see about replicating it on Windows. But if there isn't, I fear we could always need a patched libffi to maintain the interface we currently have...
Thanks, Paul
Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we?
On 25 March 2015 at 08:05, Maciej Fijalkowski <fijall@gmail.com> wrote:
Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we?
Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so. Thanks for checking for me. Paul
On Wed, 25 Mar 2015 08:15:18 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 25 March 2015 at 08:05, Maciej Fijalkowski <fijall@gmail.com> wrote:
Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we?
Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so.
I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Regards Antoine.
On 25 March 2015 at 09:09, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature.
Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Thanks, Paul Paul
On Wed, 25 Mar 2015 09:22:01 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 25 March 2015 at 09:09, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature.
Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly?
IMHO yes. Of course it's still a behaviour change, but relying on it sounds seriously broken.
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Zachary or Steve could probably answer you on that one. Regards Antoine.
On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 25 March 2015 at 09:09, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature.
Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly?
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. -- Zach On a phone
Zachary Ware wrote:
On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to.
It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. Cheers, Steve
On 26 March 2015 at 01:57, Steve Dower <Steve.Dower@microsoft.com> wrote:
Zachary Ware wrote:
On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to.
It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :)
As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system.
It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mar 25, 2015 9:28 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 26 March 2015 at 01:57, Steve Dower <Steve.Dower@microsoft.com> wrote:
Zachary Ware wrote:
On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Its not extremely clear how it's "supposed to be" done; look at the
between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to.
It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :)
As well as the configure output, we sometimes include very light
differences patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system.
It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements.
I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that. -- Zach On a phone
On 26 March 2015 at 12:40, Zachary Ware <zachary.ware+pydev@gmail.com> wrote:
On Mar 25, 2015 9:28 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 26 March 2015 at 01:57, Steve Dower <Steve.Dower@microsoft.com> wrote:
Zachary Ware wrote:
On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained.
Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to.
It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :)
As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system.
It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements.
I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that.
That sounds plenty soon enough to me - my own list of "I should really do something about that some day" items stretches back years, as the issue tracker occasionally reminds me :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
In article <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>,
For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows?
Ronald (in http://bugs.python.org/issue23534): "On OSX the internal copy of libffi that's used is based on the one in PyObjC, which in turn is based on the version of libffi on opensource.apple.com (IIRC with some small patches that fix minor issues found by the PyObjC testsuite)." -- Ned Deily, nad@acm.org
On Thu, Mar 12, 2015 at 8:35 PM, Ned Deily <nad@acm.org> wrote:
In article <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>,
For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows?
Ronald (in http://bugs.python.org/issue23534): "On OSX the internal copy of libffi that's used is based on the one in PyObjC, which in turn is based on the version of libffi on opensource.apple.com (IIRC with some small patches that fix minor issues found by the PyObjC testsuite)."
-- Ned Deily, nad@acm.org
From pypy experience, libffi installed on OS X tends to just work (we never had any issues with those)
On Thu, Mar 12, 2015 at 6:26 PM, Brett Cannon <brett@python.org> wrote:
For UNIX OSs we could probably rely on the system libffi then.
Yes, it's possible to use the system libffi -- that's what most linux distros already do -- but only if you use dynamic linking. If you want to statically link libffi (useful when embedding, making installations that are a little more portable across systems, or when you're a company like Google that wants to statically link all the things) it's a little more confusing. (The reason Brett started this thread is that I mentioned these issues in an internal Google meeting that he happened to be in ;-P) _ctypes currently contains a bunch of duplicate definitions of the ffi_type_* structs (in Modules/_ctypes/cfield.c.) It also contains a separate implementation of ffi_closure_alloc and ffi_closure_free (in Modules/_ctypes/malloc_closure.c) which is only used on OSX. I guess those redefinitions are to cope with (much) older libffis, but that's just a guess. The redefinitions aren't a problem when using dynamic linking, as long as the new definitions aren't incompatible, but if libffi were to change the ABI of the ffi_type structs, yuck. It can be a problem when statically linking, depending on how you link -- and the only reason it works with the bundled libffi is that ctypes *does not build libffi the normal way*. Instead it runs libffi's configure and then selectively builds a few source files from it, using distutils. It would be a really, really good idea to clean up that mess if we're not going to bundle libffi anymore :-P Screaming-and-stabbing-at-setup.py'ly y'rs, -- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Fri, 19 Dec 2014 09:52:26 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 19 December 2014 at 08:26, Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to add that "not doing anything" is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython).
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible.
Another strategy is to dump our private fork, link with upstream instead, and see what breaks. Presumably, our test suite should be able to catch some (most?) of that breakage. Regards Antoine.
On 19 December 2014 at 23:01, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Fri, 19 Dec 2014 09:52:26 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 19 December 2014 at 08:26, Maciej Fijalkowski <fijall@gmail.com> wrote:
I would like to add that "not doing anything" is not a good strategy either, because you accumulate bugs that get fixed upstream (I'm pretty sure all the problems from cpython got fixed in upstream libffi, but not all libffi fixes made it to cpython).
Probably the easiest way of moving this forward would be for someone to identify the CPython-specific patches in the current version, and check if they are addressed in the latest libffi version. They haven't been applied as they are, I gather, but maybe equivalent fixes have been made. I've no idea how easy that would be (presumably not trivial, or someone would already have done it). If the patches aren't needed any more, upgrading becomes a lot more plausible.
Another strategy is to dump our private fork, link with upstream instead, and see what breaks. Presumably, our test suite should be able to catch some (most?) of that breakage.
And if we're going to do something like that for 3.5, now's the time, since we still have a lot of lead time on the 3.5 release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (18)
-
Antoine Pitrou -
Benjamin Peterson -
Brett Cannon -
Christian Heimes -
francis -
Guido van Rossum -
Jim J. Jewett -
Maciej Fijalkowski -
Matthias Klose -
Ned Deily -
Nick Coghlan -
Paul Moore -
Steve Dower -
Thomas Wouters -
Victor Stinner -
Wes Turner -
Zachary Ware -
Zachary Ware