The state of our copies of libffi (was: Redoing the C API?)
On Thu, Mar 3, 2016 at 12:38 PM, Brett Cannon <brett@python.org> wrote:
[...] the maintenance issue we have with ctypes (or at least that's my hang-up with it). I think we still have not figured out what code we have patched and so no one has been willing to update to a newer version of libffi. I think Zachary looked into it and got some distance but never far enough to feel comfortable with trying to update things.
Since I've been invoked, I'll try to explain our libffi situation as far as I understand it. This is just a history lesson, I don't really have an opinion on what should be done with it. I will opine that we have some seriously old unmaintained code here, and the whole libffi situation in cpython is far more complex than is ideal.
We actually have four separate copies of libffi:
Modules/_ctypes/libffi: This is a mostly-vanilla copy of libffi-3.1
(released 19May2014), lightly patched according to
Modules/_ctypes/libffi.diff. This one is used for any non-OSX posix
build that doesn't use --with-system-ffi. doko has done a pretty
good job keeping this one relatively up to date.
Modules/_ctypes/libffi_osx: This is a "lightly patched" copy from
somewhere before libffi-2.0, probably. It has barely been touched
since 2009. I've been given to understand that it has modifications
necessary to allow building fat binaries on OSX (Ned or Ronald would
know better than I), but I don't know if such modifications may have
made it upstream since pre-2.0. This one is used for all OSX builds
that don't use --with-system-ffi.
Modules/_ctypes/libffi_msvc: This is a heavily patched copy from 27January2004 and is used for all Windows builds. The patches include additional functionality, namely (IIRC) returning the number of arguments expected when calling a foreign function with the wrong number of arguments to allow ctypes to raise a ValueError instead of crashing. I'm fairly certain those modifications have not been even submitted upstream, and the intervening twelve years (and my utter lack of experience working with asm) scare me away from trying to forward-port the patches from libffi_msvc. I did attempt to use a vanilla libffi on Windows sometime in 2014/2015, but ran into the issue that it doesn't have the features ctypes wants, which made a few fairly basic tests fail.
Modules/_ctypes/libffi_arm_wince: I don't know why we even have this. Nobody has touched it since ctypes was merged into cpython in 2006.
-- Zach
Why starting many discussions on the private python-committers mailing list? Why not discussing libffi on python-dev?
Victor
On Thu, Mar 3, 2016, 16:43 Victor Stinner <victor.stinner@gmail.com> wrote:
Why starting many discussions on the private python-committers mailing list? Why not discussing libffi on python-dev?
Because they are offshoots of your email to python-committers about the C API and no one changed the to: field.
Victor
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
On Thu, Mar 3, 2016 at 10:58 PM, Zachary Ware <zachary.ware+pydev@gmail.com> wrote:
We actually have four separate copies of libffi:
Modules/_ctypes/libffi: This is a mostly-vanilla copy of libffi-3.1 (released 19May2014), lightly patched according to Modules/_ctypes/libffi.diff. This one is used for any non-OSX posix build that doesn't use
--with-system-ffi. doko has done a pretty good job keeping this one relatively up to date.
I've opened pull requests to the libffi repository for some of the changes (I haven't looked at the ones in configure{.ac} yet) in Modules/_ctypes/libffi.diff: https://github.com/atgreen/libffi/pulls/berkerpeksag
Modules/_ctypes/libffi_osx: This is a "lightly patched" copy from somewhere before libffi-2.0, probably. It has barely been touched since 2009. I've been given to understand that it has modifications necessary to allow building fat binaries on OSX (Ned or Ronald would know better than I), but I don't know if such modifications may have made it upstream since pre-2.0. This one is used for all OSX builds that don't use
--with-system-ffi.
I did a quick check and yes, some of them have already been upstreamed. For example, https://github.com/python/cpython/commit/9dc4e927f635a08ea236c9a1e5a32a99048...
Modules/_ctypes/libffi_arm_wince: I don't know why we even have this. Nobody has touched it since ctypes was merged into cpython in 2006.
I couldn't find any reference to Modules/_ctypes/libffi_arm_wince in the codebase so I guess we can now file an issue to remove it.
Thanks for the great summary, Zachary :)
--Berker
On 03.03.2016 21:58, Zachary Ware wrote:
On Thu, Mar 3, 2016 at 12:38 PM, Brett Cannon <brett@python.org> wrote:
[...] the maintenance issue we have with ctypes (or at least that's my hang-up with it). I think we still have not figured out what code we have patched and so no one has been willing to update to a newer version of libffi. I think Zachary looked into it and got some distance but never far enough to feel comfortable with trying to update things.
Since I've been invoked, I'll try to explain our libffi situation as far as I understand it. This is just a history lesson, I don't really have an opinion on what should be done with it. I will opine that we have some seriously old unmaintained code here, and the whole libffi situation in cpython is far more complex than is ideal.
We actually have four separate copies of libffi:
Modules/_ctypes/libffi: This is a mostly-vanilla copy of libffi-3.1 (released 19May2014), lightly patched according to Modules/_ctypes/libffi.diff. This one is used for any non-OSX posix build that doesn't use
--with-system-ffi. doko has done a pretty good job keeping this one relatively up to date.
when trying to update these extra copies, I was always told that upstream was wrong/not ready, etc ... So I don't care that much about the copies. The explicit diff was intended to document the explicit and wanted changes.
Now, the last libffi release 3.2.1 is more than a year old, and getting outdated for some architectures. Upstream currently doesn't react, and the libffi copy within GCC is ahead with many changes. My plan was to update libffi with a 3.3 release when it comes out, but I don't know when this will happen.
Matthias
On 03 Mar 2016, at 21:58, Zachary Ware <zachary.ware+pydev@gmail.com> wrote:
Modules/_ctypes/libffi_osx: This is a "lightly patched" copy from somewhere before libffi-2.0, probably. It has barely been touched since 2009. I've been given to understand that it has modifications necessary to allow building fat binaries on OSX (Ned or Ronald would know better than I), but I don't know if such modifications may have made it upstream since pre-2.0. This one is used for all OSX builds that don't use
--with-system-ffi.
libffi_osx is a copy of the variant of libffi used by PyObjC, which was forked from libffi a long while ago (at the latest around the time OSX started supporting Intel processors, and probably earlier).
I don’t know if upstream libffi is good enough these days, this fork not only contains patches to make it easier to build fat binaries, but also contains a number of bug fixes that weren’t upstream at the time (mostly needed to support Apple’s interpretation of the x86_64 calling conventions in clang).
Ronald
participants (6)
-
Berker Peksağ -
Brett Cannon -
Matthias Klose -
Ronald Oussoren -
Victor Stinner -
Zachary Ware