Re: [pypy-dev] pypy-dev Digest, Vol 13, Issue 21
On 03:11 pm, pypy-dev-request@python.org wrote:
Hi all,
Fijal and me would like to raise interest among various groups of people about building a better ctypes replacement for Python.
The general background first, at least as far as we know it. People generally agree that CPython extension modules are not extremely hard to write, but also not the ultimate solution in the matter of interfacing with C code. Obviously you need to know and write C code, and learn the large CPython C API. Moreover a CPython extension module is rather specific to CPython or even a particular version of it. Even if you are lucky and your CPython extensions work on all 2.x versions, it won't work unmodified on Python 3.x, and be slow (or not work at all) on PyPy. (I'm leaving Jython and IronPython out of the picture here but it's very far from ideal for them too.)
The working alternatives nowadays are Cython and ctypes. For Cython you have to learn a whole new language; we can argue infinitely about preferences, but from my point of view it's mostly just a nicer way to write CPython C extensions using --- instead of C --- a custom language. About ctypes, I would say that the fact that its use remains marginal is enough to show that its mission kind of failed. Its problems are more than one, but most importantly the level at which you describe the information is slightly too low - it's ABI-level instead of API-level. It does not make a difference for some libraries, but for some it does make a huge difference, like when you use macros. We would like something that can be potentially used to wrap almost all C libraries.
So we would like to propose something (a bit more publically than the two subprojects of PyPy attempting to do that --- did you know about their existence?). The goal would be to come up with a better-designed project. "Better designed" in my vocabulary implies "simpler". I'm starting in this mailing list but the idea is to try to move it in its own project, and to develop at the same time at least a CPython 2.7 and 3.x and PyPy implementation.
The simplest FFI we know of for a high-level language is LuaJIT's FFI. If you are interested, read the four pages starting at http://luajit.org/ext_ffi.html . A crucial design decision was to avoid making use of tons of objects representing the C types; instead, the user mostly specifies C types as plain strings, simplifying the basic API a lot.
Hi Armin, I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work. C is C, whether you hide the representation of its structure in a string or whether you expose that representation as objects with useful, easy to use APIs. Jean-Paul
On Tue, May 15, 2012 at 9:22 PM, <exarkun@twistedmatrix.com> wrote:
On 03:11 pm, pypy-dev-request@python.org wrote:
Hi all,
Fijal and me would like to raise interest among various groups of people about building a better ctypes replacement for Python.
The general background first, at least as far as we know it. People generally agree that CPython extension modules are not extremely hard to write, but also not the ultimate solution in the matter of interfacing with C code. Obviously you need to know and write C code, and learn the large CPython C API. Moreover a CPython extension module is rather specific to CPython or even a particular version of it. Even if you are lucky and your CPython extensions work on all 2.x versions, it won't work unmodified on Python 3.x, and be slow (or not work at all) on PyPy. (I'm leaving Jython and IronPython out of the picture here but it's very far from ideal for them too.)
The working alternatives nowadays are Cython and ctypes. For Cython you have to learn a whole new language; we can argue infinitely about preferences, but from my point of view it's mostly just a nicer way to write CPython C extensions using --- instead of C --- a custom language. About ctypes, I would say that the fact that its use remains marginal is enough to show that its mission kind of failed. Its problems are more than one, but most importantly the level at which you describe the information is slightly too low - it's ABI-level instead of API-level. It does not make a difference for some libraries, but for some it does make a huge difference, like when you use macros. We would like something that can be potentially used to wrap almost all C libraries.
So we would like to propose something (a bit more publically than the two subprojects of PyPy attempting to do that --- did you know about their existence?). The goal would be to come up with a better-designed project. "Better designed" in my vocabulary implies "simpler". I'm starting in this mailing list but the idea is to try to move it in its own project, and to develop at the same time at least a CPython 2.7 and 3.x and PyPy implementation.
The simplest FFI we know of for a high-level language is LuaJIT's FFI. If you are interested, read the four pages starting at http://luajit.org/ext_ffi.html . A crucial design decision was to avoid making use of tons of objects representing the C types; instead, the user mostly specifies C types as plain strings, simplifying the basic API a lot.
Hi Armin,
I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work.
C is C, whether you hide the representation of its structure in a string or whether you expose that representation as objects with useful, easy to use APIs.
Jean-Paul ______________________________**_________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
The point is that using strings you can use (a subset of) C syntax. That means that you need to learn only one (and limitations) and not two ways to express stuff. I guess. Cheers, fijal
Maciej Fijalkowski, 15.05.2012 22:01:
On Tue, May 15, 2012 at 9:22 PM, <exarkun@twistedmatrix.com> wrote:
I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work.
C is C, whether you hide the representation of its structure in a string or whether you expose that representation as objects with useful, easy to use APIs.
The point is that using strings you can use (a subset of) C syntax. That means that you need to learn only one (and limitations) and not two ways to express stuff. I guess.
I'm with Jean-Paul here. Moving code into strings just breaks editors, specifically formatting and syntax highlighting, and requires special support in IDEs in order to look up the declarations. It also blurs the line between output, documentation and pure code internals and makes code generally harder to analyse statically (not only for compilation). I don't think there's anything wrong with requiring declarations for external C code to live in external modules. It may seem like overhead for very small C APIs, but in most cases, even the initially small ones will grow large enough over time to require a suitable code modularisation on user side. Even just a struct declaration (including documentation) tends to take so much space that it's best kept out of the main source code. So why not go all the way to move declarations into importable modules all by themselves and do the auto-wrapping (or at least the necessary preprocessing) in an import hook? That allows these files to use any kind of special syntax, including plain C and C++ (with which the Cython declaration syntax is mostly identical but easier to read and parse, BTW, so nothing new to learn here). It also makes them available to code outside of a single module so that they can be reused and even shipped as packages in their own right through PyPI. Some people have already started doing that with Cython declarations, e.g. https://github.com/twiecki/CythonGSL I think there's value in that. I'm not saying that the syntax necessarily has to be exactly that of Cython. We may eventually settle on a common declaration syntax that includes special macro declarations and maybe even additional ABI information. However, I think that Cython's declaration syntax has at least largely proven its real world applicability, so something entirely new would have to stand up against that. Also note that most major IDEs already support this syntax, specifically PyCharm, PyDev and WingIDE. Stefan
Hi Stefan, On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
I'm with Jean-Paul here.
...for a reason that might be unrelated to Jean-Paul's original complain: it's not like Cython gives first-class object manipulation on the result of parsing its .pyx files... well, too bad but Python is not Lisp :-)
I don't think there's anything wrong with requiring declarations for external C code to live in external modules.
I'm not against the idea of recognizing also Cython syntax extensions in the declarations, and indeed external files are a better idea than big inlined strings then. But I would also like to try out different paths. In one of them, these inline declarations would contain mostly just "#include <foo.h>" and the real header would be processed with the help of the C compiler. A bientôt, Armin.
2012/5/16 Armin Rigo <arigo@tunes.org>
I'm not against the idea of recognizing also Cython syntax extensions in the declarations, and indeed external files are a better idea than big inlined strings then. But I would also like to try out different paths. In one of them, these inline declarations would contain mostly just "#include <foo.h>" and the real header would be processed with the help of the C compiler.
Another incarnation of SWIG? -- Amaury Forgeot d'Arc
Hi Amaury, On Wed, May 16, 2012 at 9:58 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Another incarnation of SWIG?
...I suppose you can say so a priori. The difference is that we won't try to perform conversions for the user. Instead he gets his instances of C types as real Python objects and, if needed, needs to write conversions directly in Python. A bientôt, Armin.
Is there in the plans a Jit friendly callback mechanism (from C to pypy) for the new API? IMHO, right now callback performance in pypy is very bad (2x to 12x slower then CPython), and there is no way to do it efficiently apart from going the RPython way and retranslating all of pypy. l.
Armin Rigo, 16.05.2012 08:36:
On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel wrote:
I'm with Jean-Paul here.
...for a reason that might be unrelated to Jean-Paul's original complain: it's not like Cython gives first-class object manipulation on the result of parsing its .pyx files...
I wasn't talking about .pyx files but .pxd files. While the first contain actual (Cython) source code, which the tool we are discussing here would want to see in a pure Python source file, .pxd files are meant to contain C/C++ declarations that the Cython compiler uses to learn about external code. (Also about C-level APIs of Cython generated modules, BTW, using a PyCapsule based export mechanism. We'd eventually like to remove the need for that with the CEP that Mark mentioned.) Here is an example: http://docs.cython.org/src/tutorial/clibraries.html The first page or so should give you an idea. You can stop reading at the point where I start implementing the Queue wrapper class, as that's something that you would want to do in plain Python code instead. There are ways to generate .pxd files from header files as a one-shot operation, so that users can get going quickly and then start adapting the declaration to what they want them to look like when writing their own code against them. So this indirection allows them to take advantage of some ambiguities given by C.
I don't think there's anything wrong with requiring declarations for external C code to live in external modules.
I'm not against the idea of recognizing also Cython syntax extensions in the declarations, and indeed external files are a better idea than big inlined strings then. But I would also like to try out different paths. In one of them, these inline declarations would contain mostly just "#include <foo.h>" and the real header would be processed with the help of the C compiler.
Then take a look at some of the tools that Cython users have written to generate .pxd files from header files. I heard of at least two that use gccxml and clang respectively. Seems like Wim Lavrijsen has also started doing the same thing all over again with cppyy. SWIG also does it, but basically all on its own and without the helpful hand of a real C/C++ compiler. I would really appreciate it if there was one way to do this that everyone could just settle on and use, instead of throwing massive amounts of separate developer time on the same issue again and again, that only lead to tools that "work for what the author needed right now". We actually have a GSoC project this year that tries to get a .pxd generation tool up to a point where it becomes generally usable, based on the gcc Python plugin, which sounds like a good idea to me. There's still some time left to refocus this into something that also makes it more widely usable, outside of a plain Cython context. Stefan
Hi Stefan,
Then take a look at some of the tools that Cython users have written to generate .pxd files from header files. I heard of at least two that use gccxml and clang respectively. Seems like Wim Lavrijsen has also started doing the same thing all over again with cppyy.
not sure where the statement "all over again" comes from, given that the tech we use for CPython-bindings pre-dates Cython by half a decade or more. Now it's just being re-used and re-implemented for PyPy in a way that fits PyPy best.
I would really appreciate it if there was one way to do this that everyone could just settle on and use, instead of throwing massive amounts of separate developer time on the same issue again and again, that only lead to tools that "work for what the author needed right now".
That'd be great, but we (HEP) absolutely need C++ in all it's ugly glory, and it needs to be fully automatic from the header files. W/o that ability, which we do have today for CPython, there would be no Python in HEP. This discussion seems to be mostly restricted to C. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
wlavrijsen@lbl.gov, 16.05.2012 17:39:
Then take a look at some of the tools that Cython users have written to generate .pxd files from header files. I heard of at least two that use gccxml and clang respectively. Seems like Wim Lavrijsen has also started doing the same thing all over again with cppyy.
not sure where the statement "all over again" comes from, given that the tech we use for CPython-bindings pre-dates Cython by half a decade or more.
Hmm, are you sure? Pyrex is pretty old, too. Cython's C++ support certainly is a lot younger, though. Well, anyway... My point is that there are too many ways to extract the declarations of a C/C++ API already and each of them is tied to a specific tool and thus hard to reuse elsewhere. It would be great to have a common declaration layer that these tools can generate from the one side and code can use from the other side.
Now it's just being re-used and re-implemented for PyPy in a way that fits PyPy best.
Ok, my bad then. Get in line with SWIG. ;)
I would really appreciate it if there was one way to do this that everyone could just settle on and use, instead of throwing massive amounts of separate developer time on the same issue again and again, that only lead to tools that "work for what the author needed right now".
That'd be great, but we (HEP) absolutely need C++ in all it's ugly glory, and it needs to be fully automatic from the header files. W/o that ability, which we do have today for CPython, there would be no Python in HEP. This discussion seems to be mostly restricted to C.
I admit that I might have become a bit sloppy about all this "C/C++" repetition over time. I definitely see C++ in the loop here. Stefan
Hi Stefan,
Hmm, are you sure? Pyrex is pretty old, too. Cython's C++ support certainly is a lot younger, though. Well, anyway...
we've been parsing C++ since C++ itself was a spring chicken. I don't know when Pyrex started (and it doesn't seem to parse headers AFAICS?), but unless you tell me it was the early nineties ... The Python bindings came later for us (2002), and again, no other technology would have flown: we simply would not have been using Python today in HEP if I'd had followed any other route.
My point is that there are too many ways to extract the declarations of a C/C++ API already and each of them is tied to a specific tool and thus hard to reuse elsewhere.
This is true and most tools are completely insufficient for all tasks as C++ parsing is hard, so everyone always settles on a subset. CLang finally changes this picture, which is why we are betting on that. Problem is inertia, and the fact that the reflection info is also used for I/O (that was its original purpose). Transitioning is painful, as not everyone here is convinced that allowing the tech to read our data be an external tool is acceptable, given that we're thinking on timescales of the order of decades, and you never know when anything not in-house goes dodo-bird or is taken private.
Now it's just being re-used and re-implemented for PyPy in a way that fits PyPy best.
Ok, my bad then. Get in line with SWIG. ;)
Not sure I'm following that one: back in the day, we tried SWIG, but that wasn't up to the task (it's much better today, but not yet good enough). The point I'm making is that I could get the original CPython extension code to work on PyPy. However, then the Python side is fast (b/c of the JIT), and the C++ side is fast (b/c it's C++), leaving the bindings to become the bottle neck and we'd have achieved very little. That same issue is true for SWIG. With cppyy, which uses libffi underneath and thus readily lifts on the work done by the core PyPy folks for ctypes, virtually all call overhead can be removed. Only overhead left are a couple of guards. However, if the C++ body isn't completely trivial, then OOO and hyperthreading happily munch those guards on the side. Iow. on a modern CPU that isn't completely overcommitted, the cost of calling from Python into C++ can be as low as the cost of calling from C++ into C++. (And it's only a fraction more if overcommitted.) That's the point of cppyy, not the parsing/extracting part. The parsing part is always an external tool. Whether Reflex, CINT, Cling, or Cython++ when that becomes available for use, makes no matter. Build it (to the same level that any of those other tools are today), and I'll add a back-end. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi all, We started working on it a few days ago and it's already progressed well: http://github.com/arigo/ffi A bientôt, Armin.
Whats the problem with cppyy? Why don't extend it rather than build new solution based on pycparser?
Hi Robert, On Tue, May 29, 2012 at 3:49 PM, Robert Zaremba <robert.zaremba@zoho.com> wrote:
Whats the problem with cppyy? Why don't extend it rather than build new solution based on pycparser?
Once again the design goals are different. Note that the first mail in this thread was 15 days ago. During this time we got a working prototype. I'm sure that it would have taken us 15 days too to even figure out how to correctly interface with cppyy, not to mention that it requires a C++ compiler and --- as far as I know --- cannot handle some of the most C-specific ways headers are written, like (ab)uses of macros. A bientôt, Armin.
Hi, On Wed, 30 May 2012, Armin Rigo wrote:
On Tue, May 29, 2012 at 3:49 PM, Robert Zaremba <robert.zaremba@zoho.com> wrote:
Whats the problem with cppyy? Why don't extend it rather than build new solution based on pycparser? Once again the design goals are different.
right; to Robert: think (conceptually) of cppyy as "c++types". It has so far been presented in conjunction with Reflex, because the only way that it is useful for a Python programmer is with an automatic backend to fill in all necessary details. But for PyPy development purposes, the dispatch (cppyy) should be seen separately from the specification (Reflex/CINT/Cling/...) of C++ details.
Note that the first mail in this thread was 15 days ago. During this time we got a working prototype. I'm sure that it would have taken us 15 days too to even figure out how to correctly interface with cppyy
Actually, with Reflex it would be a bit of a hassle (you'd have to stuff the string in a file, add the required headers, run gccxml, etc., etc.) but with CINT or Cling, you could just hand the string to the C++ interpreter, run it and have the bindings generated on the fly.
not to mention that it requires a C++ compiler
Yes, you simply can't develop something this clean and dependency free for C++. Of course, it'd work for a subset of C++, but that'd have limited use.
and --- as far as I know --- cannot handle some of the most C-specific ways headers are written, like (ab)uses of macros.
Oh yes it can! :) If the compiler can parse it, then bring it on. :) Now I feel like providing bindings to cpp macro's just to prove that it can be done (did so in a limited way for PyROOT; haven't bothered for cppyy yet). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Thanks for clarification. I've been wondering why to dobule the work. I'm also really fascinated about this idea, and that i don't need to use separate tool. It looks really great! Thanks! -- Robert
So, when would users be able to test it? I'd like to compare the overhead with ctypes and cppyy for instance. I also wonder if I'd be able to send Python objects to the c++ domain for storage in a container (e.g. a vector). BR, Alex keskiviikko 30 toukokuu 2012 18:00:16 Robert Zaremba kirjoitti:
Thanks for clarification. I've been wondering why to dobule the work.
I'm also really fascinated about this idea, and that i don't need to use separate tool. It looks really great! Thanks!
Hi Alex, On Sun, Jun 3, 2012 at 10:16 AM, Alex Pyattaev <alex.pyattaev@gmail.com> wrote:
So, when would users be able to test it? I'd like to compare the overhead with ctypes and cppyy for instance.
The first version is ready for testing at https://github.com/arigo/ffi . Note that the goal is not to interface with C++ but only with C. In this pre-0.1-release, it supports the same things as a basic ctypes, minus all the Windows calling convention stuff. We're going to work now on non-ctypes issues like structs without a fixed known layout, macros, etc. (listed in the README.md file). Reading just the file names in 'ctypes/test/test_*.py', I should stress again that the scale of the project is completely different than ctypes: it's meant as a minimal interface to C offering a C-like feeling, with absolutely no extra features on top. See the 'demo/readdir.py' file for a quick introduction. If you already know independently Python, C, and (for this example) did at some point read "man openat" on Linux, then this code should be self-explanatory and kind of obvious. A bientôt, Armin.
Okay, i see the issue now. Basically i was under an impression that FFI was somehow a more elaborate thing... Sorry for the bother, the actual thing i need is more like cppyy, but working =) Anyways, might be actually interesting to see it work with SWIG, since it uses C-like functions to wrap C++ classes. But, as you have said, it is pre-alpha, so no SWIG as of now... BR, Alex. sunnuntai 03 kesäkuu 2012 16:28:52 Armin Rigo kirjoitti:
Hi Alex,
On Sun, Jun 3, 2012 at 10:16 AM, Alex Pyattaev <alex.pyattaev@gmail.com> wrote:
So, when would users be able to test it? I'd like to compare the overhead with ctypes and cppyy for instance.
The first version is ready for testing at https://github.com/arigo/ffi . Note that the goal is not to interface with C++ but only with C. In this pre-0.1-release, it supports the same things as a basic ctypes, minus all the Windows calling convention stuff. We're going to work now on non-ctypes issues like structs without a fixed known layout, macros, etc. (listed in the README.md file).
Reading just the file names in 'ctypes/test/test_*.py', I should stress again that the scale of the project is completely different than ctypes: it's meant as a minimal interface to C offering a C-like feeling, with absolutely no extra features on top.
See the 'demo/readdir.py' file for a quick introduction. If you already know independently Python, C, and (for this example) did at some point read "man openat" on Linux, then this code should be self-explanatory and kind of obvious.
A bientôt,
Armin.
Hi all, I hope I did the right thing changing the subject line, though it was the ffi C header and CPython mixed source issues that got me started... On 05/15/2012 11:36 PM Armin Rigo wrote:
Hi Stefan,
On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel<stefan_ml@behnel.de> wrote:
I'm with Jean-Paul here.
...for a reason that might be unrelated to Jean-Paul's original complain: it's not like Cython gives first-class object manipulation on the result of parsing its .pyx files... well, too bad but Python is not Lisp :-)
I don't think there's anything wrong with requiring declarations for external C code to live in external modules.
I'm not against the idea of recognizing also Cython syntax extensions in the declarations, and indeed external files are a better idea than big inlined strings then. But I would also like to try out different paths. In one of them, these inline declarations would contain mostly just "#include<foo.h>" and the real header would be processed with the help of the C compiler.
A bientôt,
Armin.
I see in this a general method of passing arbitrary sources to a translation system -- in this case with pypy as the invoking shell command. with now a possibly open-ended selection of file types. After C headers, CPython, ... ?? ;-) rfc2045[1] and related rfcs define the MIME way of specifying multipart data sets, with open-ended ways of specifying types, character encoding, transmission encoding, etc., and includes a simple delimiter system that permits nested quoting/inclusion implying tree structure possibilities (unlikely, but if needed) without too much judo. [1] http://tools.ietf.org/html/rfc2045 So I'm wondering if you could see using a new command line option something like pypy -mime mime_container_file -mime optionally_another python_source You could even optionally package everything including python sources in the container file, specifying what's what in the mime headers. Also wouldn't need encoding cookies in the source bodies. Or you could just use the mime file as a top level type and encoding specification for the rest of your sources, like a typed manifest for what your pypy invocation is going to use. I am not totally in love with MIME, but I like the abstraction behind it. And it is a pretty well defined standard in common use (which solves a big part of the documentation problem ;-) I would think that using a general-purpose multipart container file type as (optional) source would help with IDEs also, making it easy to make a folded top level MIME representation that can be expanded by clicking a [+] line to invoke plugin specialized editors selected on the basis of the metadata headers in the MIME. (I guess you could do the same with any container file, .e.g. .tgz, but I suspect you'd wind up re-inventing conventions already defined in rfc2045). Further, there's no reason you couldn't define MIME parts that were just a line or two of headers, a blank separator, and some url lines pointing to files anywhere. You could include sha1 hashes or gpg signatures etc in the header section for automated security and integrity checks, specify caching and/or automatic retrieval or update, etc. etc. But you could also use MIME very minimally ;-) This would be a way to include read-only legacy files with no modifications, while specifying encoding etc in a mixed type set that would otherwise require encoding cookie mods. You could also include or refer to patch or test stuff, but you could also use single-part MIME files like typed symbolic links with only a single line of metadata, a blank line, and an url in the file. I'm sure you are already ahead of me into other possibilities. Regards, Bengt Richter
Hi Jean-Paul, On Tue, May 15, 2012 at 9:22 PM, <exarkun@twistedmatrix.com> wrote:
I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work.
While I generally agree with this, I have doubts about this particular case. It seems to me that building normal Python objects to represent C types has mostly the effect of multiplying verbosity in the source by a factor of 3, and requiring the programmer to learn details about the new classes; and with very little gains because C types are, by definition, supposed to be used only in declarations and rarely in actual code (apart from C casts). I fear that down this path lies precisely ctypes. A bientôt, Armin.
participants (10)
-
Alex Pyattaev -
Amaury Forgeot d'Arc -
Armin Rigo -
Bengt Richter -
Eleytherios Stamatogiannakis -
exarkun＠twistedmatrix.com -
Maciej Fijalkowski -
Robert Zaremba -
Stefan Behnel -
wlavrijsen＠lbl.gov