Why CFFI is not useful - need direct ABI access 4 humans
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface. CFFI is not useful for Python programmers and here is why. The primary reason is that it requires you to know C. And knowing C requires you to know about OS architecture. And knowing about OS architecture requires you to know about ABI, which is: http://stackoverflow.com/a/3784697 This is how the compiler builds an application. It defines things (but is not limited too): How parameters are passed to functions (registers/stack). Who cleans parameters from the stack (caller/callee). Where the return value is placed for return. How exceptions propagate. The problematic part of it is that you need to think of OS ABI in terms of unusual C abstractions. Coming through several levels of them. Suppose you know OS ABI and you know that you need direct physical memory access to set bytes for a certain call in this way: 0024: 00 00 00 6C 33 33 74 00 How would you do this in Python? The most obvious way is with byte string - \x00\x00\x00\x6c\x33\x33\x74\x00 but that's not how you prepare the data for the call if, for example, 00 6C means anything to you. What is the Python way to convert 00 6C to convenient Python data structure and back and is it Pythonic (user friendly and intuitive)? import struct struct.unpack('wtf?', '\x00\x6C') If you try to lookup the magic string in struct docs: http://docs.python.org/2/library/struct.html#format-characters You'll notice that there is the mapping between possible combinations of these 2 bytes to some Python type is very mystic. First it requires you to choose either "short" or "unsigned short", but that's not enough for parsing binary data - you need to figure out the proper "endianness" and make up a magic string for it. This is just for two bytes. Imagine a definition for a binary protocol with variable message size and nested data structures. You won't be able to understand it by reading Python code. More than that - Python *by default* uses platform specific "endianness", it is uncertain (implicit) about it, so not only you should care about "endianness", but also be an expert to find out which is the correct metrics for you. Look at this: 0024: 00 00 00 6C 33 33 74 00 Where is "endianness", "alignment", "size" from this doc http://docs.python.org/2/library/struct.html#byte-order-size-and-alignment People need to *start* with this base and this concept and that's why it is harmful. CFFI proposes to provide a better interface to skip this complexity by getting back to roots and use C level. That's a pretty nice hack for C guys, I am sure it makes them completely happy, but for academic side of PyPy project, for Python interpreter and other projects build over RPython it is important to have a tool that allows to experiment with binary interfaces in convenient, readable and direct way, makes it easier for humans to understand (by reading Python code) how Python instructions are translated by JIT into binary pieces in computer memory, pieces that will be processed by operating system as a system function call on ABI level. But let's not digress, and get back to the point that struct module doesn't allow to work with structured data. In Python the only alternative standard way to define binary structure is ctypes. ctypes documentation is no better for binary guy: http://docs.python.org/2/library/ctypes.html#fundamental-data-types See how that binary guy suffered to map binary data to Python structures through ctypes: https://bitbucket.org/techtonik/discovery/src/eacd864e6542f14039c9b31eecf943... And I am saying that this is the best way available from standard library. It is pretty close to Django models, but for binary data. ctypes still is worse that struct in one thing - looking into docs, there are no size specifiers for any kind of C type, so no guarantee that 2 bytes are read as 4 bytes or worse. By looking at the ctypes code it is hard to figure out size of structure and when it may change. I can't hardly name ctypes mapping process as user friendly and resulting code as intuitive. Probably nobody could, and that's why CFFI was born. But CFFI took a different route - instead of trying to map C types to binary data (ABI level), it decided to go onto API level. While it exposes many better tool, it basically means you are dealing with C interface again - not with Pythonic interface for binary data. I am not saying that CFFI is bad - I am saying that it is good, but not enough, and that it can be fixed with cleanroom engineering approach for a broader scope of modern usage pattern for binary data than just calling OS API in C way. Why we need it? I frankly think that Stackless way of doing thing without C stack is the future, and the problem with not that not many people can see how it works, builds alternative system without classic C stack with (R)Python. Can CFFI help this? I doubt that. So, that am I proposing. Just an idea. Given the fact that I am mentally incapable of filling 100 sheet requirement to get funding under H2020, the fact that no existing commercial body could be interested to support the development as an open source project and the fact that hacking on it alone might become boring, giving this idea is the least I can do. Cleanroom engineering. http://en.wikipedia.org/wiki/Cleanroom_software_engineering "The focus of the Cleanroom process is on defect prevention, rather than defect removal." When we talk about Pythonic way of doing thing, how can we define "a defect"? Basically, we talking about user experience - the emotions that user experiences when he uses Python for the given task. What is the task at hand? For me - it is working with binary data in Python - not just parsing save games, but creating binary commands such as OS systems calls that are executed by certain CPU, GPU or whatever is on the receiver end of whatever communication interface is used. This is hardware independent and platform neutral way of doing things. So, the UX is the key, but properties of engineered product are not limited single task. The cleanroom approach allows to concentrate on the defect - when user experience will start to suffer because of the conflicts between tasks that users are trying to accomplish. For PyPy project I see the value in library for compositing of binary structures in that these operations can be pipelined and optimized at run-time in a highly effective fashion. I think that convenient binary tool is the missing brick in the basement of academic PyPy infrastructure to enable universal interoperability from (R)Python with other digital systems by providing a direct interface to the binary world. I think that 1973 year views on "high level" and "low level" systems are a little bit outdated now that we have Python, Ruby, Erlang and etc. Now C is just not a very good intermediary for "low level" access. But frankly, I do not think that with advent of networking, binary can be called a low level anymore. It is just another data format that can be as readable for humans as program structure written in Python. P.S. I have some design ideas how to make an attractive gameplay out of binary data by "coloring" regions and adding "multi-level context" to hex dumps. This falls out of scope of this issue, and requires more drawing that texting, but if somebody wants to help me with sharing the vision - I would not object. It will help to make binary world more accessible, especially for new people, who start coding with JavaScript and Python. -- anatoly t.
On Sat, Mar 29, 2014 at 1:49 PM, anatoly techtonik <techtonik@gmail.com> wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
Anatoly, such blank statements that are factually incorrect is the reason why people think your input is counterproductive. Please stop posting random ideas on pypy-dev, if you have no interest in working on them. If you want to convince someone to implement your next idea, there are places to do that, but pypy-dev is about the development of PyPy and it's the wrong place to do so. Cheers, fijal
On Sat, Mar 29, 2014 at 2:59 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Sat, Mar 29, 2014 at 1:49 PM, anatoly techtonik <techtonik@gmail.com> wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
Anatoly, such blank statements that are factually incorrect is the reason why people think your input is counterproductive.
This statement in incorrect out of context, the context in which it is correct is provided below "here is why.". If you know how to rephrase the statement differently to define the scope correctly, I am all ears. Unfortunately I am not a journalist or writer or scientists to write short and concise papers. I write how I think, and it is a problem to rewrite stuff in English to sound differently, mostly because of time constraints.
Please stop posting random ideas on pypy-dev, if you have no interest in working on them.
You're wrong in your assumptions that I am not interested. Now that it is clear, let me restate that the problem is find time to work on the problem, because productive hours are spent on paid chores.
If you want to convince someone to implement your next idea, there are places to do that, but pypy-dev is about the development of PyPy and it's the wrong place to do so.
I just want an get a feedback on the opinion. If the next idea is good or not. Maybe there are people who had this idea before. I don't ask anybody to do this. About others how can implement your ideas, I am convinced that people work on their own ideas in their free time, so collaboration only happens when ideas match. If you know the places where it is not true, I'll be interested to know about them. If you can give direct link - that will be productive, but I really doubt there are places where people spending time implementing ideas of others just for good. -- anatoly t.
On Sat, Mar 29, 2014 at 02:49:00PM +0300, anatoly techtonik wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
The primary reason is that it requires you to know C.
You're using C if you're calling it from Python. Knowing the language (to some degree) when using it is inevitable.
And knowing C requires you to know about OS architecture.
The PyPy team (especially fijal) has always strongly discouraged from porting Python code to C for performance. If you have a good reason to use C, it is not surprising that you're going to be confronted with the dangers of such a language. I am not sure if you're trying to make a point against C or CFFI here. I am also not sure if the rest of your post actually means anything, or if it is just way above my head. But given that you're throwing around with statements like "this is useless", i don't feel compelled or motivated to try to understand your ramblings. -- Markus
On Sat, Mar 29, 2014 at 3:58 PM, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
On Sat, Mar 29, 2014 at 02:49:00PM +0300, anatoly techtonik wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
The primary reason is that it requires you to know C.
You're using C if you're calling it from Python. Knowing the language (to some degree) when using it is inevitable.
This is the problem that I've tried to describe: All standard Python tools for ABI level access require C knowledge.
And knowing C requires you to know about OS architecture.
The PyPy team (especially fijal) has always strongly discouraged from porting Python code to C for performance. If you have a good reason to use C, it is not surprising that you're going to be confronted with the dangers of such a language. I am not sure if you're trying to make a point against C or CFFI here.
Against C. As I said, CFFI is good, but not enough to work conveniently with binary interfaces, and the reason for that is that it is C-centric. I support fijal - my position is that rewriting the same code in faster language is not a way to solve performance problems. Language as a problem is a failed smoke test for app architecture.
I am also not sure if the rest of your post actually means anything, or if it is just way above my head. But given that you're throwing around with statements like "this is useless", i don't feel compelled or motivated to try to understand your ramblings.
Fair point. Thanks for the feedback. Sometimes I feel like I should just stop wasting my time on ideas, and start eating some pills so that I could better concentrate on a mindless coding. -- anatoly t.
On Sun, Mar 30, 2014 at 12:05:06PM +0300, anatoly techtonik wrote:
On Sat, Mar 29, 2014 at 3:58 PM, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
On Sat, Mar 29, 2014 at 02:49:00PM +0300, anatoly techtonik wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
The primary reason is that it requires you to know C.
You're using C if you're calling it from Python. Knowing the language (to some degree) when using it is inevitable.
This is the problem that I've tried to describe:
All standard Python tools for ABI level access require C knowledge.
And knowing C requires you to know about OS architecture.
The PyPy team (especially fijal) has always strongly discouraged from porting Python code to C for performance. If you have a good reason to use C, it is not surprising that you're going to be confronted with the dangers of such a language. I am not sure if you're trying to make a point against C or CFFI here.
Against C. As I said, CFFI is good, but not enough to work conveniently with binary interfaces, and the reason for that is that it is C-centric.
I am not trying to dogmatize anything here, but i don't see a reason why efforts should be made to eliminate that property you're seeing as a problem, and i am not sure it'd be *worth it*. To me, the main usecase of CFFI seems to be embedding existing C libraries, not directly accessing ABIs.
I support fijal - my position is that rewriting the same code in faster language is not a way to solve performance problems. Language as a problem is a failed smoke test for app architecture.
I am also not sure if the rest of your post actually means anything, or if it is just way above my head. But given that you're throwing around with statements like "this is useless", i don't feel compelled or motivated to try to understand your ramblings.
Fair point. Thanks for the feedback. Sometimes I feel like I should just stop wasting my time on ideas, and start eating some pills so that I could better concentrate on a mindless coding.
That's not what i meant. It doesn't matter whether your ideas are good or bad: The way you're formulating your ideas is incredibly insulting to authors of existing solutions.
-- anatoly t.
On Sat, Mar 29, 2014 at 3:58 PM, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
On Sat, Mar 29, 2014 at 02:49:00PM +0300, anatoly techtonik wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
The primary reason is that it requires you to know C.
You're using C if you're calling it from Python. Knowing the language (to some degree) when using it is inevitable.
This is the problem that I've tried to describe:
All standard Python tools for ABI level access require C knowledge.
And knowing C requires you to know about OS architecture.
The PyPy team (especially fijal) has always strongly discouraged from porting Python code to C for performance. If you have a good reason to use C, it is not surprising that you're going to be confronted with the dangers of such a language. I am not sure if you're trying to make a point against C or CFFI here.
Against C. As I said, CFFI is good, but not enough to work conveniently with binary interfaces, and the reason for that is that it is C-centric.
Hi, (disclaimer: i have worked a lot with cffi, and i basically love it because most of my projects are in C and i need to interface with them), i am not sure to follow you. You seem to mix C with binary/structures manipulation. CFFI is for integration between PyPy/Python and C interfaces, so using C (or something really near to it) is its main purpose. If your problem is with structures manipulations, i totally agree, the python world need something better (unless i am missing some project regarding it), but this is totally irrelevant in the CFFI area. Regards -- Roberto De Ioris http://unbit.it
On Sun, Mar 30, 2014 at 12:26 PM, Roberto De Ioris <roberto@unbit.it> wrote:
Hi, (disclaimer: i have worked a lot with cffi, and i basically love it because most of my projects are in C and i need to interface with them), i am not sure to follow you. You seem to mix C with binary/structures manipulation. CFFI is for integration between PyPy/Python and C interfaces, so using C (or something really near to it) is its main purpose.
If your problem is with structures manipulations, i totally agree, the python world need something better (unless i am missing some project regarding it), but this is totally irrelevant in the CFFI area.
I mix C with binary manipulation - that's right. I feel like the problem of efficiently making C calls is a subset of a more generic problem of converting Python data into corresponding binary data, because C call looks like a chunk of binary data in memory + register setup. http://unixwiz.net/techtips/win32-callconv-asm.html I think that PyPy JIT can do faster calls that C if it was possible to manipulate with memory on this low level (for instance, providing a pre-filled memory for a series of calls and then just modifying esp for each call). If CFFI included a toolset to construct such binary calls - it would not be limited to C anymore, and will open ways to engineer more effective paradigms (Stackless + channels) and better calling conventions for interop with other useful tools (Go etc.) on binary level. I started idea with CFFI, because there is no better base to start. -- anatoly t.
Okay, just to get things right: What you want is an only-ABI solution, which abstracts completely away from technical details, in a nice pythonic wrapper? Having that idea suggestion is fine (although it is slightly off-topic on pypy-dev), but it has nothing to do with the C Foreign Function Interface (CFFI), which is C-centric as it focuses on interfacing with C. It allows for making very easy interfaces to C with very little code, which as nice bonus also appears very clean. One can also easily make the argument that, when you're glueing two languages together, you have to know both to make the proper considerations about their use. You risk making improper use of returned memory if you don't know what's going on, and you'll have no clue how to debug it. But to sum it up: - You want a language independent ABI interface that looks completely pythonic from the users point of view, and requires no knowledge of other languages - This has nothing to do with CFFI, which is very specifically - as the name implies - a C interface, which does it's job very well. Correct? My personal opinion of the idea is that it is likely to be troublesome enough to be unfeasible and very unpleasant to code. I also find it unlikely to work (without giving a load of trouble to the user), but that should not stop interested parties from trying. Regards, Kenny P.S.: Calling the work of others useless is a bad way to introduce an idea. (Unless it's an idea for "How to be hated for Dummies")
On 30/03/2014, at 11.05, anatoly techtonik <techtonik@gmail.com> wrote:
On Sat, Mar 29, 2014 at 3:58 PM, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
On Sat, Mar 29, 2014 at 02:49:00PM +0300, anatoly techtonik wrote:
I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface.
CFFI is not useful for Python programmers and here is why.
The primary reason is that it requires you to know C.
You're using C if you're calling it from Python. Knowing the language (to some degree) when using it is inevitable.
This is the problem that I've tried to describe:
All standard Python tools for ABI level access require C knowledge.
And knowing C requires you to know about OS architecture.
The PyPy team (especially fijal) has always strongly discouraged from porting Python code to C for performance. If you have a good reason to use C, it is not surprising that you're going to be confronted with the dangers of such a language. I am not sure if you're trying to make a point against C or CFFI here.
Against C. As I said, CFFI is good, but not enough to work conveniently with binary interfaces, and the reason for that is that it is C-centric.
I support fijal - my position is that rewriting the same code in faster language is not a way to solve performance problems. Language as a problem is a failed smoke test for app architecture.
I am also not sure if the rest of your post actually means anything, or if it is just way above my head. But given that you're throwing around with statements like "this is useless", i don't feel compelled or motivated to try to understand your ramblings.
Fair point. Thanks for the feedback. Sometimes I feel like I should just stop wasting my time on ideas, and start eating some pills so that I could better concentrate on a mindless coding. -- anatoly t. _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On Sun, Mar 30, 2014 at 2:40 PM, Kenny Lasse Hoff Levinsen <kennylevinsen@gmail.com> wrote:
Okay, just to get things right: What you want is an only-ABI solution, which abstracts completely away from technical details, in a nice pythonic wrapper?
Not really. I want a language independent ABI solution, yes. ABI-only implies that there is some alternative to that. I don't see any alternative - for me ABI is the necessary basis for everything else on top. So doing ABI solution design without considering these use cases is impossible. I want a decoupled ABI level. Nice pythonic wrapper that abstracts completely from technical details is not the goal. The goal is to provide practical defaults for language and hardware independent abstraction. The primary object that abstraction should work with is "platform-independent binary data", the method is "to be readable by humans". On implementation level that means that by default there is no ambiguity in syntax that defines binary data (size or endianness), and if there is a dependency on platform (CPU bitness etc.) it should be explicit, so that the behavior of structure should be clear (self-describing type names + type docs that list relevant platforms and effects on every platform). This approach inverts existing practice of using platform dependent binary structures by default.
But to sum it up: - You want a language independent ABI interface that looks completely pythonic from the users point of view, and requires no knowledge of other languages
Exactly. Python here means "intuitive and user-friendly" (which may appear different than indentations or YAML).
- This has nothing to do with CFFI, which is very specifically - as the name implies - a C interface, which does it's job very well.
Correct?
Yes CFFI does a perfect job on API level - I can't think of a better way to provide access to C API other than with C syntax. On ABI level tools can be more useful, and there is where idea intersects with CFFI. It doesn't cancel the fact that people need safe feet injury prevention interfaces. Look for my previous answer with the phrase "I mix C with binary manipulation" that covers this.
My personal opinion of the idea is that it is likely to be troublesome enough to be unfeasible and very unpleasant to code. I also find it unlikely to work (without giving a load of trouble to the user), but that should not stop interested parties from trying.
Ack. Thanks for the feedback.
On Sun, Mar 30, 2014 at 03:36:08PM +0300, anatoly techtonik wrote:
On Sun, Mar 30, 2014 at 2:40 PM, Kenny Lasse Hoff Levinsen <kennylevinsen@gmail.com> wrote:
Okay, just to get things right: What you want is an only-ABI solution, which abstracts completely away from technical details, in a nice pythonic wrapper?
Not really. I want a language independent ABI solution, yes. ABI-only implies that there is some alternative to that. I don't see any alternative - for me ABI is the necessary basis for everything else on top. So doing ABI solution design without considering these use cases is impossible. I want a decoupled ABI level.
If you want to purely binary data in python there are already some good options: struct and numpy. The first is optimised for short records, the latter for multidimensional arrays of binary data. I've used both on occasion for binary file and cross language communications.
Nice pythonic wrapper that abstracts completely from technical details is not the goal. The goal is to provide practical defaults for language and hardware independent abstraction. The primary object that abstraction should work with is "platform-independent binary data", the method is "to be readable by humans". On implementation level that means that by default there is no ambiguity in syntax that defines binary data (size or endianness), and if there is a dependency on platform (CPU bitness etc.) it should be explicit, so that the behavior of structure should be clear (self-describing type names + type docs that list relevant platforms and effects on every platform). This approach inverts existing practice of using platform dependent binary structures by default.
So struct and numpy are existing solutions to this problem, but I think you are thinking too low level for most problems. It sounds like what you really want is a schema based serialisation protocol. There are a million of these, all alike, but the two I've used the most are msgpack and thrift. Generally you want to be specifying statistical distributions rather that the specific encoding scheme; the choice between 4 byte and 8 byte ints is not particularly useful at the python programmer level, but knowing whether the number is a small int or a large one is. Thrift is the best implementation of a remote call I've used, but it still leaves a lot of room for improvment. If you are actually talking about building something that will talk to any existing code in any existing language, then you will need something more like CFFI. However, I don't think you want to take the C out of CFFI. The reason is that the packing of the data is actually one of the least important parts of that interface. As you've read on pypy-dev recently, reference counting vs gc is hard to get right. But there are many other problems which you have to address for a truly language independent ABI: Stack convention: what order are things pushed onto the stack (IIRC C and pascal grow their heaps in opposite directions in memory). Are things even pushed onto a stack? (LISP stacks are implemented with linked lists, some languages don't even bother with stack frames when they perform tail recursion removal, using conditional gotos instead) Packing conventions: different cpus like pointers to be even, or aligned to 8 bytes. Your code won't be portable if you don't handle this. Exception handling: There are as many ways to handle exceptions as there are compilers, all of them with subtle rules around lifetimes of all the objects that are being excepted over. Virtual methods and the like: In most languages (C is actually somewhat unusual here) methods or functions are actually called via a dereference or two rather than a direct memory location. In C++ virtual methods and Java this looks like a field lookup to find the vtable then a jump to a memory location from that table. This is tedious and error prone to implement correctly. Generics and types: just working out which function to call is difficult once you have C++ templates, performing the correct type checking is difficult for Java. Haskell's internal type specification alone is probably larger than all of the CFFI interfaces put together. There are no doubt whole areas I've missed here, but I hope this gives you a taste for why language developers love CFFI - it's easy to implement, easy enough to use and hides all of these complexities by pushing everything through the bottleneck which is the C ABI. Language developers would rather make their own language wonderful, compiler writers would rather make their own language efficient. Nobody really cares enough to make supporting other languages directly a goal, the biggest and best effort in this regard was .net and it also cheated by folding everything through a small bottleneck (albeit a more expressive one). This folding scars the languages around it - F# is still less pure than Haskell, J is incompatible with Java. If you want to tackle something in this area I would encourage you to look at the various serialisation tools out there and work out why they are not as good as they could be. This is probably not the right mailing list for this discussion though. njh
On Sun, Mar 30, 2014 at 6:29 PM, Nathan Hurst <njh@njhurst.com> wrote:
If you want to purely binary data in python there are already some good options: struct and numpy. The first is optimised for short records, the latter for multidimensional arrays of binary data.
numpy is not Python. About struct I wrote in the first post - it is a legacy interface that was not designed to be human friendly with usability practices of 2014.
I've used both on occasion for binary file and cross language communications.
Nice pythonic wrapper that abstracts completely from technical details is not the goal. The goal is to provide practical defaults for language and hardware independent abstraction. The primary object that abstraction should work with is "platform-independent binary data", the method is "to be readable by humans". On implementation level that means that by default there is no ambiguity in syntax that defines binary data (size or endianness), and if there is a dependency on platform (CPU bitness etc.) it should be explicit, so that the behavior of structure should be clear (self-describing type names + type docs that list relevant platforms and effects on every platform). This approach inverts existing practice of using platform dependent binary structures by default.
So struct and numpy are existing solutions to this problem, but I think you are thinking too low level for most problems. It sounds like what you really want is a schema based serialisation protocol.
If you can assemble ELF binary in that protocol, then yes. My goal is to work with low level binary data in Python with convenient tools. Not necessary high level tools - just convenient for messing with chunks.
There are a million of these, all alike, but the two I've used the most are msgpack and thrift. Generally you want to be specifying statistical distributions rather that the specific encoding scheme; the choice between 4 byte and 8 byte ints is not particularly useful at the python programmer level, but knowing whether the number is a small int or a large one is. Thrift is the best implementation of a remote call I've used, but it still leaves a lot of room for improvment.
Engineering should start with a task at hand. Statistical distribution is a too high level task. It goes above the binary manipulating logic, but the task of choosing a size of data based on other data is a corner stone in all binary format processing.
If you are actually talking about building something that will talk to any existing code in any existing language, then you will need something more like CFFI. However, I don't think you want to take the C out of CFFI. The reason is that the packing of the data is actually one of the least important parts of that interface. As you've read on pypy-dev recently, reference counting vs gc is hard to get right. But there are many other problems which you have to address for a truly language independent ABI:
Stack convention: what order are things pushed onto the stack (IIRC C and pascal grow their heaps in opposite directions in memory). Are things even pushed onto a stack? (LISP stacks are implemented with linked lists, some languages don't even bother with stack frames when they perform tail recursion removal, using conditional gotos instead)
Right. You will not be able to operate with stack registers, but if you have a clean interface to construct the contents of binary stack and read it back to friendly form - this will make the task of working with these structures easier, so maybe one day it will be possible to have a clean code that calls different libs with different calling conventions from Python.
Packing conventions: different cpus like pointers to be even, or aligned to 8 bytes. Your code won't be portable if you don't handle this.
It is good to know about that, so this should be exposed, of course. There is also a room for high level abstraction once you've covered the basics.
Exception handling: There are as many ways to handle exceptions as there are compilers, all of them with subtle rules around lifetimes of all the objects that are being excepted over.
I would be interested to see this on binary level, but I doubt such papers exist. Making a library that can produce diagrams for such structures will be a good starting point to grok at more serious performance problem when you switch CPUs.
Virtual methods and the like: In most languages (C is actually somewhat unusual here) methods or functions are actually called via a dereference or two rather than a direct memory location. In C++ virtual methods and Java this looks like a field lookup to find the vtable then a jump to a memory location from that table. This is tedious and error prone to implement correctly.
It is tedious and error prone only if it is not obvious. There are inevitable complex solutions in this world that are impossible to avoid, but it is possible to deal with complexity. If JIT can detect that vtable is not modified, it can shorten the call chain. Of course you won't be able to define all types of lookup logic in declarative form, but making this code readable is a step forward. If not Eve Online - I'd never new about Stackless and why it is awesome. Eve Online provided a user friendly interface to introduce the technology by showing its power through media of game design discipline. I think that the power of alternative low level manipulations with "data + processor" paradigm can be even better expressed in language design discipline which PyPy/RPython is all about IMHO.
Generics and types: just working out which function to call is difficult once you have C++ templates, performing the correct type checking is difficult for Java. Haskell's internal type specification alone is probably larger than all of the CFFI interfaces put together.
These are high level API and I am very interested how do they map to the low level. That gives me an ability to design a better CPU for these languages (or LPU FWIW) or see that current limitations exist.
There are no doubt whole areas I've missed here, but I hope this gives you a taste for why language developers love CFFI - it's easy to implement, easy enough to use and hides all of these complexities by pushing everything through the bottleneck which is the C ABI. Language developers would rather make their own language wonderful, compiler writers would rather make their own language efficient.
I never argued that CFFI is not loved. But I really think that it is loved only by people who know C or had to face it. I do not agree that language developers should not think about efficiency. As much as I want to simplify the problem too, in real world they should think about many things, find compromises and trade between them. On of the reasons I am so pushy about the process and tools - language design requires more scaffolding and collaborative tools that usual project issue tracker allows. Python 3 is a living proof - better greater language, but the adoption rate is very low. I am not speaking the reasons, I just state the fact. I think that executable language designers should think about low level. http://www.joelonsoftware.com/articles/fog0000000319.html
Nobody really cares enough to make supporting other languages directly a goal, the biggest and best effort in this regard was .net and it also cheated by folding everything through a small bottleneck (albeit a more expressive one). This folding scars the languages around it - F# is still less pure than Haskell, J is incompatible with Java.
If you want to tackle something in this area I would encourage you to look at the various serialisation tools out there and work out why they are not as good as they could be. This is probably not the right mailing list for this discussion though.
I'm interested to see a list of user stories and conflicts between them, like which choice was made in every conflict and (if possible) why. The problem of modern development is that people face with the same problems over and over again unable to understand the previous knowledge. Providing tools that bring greater visibility into the domain field can significantly help the progress. In context of binary manipulation I really want to see more interchangeable graphical interfaces and automated visualization tools built on top of that without overengineering by various international controlling bodies. That means - they try to complicate stuff to solve some problem no matter what. I think that the approach should be more relaxed - there are problems that can't be solved easily, but getting to the root of the problem and playing with it should be easy. -- anatoly t.
Hi Anatoly, On 04/03/2014 10:33 AM anatoly techtonik wrote: [...]
My goal is to work with low level binary data in Python with convenient tools. Not necessary high level tools - just convenient for messing with chunks.
[Nathan]
Exception handling: There are as many ways to handle exceptions as there are compilers, all of them with subtle rules around lifetimes of all the objects that are being excepted over.
I would be interested to see this on binary level, but I doubt such papers exist. Making a library that can produce diagrams for such structures will be a good starting point to grok at more serious performance problem when you switch CPUs.
I think you might enjoy reading about LLVM and their approach to representing bit-level stuff, and interfacing with various languages' definitions of exceptions etc: http://llvm.org/docs/index.html particularly http://llvm.org/docs/LangRef.html from the introduction of the latter: The LLVM code representation is designed to be used in three different forms: as an in-memory compiler IR, as an on-disk bitcode representation (suitable for fast loading by a Just-In-Time compiler), and as a human readable assembly language representation. This allows LLVM to provide a powerful intermediate representation for efficient compiler transformations and analysis, while providing a natural means to debug and visualize the transformations. The three different forms of LLVM are all equivalent. This document describes the human readable representation and notation. [...] I think you may have some interesting ideas. I am also interested in a high level language with ability to get down to the metal, so I am trying to design one. The trouble is, scouting around for ideas, you may wind up doing more reading than actually working. Ltu is particularly dangerous ;-) http://lambda-the-ultimate.org/ I think you should try to design the language you want, according to your ideas. IME there is no better way to gain respect and appreciation for other people's work in language design ;-) When you want to write your compiler, python is there. Or you may find racket's ability to define alternate syntax languages interesting: http://docs.racket-lang.org/guide/languages.html Their home pages is http://racket-lang.org/ Have fun ;-) Regards, Bengt Richter
On Sun, Mar 30, 2014 at 9:36 AM, anatoly techtonik <techtonik@gmail.com>wrote:
On ABI level tools can be more useful, and there is where idea intersects with CFFI. It doesn't cancel the fact that people need safe feet injury prevention interfaces. Look for my previous answer with the phrase "I mix C with binary manipulation" that covers this.
I think you want something like https://pypi.python.org/pypi/construct/2.5.1right? It exists and is pretty good... some guys in my company are using it to model a binary audio control protocol both to prototype and to test the embeded implementation (which is in C). -- Leonardo Santagada
On 03/30/2014 05:32 PM Leonardo Santagada wrote:
On Sun, Mar 30, 2014 at 9:36 AM, anatoly techtonik<techtonik@gmail.com>wrote:
On ABI level tools can be more useful, and there is where idea intersects with CFFI. It doesn't cancel the fact that people need safe feet injury prevention interfaces. Look for my previous answer with the phrase "I mix C with binary manipulation" that covers this.
I think you want something like https://pypi.python.org/pypi/construct/2.5.1right? It exists and is
Need to strip/separate the "right?" to isolate and make the link work: https://pypi.python.org/pypi/construct/2.5.1
pretty good... some guys in my company are using it to model a binary audio control protocol both to prototype and to test the embeded implementation (which is in C).
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hello, I'm not sure, but i think that the JIT doesn't kick in when the loops are "external", where C-code does the loop and calls Python code for each iteration, through CFFI callbacks. In our code, we have SQLite, calling Python code through callbacks. Using the loop log: PYPYLOG=log pypy mterm.py i see this summary: interpret 91% gc-collect-step 4% gc-minor 3% jit-optimize 0% gc-minor-walkroots 0% jit-tracing 0% jit-backend 0% jit-log-compiling-bridge 0% jit-resume 0% jit-backend-dump 0% gc-set-nursery-size 0% jit-log-virtualstate 0% jit-backend-addr 0% gc-hardware 0% jit-log-noopt-loop 0% jit-mem-looptoken-alloc 0% jit-log-opt-bridge 0% jit-log-rewritten-loop 0% jit-log-rewritten-bridge 0% jit-log-compiling-loop 0% jit-log-opt-loop 0% jit-log-short-preamble 0% jit-abort 0% jit-mem-collect 0% jit-summary 0% jit-backend-counts 0% If i read that correctly, most of the execution time is in the interpreter and not in the JITed code. Is there some way where i can "hint" to pypy that a callback is essentially the inside of a loop, so PyPy can JIT it?. I've attached the simple log from which above summary has been produced via: python logparser.py print-summary log.log - Regards, l.
hi "interpret" includes "in the jit". if you're having trouble with the performance, please let us know how to reproduce it and we'll try to help you pinpoint down the problem. On Mon, Mar 31, 2014 at 8:28 PM, Eleytherios Stamatogiannakis <estama@gmail.com> wrote:
Hello,
I'm not sure, but i think that the JIT doesn't kick in when the loops are "external", where C-code does the loop and calls Python code for each iteration, through CFFI callbacks. In our code, we have SQLite, calling Python code through callbacks.
Using the loop log:
PYPYLOG=log pypy mterm.py
i see this summary:
interpret 91% gc-collect-step 4% gc-minor 3% jit-optimize 0% gc-minor-walkroots 0% jit-tracing 0% jit-backend 0% jit-log-compiling-bridge 0% jit-resume 0% jit-backend-dump 0% gc-set-nursery-size 0% jit-log-virtualstate 0% jit-backend-addr 0% gc-hardware 0% jit-log-noopt-loop 0% jit-mem-looptoken-alloc 0% jit-log-opt-bridge 0% jit-log-rewritten-loop 0% jit-log-rewritten-bridge 0% jit-log-compiling-loop 0% jit-log-opt-loop 0% jit-log-short-preamble 0% jit-abort 0% jit-mem-collect 0% jit-summary 0% jit-backend-counts 0%
If i read that correctly, most of the execution time is in the interpreter and not in the JITed code. Is there some way where i can "hint" to pypy that a callback is essentially the inside of a loop, so PyPy can JIT it?.
I've attached the simple log from which above summary has been produced via:
python logparser.py print-summary log.log -
Regards,
l.
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi, On 31 March 2014 20:33, Maciej Fijalkowski <fijall@gmail.com> wrote:
hi "interpret" includes "in the jit".
"In the jit-generated machine code", precisely. Not "in the JIT while building machine code". /me renames this, as it was originally meant indeed to record only interpretation outside machine code. A bientôt, Armin.
On Sun, Mar 30, 2014 at 6:32 PM, Leonardo Santagada <santagada@gmail.com> wrote:
On Sun, Mar 30, 2014 at 9:36 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On ABI level tools can be more useful, and there is where idea intersects with CFFI. It doesn't cancel the fact that people need safe feet injury prevention interfaces. Look for my previous answer with the phrase "I mix C with binary manipulation" that covers this.
I think you want something like https://pypi.python.org/pypi/construct/2.5.1 right? It exists and is pretty good... some guys in my company are using it to model a binary audio control protocol both to prototype and to test the embeded implementation (which is in C).
Right. Construct is a starting point. What's why I included Tomer into CC list. It's primary use, as I see it - is reversing existing binary formats. I'm not quite persuaded that it has the best declarative specification for binary data. The difference is that binary data is multi-level. You can set types as UBInt8, but that's not user friendly. It says it is unsigned, it is integer, with 1 byte in size. Why should I care? In terms of binary protocol I care only about allowed rage of values. How do I work with them is the task of the next level. I decompose the problem into multiple levels: level 0: binary data location and size level 1: check/get/set binary values as-is level 2: processing nested binary values and dependent binary data level 3: converting single values into user data types and back (symmetry) level 4: serializing complex user data types into binary structures and back I believe every linker/assembler does that. If the structure can not be decomposed into these levels - it can not be statically analyzed. There are also many interesting tasks about detecting constraints, processing dependencies and most important - the user experience, interface for humans to work on each level. The basic drawback for construct is that it is hard to write tools that work on top of it. There is no obvious decoupling between different levels, that's why I can hardly think, for example, how can I compile a binary executable from a lot of bytecodes with it. Basically, why I raised the question in CFFI context is an attempt to play with a different user story - assembling binary code - the part that is currently done by tools written in C. -- anatoly t.
participants (10)
-
anatoly techtonik -
Armin Rigo -
Bengt Richter -
Eleytherios Stamatogiannakis -
Kenny Lasse Hoff Levinsen -
Leonardo Santagada -
Maciej Fijalkowski -
Markus Unterwaditzer -
Nathan Hurst -
Roberto De Ioris