Mapping RAW memory data into Python
Hi, While I had some questions concerning accessing objects in Python, that were previously created in C++ now I'd like to know whether it's possible to get access to the raw memory from Python layer. Let's suppose I have the char* ptr pointer that points to the memory chunk with IPv4 packet. I would like to be able to read (and preferably write) data from Python layer (not via wrapped object that contains char* ptr and has some methods to write/read data), but without making any memory copies. I can use struct (even define format dynamically) in Python, but I don't know how do I pass the proper pointer from C++. If I use boost::python::object I had problems that script didn't know char type. I can use boost::python::str, return it and use in Python but I suppose it does copy data into str object, right? If not Boost, maybe basic Python C API would work? BTW. Suppose I had char* txt = "some string"; How do i create boost::python::object() that would containt the whole string? Passing *txt or txt into boost::python::object(*txt) didn't work. -- regards Marek Denis
On 10/25/2010 10:51 AM, Marek Denis wrote:
Hi, While I had some questions concerning accessing objects in Python, that were previously created in C++ now I'd like to know whether it's possible to get access to the raw memory from Python layer. Let's suppose I have the char* ptr pointer that points to the memory chunk with IPv4 packet. I would like to be able to read (and preferably write) data from Python layer (not via wrapped object that contains char* ptr and has some methods to write/read data), but without making any memory copies. I can use struct (even define format dynamically) in Python, but I don't know how do I pass the proper pointer from C++. If I use boost::python::object I had problems that script didn't know char type. I can use boost::python::str, return it and use in Python but I suppose it does copy data into str object, right?
If not Boost, maybe basic Python C API would work?
BTW. Suppose I had char* txt = "some string"; How do i create boost::python::object() that would containt the whole string? Passing *txt or txt into boost::python::object(*txt) didn't work.
If it's okay for that data to be totally opaque in Python, then you probably want to look into PyCObject in the Python C API. The Python str type does not support pointing at memory it doesn't own, so it sounds like that isn't an option for you. If you want to interpret it in some way so that native Python code can understand it, you'll need to use Numpy. I've written a low-level Boost.Python interface to Numpy that you might find useful in that case here: http://svn.boost.org/svn/boost/sandbox/numpy/ ...but it will still be pretty complicated. Passing raw memory into Python is hard precisely because it's very hard to do the memory management in a safe way. Jim Bosch
On Mon, Oct 25, 2010 at 1:51 PM, Marek Denis <marek@octogan.net> wrote: now I'd like to know whether it's possible to get access to the raw memory
from Python layer. Let's suppose I have the char* ptr pointer that points to the memory chunk with IPv4 packet. I would like to be able to read (and preferably write) data from Python layer (not via wrapped object that contains char* ptr and has some methods to write/read data), but without making any memory copies.
Would it work for you to create a 'buffer' object in your C++ code? I haven't used that myself, but my impression is that the Python 'buffer' datatype exists specifically to address such issues. http://docs.python.org/release/2.5.4/lib/typesseq.html
On 27.10.2010 15:02, Nat Linden wrote:
Would it work for you to create a 'buffer' object in your C++ code? I haven't used that myself, but my impression is that the Python 'buffer' datatype exists specifically to address such issues.
This sounds very interesting, especially it has correspondencing functions in C API. I'll investigate it a bit deeper, for sure. Thank you very much! -- pozdrawiam Marek Denis
W dniu 27.10.2010 17:30, Marek pisze:
Yeah, Buffers (and especially API/C Buffer and MemoryView functions) vere what i was looking for. Thanks again! http://docs.python.org/c-api/buffer.html?highlight=buffer#PyBuffer_FromMemor... -- Marek
participants (3)
-
Jim Bosch -
Marek Denis -
Nat Linden