I would like to use (again) the buffer interface, and I have some suggestions/questions. - Only extension types can implement the buffer interface, no way for a python class. Maybe a magic method __buffer__(self) could be invented for this purpose? - Why does the builtin function buffer() always return readonly buffers, even for read/write objects? Shouldn't there also be a readwrite_buffer() function, or should buffer() return read/write buffers for read/write objects? - My bug report 216405 on sourceforge is still open (well, it is marked as closed, but it went into PEP 42) http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470 Regards, Thomas
I would like to use (again) the buffer interface, and I have some suggestions/questions.
- Only extension types can implement the buffer interface, no way for a python class. Maybe a magic method __buffer__(self) could be invented for this purpose?
How could it be made safe? The buffer interface makes raw memory addresses available. Letting a Python class decide on what addresses to use opens a big hole for abuse.
- Why does the builtin function buffer() always return readonly buffers, even for read/write objects? Shouldn't there also be a readwrite_buffer() function, or should buffer() return read/write buffers for read/write objects?
It's a lifetime issue. You can hold on to the object returned by buffer() long after the actual memory it points to is recycled for some other purpose, and while *reading* that memory is not such a big deal (assuming it is still part of your address space, you'll just get garbage), allowing it to be *written* is again an invitation to disaster.
- My bug report 216405 on sourceforge is still open (well, it is marked as closed, but it went into PEP 42) http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470
I still feel that your bug report is based on the wrong assumption about what the buffer API should do. Thomas, please first explain what you want to *do* with the buffer interface. Some of the buffer API was a mistake. It *appears* to be an interface for allocating and managing buffers, while in actuality it is only intended to provide access to buffered data that is managed by some C code. You're probably better off using the array module to manage buffers. --Guido van Rossum (home page: http://www.python.org/~guido/)
[no need to cc me directly]
I would like to use (again) the buffer interface, and I have some suggestions/questions.
- Only extension types can implement the buffer interface, no way for a python class. Maybe a magic method __buffer__(self) could be invented for this purpose?
How could it be made safe? The buffer interface makes raw memory addresses available. Letting a Python class decide on what addresses to use opens a big hole for abuse. class C: ... def __buffer__(self): # self.my_buffer is an object exposing the buffer interface return buffer(self.my_buffer) or: return self.my_buffer
- Why does the builtin function buffer() always return readonly buffers, even for read/write objects? Shouldn't there also be a readwrite_buffer() function, or should buffer() return read/write buffers for read/write objects?
It's a lifetime issue. You can hold on to the object returned by buffer() long after the actual memory it points to is recycled for some other purpose, and while *reading* that memory is not such a big deal (assuming it is still part of your address space, you'll just get garbage), allowing it to be *written* is again an invitation to disaster. Lifetimes are managed by refcounts, so it's a refcount issue, at least as long as every object exposing the buffer interface _guarantees_ that the memory address and size does not change. (Which does not seem the case for array objects).
- My bug report 216405 on sourceforge is still open (well, it is marked as closed, but it went into PEP 42) http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470
I still feel that your bug report is based on the wrong assumption about what the buffer API should do.
I only tried to fix the refcount issue.
Thomas, please first explain what you want to *do* with the buffer interface. Some of the buffer API was a mistake. It *appears* to be an interface for allocating and managing buffers, while in actuality it is only intended to provide access to buffered data that is managed by some C code. You're probably better off using the array module to manage buffers.
I want to expose memory blocks to C, wrapped in python objects/classes. These memory blocks could come from builtin python types: strings, unicode strings, array objects, mmap'd files, ... Thomas
Thomas Heller wrote:
Lifetimes are managed by refcounts, so it's a refcount issue, at least as long as every object exposing the buffer interface _guarantees_ that the memory address and size does not change. (Which does not seem the case for array objects).
Thomas, please first explain what you want to *do* with the buffer interface. Some of the buffer API was a mistake. It *appears* to be an interface for allocating and managing buffers, while in actuality it is only intended to provide access to buffered data that is managed by some C code. You're probably better off using the array module to manage buffers.
I want to expose memory blocks to C, wrapped in python objects/classes. These memory blocks could come from builtin python types: strings, unicode strings, array objects, mmap'd files, ...
If you are talking about a memory object, then I'm in agreement with you, Thomas. I'd like to see a memory object that allocates and deallocates blocks of memory and exports a pointer to its memory. It could also set privileges such are read/write, etc. Its interface would be identical, or at least similar, to the mmap object, so that they could be easily interchanged. -- Dr. Paul Barrett Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Group FAX: 410-338-4767 Baltimore, MD 21218
From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On Behalf Of Thomas Heller Sent: Thursday, 19 April 2001 2:43 AM
Better late than never!
I'd like to see a memory object that allocates and deallocates blocks of memory and exports a pointer to its memory. It could also set privileges such are read/write, etc
It's all there, in bufferobject.c. Except that the functions are not exposed to python...
I'm with Thomas too. I could have used this a number of times, and have even resorted to simple methods in my own .pyd files that wrap these APIs - eg, win32file.AllocateReadBuffer() used for overlapped IO. So while I think Thomas' bug is valid, I also understand we have to somehow handle the issue the array module has. Mark.
participants (4)
-
Guido van Rossum
-
Mark Hammond
-
Paul Barrett
-
Thomas Heller