Looks like I was a little quick sending out that message. I found more recent postings from Tim and Guido: http://mail.python.org/pipermail/python-dev/2002-July/026408.html http://mail.python.org/pipermail/python-dev/2002-July/026413.html Slippery little beast, that buffer object. :-) I'm going to go ahead and add deprecation warnings. Neil

Neil Schemenauer writes:

Looks like I was a little quick sending out that message. I found more recent postings from Tim and Guido:

http://mail.python.org/pipermail/python-dev/2002-July/026408.html http://mail.python.org/pipermail/python-dev/2002-July/026413.html

Slippery little beast, that buffer object. :-) I'm going to go ahead and add deprecation warnings.

I used it once in combination with ctypes as buffer(a-ctypes-object) to get at the raw memory whicy ctypes objects expose via the buffer API. But it was pretty obscure, and I would happily have used an external module. Like this:

...

...

import ctypes n = ctypes.c_int(12) buffer(n) str(buffer(n)) '\x0c\x00\x00\x00'

Basically, the only serious use case is getting the bytes out of objects which support the buffer API but which *don't* offer a "get the bytes out" interface. I've just realised that I could, however, also do this via the array module:

...

...

from array import array a = array('c') a.fromstring(n) # Hey - fromstring means "from buffer API"! a.tostring() '\x0c\x00\x00\x00'

There's an extra copy in there. Disaster :-) Nope, I don't think there's a good use case after all... Paul -- This signature intentionally left blank

That's a useful thing to be able to do and the buffer object does it in a safe way. I guess that's part of the reason why the buffer object has managed to survive as long as it has.

At least the builtin buffer function should go away. Even if someone had a use for it, it would not make-up for all the time lost by all the other people trying to figure what it was good for. Raymond Hettinger

On Wed, Oct 29, 2003 at 02:41:54PM +1300, Greg Ewing wrote:

There's no doubt that the current implementation of it is unacceptably dangerous, but I haven't yet seen an argument that convinces me that it couldn't be fixed if desired.

Okay. Perhaps I am missing something but would fixing it be as simple as adding another field to the tp_as_buffer struct? /* references returned by the buffer functins are valid while * the object remains alive */ #define PyBuffer_FLAG_SAFE 1 Then in stringobject.c (and elsewhere as appropriate): static PyBufferProcs buffer_as_buffer = { (getreadbufferproc)buffer_getreadbuf, (getwritebufferproc)buffer_getwritebuf, (getsegcountproc)buffer_getsegcount, (getcharbufferproc)buffer_getcharbuf, PyBuffer_FLAG_SAFE, }; Then change bufferobject so that it can only be created from objects that set PyBuffer_FLAG_SAFE. Neil

Neil Schemenauer

Okay. Perhaps I am missing something but would fixing it be as simple as adding another field to the tp_as_buffer struct?

/* references returned by the buffer functins are valid while * the object remains alive */ #define PyBuffer_FLAG_SAFE 1

Then in stringobject.c (and elsewhere as appropriate):

static PyBufferProcs buffer_as_buffer = { (getreadbufferproc)buffer_getreadbuf, (getwritebufferproc)buffer_getwritebuf, (getsegcountproc)buffer_getsegcount, (getcharbufferproc)buffer_getcharbuf, PyBuffer_FLAG_SAFE, };

Then change bufferobject so that it can only be created from objects that set PyBuffer_FLAG_SAFE.

As the essence of the solution, I think that sounds good! I think that the following should also be done: * Update the docs for the buffer functions to indicate that these are *short term* pointers, that are not guaranteed once *any* Python code is called. * Add new public buffer functions with "LongTerm" in the name (and docs that buffer is valid as long as the object). These check the flag as you propose. * Buffer object uses new LongTerm buffer functions. It points out that the buffer object itself is less at fault than the interface. I'm trying to short-circuit bugs in external extension modules that use the buffer functions without realizing the subtle assumptions made. Mark.

Neil Schemenauer:

Okay. Perhaps I am missing something but would fixing it be as simple as adding another field to the tp_as_buffer struct?

/* references returned by the buffer functins are valid while * the object remains alive */ #define PyBuffer_FLAG_SAFE 1

That's completely different from what I had in mind, which was: (1) Keep a reference to the base object in the buffer object, and (2) Use the buffer API to fetch a fresh pointer from the base object each time it's needed. Is there some reason that still wouldn't be safe enough? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

On Thu, Oct 30, 2003 at 07:21:01AM -0800, Neil Schemenauer wrote:

I don't see any problem with that.

Okay, small problem. The hash function for the buffer object is brain damaged, in more ways than one actually: >>> import array >>> a = array.array('c') >>> b = buffer(a) >>> hash(b) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 5311)] buffer_hash (self=0x40262d00) at Objects/bufferobject.c:241 241 x = *p << 7; (gdb) l 236 return -1; 237 } 238 239 len = self->b_size; 240 p = (unsigned char *) self->b_ptr; 241 x = *p << 7; 242 while (--len >= 0) 243 x = (1000003*x) ^ *p++; 244 x ^= self->b_size; 245 if (x == -1) (gdb) p len $1 = 0 (gdb) p *p Cannot access memory at address 0x0 The buffer object has 'b_readonly' and 'b_hash' fields. If readonly is true than the object is considered hashable and once computed the hash is stored in the 'hash' field. The problem is that the buffer API doesn't provide a way to determine 'readonly'. The absence of getwritebuf() is not the same thing as being read only. The buffer() builtin always sets the 'readonly' flag! I don't think the buffer hash method can depend on the data being pointed to. There is nothing in the buffer interface that tells you if the data is immutable. The hash method could return the id of the buffer object but I'm not sure how useful that would be. Neil

On Wed, Oct 29, 2003 at 08:57:19AM +0100, Troels Walsted Hansen wrote:

Raymond Hettinger wrote:

...

At least the builtin buffer function should go away. Even if someone had a use for it, it would not make-up for all the time lost by all the other people trying to figure what it was good for.

I trust you will preserve the functionality though?

I have used the buffer() function to achieve great leaps in performance in applications which send data from a string buffer to a socket. Slicing kills performance in this scenario once buffer sizes get beyond a few 100 kB.

Below is example from an asyncore.dispatcher subclass. This code sends chunks with maximum size, without ever slicing the buffer.

def handle_write(self): if self.buffer_offset: sent = self.send(buffer(self.buffer, self.buffer_offset)) else: sent = self.send(self.buffer) self.buffer_offset += sent if self.buffer_offset == len(self.buffer): del self.buffer

Twisted uses buffer() similarly. It originally sliced, by a company using the library complained of performance problems. Switching to buffer() alleviated those problems. Jp

Thus, my preference is to fix the buffer object by fixing the interface as much as possible.

Here is a sketch of a solution, incorporating both Neil and Greg's ideas:

Hang on, didn't we already go through the process of designing a new buffer interface not long ago? What was decided about the results of that? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

"Mark Hammond" writes:

That would work, be less intrusive, and allow all existing code to work unchanged. My only concern is that it does not go anywhere towards fixing the buffer interface itself.

I think that is a different issue entirely. While it may be interesting and important, can we at least try to keep them separate? Cheers, mwh -- This is the fixed point problem again; since all some implementors do is implement the compiler and libraries for compiler writing, the language becomes good at writing compilers and not much else! -- Brian Rogoff, comp.lang.functional

"Mark Hammond" writes:

Michael Hudson

...

"Mark Hammond" writes:

...

That would work, be less intrusive, and allow all existing code to work unchanged. My only concern is that it does not go anywhere towards fixing the buffer interface itself.

I think that is a different issue entirely. While it may be interesting and important, can we at least try to keep them separate?

I don't see how. The only problem I see is in the buffer interface. We could worm around the buffer interface problem in the buffer object, but I don't see how that is keeping them separate. Am I missing something?

Well, there are two things people complain about a) the buffer INTERFACE b) the buffer OBJECT are the issues plaguing both the same? I wasn't under the impression they were. It's entirely possible I'm wrong, though. Cheers, mwh -- [1] If you're lost in the woods, just bury some fibre in the ground carrying data. Fairly soon a JCB will be along to cut it for you - follow the JCB back to civilsation/hitch a lift. -- Simon Burr, cam.misc

On Fri, Oct 31, 2003 at 09:21:06AM +1100, Mark Hammond wrote:

Thus, my preference is to fix the buffer object by fixing the interface as much as possible.

Here is a sketch of a solution, incorporating both Neil and Greg's ideas:

* Type object gets a new flag - TP_HAS_BUFFER_INFO, corresponding to a new 'getbufferinfoproc' slot in the PyBufferProcs structure (note - a function pointer, not static flags as Neil suggested)

* New function 'getbufferinfoproc' returns a bitmask - Py_BUFFER_FIXED is one (and currently the only) flag that can be returned.

What does this flag mean? To my mind, there are several different types of memory buffers and the buffer interface does not distinguish between all of them. Is the size and position of the buffer fixed? Is the buffer immutable (it may be readonly by the buffer object but writable via some other mechanism)? The first question can be avoided by using Greg's idea of always refreshing the size and position. The second question cannot be answered using the current interface. I supposed if the buffer is immutable then it is implied that the its size and position is fixed.

* New buffer functions PyObject_AsFixedCharBuffer, etc. These check the new flag (and a type lacking TP_HAS_BUFFER_INFO is assumed to *not* be fixed)

* Buffer object keeps a reference to the existing object (as it does now). Its getbufferinfoproc delegates to the underlying object.

* Buffer object *never* keeps a pointer to the buffer - only to the object. Functions like tp_hash always re-fetch the buffer on demand. The buffer returned by the buffer object is then guaranteed to be as reliable as the underlying object. (This may be a semantic issue with hash(), but conceptually seems fine. Potential solution here - add Py_BUFFER_READONLY as a buffer flag, then hash() semantics could do the right thing)

You can't use the base objects hash if the buffer has a explicit size of offset. Neil