Mailman 3 Omit Py_buffer struct from Stable ABI for Python 3.2? - Python-Dev

Omit Py_buffer struct from Stable ABI for Python 3.2?

older
[issue8033] sqlite: broken long...

Nick Coghlan

5 Jan 2011 5 Jan '11

12:31 p.m.

Currently [1], the implementation and the documentation for PEP 3118's Py_buffer struct don't line up (there's an extra field in the implementation that the PEP doesn't mention). Accordingly, Mark and I think it may be a good idea to leave this structure (and possibly related APIs) out of the stable ABI for the 3.2 release. I don't *think* it needs changing, but I'm not 100% certain until we finish working through the problem and realign the implementation and documentation. Applications and extension modules that use this interface would still work - they would just have to wait until 3.3 before they could consider migrating to the stable ABI. Regards, Nick. [1] http://bugs.python.org/issue10181 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Show replies by thread

Mark Dickinson

5 Jan 5 Jan

12:55 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Wed, Jan 5, 2011 at 12:31 PM, Nick Coghlan wrote:

...

Currently [1], the implementation and the documentation for PEP 3118's Py_buffer struct don't line up (there's an extra field in the implementation that the PEP doesn't mention).

I think there are actually two such fields: smalltable and obj. The need for obj is a little ugly: as far as I can tell, it's meaningless for a 3rd-party object that wants to export buffers---it's only really used by the memoryview object and by internal Python types. Mark

Antoine Pitrou

1:13 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Wed, 5 Jan 2011 12:55:55 +0000 Mark Dickinson wrote:

...

On Wed, Jan 5, 2011 at 12:31 PM, Nick Coghlan wrote:

...
Currently [1], the implementation and the documentation for PEP 3118's Py_buffer struct don't line up (there's an extra field in the implementation that the PEP doesn't mention).

I think there are actually two such fields: smalltable and obj.

The need for obj is a little ugly: as far as I can tell, it's meaningless for a 3rd-party object that wants to export buffers---it's only really used by the memoryview object and by internal Python types.

I don't think it's ugly. It's the only way to know which object exported a Py_buffer. Otherwise you have to track the information separately, which is quite a bit uglier (especially when in conjunction with PyArg_ParseTuple and friends). Regards Antoine.

Mark Dickinson

2:03 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Wed, Jan 5, 2011 at 1:13 PM, Antoine Pitrou wrote:

...

On Wed, 5 Jan 2011 12:55:55 +0000 Mark Dickinson wrote:

...
The need for obj is a little ugly: as far as I can tell, it's meaningless for a 3rd-party object that wants to export buffers---it's only really used by the memoryview object and by internal Python types.

I don't think it's ugly. It's the only way to know which object exported a Py_buffer. Otherwise you have to track the information separately, which is quite a bit uglier (especially when in conjunction with PyArg_ParseTuple and friends).

Maybe I'm misunderstanding. What's the responsibility of a buffer export w.r.t. the obj field---i.e., what should 3rd party code be filling that obj field with in a call to getbuffer? It looks to me as though it's really the memoryview object that needs this information; that it doesn't belong in the Py_buffer struct. Isn't that what the 'base' field in PyMemoryViewObject in PEP 3118 was supposed to be for? Though I notice that that field is unused in the actual PyMemoryViewObject in Include/memoryobject.h. Mark

Mark Dickinson

2:04 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Wed, Jan 5, 2011 at 2:03 PM, Mark Dickinson wrote:

...

Maybe I'm misunderstanding. What's the responsibility of a buffer export w.r.t. the obj field---i.e., what should 3rd party code be

Grr. *buffer exporter*, not *buffer export*. Mark

Antoine Pitrou

2:24 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Wed, 5 Jan 2011 14:03:41 +0000 Mark Dickinson wrote:

...

On Wed, Jan 5, 2011 at 1:13 PM, Antoine Pitrou wrote:

...
On Wed, 5 Jan 2011 12:55:55 +0000 Mark Dickinson wrote:

...
The need for obj is a little ugly: as far as I can tell, it's meaningless for a 3rd-party object that wants to export buffers---it's only really used by the memoryview object and by internal Python types.

I don't think it's ugly. It's the only way to know which object exported a Py_buffer. Otherwise you have to track the information separately, which is quite a bit uglier (especially when in conjunction with PyArg_ParseTuple and friends).

Maybe I'm misunderstanding. What's the responsibility of a buffer export w.r.t. the obj field---i.e., what should 3rd party code be filling that obj field with in a call to getbuffer?

It would let PyBuffer_FillInfo() do the job. If it doesn't want to, it must put itself in that field, and increment its reference count.

...

It looks to me as though it's really the memoryview object that needs this information;

No, anyone wanting to release a buffer without keeping separate track of the original object needs it. As I said, this also applies to users of PyArg_ParseTuple and friends ("s*", "y*" etc. format codes).

...

Isn't that what the 'base' field in PyMemoryViewObject in PEP 3118 was supposed to be for?

Perhaps, but practice (implementing "s*" etc.) suggested it was useful in other cases. That field ('base') is removed in 3.2. Regards Antoine.

Nick Coghlan

4:07 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

On Thu, Jan 6, 2011 at 12:03 AM, Mark Dickinson wrote:

...

Maybe I'm misunderstanding. What's the responsibility of a buffer export w.r.t. the obj field---i.e., what should 3rd party code be filling that obj field with in a call to getbuffer?

It should be a pointer to the object (with the reference count incremented appropriately). GetBuffer/ReleaseBuffer should actually manage it automatically, but I'd have to look at the code to make sure that is the case (and, if it isn't, there may be backwards compatibility implications in fixing it).

...

It looks to me as though it's really the memoryview object that needs this information; that it doesn't belong in the Py_buffer struct. Isn't that what the 'base' field in PyMemoryViewObject in PEP 3118 was supposed to be for? Though I notice that that field is unused in the actual PyMemoryViewObject in Include/memoryobject.h.

If nothing else, PyObject_ReleaseBuffer needs it - otherwise the function signature would need to include a separate argument to tell it who the buffer belongs to (so it can find the appropriate function pointer to call). The implementation makes sense (since every call to GetBuffer needs to be paired with a corresponding call to ReleaseBuffer, it makes sense to keep the object reference inside the Py_buffer struct), but the fact the documentation was never corrected suggests there are going to be plenty of broken implementations of the protocol kicking around, potentially even in the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Stefan Behnel

6:21 p.m.

New subject: Omit Py_buffer struct from Stable ABI for Python 3.2?

Mark Dickinson, 05.01.2011 13:55:

...

On Wed, Jan 5, 2011 at 12:31 PM, Nick Coghlan wrote:

...
Currently [1], the implementation and the documentation for PEP 3118's Py_buffer struct don't line up (there's an extra field in the implementation that the PEP doesn't mention).

I think there are actually two such fields: smalltable and obj.

The need for obj is a little ugly: as far as I can tell, it's meaningless for a 3rd-party object that wants to export buffers---it's only really used by the memoryview object and by internal Python types.

Not at all. It's the reason why some of the buffer API functions could be changed to a simpler signature after earlier versions of the PEP had been written. Stefan

4859

Age (days ago)

4859

Last active (days ago)

List overview

Download

7 comments

4 participants

participants (4)

Antoine Pitrou
Mark Dickinson
Nick Coghlan
Stefan Behnel

Omit Py_buffer struct from Stable ABI for Python 3.2?

tags

participants (4)