PEP 688: Making the buffer protocol accessible in Python
![](https://secure.gravatar.com/avatar/57da4d2e2a527026baaaab35e6872fa5.jpg?s=120&d=mm&r=g)
It's been a longstanding problem in typeshed that we cannot fully represent the C buffer protocol in stubs, and we have to make do with annotations of `bytes` or unions of common buffer types. I just submitted PEP 688 to add a new `types.Buffer` type that will fix this problem. A preview of the rendered PEP is available at https://pep-previews--2549.org.readthedocs.build/pep-0688/. The PR is at https://github.com/python/peps/pull/2549. Summary: * types.Buffer is added to CPython, using an isinstance hook to check for the presence of the buffer protocol. * In stubs, types.Buffer can be used as a base class. * Static type checkers should not need to special case the new class in any way. Please let me know if you have any feedback.
![](https://secure.gravatar.com/avatar/7b5bbadd9baf9c6b33a053e9687ce97e.jpg?s=120&d=mm&r=g)
Thanks for moving this forward! Not having an actual way to type buffers and having memoryview duck type to bytes are both clearly problematic and it's fantastic that we may soon have a solution here. However, there is one aspect of current behaviour that I think is *not* particularly problematic, which is that we currently allow bytearray duck type to bytes (they're basically completely compatible, which is much more than int is to float). For example, see: https://github.com/niklasf/python-chess/blob/489b49d45ebd56a93ad42a3415bbffd... https://github.com/mystor/git-revise/blob/0c83d7902b2e57344326fb96f9f67cbacf... https://github.com/urllib3/urllib3/blob/e16beb210c03c6f5ce4e0908bddb6556442b... (these examples wouldn't be better served by a Buffer type) The main argument afaict for removing compatibility between just bytearray and bytes is removing a special case, but it's a special case everyone has implemented and that users rely on. And I'm not sure we'd want to get rid of special casing entirely, as you mention yourself here: https://github.com/python/mypy/pull/12661#issuecomment-1107534322 The PEP only talks about compatibility between bytes and memoryview (which is clearly problematic). It would be good to include some words that talk more directly about compatibility between bytearray and bytes and to what extent (if any) type checkers should continue their current special casing. (I personally lean on the side of keeping bytearray duck type compatible with bytes, but don't feel strongly) On Sat, 23 Apr 2022 at 10:27, Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/57da4d2e2a527026baaaab35e6872fa5.jpg?s=120&d=mm&r=g)
Thanks for bringing this up, I'd be interested in more opinions on whether bytearray should continue to be compatible with bytes. El sáb, 23 abr 2022 a las 15:46, Shantanu Jain (<hauntsaninja@gmail.com>) escribió:
Not exactly. For example, bytes is hashable but bytearray isn't. As a result, re.compile accepts bytes patterns but not bytearray. Several other stdlib functions also accept bytes but not bytearray, e.g. socket.getaddrinfo.
For context, these came from my draft mypy change at https://github.com/python/mypy/pull/12661#issuecomment-1107658028. Fair enough, but that's not a very long list considering the amount of code mypy-primer runs on.
That comment was specifically about mypy's check for overlapping comparisons. Also, it applies to both memoryview and bytearray, because bytes and memoryview objects can also compare equal. This is probably better handled by changing the __eq__ stubs on these types.
Sure, I can add some discussion.
![](https://secure.gravatar.com/avatar/57da4d2e2a527026baaaab35e6872fa5.jpg?s=120&d=mm&r=g)
El sáb, 23 abr 2022 a las 10:26, Jelle Zijlstra (<jelle.zijlstra@gmail.com>) escribió:
The PEP is now live at https://peps.python.org/pep-0688/
![](https://secure.gravatar.com/avatar/7b5bbadd9baf9c6b33a053e9687ce97e.jpg?s=120&d=mm&r=g)
Thanks for moving this forward! Not having an actual way to type buffers and having memoryview duck type to bytes are both clearly problematic and it's fantastic that we may soon have a solution here. However, there is one aspect of current behaviour that I think is *not* particularly problematic, which is that we currently allow bytearray duck type to bytes (they're basically completely compatible, which is much more than int is to float). For example, see: https://github.com/niklasf/python-chess/blob/489b49d45ebd56a93ad42a3415bbffd... https://github.com/mystor/git-revise/blob/0c83d7902b2e57344326fb96f9f67cbacf... https://github.com/urllib3/urllib3/blob/e16beb210c03c6f5ce4e0908bddb6556442b... (these examples wouldn't be better served by a Buffer type) The main argument afaict for removing compatibility between just bytearray and bytes is removing a special case, but it's a special case everyone has implemented and that users rely on. And I'm not sure we'd want to get rid of special casing entirely, as you mention yourself here: https://github.com/python/mypy/pull/12661#issuecomment-1107534322 The PEP only talks about compatibility between bytes and memoryview (which is clearly problematic). It would be good to include some words that talk more directly about compatibility between bytearray and bytes and to what extent (if any) type checkers should continue their current special casing. (I personally lean on the side of keeping bytearray duck type compatible with bytes, but don't feel strongly) On Sat, 23 Apr 2022 at 10:27, Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/57da4d2e2a527026baaaab35e6872fa5.jpg?s=120&d=mm&r=g)
Thanks for bringing this up, I'd be interested in more opinions on whether bytearray should continue to be compatible with bytes. El sáb, 23 abr 2022 a las 15:46, Shantanu Jain (<hauntsaninja@gmail.com>) escribió:
Not exactly. For example, bytes is hashable but bytearray isn't. As a result, re.compile accepts bytes patterns but not bytearray. Several other stdlib functions also accept bytes but not bytearray, e.g. socket.getaddrinfo.
For context, these came from my draft mypy change at https://github.com/python/mypy/pull/12661#issuecomment-1107658028. Fair enough, but that's not a very long list considering the amount of code mypy-primer runs on.
That comment was specifically about mypy's check for overlapping comparisons. Also, it applies to both memoryview and bytearray, because bytes and memoryview objects can also compare equal. This is probably better handled by changing the __eq__ stubs on these types.
Sure, I can add some discussion.
![](https://secure.gravatar.com/avatar/57da4d2e2a527026baaaab35e6872fa5.jpg?s=120&d=mm&r=g)
El sáb, 23 abr 2022 a las 10:26, Jelle Zijlstra (<jelle.zijlstra@gmail.com>) escribió:
The PEP is now live at https://peps.python.org/pep-0688/
participants (2)
-
Jelle Zijlstra
-
Shantanu Jain