buffer protocol interim solution(s)?
Hi, to share data, a number of libraries use the buffer protocol(s)[0 <https://docs.python.org/3/c-api/buffer.html>]. There seems to be a python issue open [1 <https://bugs.python.org/issue27501>], and a typing issue open [2 <https://github.com/python/typing/issues/593>]. Anyone have better interim solution(s) that work cross-library? The given work-around in [2 <https://github.com/python/typing/issues/593>] doesn't really work in a number of cases:
Union[bytes, bytearray, memoryview]
Consider if this would work with array.array, or <type 'numpy.ndarray'> or dozens of other libraries buffer protocol friendly types.
As a concrete example, numpy is adding the ndarray to that Union using this [3 <https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi#L2...> ]: _BufferType = Union[ndarray, bytes, bytearray, memoryview] Whilst this supports ndarray, it doesn't scale to the full list of third party types supporting buffer protocol(s) [opencv, pygame, pillow, ...]. Every suggestion appreciated -- even the strangest, ugliest, most horrific of hacks. Thanks. cheers, [0] https://docs.python.org/3/c-api/buffer.html [1] https://bugs.python.org/issue27501 [2] https://github.com/python/typing/issues/593 [3] https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi#L2...
Can you please provide concrete code examples? I am not a data scientist and I don't know all the libraries you are interested in. Also, the buffer protocol is very low level, I'd worry that the code wouldn't be very safe in practice. What's wrong with the occasional cast() call or # type: ignore comment? On Tue, May 19, 2020 at 00:36 René Dudfield <renesd@gmail.com> wrote:
Hi,
to share data, a number of libraries use the buffer protocol(s)[0 <https://docs.python.org/3/c-api/buffer.html>]. There seems to be a python issue open [1 <https://bugs.python.org/issue27501>], and a typing issue open [2 <https://github.com/python/typing/issues/593>].
Anyone have better interim solution(s) that work cross-library? The given work-around in [2 <https://github.com/python/typing/issues/593>] doesn't really work in a number of cases:
Union[bytes, bytearray, memoryview]
Consider if this would work with array.array, or <type 'numpy.ndarray'> or dozens of other libraries buffer protocol friendly types.
As a concrete example, numpy is adding the ndarray to that Union using this [3 <https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi#L2...> ]:
_BufferType = Union[ndarray, bytes, bytearray, memoryview]
Whilst this supports ndarray, it doesn't scale to the full list of third party types supporting buffer protocol(s) [opencv, pygame, pillow, ...].
Every suggestion appreciated -- even the strangest, ugliest, most horrific of hacks. Thanks.
cheers,
[0] https://docs.python.org/3/c-api/buffer.html [1] https://bugs.python.org/issue27501 [2] https://github.com/python/typing/issues/593 [3] https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi#L2...
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
Am 19.05.20 um 16:34 schrieb Guido van Rossum:
Can you please provide concrete code examples? I am not a data scientist and I don't know all the libraries you are interested in.
Also, the buffer protocol is very low level, I'd worry that the code wouldn't be very safe in practice.
What's wrong with the occasional cast() call or # type: ignore comment?
The lack of a buffer protocol is something that I came across multiple times now while reviewing typeshed PRs, although I haven't worked with the buffer protocol myself yet. But it seems to be a common problem that we can't solve on the typeshed level. The best we could do is provide an alias for Union[bytes, bytearray, memoryview, array.array] that we could use whenever the stdlib accepts buffer protocol object. But in my opinion it would be better to do it "right" and provide a buffer protocol type that could be used for such third-party classes as well. Of course it's possible to cast or # type: ignore whenever passing such an object, but that's hardly convenient and defeats the type checks. The caller must be sure that e.g. fcntl.ioctl() supports the buffer interface and can't rely on type checks. - Sebastian
On Tue, May 19, 2020 at 8:44 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 19.05.20 um 16:34 schrieb Guido van Rossum:
Can you please provide concrete code examples? I am not a data scientist and I don't know all the libraries you are interested in.
Also, the buffer protocol is very low level, I'd worry that the code wouldn't be very safe in practice.
What's wrong with the occasional cast() call or # type: ignore comment?
The lack of a buffer protocol is something that I came across multiple times now while reviewing typeshed PRs, although I haven't worked with the buffer protocol myself yet. But it seems to be a common problem that we can't solve on the typeshed level. The best we could do is provide an alias for Union[bytes, bytearray, memoryview, array.array] that we could use whenever the stdlib accepts buffer protocol object. But in my opinion it would be better to do it "right" and provide a buffer protocol type that could be used for such third-party classes as well.
Of course it's possible to cast or # type: ignore whenever passing such an object, but that's hardly convenient and defeats the type checks. The caller must be sure that e.g. fcntl.ioctl() supports the buffer interface and can't rely on type checks.
AFAIK mypy doesn't actually know anything about bytes except what's said about it in typeshed. So maybe we can just define a proper class representing the buffer protocol in an appropriate place (maybe stick it in types.pyi with an underscore name because it's not in the stdlib types.py) and ensure that bytes, bytearray etc. inherit from it? Would that kind of class hierarchy help? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
El mar., 19 may. 2020 a las 10:29, Guido van Rossum (<guido@python.org>) escribió:
On Tue, May 19, 2020 at 8:44 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 19.05.20 um 16:34 schrieb Guido van Rossum:
Can you please provide concrete code examples? I am not a data scientist and I don't know all the libraries you are interested in.
Also, the buffer protocol is very low level, I'd worry that the code wouldn't be very safe in practice.
What's wrong with the occasional cast() call or # type: ignore comment?
The lack of a buffer protocol is something that I came across multiple times now while reviewing typeshed PRs, although I haven't worked with the buffer protocol myself yet. But it seems to be a common problem that we can't solve on the typeshed level. The best we could do is provide an alias for Union[bytes, bytearray, memoryview, array.array] that we could use whenever the stdlib accepts buffer protocol object. But in my opinion it would be better to do it "right" and provide a buffer protocol type that could be used for such third-party classes as well.
Of course it's possible to cast or # type: ignore whenever passing such an object, but that's hardly convenient and defeats the type checks. The caller must be sure that e.g. fcntl.ioctl() supports the buffer interface and can't rely on type checks.
AFAIK mypy doesn't actually know anything about bytes except what's said about it in typeshed. So maybe we can just define a proper class representing the buffer protocol in an appropriate place (maybe stick it in types.pyi with an underscore name because it's not in the stdlib types.py) and ensure that bytes, bytearray etc. inherit from it? Would that kind of class hierarchy help?
Yes, that's the idea, but nobody so far has stepped up to actually do it. I would want it to be in typing(_extensions) with a public name so that third-party libraries can also use it. A slightly troublesome aspect is that the "buffer protocol" is C-only and there are no Python methods that correspond to it. A standard Protocol therefore doesn't work; instead we should create an empty class that bytes and friends inherit from. I proposed this in https://github.com/python/typing/issues/593#issuecomment-440327001.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
On Tue, May 19, 2020 at 10:43 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El mar., 19 may. 2020 a las 10:29, Guido van Rossum (<guido@python.org>) escribió:
On Tue, May 19, 2020 at 8:44 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 19.05.20 um 16:34 schrieb Guido van Rossum:
Can you please provide concrete code examples? I am not a data scientist and I don't know all the libraries you are interested in.
Also, the buffer protocol is very low level, I'd worry that the code wouldn't be very safe in practice.
What's wrong with the occasional cast() call or # type: ignore comment?
The lack of a buffer protocol is something that I came across multiple times now while reviewing typeshed PRs, although I haven't worked with the buffer protocol myself yet. But it seems to be a common problem that we can't solve on the typeshed level. The best we could do is provide an alias for Union[bytes, bytearray, memoryview, array.array] that we could use whenever the stdlib accepts buffer protocol object. But in my opinion it would be better to do it "right" and provide a buffer protocol type that could be used for such third-party classes as well.
Of course it's possible to cast or # type: ignore whenever passing such an object, but that's hardly convenient and defeats the type checks. The caller must be sure that e.g. fcntl.ioctl() supports the buffer interface and can't rely on type checks.
AFAIK mypy doesn't actually know anything about bytes except what's said about it in typeshed. So maybe we can just define a proper class representing the buffer protocol in an appropriate place (maybe stick it in types.pyi with an underscore name because it's not in the stdlib types.py) and ensure that bytes, bytearray etc. inherit from it? Would that kind of class hierarchy help?
Yes, that's the idea, but nobody so far has stepped up to actually do it. I would want it to be in typing(_extensions) with a public name so that third-party libraries can also use it.
A slightly troublesome aspect is that the "buffer protocol" is C-only and there are no Python methods that correspond to it. A standard Protocol therefore doesn't work; instead we should create an empty class that bytes and friends inherit from. I proposed this in https://github.com/python/typing/issues/593#issuecomment-440327001.
Hopefully the OP will be inspired by your proposal and submit a PR... -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Am 19.05.20 um 19:43 schrieb Jelle Zijlstra:
El mar., 19 may. 2020 a las 10:29, Guido van Rossum (<guido@python.org <mailto:guido@python.org>>) escribió:
AFAIK mypy doesn't actually know anything about bytes except what's said about it in typeshed. So maybe we can just define a proper class representing the buffer protocol in an appropriate place (maybe stick it in types.pyi with an underscore name because it's not in the stdlib types.py) and ensure that bytes, bytearray etc. inherit from it? Would that kind of class hierarchy help?
Yes, that's the idea, but nobody so far has stepped up to actually do it. I would want it to be in typing(_extensions) with a public name so that third-party libraries can also use it.
A slightly troublesome aspect is that the "buffer protocol" is C-only and there are no Python methods that correspond to it. A standard Protocol therefore doesn't work; instead we should create an empty class that bytes and friends inherit from. I proposed this in https://github.com/python/typing/issues/593#issuecomment-440327001.
I hadn't realized that the solution was this straight forward. Sounds like a good solution. - Sebastian
participants (4)
-
Guido van Rossum -
Jelle Zijlstra -
René Dudfield -
Sebastian Rittau