Can we implement eventfd(2) as documented here <https://man7.org/linux/man-pages/man2/eventfd.2.html>? It would only be available on the Linux platform, and one of the benefits would be the ability to create synchronisation primitives on said platform that can block on normal threads, and be awaited on in coroutines (without busy looping inside said coroutine). Currently the best place I can think of to put it would be in one of the Networking and Interprocess Communication modules (possibly `select` or `socket`?). The fact that it's Linux only shouldn't be an issue, since much of the contents of `select` is OS dependent.
On 16/06/2020 06:56, doodspav@gmail.com wrote:
Can we implement eventfd(2) as documented here <https://man7.org/linux/man-pages/man2/eventfd.2.html>?
EventFD is available on PyPI: https://pypi.org/project/eventfd/ -- Rhodri James *-* Kynesim Ltd
+1. Although it already exists as a package on PyPI, I've seen a non-negligible amount of user demand for a synchronization primitive that works as described above for asyncio. Adding eventfd() to the stdlib seems like a step towards eventually implementing something like that in an effective manner, and I don't think the maintenance burden would be especially high. I could also see a significant potential use case for multiprocessing (CC'd Antoine in case he has anything to add there).
The select module seems like it would be a reasonable location. On Tue, Jun 16, 2020 at 8:42 AM <doodspav@gmail.com> wrote:
Hi As said earlier, it's on PyPi: https://pypi.org/project/eventfd/ However, it's only at version 0.2 and hasn't been changed since 2016. Digging a bit deeper, there's an unresolved issue that suggests more work is required. Issue: Race condition in "set" can lead to a bad state https://github.com/palaviv/eventfd/issues/2 Perhaps a good first step would be a code review of palaviv's enventfd, and finding a fix for this issue. -- Jonathan
I've looked over the PyPi version of `eventfd`, and I feel that it is trying to be more than just a thin wrapper. The attempt to make it cross platform has given it more fields than it requires, as has the attempt to wrap it as a semaphore, and with that comes added complexity. The `eventfd` that I was suggesting be added would in my mind look something more like this: `eventfd`: - `__init__(...)` - `read(...)` - `write(...)` - `close(...)` - `closed(...)` - `fileno(...)` + some extra dunder methods We could possibly add a method to get the flags that were passed to `__init__`, and I think pickling should be disabled (while it does get inherited through `fork(2)` and likely through `execve(2)` depending on flags, I think the fd's integer value in each process can be different, though I could be wrong here). From that, one could then build more complex classes, but the base primitive should be as simple as possible. Instead of trying to fix the PyPi version, I can write a proof of concept of what I described above? doodspav
Instead of trying to fix the PyPi version, I can write a proof of concept of what I described above?
That would definitely be useful, and would help us get a more solid idea of what exactly is being proposed. I also agree that a potential version contained in the select module should be as simple as possible and rather low-level (more so than the linked PyPI project), as it's likely going to be there primarily for building other tools with. There's no guarantee of course that it will ultimately be accepted, but a proof of concept implementation would be a highly productive first step, IMO. On Tue, Jun 16, 2020 at 2:58 PM <doodspav@gmail.com> wrote:
This is my first attempt at a C implementation: https://gist.github.com/doodspav/5d96c09696fa3ef1c89b4d6426ddc338 It stores the `fd` as an atomic int, since I think threads not being able to run in parallel is an implementation detail, and I shouldn't be designing code around it? I'm not sure if all Python platforms have a C compiler that supports stdatomic, but I'll cross that bridge when I get to it. Right now `read`/`write` raise `BlockingIOError` if the `fd` is set to non-blocking, and the operation would block - I was considering adding kwargs to both operations to give the option to return `None`/`False`, but kwargs in C are annoying, so maybe at a later stage. To build: - you'll probably need a lib like `libpython3-dev` - save the gist into a file called `py_eventfd.c` - create a file called `setup.py` and paste the following into it: ```py from distutils.core import setup, Extension setup(name="eventfd", version="1.0", ext_modules=[Extension("eventfd", ["py_eventfd.c"])]) ``` - run `python3 setup.py build` and `cd b*/l*` - from there you can `import eventfd` or `from eventfd import eventfd` It should go without saying that this will only compile on Linux.
Agreed the PyPi version is not worth investing in. Do you see this as add eventfd as os.eventfd? Add definitons for the flags and that is it? Turning that into a semaphore would then be easy to do on top of that os.eventf call. You could even do the eventfd PoC using ctypes. Barry
On 16/06/2020 07.56, doodspav@gmail.com wrote:
We usually expose low-level, file descriptor-related functions in the os module and then provide high-level wrappers in Python. The approach is most flexible and allows 3rd parties to build on top of the raw file descriptor, too. I opened a BPO and created https://github.com/python/cpython/pull/20930 to implement a low-level interface to glibc's eventfd() function. Christian
I opened a BPO and created https://github.com/python/cpython/pull/20930 to implement a low-level interface to glibc's eventfd() function.
Perfect. What's the procedure now for getting that PR accepted?
Perfect. What's the procedure now for getting that PR accepted?
Next, it will go through a PR review process prior to being included for Python 3.10. Since it's being implemented as a wrapper for an existing OS API, a PR and bpo issue should be adequate (as opposed to major change proposals that typically require a PEP and/or in-depth discussion.) For more details on our GitHub PR process, see the following section from the devguide: https://devguide.python.org/pullrequest/. On Wed, Jun 17, 2020 at 6:58 AM <doodspav@gmail.com> wrote:
On 16/06/2020 06:56, doodspav@gmail.com wrote:
Can we implement eventfd(2) as documented here <https://man7.org/linux/man-pages/man2/eventfd.2.html>?
EventFD is available on PyPI: https://pypi.org/project/eventfd/ -- Rhodri James *-* Kynesim Ltd
+1. Although it already exists as a package on PyPI, I've seen a non-negligible amount of user demand for a synchronization primitive that works as described above for asyncio. Adding eventfd() to the stdlib seems like a step towards eventually implementing something like that in an effective manner, and I don't think the maintenance burden would be especially high. I could also see a significant potential use case for multiprocessing (CC'd Antoine in case he has anything to add there).
The select module seems like it would be a reasonable location. On Tue, Jun 16, 2020 at 8:42 AM <doodspav@gmail.com> wrote:
Hi As said earlier, it's on PyPi: https://pypi.org/project/eventfd/ However, it's only at version 0.2 and hasn't been changed since 2016. Digging a bit deeper, there's an unresolved issue that suggests more work is required. Issue: Race condition in "set" can lead to a bad state https://github.com/palaviv/eventfd/issues/2 Perhaps a good first step would be a code review of palaviv's enventfd, and finding a fix for this issue. -- Jonathan
I've looked over the PyPi version of `eventfd`, and I feel that it is trying to be more than just a thin wrapper. The attempt to make it cross platform has given it more fields than it requires, as has the attempt to wrap it as a semaphore, and with that comes added complexity. The `eventfd` that I was suggesting be added would in my mind look something more like this: `eventfd`: - `__init__(...)` - `read(...)` - `write(...)` - `close(...)` - `closed(...)` - `fileno(...)` + some extra dunder methods We could possibly add a method to get the flags that were passed to `__init__`, and I think pickling should be disabled (while it does get inherited through `fork(2)` and likely through `execve(2)` depending on flags, I think the fd's integer value in each process can be different, though I could be wrong here). From that, one could then build more complex classes, but the base primitive should be as simple as possible. Instead of trying to fix the PyPi version, I can write a proof of concept of what I described above? doodspav
Instead of trying to fix the PyPi version, I can write a proof of concept of what I described above?
That would definitely be useful, and would help us get a more solid idea of what exactly is being proposed. I also agree that a potential version contained in the select module should be as simple as possible and rather low-level (more so than the linked PyPI project), as it's likely going to be there primarily for building other tools with. There's no guarantee of course that it will ultimately be accepted, but a proof of concept implementation would be a highly productive first step, IMO. On Tue, Jun 16, 2020 at 2:58 PM <doodspav@gmail.com> wrote:
This is my first attempt at a C implementation: https://gist.github.com/doodspav/5d96c09696fa3ef1c89b4d6426ddc338 It stores the `fd` as an atomic int, since I think threads not being able to run in parallel is an implementation detail, and I shouldn't be designing code around it? I'm not sure if all Python platforms have a C compiler that supports stdatomic, but I'll cross that bridge when I get to it. Right now `read`/`write` raise `BlockingIOError` if the `fd` is set to non-blocking, and the operation would block - I was considering adding kwargs to both operations to give the option to return `None`/`False`, but kwargs in C are annoying, so maybe at a later stage. To build: - you'll probably need a lib like `libpython3-dev` - save the gist into a file called `py_eventfd.c` - create a file called `setup.py` and paste the following into it: ```py from distutils.core import setup, Extension setup(name="eventfd", version="1.0", ext_modules=[Extension("eventfd", ["py_eventfd.c"])]) ``` - run `python3 setup.py build` and `cd b*/l*` - from there you can `import eventfd` or `from eventfd import eventfd` It should go without saying that this will only compile on Linux.
Agreed the PyPi version is not worth investing in. Do you see this as add eventfd as os.eventfd? Add definitons for the flags and that is it? Turning that into a semaphore would then be easy to do on top of that os.eventf call. You could even do the eventfd PoC using ctypes. Barry
On 16/06/2020 07.56, doodspav@gmail.com wrote:
We usually expose low-level, file descriptor-related functions in the os module and then provide high-level wrappers in Python. The approach is most flexible and allows 3rd parties to build on top of the raw file descriptor, too. I opened a BPO and created https://github.com/python/cpython/pull/20930 to implement a low-level interface to glibc's eventfd() function. Christian
I opened a BPO and created https://github.com/python/cpython/pull/20930 to implement a low-level interface to glibc's eventfd() function.
Perfect. What's the procedure now for getting that PR accepted?
Perfect. What's the procedure now for getting that PR accepted?
Next, it will go through a PR review process prior to being included for Python 3.10. Since it's being implemented as a wrapper for an existing OS API, a PR and bpo issue should be adequate (as opposed to major change proposals that typically require a PEP and/or in-depth discussion.) For more details on our GitHub PR process, see the following section from the devguide: https://devguide.python.org/pullrequest/. On Wed, Jun 17, 2020 at 6:58 AM <doodspav@gmail.com> wrote:
participants (6)
-
Barry Scott -
Christian Heimes -
doodspav＠gmail.com -
Jonathan Fine -
Kyle Stanley -
Rhodri James