[New-bugs-announce] [issue35285] Make Proactor api extensible for reasonably any file handle

[Ignas Brašiškis]

New submission from Ignas Brašiškis <ignas.brasiskis at gmail.com>:

There was old issue (https://bugs.python.org/issue30539) requesting extension ability at proactor loop for third party libraries. It was rejected as it was asking to broad functionality and not particular api. This issue purpose is to discuss more concrete api needed to bring proactor functionally at least on par with selector.

While loop covers most valid cases it still has no ability to run any third party libraries with file descriptors. As a consequence there is workarounds with unofficial api and handle wrappers to expose and call private proactor methods (https://github.com/m-labs/asyncserial/blob/master/asyncserial/asyncserial.py#L172). 

Selector basically has the following api:

loop.add_reader(fd, callback, *args)
loop.remove_reader(fd)
loop.add_writer(fd, callback, *args)
loop.remove_writer(fd)

Nature of selector makes it easy for user to provide those operations on descriptor: make read from when it is available or make write from somewhere when you can also cancel operation.

Proactor on the other hand need to be told what and where to copy, and run callback when it's done (probably with error, also buffer passed to callback ???). It also needs to have ability to cancel operations.
 
One possibility of such api could look like this:
loop.start_read_operation(fd, buffer, done_callback, *args)
where done_callback = function(buffer, bytes_read, error)
loop.cancel_read_operation(fd, operation)
loop.start_write_operation(fd, buffer, done_callback, *args)
where done_callback = function(bytes_written, error)
loop.cancel_write_operation(fd, operation)

There probably can be a plenty refinement here to the naming buffer,
or callback workings, but in general there should be start and cancel operations. This api might even be implemented for selectors (selectors might imitate behaviour with custom writer), thought probably it this usage might be discouraged in favour of add_writer/reader logic.

As examples of this api besides IOCP (which is implementation in question here) there is linux aio (http://man7.org/linux/man-pages/man2/io_submit.2.html) api which uses similar proactor concepts.

This api obviously can be wrapped to higher level api with futures, but this probably should remain low level functionality where it is responsibility of library implementer to do right thing.

Usage is pretty limited here: mainly writing file descriptors and devices/ttys asynchronously on Win32 platform. Python itself is not exactly systems programming language, but it convenient for accessing some OS features. Sockets and pipes are covered by standard library. But still if could prevent plenty of hacks and give library writers and odd few users place to hang on.

----------
components: asyncio
messages: 330160
nosy: Ignas Brašiškis, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: Make Proactor api extensible for reasonably any file handle
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35285>
_______________________________________