[Python-ideas] Async API: some code to review

[Yury Selivanov]

On 2012-10-29, at 3:32 PM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:

> On Mon, Oct 29, 2012 at 9:10 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> On 2012-10-29, at 2:02 PM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
>> 
>>> Pollster has to support any object as file descriptor.
>>> The use case is ZeroMQ sockets: they are implemented at user level and
>>> socket is just some opaque structure wrapped by Python object.
>>> ZeroMQ has own poll function to process zmq sockets as well as regular
>>> sockets/pipes/files.
>> 
>> Well, you can use epoll/select/kqueue or whatever else with ZMQ sockets.
>> Just get the underlying file descriptor with 'getsockopt', as described
>> here: http://api.zeromq.org/master:zmq-getsockopt#toc20
> 
> Well, will take a look. I used zmq poll only.
> It works for reading only, not for writing, right?
> As I know you use proactor pattern.
> Can reactor has some problems with this approach?
> May embedded 0MQ poll be more effective via some internal optimizations?

It's officially documented and supported approach.  We haven't seen any
problem with it so far.

It works both for reading and writing, however, 99.9% EAGAIN errors occur
on reading.  When you 'send', it just stores your data in an internal
buffer and sends it itself.  When you 'read', well, if there is no data
in buffers you get EAGAIN.

As for the performance -- I haven't tested 'zmq.poll' vs (let's say) epoll, 
but I doubt there is any significant difference.  And if I would want to 
write a benchmark, I'd first compare pure blocking ZMQ sockets vs 
non-blocking ZMQ sockets with ZMQ.poll, as ZMQ uses threads heavily, and
probably, blocking threads-driven IO is faster then non-blocking with 
polling (when FDs count is relatively small), no matter whether you use
zmq.poll or epoll/etc.

-
Yury