On 11/4/12 8:11 AM, Ben Darnell wrote:
>
> The extra system calls add up. The interface of Tornado's IOLoop was
> based on epoll (where the internal state is roughly a mapping {fd:
> event_set}), so it requires more register/unregister operations when
> running on kqueue (where the internal state is roughly a set of (fd,
> event) pairs). This shows up in benchmarks of the HTTPServer; it's
> faster on platforms with epoll than platforms with kqueue. In
> low-concurrency scenarios it's actually faster to use select() even
> when kqueue is available (or maybe that's a mac-specific quirk).
>
>

Just so I have this right, you're saying that HTTPServer is slower on
kqueue because of the IOLoop design, yes?

Yes. When the server processes a request and switches from listening for readability to listening for writability, with epoll it's one call directly into the C module to set the event mask for the socket. With kqueue something in the IOLoop must store the previous state and generate the two separate actions to remove the read listener and add a write listener. I misspoke when I mentioned system call; the difference is actually the amount of python code that must be run to call the right C functions. This would get a lot better if more of the IOLoop were written in C.

I've just looked over the epoll interface and I see at least one huge
difference compared to kqueue: it requires a system call for each fd
registration event. With kevent() you can accumulate thousands of
registrations, shove them into a single kevent() call and get thousands
of events out. It's a little all-singing-all-dancing, but it's hard to
imagine a way to do it using fewer system calls. 8^)

True, although whenever I've tried to be clever and batch up kevent calls I haven't gotten the performance I'd hoped for because system calls aren't actually that expensive in comparison to python opcodes. Also at least some versions of Mac OS have a bug where you can only pass one event at a time.

-Ben

-Sam

_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
http://mail.python.org/mailman/listinfo/python-ideas