<br><div class="gmail_quote"><div class="gmail_quote"><div><snip><br><br></div><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Getting a robust and efficient message transport layer written is not<br>
easy work. It takes expertise and detailed knowledge, coupled with<br>
extensive real-world experience, to do it right. We simply decided to<br>
piggy back on some of the best that was out there, rather than trying<br>
to rewrite our own. The features we gain from zmq (it's not just the<br>
low-level performance, it's also the simple but powerful semantics of<br>
their various socket types, which we've baked into the very core of<br>
our design) are well worth the price of a C dependency in this case.<br><div></div></blockquote></div><div><br>I totally agree with you. And the more I read about zmq, the more I like it. However, the C dependency is a larger hurdle for me than it is for you. You see, IEP is an IDE that's independent of any Python installation. One can just install IEP, and use it for any python version installed on the system. I'd like to keep it that way, and that means I can only use pure Python my kernel code (and the code should work on Python 2 and Python 3).<br>
<br> </div><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>
> Further, am I right that the heartbeat is not necessary when communicating<br>
> between processes on the same box using 'localhost' (since some network<br>
> layers are bypassed)? That would give a short term solution for IEP.<br>
<br>
</div>Yes, on local host you can detect the process via other mechanisms.<br>
The question is whether the system recovers gracefully from dropped<br>
messages or incomplete connections. You do need to engineer that into<br>
the code itself, so that you don't lock up your client when the kernel<br>
becomes unresponsive, for example.<br></blockquote></div><div><br>But is it at all possible to lose a connection when you connect two processes using 'localhost'? Since it skips some of the lower layers of the networking (<a href="http://docs.python.org/howto/sockets.html" target="_blank">http://docs.python.org/howto/sockets.html</a>), I'd expect much less can go wrong. <br>
<br><br>Concluding. I think I'll stick to my implementation for the time being. For now, IEP will communicate 1to1 with a kernel, so I think it's pretty save as long as long as the kernel runs on the same box and I use 'localhost'. Later, when I implement remote processing and other more advanced stuff, I might use zmq (and require users to install it in order to use remote/parallel processing).<br>
<br>These discussion with you, and the stuff I read about zmq have got me thinking, and I'll probably improve a few things here and there. I definitely want to do some more testing (what happens if a connection is lost? Maybe I can try to recover it...).<br>
<br><br>I should probably also explain that I do not use a request/reply pattern on the socket level. I just connect the socket pair as the kernel starts up, and then it keeps sending data in both directions. So what happens if the kernel is busy running extension code, is that it will not be able to socket.send() or socket.recv(). This means:<br>
<br> * For the messages this should not be a problem, it will send the queued messages as the program returns from the extension code. <br> * There will be a couple of missed heartbeats though (but on the same box, should not be a problem, right?).<br>
* I'm not sure what happens when the client tries to send large amounts of data. Will network buffers overflow, or will this be correctly handled by TCP? (I use "select" to check whether I can send/recv over the socket.)<br>
* If the connection is lost, I'll get a socket error. So maybe I don't even need a heartbeat signal. However, I won't be able to distinguish a lost connection from the process being killed.<br><br><br><br> <br>
</div><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Thanks a lot for sharing your ideas, it's always super useful to look<br>
at these questions from multiple perspectives.<br></blockquote></div><div><br>And thank you you for discussing this stuff with me. I appreciate it a lot!<br><br> Almar<br></div></div><br><br>
</div><br>