[IPython-dev] pyzmq problems

Tue May 25 21:56:15 EDT 2010

2010/5/25 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
> Hi,
> then,
> are not pub/sub sockets  needed?

We definitely need one PUB/SUB channel.  This channel will have the
PUB socket in the kernel and the SUB socket in the frontends.  This
will be used by the kernel to send stdout, stderr, pyin, pyout, etc to
the frontends.  But your examples also has a 2nd PUB/SUB channel going
the other direction from the frontend to the kernel.  What are you
planning on sending over that channel?

> because I think that maybe with pub/sub you can to synchronize the
> frontends, sending messages that let know a kernel`s user_ns, heartbeat or
> tab_completion, because request and reply maybe have some longs processes in
> queue.

Possibly, can you explain more of the these usage cases.  So far, we
have done tab completion over the REQ/REP channel.  The heartbeat
stuff would probably be a separate socket altogether.

Brian

> O.
> El 25 de mayo de 2010 17:02, Brian Granger <ellisonbg at gmail.com> escribió:
>>
>> 2010/5/25 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> > hi.
>> > the idea is to have 2 types of channels for different types of messages.
>> > As is specified in the next file:
>> >
>> > http://github.com/ellisonbg/pyzmq/blob/master/examples/kernel/message_spec.rst
>>
>> This document is very out of date.  We wrote it before writing the
>> prototype here:
>>
>> http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/
>>
>> But never updated the design doc.  At this point, I would consider our
>> prototype as the design doc.  I don't see a need for the client to
>> have a PUB socket that the kernel is SUB'd to.
>>
>> > Another thing to discuss is the different types of messages that I don't
>> > find correct or clear on the previous link.
>> > I mean, I think there are redundancies in that proposal such as the pyin
>> > and
>> > execute request types of messages as a variable assignation in pub/sub
>> > and
>> > req/rep.
>> > What I suggest is the following:
>> > REQ/REP:
>> > Request:
>> > # msg_type = 'execute_request' content = {
>> > code : 'a = 10',
>> > }
>> > Reply:
>> > # msg_type = 'execute_reply' content = {
>> > 'status' : 'ok' OR 'error' OR 'abort' # data depends on status value
>> > 'message':'error_message' or 'output_message'
>> > }
>> > PUB/SUB:
>> > Complete:
>> > # msg_type = 'complete_request' content = {
>> > text : 'a.f', # complete on this line : 'print a.f' # full line
>> > }
>> > # msg_type = 'complete_reply' content = {
>> > matches : ['a.foo', 'a.bar']
>> > }
>> > Control:
>> > # msg_type = 'heartbeat' content = {
>> > }
>>
>> For now, please have a close look at the prototype in the link above.
>>
>> Cheers,
>>
>> Brian
>>
>> > What do you think about this?
>> > Do you think that a sole req/rep channel is enough?
>> > Best
>> > O.
>> > El 25 de mayo de 2010 15:08, Brian Granger <ellisonbg at gmail.com>
>> > escribió:
>> >>
>> >> I guess I am not clear why the kernel needs to have the SUB socket.
>> >> If the client needs to send a message to the kernel, can't it simply
>> >> use the REQ/REP channel?  But I do think the kernel needs the REP and
>> >> PUB sockets.
>> >>
>> >> Brian
>> >>
>> >> 2010/5/24 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> > I have now read the zmq doc from zmq`s website reference.
>> >> > I think we need to use for the kernel 3 ports for the communication
>> >> > system.
>> >> > Kernel description:
>> >> >
>> >> >
>> >> >  http://github.com/omazapa/ipython/blob/master/IPython/core/ipzmq_server.py
>> >> > -> port 5555 have subscribe socket into kernel class to read
>> >> > publisher
>> >> > messages from frontend.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > self._subscribe_socket = self._Context.socket(zmq.SUB)
>> >> >
>> >> > self._subscribe_socket.bind(self._subscribe_connection)
>> >> >
>> >> > self._subscribe_socket.setsockopt(zmq.SUBSCRIBE,"")
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > since the subscribe socket can not send messages "it was read in the
>> >> > reference", we need to implement another socket called publisher to
>> >> > send
>> >> > messages to frontend, then
>> >> >
>> >> >
>> >> > -> port 5556 has a socket which allow kernel class to send messages
>> >> > to
>> >> > frontend, then the subbscribe and publisher sockets will communicate.
>> >> >
>> >> >
>> >> >
>> >> > self._publisher_socket = self._Context.socket(zmq.PUB)
>> >> >
>> >> > self._publisher_socket.bind(self._publisher__connection)
>> >> >
>> >> >
>> >> > -> and 5557 will be implemented to request and publisher sockets that
>> >> > are
>> >> > working very well.
>> >> >
>> >> > do you think this 3-socket model is a good idea? You can check it
>> >> > because
>> >> > I've already implemented it an and it's working fine.
>> >> >
>> >> >
>> >> > http://github.com/omazapa/ipython/blob/master/IPython/core/ipzmq_client.py
>> >> > O.
>> >> > El 24 de mayo de 2010 13:49, Brian Granger <ellisonbg at gmail.com>
>> >> > escribió:
>> >> >>
>> >> >> Omar,
>> >> >>
>> >> >> I am busy today but here are some ideas:
>> >> >>
>> >> >> * To get to know pyzmq better, I would open up 2-3 IPython sessions,
>> >> >> import zmq on all of them and then start to create sockets and send
>> >> >> messages between the different IPython sessions.  This works really
>> >> >> well and will give you a better idea of how the different socket
>> >> >> types
>> >> >> work, how the json stuff works, etc.  This would be invaluable.
>> >> >>
>> >> >> * To simplify debugging, create a version of the code that has the
>> >> >> absolute minimal code - no objects, config, etc.  Just the raw zmq
>> >> >> messaging stuff.
>> >> >>
>> >> >> I think if you do these 2 things, the error will be more obvious.
>> >> >> Keep posting back to the list so I or Fernando can help with this
>> >> >> though.
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Brian
>> >> >>
>> >> >> 2010/5/22 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> >> > hi all
>> >> >> > I am working yet in zmq module to ipython, but I have the next
>> >> >> > problem
>> >> >> > using
>> >> >> > json.
>> >> >> > the code are in http://github.com/omazapa/ipython
>> >> >> > into the dir  ipython/IPython/core/
>> >> >> >
>> >> >> > I run my zmq server prototype
>> >> >> >
>> >> >> > the output is
>> >> >> >
>> >> >> > omazapa at tuxhome:~/MyProjects/GSoC/tmp/ipython/IPython/core$ python
>> >> >> > ipzmq_server.py
>> >> >> > reply socket= tcp://127.0.0.1:5555
>> >> >> > publisher socket = tcp://127.0.0.1:5556
>> >> >> > Server started.
>> >> >> >
>> >> >> >
>> >> >> > in this moment I am waiting json`s message in reply socket.
>> >> >> >
>> >> >> >
>> >> >> > then I run my client prototype
>> >> >> >
>> >> >> > the output is
>> >> >> >
>> >> >> > omazapa at tuxhome:~/MyProjects/GSoC/tmp/ipython/IPython/core$ python
>> >> >> > ipzmq_client.py
>> >> >> > request socket = tcp://127.0.0.1:5556
>> >> >> > subscribe socket = tcp://127.0.0.1:5555
>> >> >> >
>> >> >> >
>> >> >> > but server no recieve the message.
>> >> >> >
>> >> >> > the output is
>> >> >> >
>> >> >> > Traceback (most recent call last):
>> >> >> >   File "ipzmq_server.py", line 112, in <module>
>> >> >> >     msg=server.recieve_reply()
>> >> >> >   File "ipzmq_server.py", line 79, in recieve_reply
>> >> >> >     msg=self._reply_socket.recv_json()
>> >> >> >   File "_zmq.pyx", line 709, in zmq._zmq.Socket.recv_json
>> >> >> > (zmq/_zmq.c:5242)
>> >> >> >   File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
>> >> >> >     return _default_decoder.decode(s)
>> >> >> >   File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
>> >> >> >     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>> >> >> >   File "/usr/lib/python2.6/json/decoder.py", line 338, in
>> >> >> > raw_decode
>> >> >> >     raise ValueError("No JSON object could be decoded")
>> >> >> > ValueError: No JSON object could be decoded
>> >> >> >
>> >> >> >
>> >> >> > have you some idea?
>> >> >> > maybe, do I need encode my message before send it?
>> >> >> > I have the last version of zeromq2 from official repo and pyzmq
>> >> >> > http://github.com/ellisonbg/pyzmq/, I am using python2.6
>> >> >> >
>> >> >> > Brian  said me that the problem is that I have outdated version of
>> >> >> > zeromq
>> >> >> > and pyzmq but I update zeromq and pyzmq and It is not working yet.
>> >> >> >
>> >> >> >
>> >> >> > thks
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > IPython-dev mailing list
>> >> >> > IPython-dev at scipy.org
>> >> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Brian E. Granger, Ph.D.
>> >> >> Assistant Professor of Physics
>> >> >> Cal Poly State University, San Luis Obispo
>> >> >> bgranger at calpoly.edu
>> >> >> ellisonbg at gmail.com
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Brian E. Granger, Ph.D.
>> >> Assistant Professor of Physics
>> >> Cal Poly State University, San Luis Obispo
>> >> bgranger at calpoly.edu
>> >> ellisonbg at gmail.com
>> >
>> >
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>

-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com