[IPython-dev] pyzmq problems

Brian Granger ellisonbg at gmail.com
Fri May 28 17:20:25 EDT 2010 

2010/5/27 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
> Hi
> I think that you desing is ok, the I wrote the new implementation and put
> changes in git repo and I and doing a grafical desing of all ipython -zmq
> parts.
>
> this part are in http://gfif.udea.edu.co/ipzmq_connections.jpeg see and
> suggest me how I do heartbeat test with server.
> can I send heartbeat like a simple request?
> O.

The heartbeat capability is not quite there yet and will require
writing some C++ code in pyzmq.  So let's wait on that part of the
design.

Brian

>
> El 26 de mayo de 2010 16:16, Brian Granger <ellisonbg at gmail.com> escribió:
>>
>> 2010/5/25 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> > Ok I get it!
>> > but.
>> > can you send  messages just on request socket from frontend?
>>
>> Yes, but I think that is all we need.
>>
>> > because I can not send messages with zmq.SUB socket, and I can not recv
>> > with
>> > PUB socket.
>> > that information appears in the zmq APIs.
>>
>> The REQ/REP channel is for cases where the frontend needs to make a
>> request that the kernel will reply to immediately and there is only 1
>> reply.
>>
>> The PUB/SBU channel allows the kernel to send messages to the frontend
>> where there wasn't a corresponding reply.  A perfect example is
>> stdout, which can happen at any moment, without a corresponding
>> request from the frontend.  In this sense, the PUB/SUB channel is for
>> out of channel side effects.
>>
>> Again, I would look carefully at the example in the pyzmq source tree.
>>
>> Cheers,
>>
>> Brian
>> > http://api.zeromq.org/zmq_socket.html
>> > and I try do this in pyzmq and the message is ZMQError: Operation not
>> > supported
>> > is it the idea?
>> > O.
>> >
>> > El 25 de mayo de 2010 20:56, Brian Granger <ellisonbg at gmail.com>
>> > escribió:
>> >>
>> >> 2010/5/25 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> > Hi,
>> >> > then,
>> >> > are not pub/sub sockets  needed?
>> >>
>> >> We definitely need one PUB/SUB channel.  This channel will have the
>> >> PUB socket in the kernel and the SUB socket in the frontends.  This
>> >> will be used by the kernel to send stdout, stderr, pyin, pyout, etc to
>> >> the frontends.  But your examples also has a 2nd PUB/SUB channel going
>> >> the other direction from the frontend to the kernel.  What are you
>> >> planning on sending over that channel?
>> >>
>> >> > because I think that maybe with pub/sub you can to synchronize the
>> >> > frontends, sending messages that let know a kernel`s user_ns,
>> >> > heartbeat
>> >> > or
>> >> > tab_completion, because request and reply maybe have some longs
>> >> > processes in
>> >> > queue.
>> >>
>> >> Possibly, can you explain more of the these usage cases.  So far, we
>> >> have done tab completion over the REQ/REP channel.  The heartbeat
>> >> stuff would probably be a separate socket altogether.
>> >>
>> >> Brian
>> >>
>> >> > O.
>> >> > El 25 de mayo de 2010 17:02, Brian Granger <ellisonbg at gmail.com>
>> >> > escribió:
>> >> >>
>> >> >> 2010/5/25 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> >> > hi.
>> >> >> > the idea is to have 2 types of channels for different types of
>> >> >> > messages.
>> >> >> > As is specified in the next file:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > http://github.com/ellisonbg/pyzmq/blob/master/examples/kernel/message_spec.rst
>> >> >>
>> >> >> This document is very out of date.  We wrote it before writing the
>> >> >> prototype here:
>> >> >>
>> >> >> http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/
>> >> >>
>> >> >> But never updated the design doc.  At this point, I would consider
>> >> >> our
>> >> >> prototype as the design doc.  I don't see a need for the client to
>> >> >> have a PUB socket that the kernel is SUB'd to.
>> >> >>
>> >> >> > Another thing to discuss is the different types of messages that I
>> >> >> > don't
>> >> >> > find correct or clear on the previous link.
>> >> >> > I mean, I think there are redundancies in that proposal such as
>> >> >> > the
>> >> >> > pyin
>> >> >> > and
>> >> >> > execute request types of messages as a variable assignation in
>> >> >> > pub/sub
>> >> >> > and
>> >> >> > req/rep.
>> >> >> > What I suggest is the following:
>> >> >> > REQ/REP:
>> >> >> > Request:
>> >> >> > # msg_type = 'execute_request' content = {
>> >> >> > code : 'a = 10',
>> >> >> > }
>> >> >> > Reply:
>> >> >> > # msg_type = 'execute_reply' content = {
>> >> >> > 'status' : 'ok' OR 'error' OR 'abort' # data depends on status
>> >> >> > value
>> >> >> > 'message':'error_message' or 'output_message'
>> >> >> > }
>> >> >> > PUB/SUB:
>> >> >> > Complete:
>> >> >> > # msg_type = 'complete_request' content = {
>> >> >> > text : 'a.f', # complete on this line : 'print a.f' # full line
>> >> >> > }
>> >> >> > # msg_type = 'complete_reply' content = {
>> >> >> > matches : ['a.foo', 'a.bar']
>> >> >> > }
>> >> >> > Control:
>> >> >> > # msg_type = 'heartbeat' content = {
>> >> >> > }
>> >> >>
>> >> >> For now, please have a close look at the prototype in the link
>> >> >> above.
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Brian
>> >> >>
>> >> >> > What do you think about this?
>> >> >> > Do you think that a sole req/rep channel is enough?
>> >> >> > Best
>> >> >> > O.
>> >> >> > El 25 de mayo de 2010 15:08, Brian Granger <ellisonbg at gmail.com>
>> >> >> > escribió:
>> >> >> >>
>> >> >> >> I guess I am not clear why the kernel needs to have the SUB
>> >> >> >> socket.
>> >> >> >> If the client needs to send a message to the kernel, can't it
>> >> >> >> simply
>> >> >> >> use the REQ/REP channel?  But I do think the kernel needs the REP
>> >> >> >> and
>> >> >> >> PUB sockets.
>> >> >> >>
>> >> >> >> Brian
>> >> >> >>
>> >> >> >> 2010/5/24 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> >> >> > I have now read the zmq doc from zmq`s website reference.
>> >> >> >> > I think we need to use for the kernel 3 ports for the
>> >> >> >> > communication
>> >> >> >> > system.
>> >> >> >> > Kernel description:
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >  http://github.com/omazapa/ipython/blob/master/IPython/core/ipzmq_server.py
>> >> >> >> > -> port 5555 have subscribe socket into kernel class to read
>> >> >> >> > publisher
>> >> >> >> > messages from frontend.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > self._subscribe_socket = self._Context.socket(zmq.SUB)
>> >> >> >> >
>> >> >> >> > self._subscribe_socket.bind(self._subscribe_connection)
>> >> >> >> >
>> >> >> >> > self._subscribe_socket.setsockopt(zmq.SUBSCRIBE,"")
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > since the subscribe socket can not send messages "it was read
>> >> >> >> > in
>> >> >> >> > the
>> >> >> >> > reference", we need to implement another socket called
>> >> >> >> > publisher
>> >> >> >> > to
>> >> >> >> > send
>> >> >> >> > messages to frontend, then
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > -> port 5556 has a socket which allow kernel class to send
>> >> >> >> > messages
>> >> >> >> > to
>> >> >> >> > frontend, then the subbscribe and publisher sockets will
>> >> >> >> > communicate.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > self._publisher_socket = self._Context.socket(zmq.PUB)
>> >> >> >> >
>> >> >> >> > self._publisher_socket.bind(self._publisher__connection)
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > -> and 5557 will be implemented to request and publisher
>> >> >> >> > sockets
>> >> >> >> > that
>> >> >> >> > are
>> >> >> >> > working very well.
>> >> >> >> >
>> >> >> >> > do you think this 3-socket model is a good idea? You can check
>> >> >> >> > it
>> >> >> >> > because
>> >> >> >> > I've already implemented it an and it's working fine.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > http://github.com/omazapa/ipython/blob/master/IPython/core/ipzmq_client.py
>> >> >> >> > O.
>> >> >> >> > El 24 de mayo de 2010 13:49, Brian Granger
>> >> >> >> > <ellisonbg at gmail.com>
>> >> >> >> > escribió:
>> >> >> >> >>
>> >> >> >> >> Omar,
>> >> >> >> >>
>> >> >> >> >> I am busy today but here are some ideas:
>> >> >> >> >>
>> >> >> >> >> * To get to know pyzmq better, I would open up 2-3 IPython
>> >> >> >> >> sessions,
>> >> >> >> >> import zmq on all of them and then start to create sockets and
>> >> >> >> >> send
>> >> >> >> >> messages between the different IPython sessions.  This works
>> >> >> >> >> really
>> >> >> >> >> well and will give you a better idea of how the different
>> >> >> >> >> socket
>> >> >> >> >> types
>> >> >> >> >> work, how the json stuff works, etc.  This would be
>> >> >> >> >> invaluable.
>> >> >> >> >>
>> >> >> >> >> * To simplify debugging, create a version of the code that has
>> >> >> >> >> the
>> >> >> >> >> absolute minimal code - no objects, config, etc.  Just the raw
>> >> >> >> >> zmq
>> >> >> >> >> messaging stuff.
>> >> >> >> >>
>> >> >> >> >> I think if you do these 2 things, the error will be more
>> >> >> >> >> obvious.
>> >> >> >> >> Keep posting back to the list so I or Fernando can help with
>> >> >> >> >> this
>> >> >> >> >> though.
>> >> >> >> >>
>> >> >> >> >> Cheers,
>> >> >> >> >>
>> >> >> >> >> Brian
>> >> >> >> >>
>> >> >> >> >> 2010/5/22 Omar Andrés Zapata Mesa <andresete.chaos at gmail.com>:
>> >> >> >> >> > hi all
>> >> >> >> >> > I am working yet in zmq module to ipython, but I have the
>> >> >> >> >> > next
>> >> >> >> >> > problem
>> >> >> >> >> > using
>> >> >> >> >> > json.
>> >> >> >> >> > the code are in http://github.com/omazapa/ipython
>> >> >> >> >> > into the dir  ipython/IPython/core/
>> >> >> >> >> >
>> >> >> >> >> > I run my zmq server prototype
>> >> >> >> >> >
>> >> >> >> >> > the output is
>> >> >> >> >> >
>> >> >> >> >> > omazapa at tuxhome:~/MyProjects/GSoC/tmp/ipython/IPython/core$
>> >> >> >> >> > python
>> >> >> >> >> > ipzmq_server.py
>> >> >> >> >> > reply socket= tcp://127.0.0.1:5555
>> >> >> >> >> > publisher socket = tcp://127.0.0.1:5556
>> >> >> >> >> > Server started.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > in this moment I am waiting json`s message in reply socket.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > then I run my client prototype
>> >> >> >> >> >
>> >> >> >> >> > the output is
>> >> >> >> >> >
>> >> >> >> >> > omazapa at tuxhome:~/MyProjects/GSoC/tmp/ipython/IPython/core$
>> >> >> >> >> > python
>> >> >> >> >> > ipzmq_client.py
>> >> >> >> >> > request socket = tcp://127.0.0.1:5556
>> >> >> >> >> > subscribe socket = tcp://127.0.0.1:5555
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > but server no recieve the message.
>> >> >> >> >> >
>> >> >> >> >> > the output is
>> >> >> >> >> >
>> >> >> >> >> > Traceback (most recent call last):
>> >> >> >> >> >   File "ipzmq_server.py", line 112, in <module>
>> >> >> >> >> >     msg=server.recieve_reply()
>> >> >> >> >> >   File "ipzmq_server.py", line 79, in recieve_reply
>> >> >> >> >> >     msg=self._reply_socket.recv_json()
>> >> >> >> >> >   File "_zmq.pyx", line 709, in zmq._zmq.Socket.recv_json
>> >> >> >> >> > (zmq/_zmq.c:5242)
>> >> >> >> >> >   File "/usr/lib/python2.6/json/__init__.py", line 307, in
>> >> >> >> >> > loads
>> >> >> >> >> >     return _default_decoder.decode(s)
>> >> >> >> >> >   File "/usr/lib/python2.6/json/decoder.py", line 319, in
>> >> >> >> >> > decode
>> >> >> >> >> >     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>> >> >> >> >> >   File "/usr/lib/python2.6/json/decoder.py", line 338, in
>> >> >> >> >> > raw_decode
>> >> >> >> >> >     raise ValueError("No JSON object could be decoded")
>> >> >> >> >> > ValueError: No JSON object could be decoded
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > have you some idea?
>> >> >> >> >> > maybe, do I need encode my message before send it?
>> >> >> >> >> > I have the last version of zeromq2 from official repo and
>> >> >> >> >> > pyzmq
>> >> >> >> >> > http://github.com/ellisonbg/pyzmq/, I am using python2.6
>> >> >> >> >> >
>> >> >> >> >> > Brian  said me that the problem is that I have outdated
>> >> >> >> >> > version
>> >> >> >> >> > of
>> >> >> >> >> > zeromq
>> >> >> >> >> > and pyzmq but I update zeromq and pyzmq and It is not
>> >> >> >> >> > working
>> >> >> >> >> > yet.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > thks
>> >> >> >> >> >
>> >> >> >> >> > _______________________________________________
>> >> >> >> >> > IPython-dev mailing list
>> >> >> >> >> > IPython-dev at scipy.org
>> >> >> >> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> Brian E. Granger, Ph.D.
>> >> >> >> >> Assistant Professor of Physics
>> >> >> >> >> Cal Poly State University, San Luis Obispo
>> >> >> >> >> bgranger at calpoly.edu
>> >> >> >> >> ellisonbg at gmail.com
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Brian E. Granger, Ph.D.
>> >> >> >> Assistant Professor of Physics
>> >> >> >> Cal Poly State University, San Luis Obispo
>> >> >> >> bgranger at calpoly.edu
>> >> >> >> ellisonbg at gmail.com
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Brian E. Granger, Ph.D.
>> >> >> Assistant Professor of Physics
>> >> >> Cal Poly State University, San Luis Obispo
>> >> >> bgranger at calpoly.edu
>> >> >> ellisonbg at gmail.com
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at scipy.org
>> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Brian E. Granger, Ph.D.
>> >> Assistant Professor of Physics
>> >> Cal Poly State University, San Luis Obispo
>> >> bgranger at calpoly.edu
>> >> ellisonbg at gmail.com
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >
>> >
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the IPython-dev mailing list