[IPython-dev] Notebook websocket change

Wed Jan 14 19:50:46 EST 2015

The PR is 2321 <https://github.com/ipython/ipython/pull/2321>, switching
IPython to use SockJS instead of WebSockets. I’m not sure how informative
it is, since things have moved around *a lot* since then, and it is from a
time when we were shipping more things in external, rather than accepting
them as dependencies. I think a similar patch today would be a great deal
simpler, especially now that it doesn’t need to include merging sockets. I
also had to include a hack to workaround the fact that SockJS didn’t allow
empty messages, and we were sending empty messages as part of the initial
handshake (also not true anymore).

-MinRK

On Wed, Jan 14, 2015 at 4:12 PM, William Stein <wstein at gmail.com> wrote:

> On Thu, Jan 8, 2015 at 11:18 AM, MinRK <benjaminrk at gmail.com> wrote:
> > Some time ago, I made a PR to IPython switching to socket.io, which
> required
> > the same basic changes since it only allowed a single connection per
> page,
> > so I know it's not too hard. We may start to see more pressure as hosted
> > instances, and reconsider the fallback in the future, but at this point,
> we
> > don't feel that pressure. Wind proper websockets to be sufficiently
> > preferable that we will wait as long as we can before appeasing old,
> broken,
> > or misconfigured network environments that prevent websockets from
> working.
> >
> > The majority of notebook users seem to be running as a desktop app via
> > localhost, where websockets are rarely a problem (The nightmare of Sophos
> > AV/firewall on Windows excluded).
>
> Do you have a pointer to the above pull request?
>
> In my experience battling with websockets and the cloud over the last
> few years, it's not just an issue of websockets versus fallbacks --
> there's also issues with the robustness of the websocket connection,
> how it's maintained,  automatically reconnecting if the connection has
> problems, heartbeats, etc.   If you only use IPython locally then none
> of this is a problem.  However, if you use IPython over a complicated
> network that drops packets sometimes, flakes out, etc., it can be.   I
> get fairly regular emails from people having trouble specifically with
> the robustness of the iPython connection inside of SMC.    Libraries
> like socket.io and engine.io [1] seem to have pretty good debuged and
> thought out code for dealing with these situations (and for fallbacks
> in case websockets doesn't work).
>
> Since this use case is only a tiny part of the total IPython notebook
> usage, it makes sense for me to take this on myself.   I will
> definitely highly prioritize trying to hack on IPython in order to get
> it to use engine.io+primus, since I have to deal with this websocket
> robustness situation to improve my user's experience.   For example,
> I'm getting fairly regular complaints from Randy Leveque and his
> colleagues at UW.     So I'd greatly appreciate a link to that pull
> request, and of course I'll share anything I do here.
>
> [1] https://github.com/Automattic/engine.io
>
> >
> > -MinRK
> >
> > On Thu, Jan 8, 2015 at 10:55 AM, William Stein <wstein at gmail.com> wrote:
> >>
> >> On Thu, Jan 8, 2015 at 10:19 AM, Brian Granger <ellisonbg at gmail.com>
> >> wrote:
> >> > So far we have felt that it isn't worth the complexity cost for us to
> >> > support people that deliberately choose to break the internet. It is
> not
> >> > like WebSockets is some crazy, insecure, unsupported hack over a weird
> >> > port.
> >> > It is just different bytes going over the same HTTP/HTTPS socket
> >> > connection
> >> > and is essentially a formal standard (I know you know this - this is
> our
> >> > messaging to companies who complain).
> >>
> >> I know -- that's why I switched SMC to pure websockets for a while.
> >> Then I received complaints at a rate of about 2 very desperate users
> >> per week, and these were the people who cared enough to complain
> >> repeatedly.
> >>
> >> > Our situation is a bit different though than SMC because most of our
> >> > current
> >> > users run the notebook on their own computers. This may be something
> >> > that we
> >> > eventually want to rethink as more people run this in the cloud.
> >> > However, I
> >> > would like to push back on the companies choosing to break the
> internet
> >> > really hard before giving in. I am guessing that in most cases,
> >> > WebSockets
> >> > are broken in enterprise setting not because of some important
> >> > deliberate
> >> > choice, but rather because people don't understand it and are using
> >> > default
> >> > settings that disable them (I could be wrong though)....
> >>
> >> In my experience, for some people that's exactly the problem.  For one
> >> person I helped it turned out their specific problem was the Windows
> >> install they had access to had websockets explicitly disabled --
> >> perhaps that was an IT policy.  For other people the problem is
> >> further along the stack (e.g., possibly involving caching).
> >>
> >> Anyway, this change so that "there is a single websocket connection
> >> between the server and client. " makes implementing an alternative to
> >> WebSocket as a fallback dramatically easier, which I greatly
> >> appreciate.
> >>
> >> > Our situation is a bit different though than SMC because most of our
> >> > current
> >> > users run the notebook on their own computers. This may be something
> >> > that we
> >>
> >> Just curious -- how do you know that most people using IPython aren't
> >> already using it through SMC?
> >>
> >> William
> >>
> >> >
> >> > Cheers,
> >> >
> >> > Brian
> >> >
> >> >
> >> >
> >> > On Thu, Jan 8, 2015 at 10:10 AM, William Stein <wstein at gmail.com>
> wrote:
> >> >>
> >> >> On Thu, Jan 8, 2015 at 10:04 AM, MinRK <benjaminrk at gmail.com> wrote:
> >> >> > Alternative notebook frontend authors:
> >> >> >
> >> >> > There is an upcoming change to websocket connections to the
> notebook
> >> >> > server.
> >> >> > Instead of three websockets per notebook, one for each zmq channel,
> >> >> > there is
> >> >> > a single websocket connection between the server and client. The
> >> >> > channel
> >> >> > associated with each message is identified by a channel key in the
> >> >> > message
> >> >> > dict itself.
> >> >> >
> >> >> > Relevant changes:
> >> >> >
> >> >> > url is /channels instead of /shell, etc.
> >> >> > when sending a message, add a channel key with the channel name
> >> >> > ('shell',
> >> >> > 'stdin')
> >> >> > when receiving a message, check the channel key for routing
> >> >> >
> >> >> > -MinRK
> >> >>
> >> >> Thanks.
> >> >>
> >> >> Of related interest, what the situation involving providing a
> fallback
> >> >> to websockets?
> >> >>
> >> >> I tried to only support websockets connections with SageMathCloud for
> >> >> a few weeks, and immediately ran into many complaints from users who
> >> >> couldn't connect as a result.   This is even with https and so on --
> >> >> it's just a sad fact that in some corporate or otherwise constrained
> >> >> environments that websockets don't work.   By far the best approach I
> >> >> found was to use Primus + Engine.io, which uses websockets if
> >> >> possible, but will fallback to polling.   So this is what SMC does
> >> >> now, and everything works, even for people that don't have websockets
> >> >> as an option... except for IPython-in-SMC of course, which doesn't
> >> >> work.
> >> >>
> >> >> Another unrelated issue is that I'm now hardcoding making this change
> >> >> around line 171 of IPython/html/notebookapp.py, since I want to serve
> >> >> the purely static html of IPython from a completely different web
> >> >> server:
> >> >>
> >> >>             #static_url_prefix = url_path_join(base_url,'/static/'),
> >> >>             static_url_prefix = '/static/ipython/',
> >> >>
> >> >> (Sorry for derailing the thread.)
> >> >>
> >> >>  -- William
> >> >>
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > IPython-dev mailing list
> >> >> > IPython-dev at scipy.org
> >> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> William Stein
> >> >> Professor of Mathematics
> >> >> University of Washington
> >> >> http://wstein.org
> >> >> _______________________________________________
> >> >> IPython-dev mailing list
> >> >> IPython-dev at scipy.org
> >> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Brian E. Granger
> >> > Cal Poly State University, San Luis Obispo
> >> > @ellisonbg on Twitter and GitHub
> >> > bgranger at calpoly.edu and ellisonbg at gmail.com
> >> >
> >> > _______________________________________________
> >> > IPython-dev mailing list
> >> > IPython-dev at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >> >
> >>
> >>
> >>
> >> --
> >> William Stein
> >> Professor of Mathematics
> >> University of Washington
> >> http://wstein.org
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
> >
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
>
>
>
> --
> William Stein
> Professor of Mathematics
> University of Washington
> http://wstein.org
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150114/4a3f1ac5/attachment.html>