[IPython-dev] ipython1 remote exceptions causing local crashes

[Douglas Jones]

Fernando,

Thank you for the detailed response.

One thing I meant to mention in my previous post, is that the crash
only happens when using %autopx mode. Doing the same thing in using
rc.executeAll(...) causes the exception and the Traceback to be
displayed. Since you said this is a feature, I suppose you know this.
What I don't understand is why the behavior between the two should be
different.

That said, I understand the issue you are trying to solve by having
the client behave in this manner, however I think it might be
undesirable behavior for some projects.

A lot of my experience is based on the default python (CPython) shell,
which I think the previous "chainsaw" version of ipython1 did a good
job of emulating in regard to exceptions coming from the cluster.

Please forgive me if some of the issues I bring up have already been
discussed or have already been addressed in the code, as I haven't
been able to look through all of the new features and changes.

For the project that I am working on, it seems that one of the big
features is to use your cluster in an interactive manner. The "feel"
of this for our current implementation is much like typing any other
command into your python shell. Ipython1 goes a long way to providing
the features to make this necessary.

That said, it seems like an overreaction to have the user, more or
less, ejected from their interactive session for what may have been a
trivial mistake. I think one of the nice features of working
interactively is that you are given a chance to fix the mistake and
continue along with minimal fuss.

So, I can see the behavior that you have being useful for modes where
there is not a human at the keyboard. I agree that you wouldn't want
code to continue to run on the assumption that a previous command was
successful when it really wasn't.

Because I think IPython will be used for both situations (interactive
and "batch"), the handling of exceptions should at least be
configurable or changed internally based on the mode of use.

I realize this is a work in progress, but the current handling of
exceptions is misleading, I think. The user is presented with the
normal IPython has crashed error messages with instructions to mail
the developers along with a long traceback. This really makes it seem
like the error was internal to IPython and not something that was a
result of the user's own code being executed on the cluster. It was
this confusion that prompted me to mail the list with my "issue".

I hope you'll take into consideration the things I've said here. I'd
be happy to provide more information or go into more detail about how
I envision things working. I'd also be interested in hearing some of
what has already been decided and discussed as well as any future
development along this line that has been planned, but not implemented
yet.

Thank you,
~Doug


On 2/20/07, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi Douglas,
>
> On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:
> > Hi all,
> >
> > I've been looking at the new ipython1 saw branch and the implications
> > of the changes to a project that I'm working on.
> >
> > I've noticed in my testing that running some code on the engine that
> > causes an exception crashes my ipython client. I'm using recent svn
> > checkouts of both ipython1(revision 2104) and the ipython shell (which
> > lists itself as version IPython 0.7.4.svn.r2010).
> >
> > Is this a known issue?
>
> Well, it's a feature, not a bug :)  It was actually something that was
> deliberately added recently, to ensure that remote exceptions aren't
> silently swallowed and that you are forced to know about them.  The
> reason is that otherwise swallowing remote exceptions makes for
> incredibly hard debugging: your engine threw an exception which went
> silently into a log file only, and the client keeps on happily feeding
> it new commands to execute, most of which are likely to produce
> garbage because some previous quantity you expected to have computed
> is not there.
>
> We are trying to bring, within the realm of what's possible, a smooth
> transition from a model of single-process synchronous programming to
> distributed computing, and exceptions are a particularly thorny issue
> here.  We recently spoke to DOE people about  the next-generation
> language projects for distributed computing that the DOE/DARPA are
> looking into (X10, Fortress and Chapel, see
> http://www.ahpcrc.org/conferences/PGAS2006/presentations/Yelick.pdf).
> As far as we can tell, no system yet has a completely satisfactory
> answer for this, partly because I don't think you /can/ have such an
> answer.
>
> We hope we'll offer a model that's reasonable for end users (we are
> end users of this, so we're eating our own dog food here), and that
> will hopefully balance:
>
> - Feeling like an analytic continuation of the serial programming
> model (though remember that you may always find surprises off the real
> axis :)
>
> - Offering some utilities that make the common synchronization tasks
> needed for regular algorithmic development easy to achieve.
>
> - Exposing enough of the distributed objects to allow more
> sophisticated uses. It is still possible that we'll limit access to
> parts of the underlying asynchronous machinery for reasons I don't
> quite have time to explain right now.  I can go into more detail later
> if needed.
>
>
> But in all of this, feedback from users will be critically important.
> We've put a LOT of thought into this, but it's very easy to travel
> down a dead end without noticing it, so feel free to speak up.  If we
> get this right, this system should be very useful for many people and
> for a while, so this is the time to complain if you think we're
> barking up the wrong tree on any issue.
>
> > If you need more information, I'd be happy to provide some of the
> > traceback information.
> >
> > The only thing I haven't updated is the version of Twisted installed
> > on the machine. Is it necessary to have a newer version of Twisted for
> > the ipython1 saw branch? I think Twisted that is installed is version
> > 2.4.0.
>
> Yes, saw relies of bugfixes to Twisted that are only part of Twisted
> 2.5.0.  We do now ship twisted's web2 module internally so you don't
> have to track SVN for that.  Basically if you get Twisted 2.5.0
> installed (which requires zope interfaces), you should be good to go.
>
> Cheers,
>
> f
>