[IPython-dev] Comments on IPython Qt widgets

Sat Feb 12 15:16:26 EST 2011

On Sat, Feb 12, 2011 at 11:38 AM, Brian Granger <ellisonbg at gmail.com> wrote:

> I think there are two possible approaches:
>
> 1.  Still use the pyzmq interface through KernelManager and the
> SocketChannel objects, just use in process (threads) zeromq sockets.
> While complicated, this might be the best approach.  But, it might
> take some GUI event loop magic to get working (because you can't run
> anything in a thread.
>

The problem with ZMQ for me is that the shell still has poor responsiveness
when embedded in QGIS.  I still don't know if this problem is specific to
QGIS or would affect other C++ Qt applications that run embedded Python
interpreters.  I don't have another C++ app handy that does this to
cross-validate with.

I did some hacking over the week to try and work around the problem that
included: switching the ZmqSocketChannel to a subclass of QThread, tweaking
the thread run priorities, fiddling with the signal connections, and nothing
gave good results.  So, the problem could be:

   1. With the way QGIS handles threads controlled by Python
   2. With the way the ZeroMQ event loop behaves when embedded in a C++ app
   3. Other

I noticed that if I interact with the console window somehow, like twiddling
the scroll wheel, while it is waiting then a refresh occurs that brings in
the results from the external kernel.  Also, I wrote a quick example in pure
PyQGIS where Python is running the whole show and it is just as responsive
as `ipython-qtconsole`. So, I suspect explanation #1 is still the most
likely cause for this behavior.

I haven't heard anything back from the QGIS-dev mailing list yet, so the ZMQ
route is currently cold for me at this point.

2.  Get rid of the zeromq stuff entirely.  This is probably easier.
> To do this, I would check out the subclass of InteractiveShell in
> IPython.zmq.zmqshell.  It should be pretty easy to build a subclass of
> InteractiveShell that works, *but*, it leaves you without a GUI.
>

I also gave this a shot but creating a very primitive "kernel manager" that
ran an InteractiveShell inside an OutputTrap that was subclassed to also
redirect io.Term.  This worked fairly well and I was able to get something
working that was responsive in QGIS and supported tab-completion.  There
were two main difficulties I ran into going this route:

- The frontend widgets seem to know a lot about the backend implementation
which makes it difficult to create a new kernel manager that does something
different under the hood.

- The ZMQ kernel manager currently forms the base class for all kernel
managers.

It seems like it might be a good idea to:

- Create an ABC for kernel managers that defines everything a frontend
widget really needs to know in order to use them.

- Replace specific references in FrontendWidgets
like self.kernel_manager.xreq_channel.execute with something a little more
abstract such as self.kernel_manager.execute and let the kernel manager
worry about the details of which socket, channel, QSignal, etc is actually
carrying the message.

These are just my thoughts, and I reserve the right to be wrong :)

To summarize a bit of a rambling email: embedding an InteractiveShell has
proven to be a very promising route.  At this point, the main question I
have concerning this implementation is:

   - Is there a better way to set the stout/stderr streams of  kernel than
having the kernel manager run every call to "run_source" inside an
OutputTrap?

Final note: you guys are doing an awesome job with the rewrite, on the whole
it has been very easy to hack on IPython without breaking everything since
things are well componentized.

-Charlie

Min, Fernando or Evan, any thoughts on this?
>
> Cheers,
>
> Brian
>
>
> On Tue, Feb 8, 2011 at 10:51 AM, Charlie Sharpsteen
> <source at sharpsteen.net> wrote:
> > Thanks for the reply Brian,
> >
> > On Mon, Feb 7, 2011 at 8:15 PM, Brian Granger <ellisonbg at gmail.com
> >> wrote:
> >
> >> Charlie,
> >>
> >> > I was playing around with the new Qt  widgets in 0.11-dev the other
> day
> >> and
> >> > was very impressed.  I tried throwing together a quick plugin to add
> an
> >> > IPython console to the Qt-based program Quantum GIS [QGIS][1].
>  Creating
> >> the
> >> > plugin was easy enough as QGIS runs an embedded Python interpreter and
> >> > exposes a lot of PyQt4 functionality.  However, I noticed a couple of
> >> things
> >> > and was wondering if anyone on this list had any comments to share:
> >> >
> >> > - The shell is based on a stripped down version of `ipython-qtconsole`
> >> and
> >> > works but is very, very slow for some reason.  The is a delay of a
> >> > couple
> >> of
> >> > seconds between when an action is triggered and a response comes back
> to
> >> the
> >> > terminal.  The only guess I have that may explain this behavior is
> that
> >> QGIS
> >> > is a C++ application that embeds a Python interpreter whereas
> >> > ipython-qtconsole is a pure PyQt4 program.  Perhaps QGIS is not
> letting
> >> the
> >> > Python threads run as often as a pure PyQt4 program would.
> >> > This could be a QGIS-specific problem, but I was wondering if anyone
> >> > else
> >> > had the chance to run the IPython Qt widget in an embedded interpreter
> >> and
> >> > noticed similar results?
> >>
> >> In general the ipython-qtconsole should be quite fast.  But for that
> >> to be the case, the qt event loop has to be running.  If the C++ is
> >> not letting the qt event loop run often enough, you would definitely
> >> see quite slow behavior.
> >
> >
> > This is what I suspect, I would probably have to spend some quality time
> > with the C++ side of the code to be sure.
> >
> >
> >
> >> > - Currently, the only IPython kernel available for use with the Qt
> >> > widget
> >> > runs the IPython interpreter in a separate process.  For QGIS, it
> would
> >> be
> >> > very nice to have the option of running an IPython kernel inside the
> >> > same
> >> > process as the Qt widgets.  IPython would then have direct access to
> the
> >> > QGIS interface and would be able to inspect and operatie on the PyQt4
> >> > objects that compose the GUI providing a powerful tool for users
> >> performing
> >> > spatial analysis and developers building plugins.
> >> > Are there any plans to develop IPython kernels that do not run
> >> > in separate processes?  I spent a few days trying to build one myself
> by
> >> > wrapping an InteractiveShell, but I feel like I am flailing a little
> bit
> >> > when it comes to creating a kernel_manager that will integrate with
> the
> >> Qt
> >> > frontend.  If anyone has any design advice on how to implement an
> >> in-process
> >> > kernel I would love to hear it.
> >>
> >> There is interest in having an in process IPython, but I honestly
> >> think it will not work very well at all.  This is mostly because of
> >> the GIL.  I would simply turn the model inside out.  If you run the
> >> QGIS code that has all the PyQt widgets in the IPython kernel (start
> >> the kernel with the pyqt event loop running using --pylab qt) your
> >> ipython session will have all the access that it needs.  IOW, don't
> >> embed IPython in your app, embed your app in IPython.  If this model
> >> doesn't work for you, we can talk more about how to get started on an
> >> in process IPython widget, but it will be a significant amount of work
> >> I'm afraid.
> >
> >
> > The problem with embedding QGIS in IPython is that the main app is
> written
> > in C++ and `import qgis` only works out of the box on Linux---for Windows
> > and OS X it is a bit of a hassle to get paths set up right.  Many things
> are
> > wrapped via SIP and it is possible to run a lot of widgets from IPython
> but
> > the main windows are not so it is currently not possible to re-launch the
> > app under Python.
> >
> > QGIS currently includes a Python console that makes use of
> > code.InteractiveInterpreter from the standard library. This console it
> works
> > very well with the exception that I always find myself pounding on the
> tab
> > key in vain when I forget what I can do with a random Python object.
> >  Replacing the QGIS Python console with an IPython console would be
> awesome
> > and would cut out a lot of trips to the QGIS source code and various
> > reference manuals.
> >
> > If an in-process kernel could be implemented that shared the same
> interface
> > as the QtKernelManager, then users could easily be given the option of
> > launching an external process for tasks where the GIL would be a bother.
> >
> > I am willing to take a stab at developing an in-process widget but after
> > thrashing around a bit this weekend I feel like some guidance would help
> me
> > focus my efforts.
> >
> >
> >
> >> > The plugin is available on [GitHub][2] under the tag `0.1` if anyone
> >> feels
> >> > interested in checking it out as an IPython use case.  The 0.1 version
> >> > is
> >> > really quite simple---about 5-10 lines in console.py that do the
> actual
> >> > implementation.  To use:
> >> >   - Install QGIS
> >> >   - Clone project as a sub-directory of ~/.qgis/python/plugins/
> >> >   - Run `make` (runs PyQt4 dev tools on ui and qrc files)
> >> >   - Launch QGIS and enable Plugin through Plugins -> Manage Plugins
> menu
> >> >
> >> > Keep up the good work on IPython development!
> >>
> >> Thanks!
> >>
> >> Brian
> >>
> >> > -Charlie
> >> > [1]: http://www.qgis.org
> >> > [2]: http://www.github.com/Sharpie/qgis-ipython/tree/0.1
> >> > _______________________________________________
> >> > IPython-dev mailing list
> >> > IPython-dev at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Brian E. Granger, Ph.D.
> >> Assistant Professor of Physics
> >> Cal Poly State University, San Luis Obispo
> >> bgranger at calpoly.edu
> >> ellisonbg at gmail.com
> >>
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110212/3552f71a/attachment.html>