[IPython-dev] variable information request

Brian Granger ellisonbg at gmail.com
Mon Apr 11 12:45:22 EDT 2011


On Mon, Apr 11, 2011 at 3:28 AM, Jason Grout <jason.grout at drake.edu> wrote:
> On 4/10/11 12:59 AM, Brian Granger wrote:
>> Jason,
>> On Sat, Apr 9, 2011 at 4:53 AM, Jason Grout<jason-sage at creativetrax.com>
>>  wrote:
>>> Here's a crazy idea that could lead to some nice interactively-updating
>>> components of an IPython frontend.  Right now, in the new messaging
>>> protocol, it appears that I can send a computation request and then ask
>>> for the output to contain the values of several variables (the
>>> "user_variables" field of a execute_reply message).  However, what if I
>>> want to check the value of a variable *during* a computation and get an
>>> immediate response?  I might, for example, have a box that prints out
>>> the current value of a root approximation, for example, or a slider that
>>> contains the current iteration number, and I want these to be updated
>>> fairly frequently while the computation is running.
>> I think this is where the display system comes in.  We have extended
>> the model of the displayhook to allow that macheniery to be triggered
>> by users anywhere in their code by simply calling our top-level
>> display functions:
>> https://github.com/ipython/ipython/blob/master/IPython/core/display.py
>> Calling these functions causes a JSON message to be sent to all
>> frontends instantly (it doesn't wait for the code to finish).  These
>> messages can have extra metedata and a A given frontend could easily
>> look at that metadata and decide how to display the JSON data.
> But that requires that I actually call the display function from within
> code.  That answers the case where I am actually writing all of the relevant
> code and can anticipate in advance what information a user will want.
>  Another use-case is allowing the user to just query about whatever
> variables they want, at any time, without needing to anticipate what
> information needs to be sent ahead of time.

Ahh, yes, you want a pull mechanism that allows users to request a
variable on the fly, rather than the push approach I described that
does require statements in the code.

>>> Here's one way to do it, I think:  Run a separate thread that just
>>> answers these queries.  Since the GIL handles access to variables, I
>>> think it's okay for the separate thread to retrieve the value of a
>>> variable and return that, and it seamlessly does this while the main
>>> thread is carrying on a computation.  For example, the following code
>>> will print out the current iteration number every time enter is pressed:
>> Almost always, in Python, threads are not the answer.  This type of
>> service would go down the second that non-GIL releasing extension code
>> is run.  This is one of the main reasons we have gone to pyzmq, as all
>> of its networking stuff is handled in GIL releasing C++ threads.  The
>> benefit of the existing display logic that uses pyzmq is that it will
>> continue to work fine in these situations.
> It seems that the service wouldn't go down as much as the service would be
> delayed, right?  That seems rather unavoidable if we want to access a Python
> variable without corruption.    As soon as the GIL was released, the
> variables would be queried and values sent.  We face the same issue in doing
> a display() inside the code---the display() has to run after the non-GIL
> releasing extension code is run too.  So I don't see the advantage of using
> a display() in the code we are trying to investigate and query, versus using
> a separate thread to query values in the currently running code.

I agree that for what you are wanting, display is not the right solution.

In the threading approach, the user namespace would have to be
protected with a lock.  The main challenge is that we are extremely
weary of growing the number of threads in IPython.  Currently we only
have one place where we use a thread in the core of IPython:  for
saving the users history.  This is done to hide the latency of sqlite
writes and it appears that the GIL is released during this io.  While
I am not ready to add another thread to enable the feature you are
asking about, I don't see any reason we couldn't make is possible to
implement as some sort of IPython extension.  It would also require a
new pyzmq channel.



> Thanks,
> Jason
> --
> Jason Grout
> jason.grout at drake.edu
> Math and Computer Science Department
> Drake University

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com

More information about the IPython-dev mailing list