[IPython-dev] IPEP 11: Tab Completion System Refactor

[Robert McGibbon]

Takafumi,

Thanks a lot for your thoughts! I will definitely look closely at the Jedi library.

In both the the way it currently exists, and in the way I've been thinking about
IPEP 11, the completion system is very much based around lines. A lot of
this has to do with GNU readline and the terminal IPython client -- you're entering
lines one at a time, so the widest context that the completion can occur in is the line.
Obviously, for non-readline frontends, especially the notebook, the wider context
is now not just the line but the entire code cell. I will put more thought into how exactly
the cursor position should be denoted within these multi-line environments.

As I started implementing the first draft of IPEP 11, I realized some of the subtleties of
the namespacing that aren't reflected in the IPEP -- they should be added. I'm still not
exactly what is the most appropriate. What I've been doing is holding a reference t
the IPython shell object (instance of an InteractiveShellABC subclass) in the
BaseMatcher. The shell contains both the `user_ns` and a `user_global_ns` 
namespaces. They're generally the same, except in cases where you embed IPython
in another process or something.

Maybe it would be cleaner for the BaseMatcher to hold a reference to the namespaces directly,
so you don't have to "tunnel" though the shell object to get them. What do you think?

As to the typing of the returned completions {str -> set(str)}, I chatted with Fernando a little bit about
this. There's a tension between flexibility and simplicity here, and also speed. Creating a lot of little
container objects might be a bit of a performance problem. The final set of completions is going to
have to be sent, presumably as JSON, over the wire to non-readline frontends, and the results are
going to have to be displayed to the user. If we make the return 'attributes' of the completion too 
flexible, I wonder how we're going to effectively display them to the user? It'll be hard to design a
consistent UI/UX around a spec that says that each completion can be infinitely variable and flexible.

But perhaps you're right that more than one 'attribute' per match is desirable. Do you have an idea for
what an effective UI would look like that would be able to take advantage of all of that information?

I will definitely look closely at jedi and its features / design. I don't want to "overbuild" this IPython
feature, but at the same time, this type of thing could be a HUGE win for the IPython notebook!

Cheers,
-Robert

On Mar 19, 2013, at 11:20 AM, Takafumi Arakaki wrote:

> Hi,
> 
> I am also interested in extending IPython completion.  I wrote a patch
> for Jedi (an awesome Python auto-completion library) to use it with
> Python interpreter and it is in review process here:
> https://github.com/davidhalter/jedi/pull/145.
> Jedi statically analyzes Python sources so you can complete instances
> or returned values, even before actually creating the object.  For example,
> you can complete `os.path.join("a", "b").<TAB>`.
> 
> I read IPEP 11 to see if it provides sufficient information for Jedi API.
> First I want to make sure about what `cursor_position` is.  The docstring
> says:
> 
>> cursor_position : int, optional
>>    The position of the cursor within the line, at the time that
>>    the completion key was activated. If not supplied, the cursor
>>    will be assumed to have been at the end of the line.
> 
> I think it should be clear if it is zero-origin or not.  Also, I am not
> sure if I understand what does it mean by "within the line".  If I am
> at the first column of the second line, is it zero or the length of the
> first line?
> 
> Also, I am not sure how CompletionManager gets namespaces (e.g.,
> `locals()`).  I assume you get it from a singleton IPython instance,
> but I think passing namespaces around is cleaner.
> 
> Same goes for `BaesMatcher` and `CompletionEvent`.  It seems there is
> no way to get the cursor position and namespaces from `BaesMatcher`
> subclasses.  I suggest:
> 
> 1. Add attribute(s) to hold cursor position in `CompletionEvent`.
>   It can be `lineno` and `column` or just `position`.
> 
> 2. Add `namespaces` attribute (list of dict) to `CompletionEvent`
>   or pass `namespaces` to `BaseMatcher.match`.
> 
> 
> Regarding the returned value `completions` (dict, {str -> list(str)}),
> why not a list of `Completion` (another class) instance?  One
> completion can have many "attributes".  If we follow the current
> suggestion, we can attach only one "attribute" (e.g., file, directory,
> etc.) and they are exclusive.  Probably it is OK enough, but I think
> it is hard to extend.  For example, Jedi's completion has
> many attributes:
> - https://jedi.readthedocs.org/en/dev/docs/plugin-api.html#api_classes.Completion
> - https://jedi.readthedocs.org/en/dev/docs/plugin-api.html#api_classes.BaseDefinition
> So, it is not possible to send all these information to clients.
> 
> 
> BTW, I only read this thread and the IPEP on the wiki.  Please tell me
> if there are discussions I am missing.
> 
> 
> [1] Here is list of supported cases:
>    https://github.com/davidhalter/jedi/pull/145#issuecomment-14346096.
> 
> 
> -- Takafumi
> 
> 
> On Tue, Mar 12, 2013 at 12:20 AM, Robert McGibbon <rmcgibbo at gmail.com> wrote:
>> Hey,
>> 
>> After talking with Fernando on Friday, I've updated IPEP11. The most
>> substantial change to the document is that we're proposing a modification to
>> the completion messaging protocol
>> (http://ipython.org/ipython-doc/dev/development/messaging.html#complete).
>> 
>> The goal is that each tab completion option be associated with a "kind",
>> e.g: file, directory, object, magic, etc. This will enable non-readline
>> clients to display richer, more contextual information to the user.  Another
>> goal is to simplify the code base. It's pretty messy and incomprehensible
>> now.
>> 
>> At this point, I think that the plan is pretty much fleshed out. Any input
>> would really be appreciated.
>> 
>> -Robert
>> 
>> On Feb 25, 2013, at 8:51 PM, Robert McGibbon wrote:
>> 
>> Hi,
>> 
>> I've posted IPEP 11, which is a proposal to refactor the kernel side tab
>> completion machinery. There are two three for refactoring: the first is to
>> provide a richer API for new tab completion matchers to interact with
>> IPython, enabling, for example, projects like PR2701 to be done more
>> cleanly. The second goal is to make the tab completion system less tied to
>> GNU readline and capable of delivering richer contextual information to
>> non-readline frontends like the notebook. The third is to clean up and
>> simplify the existing code.
>> 
>> https://github.com/ipython/ipython/wiki/IPEP-11%3A-Tab-Completion-System-Refactor
>> 
>> Any and all thoughts are appreciated.
>> 
>> -Robert
>> 
>> 
>> 
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev