[IPython-dev] IPEP 11: Tab Completion System Refactor

Takafumi Arakaki aka.tkf at gmail.com
Wed Mar 20 09:11:47 EDT 2013 

Hi Robert

If the completion system must be readline-friendly, probably providing
column number is a better idea.  To prevent loosing information about
the cursor position, we should add line number.  As calculating cursor
position as a number of characters from the beginning of code (cell)
from a pair of line number and column number is hard, and as some
library may need it, additionally having position also makes sense.

> Maybe it would be cleaner for the BaseMatcher to hold a reference to the namespaces directly,
> so you don't have to "tunnel" though the shell object to get them. What do you think?

I think the question is whether we need to access other attributes and
methods of InteractiveShellABC.  If not, having a list of namespaces
is better because then the components are well separated.

Regarding the completion objects, I think speed is not so much problem
here.  I am using Jedi over an PRC protocol and sending many
attributes (even docstrings!) with it but I feel no speed issue so
far.  I agree having a good UI to show completions with any attributes
is impossible.  My point is that if you can extend it or not without
breaking existing completion modules.  But I have at least two useful
attributes in mind.  One is "annotation" where you can put short
information for the completion.  The other one is the "kind" you are
talking about in the IPEP.  For example of "annotation", see this
screenshot of the Emacs plugin of Jedi:
http://stackoverflow.com/a/14710037/727827

Besides extensibility, a list of completion objects has other benefit.
If matcher can order completions, you can pass that information to
clients.  If you have separated lists for each "kind", you can't order
completions across "kinds".

Takafumi


On Wed, Mar 20, 2013 at 5:12 AM, Robert McGibbon <rmcgibbo at gmail.com> wrote:
> Takafumi,
>
> Thanks a lot for your thoughts! I will definitely look closely at the Jedi library.
>
> In both the the way it currently exists, and in the way I've been thinking about
> IPEP 11, the completion system is very much based around lines. A lot of
> this has to do with GNU readline and the terminal IPython client -- you're entering
> lines one at a time, so the widest context that the completion can occur in is the line.
> Obviously, for non-readline frontends, especially the notebook, the wider context
> is now not just the line but the entire code cell. I will put more thought into how exactly
> the cursor position should be denoted within these multi-line environments.
>
> As I started implementing the first draft of IPEP 11, I realized some of the subtleties of
> the namespacing that aren't reflected in the IPEP -- they should be added. I'm still not
> exactly what is the most appropriate. What I've been doing is holding a reference t
> the IPython shell object (instance of an InteractiveShellABC subclass) in the
> BaseMatcher. The shell contains both the `user_ns` and a `user_global_ns`
> namespaces. They're generally the same, except in cases where you embed IPython
> in another process or something.
>
> Maybe it would be cleaner for the BaseMatcher to hold a reference to the namespaces directly,
> so you don't have to "tunnel" though the shell object to get them. What do you think?
>
> As to the typing of the returned completions {str -> set(str)}, I chatted with Fernando a little bit about
> this. There's a tension between flexibility and simplicity here, and also speed. Creating a lot of little
> container objects might be a bit of a performance problem. The final set of completions is going to
> have to be sent, presumably as JSON, over the wire to non-readline frontends, and the results are
> going to have to be displayed to the user. If we make the return 'attributes' of the completion too
> flexible, I wonder how we're going to effectively display them to the user? It'll be hard to design a
> consistent UI/UX around a spec that says that each completion can be infinitely variable and flexible.
>
> But perhaps you're right that more than one 'attribute' per match is desirable. Do you have an idea for
> what an effective UI would look like that would be able to take advantage of all of that information?
>
> I will definitely look closely at jedi and its features / design. I don't want to "overbuild" this IPython
> feature, but at the same time, this type of thing could be a HUGE win for the IPython notebook!
>
> Cheers,
> -Robert
>
> On Mar 19, 2013, at 11:20 AM, Takafumi Arakaki wrote:
>
>> Hi,
>>
>> I am also interested in extending IPython completion.  I wrote a patch
>> for Jedi (an awesome Python auto-completion library) to use it with
>> Python interpreter and it is in review process here:
>> https://github.com/davidhalter/jedi/pull/145.
>> Jedi statically analyzes Python sources so you can complete instances
>> or returned values, even before actually creating the object.  For example,
>> you can complete `os.path.join("a", "b").<TAB>`.
>>
>> I read IPEP 11 to see if it provides sufficient information for Jedi API.
>> First I want to make sure about what `cursor_position` is.  The docstring
>> says:
>>
>>> cursor_position : int, optional
>>>    The position of the cursor within the line, at the time that
>>>    the completion key was activated. If not supplied, the cursor
>>>    will be assumed to have been at the end of the line.
>>
>> I think it should be clear if it is zero-origin or not.  Also, I am not
>> sure if I understand what does it mean by "within the line".  If I am
>> at the first column of the second line, is it zero or the length of the
>> first line?
>>
>> Also, I am not sure how CompletionManager gets namespaces (e.g.,
>> `locals()`).  I assume you get it from a singleton IPython instance,
>> but I think passing namespaces around is cleaner.
>>
>> Same goes for `BaesMatcher` and `CompletionEvent`.  It seems there is
>> no way to get the cursor position and namespaces from `BaesMatcher`
>> subclasses.  I suggest:
>>
>> 1. Add attribute(s) to hold cursor position in `CompletionEvent`.
>>   It can be `lineno` and `column` or just `position`.
>>
>> 2. Add `namespaces` attribute (list of dict) to `CompletionEvent`
>>   or pass `namespaces` to `BaseMatcher.match`.
>>
>>
>> Regarding the returned value `completions` (dict, {str -> list(str)}),
>> why not a list of `Completion` (another class) instance?  One
>> completion can have many "attributes".  If we follow the current
>> suggestion, we can attach only one "attribute" (e.g., file, directory,
>> etc.) and they are exclusive.  Probably it is OK enough, but I think
>> it is hard to extend.  For example, Jedi's completion has
>> many attributes:
>> - https://jedi.readthedocs.org/en/dev/docs/plugin-api.html#api_classes.Completion
>> - https://jedi.readthedocs.org/en/dev/docs/plugin-api.html#api_classes.BaseDefinition
>> So, it is not possible to send all these information to clients.
>>
>>
>> BTW, I only read this thread and the IPEP on the wiki.  Please tell me
>> if there are discussions I am missing.
>>
>>
>> [1] Here is list of supported cases:
>>    https://github.com/davidhalter/jedi/pull/145#issuecomment-14346096.
>>
>>
>> -- Takafumi
>>
>>
>> On Tue, Mar 12, 2013 at 12:20 AM, Robert McGibbon <rmcgibbo at gmail.com> wrote:
>>> Hey,
>>>
>>> After talking with Fernando on Friday, I've updated IPEP11. The most
>>> substantial change to the document is that we're proposing a modification to
>>> the completion messaging protocol
>>> (http://ipython.org/ipython-doc/dev/development/messaging.html#complete).
>>>
>>> The goal is that each tab completion option be associated with a "kind",
>>> e.g: file, directory, object, magic, etc. This will enable non-readline
>>> clients to display richer, more contextual information to the user.  Another
>>> goal is to simplify the code base. It's pretty messy and incomprehensible
>>> now.
>>>
>>> At this point, I think that the plan is pretty much fleshed out. Any input
>>> would really be appreciated.
>>>
>>> -Robert
>>>
>>> On Feb 25, 2013, at 8:51 PM, Robert McGibbon wrote:
>>>
>>> Hi,
>>>
>>> I've posted IPEP 11, which is a proposal to refactor the kernel side tab
>>> completion machinery. There are two three for refactoring: the first is to
>>> provide a richer API for new tab completion matchers to interact with
>>> IPython, enabling, for example, projects like PR2701 to be done more
>>> cleanly. The second goal is to make the tab completion system less tied to
>>> GNU readline and capable of delivering richer contextual information to
>>> non-readline frontends like the notebook. The third is to clean up and
>>> simplify the existing code.
>>>
>>> https://github.com/ipython/ipython/wiki/IPEP-11%3A-Tab-Completion-System-Refactor
>>>
>>> Any and all thoughts are appreciated.
>>>
>>> -Robert
>>>
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

More information about the IPython-dev mailing list