[IPython-dev] Function specific hooks into the ipython tab completion system
Aaron Meurer
asmeurer at gmail.com
Fri Nov 30 10:52:45 EST 2012
On Nov 30, 2012, at 5:49 AM, Robert McGibbon <rmcgibbo at gmail.com> wrote:
Hey,
*[tl;dr (1) Prototype working (2) Dynamic language = limited potential (3)
Function annotations and*
*getting the ball rolling on annotation interoperability schemes]*
I have a working prototype of the system at
https://github.com/rmcgibbo/ipython/tree/functab.
Currently, the only type of matches that are supported are for enumerated
string literals.
I will be adding more soon, and making the interface such that it is easy
to extend the code with
new argument specific tab completion logic. The heavy lifting is done I
think.
Here's what you can do currently:
In [1]: from IPython.extensions.customtab import tab_completion
In [2]: @tab_completion(mode=['read', 'write'])
...: def my_io(fname, mode):
...: print 'lets go now!'
...:
In [3]: my_io('f.tgz', <TAB>
'read' 'write'
(or if there's only one matching completion, it'll obviously just fill in)
So when your cursor is in place to be adding the second argument, it will
do tab completions based on
two static strings read from the decorator. It pretty much as you'd expect,
supporting keyword args, etc.
The one thing that's tricky is dealing with quotation mark literals (' and
") in readline. This was my first
time coding against readline, so maybe I was making some rookie mistakes,
but it seems to me like if
you don't play any tricks, the quotation marks are not treated correctly.
When the current token is '"rea',
as if you've just started typing "read", and you hit tab, readline seems to
tokenize the line and associate
the opening quotation mark with the previous token, not the current one.
This means that if you include it
in the completion that you give, and you hit tab, the line will end up with
TWO opening quotation marks,
ala '""read'. (And it'll keep going if you keep hitting tab). I had to hack
my way around this, but if
anyone has any suggestions I'm all ears.
---
As Tom said, there's a lot you could do with this -- enumerated string
literals are just the tip of the
iceberg. Some other ones include glob-completed string literals for
filenames, type-based
matching of python objects in the current namespace (i.e. only show tab
completions for ndarray
objects in your session, etc).
You obviously can't do as much as you could in a statically typed language
-- if you have type-based tab
completion looking for ndarray objects, you're not going to be able to
infer that since the return value of
some random function like np.random.randn is going to be an ndarray, you
could tab complete that too.
While things like type checking can be done with decorators / annotations,
this isn't really the place. In particular,
the code for tab completion gets evaluated before you as a user ever hit
enter and execute your line of
code -- and itsnot appropriate to actually exec anything -- so there's no
way to know the type of the first
argument to a function like foo(bar(x), 1<TAB> at the time that the tab is
executed.
Function annotations also let you annotate the return type of a function.
----
As to the decorator vs function annotation syntax, I am hesitant to use
annotations for a few reasons. First, selfishly,
I don't use python3. (I would, but I depend on PyTabes for everything, and
it isn't py3k ready). Without the
syntactical support, manually attaching __annotations__ to functions seems
like a lame api.
It would be easy to make the decorators discussed before use
__annotations__ to store the data.
Second, in order to do annotations "right" in such a way that two
independent project don't have annotation semantics
that fail to interoperate, everyone needs to agree on some system. If one
library expects the annotations to be strings
(or to mean a specific thing, like a type for typechecking) and another
library expects something different, then everyone is
in a bad spot. PEP3107 <http://www.python.org/dev/peps/pep-3107/> seems to
want some convention to "develop organically". As the author of the PEP
said on
Python-ideas:
When the first big project like Zope or Twisted announces that "we will do
annotation interop like so...", everyone else will be pressured to
line up behind them.
I don't want to pick a mechanism, only to have to roll
another release when Zope or Twisted or some other big project pushes
out their own solution.
But, on the other hand, maybe there's no time like the present to start,
and although the PEP was finalized in 2010,
and it doesn't seem like Django or any other big package is using
annotations, so maybe we should lead the way?
This is exactly the point I was trying to make. Unless I'm mistaken, no
one in the scientific python community uses annotations yet. IPython is the
perfect project to start to standardize it for this community, and
hopefully push other communities as well.
In that spirit, I propose two possible schemes for annotation
interoperability:
(1) All the annotations should be dicts. That way we can put tab-complete
specific stuff under the 'tab' key and other
libraries/features can put their stuff under a different key, like
'type'or what have you. One advantage are that it's
simple. On the other hand, its kind of verbose, and IMO ugly.
def my_io(fname, mode : {'tab': ['read', 'write']}):
pass
def my_io(fname, mode : {'tab': ['read', 'write'], 'foo':'bar'}):
pass
I like this idea. In general, the elements of the list would be
TabCompletion objects that describe arbitrary tab completions (e.g., files
in the current directory).
(2). Inspired by ggplot <https://en.wikipedia.org/wiki/Ggplot2>, where you
compose plots by *summing* components, the annotations could be the sum of
objects that overload __add__, like:
def my_io(fname, mode : tab('read', 'write'))
pass
or
def my_io(fname, mode : tab('read', 'write') + typecheck(str)):
pass
Where tab and typecheck are something like:
class tab(object):
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
self.next = None
self.prev = None
def __add__(self, other):
if not hasattr(other, '__add__'):
raise ValueError('"{}" must also be composable'.format(other))
self.next = other
other.prev = self
return other
Then you could basically recover something analogous to the dictionary
in proposal one
by walking through the next/prev pointers. The advantages of this are
chiefly visual.
You could actually make this identical to option 1 by subclassing dict and
overriding __add__.
I'm not sure it's necessary for most libraries, though. If you want simple
syntax, just write a simple helper function so you can just do mode:
tab(["read", "write"]), where tab(a) returns {'tab': a}.
But if you just require the API __annotations__["arg"]['tab'], it doesn't
matter if you use a dict or a custom object.
Aaron Meurer
Obviously other things like lists or tuples or generic iterables could be
done too.
def my_io(fname, mode : (tab('read', 'write'), typecheck(str))):
pass
or
def my_io(fname, mode : [tab('read', 'write'), typecheck(str)]):
pass
With some cleverness, it might be possible to add an __iter__ method to the
tab class that
would let you turn the sum into a list via list(iterable), such that via
polymorphism the ggplot
style and the tuple/list style could exist side-by-side.
Basically, IMHO, using function annotation system requires some serious
thought -- probably
above my pay grade -- to do right.
(Sorry this email was so long)
-Robert
On Nov 29, 2012, at 6:38 AM, Tom Dimiduk wrote:
That looks pretty great. I would use something like that.
For the filename('.txt'), it could be handy to be able to pass arbitrary
globs (as in for glob.glob). You could still default to matching against
the end of the string for extensions, but adding the glob support costs
little (since you probably want to use glob internally anyway.
For extra (and unnecessary) fancyness, I could also see use cases for
from from tablib import tabcompletion, instance
class bar:
pass
@tabcompletion(foo=instance(bar))
to be able to only complete for specific types of objects for other
parameters (it would do an isinstance test).
Even more bonus points if the decorator could parse numpy styled docstrings
to grab that kind of information about parameters. I guess at this point
you could do type checking, file existence checking, and a variety of other
fun stuff there as well once you have that information, but that is almost
certainly going out of the scope of your proposal.
Sorry if I am growing your proposal too much, the basic thing you proposed
would still be very useful. If I can grab some spare mental cycles, I
would collaborate with you on it if you end up writing it.
Tom
On 11/29/2012 05:28 AM, Robert McGibbon wrote:
Hi,
Good spot, Matthias. I didn't see that method was already exposed -- I was
just looking at IPCompleter.matchers, which what that method inserts into.
Annotations are cool, but they're not obviously composable. I worry that
if I use them for this and then fix one syntax of how the annotation is
parsed, and somebody else
is using annotations in their lib for something else, the two schemes won't
be able to interoperate. Also they're py3k only.
My preferred syntax would be
from tablib import tabcompletion, filename,
@tabcompletion(fname=filename, mode=['r', 'w'])
def load(fname, mode, other_argument):
pass
or maybe with a parameterized filename to get specific extensions
@tabcompletion(fname=filename('.txt'))
def f(fname, other_argument):
pass
Is this something that other people would be interested in?
-Robert
On Nov 29, 2012, at 2:02 AM, Matthias BUSSONNIER wrote:
Hi,
I may be wrong, but IIRC you can insert your own completer in the IPython
completer chain and decide to filter the previous completion.
You should be able to have a custom completer that just forward the
previous completion in most cases,
And just do a dir completion if the object is np.loadtxt in your case (or
look at __annotations__ if you wish).
I've found one reference to inserting custom completer here
http://ipython.org/ipython-doc/dev/api/generated/IPython.core.interactiveshell.html?highlight=interactiveshell#IPython.core.interactiveshell.InteractiveShell.set_custom_completer
--
Matthias
Le 29 nov. 2012 à 10:27, Aaron Meurer a écrit :
I've often thought this as well. Probably a full-blown IPEP is in
order here. Perhaps __annotations__ would be the correct way to go
here.
Aaron Meurer
On Thu, Nov 29, 2012 at 12:58 AM, Robert McGibbon <rmcgibbo at gmail.com>
wrote:
Hey,
Tab completion in IPython is one of the things that makes it so useful,
especially the context specific tab completion for things like "from ..."
where only packages, or obviously the special completion for attributes when
the line contains a dot.
I use IPython for interactive data analysis a lot, and one of the most
frequent operations is loading up data with something like numpy.loadtxt()
or various related functions.
It would be really awesome if we could annotate functions to interact with
the tab completion system, perhaps for instance saying that argument 0 to
numpy.loadtxt() is supposed to be a filename, so let's give tab-complete
suggestions that try to look for directories/files. Some functions only
files with specific extensions, so you could filter based on that or
whatever.
By hacking on the code for completerlib.py:cd_completer, I sketched out a
little demo of what you could do with this: https://gist.github.com/4167151.
The architecture is totally wrong, but it lets you get behavior like:
```
In [1]: ls
datfile.dat dir1/ dir2/ file.gz random_junk test.py
In [2]: directory_as_a_variable = 'sdfsfsd'
In [3]: f = np.loadtxt(<TAB>
datfile.dat dir1/ dir2/ file.gz
In [4]: normal_function(<TAB>
Display all 330 possibilities? (y or n)
In [5]: g = np.loadtxt(di<TAB>
dict dir1/ directory_as_a_variable
divmod
dir dir2/ directory_of_my_choosing
```
Basically hitting the tab completion, when np.loadtxt is on the input line,
only shows directories and files that end with a certain extension. If you
start to type in the name of an object in your namespace, it'll show up too,
but only once you've typed in more than 1 matching character.
The implementation in my gist is pretty lame. The way I've coded it up, the
special behavior is based on simply finding the string "np.loadtxt" on the
input line, not on the actual function. This means you can't really make the
behavior specific to your position in the argument list (i.e. I know that
the first arg is a filename, and so should be tab completed like this, but
the other ones are not). I suspect the right way to do the implementation is
via function decorators to specify the behavior and then adding to
IPCompleter instead.
I think I'm up for giving this a shot.
Thoughts? Is this a feature anyone else would find interesting?
-Robert
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing
listIPython-dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20121130/cf47dec8/attachment.html>
More information about the IPython-dev
mailing list