[IPython-dev] Input transformation rework - input requested

[Aaron Meurer]

I believe you're right.

I suppose another potential transform would be to allow x^2 to
represent x**2. This would have be done at the token level since ^ has
a different precedence than **.

Only operating on valid Python seems fine. If someone wants to
implement a DSL they should just write a kernel.

Have you considered using something like parso, which can manipulate
an AST and tokens losslessly?

Aaron Meurer

On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> Thanks Aaron. I am aware that Sympy uses input transformation, but it would
> be good to check some more details of exactly how:
>
> 1. I believe wrapping integers and defining symbols can both be done with
> AST transformations. Let me know if I'm missing something.
> 2. Wrapping floats can't, because the string form is discarded before we get
> the AST. But this is still an operation that changes one syntactically valid
> piece of Python code to another, right? I'm considering limiting the API to
> only operate on already valid Python code, and using private APIs to handle
> IPython's special syntax (like %magics).
> 3. Are there any other transformations that Sympy does (or wants to do)
> besides these three?
>
> On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
>>
>> I'm sure you're already aware, but we use them for SymPy for some
>> (optional) transformers that make Python more symbolic friendly
>> (auto-replace undefined variables with Symbols, wrap integer literals
>> with Integer so that exact rational numbers work, and so on). The AST
>> transformer is nice but limiting. We can't use it to wrap float
>> literals with higher precision for instance because Python drops the
>> precision of floats in the AST.
>>
>> There are already some open issues about some limitations we've seen
>> (https://github.com/sympy/sympy/issues/14440 and
>> https://github.com/ipython/ipython/issues/10893). I'm not fully
>> updated on the specifics of what the limitations were, but from what I
>> remember, there were issues with IPython doing things line-by-line.
>>
>> I don't know how the new code is architected, but it would be nice if
>> the transformers could just access the raw input from IPython, and
>> various transformers could be plugged in to that to make things
>> simpler if desired (the AST transformer could be one such instance).
>>
>> At the very least, if you want to test the new design, you could look
>> at translating our transformers in SymPy (including the work in
>> progress float transformer at
>> https://github.com/sympy/sympy/pull/13300).
>>
>> Aaron Meurer
>>
>> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
>> wrote:
>> > Thomas this is great news! I think that part of the code base could
>> > benefit
>> > from being simpler and more extensible. Thanks for tackling this!
>> >
>> > Sent from my iPhone
>> >
>> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
>> >
>> > Hi all,
>> >
>> > For IPython 7, I'm planning a rework of the input transformation
>> > framework:
>> > https://github.com/ipython/ipython/pull/11041
>> >
>> > The new framework is - I hope - simpler than the one it replaces, and
>> > will
>> > hopefully have fewer weird corner cases. But it's still a pretty complex
>> > beast, and I don't feel like it's a good platform for people to add more
>> > transformations to. One option to improve this is to make a smaller
>> > extensible API, restricting the kind of things that third party code can
>> > do.
>> >
>> > I don't think there's ever been much third party code extending
>> > IPython's
>> > input transformation, but I'd like to find out about the code that does.
>> > So
>> > if you're aware of a project that does use our input transformation API:
>> >
>> > - What project?
>> > - What does it want to transform?
>> > - Can you point me to the code?
>> > - What fallback options would it have if IPython didn't support the
>> > transformation it wanted?
>> >
>> > This is specifically about text transformations; AST transformations are
>> > not
>> > changed.
>> >
>> > Thanks,
>> > Thomas
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at python.org
>> https://mail.python.org/mailman/listinfo/ipython-dev
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>