From vinay_sajip at yahoo.co.uk Tue Dec 13 23:56:37 2011
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 13 Dec 2011 22:56:37 +0000 (UTC)
Subject: [Python-porting] lib2to3 and matching of fixers
Message-ID: <loom.20111213T235106-363@post.gmane.org>
I'm trying to write a custom lib2to3 fixer which, when processing the source
x = b'foo' b'bar'
will match the right hand side of the expression. From what I can see I should
be able to match it with 'atom' or 'power', but it doesn't seem to match either
of those. The individual byte-string tokens are matched using STRING, as
expected.
What should I use to match the right-hand side, for example if I want to do
constant folding and transform to something like b'foobar'?
Regards,
Vinay Sajip
From benjamin at python.org Wed Dec 14 02:30:16 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 13 Dec 2011 20:30:16 -0500
Subject: [Python-porting] lib2to3 and matching of fixers
In-Reply-To: <loom.20111213T235106-363@post.gmane.org>
References: <loom.20111213T235106-363@post.gmane.org>
Message-ID: <CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
2011/12/13 Vinay Sajip <vinay_sajip at yahoo.co.uk>:
> I'm trying to write a custom lib2to3 fixer which, when processing the source
>
> x = b'foo' b'bar'
>
> will match the right hand side of the expression. From what I can see I should
> be able to match it with 'atom' or 'power', but it doesn't seem to match either
> of those. The individual byte-string tokens are matched using STRING, as
> expected.
What are you using exactly? I would expect "atom< STRING+>" to work.
>
> What should I use to match the right-hand side, for example if I want to do
> constant folding and transform to something like b'foobar'?
--
Regards,
Benjamin
From vinay_sajip at yahoo.co.uk Wed Dec 14 12:43:10 2011
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 14 Dec 2011 11:43:10 +0000 (GMT)
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
In-Reply-To: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
Message-ID: <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
>> What are you using exactly? I would expect "atom< STRING+>" to
>> work.
>
>Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching
> language, but hopefully I'll get more familiar as I work with it!
I'm now trying to match a tuple using the following pattern:
PATTERN = "atom< '(' args=testlist_gexp< any+ > ')' >"
This is a subset of the pattern used in the isinstance fixer that Armin Ronacher wrote. However, the pattern fails to match the source
x = (int, long)
The parse tree for this source looks like this:
Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'x'), Leaf(22, '='), Node(atom, [Leaf(7, '('), Node(testlist_gexp, [Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long')]), Leaf(8, ')')])]), Leaf(4, '\n')]), Leaf(0, '')])
Since it contains an "atom" node with children consisting of a parenthesised "testlist_gexp" node, I'm not sure why there is no match. What am I missing?
Thanks and regards,
Vinay Sajip
From benjamin at python.org Wed Dec 14 18:35:25 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 14 Dec 2011 12:35:25 -0500
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
In-Reply-To: <1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
<1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
Message-ID: <CAPZV6o-+y3QX-ECExj1ebYyM-cZi0CVs_Q+1p7=YjSsH2goG6Q@mail.gmail.com>
2011/12/14 Vinay Sajip <vinay_sajip at yahoo.co.uk>:
>>> What are you using exactly? I would expect "atom< STRING+>" to
>
>>> work.
>>
>>Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching
>> language, but hopefully I'll get more familiar as I work with it!
>
>
> I'm now trying to match a tuple using the following pattern:
>
> PATTERN = "atom< '(' args=testlist_gexp< any+ > ')' >"
>
>
> This is a subset of the pattern used in the isinstance fixer that Armin Ronacher wrote. However, the pattern fails to match the source
>
> x = (int, long)
>
>
> The parse tree for this source looks like this:
>
> Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'x'), Leaf(22, '='), Node(atom, [Leaf(7, '('), Node(testlist_gexp, [Leaf(1, 'int'), Leaf(12, ','), Leaf(1, 'long')]), Leaf(8, ')')])]), Leaf(4, '\n')]), Leaf(0, '')])
You have to account for the newline.
--
Regards,
Benjamin
From vinay_sajip at yahoo.co.uk Wed Dec 14 21:38:35 2011
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 14 Dec 2011 20:38:35 +0000 (UTC)
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
<1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
<CAPZV6o-+y3QX-ECExj1ebYyM-cZi0CVs_Q+1p7=YjSsH2goG6Q@mail.gmail.com>
Message-ID: <loom.20111214T191055-653@post.gmane.org>
Benjamin Peterson <benjamin at ...> writes:
>
> You have to account for the newline.
>
Thanks for the response, but it's not clear how or why. As to the "why": the
portion of the tree I get by looking at
tree.children[0].children[0].children[2]
is
Node(atom, [
Leaf(7, '('),
Node(testlist_gexp, [
Leaf(1, 'int'),
Leaf(12, ','),
Leaf(1, 'long')
]),
Leaf(8, ')')
])
which is the bit I expect to match. It does not contain the newline (that's
present as the last child the enclosing grandparent node (simple_stmt) obtained
by looking at tree.children[0]:
Node(simple_stmt, [
Node(expr_stmt, [
Leaf(1, 'x'),
Leaf(22, '='),
Node(atom, [
Leaf(7, '('),
Node(testlist_gexp, [
Leaf(1, 'int'),
Leaf(12, ','),
Leaf(1, 'long')
]),
Leaf(8, ')')
])
]),
Leaf(4, '\n')
])
The pattern I used appears consistent with the subset of the pattern used in
the is_instance fixer to match a tuple in the isinstance call, other than
whitespace in the pattern string itself:
power<
'isinstance'
trailer< '(' arglist< any ',' atom< '('
args=testlist_gexp< any+ >
')' > > ')' >
>
in the is_instance fixer, as against my fixer's
atom< '(' args=testlist_gexp< any+ > ')' >
So I can't see why the newline needs to be accounted for, nor how to do it.
Arranging the whitespace in my pattern to exactly match that segment in the
is_instance fixer's pattern made no difference, as I would have expected.
So, how do I account for the newline, and why do I need to when it's two levels
up in the tree?
Thanks for your patience with lib2to3 newbie.
Regards,
Vinay Sajip
From benjamin at python.org Wed Dec 14 23:18:19 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 14 Dec 2011 17:18:19 -0500
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
In-Reply-To: <loom.20111214T191055-653@post.gmane.org>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
<1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
<CAPZV6o-+y3QX-ECExj1ebYyM-cZi0CVs_Q+1p7=YjSsH2goG6Q@mail.gmail.com>
<loom.20111214T191055-653@post.gmane.org>
Message-ID: <CAPZV6o8-cGPQcLdQv-C8+x_-9HEAUQBPg8uvcb_sCEP8+bKTog@mail.gmail.com>
2011/12/14 Vinay Sajip <vinay_sajip at yahoo.co.uk>:
> Benjamin Peterson <benjamin at ...> writes:
>
>>
>> You have to account for the newline.
>>
>
> Thanks for the response, but it's not clear how or why. As to the "why": the
> portion of the tree I get by looking at
Sorry. I was trying to read the parse tree from the linear version in
your last mail and failed. You are correct.
Does your fixer us BM_compatible = True? Try setting that to false.
--
Regards,
Benjamin
From vinay_sajip at yahoo.co.uk Thu Dec 15 00:01:52 2011
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 14 Dec 2011 23:01:52 +0000 (GMT)
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
In-Reply-To: <CAPZV6o8-cGPQcLdQv-C8+x_-9HEAUQBPg8uvcb_sCEP8+bKTog@mail.gmail.com>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
<1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
<CAPZV6o-+y3QX-ECExj1ebYyM-cZi0CVs_Q+1p7=YjSsH2goG6Q@mail.gmail.com>
<loom.20111214T191055-653@post.gmane.org>
<CAPZV6o8-cGPQcLdQv-C8+x_-9HEAUQBPg8uvcb_sCEP8+bKTog@mail.gmail.com>
Message-ID: <1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com>
>
> Does your fixer us BM_compatible = True? Try setting that to false.
>
With BM_compatible = False, the match is indeed made, and my fixer's transform method called. Thanks!
Is the meaning of the BM_compatible flag, or the overall matching and transformation algorithm, documented somewhere? Most of the fixers I looked at had BM_compatible = True, and I took my cue from the fix_isinstance fixer as it appeared closest to what I was trying to do.
Regards,
Vinay Sajip
From benjamin at python.org Thu Dec 15 00:05:31 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 14 Dec 2011 18:05:31 -0500
Subject: [Python-porting] Fw: lib2to3 and matching of fixers
In-Reply-To: <1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
<1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
<1323862990.653.YahooMailNeo@web25807.mail.ukl.yahoo.com>
<CAPZV6o-+y3QX-ECExj1ebYyM-cZi0CVs_Q+1p7=YjSsH2goG6Q@mail.gmail.com>
<loom.20111214T191055-653@post.gmane.org>
<CAPZV6o8-cGPQcLdQv-C8+x_-9HEAUQBPg8uvcb_sCEP8+bKTog@mail.gmail.com>
<1323903712.50651.YahooMailNeo@web25801.mail.ukl.yahoo.com>
Message-ID: <CAPZV6o96YaRg6_7Ysf6AxYtsW+iprgVU=uYQDvReQxjFvWPx0Q@mail.gmail.com>
2011/12/14 Vinay Sajip <vinay_sajip at yahoo.co.uk>:
>>
>
>> Does your fixer us BM_compatible = True? Try setting that to false.
>>
>
> With BM_compatible = False, the match is indeed made, and my fixer's transform method called. Thanks!
>
> Is the meaning of the BM_compatible flag, or the overall matching and transformation algorithm, documented somewhere? Most of the fixers I looked at had BM_compatible = True, and I took my cue from the fix_isinstance fixer as it appeared closest to what I was trying to do.
No, unfortunately. The BM matcher is faster, but has weird behavior on
some things (as you can see).
--
Regards,
Benjamin
From vinay_sajip at yahoo.co.uk Wed Dec 14 10:23:23 2011
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 14 Dec 2011 09:23:23 +0000 (GMT)
Subject: [Python-porting] lib2to3 and matching of fixers
In-Reply-To: <CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
References: <loom.20111213T235106-363@post.gmane.org>
<CAPZV6o_=uZiH=BBFQPnn1ig1oJMadqOdKpMWFQ09YHfG+7YnYg@mail.gmail.com>
Message-ID: <1323854603.84819.YahooMailNeo@web25802.mail.ukl.yahoo.com>
> What are you using exactly? I would expect "atom< STRING+>" to
> work.
Thanks, that worked. I tried "atom" and "STRING+" separately - sorry, I'm unfamiliar with the pattern-matching language, but hopefully I'll get more familiar as I work with it!
Regards,
Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20111214/f5f45076/attachment.html>