[Patches] [ python-Patches-468457 ] Minor changes to sre.Scanner

[noreply@sourceforge.net]

Patches item #468457, was opened at 2001-10-05 18:59
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=468457&group_id=5470

Category: Library (Lib)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Greg Chapman (glchapman)
>Assigned to: Fredrik Lundh (effbot)
Summary: Minor changes to sre.Scanner

Initial Comment:
I've been experimenting with sre.Scanner as a token 
generator for the spark parser.  The following patch 
makes three changes.  The first I believe is a bug fix 
(though perhaps I am not using the Scanner class as 
expected); the other two are enhancement suggestions:

1) In my experience with the current sre.Scanner, if 
the first pattern in the lexicon matches, an exception 
is raised because match.lastindex == None (and so 
cannot be used as an index into the lexicon list).  I 
think this is because the first pattern's subgroup id 
equals 0.  Changing the subgroup setup loop so that 
the group ids start with 1 eliminates this exception.

2) I needed to use case-insensitive matching for the 
tokens, and adding (?i) to the patterns didn't work.  
I believe that to get the pattern compiled using 
flags, they have to be set in the sre_parse.Pattern 
object which is allocated in Scanner.__init__; this 
will not happen with the current Scanner because the 
calls to sre_parse.parse(phrase) don't have access to 
the top-level Pattern object, so they can't set the 
flags in it.  I added an optional flags parameter to 
the Scanner class's constructor which allows setting 
global flags for compilation of the lexicon patterns.

3) Looking in _sre.c, I found the internal scanner 
object, which allows repeated matching over a string.  
This object looks ready-made for this kind of 
scanning, so I changed Scanner.scan to use it.

I've marked each change line with the number of the 
change (1, 2, or 3) with which it is associated.  As 
for docs and tests, I'm not sure what (if anything) to 
add, since none exist for the current Scanner class.




----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=468457&group_id=5470