[New-bugs-announce] [issue7255] "Default" word boundaries for Unicode data?

[daniel mccloy]

New submission from daniel mccloy <bangalore.online at gmail.com>:

Regarding UTS #18 (Unicode Standards for RegEx Engines), which can be
found at:
http://www.unicode.org/reports/tr18/

Is there a plan or commitment for Python to implement at least "default
word boundaries" (a Level 2 feature), rather than the current "simple
word boundaries"?  I don't believe that the algorithm for this is a
whole lot more complicated, but it certainly makes a huge difference for
processing non-Roman text.

For example, to match the whole word રત without matching the word રતા
(which has an additional vowel at the end, the vertical line), with
"default word boundary" recognition, you could use the pattern \bરત\b. 
With Python's current "simple word boundary" recognition, however, the
\b assertion is pretty much useless here, and I have yet to see a decent
zero-width pattern that can take its place.

BTW, the ICU regex libraries do provide this level of Unicode support:
http://userguide.icu-project.org/strings/regexp
It seems to work perfectly on Indic text, based on the tests I've done.

Being open-source, it may be a helpful reference for the algorithm needed.

Dan

----------
components: Regular Expressions
messages: 94856
nosy: RegEx4All
severity: normal
status: open
title: "Default" word boundaries for Unicode data?
type: feature request
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7255>
_______________________________________