when looking through skip's coverage listing, I noted a bug in SRE: #define SRE_UNI_IS_ALNUM(ch) ((ch) < 256 ? isalnum((ch)) : 0) this predicate is used for \w when a pattern is compiled using the "unicode locale" (flag U), and should definitely not use 8-bit locale stuff. however, there's no such thing as a Py_UNICODE_ISALNUM (or even a Py_UNICODE_ISALPHA). what should I do? how about using: Py_UNICODE_ISLOWER || Py_UNICODE_ISUPPER || Py_UNICODE_ISTITLE || Py_UNICODE_ISDIGIT </F>
Fredrik Lundh wrote:
when looking through skip's coverage listing, I noted a bug in SRE:
#define SRE_UNI_IS_ALNUM(ch) ((ch) < 256 ? isalnum((ch)) : 0)
this predicate is used for \w when a pattern is compiled using the "unicode locale" (flag U), and should definitely not use 8-bit locale stuff.
however, there's no such thing as a Py_UNICODE_ISALNUM (or even a Py_UNICODE_ISALPHA). what should I do? how about using:
Py_UNICODE_ISLOWER || Py_UNICODE_ISUPPER || Py_UNICODE_ISTITLE || Py_UNICODE_ISDIGIT
This will give you all cased chars along with all digits; it ommits the non-cased ones. It's a good start, but probably won't cover the full range of letters + numbers. Perhaps we need another table for isalpha in unicodectype.c ? (Or at least one which defines all non-cased letters.) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
mal wrote:
Py_UNICODE_ISLOWER || Py_UNICODE_ISUPPER || Py_UNICODE_ISTITLE || Py_UNICODE_ISDIGIT
This will give you all cased chars along with all digits; it ommits the non-cased ones.
but of course...
It's a good start, but probably won't cover the full range of letters + numbers.
Perhaps we need another table for isalpha in unicodectype.c ? (Or at least one which defines all non-cased letters.)
+1 from me (SRE needs this, and it doesn't really make much sense to add unicode tables to SRE just because the built-in ones are slightly incomplete...) how about this plan: -- you add a Py_UNICODE_ALPHA to unicodeobject.h asap, which does exactly that (or I can do that, if you prefer). (and maybe even a Py_UNICODE_ALNUM) -- I change SRE to use that asap. -- you, I, or someone else add a better implementation, some other day. </F>
Fredrik Lundh wrote:
mal wrote:
Py_UNICODE_ISLOWER || Py_UNICODE_ISUPPER || Py_UNICODE_ISTITLE || Py_UNICODE_ISDIGIT
This will give you all cased chars along with all digits; it ommits the non-cased ones.
but of course...
It's a good start, but probably won't cover the full range of letters + numbers.
Perhaps we need another table for isalpha in unicodectype.c ? (Or at least one which defines all non-cased letters.)
+1 from me (SRE needs this, and it doesn't really make much sense to add unicode tables to SRE just because the built-in ones are slightly incomplete...)
how about this plan:
-- you add a Py_UNICODE_ALPHA to unicodeobject.h asap, which does exactly that (or I can do that, if you prefer). (and maybe even a Py_UNICODE_ALNUM)
Ok, I'll add Py_UNICODE_ISALPHA and Py_UNICODE_ISALNUM (first with approximations of the sort you give above and later with true implementations using tables in unicodectype.c) on Monday... gotta run now.
-- I change SRE to use that asap.
-- you, I, or someone else add a better implementation, some other day.
</F>
Nice weekend :) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
Fredrik Lundh wrote:
how about this plan:
-- you add a Py_UNICODE_ALPHA to unicodeobject.h asap, which does exactly that (or I can do that, if you prefer). (and maybe even a Py_UNICODE_ALNUM)
Ok, I'll add Py_UNICODE_ISALPHA and Py_UNICODE_ISALNUM (first with approximations of the sort you give above and later with true implementations using tables in unicodectype.c) on Monday... gotta run now.
-- I change SRE to use that asap.
-- you, I, or someone else add a better implementation, some other day.
I've just looked into this... the problem here is what to consider as being "alpha" and what "numeric". I could add two new tables for the characters with category 'Lo' (other letters, not cased) and 'Lm' (letter modifiers) to match all letters in the Unicode database, but those tables have some 5200 entries (note that there are only 804 lower case letters and 686 upper case ones). Note that there seems to be no definition of what is to be considered alphanumeric in Unicode. The only quote I found was in http://www.w3.org/TR/xslt#convert which says: """ Alphanumeric means any character that has a Unicode category of Nd, Nl, No, Lu, Ll, Lt, Lm or Lo. """ Here's what the glibc has to say about these chars: /* Test for any wide character for which `iswupper' or 'iswlower' is true, or any wide character that is one of a locale-specific set of wide-characters for which none of `iswcntrl', `iswdigit', `iswpunct', or `iswspace' is true. */ extern int iswalpha __P ((wint_t __wc)); Question: Should I go ahead and add the Lo and Lm tables to unicodectype.c ? Pros: standards confrom Cons: huge in size -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
FYI, I've added two new macros which allow querying alphabetic and alphanumeric characters: Py_UNICODE_ISALPHA() and Py_UNICODE_ISALNUM() The implementation is currently only experimental -- some 5200 chars are missing from being correctly identified as being alphanumeric (see my other post on the topic). -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[M.-A. Lemburg]
"M.-A. Lemburg" wrote:
Fredrik Lundh wrote:
how about this plan:
-- you add a Py_UNICODE_ALPHA to unicodeobject.h asap, which does exactly that (or I can do that, if you prefer). (and maybe even a Py_UNICODE_ALNUM)
Ok, I'll add Py_UNICODE_ISALPHA and Py_UNICODE_ISALNUM (first with approximations of the sort you give above and later with true implementations using tables in unicodectype.c) on Monday... gotta run now.
-- I change SRE to use that asap.
-- you, I, or someone else add a better implementation, some other day.
I've just looked into this... the problem here is what to consider as being "alpha" and what "numeric".
I could add two new tables for the characters with category 'Lo' (other letters, not cased) and 'Lm' (letter modifiers) to match all letters in the Unicode database, but those tables have some 5200 entries (note that there are only 804 lower case letters and 686 upper case ones).
In JDK1.3, Character.isLetter(..) and Character.isDigit(..) are documented as: http://java.sun.com/j2se/1.3/docs/api/java/lang/Character.html#isLetter(char) http://java.sun.com/j2se/1.3/docs/api/java/lang/Character.html#isDigit(char) http://java.sun.com/j2se/1.3/docs/api/java/lang/Character.html#isLetterOrDig...) I guess that java uses the extra huge tables. regards, finn
participants (3)
-
bckfnn＠worldonline.dk -
Fredrik Lundh -
M.-A. Lemburg