Mailman 3 August 2007 - Python-Dev

Re: [Python-Dev] Regular expressions, Unicode etc.
by Nick Maclaren Aug. 10, 2007

Aug. 10, 2007

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > Your specification was "For Unicode, whatever people agree!" > > I would not call that "Unicode-based". Can we drop this, please? I am happy to agree that I was being unclear (it is a common failing of mine), but I did provide the specification I coded. Specifically, and in full, I said: For Unicode, whatever people agree! I use the criterion that it has a defined category that doesn't … [View More]

1 0

Re: [Python-Dev] Regular expressions, Unicode etc.
by Nick Maclaren Aug. 10, 2007

Aug. 10, 2007

I am not on "Python 3000", so am restricting. Mike Klaas <mike.klaas(a)gmail.com> wrote: > > > I have needed to push my stack to teach REs (don't ask), and am > > taking a look at the RE code. I may be able to extend it to support > > RFE 694374 and (more importantly) atomic groups and possessive > > quantifiers. While I regard such things as revolting beyond belief, > > they make a HELL of a difference to the efficiency of recognising > > things … [View More]

4 3

Re: [Python-Dev] Regular expressions, Unicode etc.
by Nick Maclaren Aug. 10, 2007

Aug. 10, 2007

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > There is no problem for isalnum: it will just go away if > byte-oriented characters go away. Fortunately, we have a > replacement for the Unicode case. As we do for isprint. > The relevance is that your specification of "printing character" > as "isprint returns true" is nearly useless, as it only applies > to byte-oriented characters. Eh? That's ALL I used it to specify! I used a … [View More]Unicode-based specification for Unicode. > Unicode-isalnum is defined as isalpha|isdecimal|isdigit|isnumeric. > isalpha means categories Ll, Lu, Lt, Lo, Lm. isdecimal means > character has the decimal property. isigit means the character has > the digit property. isnumeric means the character has the numeric > property. I sincerely hope it isn't! Using a mixture of categories and properties is truly horrible, because it isn't unlikely that some future version of Unicode will introduce anomalies, even if there aren't any there already. And the character aliases file doesn't include any properties called 'digit' or 'decimal' or anything much like them, so they need a painful amount of reverse engineering to determine what characters they bind to. It LOOKS as if they are the subcategories, which would be OK. A much cleaner and more future-proof specification would be any category beginning with 'L' or 'N'. For example, Unicode doesn't CURRENTLY have a category for indeterminate numbers or sacred case, such as are used in some languages, but it isn't implausible that it would add them :-) > It was a proposal for a definition. English is not my native > language, and "printing character" means nothing to me. So > I kindly asked for a definition, and suggested one possibility. > I would not have guessed that you consider white-space characters > as "printing", as they don't actually print anything. Ah. It's not an ordinary English term. It's a computer language one, so I assumed that you would know it. It is older than C, but C standardised its use to mean any of the characters which are intended to display (or leave a blank) with standard, single positioning semantics. Almost all languages derived from C use it in the same sense, and Python has a fair amount of C ancestry. > The problem is that you did not quite mention a rule, or else > I missed it. I did, and you did! I said that it should be any character with a defined category that is not 'control'. > You seem to be asking for being able to express "not a control > character". I propose that this is best done with UTS#18, > in which you would write > > [\P{C}] # or \P{Other} > > If this is what you want, I'm all in favor of having it > implemented. Excellent! We are agreed. Yes, that is equivalent. I am NOT volunteering to add the support of that to the parser, especially now I have discovered the format of the intermediate data :-( It would be a foul task, and it isn't clear what syntax to use, anyway. There is the horrible POSIX syntax, which I blame (perhaps wrongly) on HP-UX, and the Java one, which I believe is a modified subset of the example in UTS#8. But that says: All syntax and API presented in this document is only for the purpose of illustration; there is absolutely no requirement to follow such syntax or API. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 [View Less]

3 2

Re: [Python-Dev] Unicode database
by Nick Maclaren Aug. 10, 2007

Aug. 10, 2007

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > > I think that you will find that you are using a non-standard > > environment and set of Python sources. > > Please trust me that I didn't. See below. I always trust people as much as I trust myself, but I do tend to check up. See below. > Ah, the makefile. I don't think you use it create the Unicode database. > > It's only good for generating the codecs (Lib/encodings) Yes, … [View More]but it DOES attempt to download the mappings, and is the ONLY script which attempts to do so. beelzebub$find Python-2.5.1 -type f | wc 3458 3460 135981 beelzebub$find Python-2.5.1 -type f | xargs grep ftp.unicode.org Python-2.5.1/Doc/lib/libunicodedata.tex:4.1.0 which is publicly available from \url{ftp://ftp.unicode.org/}. grep: Python-2.5.1/Mac/Icons/Disk: No such file or directory grep: Image.icns: No such file or directory grep: Python-2.5.1/Mac/Icons/Python: No such file or directory grep: Folder.icns: No such file or directory Python-2.5.1/Misc/NEWS: at ftp.unicode.org and contain a few updates (e.g. the Mac OS Python-2.5.1/Tools/unicode/Makefile:# files available at ftp://ftp.unicode.org/ Python-2.5.1/Tools/unicode/Makefile: ncftpget -R ftp.unicode.org . Public/MAPPINGS Python-2.5.1/Tools/unicode/gencodec.py:site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:# ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT the Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:# ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT Python-2.5.1/Tools/unicode/python-mappings/KOI8-U.TXT:# ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT Python-2.5.1/Tools/unicode/python-mappings/CP1140.TXT:# ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT Python-2.5.1/Modules/unicodedata.c:4.1.0 which is publically available from ftp://ftp.unicode.org/.\n > AFAICT, the mappings are still where they always were: at the > location given in the Makefile. (e.g. > ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT > ) Then you DEFINITELY are using a non-standard set of files. That above was from the source of Python 2.5.1 that I have just downloaded. > Did you really believe the Unicode consortium doesn't have the > old versions of the character database online? Do you think > they are complete fools? Please don't be offensive. I said that I had failed to find them, after searching the Unicode Web site. Now that you have give me the actual file name, I can find them, but searching on the version and request for that database leads to unhelpful files. > Googling for "unicode 3.2 ucd" gives me > > http://unicode.org/Public/3.2-Update/ > > as the top hit (of course, you have to know that they call > the character database "ucd" to invoke that query). Generally, I distrust Google for such things, as it is as likely to lead to you the wrong information as the right one. For example, that hit you found was on a different logical server, and could well be an incorrect version of the database. It is VERY common for such things to 'escape' into Google. Have you checked whether or not that file is correct with the Unicode consortium? Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 [View Less]

3 2

Move to a "py3k" branch *DONE*
by Guido van Rossum Aug. 9, 2007

Aug. 9, 2007

Please spread the word. The py3k-struni branch is dead! Don't use it any more. --Guido ---------- Forwarded message ---------- From: Guido van Rossum <guido(a)python.org> Date: Aug 9, 2007 7:43 AM Subject: Move to a "py3k" branch *DONE* To: Python 3000 <python-3000(a)python.org> Cc: Neal Norwitz <nnorwitz(a)gmail.com> This is done. The new py3k branch is ready for business. If you currently have the py3k-struni branch checked out (at its top level), *don't update*, but … [View More]issue the following commands: svn switch svn+ssh://pythondev@svn.python.org/python/branches/py3k svn update Only a small amount of activity should result (unless you didn't svn update for a long time). For the p3yk branch, the same instructions will work, but the svn update will update most of your tree. A "make clean" is recommended in this case. Left to do: - update the wikis - clean out the old branches - switch the buildbot and the doc builder to use the new branch (Neal) There are currently about 7 failing unit tests left: test_bsddb test_bsddb3 test_email test_email_codecs test_email_renamed test_sqlite test_urllib2_localnet See http://wiki.python.org/moin/Py3kStrUniTests for detailed status regarding these. --Guido On 8/9/07, Guido van Rossum <guido(a)python.org> wrote: > I am starting now. Please, no more checkins to either p3yk ot py3k-struni. > > On 8/8/07, Guido van Rossum <guido(a)python.org> wrote: > > I would like to move to a new branch soon for all Py3k development. > > > > I plan to name the branch "py3k". It will be branched from > > py3k-struni. I will do one last set of merges from the trunk via p3yk > > (note typo!) and py3k-struni, and then I will *delete* the old py3k > > and py3k-struni branches (you will still be able to access their last > > known good status by syncing back to a previous revision). I will > > temporarily shut up some unit tests to avoid getting endless spam from > > Neal's buildbot. > > > > After the switch, you should be able to switch your workspaces to the > > new branch using the "svn switch" command. > > > > If anyone is in the middle of something that would become painful due > > to this changeover, let me know ASAP and I'll delay. > > > > I will send out another message when I start the move, and another > > when I finish it. > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- --Guido van Rossum (home page: http://www.python.org/~guido/) -- --Guido van Rossum (home page: http://www.python.org/~guido/) [View Less]

1 0

cc: "Martin v. Löwis" <martin@v.loewis.de>
by Nick Maclaren Aug. 9, 2007

Aug. 9, 2007

Re: [Python-Dev] Regular expressions, Unicode etc. =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > I recommend you use the 4.1 version of the database; this should > work out of the box, with no change to the build environment at > all. I tried that, of course. See below. > As for updating it - that has to wait until the next release > of Python. At that point, 5.1 might be releasesd, so 5.0 might > get skipped altogether. Very true. … [View More]> I would likely close such a report as "works for me" (after testing > it does - it did when I last ran it, which was before the release > of Python 2.5). I think that you will find that you are using a non-standard environment and set of Python sources. I started off with the standard distribution. > It did not suffer from bit-rot - it still works just fine for > the version of the database that is supported. Really? I have just checked 2.5.1, and the same defects are there. > As for the need for redesigning - I don't see that need. What specific > aspect do you think needs redesigning? If you merely meant to say > "I don't understand the code" - this is not enough reason, I > remember it took me some time to understand it as well, but now > I see that it does precisely what it needs to do, and precisely > in the way it needs to do that. Well, here are a selection of the issues that I found: The Makefile includes the command: ncftpget -R ftp.unicode.org . Public/MAPPINGS Not merely is ncftpget not a standard utility, the current mappings are no longer at that location. Indeed, I can see nothing useful in that directory at present, though I haven't searched it in depth! Looking through www.unicode.org, I could find the relevant files for 5.0.0, but for no other version. No, I am NOT going to type in over a megabyte of data from the PDF! makeunicodedata.py has a reference to the Unicode 3.2 files, but they are not present in the standard distribution, the Makefile doesn't fetch them, and I can't find them. makeunicodedata.py refers to (for example) UnicodeData.txt and Modules/unicodedata_db.h as such, which rather requires it to be run in a particular directory. I can find nothing in any file even referring to this. Having run it, running 'make all' does not rebuild Python correctly. I couldn't be bothered to work out why, so I hit it with the usual trick, 'make distclean'. And, of course, it SHOULD be possible to upgrade the Unicode data without having to change version of Python! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 [View Less]

2 1

Re: [Python-Dev] Regular expressions, Unicode etc.
by Nick Maclaren Aug. 9, 2007

Aug. 9, 2007

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: > > >> Before discussing the escape, I'd like to see a specification of > >> it first - what characters precisely would classify as "printing"? > > > > For basic ASCII and locale-based testing, whatever isprint() says. > > Just as for isalpha(). > > In the mediate term, locale-based testing will go away/be not > implementable (in particular, Py3k won't have a byte-… [View More]oriented > character string type, so we can't use isprint). In general, > isprint is unsuitable since it doesn't support multi-byte > character sets. Well, iswprint isn't so restricted :-) I don't see the relevance of this, as EXACTLY the same problem applies to isalnum and \w. If you can solve one problem (and you have to solve the latter), you can solve the other. > > For Unicode, whatever people agree! I use the criterion that it > > has a defined category that doesn't start with 'C' - which is what > > I think that most people will accept. > > -1. There must be a better specification than that. > > Can you please explain the concept of "printing character"? If > you have a Unicode code point, how do you determine whether it > is printing? If rendering it would generate black pixels on white > background? Eh? This is a character set we are talking about. The proposed extensions to include font and colour are an aberration that I shall thankfully be long retired before they hit. Unicode has a two letter classification of each character, with the main category being in upper case and the subsidiary one in lower. Let's ignore the latter, as it is irrelevant. The main categories are 'Z' (spaces), 'L' (letters), 'N' (numbers), 'S' (Symbols), 'P' (punctuation), 'M' (marks) and 'C' control characters. There are some pretty weird entries in 'L' and 'N' and the difference between 'S', P' and 'M' is arcane, to a degree. But all of the categories except 'C' are things that display, and 'C' is mainly the ASCII controls we know and, er, love - with some similar extras. Obviously, unclassified characters should not be called printing, and equally obviously controls shouldn't. There is no clear reason why the others should not be - especially as the difference between a modifying accent and a free-standing one is something so obscure that most people don't even know that there IS one. The point about an escape for printing characters is to check for bad characters in text input, and the rule I mentioned is fine for that. What's the problem with it? Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 [View Less]

2 1

Regular expressions, Unicode etc.
by Nick Maclaren Aug. 9, 2007

Aug. 9, 2007

I have needed to push my stack to teach REs (don't ask), and am taking a look at the RE code. I may be able to extend it to support RFE 694374 and (more importantly) atomic groups and possessive quantifiers. While I regard such things as revolting beyond belief, they make a HELL of a difference to the efficiency of recognising things like HTML tags in a morass of mixed text. The other approach, which is to stick to true regular expressions, and wholly or partially convert to DFAs, has … [View More]

3 4

Please help verify SF data dump imported into (future) new tracker
by Brett Cannon Aug. 9, 2007

Aug. 9, 2007

We are getting very close to moving over to the new tracker (hopefully by the end of the month; no firm date yet, though, as we are still planning things out)! Part of the transition is taking a data dump provided by SourceForge and loading it into our Roundup instance. But we need to make some effort to make sure SF's data dump is accurate and that our import is good. If you can, please go to SourceForge and choose some issue (bug, patch, whatever), and then look up the corresponding issue … [View More]

1 0

Re: [Python-Dev] Regular expressions, Unicode etc.
by Nick Maclaren Aug. 9, 2007

Aug. 9, 2007

Further to the above, I found the Unicode sources, have rebuilt the files, but it involved some fairly serious hacking to the building mechanism and I have had to disable the Unicode 3.2 support. And, of course, that means that 4 of the tests fail. This area needs addressing, not least because Python should clearly be upgraded to Unicode 5.0.0 (which is what I am using) at some stage. I am not sure how best to report a bug that essentially says "The build mechanisms for Unicode have suffered … [View More]

3 2