Mailman 3 May 2000 - Python-Dev

Re: [XML-SIG] Re: Unicode debate
by ht＠cogsci.ed.ac.uk 05 May '00

05 May '00

"Fredrik Lundh" <fredrik(a)pythonware.com> writes: > Guido van Rossum <guido(a)python.org> wrote: > > Thanks -- but that's really Tcl's fault, since the only way to get > > character data *into* Tcl (or out of it) is through the UTF-8 > > encoding. > > from http://dev.scriptics.com/man/tcl8.3/TclLib/StringObj.htm > > Tcl_NewUnicodeObj(Tcl_UniChar* unicode, int numChars) > > Tcl_NewUnicodeObj and Tcl_SetUnicodeObj create a new > object or modify an existing object to hold a copy of the > Unicode string given by unicode and numChars. > > (Tcl_UniChar* is currently the same thing as Py_UNICODE*) > Any way this can be exploited in Tkinter? ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

1 0

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by ht＠cogsci.ed.ac.uk 05 May '00

05 May '00

Guido van Rossum <guido(a)python.org> writes: > > I think I hear a moderate consensus developing that the 'ASCII > > proposal' is a reasonable compromise given the time constraints. But > > let's not fail to come back to this ASAP -- it _really_ narcs me that > > every time I load XML into my Python-based editor I'm going to convert > > large amounts of wide-string data into UTF-8 just so Tk can convert it > > back to wide-strings in order to display it! > > Thanks -- but that's really Tcl's fault, since the only way to get > character data *into* Tcl (or out of it) is through the UTF-8 > encoding. > > And is your XML really stored on disk in its 16-bit format? No, I have no idea what encoding it's in, my XML parser supports over a dozen encodings, and quite sensibly always delivers the content, as per the XML REC, as wide-strings. ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

1 0

FW: weird bug in test_winreg
by Mark Hammond 04 May '00

04 May '00

Oops - I didnt notice the CC - a copy of what I sent to Guido: -----Original Message----- From: Mark Hammond [mailto:mhammond@skippinet.com.au] Sent: Thursday, 4 May 2000 11:13 AM To: Guido van Rossum Subject: RE: weird bug in test_winreg Hah - I was just thinking about this this myself. If I wasnt waiting 24 hours, I would have beaten you to the test_fork1 patch :-) However, there is something bad going on. If you remove your test_fork1 patch, and run it from regrtest (_not_ stand alone) you will see the children threads die with: File "L:\src\Python-cvs\Lib\test\test_fork1.py", line 30, in f alive[id] = os.getpid() AttributeError: 'None' object has no attribute 'getpid' Note the error - os is None! [The reason is only happens as part of the test is because the children are created before the main thread fails with the attribute error] Similarly, I get spurious: Traceback (most recent call last): File ".\test_thread.py", line 103, in task2 mutex.release() AttributeError: 'None' object has no attribute 'release' (Only rarely, and never when run stand-alone - the test_fork1 exception happens 100% of the time from the test suite) And of course the test_winreg one. test_winreg, I guessed, may be caused by the import lock (but its certainly not obvious how or why!?). However, that doesnt explain the others. I also saw these _before_ I applied the threading patches (and after!) So I think the problem may be a little deeper? Mark.

2 2

Minimal (single-file) Python?
by gvwilson＠nevex.com 04 May '00

04 May '00

Hi. Has anyone ever built, or thought about building, a single-file Python, in which all the "basic" capabilities are included in a single executable (where "basic" means "can do as much as the Bourne shell")? Some of the entries in the Software Carpentry competition would like to be able to bootstrap from as small a starting point as possible. Thanks, Greg p.s. I don't think this is the same problem as moving built-in features of Python into optionally-loaded libraries, as some of the things in the 'sys', 'string', and 'os' modules would have to move in the other direction to ensure Bourne shell equivalence.

1 0

Unicode strings: an alternative
by Just van Rossum 04 May '00

04 May '00

Today I had a relatively simple idea that unites wide strings and narrow strings in a way that is more backward comatible at the C level. It's quite possible this has already been considered and rejected for reasons that are not yet obvious to me, but I'll give it a shot anyway. The main concept is not to provide a new string type but to extend the existing string object like so: - wide strings are stored as if they were narrow strings, simply using two bytes for each Unicode character. - there's a flag that specifies whether the string is narrow or wide. - the ob_size field is the _physical_ length of the data; if the string is wide, len(s) will return ob_size/2, all other string operations will have to do similar things. - there can possibly be an encoding attribute which may specify the used encoding, if known. Admittedly, this is tricky and involves quite a bit of effort to implement, since all string methods need to have narrow/wide switch. To make it worse, it hardly offers anything the current solution doesn't. However, it offers one IMHO _big_ advantage: C code that just passes strings along does not need to change: wide strings can be seen as narrow strings without any loss. This allows for __str__() & str() and friends to work with unicode strings without any change. Any thoughts? Just

4 6

weird bug in test_winreg
by Guido van Rossum 04 May '00

04 May '00

I just noticed a weird traceback in test_winreg. When I import test.autotest on Windows, I get a "test failed" notice for test_winreg. When I run it by itself the test succeeds. But when I first import test.autotest and then import test.test_winreg (which should rerun the latter, since test.regrtest unloads all test modules after they have run), I get an AttributeError telling me that 'None' object has no attribute 'get'. This is in encodings.__init__.py in the first call to _cache.get() in search_function. Somehow this is called by SetValueEx() in WriteTestData() in test/test_winreg.py. But inspection of the encodings module shows that _cache is {}, not None, and the source shows no evidence of how this could have happened. Any suggestions? --Guido van Rossum (home page: http://www.python.org/~guido/)

1 1

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by tpassin＠home.com 03 May '00

03 May '00

Guido van Rossum wrote, about how to represent strings: > Paul, we're both just saying the same thing over and over without > convincing each other. I'll wait till someone who wasn't in this > debate before chimes in. I'm with Paul and Federick on this one - at least about characters being the atoms of a string. We **have** to be able to refer to **characters** in a string, and without guessing. Otherwise, how could you ever construct a test, like theString[3]==[a particular japanese ideograph]? If we do it by having a "string" datatype, which is really a byte list, and a "unicodeString" datatype which is a list of abstract characters, I'd say everyone could get used to working with them. We'd have to supply conversion functions, of course. This route might be the easiest to understand for users. We'd have to be very clear about what file.read() would return, for example, and all those similar read and write functions. And we'd have to work out how real 8-bit calls (like writing to a socket?) would play with the new types. For extra clarity, we could leave string the way it is, introduce stringU (unicode string) **and** string8 (Latin-1 or byte list, whichever seems to be the best equivalent to the current string). Then we would deprecate string in favor of string8. Then if tcl and perl go to unicode strings we pass them a stringU, and if they go some other way, we pass them something else. COme to think of it, we need some some data type that will continue to work with c and c++. Would that be string8 or would we keep string for that purpose? Clarity and ease of use for the user should be primary, fast implementations next. If we didn't care about ease of use and clarity, we could all use Scheme or c, don't use sight of it. I'd suggest we could create some use cases or scenarios for this area - needs input from those who know encodings and low level Python stuff better than I. Then we could examine more systematically how well various approaches would work out. Regards, Tom Passin

13 50

Re: [Python-checkins] CVS: python/dist/src/Python bltinmodule.c,2.154,2.155
by Jack Jansen 03 May '00

03 May '00

> _PyBuiltin_Init_2(): Don't test Py_UseClassExceptionsFlag, just go > ahead and initialize the class-based standard exceptions. If this > fails, we throw a Py_FatalError. Isn't a Py_FatalError overkill? Or will not having the class-based standard exceptions lead to so much havoc later on that it is better than limping on? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

Unicode comparisons & normalization
by Just van Rossum 03 May '00

03 May '00

After quickly browsing through the unicode.org URLs I posted earlier, I reach the following (possibly wrong) conclusions: - there is a script and language independent canonical form (but automatic normalization is indeed a bad idea) - ideally, unicode comparisons should follow the rules from http://www.unicode.org/unicode/reports/tr10/ (But it seems hardly realistic for 1.6, if at all...) - this would indeed mean that it's possible for u == v even though type(u) is type(v) and len(u) != len(v). However, I don't see how this would collapse /F's world, as the two strings are at most semantically equivalent. Their physical difference is real, and still follows the a-string-is-a-sequence-of-characters rule (!). - there may be additional customized language-specific sorting rules. I currently don't see how to implement that without some global variable. - the sorting rules are very complicated, and should be implemented by calculating "sort keys". If I understood it correctly, these can take up to 4 bytes per character in its most compact form. Still, for it to be somewhat speed-efficient, they need to be cached... - u.find() may need an alternative API, which returns a (begin, end) tuple, since the match may not have the same length as the search string... (This is tricky, since you need the begin and end indices in the non-canonical form...) Just

4 4

PROPOSAL: exposure of values in limits.h and float.h
by Trent Mick 03 May '00

03 May '00

I apologize if I am hitting covered ground. What about a module (called limits or something like that) that would expose some appropriate #define's in limits.h and float.h. For example: limits.FLT_EPSILON could expose the C DBL_EPSILON limits.FLT_MAX could expose the C DBL_MAX limits.INT_MAX could expose the C LONG_MAX (although that particulay name would cause confusion with the actual C INT_MAX) - Does this kind of thing already exist somewhere? Maybe in NumPy. - If we ever (perhaps in Py3K) turn the basic types into classes then these could turn into constant attributes of those classes, i.e.: f = 3.14159 f.EPSILON = <as set by C's DBL_EPSILON> - I thought of these values being useful when I thought of comparing two floats for equality. Doing a straight comparison of floats is dangerous/wrong but is it not okay to consider two floats reasonably equal iff: -EPSILON < float2 - float1 < EPSILON Or maybe that should be two or three EPSILONs. It has been a while since I've done any numerical analysis stuff. I suppose the answer to my question is: "It depends on the situation." Could this algorithm for float comparison be a better default than the status quo? I know that Mark H. and others have suggested that Python should maybe not provide a float comparsion operator at all to beginners. Trent -- Trent Mick trentm(a)activestate.com

2 1

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by jkraai＠murlmail.com 02 May '00

02 May '00

The ever quotable Guido: > English is the one natural language that we can all sue to communicate ------------------------------------------------------------------ You've received MurlMail! -- FREE, web-based email, accessible anywhere, anytime from any browser-enabled device. Sign up now at http://murl.com Murl.com - At Your Service

1 0

Re: [Python-checkins] CVS: python/dist/src/Doc/lib libos.tex,1.38,1.39
by pf＠artcom-gmbh.de 02 May '00

02 May '00

Barry A. Warsaw: > Update of /projects/cvsroot/python/dist/src/Doc/lib [...] > libos.tex [...] > Availability: Macintosh, \UNIX{}, Windows. > \end{funcdesc} > --- 703,712 ---- > \end{funcdesc} > > ! \begin{funcdesc}{utime}{path, times} > ! Set the access and modified times of the file specified by \var{path}. > ! If \var{times} is \code{None}, then the file's access and modified > ! times are set to the current time. Otherwise, \var{times} must be a > ! 2-tuple of numbers, of the form \var{(atime, mtime)} which is used to > ! set the access and modified times, respectively. > Availability: Macintosh, \UNIX{}, Windows. > \end{funcdesc} I may have missed something, but I haven't seen a patch to the WinXX and MacOS implementation of the 'utime' function. So either the documentation should explicitly point out, that the new additional signature is only available on Unices or even better it should be implemented on all platforms so that programmers intending to write portable Python have not to worry about this. I suggest an additional note saying that this signature has been added in Python 1.6. There used to be several such notes all over the documentation saying for example: "New in version 1.5.2." which I found very useful in the past! Regards, Peter

4 4

Traceback style
by Ka-Ping Yee 02 May '00

02 May '00

This was also going to go out after i posted the display/displaytb patch. But anyway, let's see what you all think. I propose the following stylistic changes to traceback printing: 1. If there is no function name for a given level in the traceback, just omit the ", in ?" at the end of the line. 2. If a given level of the traceback is in a method, instead of just printing the method name, print the class and the method name. 3. Instead of beginning each line with: File "foo.py", line 5 print the line first and drop the quotes: Line 5 of foo.py In the common interactive case that the file is a typed-in string, the current printout is File "<stdin>", line 1 and the following is easier to read in my opinion: Line 1 of <stdin> Here is an example: >>> class Spam: ... def eggs(self): ... return self.ham ... >>> s = Spam() >>> s.eggs() Traceback (innermost last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in eggs AttributeError: ham With the suggested changes, this would print as Traceback (innermost last): Line 1 of <stdin> Line 3 of <stdin>, in Spam.eggs AttributeError: ham -- ?!ng "In the sciences, we are now uniquely privileged to sit side by side with the giants on whose shoulders we stand." -- Gerald Holton

7 11

documentation for new modules
by Fred L. Drake, Jr. 02 May '00

02 May '00

The "winreg" module needs some documentation; is anyone here up to the task? I don't think I know enough about the registry to write something reasonable. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives

3 5

fun with unicode, part 1
by Fredrik Lundh 02 May '00

02 May '00

>>> filename = u"gröt" >>> file = open(filename, "w") >>> file.close() >>> import glob >>> print glob.glob("gr*") ['gr\303\266t'] >>> print glob.glob(u"gr*") [u'gr\366t'] >>> import os >>> os.system("dir gr*") ... GRÇôT 0 01-02-03 12.34 grÇôt 1 fil(es) 0 byte 0 dir 12 345 678 byte free hmm. </F>

9 12

At the interactive port
by Moshe Zadka 02 May '00

02 May '00

Continuing the recent debate about what is appropriate to the interactive prompt printing, and the wide agreement that whatever we decide, users might think otherwise, I've written up a patch to have the user control via a function in __builtin__ the way things are printed at the prompt. This is not patches@python level stuff for two reasons: 1. I'm not sure what to call this function. Currently, I call it __print_expr__, but I'm not sure it's a good name 2. I haven't yet supplied a default in __builtin__, so the user *must* override this. This is unacceptable, of course. I'd just like people to tell me if they think this is worth while, and if there is anything I missed. *** ../python/dist/src/Python/ceval.c Fri Mar 31 04:42:47 2000 --- Python/ceval.c Sat Apr 29 03:55:36 2000 *************** *** 1014,1047 **** case PRINT_EXPR: v = POP(); ! /* Print value except if None */ ! /* After printing, also assign to '_' */ ! /* Before, set '_' to None to avoid recursion */ ! if (v != Py_None && ! (err = PyDict_SetItemString( ! f->f_builtins, "_", Py_None)) == 0) { ! err = Py_FlushLine(); ! if (err == 0) { ! x = PySys_GetObject("stdout"); ! if (x == NULL) { ! PyErr_SetString( ! PyExc_RuntimeError, ! "lost sys.stdout"); ! err = -1; ! } ! } ! if (err == 0) ! err = PyFile_WriteObject(v, x, 0); ! if (err == 0) { ! PyFile_SoftSpace(x, 1); ! err = Py_FlushLine(); ! } ! if (err == 0) { ! err = PyDict_SetItemString( ! f->f_builtins, "_", v); ! } } ! Py_DECREF(v); break; case PRINT_ITEM: --- 1014,1035 ---- case PRINT_EXPR: v = POP(); ! x = PyDict_GetItemString(f->f_builtins, ! "__print_expr__"); ! if (x == NULL) { ! PyErr_SetString(PyExc_SystemError, ! "__print_expr__ not found"); ! Py_DECREF(v); ! break; ! } ! t = PyTuple_New(1); ! if (t != NULL) { ! PyTuple_SET_ITEM(t, 0, v); ! w = PyEval_CallObject(x, t); ! Py_XDECREF(w); } ! /*Py_DECREF(x);*/ ! Py_XDECREF(t); break; case PRINT_ITEM: -- Moshe Zadka <mzadka(a)geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com

4 7

Traceback behaviour in exceptional cases
by Ka-Ping Yee 02 May '00

02 May '00

Here is how i was planning to take care of exceptions in sys.displaytb... 1. When the 'sys' module does not contain a 'stderr' attribute, Python currently prints 'lost sys.stderr' to the original stderr instead of printing the traceback. I propose that it proceed to try to print the traceback to the real stderr in this case. 2. If 'sys.stderr' is buffered, the traceback does not appear in the file. I propose that Python flush 'sys.stderr' immediately after printing a traceback. 3. Tracebacks get printed to whatever object happens to be in 'sys.stderr'. If the object is not a file (or other problems occur during printing), nothing gets printed anywhere. I propose that Python warn about this on stderr, then try to print the traceback to the real stderr as above. 4. Similarly, 'sys.displaytb' may cause an exception. I propose that when this happens, Python invoke its default traceback printer to print the exception from 'sys.displaytb' as well as the original exception. #4 may seem a little convoluted, so here is the exact logic i suggest (described here in Python but to be implemented in C), where 'handle_exception()' is the routine the interpreter uses to handle an exception, 'print_exception' is the built-in exception printer currently implemented in PyErr_PrintEx and PyTraceBack_Print, and 'err' is the actual, original stderr. def print_double_exception(tb, exc, disptb, dispexc, file): file.write("Exception occured during traceback display:\n") print_exception(disptb, dispexc, file) file.write("\n") file.write("Original exception passed to display routine:\n") print_exception(tb, exc, file) def handle_double_exception(tb, exc, disptb, dispexc): if hasattr(sys, 'stderr'): err.write("Missing sys.stderr; printing exception to stderr.\n") print_double_exception(tb, exc, disptb, dispexc, err) return try: print_double_exception(tb, exc, disptb, dispexc, sys.stderr) except: err.write("Error on sys.stderr; printing exception to stderr.\n") print_double_exception(tb, exc, disptb, dispexc, err) def handle_exception(): tb, exc = sys.exc_traceback, sys.exc_value try: sys.displaytb(tb, exc) except: disptb, dispexc = sys.exc_traceback, sys.exc_value try: handle_double_exception(tb, exc, disptb, dispexc) except: pass def default_displaytb(tb, exc): if hasattr(sys, 'stderr'): print_exception(tb, exc, sys.stderr) else: print "Missing sys.stderr; printing exception to stderr." print_exception(tb, exc, err) sys.displaytb = sys.__displaytb__ = default_displaytb -- ?!ng "In the sciences, we are now uniquely privileged to sit side by side with the giants on whose shoulders we stand." -- Gerald Holton

1 0

Neil Hodgson on python-dev?
by Mark Hammond 02 May '00

02 May '00

I'd like to propose that we invite Neil Hodgson to join the python-dev family. Neil is the author of the Scintilla editor control, now used by wxPython and Pythonwin... Smart guy, and very experienced with Python (scintilla was originally written because he had trouble converting Pythonwin to be a color'd editor :-) But most relevant at the moment is his Unicode experience. He worked for along time with Fujitsu, working with Japanese and all the encoding issues there. I have heard him echo the exact sentiments of Andy. He is also in the process of polishing the recent Unicode support in Scintilla. As this Unicode debate seems to be going nowhere fast, and appears to simply need more people with _experience_, I think he would be valuable. Further, he is a pretty quiet guy - you wont find him offering his opinion on every post that moves through here :-) Thoughts? Mark.

2 1