Mailman 3 April 2000 - Python-Dev

Re: Python 1.6a2 Unicode bug (was Re: comparing strings and ints)
by Just van Rossum 09 May '00

09 May '00

I wrote: >A utf-8-encoded 8-bit string in Python is *not* a string, but a "ByteArray". Another way of putting this is: - utf-8 in an 8-bit string is to a unicode string what a pickle is to an object. - defaulting to utf-8 upon coercing is like implicitly trying to unpickle an 8-bit string when comparing it to an instance. Bad idea. Defaulting to Latin-1 is the only logical choice, no matter how western-culture-centric this may seem. Just

16 66

issues with int/long on 64bit platforms - eg stringobject (PR#306)
by Guido van Rossum 06 May '00

06 May '00

The email below is a serious bug report. A quick analysis shows that UserString.count() calls the count() method on a string object, which calls PyArg_ParseTuple() with the format string "O|ii". The 'i' format code truncates integers. It probably should raise an overflow exception instead. But that would still cause the test to fail -- just in a different way (more explicit). Then the string methods should be fixed to use long ints instead -- and then something else would probably break... --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Mon, 24 Apr 2000 19:26:27 -0400 From: mark.favas(a)per.dem.csiro.au To: python-bugs-list(a)python.org cc: bugs-py(a)python.org Subject: [Python-bugs-list] 1.6a2 issues with int/long on 64bit platforms - eg stringobject (PR#306) Full_Name: Mark Favas Version: 1.6a2 CVS of 25 April OS: DEC Alpha, Tru64 Unix 4.0F Submission from: wa107.dialup.csiro.au (130.116.4.107) There seems to be issues (and perhaps lurking cans of worms) on 64-bit platforms where sizeof(long) != sizeof(int). For example, the CVS version of 1.6a2 of 25 April fails the UserString regression test. The tests fail as follows (verbose set to 1): abcabcabc.count(('abc',)) no 'abcabcabc' <method UserString.count of UserString instance at 140217e40> 3 <> 2 abcabcabc.count(('abc', 1)) no 'abcabcabc' <method UserString.count of UserString instance at 140216580> 2 <> 1 abcdefghiabc.find(('abc', 1)) no 'abcdefghiabc' <method UserString.find of UserString instance at 140216730> 9 < > - -1 abcdefghiabc.rfind(('abc',)) no 'abcdefghiabc' <method UserString.rfind of UserString instance at 140216580> 9 <> 0 abcabcabc.rindex(('abc',)) no 'abcabcabc' <method UserString.rindex of UserString instance at 140216dc0> 6 <> 3 abcabcabc.rindex(('abc', 1)) no 'abcabcabc' <method UserString.rindex of UserString instance at 140216730> 6 <> 3 These tests are failing because the calls from the UserString methods to the underlying string methods are setting the default value of the end-of-string parameter to sys.maxint, which is defined as LONG_MAX (9223372036854775807), whereas the string methods in stringobject.c are using ints and expecting them to be no larger than INT_MAX (2147483647). Thus the end-of-string parameter becomes -1 in the default case. The size of an int on my platform is 4, and the size of a long is 8, so the "natural size of a Python integer" should be 8, by my understanding. The obvious fix is to change stringobject.c to use longs, rather than ints, but the problem might be more widespread than that. INT_MAX is used in unicodeobject.c, pypcre.c, _sre.c, stropmodule.c, and ceval.c as well as stringobject.c. Some of these look as though LONG_MAX should have been used (variables compared to INT_MAX are longs, but I am not confident enough to submit patches for them... Mark _______________________________________________ Python-bugs-list maillist - Python-bugs-list(a)python.org http://www.python.org/mailman/listinfo/python-bugs-list ------- End of Forwarded Message

5 8

fun with unicode, part 1
by Fredrik Lundh 02 May '00

02 May '00

>>> filename = u"gröt" >>> file = open(filename, "w") >>> file.close() >>> import glob >>> print glob.glob("gr*") ['gr\303\266t'] >>> print glob.glob(u"gr*") [u'gr\366t'] >>> import os >>> os.system("dir gr*") ... GRÇôT 0 01-02-03 12.34 grÇôt 1 fil(es) 0 byte 0 dir 12 345 678 byte free hmm. </F>

9 12

At the interactive port
by Moshe Zadka 02 May '00

02 May '00

Continuing the recent debate about what is appropriate to the interactive prompt printing, and the wide agreement that whatever we decide, users might think otherwise, I've written up a patch to have the user control via a function in __builtin__ the way things are printed at the prompt. This is not patches@python level stuff for two reasons: 1. I'm not sure what to call this function. Currently, I call it __print_expr__, but I'm not sure it's a good name 2. I haven't yet supplied a default in __builtin__, so the user *must* override this. This is unacceptable, of course. I'd just like people to tell me if they think this is worth while, and if there is anything I missed. *** ../python/dist/src/Python/ceval.c Fri Mar 31 04:42:47 2000 --- Python/ceval.c Sat Apr 29 03:55:36 2000 *************** *** 1014,1047 **** case PRINT_EXPR: v = POP(); ! /* Print value except if None */ ! /* After printing, also assign to '_' */ ! /* Before, set '_' to None to avoid recursion */ ! if (v != Py_None && ! (err = PyDict_SetItemString( ! f->f_builtins, "_", Py_None)) == 0) { ! err = Py_FlushLine(); ! if (err == 0) { ! x = PySys_GetObject("stdout"); ! if (x == NULL) { ! PyErr_SetString( ! PyExc_RuntimeError, ! "lost sys.stdout"); ! err = -1; ! } ! } ! if (err == 0) ! err = PyFile_WriteObject(v, x, 0); ! if (err == 0) { ! PyFile_SoftSpace(x, 1); ! err = Py_FlushLine(); ! } ! if (err == 0) { ! err = PyDict_SetItemString( ! f->f_builtins, "_", v); ! } } ! Py_DECREF(v); break; case PRINT_ITEM: --- 1014,1035 ---- case PRINT_EXPR: v = POP(); ! x = PyDict_GetItemString(f->f_builtins, ! "__print_expr__"); ! if (x == NULL) { ! PyErr_SetString(PyExc_SystemError, ! "__print_expr__ not found"); ! Py_DECREF(v); ! break; ! } ! t = PyTuple_New(1); ! if (t != NULL) { ! PyTuple_SET_ITEM(t, 0, v); ! w = PyEval_CallObject(x, t); ! Py_XDECREF(w); } ! /*Py_DECREF(x);*/ ! Py_XDECREF(t); break; case PRINT_ITEM: -- Moshe Zadka <mzadka(a)geocities.com>. http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com

4 7

Changing PYC-Magic
by M.-A. Lemburg 29 Apr '00

29 Apr '00

I have just looked at the Python/import.c file and the hard coded PYC magic number... /* Magic word to reject .pyc files generated by other Python versions */ /* Change for each incompatible change */ /* The value of CR and LF is incorporated so if you ever read or write a .pyc file in text mode the magic number will be wrong; also, the Apple MPW compiler swaps their values, botching string constants */ /* XXX Perhaps the magic number should be frozen and a version field added to the .pyc file header? */ /* New way to come up with the magic number: (YEAR-1995), MONTH, DAY */ #define MAGIC (20121 | ((long)'\r'<<16) | ((long)'\n'<<24)) A bit outdated, I'd say. With the addition of Unicode, the PYC files will contain marshalled Unicode objects which are not readable by older versions. I'd suggest bumping the magic number to 50428 ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

1 0

Brian Hooper's patch to add u & u# to Py_BuildValue
by Fred L. Drake, Jr. 28 Apr '00

28 Apr '00

Brian Hooper submitted a patch to add U and U# to the format strings for Py_BuildValue(), and there were comments that indicated u and u# would be better. He's submitted a documentation update for this as well the implementation. If there are no objections, I'll incorporate these changes. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives

2 1

Re: [XML-SIG] Python 1.6a2 Unicode experiences?
by Paul Prescod 28 Apr '00

28 Apr '00

Andy Robinson wrote: > > - you can work with old fashioned strings, which are understood > by everyone to be arrays of bytes, and there is no magic > conversion going on. The bytes in literal strings in your script file > are the bytes that end up in the program. Who is "everyone"? Are you saying that CP4E hordes are going to understand that the syntax "abcde" is constructing a *byte array*? It seems like you think that Python users are going to be more sophisticated in their understanding of these issues than Java programmers. In most other things, Python is simpler. > ... > > I'm also convinced that the majority of Python scripts won't need > to work in Unicode. Anything working with XML will need to be Unicode. Anything working with the Win32 API (especially COM) will want to do Unicode. Over time the entire Web infrastructure will move to Unicode. Anything written in JPython pretty much MOST use Unicode (doesn't it?). > Even working with exotic languages, there is always a native > 8-bit encoding. Unicode has many encodings: Shift-JIS, Big-5, EBCDIC ... You can use 8-bit encodings of Unicode if you want. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html

5 4

Re: [Python-Dev] Where the speed is lost! (was: 1.6 speed)
by Jeremy Hylton 28 Apr '00

28 Apr '00

>>>>> "CT" == Christian Tismer <tismer(a)tismer.com> writes: CT> Summary: We had two effects here. Effect 1: Wasting time with CT> extra errors in instance creation. Effect 2: Loss of locality CT> due to code size increase. CT> Solution to 1 is Jeremy's patch. Solution to 2 could be a CT> little renaming of the one or the other module, in order to get CT> the default link order to support locality better. CT> Now everything is clear to me. My first attempts with reordering CT> could not reveal the loss with the instance stuff. CT> All together, Python 1.6 is a bit faster than 1.5.2 if we try to CT> get related code ordered better. I reach a different conclusion. The performance difference 1.5.2 and 1.6, measured with pystone and pybench, is so small that effects like the order in which the compiler assembles the code make a difference. I don't think we should make any non-trivial effort to improve performance based on this kind of voodoo. I also question the claim that the two effects here explain the performance difference between 1.5.2 and 1.6. Rather, they explain the performance difference of pystone and pybench running on different versions of the interpreter. Saying that pystone is the same speed is a far cry from saying that python is the same speed! Remember that performance on a benchmark is just that. (It's like the old joke about a person's IQ: It is a very good indicator of how well they did on the IQ test.) I think we could use better benchmarks of two sorts. The pybench microbenchmarks are quite helpful individually, though the overall number isn't particularly meaningful. However, these benchmarks are sometimes a little too big to be useful. For example, the instance creation effect was tracked down by running this code: class Foo: pass for i in range(big_num): Foo() The pybench test "CreateInstance" does all sorts of other stuff. It tests creation with and without an __init__ method. It tests instance deallocation (because all the created objected need to be dealloced, too). It also tests attribute assignment, since many of the __init__ methods make assignments. What would be better (and I'm not sure what priority should be placed on doing it) is a set of nano-benchmarks that try to limit themselves to a single feature or small set of features. Guido suggested having a hierarchy so that there are multiple nano-benchmarks for instance creation, each identifying a particular effect, and a micro-benchmark that is the aggregate of all these nano-benchmarks. We could also use some better large benchmarks. Using pystone is pretty crude, because it doesn't necessarily measure the performance of things we care about. It would be better to have a collection of 5-10 apps that each do something we care about -- munging text files or XML data, creating lots of objects, etc. For example, I used the compiler package (in nondist/src/Compiler) to compile itself. Based on that benchmark, an interpreter built from the current CVS tree is still 9-11% slower than 1.5. Jeremy

2 1

Python 1.6a2 Unicode bug (was Re: comparing strings and ints)
by andy＠reportlab.python.org 28 Apr '00

28 Apr '00

>Alisa Pasic Robinson Drat! my wife's been hacking my email headers! Sorry... - Andy Robinson

1 0

Encoding of 8-bit strings and Python source code
by M.-A. Lemburg 27 Apr '00

27 Apr '00

After the discussion about #pragmas two weeks ago and some interesting ideas in the direction of source code encodings and ways to implement them, I would like to restart the talk about encodings in source code and runtime auto-conversions. Fredrik recently posted patches to the patches list which loosen the currently hard-coded default encoding used throughout the Unicode design and add a layer of abstraction which would make it easily possible to change the default encoding at some later point. While making things more abstract is certainly a wise thing to do, I am not sure whether this particular case fits into the design decisions made a few months ago. Here's a short summary of what was discussed recently: 1. Fredrik posted the idea of changing the default encoding from UTF-8 to Latin-1 (he calls this 8-bit Unicode which points to the motivation behind this: 8-bit strings should behave like 8-bit Unicode). His recent patches work into this direction. 2. Fredrik also posted an interesting idea which enables writing Python source code in any supported encoding by having the Python tokenizer read Py_UNICODE data instead of char data. A preprocessor would take care of converting the input to Py_UNICODE; the parser would assure that 8-bit string data gets converted back to char data (using e.g. UTF-8 or Latin-1 for the encoding) 3. Regarding the addition of pragmas to allow specifying the used source code encoding several possibilities were mentioned: - addition of a keyword "pragma" to define pragma dictionaries - usage of a "global" as basis for this - adding a new keyword "decl" which also allows defining other things such as type information - XML like syntax embedded into Python comments Some comments: Ad 1. UTF-8 is used as basis in many other languages such as TCL or Perl. It is not an intuitive way of writing strings and causes problems due to one character spanning 1-6 bytes. Still, the world seems to be moving into this direction, so going the same way can't be all wrong... Note that stream IO can be recoded in a way which allows Python to print and read e.g. Latin-1 (see below). The general idea behind the fixed default encoding design was to give all the power to the user, since she eventually knows best which encoding to use or expect. Ad 2. I like this idea because it enables writing Unicode- aware programs *in* Unicode... the only problem which remains is again the encoding to use for the classic 8-bit strings. Ad 3. For 2. to work, the encoding would have to appear close to the top of the file. The preprocessor would have to be BOM-mark aware to tell whether UTF-16 or some ASCII extension is used by the file. Guido asked me for some code which demonstrates Latin-1 recoding using the existing mechanisms. I've attached a simple script to this mail. It is not much tested yet, so please give it a try. You can also change it to use any other encoding you like. Together with the Japanese codecs provided by Tamito Kajiyama (http://pseudo.grad.sccs.chukyo-u.ac.jp/~kajiyama/tmp/japanese-codecs.tar.gz) you should be able to type Shift-JIS at the raw_input() or interactive prompt, have it stored as UTF-8 and then printed back as Shift-JIS, provided you put add a recoder similar to the attached one for Latin-1 to your PYTHONSTARTUP or site.py script. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

7 9