Mailman 3 May 2000 - Python-Dev

issues with int/long on 64bit platforms - eg stringobject (PR#306)
by Guido van Rossum 05 May '00

05 May '00

The email below is a serious bug report. A quick analysis shows that UserString.count() calls the count() method on a string object, which calls PyArg_ParseTuple() with the format string "O|ii". The 'i' format code truncates integers. It probably should raise an overflow exception instead. But that would still cause the test to fail -- just in a different way (more explicit). Then the string methods should be fixed to use long ints instead -- and then something else would probably break... --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Mon, 24 Apr 2000 19:26:27 -0400 From: mark.favas(a)per.dem.csiro.au To: python-bugs-list(a)python.org cc: bugs-py(a)python.org Subject: [Python-bugs-list] 1.6a2 issues with int/long on 64bit platforms - eg stringobject (PR#306) Full_Name: Mark Favas Version: 1.6a2 CVS of 25 April OS: DEC Alpha, Tru64 Unix 4.0F Submission from: wa107.dialup.csiro.au (130.116.4.107) There seems to be issues (and perhaps lurking cans of worms) on 64-bit platforms where sizeof(long) != sizeof(int). For example, the CVS version of 1.6a2 of 25 April fails the UserString regression test. The tests fail as follows (verbose set to 1): abcabcabc.count(('abc',)) no 'abcabcabc' <method UserString.count of UserString instance at 140217e40> 3 <> 2 abcabcabc.count(('abc', 1)) no 'abcabcabc' <method UserString.count of UserString instance at 140216580> 2 <> 1 abcdefghiabc.find(('abc', 1)) no 'abcdefghiabc' <method UserString.find of UserString instance at 140216730> 9 < > - -1 abcdefghiabc.rfind(('abc',)) no 'abcdefghiabc' <method UserString.rfind of UserString instance at 140216580> 9 <> 0 abcabcabc.rindex(('abc',)) no 'abcabcabc' <method UserString.rindex of UserString instance at 140216dc0> 6 <> 3 abcabcabc.rindex(('abc', 1)) no 'abcabcabc' <method UserString.rindex of UserString instance at 140216730> 6 <> 3 These tests are failing because the calls from the UserString methods to the underlying string methods are setting the default value of the end-of-string parameter to sys.maxint, which is defined as LONG_MAX (9223372036854775807), whereas the string methods in stringobject.c are using ints and expecting them to be no larger than INT_MAX (2147483647). Thus the end-of-string parameter becomes -1 in the default case. The size of an int on my platform is 4, and the size of a long is 8, so the "natural size of a Python integer" should be 8, by my understanding. The obvious fix is to change stringobject.c to use longs, rather than ints, but the problem might be more widespread than that. INT_MAX is used in unicodeobject.c, pypcre.c, _sre.c, stropmodule.c, and ceval.c as well as stringobject.c. Some of these look as though LONG_MAX should have been used (variables compared to INT_MAX are longs, but I am not confident enough to submit patches for them... Mark _______________________________________________ Python-bugs-list maillist - Python-bugs-list(a)python.org http://www.python.org/mailman/listinfo/python-bugs-list ------- End of Forwarded Message

5 8

Re: [I18n-sig] Unicode strings: an alternative
by Florian Weimer 05 May '00

05 May '00

Just van Rossum <just(a)letterror.com> writes: > Good point. All this taken together still means to me that comparisons > between wide and narrow strings should take place at the character level, > which implies that coercion from narrow to wide is done at the character > level, without looking at the encoding. (Which in my book in turn still > implies that as long as we're talking about Unicode, narrow strings are > effectively Latin-1.) Sorry for jumping in, I've only recently discovered this list. :-/ At the moment, most of the computing world is not Latin-1 but Windows-12??. That's why I don't think this is a good idea at all.

1 0

Moving Unicode debate to i18n-sig@python.org
by Guido van Rossum 05 May '00

05 May '00

I've moved all my responses to the Unicode debate to the i18n-sig mailing list, where it belongs. Please don't cross-post any more. If you're interested in this issue but aren't subscribed to the i18n-sig list, please subscribe at http://www.python.org/mailman/listinfo/i18n-sig/. To view the archives, go to http://www.python.org/pipermail/i18n-sig/. See you there! --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [I18n-sig] Unicode strings: an alternative
by Just van Rossum 05 May '00

05 May '00

(Boy, is it quiet here all of a sudden ;-) Sorry for the duplication of stuff, but I'd like to reiterate my points, to separate them from my implementation proposal, as that's just what it is: an implementation detail. These things are important to me: - get rid of the Unicode-ness of wide strings, in order to - make narrow and wide strings as similar as possible - implicit conversion between narrow and wide strings should happen purely on the basis of the character codes; no assumption at all should be made about the encoding, ie. what the character code _means_. - downcasting from wide to narrow may raise OverflowError if there are characters in the wide string that are > 255 - str(s) should always return s if s is a string, whether narrow or wide - file objects need to be responsible for handling wide strings - the above two points should make it possible for - if no encoding is known, Unicode is the default, whether narrow or wide The above points seem to have the following consequences: - the 'u' in \uXXXX notation no longer makes much sense, since it is not neccesary for the character to be a Unicode code point: it's just a 2-byte int. \wXXXX might be an option. - the u"" notation is no longer neccesary: if a string literal contains a character > 255 the string should automatically become a wide string. - narrow strings should also have an encode() method. - the builtin unicode() function might be redundant if: - it is possible to specify a source encoding. I'm not sure if this is best done through an extra argument for encode() or that it should be a new method, eg. transcode(). - s.encode() or s.transcode() are allowed to output a wide string, as in aNarrowString.encode("UCS-2") and s.transcode("Mac-Roman", "UCS-2"). My proposal to extend the "old" string type to be able to contain wide strings is of course largely unrelated to all this. Yet it may provide some additional C compatibility (especially now that silent conversion to utf-8 is out) as well as a workaround for the str()-having-to-return-a-narrow-string bottleneck. Just

6 8

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by ht＠cogsci.ed.ac.uk 05 May '00

05 May '00

Guido van Rossum <guido(a)python.org> writes: <snip/> > My ASCII proposal is a compromise that tries to be fair to both uses > for strings. Introducing byte arrays as a more fundamental type has > been on the wish list for a long time -- I see no way to introduce > this into Python 1.6 without totally botching the release schedule > (June 1st is very close already!). I'd like to be able to move on, > there are other important things still to be added to 1.6 (Vladimir's > malloc patches, Neil's GC, Fredrik's completed sre...). > > For 1.7 (which should happen later this year) I promise I'll reopen > the discussion on byte arrays. I think I hear a moderate consensus developing that the 'ASCII proposal' is a reasonable compromise given the time constraints. But let's not fail to come back to this ASAP -- it _really_ narcs me that every time I load XML into my Python-based editor I'm going to convert large amounts of wide-string data into UTF-8 just so Tk can convert it back to wide-strings in order to display it! ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

5 4

Re: [XML-SIG] Re: Unicode debate
by ht＠cogsci.ed.ac.uk 05 May '00

05 May '00

"Fredrik Lundh" <fredrik(a)pythonware.com> writes: > Guido van Rossum <guido(a)python.org> wrote: > > Thanks -- but that's really Tcl's fault, since the only way to get > > character data *into* Tcl (or out of it) is through the UTF-8 > > encoding. > > from http://dev.scriptics.com/man/tcl8.3/TclLib/StringObj.htm > > Tcl_NewUnicodeObj(Tcl_UniChar* unicode, int numChars) > > Tcl_NewUnicodeObj and Tcl_SetUnicodeObj create a new > object or modify an existing object to hold a copy of the > Unicode string given by unicode and numChars. > > (Tcl_UniChar* is currently the same thing as Py_UNICODE*) > Any way this can be exploited in Tkinter? ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

1 0

Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
by ht＠cogsci.ed.ac.uk 05 May '00

05 May '00

Guido van Rossum <guido(a)python.org> writes: > > I think I hear a moderate consensus developing that the 'ASCII > > proposal' is a reasonable compromise given the time constraints. But > > let's not fail to come back to this ASAP -- it _really_ narcs me that > > every time I load XML into my Python-based editor I'm going to convert > > large amounts of wide-string data into UTF-8 just so Tk can convert it > > back to wide-strings in order to display it! > > Thanks -- but that's really Tcl's fault, since the only way to get > character data *into* Tcl (or out of it) is through the UTF-8 > encoding. > > And is your XML really stored on disk in its 16-bit format? No, I have no idea what encoding it's in, my XML parser supports over a dozen encodings, and quite sensibly always delivers the content, as per the XML REC, as wide-strings. ht -- Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh W3C Fellow 1999--2001, part-time member of W3C Team 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: ht(a)cogsci.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/

1 0

FW: weird bug in test_winreg
by Mark Hammond 04 May '00

04 May '00

Oops - I didnt notice the CC - a copy of what I sent to Guido: -----Original Message----- From: Mark Hammond [mailto:mhammond@skippinet.com.au] Sent: Thursday, 4 May 2000 11:13 AM To: Guido van Rossum Subject: RE: weird bug in test_winreg Hah - I was just thinking about this this myself. If I wasnt waiting 24 hours, I would have beaten you to the test_fork1 patch :-) However, there is something bad going on. If you remove your test_fork1 patch, and run it from regrtest (_not_ stand alone) you will see the children threads die with: File "L:\src\Python-cvs\Lib\test\test_fork1.py", line 30, in f alive[id] = os.getpid() AttributeError: 'None' object has no attribute 'getpid' Note the error - os is None! [The reason is only happens as part of the test is because the children are created before the main thread fails with the attribute error] Similarly, I get spurious: Traceback (most recent call last): File ".\test_thread.py", line 103, in task2 mutex.release() AttributeError: 'None' object has no attribute 'release' (Only rarely, and never when run stand-alone - the test_fork1 exception happens 100% of the time from the test suite) And of course the test_winreg one. test_winreg, I guessed, may be caused by the import lock (but its certainly not obvious how or why!?). However, that doesnt explain the others. I also saw these _before_ I applied the threading patches (and after!) So I think the problem may be a little deeper? Mark.

2 2

Minimal (single-file) Python?
by gvwilson＠nevex.com 04 May '00

04 May '00

Hi. Has anyone ever built, or thought about building, a single-file Python, in which all the "basic" capabilities are included in a single executable (where "basic" means "can do as much as the Bourne shell")? Some of the entries in the Software Carpentry competition would like to be able to bootstrap from as small a starting point as possible. Thanks, Greg p.s. I don't think this is the same problem as moving built-in features of Python into optionally-loaded libraries, as some of the things in the 'sys', 'string', and 'os' modules would have to move in the other direction to ensure Bourne shell equivalence.

1 0

Unicode strings: an alternative
by Just van Rossum 04 May '00

04 May '00

Today I had a relatively simple idea that unites wide strings and narrow strings in a way that is more backward comatible at the C level. It's quite possible this has already been considered and rejected for reasons that are not yet obvious to me, but I'll give it a shot anyway. The main concept is not to provide a new string type but to extend the existing string object like so: - wide strings are stored as if they were narrow strings, simply using two bytes for each Unicode character. - there's a flag that specifies whether the string is narrow or wide. - the ob_size field is the _physical_ length of the data; if the string is wide, len(s) will return ob_size/2, all other string operations will have to do similar things. - there can possibly be an encoding attribute which may specify the used encoding, if known. Admittedly, this is tricky and involves quite a bit of effort to implement, since all string methods need to have narrow/wide switch. To make it worse, it hardly offers anything the current solution doesn't. However, it offers one IMHO _big_ advantage: C code that just passes strings along does not need to change: wide strings can be seen as narrow strings without any loss. This allows for __str__() & str() and friends to work with unicode strings without any change. Any thoughts? Just

4 6