Mailman 3 November 1999 - Python-Dev

Unicode proposal: %-formatting ?
by M.-A. Lemburg 17 Nov '99

17 Nov '99

I wonder how we could add %-formatting to Unicode strings without duplicating the PyString_Format() logic. First, do we need Unicode object %-formatting at all ? Second, here is an emulation using strings and <default encoding> that should give an idea of one could work with the different encodings: s = '%s %i abcäöü' # a Latin-1 encoded string t = (u,3) # Convert Latin-1 s to a <default encoding> string via Unicode s1 = unicode(s,'latin-1').encode() # The '%s' will now add u in <default encoding> s2 = s1 % t # Finally, convert the <default encoding> encoded string to Unicode u1 = unicode(s2) Note that .encode() defaults to the current setting of <default encoding>. Provided u maps to Latin-1, an alternative would be: u1 = unicode('%s %i abcäöü' % (u.encode('latin-1'),3), 'latin-1') -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 50 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

3 4

RE: [Python-Dev] Internationalization Toolkit
by Tim Peters 17 Nov '99

17 Nov '99

[Andy Robinson] > ... > I presume no one is actually advocating dropping > ordinary Python strings, or the ability to do > rawdata = open('myfile.txt', 'rb').read() > without any transformations? If anyone has advocated either, they've successfully hidden it from me. Anyone?

2 1

mmapfile module
by Andrew M. Kuchling 16 Nov '99

16 Nov '99

Guido van Rossum writes: >Hm, this seems to require a higher-level Python module to hide the >differences. Maybe the Unix version could also use a filename? I >would think that mmap'ed files should always be backed by a file (not >by a pipe, socket etc.). Or is there an issue with secure creation of >temp files? This is a question for a separate thread. Hmm... I don't know of any way to use mmap() on non-file things, either; there are odd special cases, like using MAP_ANONYMOUS on /dev/zero to allocate memory, but that's still using a file. On the other hand, there may be some special case where you need to do that. We could add a fileno() method to get the file descriptor, but I don't know if that's useful to Windows. (Is Sam Rushing, the original author of the Win32 mmapfile, on this list?) What do we do about the tagname, which is a Win32 argument that has no Unix counterpart -- I'm not even sure what its function is. -- A.M. Kuchling http://starship.python.net/crew/amk/ I had it in me to be the Pierce Brosnan of my generation. -- Vincent Me's past career plans in EGYPT #1

3 2

Re: [Python-Dev] Some thoughts on the codecs...
by Andy Robinson 16 Nov '99

16 Nov '99

--- "M.-A. Lemburg" <mal(a)lemburg.com> wrote: > So I can drop JIS ? [I won't be able to drop the > escaped unicode > codec because this is needed for u"" and ur"".] Drop Japanese from the core language. JIS0208 is a big character set with three popular encodings (Shift-JIS, EUC-JP and JIS), and a host of slight variations; it has 6879 characters, and there are a range of options a user might need to set for it to be useful. So let's assume for now this a separate package. There's a good chance I'll do it but it is not a small job. If you start statically linking in tables of 7000 characters for one Asian language, you'll have to do the lot. As for the single-byte Latin ones, a prototype Python module could be whipped up in a couple of evenings, and a tiny C function which does single-byte to double-byte mappings and vice versa could make it fast. We can have an extensible, data driven solution in no time without having to build it into the core. The way I see it, to claim that python has i18n, a serious effort is needed to ensure every major encoding in the world is available to Python users. But that's separate to the core languages. Your spec should only cover what is going to be hard-coded into Python. I'd like to see one paragraph in your spec stating that our architecture seperates the encodings themselves from the core language changes, and that getting them sorted is a logically separate (but important) project. Ideally, we could put together a separate proposal for the encoding library itself and run it by some world class experts in that field, but after yours is done. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

2 1

Re: [Python-Dev] Internationalization Toolkit
by Da Silva, Mike 16 Nov '99

16 Nov '99

Fredrik Lundh wrote: > 5. UTF-16 requires string operations that do not make assumptions about > nulls - this means re-implementing most of the C runtime functions to work > with unsigned shorts. footnote: the mad scientist has been there and done that: http://www.pythonware.com/madscientist/ <http://www.pythonware.com/madscientist/> (and you can replace "unsigned short" with "whatever's suitable on this platform") Surely using a different type on different platforms means that we throw away the concept of a platform independent Unicode string? I.e. on Solaris, wchar_t is 32 bits, on Windows it is 16 bits. Does this mean that to transfer a file between a Windows box and Solaris, an implicit conversion has to be done to go from 16 bits to 32 bits (and vice versa)? What about byte ordering issues? Or do you mean whatever 16 bit data type is available on the platform, with a standard (platform independent) byte ordering maintained? Mike da S

7 12

RE: [Python-Dev] Internationalization Toolkit
by Da Silva, Mike 16 Nov '99

16 Nov '99

Most of the ASCII string functions do indeed work for UTF-8. I have made extensive use of this feature when writing translation logic to harmonize ASCII text (an SQL statement) with substitution parameters that must be converted from IBM EBCDIC code pages (5035, 1027) into UTF8. Since UTF-8 is a superset of ASCII, this all works fine. Some of the character classification functions etc can be flaky when used with UTF8 characters outside the ASCII range, but simple string operations work fine. As I see it, the relative pros and cons of UTF-8 versus UTF-16 for use as an internal string representation are: 1. UTF-8 allows all characters to be displayed (in some form or other) on the users machine, with or without native fonts installed. Naturally anything outside the ASCII range will be garbage, but it is an immense debugging aid when working with character encodings to be able to touch and feel something recognizable. Trying to decode a block of raw UTF-16 is a pain. 2. UTF-8 works with most existing string manipulation libraries quite happily. It is also portable (a char is always 8 bits, regardless of platform; wchar_t varies between 16 and 32 bits depending on the underlying operating system (although unsigned short does seems to work across platforms, in my experience). 3. UTF-16 has some advantages in providing fixed width characters and, (ignoring surrogate pairs etc) a modeless encoding space. This is an advantage for fast string operations, especially on CPU's that have efficient operations for handling 16bit data. 4. UTF-16 would directly support a tightly coupled character properties engine, which would enable Unicode compliant case folding and character decomposition to be performed without an intermediate UTF-8 <----> UTF-16 translation step. 5. UTF-16 requires string operations that do not make assumptions about nulls - this means re-implementing most of the C runtime functions to work with unsigned shorts. Regards, Mike da Silva -----Original Message----- From: Greg Stein [SMTP:gstein@lyra.org] Sent: 12 November 1999 10:30 To: Tim Peters Cc: python-dev(a)python.org Subject: RE: [Python-Dev] Internationalization Toolkit On Fri, 12 Nov 1999, Tim Peters wrote: >... > Using UTF-8 internally is also reasonable, and if it's being rejected on the > grounds of supposed slowness No... my main point was interaction with the underlying OS. I made a SWAG (Scientific Wild Ass Guess :-) and stated that UTF-8 is probably slower for various types of operations. As always, your infernal meddling has dashed that hypothesis, so I must retreat... >... > I expect either would work well. It's at least curious that Perl and Tcl > both went with UTF-8 -- does anyone think they know *why*? I don't. The > people here saying UCS-2 is the obviously better choice are all from the > Microsoft camp <wink>. It's not obvious to me, but then neither do I claim > that UTF-8 is obviously better. Probably for the exact reason that you stated in your messages: many 8-bit (7-bit?) functions continue to work quite well when given a UTF-8-encoded string. i.e. they didn't have to rewrite the entire Perl/TCL interpreter to deal with a new string type. I'd guess it is a helluva lot easier for us to add a Python Type than for Perl or TCL to whack around with new string types (since they use strings so heavily). Cheers, -g -- Greg Stein, http://www.lyra.org/ _______________________________________________ Python-Dev maillist - Python-Dev(a)python.org http://www.python.org/mailman/listinfo/python-dev

4 4

PyErr_Format security note
by A.M. Kuchling 15 Nov '99

15 Nov '99

I noticed this in PyErr_Format(exception, format, va_alist): char buffer[500]; /* Caller is responsible for limiting the format */ ... vsprintf(buffer, format, vargs); Making the caller responsible for this is error-prone. The danger, of course, is a buffer overflow caused by generating an error string that's larger than the buffer, possibly letting people execute arbitrary code. We could add a test to the configure script for vsnprintf() and use it when possible, but that only fixes the problem on platforms which have it. Can we find an implementation of vsnprintf() someplace? -- A.M. Kuchling http://starship.python.net/crew/amk/ One form to rule them all, one form to find them, one form to bring them all and in the darkness rewrite the hell out of them. -- Digital Equipment Corporation, in a comment from SENDMAIL Ruleset 3

8 13

Some thoughts on the codecs...
by Da Silva, Mike 15 Nov '99

15 Nov '99

Andy Robinson wrote: 1. Stream interface At the moment a codec has dump and load methods which read a (slice of a) stream into a string in memory and vice versa. As the proposal notes, this could lead to errors if you take a slice out of a stream. This is not just due to character truncation; some Asian encodings are modal and have shift-in and shift-out sequences as they move from Western single-byte characters to double-byte ones. It also seems a bit pointless to me as the source (or target) is still a Unicode string in memory. This is a real problem - a filter to convert big files between two encodings should be possible without knowledge of the particular encoding, as should one on the input/output of some server. We can still give a default implementation for single-byte encodings. What's a good API for real stream conversion? just Codec.encodeStream(infile, outfile) ? or is it more useful to feed the codec with data a chunk at a time? A user defined chunking factor (suitably defaulted) would be useful for processing large files. 2. Data driven codecs I really like codecs being objects, and believe we could build support for a lot more encodings, a lot sooner than is otherwise possible, by making them data driven rather making each one compiled C code with static mapping tables. What do people think about the approach below? First of all, the ISO8859-1 series are straight mappings to Unicode code points. So one Python script could parse these files and build the mapping table, and a very small data file could hold these encodings. A compiled helper function analogous to string.translate() could deal with most of them. Secondly, the double-byte ones involve a mixture of algorithms and data. The worst cases I know are modal encodings which need a single-byte lookup table, a double-byte lookup table, and have some very simple rules about escape sequences in between them. A simple state machine could still handle these (and the single-byte mappings above become extra-simple special cases); I could imagine feeding it a totally data-driven set of rules. Third, we can massively compress the mapping tables using a notation which just lists contiguous ranges; and very often there are relationships between encodings. For example, "cpXYZ is just like cpXYY but with an extra 'smiley' at 0XFE32". In these cases, a script can build a family of related codecs in an auditable manner. The problem here is that we need to decide whether we are Unicode-centric, or whether Unicode is just another supported encoding. If we are Unicode-centric, then all code-page translations will require static mapping tables between the appropriate Unicode character and the relevant code points in the other encoding. This would involve (worst case) 64k static tables for each supported encoding. Unfortunately this also precludes the use of algorithmic conversions and or sparse conversion tables because most of these transformations are relative to a source and target non-Unicode encoding, eg JIS <---->EUCJIS. If we are taking the IBM approach (see CDRA), then we can mix and match approaches, and treat Unicode strings as just Unicode, and normal strings as being any arbitrary MBCS encoding. To guarantee the utmost interoperability and Unicode 3.0 (and beyond) compliance, we should probably assume that all core encodings are relative to Unicode as the pivot encoding. This should hopefully avoid any gotcha's with roundtrips between any two arbitrary native encodings. The downside is this will probably be slower than an optimised algorithmic transformation. 3. What encodings to distribute? The only clean answers to this are 'almost none', or 'everything that Unicode 3.0 has a mapping for'. The latter is going to add some weight to the distribution. What are people's feelings? Do we ship any at all apart from the Unicode ones? Should new encodings be downloadable from www.python.org <http://www.python.org> ? Should there be an optional package outside the main distribution? Ship with Unicode encodings in the core, the rest should be an add on package. If we are truly Unicode-centric, this gives us the most value in terms of accessing a Unicode character properties database, which will provide language neutral case folding, Hankaku <----> Zenkaku folding (Japan specific), and composition / normalisation between composed characters and their component nonspacing characters. Regards, Mike da Silva

2 1

Re: [Python-Dev] just say no...
by Andy Robinson 15 Nov '99

15 Nov '99

--- Guido van Rossum <guido(a)CNRI.Reston.VA.US> wrote: > Did you read Andy Robinson's case study? He > suggested that for certain encodings there may be > other things you can do that are more > user-friendly than raising an exception, depending > on the application. I am proposing to leave this a > detail of each specific translation. > There may even be translations that do the same thing > except they have a different behavior for > untranslatable cases -- e.g. a strict version > that raises an exception and a non-strict version > that replaces bad characters with '?'. I think this > is one of the powers of having an extensible set of > encodings. This would be a desirable option in almost every case. Default is an exception (I want to know my data is not clean), but an option to specify an error character. It is usually a question mark but Mike tells me that some encodings specify the error character to use. Example - I query a Sybase Unicode database containing European accents or Japanese. By default it will give me question marks. If I issue the command 'set char_convert utf8', then I see the lot (as garbage, but never mind). If it always errored whenever a query result contained unexpected data, it would be almost impossible to maintain the database. If I wrote my own codec class for a family of encodings, I'd give it an even wider variety of error-logging options - maybe a mode where it told me where in the file the dodgy characters were. We've already taken the key step by allowing codecs to be separate objects registered at run-time, implemented in either C or Python. This means that once again Python will have the most flexible solution around. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

1 0

Re: [Python-Dev] just say no...
by Jack Jansen 13 Nov '99

13 Nov '99

Recently, Greg Stein <gstein(a)lyra.org> said: > This was done last year!! We have "s#" meaning "give me some bytes." We > have "t#" meaning "give me some 8-bit characters." The Python distribution > has been completely updated to use the appropriate format in each call. Oops... I remember the discussion but I wasn't aware that somone had actually _implemented_ this:-). Part of my misunderstanding was also caused by the fact that I inspected what I thought would be the prime candidate for t#: file.write() to a non-binary file, and it doesn't use the new format. I also noted a few inconsistencies at first glance, by the way: most modules seem to use s# for things like filenames and other data-that-is-readable-but-shouldn't-be-messed-with, but binascii is an exception and it uses t# for uuencoded strings... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1