Mailman 3 November 1999 - Python-Dev

Python 1.6 status
by Guido van Rossum Dec. 7, 1999

Dec. 7, 1999

Greg Stein recently reminded me that he was holding off on 1.6 patches because he was under the impression that I wasn't accepting them yet. The situation is rather more complicated than that. There are a great deal of things that need to be done, and for many of them I'd be most happy to receive patches! For other things, however, I'm still in the requirements analysis phase, and patches might be premature (e.g., I want to redesign the import mechanisms, and while I like some of the … [View More]prototypes that have been posted, I'm not ready to commit to any specific implementation). How do you know for which things I'm ready for patches? Ask me. I've tried to make lists before, and there are probably some hints in the TODO FAQ wizard as well as in the "requests" section of the Python Bugs List. Greg also suggested that I might receive more patches if I opened up the CVS tree for checkins by certain valued contributors. On the one hand I'm reluctant to do that (I feel I have a pretty good track record of checking in patches that are mailed to me, assuming I agree with them) but on the other hand there might be something to say for this, because it gives contributors more of a sense of belonging to the inner core. Of course, checkin privileges don't mean you can check in anything you like -- as in the Apache world, changes must be discussed and approved by the group, and I would like to have a veto. However once a change is approved, it's much easier if the contributor can check the code in without having to go through me all the time. A drawback may be that some people will make very forceful requests to be given checkin privileges, only to never use them; just like there are some members of python-dev who have never contributed. I definitely want to limit the number of privileged contributors to a very small number (e.g. 10-15). One additional detail is the legal side -- contributors will have to sign some kind of legal document similar to the current (wetsign.html) release form, but guiding all future contributions. I'll have to discuss this with CNRI's legal team. Greg, I understand you have checkin privileges for Apache. What is the procedure there for handing out those privileges? What is the procedure for using them? (E.g. if you made a bogus change to part of Apache you're not supposed to work on, what happens?) I'm hoping for several kind of responses to this email: - uncontroversial patches - questions about whether specific issues are sufficiently settled to start coding a patch - discussion threads opening up some issues that haven't been settled yet (like the current, very productive, thread in i18n) - posts summarizing issues that were settled long ago in the past, requesting reverification that the issue is still settled - suggestions for new issues that maybe ought to be settled in 1.6 - requests for checkin privileges, preferably with a specific issue or area of expertise for which the requestor will take responsibility --Guido van Rossum (home page: http://www.python.org/~guido/) [View Less]

14 54

Re: [Python-Dev] Import redesign [LONG]
by Greg Stein Dec. 2, 1999

Dec. 2, 1999

[ taking the liberty to CC: this back to python-dev ] On Fri, 19 Nov 1999, David Ascher wrote: > > > (2) a file in a directory that's on sys.path can be a zip/jar file; > > > its contents will be considered as a package (note that this is > > > different from (1)!) > > > > No problem. This will slow things down, as a stat() for *.zip and/or *.jar > > must be done, in addition to *.py, *.pyc, and *.pyo. > > Aside: it strikes me … [View More]

5 9

updated imputil
by Greg Stein Dec. 2, 1999

Dec. 2, 1999

I've updated imputil... The main changes is that I added SysPathImporter and BuiltinImporter. I also did some restructing to help with bootstrapping the module (remove dependence on os.py). For testing a revamped Python import system, you can importing the thing and call imputil._test_revamp() to set it up. This will load normal, builtin, and frozen modules via imputil. Dynamic modules are still handled by Python, however. I ran a timing comparisons of importing all modules in /usr/lib/… [View More]

8 23

RE: [Python-Dev] Another 1.6 wish
by Asbahr, Jason Dec. 1, 1999

Dec. 1, 1999

My first Python-Dev post. :-) >We had some discussion a while back about enabling thread support by >default, if the underlying OS supports it obviously. What's the consensus about Python microthreads -- a likely candidate for incorporation in 1.6 (or later)? Also, we have a couple minor convenience functions for Python in an MSDEV environment, an exposure of OutputDebugString for writing to the DevStudio log window and a means of tripping DevStudio C/C++ layer breakpoints from … [View More]

5 7

CFP Developers' Day - 8th International Python Conference
by bwarsaw＠python.org Nov. 30, 1999

Nov. 30, 1999

Hello Python Developers! Thursday January 27 2000, the final day of the 8th International Python Conference is Developers' Day, where Python hackers get together to discuss and reach agreements on the outstanding issues facing Python. This is also your once-a-year chance for face-to-face interactions with Python's creator Guido van Rossum and other experienced Python developers. To make Developers' Day a success, we need you! We're looking for a few good champions to lead topic sessions. … [View More]

1 0

Import Design
by M.-A. Lemburg Nov. 24, 1999

Nov. 24, 1999

Before hooking on to some more PathBuiltinImporters ;-), I'd like to spawn a thread leading in a different direction... There has been some discussion on what we really expect of the import mechanism to be able to do. Here's a summary of what I think we need: * compatibility with the existing import mechanism * imports from library archives (e.g. .pyl or .par-files) * a modified intra package import lookup scheme (the thingy which I call "walk-me-up-Scotty" patch -- see previous posts) … [View More]

2 1

RE: [Python-Dev] Unicode Proposal: Version 0.8
by Andy Robinson Nov. 24, 1999

Nov. 24, 1999

--- Mark Hammond <mhammond(a)skippinet.com.au> wrote: > > Pretty quiet around here lately... > > My guess is that most positions and opinions have > been covered. It is > now probably time for less talk, and more code! > > It is time to start an implementation plan? Do we > start with /F's > Unicode implementation (which /G *smirk* seemed to > approve of)? Who > does what? When can we start to play with it? > > And a key point that seems to … [View More]

1 0

Unicode Proposal: Version 0.4
by M.-A. Lemburg Nov. 24, 1999

Nov. 24, 1999

I've uploaded a new version of the proposal which incorporates a lot of what has been discussed on the list. Thanks to everybody who helped so far. Note that I have extended the list of references for those who want to join in, but are in need of more background information. The latest version of the proposal is available at: http://starship.skyport.net/~lemburg/unicode-proposal.txt Older versions are available as: http://starship.skyport.net/~lemburg/unicode-proposal-X.X.txt Some POD (… [View More]

3 8

breaking out code for dynamic loading
by Greg Stein Nov. 24, 1999

Nov. 24, 1999

Guido, I can't find the message, but it seems that at some point you mentioned wanting to break out importdl.c into separate files. The configure process could then select the appropriate one to use for the platform. Sounded great until I looked at importdl.c. There are a 13 variants of dynamic loading. That would imply 13 separate files/modules. I'd be happy to break these out, but are you actually interested in that many resulting modules? If so, then any suggestions for naming? (e.g. … [View More]

1 0

New thread ideas in Perl-land
by Jean-Claude Wippler Nov. 23, 1999

Nov. 23, 1999

Just got a note about a paper on a new way of dealing with threads, as presented to the Perl-Porters list. The idea is described in: http://www.cpan.org/modules/by-authors/id/G/GB/GBARTELS/thread_0001.txt I have no time to dive in, comment, or even judge the relevance of this, but perhaps someone else on this list wishes to check it out. The author of this is Greg London <bartels(a)pixelmagic.com>. -- Jean-Claude

1 0

Another 1.6 wish
by Barry A. Warsaw Nov. 19, 1999

Nov. 19, 1999

We had some discussion a while back about enabling thread support by default, if the underlying OS supports it obviously. I'd like to see that happen for 1.6. IIRC, this shouldn't be too hard -- just a few tweaks of the configure script (and who knows what for those minority platforms that don't use configure :). -Barry

4 3

Re: [Python-Dev] Codecs and StreamCodecs
by Fredrik Lundh Nov. 19, 1999

Nov. 19, 1999

Andy Robinson <captainrobbo(a)yahoo.com> wrote: > So I would take flush() to mean 'shift back to > ASCII now'. if we're still talking about my "just one codec, please" proposal, that's exactly what encoder.flush should do. while decoder.flush should raise an ex- ception if you're still in double byte mode (at least if running in 'strict' mode). > Calling flush and reopen would thus "almost" get the > same data across. > > I'm trying to think if it would be dangerous. … [View More]

1 0

Re: [Python-Dev] Codecs and StreamCodecs
by Andy Robinson Nov. 19, 1999

Nov. 19, 1999

--- "M.-A. Lemburg" <mal(a)lemburg.com> wrote: > Guido van Rossum wrote: > > I don't know where this came from, but a flush() > should work like > > flush() on a file. > > It came from Fredrik's proposal. > > > It doesn't return a value, it just sends any > > remaining data to the underlying stream (for > output). For input it > > shouldn't be supported at all. > > > > The idea is that flush() should do the same to the > … [View More]

1 0

Codecs and StreamCodecs
by M.-A. Lemburg Nov. 19, 1999

Nov. 19, 1999

Here is a new proposal for the codec interface: class Codec: def encode(self,u,slice=None): """ Return the Unicode object u encoded as Python string. If slice is given (as slice object), only the sliced part of the Unicode object is encoded. The method may not store state in the Codec instance. Use SteamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ ... def decode(self,s,slice=None): """ Return an … [View More]equivalent Unicode object for the encoded Python string s. If slice is given (as slice object), only the sliced part of the Python string is decoded and returned as Unicode object. Note that this can cause the decoding algorithm to fail due to truncations in the encoding. The method may not store state in the Codec instance. Use SteamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ ... class StreamCodec(Codec): def __init__(self,stream=None,errors='strict'): """ Creates a StreamCodec instance. stream must be a file-like object open for reading and/or writing binary data depending on the intended codec action or None. The StreamCodec may implement different error handling schemes by providing the errors argument. These parameters are known (they need not all be supported by StreamCodec subclasses): 'strict' - raise an UnicodeError (or a subclass) 'ignore' - ignore the character and continue with the next (a single character) - replace errorneous characters with the given character (may also be a Unicode character) """ self.stream = stream def write(self,u,slice=None): """ Writes the Unicode object's contents encoded to self.stream. stream must be a file-like object open for writing binary data. If slice is given (as slice object), only the sliced part of the Unicode object is written. """ ... the base class should provide a default implementation of this method using self.encode ... def read(self,length=None): """ Reads an encoded string from the stream and returns an equivalent Unicode object. If length is given, only length Unicode characters are returned (the StreamCodec instance reads as many raw bytes as needed to fulfill this requirement). Otherwise, all available data is read and decoded. """ ... the base class should provide a default implementation of this method using self.decode ... It is not required by the unicodec.register() API to provide a subclass of these base class, only the given methods must be present; this allows writing Codecs as extensions types. All Codecs must provide the .encode()/.decode() methods. Codecs having the .read() and/or .write() methods are considered to be StreamCodecs. The Unicode implementation will by itself only use the stateless .encode() and .decode() methods. All other conversion have to be done by explicitly instantiating the appropriate [Stream]Codec. -- Feel free to beat on this one ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 45 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ [View Less]

7 27

Internationalization Toolkit
by Andy Robinson Nov. 19, 1999

Nov. 19, 1999

Here are the features I'd like to see in a Python Internationalisation Toolkit. I'm very open to persuasion about APIs and how to do it, but this is roughly the functionality I would have wanted for the last year (see separate post "Internationalization Case Study"): Built-in types: --------------- "Unicode String" and "Normal String". The normal string is can hold all 256 possible byte values and is analogous to java's Byte Array - in other words an ordinary Python string. Unicode … [View More]strings iterate (and are manipulated) per character, not per byte. You knew that already. To manipulate anything in a funny encoding, you convert it to Unicode, manipulate it there, then convert it back. Easy Conversions ---------------------- This is modelled on Java which I think has it right. When you construct a Unicode string, you may supply an optional encoding argument. I'm not bothered if conversion happens in a global function, a constructor method or whatever. MyUniString = ToUnicode('hello') # assumes ASCII MyUniString = ToUnicode('pretend this is Japanese', 'ShiftJIS') #specified The converse applies when converting back. The encoding designators should agree with Java. If data is encountered which is not valid for the encoding, there are several strategies, and it would be nice if they could be specified explicitly: 1. replace offending characters with a question mark 2. try to recover intelligently (possible in some cases) 3. raise an exception A 'Unicode' designator is needed which performs a dummy conversion. File Opening: --------------- It should be possible to work with files as we do now - just streams of binary data. It should also be possible to read, say, a file of locally endoded addresses into a Unicode string. e.g. open(myfile, 'r', 'ShiftJIS'). It should also be possible to open a raw Unicode file and read the bytes into ordinary Python strings, or Unicode strings. In this case one needs to watch out for the byte-order marks at the beginning of the file. Not sure of a good API to do this. We could have OrdinaryFile objects and UnicodeFile objects, or proliferate the arguments to 'open. Doing the Conversions ---------------------------- All conversions should go through Unicode as the central point. Here is where we can start to define the territory. Some conversions are algorithmic, some are lookups, many are a mixture with some simple state transitions (e.g. shift characters to denote switches from double-byte to single-byte). I'd like to see an 'encoding engine' modelled on something like mxTextTools - a state machine with a few simple actions, effectively a mini-language for doing simple operations. Then a new encoding can be added in a data-driven way, and still go at C-like speeds. Making this open and extensible (and preferably not needing to code C to do it) is the only way I can see to get a really good solid encodings library. Not all encodings need go in the standard distribution, but all should be downloadable from www.python.org. A generalized two-byte-to-two-byte mapping is 128kb. But there are compact forms which can reduce these to a few kb, and also make the data intelligible. It is obviously desirable to store stuff compactly if we can unpack it fast. Typed Strings ---------------- When you are writing data conversion tools to sit in the middle of a bunch of databases, you could save a lot of grief with a string that knows its encoding. What follows could be done as a Python wrapper around something ordinary strings rather than as a new type, and thus need not be part of the language. This is analogous to Martin Fowler's Quantity pattern in Analysis Patterns, where a number knows its units and you cannot add dollars and pounds accidentally. These would do implicit conversions; and they would stop you assigning or confusing differently encoded strings. They would also validate when constructed. 'Typecasting' would be allowed but would require explicit code. So maybe something like... >>>ts1 = TypedString('hello', 'cp932ms') # specify encoding, it remembers it >>>ts2 = TypedString('goodbye','cp5035') >>>ts1 + ts2 #or any of a host of other encoding options EncodingError >>>ts3 = TypedString(ts1, 'cp5035') #converts it implicitly going via Unicode >>>ts4 = ts1.cast('ShiftJIS') #the developer knows that in this case the string is compatible. Going Deeper ---------------- The project I describe involved many more issues than just a straight conversion. I envisage an encodings package or module which power users could get at directly. We have be able to answer the questions: 'is string X a valid instance of encoding Y?' 'is string X nearly a valid instance of encoding Y, maybe with a little corruption, or is it something totally different?' - this one might be a task left to a programmer, but the toolkit should help where it can. 'can string X be converted from encoding Y to encoding Z without loss of data? If not, exactly what will get trashed' ? This is a really useful utility. More generally, I want tools to reason about character sets and encodings. I have 'Character Set' and 'Character Mapping' classes - very app-specific and proprietary - which let me express and answer questions about whether one character set is a superset of another and reason about round trips. I'd like to do these properly for the toolkit. They would need some C support for speed, but I think they could still be data driven. So we could have an Endoding object which could be pickled, and we could keep a directory full of them as our database. There might actually be two encoding objects - one for single-byte, one for multi-byte, with the same API. There are so many subtle differences between encodings (even within the Shift-JIS family) - company X has ten extra characters, and that is technically a new encoding. So it would be really useful to reason about these and say 'find me all JIS-compatible encodings', or 'report on the differences between Shift-JIS and 'cp932ms'. GUI Issues ------------- The new Pythonwin breaks somewhat on Japanese - editor windows are fine but console output is show as single-byte garbage. I will try to evaluate IDLE on a Japanese test box this week. I think these two need to work for double-byte languages for our credibility. Verifiability and printing ----------------------------- We will need to prove it all works. This means looking at text on a screen or on paper. A really wicked demo utility would be a GUI which could open files and convert encodings in an editor window or spreadsheet window, and specify conversions on copy/paste. If it could save a page as HTML (just an encoding tag and data between <PRE> tags, then we could use Netscape/IE for verification. Better still, a web server demo could convert on python.org and tag the pages appropriately - browsers support most common encodings. All the encoding stuff is ultimately a bit meaningless without a way to display a character. I am hoping that PDF and PDFgen may add a lot of value here. Adobe (and Ken Lunde) have spent years coming up with a general architecture for this stuff in PDF. Basically, the multi-byte fonts they use are encoding independent, and come with a whole bunch of mapping tables. So I can ask for the same Japanese font in any of about ten encodings - font name is a combination of face name and encoding. The font itself does the remapping. They make available downloadable font packs for Acrobat 4.0 for most languages now; these are good places to raid for building encoding databases. It also means that I can write a Python script to crank out beautiful-looking code page charts for all of our encodings from the database, and input and output to regression tests. I've done it for Shift-JIS at Fidelity, and would have to rewrite it once I am out of here. But I think that some good graphic design here would lead to a product that blows people away - an encodings library that can print out its own contents for viewing and thus help demonstrate its own correctness (or make errors stick out like a sore thumb). Am I mad? Have I put you off forever? What I outline above would be a serious project needing months of work; I'd be really happy to take a role, if we could find sponsors for the project. But I believe we could define the standard for years to come. Furthermore, it would go a long way to making Python the corporate choice for data cleaning and transformation - territory I think we should own. Regards, Andy Robinson Robinson Analytics Ltd. ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com [View Less]

15 122

Re: [Python-Dev] Some thoughts on the codecs...
by Jack Jansen Nov. 18, 1999

Nov. 18, 1999

> I would propose to only add some very basic encodings to > the standard distribution, e.g. the ones mentioned under > Standard Codecs in the proposal: > > 'utf-8': 8-bit variable length encoding > 'utf-16': 16-bit variable length encoding (litte/big endian) > 'utf-16-le': utf-16 but explicitly little endian > 'utf-16-be': utf-16 but explicitly big endian > 'ascii': 7-bit ASCII codepage > 'latin-1': Latin-1 codepage > 'html-entities': Latin-… [View More]

7 10

RE: [Python-Dev] Some thoughts on the codecs...
by Andy Robinson Nov. 17, 1999

Nov. 17, 1999

--- Mark Hammond <mhammond(a)skippinet.com.au> wrote: > Actually, I was thinking even more radically - drop > the codec registry > all together, and use modules with "well-known" > names (a slight > precedent, but Python isnt adverse to well-known > names in general) > > eg: > iso-8859-1.py: > > import unicodec > def encode(...): > ... > def decode(...): > ... > > iso-8859-2.py: > from iso-8859-1 import * > This is the … [View More]

2 1

Rosette i18n API
by Andy Robinson Nov. 17, 1999

Nov. 17, 1999

There is a very capable C++ library at http://rosette.basistech.com/ It is well worth looking at the things this API actually lets you do for ideas on patterns. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

1 0

Some thoughts on the codecs...
by Andy Robinson Nov. 17, 1999

Nov. 17, 1999

Some thoughts on the codecs... 1. Stream interface At the moment a codec has dump and load methods which read a (slice of a) stream into a string in memory and vice versa. As the proposal notes, this could lead to errors if you take a slice out of a stream. This is not just due to character truncation; some Asian encodings are modal and have shift-in and shift-out sequences as they move from Western single-byte characters to double-byte ones. It also seems a bit pointless to me as the … [View More]source (or target) is still a Unicode string in memory. This is a real problem - a filter to convert big files between two encodings should be possible without knowledge of the particular encoding, as should one on the input/output of some server. We can still give a default implementation for single-byte encodings. What's a good API for real stream conversion? just Codec.encodeStream(infile, outfile) ? or is it more useful to feed the codec with data a chunk at a time? 2. Data driven codecs I really like codecs being objects, and believe we could build support for a lot more encodings, a lot sooner than is otherwise possible, by making them data driven rather making each one compiled C code with static mapping tables. What do people think about the approach below? First of all, the ISO8859-1 series are straight mappings to Unicode code points. So one Python script could parse these files and build the mapping table, and a very small data file could hold these encodings. A compiled helper function analogous to string.translate() could deal with most of them. Secondly, the double-byte ones involve a mixture of algorithms and data. The worst cases I know are modal encodings which need a single-byte lookup table, a double-byte lookup table, and have some very simple rules about escape sequences in between them. A simple state machine could still handle these (and the single-byte mappings above become extra-simple special cases); I could imagine feeding it a totally data-driven set of rules. Third, we can massively compress the mapping tables using a notation which just lists contiguous ranges; and very often there are relationships between encodings. For example, "cpXYZ is just like cpXYY but with an extra 'smiley' at 0XFE32". In these cases, a script can build a family of related codecs in an auditable manner. 3. What encodings to distribute? The only clean answers to this are 'almost none', or 'everything that Unicode 3.0 has a mapping for'. The latter is going to add some weight to the distribution. What are people's feelings? Do we ship any at all apart from the Unicode ones? Should new encodings be downloadable from www.python.org? Should there be an optional package outside the main distribution? Thanks, Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com [View Less]

12 29

RE: [Python-Dev] Unicode proposal: %-formatting ?
by Andy Robinson Nov. 17, 1999

Nov. 17, 1999

--- Tim Peters <tim_one(a)email.msn.com> wrote: > I'm more baffled about what "abc" should do. I > didn't understand the t#/s# > etc arguments, and how those do or don't relate to > what str() does. On the > face of it, the idea that a gazillion and one > distinct encodings all get > lumped into "a string object" without remembering > their nature makes about > as much sense as if Python were to treat all > instances of all user-defined > classes as … [View More]

1 0

Unicode proposal: %-formatting ?
by M.-A. Lemburg Nov. 17, 1999

Nov. 17, 1999

I wonder how we could add %-formatting to Unicode strings without duplicating the PyString_Format() logic. First, do we need Unicode object %-formatting at all ? Second, here is an emulation using strings and <default encoding> that should give an idea of one could work with the different encodings: s = '%s %i abcäöü' # a Latin-1 encoded string t = (u,3) # Convert Latin-1 s to a <default encoding> string via Unicode s1 = unicode(s,'latin-1').encode() # The '%… [View More]

3 4

RE: [Python-Dev] Internationalization Toolkit
by Tim Peters Nov. 17, 1999

Nov. 17, 1999

[Andy Robinson] > ... > I presume no one is actually advocating dropping > ordinary Python strings, or the ability to do > rawdata = open('myfile.txt', 'rb').read() > without any transformations? If anyone has advocated either, they've successfully hidden it from me. Anyone?

2 1

mmapfile module
by Andrew M. Kuchling Nov. 16, 1999

Nov. 16, 1999

Guido van Rossum writes: >Hm, this seems to require a higher-level Python module to hide the >differences. Maybe the Unix version could also use a filename? I >would think that mmap'ed files should always be backed by a file (not >by a pipe, socket etc.). Or is there an issue with secure creation of >temp files? This is a question for a separate thread. Hmm... I don't know of any way to use mmap() on non-file things, either; there are odd special cases, like using … [View More]

3 2

Re: [Python-Dev] Some thoughts on the codecs...
by Andy Robinson Nov. 16, 1999

Nov. 16, 1999

--- "M.-A. Lemburg" <mal(a)lemburg.com> wrote: > So I can drop JIS ? [I won't be able to drop the > escaped unicode > codec because this is needed for u"" and ur"".] Drop Japanese from the core language. JIS0208 is a big character set with three popular encodings (Shift-JIS, EUC-JP and JIS), and a host of slight variations; it has 6879 characters, and there are a range of options a user might need to set for it to be useful. So let's assume for now this a separate package. … [View More]

2 1

Re: [Python-Dev] Internationalization Toolkit
by Da Silva, Mike Nov. 16, 1999

Nov. 16, 1999

Fredrik Lundh wrote: > 5. UTF-16 requires string operations that do not make assumptions about > nulls - this means re-implementing most of the C runtime functions to work > with unsigned shorts. footnote: the mad scientist has been there and done that: http://www.pythonware.com/madscientist/ <http://www.pythonware.com/madscientist/> (and you can replace "unsigned short" with "whatever's suitable on this platform") Surely using a different type on different platforms means … [View More]

7 12

RE: [Python-Dev] Internationalization Toolkit
by Da Silva, Mike Nov. 16, 1999

Nov. 16, 1999

Most of the ASCII string functions do indeed work for UTF-8. I have made extensive use of this feature when writing translation logic to harmonize ASCII text (an SQL statement) with substitution parameters that must be converted from IBM EBCDIC code pages (5035, 1027) into UTF8. Since UTF-8 is a superset of ASCII, this all works fine. Some of the character classification functions etc can be flaky when used with UTF8 characters outside the ASCII range, but simple string operations work fine. … [View More] As I see it, the relative pros and cons of UTF-8 versus UTF-16 for use as an internal string representation are: 1. UTF-8 allows all characters to be displayed (in some form or other) on the users machine, with or without native fonts installed. Naturally anything outside the ASCII range will be garbage, but it is an immense debugging aid when working with character encodings to be able to touch and feel something recognizable. Trying to decode a block of raw UTF-16 is a pain. 2. UTF-8 works with most existing string manipulation libraries quite happily. It is also portable (a char is always 8 bits, regardless of platform; wchar_t varies between 16 and 32 bits depending on the underlying operating system (although unsigned short does seems to work across platforms, in my experience). 3. UTF-16 has some advantages in providing fixed width characters and, (ignoring surrogate pairs etc) a modeless encoding space. This is an advantage for fast string operations, especially on CPU's that have efficient operations for handling 16bit data. 4. UTF-16 would directly support a tightly coupled character properties engine, which would enable Unicode compliant case folding and character decomposition to be performed without an intermediate UTF-8 <----> UTF-16 translation step. 5. UTF-16 requires string operations that do not make assumptions about nulls - this means re-implementing most of the C runtime functions to work with unsigned shorts. Regards, Mike da Silva -----Original Message----- From: Greg Stein [SMTP:gstein@lyra.org] Sent: 12 November 1999 10:30 To: Tim Peters Cc: python-dev(a)python.org Subject: RE: [Python-Dev] Internationalization Toolkit On Fri, 12 Nov 1999, Tim Peters wrote: >... > Using UTF-8 internally is also reasonable, and if it's being rejected on the > grounds of supposed slowness No... my main point was interaction with the underlying OS. I made a SWAG (Scientific Wild Ass Guess :-) and stated that UTF-8 is probably slower for various types of operations. As always, your infernal meddling has dashed that hypothesis, so I must retreat... >... > I expect either would work well. It's at least curious that Perl and Tcl > both went with UTF-8 -- does anyone think they know *why*? I don't. The > people here saying UCS-2 is the obviously better choice are all from the > Microsoft camp <wink>. It's not obvious to me, but then neither do I claim > that UTF-8 is obviously better. Probably for the exact reason that you stated in your messages: many 8-bit (7-bit?) functions continue to work quite well when given a UTF-8-encoded string. i.e. they didn't have to rewrite the entire Perl/TCL interpreter to deal with a new string type. I'd guess it is a helluva lot easier for us to add a Python Type than for Perl or TCL to whack around with new string types (since they use strings so heavily). Cheers, -g -- Greg Stein, http://www.lyra.org/ _______________________________________________ Python-Dev maillist - Python-Dev(a)python.org http://www.python.org/mailman/listinfo/python-dev [View Less]

4 4

PyErr_Format security note
by A.M. Kuchling Nov. 15, 1999

Nov. 15, 1999

I noticed this in PyErr_Format(exception, format, va_alist): char buffer[500]; /* Caller is responsible for limiting the format */ ... vsprintf(buffer, format, vargs); Making the caller responsible for this is error-prone. The danger, of course, is a buffer overflow caused by generating an error string that's larger than the buffer, possibly letting people execute arbitrary code. We could add a test to the configure script for vsnprintf() and use it when possible, but that only fixes the … [View More]

8 13

Some thoughts on the codecs...
by Da Silva, Mike Nov. 15, 1999

Nov. 15, 1999

Andy Robinson wrote: 1. Stream interface At the moment a codec has dump and load methods which read a (slice of a) stream into a string in memory and vice versa. As the proposal notes, this could lead to errors if you take a slice out of a stream. This is not just due to character truncation; some Asian encodings are modal and have shift-in and shift-out sequences as they move from Western single-byte characters to double-byte ones. It also seems a bit pointless to me as the source (or … [View More]target) is still a Unicode string in memory. This is a real problem - a filter to convert big files between two encodings should be possible without knowledge of the particular encoding, as should one on the input/output of some server. We can still give a default implementation for single-byte encodings. What's a good API for real stream conversion? just Codec.encodeStream(infile, outfile) ? or is it more useful to feed the codec with data a chunk at a time? A user defined chunking factor (suitably defaulted) would be useful for processing large files. 2. Data driven codecs I really like codecs being objects, and believe we could build support for a lot more encodings, a lot sooner than is otherwise possible, by making them data driven rather making each one compiled C code with static mapping tables. What do people think about the approach below? First of all, the ISO8859-1 series are straight mappings to Unicode code points. So one Python script could parse these files and build the mapping table, and a very small data file could hold these encodings. A compiled helper function analogous to string.translate() could deal with most of them. Secondly, the double-byte ones involve a mixture of algorithms and data. The worst cases I know are modal encodings which need a single-byte lookup table, a double-byte lookup table, and have some very simple rules about escape sequences in between them. A simple state machine could still handle these (and the single-byte mappings above become extra-simple special cases); I could imagine feeding it a totally data-driven set of rules. Third, we can massively compress the mapping tables using a notation which just lists contiguous ranges; and very often there are relationships between encodings. For example, "cpXYZ is just like cpXYY but with an extra 'smiley' at 0XFE32". In these cases, a script can build a family of related codecs in an auditable manner. The problem here is that we need to decide whether we are Unicode-centric, or whether Unicode is just another supported encoding. If we are Unicode-centric, then all code-page translations will require static mapping tables between the appropriate Unicode character and the relevant code points in the other encoding. This would involve (worst case) 64k static tables for each supported encoding. Unfortunately this also precludes the use of algorithmic conversions and or sparse conversion tables because most of these transformations are relative to a source and target non-Unicode encoding, eg JIS <---->EUCJIS. If we are taking the IBM approach (see CDRA), then we can mix and match approaches, and treat Unicode strings as just Unicode, and normal strings as being any arbitrary MBCS encoding. To guarantee the utmost interoperability and Unicode 3.0 (and beyond) compliance, we should probably assume that all core encodings are relative to Unicode as the pivot encoding. This should hopefully avoid any gotcha's with roundtrips between any two arbitrary native encodings. The downside is this will probably be slower than an optimised algorithmic transformation. 3. What encodings to distribute? The only clean answers to this are 'almost none', or 'everything that Unicode 3.0 has a mapping for'. The latter is going to add some weight to the distribution. What are people's feelings? Do we ship any at all apart from the Unicode ones? Should new encodings be downloadable from www.python.org <http://www.python.org> ? Should there be an optional package outside the main distribution? Ship with Unicode encodings in the core, the rest should be an add on package. If we are truly Unicode-centric, this gives us the most value in terms of accessing a Unicode character properties database, which will provide language neutral case folding, Hankaku <----> Zenkaku folding (Japan specific), and composition / normalisation between composed characters and their component nonspacing characters. Regards, Mike da Silva [View Less]

2 1

Re: [Python-Dev] just say no...
by Andy Robinson Nov. 15, 1999

Nov. 15, 1999

--- Guido van Rossum <guido(a)CNRI.Reston.VA.US> wrote: > Did you read Andy Robinson's case study? He > suggested that for certain encodings there may be > other things you can do that are more > user-friendly than raising an exception, depending > on the application. I am proposing to leave this a > detail of each specific translation. > There may even be translations that do the same thing > except they have a different behavior for > untranslatable cases … [View More]

1 0

Re: [Python-Dev] just say no...
by Jack Jansen Nov. 13, 1999

Nov. 13, 1999

Recently, Greg Stein <gstein(a)lyra.org> said: > This was done last year!! We have "s#" meaning "give me some bytes." We > have "t#" meaning "give me some 8-bit characters." The Python distribution > has been completely updated to use the appropriate format in each call. Oops... I remember the discussion but I wasn't aware that somone had actually _implemented_ this:-). Part of my misunderstanding was also caused by the fact that I inspected what I thought would be the prime … [View More]

2 1

Re: [Python-Dev] just say no...
by Jack Jansen Nov. 13, 1999

Nov. 13, 1999

The problem with "s" and "s#" is that they're already semantically overloaded, and will become more so with support for multiple charsets. Some modules use "s#" when they mean "give me a pointer to an area of memory and its length". Writing to binary files is an example of this. Some modules use it to mean "give me a pointer to a string". Writing to a text file is (probably) an example of this. Some modules use it to mean "give me a pointer to an 8-bit ASCII string". This is the case if … [View More]

3 4

RE: [Python-Dev] Internationalization Toolkit
by Andy Robinson Nov. 12, 1999

Nov. 12, 1999

--- "Da Silva, Mike" <Mike.Da.Silva(a)uk.fid-intl.com> wrote: > As I see it, the relative pros and cons of UTF-8 > versus UTF-16 for use as an > internal string representation are: > [snip] > Regards, > Mike da Silva > Note that by going with UTF16, we get both. We will certainly have a codec for utf8, just as we will for ISO-Latin-1, Shift-JIS or whatever. And a perfectly ordinary Python string is a great place to hold UTF8; you can look at it and use most of the … [View More]

1 0

RE: [Python-Dev] Internationalization Toolkit
by Andy Robinson Nov. 12, 1999

Nov. 12, 1999

--- Gordon McMillan <gmcm(a)hypernet.com> wrote: > [per-thread defaults] > > C'mon guys, hasn't anyone ever played consultant > before? The > idea is obviously brain-dead. OTOH, they asked for > it > specifically, meaning they have some assumptions > about how > they think they're going to use it. If you give them > what they > ask for, you'll only have to fix it when they > realize there are > other ways of doing things that don't work with &… [View More]

2 1

Internationalization Toolkit
by Andy Robinson Nov. 12, 1999

Nov. 12, 1999

In general, I like this proposal a lot, but I think it only covers half the story. How we actually build the encoder/decoder for each encoding is a very big issue. Thoughts on this below. First, a little nit > u = u'<utf-8 encoded Python string>' I don't like using funny prime characters - why not an explicit function like "utf8()" On to the important stuff:> > unicodec.register(<encname>,<encoder>,<decoder> > [,<stream_encoder>, <… [View More]stream_decoder>]) > This registers the codecs under the given encoding > name in the module global dictionary > unicodec.codecs. Stream codecs are optional: > the unicodec module will provide appropriate > wrappers around <encoder> and > <decoder> if not given. I would MUCH prefer a single 'Encoding' class or type to wrap up these things, rather than up to four disconnected objects/functions. Essentially it would be an interface standard and would offer methods to do the four things above. There are several reasons for this. (1) there are quite a lot of things you might want to do with an encoding object, and we could extend the interface in future easily. As a minimum, give it the four methods implied by the above, two of which can be defaults. But I'd like an encoding to be able to tell me the set of characters to which it applies; validate a string; and maybe tell me if it is a subset or superset of another. (2) especially with double-byte encodings, they will need to load up some kind of database on startup and use this for both encoding and decoding - much better to share it and encapsulate it inside one object (3) for some languages, there are extra functions wanted. For Japanese, you need two or three functions to expand half-width to full-width katakana, convert double-byte english to single-byte and vice versa. A Japanese encoding object would be a handy place to put this knowledge. (4) In the real world you get many encodings which are subtle variations of the same thing, plus or minus a few characters. One bit of code might be able to share the work of several encodings, by setting a few flags. Certainly true of Japanese. (5) encoding/decoding algorithms can be program or data or (very often) a bit of both. We have not yet discussed where to keep all the mapping tables, but if data is involved it should be hidden in an object. (6) See my comments on a state machine for doing the encodings. If this is done well, we might two different standard objects which conform to the Encoding interface (a really light one for single-byte encodings, and a bigger one for multi-byte), and everything else could be data driven. (6) Easy to grow - encodings can be prototyped and proven in Python, ported to C if needed or when ready. In summary, firm up the concept of an Encoding object and give it room to grow - that's the key to real-world usefulness. If people feel the same way I'll have a go at an interface for that, and try show how it would have simplified specific problems I have faced. We also need to think about where encoding info will live. You cannot avoid mapping tables, although you can hide them inside code modules or pickled objects if you want. Should there be a standard "..\Python\Enc" directory? And we're going to need some kind of testing and certification procedure when adding new encodings. This stuff has to be right. Guido asked about TypedString. This can probably be done on top of the built-in stuff - it is just a convenience which would clarify intent, reduce lines of code and prevent people shooting themselves in the foot when juggling a lot of strings in different (non-Unicode) encodings. I can do a Python module to implement that on top of whatever is built. Regards, Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com [View Less]

5 6

Re: [Python-Dev] Internationalization Toolkit
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

> See my other post on the subject... > > Note that if we make UTF-8 the standard encoding, > nearly all > special Latin-1 characters will produce UTF-8 errors > on input > and unreadable garbage on output. That will probably > be unacceptable > in Europe. To remedy this, one would *always* have > to use > u.encode('latin-1') to get readable output for > Latin-1 strings > repesented in Unicode. You beat me to it - a colleague and I were just discussing … [View More]

2 1

Re: [Python-Dev] Internationalization Toolkit
by Jack Jansen Nov. 11, 1999

Nov. 11, 1999

> > [MAL, on Unicode chr() and ord() > > > ... > > > Because unichr() will always have to return Unicode objects. You don't > > > want chr(i) to return Unicode for i>255 and strings for i<256. > > > OTOH, ord() could probably be extended to also work on Unicode objects. > Fine. So I'll drop the uniord() API and extend ord() instead. Hmm, then wouldn't it be more logical to drop unichr() too, but add an optional parameter to chr() to specify … [View More]

2 1

Re: [Python-Dev] Internationalization Toolkit
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

> I say axe it and say "UTF-8" is the fixed, default > encoding. If you want > something else, then do that explicitly. > Let me tell you why you would want to have an encoding which can be set: (1) sday I am on a Japanese Windows box, I have a string called 'address' and I do 'print address'. If I see utf8, I see garbage. If I see Shift-JIS, I see the correct Japanese address. At this point in time, utf8 is an interchange format but 99% of the world's data is in various native … [View More]

3 2

Re: [Python-Dev] Internationalization Toolkit
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

> You almost convinced me there, but I think this can > still be done > without changing the default encoding: simply reopen > stdout with a > different encoding. This is how Java does it. I/O > streams with an > encoding specified at open() are a very powerful > feature. You can > hide this in your $PYTHONSTARTUP. Good point, I'm happy with this. Make sure we specify it in the docs as the right way to do it. In an IDE, we'd have an Options screen somewhere for … [View More]

1 0

Re: [Python-Dev] Internationalization Toolkit (fwd)
by Greg Stein Nov. 11, 1999

Nov. 11, 1999

Andy originally sent this just to me... I replied in kind, but saw that he sent another copy to python-dev. Sending my reply there... ---------- Forwarded message ---------- Date: Thu, 11 Nov 1999 04:00:38 -0800 (PST) From: Greg Stein <gstein(a)lyra.org> To: andy(a)robanal.demon.co.uk Subject: Re: [Python-Dev] Internationalization Toolkit [ note: you sent direct to me; replying in kind in case that was your intent ] On Wed, 10 Nov 1999, [iso-8859-1] Andy Robinson wrote: >... > … [View More]

1 0

Re: [Python-Dev] default encodings (was: Internationalization Toolkit)
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

> > What about just explaining the rationale for the > default-less point of > view to whoever is in charge of this at HP and see > why they came up with > their rationale in the first place? They might have > a good reason, or > they might be willing to change said requirement. > > --david For that matter (I came into this a bit late), is there a statement somewhere of what HP actually want to do? - Andy ===== Andy Robinson Robinson Analytics Ltd. ---------… [View More]

2 1

Re: cached encoding (Re: [Python-Dev] Internationalization Toolkit)
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

--- "Barry A. Warsaw" <bwarsaw(a)cnri.reston.va.us> wrote: > > I'm starting to think about devday topics. Sounds > like an I18n > session would be very useful. Champions? > I'm willing to explain what the fuss is about to bemused onlookers and give some examples of problems it should be able to solve - plenty of good slides and screen shots. I'll stay well away from the C implementation issues. Regards, Andy ===== Andy Robinson Robinson Analytics Ltd. ---------------… [View More]

1 0

Re: cached encoding (Re: [Python-Dev] Internationalization Toolkit)
by Andy Robinson Nov. 11, 1999

Nov. 11, 1999

> 2. Are there plans for an internationalization > session at IPC8? Perhaps a > few key players could be locked into a room for a > couple days, to emerge > bloodied, but with an implementation in-hand... Excellent idea. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for … [View More]

1 0

Re: [Python-Dev] I18N Toolkit
by Jack Jansen Nov. 10, 1999

Nov. 10, 1999

> a slightly hairer design issue is what combinations > of pattern and string the new 're' will handle. > > the first two are obvious: > > ordinary pattern, ordinary string > unicode pattern, unicode string > > but what about these? > > ordinary pattern, unicode string > unicode pattern, ordinary string I think the logical thing to do would be to "promote" the ordinary pattern or string to unicode, in a similar way to what … [View More]

1 0

Internationalisation Case Study
by Andy Robinson Nov. 9, 1999

Nov. 9, 1999

Guido has asked me to get involved in this discussion, as I've been working practically full-time on i18n for the last year and a half and have done quite a bit with Python in this regard. I thought the most helpful thing would be to describe the real-world business problems I have been tackling so people can understand what one might want from an encoding toolkit. In this (long) post I have included: 1. who I am and what I want to do 2. useful sources of info 3. a real world i18n project 4. … [View More]what I'd like to see in an encoding toolkit Grab a coffee - this is a long one. 1. Who I am -------------- Firstly, credentials. I'm a Python programmer by night, and when I can involve it in my work which happens perhaps 20% of the time. More relevantly, I did a postgrad course in Japanese Studies and lived in Japan for about two years; in 1990 when I returned, I was speaking fairly fluently and could read a newspaper with regular reference tio a dictionary. Since then my Japanese has atrophied badly, but it is good enough for IT purposes. For the last year and a half I have been internationalizing a lot of systems - more on this below. My main personal interest is that I am hoping to launch a company using Python for reporting, data cleaning and transformation. An encoding library is sorely needed for this. 2. Sources of Knowledge ------------------------------ We should really go for world class advice on this. Some people who could really contribute to this discussion are: - Ken Lunde, author of "CJKV Information Processing" and head of Asian Type Development at Adobe. - Jeffrey Friedl, author of "Mastering Regular Expressions", and a long time Japan resident and expert on things Japanese - Maybe some of the Ruby community? I'll list up books URLs etc. for anyone who needs them on request. 3. A Real World Project ---------------------------- 18 months ago I was offered a contract with one of the world's largest investment management companies (which I will nickname HugeCo) , who (after many years having analysts out there) were launching a business in Japan to attract savers; due to recent legal changes, Japanese people can now freely buy into mutual funds run by foreign firms. Given the 2% they historically get on their savings, and the 12% that US equities have returned for most of this century, this is a business with huge potential. I've been there for a while now, rotating through many different IT projects. HugeCo runs its non-US business out of the UK. The core deal-processing business runs on IBM AS400s. These are kind of a cross between a relational database and a file system, and speak their own encoding called EBCDIC. Five years ago the AS400 had limited connectivity to everything else, so they also started deploying Sybase databases on Unix to support some functions. This means 'mirroring' data between the two systems on a regular basis. IBM has always included encoding information on the AS400 and it converts from EBCDIC to ASCII on request with most of the transfer tools (FTP, database queries etc.) To make things work for Japan, everyone realised that a double-byte representation would be needed. Japanese has about 7000 characters in most IT-related character sets, and there are a lot of ways to store it. Here's a potted language lesson. (Apologies to people who really know this field -- I am not going to be fully pedantic or this would take forever). Japanese includes two phonetic alphabets (each with about 80-90 characters), the thousands of Kanji, and English characters, often all in the same sentence. The first attempt to display something was to make a single -byte character set which included ASCII, and a simplified (and very ugly) katakana alphabet in the upper half of the code page. So you could spell out the sounds of Japanese words using 'half width katakana'. The basic 'character set' is Japan Industrial Standard 0208 ("JIS"). This was defined in 1978, the first official Asian character set to be defined by a government. This can be thought of as a printed chart showing the characters - it does not define their storage on a computer. It defined a logical 94 x 94 grid, and each character has an index in this grid. The "JIS" encoding was a way of mixing ASCII and Japanese in text files and emails. Each Japanese character had a double-byte value. It had 'escape sequences' to say 'You are now entering ASCII territory' or the opposite. In 1978 Microsoft quickly came up with Shift-JIS, a smarter encoding. This basically said "Look at the next byte. If below 127, it is ASCII; if between A and B, it is a half-width katakana; if between B and C, it is the first half of a double-byte character and the next one is the second half". Extended Unix Code (EUC) does similar tricks. Both have the property that there are no control characters, and ASCII is still ASCII. There are a few other encodings too. Unfortunately for me and HugeCo, IBM had their own standard before the Japanese government did, and it differs; it is most commonly called DBCS (Double-Byte Character Set). This involves shift-in and shift-out sequences (0x16 and 0x17, cannot remember which way round), so you can mix single and double bytes in a field. And we used AS400s for our core processing. So, back to the problem. We had a FoxPro system using ShiftJIS on the desks in Japan which we wanted to replace in stages, and an AS400 database to replace it with. The first stage was to hook them up so names and addresses could be uploaded to the AS400, and data files consisting of daily report input could be downloaded to the PCs. The AS400 supposedly had a library which did the conversions, but no one at IBM knew how it worked. The people who did all the evaluations had basically proved that 'Hello World' in Japanese could be stored on an AS400, but never looked at the conversion issues until mid-project. Not only did we need a conversion filter, we had the problem that the character sets were of different sizes. So it was possible - indeed, likely - that some of our ten thousand customers' names and addresses would contain characters only on one system or the other, and fail to survive a round trip. (This is the absolute key issue for me - will a given set of data survive a round trip through various encoding conversions?) We figured out how to get the AS400 do to the conversions during a file transfer in one direction, and I wrote some Python scripts to make up files with each official character in JIS on a line; these went up with conversion, came back binary, and I was able to build a mapping table and 'reverse engineer' the IBM encoding. It was straightforward in theory, "fun" in practice. I then wrote a python library which knew about the AS400 and Shift-JIS encodings, and could translate a string between them. It could also detect corruption and warn us when it occurred. (This is another key issue - you will often get badly encoded data, half a kanji or a couple of random bytes, and need to be clear on your strategy for handling it in any library). It was slow, but it got us our gateway in both directions, and it warned us of bad input. 360 characters in the DBCS encoding actually appear twice, so perfect round trips are impossible, but practically you can survive with some validation of input at both ends. The final story was that our names and addresses were mostly safe, but a few obscure symbols weren't. A big issue was that field lengths varied. An address field 40 characters long on a PC might grow to 42 or 44 on an AS400 because of the shift characters, so the software would truncate the address during import, and cut a kanji in half. This resulted in a string that was illegal DBCS, and errors in the database. To guard against this, you need really picky input validation. You not only ask 'is this string valid Shift-JIS', you check it will fit on the other system too. The next stage was to bring in our Sybase databases. Sybase make a Unicode database, which works like the usual one except that all your SQL code suddenly becomes case sensitive - more (unrelated) fun when you have 2000 tables. Internally it stores data in UTF8, which is a 'rearrangement' of Unicode which is much safer to store in conventional systems. Basically, a UTF8 character is between one and three bytes, there are no nulls or control characters, and the ASCII characters are still the same ASCII characters. UTF8<->Unicode involves some bit twiddling but is one-to-one and entirely algorithmic. We had a product to 'mirror' data between AS400 and Sybase, which promptly broke when we fed it Japanese. The company bought a library called Unilib to do conversions, and started rewriting the data mirror software. This library (like many) uses Unicode as a central point in all conversions, and offers most of the world's encodings. We wanted to test it, and used the Python routines to put together a regression test. As expected, it was mostly right but had some differences, which we were at least able to document. We also needed to rig up a daily feed from the legacy FoxPro database into Sybase while it was being replaced (about six months). We took the same library, built a DLL wrapper around it, and I interfaced to this with DynWin , so we were able to do the low-level string conversion in compiled code and the high-level control in Python. A FoxPro batch job wrote out delimited text in shift-JIS; Python read this in, ran it through the DLL to convert it to UTF8, wrote that out as UTF8 delimited files, ftp'ed them to an in directory on the Unix box ready for daily import. At this point we had a lot of fun with field widths - Shift-JIS is much more compact than UTF8 when you have a lot of kanji (e.g. address fields). Another issue was half-width katakana. These were the earliest attempt to get some form of Japanese out of a computer, and are single-byte characters above 128 in Shift-JIS - but are not part of the JIS0208 standard. They look ugly and are discouraged; but when you ar enterinh a long address in a field of a database, and it won't quite fit, the temptation is to go from two-bytes-per -character to one (just hit F7 in windows) to save space. Unilib rejected these (as would Java), but has optional modes to preserve them or 'expand them out' to their full-width equivalents. The final technical step was our reports package. This is a 4GL using a really horrible 1980s Basic-like language which reads in fixed-width data files and writes out Postscript; you write programs saying 'go to x,y' and 'print customer_name', and can build up anything you want out of that. It's a monster to develop in, but when done it really works - million page jobs no problem. We had bought into this on the promise that it supported Japanese; actually, I think they had got the equivalent of 'Hello World' out of it, since we had a lot of problems later. The first stage was that the AS400 would send down fixed width data files in EBCDIC and DBCS. We ran these through a C++ conversion utility, again using Unilib. We had to filter out and warn about corrupt fields, which the conversion utility would reject. Surviving records then went into the reports program. It then turned out that the reports program only supported some of the Japanese alphabets. Specifically, it had a built in font switching system whereby when it encountered ASCII text, it would flip to the most recent single byte text, and when it found a byte above 127, it would flip to a double byte font. This is because many Chinese fonts do (or did) not include English characters, or included really ugly ones. This was wrong for Japanese, and made the half-width katakana unprintable. I found out that I could control fonts if I printed one character at a time with a special escape sequence, so wrote my own bit-scanning code (tough in a language without ord() or bitwise operations) to examine a string, classify every byte, and control the fonts the way I wanted. So a special subroutine is used for every name or address field. This is apparently not unusual in GUI development (especially web browsers) - you rarely find a complete Unicode font, so you have to switch fonts on the fly as you print a string. After all of this, we had a working system and knew quite a bit about encodings. Then the curve ball arrived: User Defined Characters! It is not true to say that there are exactly 6879 characters in Japanese, and more than counting the number of languages on the Indian sub-continent or the types of cheese in France. There are historical variations and they evolve. Some people's names got missed out, and others like to write a kanji in an unusual way. Others arrived from China where they have more complex variants of the same characters. Despite the Japanese government's best attempts, these people have dug their heels in and want to keep their names the way they like them. My first reaction was 'Just Say No' - I basically said that it one of these customers (14 out of a database of 8000) could show me a tax form or phone bill with the correct UDC on it, we would implement it but not otherwise (the usual workaround is to spell their name phonetically in katakana). But our marketing people put their foot down. A key factor is that Microsoft has 'extended the standard' a few times. First of all, Microsoft and IBM include an extra 360 characters in their code page which are not in the JIS0208 standard. This is well understood and most encoding toolkits know what 'Code Page 932' is Shift-JIS plus a few extra characters. Secondly, Shift-JIS has a User-Defined region of a couple of thousand characters. They have lately been taking Chinese variants of Japanese characters (which are readable but a bit old-fashioned - I can imagine pipe-smoking professors using these forms as an affectation) and adding them into their standard Windows fonts; so users are getting used to these being available. These are not in a standard. Thirdly, they include something called the 'Gaiji Editor' in Japanese Win95, which lets you add new characters to the fonts on your PC within the user-defined region. The first step was to review all the PCs in the Tokyo office, and get one centralized extension font file on a server. This was also fun as people had assigned different code points to characters on differene machines, so what looked correct on your word processor was a black square on mine. Effectively, each company has its own custom encoding a bit bigger than the standard. Clearly, none of these extensions would convert automatically to the other platforms. Once we actually had an agreed list of code points, we scanned the database by eye and made sure that the relevant people were using them. We decided that space for 128 User-Defined Characters would be allowed. We thought we would need a wrapper around Unilib to intercept these values and do a special conversion; but to our amazement it worked! Somebody had already figured out a mapping for at least 1000 characters for all the Japanes encodings, and they did the round trips from Shift-JIS to Unicode to DBCS and back. So the conversion problem needed less code than we thought. This mapping is not defined in a standard AFAIK (certainly not for DBCS anyway). We did, however, need some really impressive validation. When you input a name or address on any of the platforms, the system should say (a) is it valid for my encoding? (b) will it fit in the available field space in the other platforms? (c) if it contains user-defined characters, are they the ones we know about, or is this a new guy who will require updates to our fonts etc.? Finally, we got back to the display problems. Our chosen range had a particular first byte. We built a miniature font with the characters we needed starting in the lower half of the code page. I then generalized by name-printing routine to say 'if the first character is XX, throw it away, and print the subsequent character in our custom font'. This worked beautifully - not only could we print everything, we were using type 1 embedded fonts for the user defined characters, so we could distill it and also capture it for our internal document imaging systems. So, that is roughly what is involved in building a Japanese client reporting system that spans several platforms. I then moved over to the web team to work on our online trading system for Japan, where I am now - people will be able to open accounts and invest on the web. The first stage was to prove it all worked. With HTML, Java and the Web, I had high hopes, which have mostly been fulfilled - we set an option in the database connection to say 'this is a UTF8 database', and Java converts it to Unicode when reading the results, and we set another option saying 'the output stream should be Shift-JIS' when we spew out the HTML. There is one limitations: Java sticks to the JIS0208 standard, so the 360 extra IBM/Microsoft Kanji and our user defined characters won't work on the web. You cannot control the fonts on someone else's web browser; management accepted this because we gave them no alternative. Certain customers will need to be warned, or asked to suggest a standard version of a charactere if they want to see their name on the web. I really hope the web actually brings character usage in line with the standard in due course, as it will save a fortune. Our system is multi-language - when a customer logs in, we want to say 'You are a Japanese customer of our Tokyo Operation, so you see page X in language Y'. The language strings all all kept in UTF8 in XML files, so the same file can hold many languages. This and the database are the real-world reasons why you want to store stuff in UTF8. There are very few tools to let you view UTF8, but luckily there is a free Word Processor that lets you type Japanese and save it in any encoding; so we can cut and paste between Shift-JIS and UTF8 as needed. And that's it. No climactic endings and a lot of real world mess, just like life in IT. But hopefully this gives you a feel for some of the practical stuff internationalisation projects have to deal with. See my other mail for actual suggestions - Andy Robinson ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com [View Less]

1 0

wish list
by Guido van Rossum Nov. 8, 1999

Nov. 8, 1999

I got the wish list below. Anyone care to comment on how close we are on fulfilling some or all of this? --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Thu, 04 Nov 1999 20:26:54 +0700 From: "Claudio Ramón" <rmn70(a)hotmail.com> To: guido(a)python.org Hello, I'm a python user (excuse my english, I'm spanish and...). I think it is a very complete language and I use it in solve statistics, phisics, mathematics, chemistry and … [View More]

5 4

updated modules
by Greg Stein Nov. 7, 1999

Nov. 7, 1999

Hi all... I've updated some of the modules at http://www.lyra.org/greg/python/. Specifically, there is a new httplib.py, davlib.py, qp_xml.py, and a new imputil.py. The latter will be updated again RSN with some patches from Jim Ahlstrom. Besides some tweaks/fixes/etc, I've also clarified the ownership and licensing of the things. httplib and davlib are (C) Guido, licensed under the Python license (well... anything he chooses :-). qp_xml and imputil are still Public Domain. I also added some … [View More]

1 0

Re: [Python-Dev] wish list
by Sam Rushing Nov. 5, 1999

Nov. 5, 1999

James C. Ahlstrom writes: > Guido van Rossum wrote: > > I got the wish list below. Anyone care to comment on how close we are > > on fulfilling some or all of this? > > > * GNU CC for Win32 compatibility (compilation of python interpreter and > > "Freeze" utility). I think MingWin32 (Mummint Khan) is a good alternative > > eviting the cygwin dll user. > > I don't know what this means. mingw32: 'minimalist gcc for win32'. it's gcc on win32 … [View More]

3 2

paper available
by Vladimir Marangozov Nov. 5, 1999

Nov. 5, 1999

I've OCR'd Saltzer's paper. It's available temporarily (in MS Word format) at http://sirac.inrialpes.fr/~marangoz/tmp/Saltzer.zip Since there may be legal problems with LNCS, I will disable the link shortly (so those of you who have not received a copy and are interested in reading it, please grab it quickly) If prof. Saltzer agrees (and if he can, legally) put it on his web page, I guess that the paper will show up at http://mit.edu/saltzer/ Jeremy, could you please check this with prof. … [View More]

2 1

Benevolent dictator versus the bureaucratic committee?
by Mark Hammond Nov. 2, 1999

Nov. 2, 1999

I have for some time been wondering about the usefulness of this mailing list. It seems to have produced staggeringly few results since inception. This is not a critisism of any individual, but of the process. It is proof in my mind of how effective the benevolent dictator model is, and how ineffective a language run by committee would be. This "committee" never seems to be capable of reaching a consensus on anything. A number of issues dont seem to provoke any responses. As a result, … [View More]many things seem to die a slow and lingering death. Often there is lots of interesting discussion, but still precious few results. In the pre python-dev days, the process seemed easier - we mailed Guido directly, and he either stated "yea" or "nay" - maybe we didnt get the response we hoped for, but at least we got a response. Now, we have the result that even if Guido does enter into a thread, the noise seems to drown out any hope of getting anything done. Guido seems to be faced with the dilemma of asserting his dictatorship in the face of many dissenting opinions from many people he respects, or putting it in the too hard basket. I fear the latter is the easiest option. At the end of this mail I list some of the major threads over the last few months, and can't see a single thread that has resulted in a CVS checkin, and only one that has resulted in agreement. This, to my mind at least, is proof that things are really not working. I long for the "good old days" - take the replacement of "ni" with built-in functionality, for example. I posit that if this was discussed on python-dev, it would have caused a huge flood of mail, and nothing remotely resembling a consensus. Instead, Guido simply wrote an essay and implemented some code that he personally liked. No debate, no discussion. Still an excellent result. Maybe not a perfect result, but a result nonetheless. However, Guido's time is becoming increasingly limited. So should we consider moving to a "benevolent lieutenent" model, in conjunction with re-ramping up the SIGS? This would provide 2 ways to get things done: * A new SIG. Take relative imports, for example. If we really do need a change in this fairly fundamental area, a SIG would be justified ("import-sig"). The responsibility of the SIG is to form a consensus (and code that reflects it), and report back to Guido (and the main newsgroup) with the result of this. It worked well for RE, and allowed those of us not particularly interested to keep out of the debate. If the SIG can not form consensus, then tough - it dies - and should not be mourned. Presumably Guido would keep a watchful eye over the SIG, providing direction where necessary, but in general stay out of the day to day traffic. New SIGs seem to have stopped since this list creation, and it seems that issues that should be discussed in new SIGS are now discussed here. * Guido could delegate some of his authority to a single individual responsible for a certain limited area - a benevolent lieutenent. We may have a lieutentant responsible for different areas, and could only exercise their authority with small, trivial changes. Eg, the "getopt helper" thread - if a lieutenant was given authority for the "standard library", they could simply make a yea or nay decision, and present it to Guido. Presumably Guido trusts this person he delegated to enough that the majority of the lieutenant's recommendations would be accepted. Presumably there would be a small number of lieutentants, and they would then become the new "python-dev" - say up to 5 people. This list then discusses high level strategies and seek direction from each other when things get murky. This select group of people may not (indeed, probably would not) include me, but I would have no problem with that - I would prefer to see results achieved than have my own ego stroked by being included in a select, but ineffective group. In parting, I repeat this is not a direct critisism, simply an observation of the last few months. I am on this list, so I am definately as guilty as any one else - which is "not at all" - ie, no one is guilty, I simply see it as endemic to a committee with people of diverse backgrounds, skills and opinions. Any thoughts? Long live the dictator! :-) Mark. Recent threads, and my take on the results: * getopt helper? Too much noise regarding semantic changes. * Alternative Approach to Relative Imports * Relative package imports * Path hacking * Towards a Python based import scheme Too much noise - no one could really agree on the semantics. Implementation thrown in the ring, and promptly forgotten. * Corporate installations Very young, but no result at all. * Embedding Python when using different calling conventions Quite young, but no result as yet, and I have no reason to believe there will be. * Catching "return" and "return expr" at compile time Seemed to be blessed - yay! Dont believe I have seen a check-in yet. * More Python command-line features Seemed general agreement, but nothing happened? * Tackling circular dependencies in 2.0? Lots of noise, but no results other than "GC may be there in 2.0" * Buffer interface in abstract.c Determined it could break - no solution proposed. Lots of noise regarding if is is a good idea at all! * mmapfile module No result. * Quick-and-dirty weak references No result. * Portable "spawn" module for core? No result. * Fake threads Seemed to spawn stackless Python, but in the face of Guido being "at best, lukewarm" about this issue, I would again have to conclude "no result". An authorative "no" in this area may have saved lots of effort and heartache. * add Expat to 1.6 No result. * I'd like list.pop to accept an optional second argument giving a default value No result * etc No result. [View Less]

4 3

Misleading syntax error text
by M.-A. Lemburg Nov. 1, 1999

Nov. 1, 1999

[Extracted from the psa-members list...] Gordon McMillan wrote: > > Chris Fama wrote, > > And now the rub: the exact same function definition has passed > > through byte-compilation perfectly OK many times before with no > > problems... of course, this points rather clearly to the > > preceding code, but it illustrates a failing in Python's syntax > > error messages, and IMHO a fairly serious one at that, if this is > > indeed so. > > My simple … [View More]

2 2

dircache.py
by Gordon McMillan Nov. 1, 1999

Nov. 1, 1999

Pursuant to my volunteering to implement Guido's plan to combine cmp.py, cmpcache.py, dircmp.py and dircache.py into filecmp.py, I did some investigating of dircache.py. I find it completely unreliable. On my NT box, the mtime of the directory is updated (on average) 2 secs after a file is added, but within 10 tries, there's always one in which it takes more than 100 secs (and my test script quits). My Linux box hardly ever detects a change within 100 secs. I've tried a number of ways … [View More]

2 1

Re: [Python-Dev] Benevolent dictator versus the bureaucratic committee?
by Jack Jansen Nov. 1, 1999

Nov. 1, 1999

I think I agree with Mark's post, although I do see a little more light (the relative imports dicussion resulted in working code, for instance). The benevolent lieutenant idea may work, _if_ the lieutenants can be found. I myself will quickly join Mark in wishing the new python-dev well and abandoning ship (half a :-). If that doesn't work maybe we should try at the very least to create a "memory". If you bring up a subject for discussion and you don't have working code that's fine the … [View More]

1 0

Re: [Python-Dev] Embedding Python when using different calling conventions.
by Jack Jansen Nov. 1, 1999

Nov. 1, 1999

> OTOH, we could take chance to reorganize these macros from bottom > up: when I started coding extensions I found them not very useful > mostly because I didn't have control over them meaning "export > this symbol" or "import the symbol". Especially the DL_IMPORT > macro is strange because it seems to handle both import *and* > export depending on whether Python is compiled or not. This would be very nice. The DL_IMPORT/DL_EXPORT stuff is really weird unless you're working … [View More]

1 0