Mailman 3 November 1999 - Python-Dev

Another 1.6 wish
by Barry A. Warsaw 19 Nov '99

19 Nov '99

We had some discussion a while back about enabling thread support by default, if the underlying OS supports it obviously. I'd like to see that happen for 1.6. IIRC, this shouldn't be too hard -- just a few tweaks of the configure script (and who knows what for those minority platforms that don't use configure :). -Barry

4 3

Re: [Python-Dev] Codecs and StreamCodecs
by Fredrik Lundh 19 Nov '99

19 Nov '99

Andy Robinson <captainrobbo(a)yahoo.com> wrote: > So I would take flush() to mean 'shift back to > ASCII now'. if we're still talking about my "just one codec, please" proposal, that's exactly what encoder.flush should do. while decoder.flush should raise an ex- ception if you're still in double byte mode (at least if running in 'strict' mode). > Calling flush and reopen would thus "almost" get the > same data across. > > I'm trying to think if it would be dangerous. Do web > and ftp servers often call flush() in the middle of > transmitting a block of text? again, if we're talking about my proposal, these flush methods are only called by the string or stream wrappers, never by the applications. see the original post for de- tails. </F>

1 0

Re: [Python-Dev] Codecs and StreamCodecs
by Andy Robinson 19 Nov '99

19 Nov '99

--- "M.-A. Lemburg" <mal(a)lemburg.com> wrote: > Guido van Rossum wrote: > > I don't know where this came from, but a flush() > should work like > > flush() on a file. > > It came from Fredrik's proposal. > > > It doesn't return a value, it just sends any > > remaining data to the underlying stream (for > output). For input it > > shouldn't be supported at all. > > > > The idea is that flush() should do the same to the > encoder state that > > close() followed by a reopen() would do. Well, > more or less. But if > > the process were to be killed right after a > flush(), the data written > > to disk should be a complete encoding, and not > have a lingering shift > > state. > This could be useful in real life. For example, iso-2022-jp has a 'single-byte-mode' and a 'double-byte-mode' with shift-sequences to separate them. The rule is that each line in the text file or email message or whatever must begin and end in single-byte mode. So I would take flush() to mean 'shift back to ASCII now'. Calling flush and reopen would thus "almost" get the same data across. I'm trying to think if it would be dangerous. Do web and ftp servers often call flush() in the middle of transmitting a block of text? - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

1 0

Codecs and StreamCodecs
by M.-A. Lemburg 19 Nov '99

19 Nov '99

Here is a new proposal for the codec interface: class Codec: def encode(self,u,slice=None): """ Return the Unicode object u encoded as Python string. If slice is given (as slice object), only the sliced part of the Unicode object is encoded. The method may not store state in the Codec instance. Use SteamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ ... def decode(self,s,slice=None): """ Return an equivalent Unicode object for the encoded Python string s. If slice is given (as slice object), only the sliced part of the Python string is decoded and returned as Unicode object. Note that this can cause the decoding algorithm to fail due to truncations in the encoding. The method may not store state in the Codec instance. Use SteamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ ... class StreamCodec(Codec): def __init__(self,stream=None,errors='strict'): """ Creates a StreamCodec instance. stream must be a file-like object open for reading and/or writing binary data depending on the intended codec action or None. The StreamCodec may implement different error handling schemes by providing the errors argument. These parameters are known (they need not all be supported by StreamCodec subclasses): 'strict' - raise an UnicodeError (or a subclass) 'ignore' - ignore the character and continue with the next (a single character) - replace errorneous characters with the given character (may also be a Unicode character) """ self.stream = stream def write(self,u,slice=None): """ Writes the Unicode object's contents encoded to self.stream. stream must be a file-like object open for writing binary data. If slice is given (as slice object), only the sliced part of the Unicode object is written. """ ... the base class should provide a default implementation of this method using self.encode ... def read(self,length=None): """ Reads an encoded string from the stream and returns an equivalent Unicode object. If length is given, only length Unicode characters are returned (the StreamCodec instance reads as many raw bytes as needed to fulfill this requirement). Otherwise, all available data is read and decoded. """ ... the base class should provide a default implementation of this method using self.decode ... It is not required by the unicodec.register() API to provide a subclass of these base class, only the given methods must be present; this allows writing Codecs as extensions types. All Codecs must provide the .encode()/.decode() methods. Codecs having the .read() and/or .write() methods are considered to be StreamCodecs. The Unicode implementation will by itself only use the stateless .encode() and .decode() methods. All other conversion have to be done by explicitly instantiating the appropriate [Stream]Codec. -- Feel free to beat on this one ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 45 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

7 27

Internationalization Toolkit
by Andy Robinson 19 Nov '99

19 Nov '99

Here are the features I'd like to see in a Python Internationalisation Toolkit. I'm very open to persuasion about APIs and how to do it, but this is roughly the functionality I would have wanted for the last year (see separate post "Internationalization Case Study"): Built-in types: --------------- "Unicode String" and "Normal String". The normal string is can hold all 256 possible byte values and is analogous to java's Byte Array - in other words an ordinary Python string. Unicode strings iterate (and are manipulated) per character, not per byte. You knew that already. To manipulate anything in a funny encoding, you convert it to Unicode, manipulate it there, then convert it back. Easy Conversions ---------------------- This is modelled on Java which I think has it right. When you construct a Unicode string, you may supply an optional encoding argument. I'm not bothered if conversion happens in a global function, a constructor method or whatever. MyUniString = ToUnicode('hello') # assumes ASCII MyUniString = ToUnicode('pretend this is Japanese', 'ShiftJIS') #specified The converse applies when converting back. The encoding designators should agree with Java. If data is encountered which is not valid for the encoding, there are several strategies, and it would be nice if they could be specified explicitly: 1. replace offending characters with a question mark 2. try to recover intelligently (possible in some cases) 3. raise an exception A 'Unicode' designator is needed which performs a dummy conversion. File Opening: --------------- It should be possible to work with files as we do now - just streams of binary data. It should also be possible to read, say, a file of locally endoded addresses into a Unicode string. e.g. open(myfile, 'r', 'ShiftJIS'). It should also be possible to open a raw Unicode file and read the bytes into ordinary Python strings, or Unicode strings. In this case one needs to watch out for the byte-order marks at the beginning of the file. Not sure of a good API to do this. We could have OrdinaryFile objects and UnicodeFile objects, or proliferate the arguments to 'open. Doing the Conversions ---------------------------- All conversions should go through Unicode as the central point. Here is where we can start to define the territory. Some conversions are algorithmic, some are lookups, many are a mixture with some simple state transitions (e.g. shift characters to denote switches from double-byte to single-byte). I'd like to see an 'encoding engine' modelled on something like mxTextTools - a state machine with a few simple actions, effectively a mini-language for doing simple operations. Then a new encoding can be added in a data-driven way, and still go at C-like speeds. Making this open and extensible (and preferably not needing to code C to do it) is the only way I can see to get a really good solid encodings library. Not all encodings need go in the standard distribution, but all should be downloadable from www.python.org. A generalized two-byte-to-two-byte mapping is 128kb. But there are compact forms which can reduce these to a few kb, and also make the data intelligible. It is obviously desirable to store stuff compactly if we can unpack it fast. Typed Strings ---------------- When you are writing data conversion tools to sit in the middle of a bunch of databases, you could save a lot of grief with a string that knows its encoding. What follows could be done as a Python wrapper around something ordinary strings rather than as a new type, and thus need not be part of the language. This is analogous to Martin Fowler's Quantity pattern in Analysis Patterns, where a number knows its units and you cannot add dollars and pounds accidentally. These would do implicit conversions; and they would stop you assigning or confusing differently encoded strings. They would also validate when constructed. 'Typecasting' would be allowed but would require explicit code. So maybe something like... >>>ts1 = TypedString('hello', 'cp932ms') # specify encoding, it remembers it >>>ts2 = TypedString('goodbye','cp5035') >>>ts1 + ts2 #or any of a host of other encoding options EncodingError >>>ts3 = TypedString(ts1, 'cp5035') #converts it implicitly going via Unicode >>>ts4 = ts1.cast('ShiftJIS') #the developer knows that in this case the string is compatible. Going Deeper ---------------- The project I describe involved many more issues than just a straight conversion. I envisage an encodings package or module which power users could get at directly. We have be able to answer the questions: 'is string X a valid instance of encoding Y?' 'is string X nearly a valid instance of encoding Y, maybe with a little corruption, or is it something totally different?' - this one might be a task left to a programmer, but the toolkit should help where it can. 'can string X be converted from encoding Y to encoding Z without loss of data? If not, exactly what will get trashed' ? This is a really useful utility. More generally, I want tools to reason about character sets and encodings. I have 'Character Set' and 'Character Mapping' classes - very app-specific and proprietary - which let me express and answer questions about whether one character set is a superset of another and reason about round trips. I'd like to do these properly for the toolkit. They would need some C support for speed, but I think they could still be data driven. So we could have an Endoding object which could be pickled, and we could keep a directory full of them as our database. There might actually be two encoding objects - one for single-byte, one for multi-byte, with the same API. There are so many subtle differences between encodings (even within the Shift-JIS family) - company X has ten extra characters, and that is technically a new encoding. So it would be really useful to reason about these and say 'find me all JIS-compatible encodings', or 'report on the differences between Shift-JIS and 'cp932ms'. GUI Issues ------------- The new Pythonwin breaks somewhat on Japanese - editor windows are fine but console output is show as single-byte garbage. I will try to evaluate IDLE on a Japanese test box this week. I think these two need to work for double-byte languages for our credibility. Verifiability and printing ----------------------------- We will need to prove it all works. This means looking at text on a screen or on paper. A really wicked demo utility would be a GUI which could open files and convert encodings in an editor window or spreadsheet window, and specify conversions on copy/paste. If it could save a page as HTML (just an encoding tag and data between <PRE> tags, then we could use Netscape/IE for verification. Better still, a web server demo could convert on python.org and tag the pages appropriately - browsers support most common encodings. All the encoding stuff is ultimately a bit meaningless without a way to display a character. I am hoping that PDF and PDFgen may add a lot of value here. Adobe (and Ken Lunde) have spent years coming up with a general architecture for this stuff in PDF. Basically, the multi-byte fonts they use are encoding independent, and come with a whole bunch of mapping tables. So I can ask for the same Japanese font in any of about ten encodings - font name is a combination of face name and encoding. The font itself does the remapping. They make available downloadable font packs for Acrobat 4.0 for most languages now; these are good places to raid for building encoding databases. It also means that I can write a Python script to crank out beautiful-looking code page charts for all of our encodings from the database, and input and output to regression tests. I've done it for Shift-JIS at Fidelity, and would have to rewrite it once I am out of here. But I think that some good graphic design here would lead to a product that blows people away - an encodings library that can print out its own contents for viewing and thus help demonstrate its own correctness (or make errors stick out like a sore thumb). Am I mad? Have I put you off forever? What I outline above would be a serious project needing months of work; I'd be really happy to take a role, if we could find sponsors for the project. But I believe we could define the standard for years to come. Furthermore, it would go a long way to making Python the corporate choice for data cleaning and transformation - territory I think we should own. Regards, Andy Robinson Robinson Analytics Ltd. ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

15 122

Re: [Python-Dev] Some thoughts on the codecs...
by Jack Jansen 18 Nov '99

18 Nov '99

> I would propose to only add some very basic encodings to > the standard distribution, e.g. the ones mentioned under > Standard Codecs in the proposal: > > 'utf-8': 8-bit variable length encoding > 'utf-16': 16-bit variable length encoding (litte/big endian) > 'utf-16-le': utf-16 but explicitly little endian > 'utf-16-be': utf-16 but explicitly big endian > 'ascii': 7-bit ASCII codepage > 'latin-1': Latin-1 codepage > 'html-entities': Latin-1 + HTML entities; > see htmlentitydefs.py from the standard Pythin Lib > 'jis' (a popular version XXX): > Japanese character encoding > 'unicode-escape': See Unicode Constructors for a definition > 'native': Dump of the Internal Format used by Python I would suggest adding the Dos, Windows and Macintosh standard 8-bit charsets (their equivalents of latin-1) too, as documents in these encoding are pretty ubiquitous. But maybe these should only be added on the respective platforms. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

7 10

RE: [Python-Dev] Some thoughts on the codecs...
by Andy Robinson 17 Nov '99

17 Nov '99

--- Mark Hammond <mhammond(a)skippinet.com.au> wrote: > Actually, I was thinking even more radically - drop > the codec registry > all together, and use modules with "well-known" > names (a slight > precedent, but Python isnt adverse to well-known > names in general) > > eg: > iso-8859-1.py: > > import unicodec > def encode(...): > ... > def decode(...): > ... > > iso-8859-2.py: > from iso-8859-1 import * > This is the simplest if each codec really is likely to be implemented in a separate module. But just look at the data! All the iso-8859 encodings need identical functionality, and just have a different mapping table with 256 elements. It would be trivial to implement these in one module. And the wide variety of Japanese encodings (mostly corporate or historical variants of the same character set) are again best treated from one code base with a bunch of mapping tables and routines to generate the variants - basically one can store the deltas. So the choice is between possibly having a lot of almost-dummy modules, or having Python modules which generate and register a logical family of encodings. I may have some time next week and will try to code up a few so we can pound on something. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

2 1

Rosette i18n API
by Andy Robinson 17 Nov '99

17 Nov '99

There is a very capable C++ library at http://rosette.basistech.com/ It is well worth looking at the things this API actually lets you do for ideas on patterns. - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

1 0

Some thoughts on the codecs...
by Andy Robinson 17 Nov '99

17 Nov '99

Some thoughts on the codecs... 1. Stream interface At the moment a codec has dump and load methods which read a (slice of a) stream into a string in memory and vice versa. As the proposal notes, this could lead to errors if you take a slice out of a stream. This is not just due to character truncation; some Asian encodings are modal and have shift-in and shift-out sequences as they move from Western single-byte characters to double-byte ones. It also seems a bit pointless to me as the source (or target) is still a Unicode string in memory. This is a real problem - a filter to convert big files between two encodings should be possible without knowledge of the particular encoding, as should one on the input/output of some server. We can still give a default implementation for single-byte encodings. What's a good API for real stream conversion? just Codec.encodeStream(infile, outfile) ? or is it more useful to feed the codec with data a chunk at a time? 2. Data driven codecs I really like codecs being objects, and believe we could build support for a lot more encodings, a lot sooner than is otherwise possible, by making them data driven rather making each one compiled C code with static mapping tables. What do people think about the approach below? First of all, the ISO8859-1 series are straight mappings to Unicode code points. So one Python script could parse these files and build the mapping table, and a very small data file could hold these encodings. A compiled helper function analogous to string.translate() could deal with most of them. Secondly, the double-byte ones involve a mixture of algorithms and data. The worst cases I know are modal encodings which need a single-byte lookup table, a double-byte lookup table, and have some very simple rules about escape sequences in between them. A simple state machine could still handle these (and the single-byte mappings above become extra-simple special cases); I could imagine feeding it a totally data-driven set of rules. Third, we can massively compress the mapping tables using a notation which just lists contiguous ranges; and very often there are relationships between encodings. For example, "cpXYZ is just like cpXYY but with an extra 'smiley' at 0XFE32". In these cases, a script can build a family of related codecs in an auditable manner. 3. What encodings to distribute? The only clean answers to this are 'almost none', or 'everything that Unicode 3.0 has a mapping for'. The latter is going to add some weight to the distribution. What are people's feelings? Do we ship any at all apart from the Unicode ones? Should new encodings be downloadable from www.python.org? Should there be an optional package outside the main distribution? Thanks, Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

12 29

RE: [Python-Dev] Unicode proposal: %-formatting ?
by Andy Robinson 17 Nov '99

17 Nov '99

--- Tim Peters <tim_one(a)email.msn.com> wrote: > I'm more baffled about what "abc" should do. I > didn't understand the t#/s# > etc arguments, and how those do or don't relate to > what str() does. On the > face of it, the idea that a gazillion and one > distinct encodings all get > lumped into "a string object" without remembering > their nature makes about > as much sense as if Python were to treat all > instances of all user-defined > classes as being of a single InstanceType type > <wink> -- except in the > latter case you at least get a __class__ attribute > to find your way home > again. Well said. When the core stuff is done, I'm going to implement a set of "TypedString" helper routines which will remember what they are encoded in and won't let you abuse them by concatenating or otherwise mixing different encodings. If you are consciously working with multi-encoding data, this higher level of abstraction is really useful. But I reckon that can be done in pure Python (just overload '%;, '+' etc. with some encoding checks). - Andy ===== Andy Robinson Robinson Analytics Ltd. ------------------ My opinions are the official policy of Robinson Analytics Ltd. They just vary from day to day. __________________________________________________ Do You Yahoo!? Bid and sell for free at http://auctions.yahoo.com

1 0