Proposed PEP: Codec error handling callbacks

Walter Dörwald walter at
Thu Oct 4 20:22:01 CEST 2001

Here's a first try at a PEP for making Python's codec error
handling customizable. This has been briefly discussed before
on the I18N-SIG mailing list.

A sample implementation of the proposed feature is available
as a SourceForge patch.

    Walter Dörwald

PEP: ???
Title: Codec error handling callbacks
Version: $Revision$
Last-Modified: $Date$
Author: walter at (Walter Dörwald)
Status: Draft
Type: Standards Track
Python-Version: 2.3
Created: ??-???-2001
Post-History: 04-Oct-2001


     This PEP aims at extending Python's fixed codec error handling
     schemes with a more flexible callback based approach.

     Python currently uses a fixed error handling for codec error
     handlers. This PEP describes a mechanism which allows Python to
     use function callbacks as error handlers.  With these more flexible
     error handlers it is possible to add new functionality to existing
     codecs by e.g. providing fallback solutions or different encodings
     for cases where the standard codec mapping does not apply.


     A typical case is where a character encoding does not support
     the full range of Unicode characters.  For these cases many high
     level protocols support a way of escaping a Unicode character
     (e.g. Python itself support the \x, \u and \U convention, XML
     supports character references via &#xxxx; etc.).

     When implementing such an encoding algorithm, a problem with the
     current implementation of the encode method of Unicode objects
     becomes apparent: For determining which characters are unencodable
     by a certain encoding, every single character has to be tried,
     because encode does not provide any information about the location
     of the error(s), so

         # (1)
         us = u"xxx"
         s = us.encode(encoding)

     has to be replaced by

         # (2)
         us = u"xxx"
         v = []
         for c in us:
             except UnicodeError:
                 v.append("&#" + ord(c) + ";")
         s = "".join(v)

     This slows down encoding dramatically as now the loop through
     the string is done in Python code and no longer in C code.

     Furthermore this solution poses problems with stateful encodings.
     For example UTF16 uses a Byte Order Mark at the start of the
     encoded byte string to specify the byte order.  Using (2) with
     UTF16, results in an 8 bit string with a BOM between every

     To work around this problem, a stream writer - which keeps
     state between calls to the encoding function - has to be used:

         # (3)
         us = u"xxx"
         import codecs, cStringIO as StringIO
         writer = codecs.lookup(encoding)[3]

         v = StringIO.StringIO()
         uv = writer(v)
         for c in us:
             except UnicodeError:
                 uv.write(u"&#%d;" % ord(c))
         s = str(v)

     To compare the speed of (1) and (3) the following test script
     has been used:

         # (4)
         import time
         us = u"äa"*1000000
         encoding = "ascii"
         import codecs, cStringIO as StringIO

         t1 = time.time()

         s1 = us.encode(encoding, "replace")

         t2 = time.time()

         writer = codecs.lookup(encoding)[3]

         v = StringIO.StringIO()
         uv = writer(v)
         for c in us:
             except UnicodeError:
         s2 = v.getvalue()

         t3 = time.time()

         print "1:", t2-t1
         print "2:", t3-t2
         print "factor:", (t3-t2)/(t2-t1)

     On Linux with Python 2.1 this gives the
     following output:
         1: 0.395456075668
         2: 126.909595966
         factor: 320.919575586
     i.e. (3) is 320 times slower than (1).

Solution 1

     Add a position attribute to UnicodeError instances.

     When the encoder encounters an unencodable character
     it raises an exception that specifies the position in the
     Unicode object where the unencodable character occurs.
     The application can then reencode the Unicode object up
     to the offending character, replace the offending character
     with something appropriate and retry encoding with the
     rest of the Unicode string until encoding is finished.

     A stream writer will write everything up to the offending
     character and then raise an exception, so the application
     only has to replace the offending character and retry the rest
     of the string.


     Requires minor changes to all the encoders/stream writers.


     As the encoder has to be called multiple times, this
     won't work with stateful encoding, so a stream writer
     has to be used in all cases.

     If unencodable characters occur very often Solution 1
     will probably not be much faster than (3).  E.g. for the
     string u"a"*10000+u"ä" all the characters but the last one
     will have to be encoded twice when using an encoder
     (but only once when using a stream writer).

     This solution is specific to encoding and can't
     be extended to decoding easily.

Solution 2

     Add additional error handling names for every needed
     replacement scheme (e.g. "xmlcharrefreplace" for "&#%d;"
     or "escapereplace" for "\\x%02x" / "\\u%04x" / "\\U%08x")


     Requires minor changes to all encoders/stream writers.

     As the replacement scheme is implemented directly in
     the encoder this should be the fastest solution.


     The available replacement schemes are hardwired.
     Additional replacement schemes require that all
     encoders/decoders are changed again.  This is especially
     problematic for decoding where users might want to
     implement various error handlers for handling broken
     text files.

Solution 3

     The errors argument is no longer a string, but a callback
     function: This callback function gets passed the original
     unicode object and the position of the unencodable character
     and returns a new unicode object that should be encoded instead
     of the unencodable one. (Note that this requires that the encoder
     *must* be able to encode what is returned from the handler.  If
     not a normal UnicodeError will be raised.)

     Example code could look like this:

         us = u"xxx"

         def xmlescape(uni, pos):
             return u"&#%d;" % ord(uni[pos])

         s = us.encode(encode, xmlescape)


     This makes the error handling completely customizable. The error
     handler may choose to raise an exception or do any kind of
     replacement required.

     The interface between the encoder and the error handler can be
     designed in a way that this mechanism can be used for decoding too:
     The original 8bit string is passed to the error handler and the
     error handler returns a replacement unicode object and a
     resyncronization position where decoding should continue.


     This solutions requires changes to every C function
     that has "const char *errors" arguments. (e.g.

             const Py_UNICODE *p,
             int size,
             const char *errors)

     has to be changed to

             const Py_UNICODE *p,
             int size,
             PyObject *errors)

     (To provide backwards compatibility the PyUnicode_EncodeLatin1
     signature remains the same, a new function
     PyUnicode_EncodeLatin1Ex is introduced with the new
     signature.  PyUnicode_EncodeLatin1 simply calls the new function.)

Solution 4

     The errors argument is still a string, but this string
     is used to lookup a error handling callback function
     from a callback registry.


     No changes to the Python C API are required.  Well known
     error handling schemes could be implemented directly in
     the encoder/decoder for maximum speed.  Like solution 3
     this can be done for encoders and decoders.


     Currently all encoding/decoding functions have arguments
         const Py_UNICODE *p, int size
         const char *p, int size
     to specify the unicode characters/8bit characters to be
     encoded/decoded.  In case of a error a new unicode or str
     object has to be created from this information and passed
     to the error handler.  This has to be done either for every
     occuring error or once on the first error and the result object
     must be kept until the end of the function in case another
     error occurs. Most functions that call the codec functions
     work with unicode/str objects anyway, so they have to extract
     the const Py_UNICODE */int arguments and pass it to the functions,
     which has to reconstruct the object in case of an error.

Sample implementation

     A sample implementation is available on SourceForge [1].  This
     patch implements a combination of solutions 3 and 4, i.e. it
     is possible to pass functions *and* registered callback names
     to unicode.encode.

     The interface between the encoder and the handler has been
     extended to be able to support the same interface for encoding
     and decoding.  The encoder/decoder passes a tuple to the callback
     with the following entries:
         0: the name of the encoding
         1: the original Unicode object/the original 8bit string
         2: the position of the unencodable character/byte
         3: the reason why the character/byte can't be
            encoded/decoded as a string (e.g.
            "character not in range(128)" for encoding or
            "unexpected end of data", "invalid code byte" etc.
            for decoding)
         4: an additional object that can be used to pass
            state information to the callback. All implemented
            encoders/decoders currently pass None for this.

     The callback must return a tuple with the following info:

         0: a Unicode object which will be encoded instead
            of the offending character (for encoders).
            A Unicode object that will be used as a replacement
            for the undecodable bytes in the decoded Unicode
            object (for decoders)
         1: a position where the encoding/decoding process should
            continue after processing the replacement string.

     The patch includes several preregistered encode
     error handlers schemes with the following names:
         "strict", "ignore", "replace", "xmlcharrefreplace",
     and decode error handlers with the names:
         "strict", "ignore", "replace"

     The patch includes the other change to the C API described in
     Solution 4.  The parameters
         const Py_UNICODE *p, int size
     have been replace by
         PyObject *p
     so that all functions get the Unicode object directly
     as a PyObject * and pass this directly along to the error

     For further details see the patch on SourceForge.


     [1] Python patch #432401 "unicode encoding error callbacks"

     [2] Previous discussions on this topics in the I18N-SIG mailing list


     This document has been placed in the public domain.

Local Variables:
mode: indented-text
indent-tabs-mode: nil

More information about the Python-list mailing list