[Python-Dev] Decoding incomplete unicode
M.-A. Lemburg
mal at egenix.com
Thu Aug 19 12:15:49 CEST 2004
Walter Dörwald wrote:
> Let's compare example uses:
>
> 1) Having feed() as part of the StreamReader API:
> ---
> s = u"???".encode("utf-8")
> r = codecs.getreader("utf-8")()
> for c in s:
> print r.feed(c)
> ---
I consider adding a .feed() method to the stream codec
bad design. .feed() is something you do on a stream, not
a codec.
> 2) Explicitely using a queue object:
> ---
> from whatever import StreamQueue
>
> s = u"???".encode("utf-8")
> q = StreamQueue()
> r = codecs.getreader("utf-8")(q)
> for c in s:
> q.write(c)
> print r.read()
> ---
This is probably how an advanced codec writer would use the APIs
to build new stream interfaces.
> 3) Using a special wrapper that implicitely creates a queue:
> ----
> from whatever import StreamQueueWrapper
> s = u"???".encode("utf-8")
> r = StreamQueueWrapper(codecs.getreader("utf-8"))
> for c in s:
> print r.feed(c)
> ----
This could be turned into something more straight forward,
e.g.
from codecs import EncodedStream
# Load data
s = u"???".encode("utf-8")
# Write to encoded stream (one byte at a time) and print
# the read output
q = EncodedStream(input_encoding="utf-8", output_encoding="unicode")
for c in s:
q.write(c)
print q.read()
# Make sure we have processed all data:
if q.has_pending_data():
raise ValueError, 'data truncated'
> I very much prefer option 1).
I prefer the above example because it's easy to read and
makes things explicit.
> "If the implementation is hard to explain, it's a bad idea."
The user usually doesn't care about the implementation, only it's
interfaces.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Aug 19 2004)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
More information about the Python-Dev
mailing list