[Python-Dev] Decoding incomplete unicode

[M.-A. Lemburg]

Walter Dörwald wrote:
> M.-A. Lemburg wrote:
>> Overall, I don't like the idea of adding extra
>> APIs breaking the existing codec API.
> 
> Adding a final argument that defaults to False doesn't
> break the API for the callers, only for the implementors.
> 
> And if we drop the final argument, the API is completely
> backwards compatible both for users and implementors.
> The only thing that gets added is the feed() method,
> that implementors don't have to overwrite.
> 
>> I believe that we can
>> extend stream codecs to also work in a feed mode without
>> breaking the API.
> 
> Abandoning the final argument and adding a feed() method
> would IMHO be the simplest way to do this.
> 
> But then there's no way to make sure that every byte from
> the input stream is really consumed.

I've thought about this some more. Perhaps I'm still missing
something, but wouldn't it be possible to add a feeding
mode to the existing stream codecs by creating a new queue
data type (much like the queue you have in the test cases of
your patch) and using the stream codecs on these ?

I think such a queue would be generally useful in other
contexts as well, e.g. for implementing fast character based
pipes between threads, non-Unicode feeding parsers, etc.
Using such a type you could potentially add a feeding
mode to stream or file-object API based algorithms very
easily.

We could then have a new class, e.g. FeedReader, which
wraps the above in a nice API, much like StreamReaderWriter
and StreamRecoder.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 17 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::