[Python-3000] iostack and sock2

Josiah Carlson jcarlson at uci.edu
Tue Jun 6 18:38:05 CEST 2006

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Ronald Oussoren wrote:
> > I'm slighly worried about this thread. Async I/O and "read exactly N  
> > bytes" don't really match up. I don't know about the other  mechanisms, 
> > but at least with select and poll when the system says  you can read 
> > from a file descriptor you're only guaranteed that one  call to 
> > read(2)/recv(2)/... won't block. The implementation of a  python read 
> > method that returns exactly the number of bytes that you  requested will 
> > have to call the read system call in a loop and hence  might block.
> This is one case where the callback model of async i/o may
> help. If there were a way to say "don't call me until you've
> got n bytes ready", the descriptor could become ready
> multiple times and multiple reads performed behind the
> scenes, then when enough bytes are there, your callback
> is called.

class ReadExactly:
    def __init__(self, callback, current_count):
        self.remaining = current_count
        self.callback = callback
        self.buffer = []
    def __call__(self, data):
        while data:
            if len(data) >= self.remaining:
                b, self.buffer = self.buffer, []
                data = data[:self.remaining]
                self.remaining = 0
                self.callback(''.join(b), reader=self)
                self.remaining -= len(data)

Generally though, it's a bit easier to handle the piecewise reading, etc.,
as part of the async socket class.  The asynchat module uses,
handle_read(), collect_incoming_data(data), and found_terminator(); a
semantic I've borrowed for my own asynchronous socket classes and have
been fairly happy with.  asynchat implements the handle_read() portion,
which knows about line-terminated protocols (pop, smtp, http, ...) as
well as protocols using the 'read X bytes semantic', where X can be
fixed or variable.  (read 4 bytes, decode, read X bytes, decode, read 4...)

If the new asynchronous class had some equivalent functionality, and
some reasonable set of default behavior (overridable via subclass or
flags), you could get arbitrarily desired behavior; from "call this
thing whenever you get data", to "call this thing when you have gotten X
bytes", to "call this thing when you have found this ''line'' terminator
X", etc.

 - Josiah

More information about the Python-3000 mailing list