Buffer pair for lexical analysis of raw binary data

[Angus Rodgers]

On 28 Jun 2009 08:00:23 -0700, aahz at pythoncraft.com (Aahz) wrote:

>In article <0qec45lho8lkng4n20sb1ad4eguat675pb at 4ax.com>,
>Angus Rodgers  <twirlip at bigfoot.com> wrote:
>>
>>Partly as an educational exercise, and partly for its practical
>>benefit, I'm trying to pick up a programming project from where
>>I left off in 2001.  It implemented in slightly generalised form
>>the "buffer pair" scheme for lexical analysis described on pp.
>>88--92 of Aho et al., /Compilers: Principles, Techniques and 
>>Tools/ (1986). (I'm afraid I don't have a page reference for the
>>2007 second edition.  Presumably it's also in Knuth somewhere.)
>>
>>  [...]
>>
>>Does some Python library already provide some functionality like
>>this?  (It's enough to do it with nblocks = 2, as in Aho et al.)
>
>Not AFAIK, but there may well be something in the recipes or PyPI; have
>you tried searching them?

Searching for "buffer" at <http://pypi.python.org/pypi> (which I
didn't know about) gives quite a few hits (including reflex 0.1,
"A lightweight regex-based lexical scanner library").

By "recipes", do you mean
<http://code.activestate.com/recipes/langs/python/> (also new to me)?

There is certainly a lot of relevant code there (e.g. "Recipe 392150:
Buffered Stream with Multiple Forward-Only Readers"), which I can try
to learn from, even if I can't use it directly.

Thanks!
-- 
Angus Rodgers