ANN: pytoken 1.0 - native 86 machine code scanner generator

ramb at ramb at
Sat Jun 14 09:50:55 CEST 2008

I am pleased to announce the 1.0 version of pytoken.

It is available here:

What is pytoken

Pytoken is a scanner generator. Given an input specification - a
bunch of regular expressions - pytoken will generate x86 machine
code that recognizes those regular expressions. Pytoken will be
most useful for programmers that want to parse complex text files.

Pytoken has separate objects for scanners and buffers.
Here is a simple example:

  import pytoken

  lexer_obj = pytoken.lexer()
  lexer_obj.add_pattern("a", 1)
  lexer_obj.add_pattern("b", 2)

  buf = pytoken.lexer_state()

  tok = lexer_obj.get_token(buf)
  assert tok == 1
  tok = lexer_obj.get_token(buf)
  assert tok == 2

Pytoken has been written in a portable fashion - it is designed to support
multiple CPU types, even though only the x86 (32 bit) is supported now.


More information about the Python-list mailing list