[Python-Dev] Re: Automatic flex interface for Python?

[Greg Ewing]

Aahz <aahz@pythoncraft.com>:

> Can someone suggest a good simple reference on the
> distinctions between parsing / lexing / tokenizing

Lexical analysis, otherwise known as "lexing" or
"tokenising", is the process of splitting the input
up into a sequence of "tokens", such as (in the case
of a programming language) identifiers, operators, 
string literals, etc.

Parsing is the next higher level in the process,
which takes the sequence of tokens and recognises
language constructs -- statements, expressions, etc.

> particularly in the context of general string processing (e.g. XML)
> rather than the arcane art of compiler technology?

The lexing and parsing part of compiler technology isn't really any
more arcane than it is for XML or anything else -- exactly the same
principles apply.

It's more a matter of how deeply you want to get into the theory. The
standard text on this stuff around here seems to be Aho, Hopcroft and
Ullman, "The Theory of Parsing, Translation and Compiling", but you
might find that a bit much if all you want to do is parse XML. It
will, however, give you a good grounding in the theory of REs, various
classes of grammar, different parsing techniques, etc., after which
writing an XML parser will seem like quite a trivial task. :-)

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+