speed

[Ayose]

Hi,

On Thu, Aug 19, 2004 at 03:37:26PM +0200, Peter Kleiweg wrote:
> 
> I implemented a lexer in Pylly and compared it to the version I
> had written in Flex. Processing 219062 lines took 0.9 seconds in
> C (from Flex), and 5 minutes 54 second in Python (from Pylly), a
> ratio of 393 to 1.
> 
> Is this normal for Python, or does Flex produce better parsers
> than Pylly? I have been looking at the code produced by Flex to
> see if I could translate it to Python automaticly. But it has a
> lot of goto statements, and I haven't figured out how to
> translate those to Python efficiently.

Don't try to translate the generated code to python. Python code is
(almost) always slower than C code, because C is converted into machine
code, and Python has to be interpreted by the VM. Besides, python does a
lot of checks.

Try with PLY, <http://systems.cs.uchicago.edu/ply/>. If you have
experience with flex/yacc in C, this module should be easy to use.

You can also play with Psyco (a JIT compiler for x86) or even with
Pyrex.

But, IMHO, if you has to process very big files, don't do it with
python. Instead, write a simple C-module, which uses your Flex parser
and creates python objects with that information. It should be trivial
if you have experience with the C API. :-)

> 
> What are the average times used for text processing of Python
> compared to C?
> 

IMO, Python is a powerful language to do almost everything, but in some
cases it is bad. One of this cases is intensive computing (like parsing a
big file). Use the correct tool =)

-- 
Ayose Cazorla León
Debian GNU/Linux - setepo