<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
Roy,
<p>For an alternative Python solution (without resorting to C) you could
try using compile() to create code objects for eval or exec rather than
using a while or for loop. On my Intel Pentium 200 machine the exec and
eval methods in the following code ran about 4% faster than the for loop.
Please note that I have only tested this on Python 2.0
<p><tt>####</tt>
<br><tt>filename = 'logfile.log'</tt><tt></tt>
<p><tt># Old FORTRAN technique. Static allocation of list length.</tt>
<br><tt># If an IndexError occurs, change the value and start again.</tt>
<br><tt>MAX_LEN = 20</tt><tt></tt>
<p><tt># pre-create a list of code objects that return list 'L' as a dictionary</tt>
<br><tt>eval_code_object_list = [None] * MAX_LEN</tt>
<br><tt>exec_code_object_list = [None] * MAX_LEN</tt>
<br><tt>for i in range(0, MAX_LEN, 2): # assume list of even length</tt>
<br><tt> text = ''</tt>
<br><tt> for j in range(0, i, 2):</tt>
<br><tt> text='%sL[%d]:L[%d],'
% (text, j, j+1)</tt>
<br><tt> text = '{%s}' % text</tt><tt></tt>
<p><tt> eval_code_object_list[i] = compile(text, '',
'eval')</tt><tt></tt>
<p><tt> text = 'dict=%s' % text</tt>
<br><tt> exec_code_object_list[i] = compile(text, '',
'exec')</tt>
<br><tt></tt> <tt></tt>
<p><tt>import time # for some stats</tt><tt></tt>
<p><tt>start_time = time.clock()</tt><tt></tt>
<p><tt># for Python 1.5.2</tt>
<br><tt>print 'Python readlines():',</tt>
<br><tt>lines = open(filename, 'rt').readlines()</tt><tt></tt>
<p><tt>print time.clock() - start_time</tt>
<br><tt>start_time = time.clock()</tt><tt></tt>
<p><tt># an alternative method for Python 2.0 (not as fast), will fail
on Python 1.5.2</tt>
<br><tt>print 'Python read().splitlines():',</tt>
<br><tt>lines = open(filename, 'rt').read().splitlines()</tt><tt></tt>
<p><tt>print time.clock() - start_time</tt><tt></tt>
<p><tt># Python 2.1 could use the 'for line in open(filename).xreadlines():'
method</tt>
<br><tt># for large files</tt><tt></tt>
<p><tt>start_time = time.clock()</tt>
<br><tt>print 'for loop:',</tt><tt></tt>
<p><tt>## The list 'L' is converted to a dictionary 'dict' using a simple
for loop.</tt>
<br><tt>for line in lines:</tt>
<br><tt> L = line.split()</tt>
<br><tt> dict = {}</tt>
<br><tt> for i in range(0, len(L), 2):</tt>
<br><tt> dict[L[i]] = L[i+1]</tt><tt></tt>
<p><tt>print time.clock() - start_time</tt>
<br><tt>start_time = time.clock()</tt>
<br><tt>print 'eval:',</tt><tt></tt>
<p><tt>## The list 'L' is converted to a dictionary 'dict' using an index
into the list</tt>
<br><tt>## of code objects. Lookups and no iteration during the creation
of 'dict'</tt>
<br><tt>for line in lines:</tt>
<br><tt> L = line.split()</tt>
<br><tt> dict = eval(eval_code_object_list[len(L)])</tt><tt></tt>
<p><tt>print time.clock() - start_time</tt>
<br><tt>start_time = time.clock()</tt>
<br><tt>print 'exec:',</tt><tt></tt>
<p><tt>## as above but use exec to produce dict</tt>
<br><tt>for line in lines:</tt>
<br><tt> L = line.split()</tt>
<br><tt> exec exec_code_object_list[len(L)]</tt>
<br><tt> # dict is here !</tt><tt></tt>
<p><tt>print time.clock() - start_time</tt><tt></tt>
<p><tt>####</tt>
<p>I normally do not bother with optimisations that give less than 10%
improvement.
<p>There has been a request for a Python equivalent of the Perl split mentioned
<br>somewhere in the Python world...
<p>Stephen D Evans
<p>Roy.Culley@switzerland.org wrote:
<blockquote TYPE=CITE>I'm new to python and am trying to convert a perl
script which analyses
<br>firewall logs to python as a learning exercise.
<p>The log files contain lines of multiple key / value pairs such as:
<p> interface qfe0 proto tcp src 1.1.1.1 dst 2.2.2.2
service smtp \
<br> s_port 44008 len 44 rule
7
<p>Not all records are the same and the key / value pairs are not at
<br>fixed positions. In perl, assuming the line is in $_, I can do:
<p> %Rec = split
<p>Is there an equivalent simple way to do this with python? I've done
<br>it by converting the data into a list and using a while loop to set
<br>the dictionary entries. However, the log files have about 4 million
<br>entries per day so I need something that is fast.
<p>Any help / suggestions most appreciated.</blockquote>
</html>