I just wrote something. I could not run a profiler or analyze the timing but<div>I felt it was effecient. Have a look and see if it helps:</div><div><br></div><div><div>from itertools import *</div><div><br></div><div>def sep_add(line1, line2):</div>
<div> if line1 and line2:</div><div> val1 = line1.split()</div><div> val2 = line2.split()</div><div> if (val1 and val2) and (len(val1) == len(val2) == 2):</div><div> return (val1[0] == val2[0])</div>
<div> </div><div>def add_col_files(file1, file2):</div><div> fs1 = open(file1, "r")</div><div> fs2 = open(file2, "r")</div><div><br></div><div> fsn = open("new_sample.txt", "w")</div>
<div> </div><div> # Zip the files together and find the ones that match the index</div><div> # process the tuple accordingly</div><div> # output should be what you want to write to the file</div><div> </div>
<div> for k in ifilter(lambda (i,j): (sep_add(i,j)), izip(fs1, fs2)):</div><div> if k:</div><div> output = k[0] + " " + k[1] + "\n" # sample output</div><div> fsn.write(output)</div>
<div> </div><div> fsn.close()</div><div> fs1.close()</div><div> fs2.close()</div><div><br></div><div><div>if __name__ == "__main__":</div><div> import time</div><div> start = time.localtime(time.time())</div>
<div> print time.asctime(start)</div><div> </div><div> #add_col_files("sample1.txt", "sample3.txt")</div><div><br></div><div> end = time.localtime(time.time())</div><div> print time.asctime(end)</div>
</div><div><br></div><div>It took about a minute on my comp for comparing about 50-100M sized files.</div><div>As I said not done too much of testing on this code.</div><br><div class="gmail_quote">2011/6/21 Ken Seehart <span dir="ltr"><<a href="mailto:ken@seehart.com">ken@seehart.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><u></u>
<div text="#000000" bgcolor="#ffffff"><div><div></div><div class="h5">
On 6/20/2011 10:31 PM, Ken Seehart wrote:
<blockquote type="cite">
On 6/20/2011 7:59 PM, <a href="mailto:king6cong@gmail.com" target="_blank">king6cong@gmail.com</a> wrote:
<blockquote type="cite">
<div>Hi£¬</div>
<div> I have two large files,each has more than 200000000
lines,and each line consists of two fields,one is the id and
the other a value,</div>
<div>the ids are sorted.</div>
<div><br>
</div>
<div>for example:</div>
<div><br>
</div>
<div>file1</div>
<div>(uin_a y)</div>
<div>1 10000245</div>
<div>2 12333</div>
<div>3 324543</div>
<div>5 3464565</div>
<div>....</div>
<div><br>
</div>
<div><br>
</div>
<div>file2 </div>
<div>(uin_b gift)</div>
<div>1 34545</div>
<div>3 6436466</div>
<div>4 35345646</div>
<div>5 463626</div>
<div>....</div>
<div><br>
</div>
<div>I want to merge them and get a file,the lines of which
consists of an id and the sum of the two values in file1 and
file2¡£</div>
<div>the codes are as below: </div>
<div><br>
</div>
<div>uin_y=open('file1')</div>
<div>uin_gift=open(file2')</div>
<div><br>
</div>
<div>y_line=uin_y.next()</div>
<div>gift_line=uin_gift.next()</div>
<div><br>
</div>
<div> while 1:</div>
<div> try:</div>
<div> uin_a,y=[int(i) for i in y_line.split()]</div>
<div> uin_b,gift=[int(i) for i in gift_line.split()]</div>
<div> if uin_a==uin_b:</div>
<div> score=y+gift</div>
<div> print uin_a,score</div>
<div> y_line=uin_y.next()</div>
<div> gift_line=uin_gift.next()</div>
<div> if uin_a<uin_b:</div>
<div> print uin_a,y</div>
<div> y_line=uin_y.next()</div>
<div> if uin_a>uin_b:</div>
<div> print uin_b,gift</div>
<div> gift_line=uin_gift.next()</div>
<div> except StopIteration:</div>
<div> break</div>
<div><br>
</div>
<div><br>
</div>
<div>the question is that those code runs 40+ minutes on a
server(16 core,32G mem),</div>
<div>the <span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">time complexity is O(n),and there are not too
much operations,</span></span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">I think it should be faster.So I want to ask
which part costs so much.</span></span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">I tried the cProfile module </span></span><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px">but didn't get too much.</span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px">I guess maybe it is the int() operation that cost so
much,but I'm not sure </span></div>
<div> <span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">and don't know how to solve this.</span></span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">Is there a way to avoid type convertion in
Python such as scanf in C?</span></span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif;font-size:13px;line-height:18px"><span style="word-wrap:break-word">Thanks for your help £º£©</span></span></div>
</blockquote>
<br>
Unfortunately python does not have a scanf equivalent AFAIK. Most
use cases for scanf can be handled by regular expressions, but
that would clearly useless for you, and just slow you down more
since it does not perform the int conversion for you.<br>
<br>
Your code appears to have a bug: I would expect that the last
entry will be lost unless both files end with the same index
value. Be sure to test your code on a few short test files.<br>
<br>
I recommend psyco to make the whole thing faster.<br>
<br>
Regards,<br>
Ken Seehart<br>
<br>
</blockquote></div></div>
Another thought (a bit of extra work, but you might find it
worthwhile if psyco doesn't yield a sufficient boost):<br>
<br>
Write a couple filter programs to convert to and from binary data
(pairs of 32 or 64 bit integers depending on your requirements).<br>
<br>
Modify your program to use the subprocess module to open two
instances of the binary conversion process with the two input files.
Then pipe the output of that program into the binary to text filter.<br>
<br>
This might turn out to be faster since each process would make use
of a core. Also it gives you other options, such as keeping your
data in binary form for processing, and converting to text only as
needed.<br>
<br>
Ken Seehart<br>
<br>
</div>
<br>--<br>
<a href="http://mail.python.org/mailman/listinfo/python-list" target="_blank">http://mail.python.org/mailman/listinfo/python-list</a><br>
<br></blockquote></div><br></div>