<div dir="ltr">I think I must have screwed this up, can someone point out my errors?<div><br></div><div>I worked based off Doug's code, then attempted to dictify the results to minimize lookup times in that filter function.</div><div><br></div><div>Full disclosure: I was only working based off no errors, with no knowledge of the algorithm implementation.</div><div><br></div><div>code:</div><div><br></div><div><a href="https://gist.github.com/gravesmedical/58a6b665b553c1294b56">https://gist.github.com/gravesmedical/58a6b665b553c1294b56</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 10, 2015 at 8:57 AM, Ross Heflin <span dir="ltr"><<a href="mailto:heflin.rosst@gmail.com" target="_blank">heflin.rosst@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Might be time to profile.<div>Run your similarity matrix builder with the large dataset against cProfile (or whatever works on PyPy) for some time (30 min) and see where its spending the majority of its time.<div><br></div><div>-Ross</div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Nov 9, 2015 at 7:44 PM, Lewit, Douglas <span dir="ltr"><<a href="mailto:d-lewit@neiu.edu" target="_blank">d-lewit@neiu.edu</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><font size="4">Hey guys,</font><div><font size="4"><br></font></div><div><font size="4">I need some advice on this one. I'm attaching the homework assignment so that you understand what I'm trying to do. I went as far as the construction of the Similarity Matrix, which is a matrix of Pearson correlation coefficients.</font></div><div><font size="4"><br></font></div><div><font size="4">My problem is this. u1.base (which is also attached) contains Users (first column), Items (second column), Ratings (third column) and finally the time stamp in the 4th and final column. (Just discard the 4th column. We're not using it for anything. )</font></div><div><font size="4"><br></font></div><div><font size="4">It's taking HOURS for Python to build the similarity matrix. So what I did was:</font></div><div><font size="4"><br></font></div><div><font size="4"><b>head -n 5000 u1.base > practice.base</b></font></div><div><font size="4"><br></font></div><div><font size="4">and I also downloaded the PyPy interpreter for Python 3. Then using PyPy (or pypy or whatever) I ran my program on the first ten thousand lines of data from u1.base stored in the new text file, practice.base. Not a problem!!! I still had to wait a couple minutes, but not a couple hours!!! </font></div><div><font size="4"><br></font></div><div><font size="4">Is there a way to make this program work for such a large set of data? I know my program successfully constructs the Similarity Matrix (i.e. similarity between users) for 5,000, 10,000, 20,000 and even 25,000 lines of data. But for 80,000 lines of data the program becomes very slow and overtaxes my CPU. (The fan turns on and the bottom of my laptop starts to get very hot.... a bad sign! )</font></div><div><font size="4"><br></font></div><div><font size="4">Does anyone have any recommendations? ( I'm supposed to meet with my prof on Tuesday. I may just explain the problem to him and request a smaller data set to work with. And unfortunately he knows very little about Python. He's primarily a C++ and Java programmer. )</font></div><div><font size="4"><br></font></div><div><font size="4">I appreciate the feedback. Thank you!!!</font></div><div><font size="4"><br></font></div><div><font size="4">Best,</font></div><div><font size="4"><br></font></div><div><font size="4">Douglas Lewit</font></div><div><font size="4"><br></font></div><div><font size="4"><br></font></div></div>
<br></div></div><span class="">_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org" target="_blank">Chicago@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/chicago" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/chicago</a><br>
<br></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>From the "desk" of Ross Heflin<br>phone number: (847) <23,504,826th decimal place of pi></div>
</font></span></div>
<br>_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org">Chicago@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/chicago" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/chicago</a><br>
<br></blockquote></div><br></div>