With more than 15000 records you would be better off using a relational database.<br>Although it will create more work to start with (you'll have to learn it), it will save you a lot of work in the medium and long term.<br>
<br>Almost any relational database can be accessed from python.As it is just for your own use SQLite might be the most appropiate (it has a very small footprint) but MySQL is excellent and so are many others.<br><br>To use a relational database you might think about learning SQL. It is very easy (especially if you you know any Boolean algebra) and is a language that has been used almost unchanged for decades and shows every sign of staying here for a long time. In computing it is one of the most useful things you can learn. There is a good introductory, interactive tutorial at
<a href="http://sqlcourse.com/"> http://sqlcourse.com/</a><br><br>If you feel you need another abstraction layer on top of this you could look at <a href="http://www.sqlobject.org/">SQLObject</a>.<br><br>Personally I would recommend that you start with
<a href="http://www.mysql.com">MySQL</a>. It is open source, easy to install and use, stable and fast. But with SQL motors you have lots of good choices. <br><br>Peter Jessop<br><br><br>On 12/13/06, Thomas <<a href="mailto:tavspam@gmail.com">
tavspam@gmail.com</a>> wrote:<br>> I'm writing a program to analyse the profiles of the 15500 users of my<br>> forum. I have the profiles as html files stored locally and I'm using<br>> ClientForm to extract the various details from the html form in each
<br>> file.<br>> <br>> My goal is to identify lurking spammers but also to learn how to<br>> better spot spammers by calculating statistical correlations in the<br>> data against known spammers.<br>> <br>
> I need advise with how to organise my data. There are 50 fields in<br>> each profile, some fields will be much more use than others so I<br>> though about creating say 10 files to start off with that contained<br>
> dictionaries of userid to field value. That way I'm dealing with 10 to<br>> 50 files instead of 15500.<br>> <br>> Also, I am inexperienced with using classes but eager to learn and<br>> wonder if they would be any help in this case.
<br>> <br>> Any advise much appreciated and thanks in advance,<br>> Thomas<br>> _______________________________________________<br>> Tutor maillist - <a href="mailto:Tutor@python.org">Tutor@python.org</a>
<br>> <a href="http://mail.python.org/mailman/listinfo/tutor">http://mail.python.org/mailman/listinfo/tutor</a><br>> <br>