[DB-SIG] Managing Large/Huge Data Base + First Steps

M.-A. Lemburg mal at egenix.com
Mon Jul 6 22:40:15 CEST 2009

> 2009/6/29 Nicolò Lazzaro <n.lazzaro at hotmail.it>
>>  Hi all! I'm trying to build a program that works in some way like a "life
>> evolution simulator". This "planet" should be made of - at least - 10^8
>> cells, that continuously change their state and are subject to mathematical
>> functions. I've just came to python, after realizing that PHP and SQL (of
>> what I know some basic scripting) could never fit this project, thinking of
>> moving from databases to text files.
>> So I've come to this mailing list, hoping to learn something useful for my
>> purpose and willing to know were should I look (after some beginner tutorial
>> of course) to get the right skills in this specific field.
>> Thank you for reading! I hope to find some answer, have a nice day! :-D
>> Nicolo'

Any decent modern SQL database system should be able to handle the
data of 100 million "cells". Most databases easily scale up to 1 TB
of data (ie. each cell could well use 10kB data). Beyond that it's
more a matter of the right hardware.

It's generally not a good idea to try to reinvent database technology
from scratch by e.g. using text files. Instead, you should leverage
all the features available in modern databases, such as indexes (of
various kinds), search, transactions, compression, careful caching,
etc. etc.

Marc-Andre Lemburg

Professional Python Services directly from the Source  (#1, Jul 06 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the DB-SIG mailing list