Help with large data set and fatal python error (pysqlite)

[Slawomir Nowaczyk]

On Wed, 19 May 2004 17:56:37 -0400
"Vineet Jain" <vineet at eswap.com> wrote:

#> I've been able to recreate the problem with the following code segment. It
#> seems that the problem comes when you have multple db files open at the same
#> time. In my application I have many database files open at the same time.
#> I'm not using threads so all access to the db files are done sequenetially.

#> At the end of the program I get the following error:

#> Fatal Python error: deallocating None

#> This application has requested the Runtime to terminate it in an unusual
#> way.
#> Please contact the application's support team for more information.

#> If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
#> over 35 gives the above error. In my actual application, when run for a
#> larger set, the error can come in the middle of the program and is causing
#> me a lot of grief.

#> Any help would be really appreciated.

I don't know how much help would it be, but the code below is working
fine for me - for as much as "numInnerLoop = 150" and "k in range(5)".
So it would seem like a problem specific to your system.

I am using sqlite.version '0.5.0' on Windows XP. What is your
configuration?

One wild guess would be that you are reaching the system limit of open
files (assuming each connection means opening a file, which seems
likely). I recall Windows 9x having something like this, but I do not
remember details and don't know how it looks like in newer versions.

You may also want to check if you are able to open many connections
with stand-alone sqlite program. That should, at least, give you a
hint whom should you send a bug report :-) Even if there is a reason
for sqlite to fail, it should do it more gracefully.

-------------------Problem Code----------------------------
import sqlite
import gc
import sys, os

#initialize database

dbFileName      = "c:/test.db"
numRecords      = 1000

os.remove(dbFileName)
conn    = sqlite.connect(database=dbFileName,autocommit=0)
cur     = conn.cursor()
cur.execute("create table t (field1,field2,field3,field4,field5,field6,field7)")
for i in range(numRecords):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
conn.commit()
conn.close()

numInnerLoop = 50
conn         = [None]*numInnerLoop
cur          = [None]*numInnerLoop
tempStore    = [None]*numInnerLoop
qry          = "select * from t"

for k in range(1):
    for i in range(numInnerLoop):
        conn[i]   = sqlite.connect(database=dbFileName,autocommit=1)
        cur[i]    = conn[i].cursor()

        sys.stdout.write('OuterLoop: %d   Inner Loop :  %d\r' % (k, i))

        tempStore[i] = []
        cur[i].execute(qry)
        dbRowList = cur[i].fetchall()
        for j in range(numRecords):
            for l in range(6):
                tempStore[i].append(float(dbRowList[j][l]))

        dbRowList = None
        gc.collect()

for i in range(numInnerLoop):
    conn[i].close()

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( Slawomir.Nowaczyk at cs.lth.se )

My goal in life is to be the sort of person my dog thinks I am.