[Tutor] a db question

[Glen Wheeler]

> Glen,
>
> I tried your suggestions, putting the parentheses after "sync" and
> getting rid of the print statement... it didn't help.  It went much
> faster with no print statement, but it blew up at exactly the same
> spot.  I carefully checked both the key value and the record (text
> string), and they were fine.  I think it may be that I only have 256M
> of RAM, and I'm getting some kind of caching problem.

  Indeed, like some sort of virtual memory problem...good old windows.

> Also, if I restart the program a couple of records before the one that
> it  blew up on, it continues fine for another 100,000 records or so,
> so it doesn't seem to be a problem with my data.
>

  That's good news.  In that case, I would suggest using multiple files -
your code looks like this...

import os, anydbm
ofile = anydbm.open("e:\\python20\\lib\\wnctrans","c")
ifile = open("e:\\python20\\lib\\wncdet","rb")
fin = ifile.fileno()
for x in range(0,783284):
   if x % 10000 == 0:
      ofile.sync
   print x
   ptr = os.lseek(fin,x * 50, 0)
   wstr = os.read(fin,50)
   if len(wstr) == 50:
      key12 = str(wstr[:12])
      item = str(wstr[12:])
      ofile[key12] = item
os.close(fin)
print "Done..."
dum = raw_input("any key")
ofile.close()

  Right?  I suppose you could try doing the same thing, but split it up into
smaller loops.  For example...

t = 0
for x in range(1000):
  t += x

  Could become...

t = 0
for y in range((1000/10), (1000+(1000/10)), (1000/10)): ## that last thing
is called the 'step'
  st = 0
  for x in range((y-(1000/10)), y):
    st += x
  t += st
  del st

  I left the range things unsimplified because although the look a little
ugly now, I hope that this way it is easier to see what is happening.  In
both cases t will have the same end value.
  Or, even better, look into the os() module and try to find stuff on memory
allocation.  I'm afraid that the multiple loops thing may just be a dead
end, so it looks like either you get thrown a bone from people more
knowledgeable than me (come on guys :) or have a bit of reading ahead of
you.  Either way, alot of fun.

  Hope that helped,
  Glen.