Performance Problems when selecting HUGE amounts of data from MySQL dbs
gabriel_ambuehl at buz.ch
Wed Jan 9 10:25:59 EST 2002
-----BEGIN PGP SIGNED MESSAGE-----
I need to export HUGE amounts (upto 1Mio, but normally some 10000) of
say 150 byte fields from a MySQL db and in the end, I need them as
ONE giant concatenated
string. Working with mysql-python, the only way I see is
db.execute("SELECT myfield FROM table")
for record in data:
but this is way too slow. Now I wondering whether there's any other
approach I can't seem to able to find or some other trick to speed
things up (maybe there's some way to have MySQL concatenate the
itself and return only giantstring?)?
An approach for optimizing might be to select only X fields at a time
(using LIMIT) and do multiple selects, but I feel the real problem is
the concatenation, not the select itself...
Any comments, thoughts, tricks would be greatly appreciated.
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5i
-----END PGP SIGNATURE-----
More information about the Python-list