Re: [SciPy-user] handling of huge files for post-processing
Hello David, indeed data in file a depends on data in file b... that the biggest problem and consequently I guess I need something that operates better on the file-system than in main memory. Do you think, it's possible to use PyTables to tackle the problem? I would need something that can group together such enormous data-sets. sqlite is nice to group data of a table together, but I guess my data-sets are just to big... Acutally I unfortunately don't see the possibility to iterate over the entries of the files in the manner you described below.... Thanks, Christoph ------------------------------ Message: 3 Date: Tue, 26 Feb 2008 09:17:00 -0500 From: "David Huard" <david.huard@gmail.com> Subject: Re: [SciPy-user] handling of huge files for post-processing To: "SciPy Users List" <scipy-user@scipy.org> Message-ID: <91cf711d0802260617o4d768824wbf5fae702b59f00a@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Cristoph, Do you mean that b depends on the entire dataset a ? In this case, you might consider buying additional memory; this is often way cheaper in terms of time than trying to optimize the code. What I mean by iterators is that when you open a binary file, you generally have the possibility to iterate over each element in the file. For instance, when reading an ascii file: for line in f.readline(): some operation on the current line. instead of loading all the file in memory: lines = f.readlines() This way, only one line is kept in memory at a time. If you can write your code in this manner, this might solve your memory problem. For instance, here is a generator that opens two files and will return the current line of each file each time it's next() method is called def read(): a = open('filea', 'r') b = open('fileb', 'r') la = a.readline() lb = b.readline() while (la and lb): yield la,lb la = a.readline() lb = b.readline() for a, b in read(): some operation on a,b HTH, David
Whether or not PyTables is going to make a difference really depends on how much data you need at a given time to perform the computation. If this exceeds your RAM, it doesn't matter what binary format you are using. That being said, I am not familiar with sqlite, so I don't know if there is some limitations regarding the database size. Storing your data using PyTables will allow you to store as many GB in a single file as you wish. The tricky part will then be to extract only the data that you need to perform your computations and make sure this always stays below the RAM limit, or else the swap memory will be used and it will slow down things considerably. I suggest you try to estimate how much memory you'll be needing for your computations, see how much RAM you have, and decide whether or not you should just spend some euros and install additional RAM. Servus, David 2008/2/26, Christoph Scheit <Christoph.Scheit@lstm.uni-erlangen.de>:
Hello David,
indeed data in file a depends on data in file b... that the biggest problem and consequently I guess I need something that operates better on the file-system than in main memory.
Do you think, it's possible to use PyTables to tackle the problem? I would need something that can group together such enormous data-sets. sqlite is nice to group data of a table together, but I guess my data-sets are just to big...
Acutally I unfortunately don't see the possibility to iterate over the entries of the files in the manner you described below....
Thanks,
Christoph ------------------------------
Message: 3 Date: Tue, 26 Feb 2008 09:17:00 -0500
From: "David Huard" <david.huard@gmail.com> Subject: Re: [SciPy-user] handling of huge files for post-processing To: "SciPy Users List" <scipy-user@scipy.org> Message-ID:
<91cf711d0802260617o4d768824wbf5fae702b59f00a@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Cristoph,
Do you mean that b depends on the entire dataset a ? In this case, you might consider buying additional memory; this is often way cheaper in terms of time than trying to optimize the code.
What I mean by iterators is that when you open a binary file, you generally have the possibility to iterate over each element in the file. For instance, when reading an ascii file:
for line in f.readline(): some operation on the current line.
instead of loading all the file in memory: lines = f.readlines()
This way, only one line is kept in memory at a time. If you can write your code in this manner, this might solve your memory problem. For instance, here is a generator that opens two files and will return the current line of each file each time it's next() method is called def read(): a = open('filea', 'r') b = open('fileb', 'r') la = a.readline() lb = b.readline() while (la and lb): yield la,lb la = a.readline() lb = b.readline()
for a, b in read(): some operation on a,b
HTH,
David
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Hi, Christoph, I am unclear exactly what you are really doing, are you just reading, converting, grouping and summing across files? (A small example always helps.) Based on what you have indicated, I doubt that just switching PyTables will be sufficient. Your emails suggest that a different scheme is required than what you are currently doing. It should be expected that large or many files will be resource intensive - the key is to determine which bottlenecks can be removed. Depending on what need to be done, you can process files one at a one and accumulate the results which means that you only deal with one file at a time. Alternatively you can process chunks where you only use specific chunks of the files but requires you to reread files multiple times. Regards, Bruce On Tue, Feb 26, 2008 at 11:23 AM, David Huard <david.huard@gmail.com> wrote:
Whether or not PyTables is going to make a difference really depends on how much data you need at a given time to perform the computation. If this exceeds your RAM, it doesn't matter what binary format you are using. That being said, I am not familiar with sqlite, so I don't know if there is some limitations regarding the database size.
Storing your data using PyTables will allow you to store as many GB in a single file as you wish. The tricky part will then be to extract only the data that you need to perform your computations and make sure this always stays below the RAM limit, or else the swap memory will be used and it will slow down things considerably.
I suggest you try to estimate how much memory you'll be needing for your computations, see how much RAM you have, and decide whether or not you should just spend some euros and install additional RAM.
Servus,
David
2008/2/26, Christoph Scheit <Christoph.Scheit@lstm.uni-erlangen.de>:
Hello David,
indeed data in file a depends on data in file b... that the biggest problem and consequently I guess I need something that operates better on the file-system than in main memory.
Do you think, it's possible to use PyTables to tackle the problem? I would need something that can group together such enormous data-sets. sqlite is nice to group data of a table together, but I guess my data-sets are just to big...
Acutally I unfortunately don't see the possibility to iterate over the entries of the files in the manner you described below....
Thanks,
Christoph ------------------------------
Message: 3 Date: Tue, 26 Feb 2008 09:17:00 -0500
From: "David Huard" <david.huard@gmail.com> Subject: Re: [SciPy-user] handling of huge files for post-processing To: "SciPy Users List" <scipy-user@scipy.org> Message-ID:
<91cf711d0802260617o4d768824wbf5fae702b59f00a@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Cristoph,
Do you mean that b depends on the entire dataset a ? In this case, you might consider buying additional memory; this is often way cheaper in terms of time than trying to optimize the code.
What I mean by iterators is that when you open a binary file, you generally have the possibility to iterate over each element in the file. For instance, when reading an ascii file:
for line in f.readline(): some operation on the current line.
instead of loading all the file in memory: lines = f.readlines()
This way, only one line is kept in memory at a time. If you can write your code in this manner, this might solve your memory problem. For instance, here is a generator that opens two files and will return the current line of each file each time it's next() method is called def read(): a = open('filea', 'r') b = open('fileb', 'r') la = a.readline() lb = b.readline() while (la and lb): yield la,lb la = a.readline() lb = b.readline()
for a, b in read(): some operation on a,b
HTH,
David
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
Hello All, Most likely this is really just a question for David Cournapeau. . . Am having a bit of trouble apparently resulting from being a python newbie. I have numpy, scipy, and pyaudiolab up and going, but am having trouble getting pysamplerate to happen. In attempting to install pysamplerate, I have run pysamplerate's setup.py in a python interpreter, choosing task [2], the install option. That does the following: samplerate_info: libraries samplerate not found in /Library/Frameworks/Python.framework/Versions/2.5/lib FOUND: libraries = ['samplerate'] library_dirs = ['/usr/local/lib'] fulllibloc = /usr/local/lib/libsamplerate.so.0 fullheadloc = /usr/local/include/samplerate.h include_dirs = ['/usr/local/include'] running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_py copying pysamplerate.py -> build/lib/pysamplerate running install_lib creating /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/__init__.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/generate_const.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/header_parser.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/info.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/pysamplerate.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/setup.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/__init__.py to __init__.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/generate_const.py to generate_const.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/header_parser.py to header_parser.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/info.py to info.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/pysamplerate.py to pysamplerate.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/setup.py to setup.pyc running install_egg_info Writing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate-0.1-py2.5.egg-info **************************************** The latest version of libsamplerate has been installed in /usr/local/lib/, with ls libsamplerate* listing the following: libsamplerate.0.1.1.dylib libsamplerate.dylib libsamplerate.0.dylib libsamplerate.la libsamplerate.a Starting a new python interpreter, attempting to import pysamplerate, i get:
import pysamplerate Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pysamplerate.py", line 23, in <module> _src = cdll.LoadLibrary('/usr/local/lib/libsamplerate.so.0') File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/__init__.py", line 423, in LoadLibrary return self._dlltype(name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/__init__.py", line 340, in __init__ self._handle = _dlopen(self._name, mode) OSError: dlopen(/usr/local/lib/libsamplerate.so.0, 6): image not found
**************************************** What I see is that libsamplerate.so.0 is missing from the /usr/local/lib directory. Is this a file that should be created by pysamplerate's setup.py? Anyway, I'm doing something wrong. Sure it is rather simple. Thanks for the help. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Joseph Anderson Lecturer in Music School of Arts and New Media University of Hull, Scarborough Campus, Scarborough, North Yorkshire, YO11 3AZ, UK T: +44.(0)1723.357341 T: +44.(0)1723.357370 F: +44.(0)1723.350815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ***************************************************************************************** To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html *****************************************************************************************
Hello All, Don't know if this went through the first time or not. . . -----Original Message----- From: scipy-user-bounces@scipy.org on behalf of Joseph Anderson Sent: Sat 03/01/2008 3:12 PM To: SciPy Users List Subject: Newbie help for installing pysamplerate Hello All, Most likely this is really just a question for David Cournapeau. . . Am having a bit of trouble apparently resulting from being a python newbie. I have numpy, scipy, and pyaudiolab up and going, but am having trouble getting pysamplerate to happen. In attempting to install pysamplerate, I have run pysamplerate's setup.py in a python interpreter, choosing task [2], the install option. That does the following: samplerate_info: libraries samplerate not found in /Library/Frameworks/Python.framework/Versions/2.5/lib FOUND: libraries = ['samplerate'] library_dirs = ['/usr/local/lib'] fulllibloc = /usr/local/lib/libsamplerate.so.0 fullheadloc = /usr/local/include/samplerate.h include_dirs = ['/usr/local/include'] running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_py copying pysamplerate.py -> build/lib/pysamplerate running install_lib creating /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/__init__.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/generate_const.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/header_parser.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/info.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/pysamplerate.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate copying build/lib/pysamplerate/setup.py -> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/__init__.py to __init__.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/generate_const.py to generate_const.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/header_parser.py to header_parser.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/info.py to info.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/pysamplerate.py to pysamplerate.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate/setup.py to setup.pyc running install_egg_info Writing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pysamplerate-0.1-py2.5.egg-info **************************************** The latest version of libsamplerate has been installed in /usr/local/lib/, with ls libsamplerate* listing the following: libsamplerate.0.1.1.dylib libsamplerate.dylib libsamplerate.0.dylib libsamplerate.la libsamplerate.a Starting a new python interpreter, attempting to import pysamplerate, i get:
import pysamplerate Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pysamplerate.py", line 23, in <module> _src = cdll.LoadLibrary('/usr/local/lib/libsamplerate.so.0') File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/__init__.py", line 423, in LoadLibrary return self._dlltype(name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/__init__.py", line 340, in __init__ self._handle = _dlopen(self._name, mode) OSError: dlopen(/usr/local/lib/libsamplerate.so.0, 6): image not found
**************************************** What I see is that libsamplerate.so.0 is missing from the /usr/local/lib directory. Is this a file that should be created by pysamplerate's setup.py? Anyway, I'm doing something wrong. Sure it is rather simple. Thanks for the help. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Joseph Anderson Lecturer in Music School of Arts and New Media University of Hull, Scarborough Campus, Scarborough, North Yorkshire, YO11 3AZ, UK T: +44.(0)1723.357341 T: +44.(0)1723.357370 F: +44.(0)1723.350815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ***************************************************************************************** To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html *****************************************************************************************
Joseph Anderson wrote:
What I see is that libsamplerate.so.0 is missing from the /usr/local/lib directory. Is this a file that should be created by pysamplerate's setup.py?
Anyway, I'm doing something wrong. Sure it is rather simple.
Well, pysamplerate should not try to load *.so files, it cannot work on Mac OS X. I would have believed this was a stupid mistake of mine, but I am surprised, because the system does find the libsamplerate.so.0 file, which does not seem to exist on your platform... Anyway, you are not doing anything wrong, that's a mistake of mine. Let me check tonight on my macbook to see what's going on on Mac OS X. Also, note that the most up-to-date version is available in scikits under the name samplerate (I should have mentioned it on pysamplerate webpage). cheers, David
Hello David, Ah, ok. Glad to hear it isn't exactly my newbie misconceptions causing things not to work. Thanks for having a look at what is going on. I hadn't bee aware of scikits, but found my way to: http://scipy.org/scipy/scikits/wiki Along with trying to get (py)samplerate running, I'm using pyaudiolab. (I think I have version 0.6.7.) From looking around scikits and your comments below, looks like I should go ahead and reinstall using the scikits versions--these being the current/up to date? Thanks for having a look at samplerate. I'll go ahead and have another try when you've got things going. My best, Jo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Joseph Anderson Lecturer in Music School of Arts and New Media University of Hull, Scarborough Campus, Scarborough, North Yorkshire, YO11 3AZ, UK T: +44.(0)1723.357341 T: +44.(0)1723.357370 F: +44.(0)1723.350815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -----Original Message----- From: scipy-user-bounces@scipy.org on behalf of David Cournapeau Sent: Mon 03/03/2008 12:33 PM To: SciPy Users List Subject: Re: [SciPy-user] Newbie help for installing pysamplerate Joseph Anderson wrote:
What I see is that libsamplerate.so.0 is missing from the /usr/local/lib directory. Is this a file that should be created by pysamplerate's setup.py?
Anyway, I'm doing something wrong. Sure it is rather simple.
Well, pysamplerate should not try to load *.so files, it cannot work on Mac OS X. I would have believed this was a stupid mistake of mine, but I am surprised, because the system does find the libsamplerate.so.0 file, which does not seem to exist on your platform... Anyway, you are not doing anything wrong, that's a mistake of mine. Let me check tonight on my macbook to see what's going on on Mac OS X. Also, note that the most up-to-date version is available in scikits under the name samplerate (I should have mentioned it on pysamplerate webpage). cheers, David _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user ***************************************************************************************** To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html *****************************************************************************************
Joseph Anderson wrote:
Hello David,
Ah, ok. Glad to hear it isn't exactly my newbie misconceptions causing things not to work. Thanks for having a look at what is going on.
I hadn't bee aware of scikits, but found my way to:
http://scipy.org/scipy/scikits/wiki
Along with trying to get (py)samplerate running, I'm using pyaudiolab. (I think I have version 0.6.7.) From looking around scikits and your comments below, looks like I should go ahead and reinstall using the scikits versions--these being the current/up to date?
Yes. Since their inclusion, just be aware that py prefix was dropped: pyaudiolab becomes audiolab, same for pysamplerate becoming samplerate. I did a quick hack to solve the problem on mac os X, available in revision 890. cheers, David
Great. Thanks David. I'll go ahead and try both audiolab and the updated samplerate. My best, Jo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Joseph Anderson Lecturer in Music School of Arts and New Media University of Hull, Scarborough Campus, Scarborough, North Yorkshire, YO11 3AZ, UK T: +44.(0)1723.357341 T: +44.(0)1723.357370 F: +44.(0)1723.350815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On 5/3/08 03:30, "David Cournapeau" <david@ar.media.kyoto-u.ac.jp> wrote:
Joseph Anderson wrote:
Hello David,
Ah, ok. Glad to hear it isn't exactly my newbie misconceptions causing things not to work. Thanks for having a look at what is going on.
I hadn't bee aware of scikits, but found my way to:
http://scipy.org/scipy/scikits/wiki
Along with trying to get (py)samplerate running, I'm using pyaudiolab. (I think I have version 0.6.7.) From looking around scikits and your comments below, looks like I should go ahead and reinstall using the scikits versions--these being the current/up to date?
Yes. Since their inclusion, just be aware that py prefix was dropped: pyaudiolab becomes audiolab, same for pysamplerate becoming samplerate.
I did a quick hack to solve the problem on mac os X, available in revision 890.
cheers,
David _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
***************************************************************************************** To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html *****************************************************************************************
participants (6)
-
Bruce Southey -
Christoph Scheit -
David Cournapeau -
David Huard -
Joseph Anderson -
Joseph Anderson