Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 88, Issue 1
On 31.12.2013 14:13, Amira Chekir wrote:
Hello together,
I try to load a (large) NIfTI file (DMRI from Human Connectome Project, about 1 GB) with NiBabel.
import nibabel as nib img = nib.load("dmri.nii.gz") data = img.get_data()
The program crashes during "img.get_data()" with an "MemoryError" (having 4 GB of RAM in my machine).
Any suggestions?
are you using a 64 bit operating system? which version of numpy?
assuming nibabel uses np.load under the hood you could try it with numpy 1.8 which reduces excess memory usage when loading compressed files.
Hi, Thanks for your answer. I use ubuntu 12.04 32 bits and python 2.7 I upgrade numpy to 1.8, but the error persists I think that the problem is in gzip.py : max_read_chunk = 10 * 1024 * 1024 # 10Mb What do you think? Best regards, AMIRA 2014/1/1 <numpy-discussion-request@scipy.org>
Send NumPy-Discussion mailing list submissions to numpy-discussion@scipy.org
To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to numpy-discussion-request@scipy.org
You can reach the person managing the list at numpy-discussion-owner@scipy.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of NumPy-Discussion digest..."
Today's Topics:
1. Re: proposal: min, max of complex should give warning (Ralf Gommers) (David Goldsmith) 2. Re: NumPy-Discussion Digest, Vol 87, Issue 35 (Amira Chekir)
----------------------------------------------------------------------
Message: 1 Date: Tue, 31 Dec 2013 11:43:49 -0800 From: David Goldsmith <d.l.goldsmith@gmail.com> Subject: Re: [Numpy-discussion] proposal: min, max of complex should give warning (Ralf Gommers) To: numpy-discussion@scipy.org Message-ID: <CAFtPsZr9FN6wwr85ScDNhqdkWZhG7C+= rWU6EVuBMG+mY-XJdA@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"
As for your proposal, it would be good to know if adding a warning would actually catch any bugs. For the truncation warning it caught several in scipy and other libs IIRC.
Ralf
In light of this, perhaps the pertinent unit tests should be modified (even if the warning suggestion isn't adopted, about which I'm neutral...but I'm a little surprised that there isn't a generic way to globally turn off specific warnings).
DG
On 01.01.2014 16:50, Amira Chekir wrote:
On 31.12.2013 14:13, Amira Chekir wrote:
Hello together,
I try to load a (large) NIfTI file (DMRI from Human Connectome Project, about 1 GB) with NiBabel.
import nibabel as nib img = nib.load("dmri.nii.gz") data = img.get_data()
The program crashes during "img.get_data()" with an "MemoryError" (having 4 GB of RAM in my machine).
Any suggestions?
are you using a 64 bit operating system? which version of numpy?
assuming nibabel uses np.load under the hood you could try it with numpy 1.8 which reduces excess memory usage when loading compressed files.
Hi, Thanks for your answer. I use ubuntu 12.04 32 bits and python 2.7 I upgrade numpy to 1.8, but the error persists I think that the problem is in gzip.py : max_read_chunk = 10 * 1024 * 1024 # 10Mb What do you think?
On a 32 bit system you can only use 2GB of ram (even if you have 4GB). A single copy of your data will already exhaust this and this can be hard to avoid with numpy. Use an 64 bit operating system with more RAM or somehow try to chunk your workload into smaller sizes.
participants (2)
-
Amira Chekir -
Julian Taylor