[Numpy-discussion] NumPy-Discussion Digest, Vol 88, Issue 1
Amira Chekir
chekir.amira at gmail.com
Wed Jan 1 10:50:11 EST 2014
On 31.12.2013 14:13, Amira Chekir wrote:
> > Hello together,
> >
> > I try to load a (large) NIfTI file (DMRI from Human Connectome Project,
> > about 1 GB) with NiBabel.
> >
> > import nibabel as nib
> > img = nib.load("dmri.nii.gz")
> > data = img.get_data()
> >
> > The program crashes during "img.get_data()" with an "MemoryError"
> > (having 4 GB of RAM in my machine).
> >
> > Any suggestions?
>
> are you using a 64 bit operating system?
> which version of numpy?
>
> assuming nibabel uses np.load under the hood you could try it with numpy
> 1.8 which reduces excess memory usage when loading compressed files.
Hi,
Thanks for your answer.
I use ubuntu 12.04 32 bits and python 2.7
I upgrade numpy to 1.8, but the error persists
I think that the problem is in gzip.py :
max_read_chunk = 10 * 1024 * 1024 # 10Mb
What do you think?
Best regards,
AMIRA
2014/1/1 <numpy-discussion-request at scipy.org>
> Send NumPy-Discussion mailing list submissions to
> numpy-discussion at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> or, via email, send a message with subject or body 'help' to
> numpy-discussion-request at scipy.org
>
> You can reach the person managing the list at
> numpy-discussion-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of NumPy-Discussion digest..."
>
>
> Today's Topics:
>
> 1. Re: proposal: min, max of complex should give warning (Ralf
> Gommers) (David Goldsmith)
> 2. Re: NumPy-Discussion Digest, Vol 87, Issue 35 (Amira Chekir)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 31 Dec 2013 11:43:49 -0800
> From: David Goldsmith <d.l.goldsmith at gmail.com>
> Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> give warning (Ralf Gommers)
> To: numpy-discussion at scipy.org
> Message-ID:
> <CAFtPsZr9FN6wwr85ScDNhqdkWZhG7C+=
> rWU6EVuBMG+mY-XJdA at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> >
> > As for your proposal, it would be good to know if adding a warning would
> > actually catch any bugs. For the truncation warning it caught several in
> > scipy and other libs IIRC.
> >
> > Ralf
> >
>
> In light of this, perhaps the pertinent unit tests should be modified (even
> if the warning suggestion isn't adopted, about which I'm neutral...but I'm
> a little surprised that there isn't a generic way to globally turn off
> specific warnings).
>
> DG
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/ac17f43e/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Wed, 1 Jan 2014 16:45:09 +0100
> From: Amira Chekir <chekir.amira at gmail.com>
> Subject: Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 87, Issue
> 35
> To: numpy-discussion at scipy.org
> Message-ID:
> <
> CAB-foYhZMYH+asXUC_SnO6bjCDSOji+d8J6tyqSvucNOv_dyiQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi,
> Thanks for your answer.
> I use ubuntu 12.04 32 bits and python 2.7
> I upgrade numpy to 1.8, but the error persists
> I think that the problem is in gzip.py :
> max_read_chunk = 10 * 1024 * 1024 # 10Mb
> What do you think?
>
> Best regards,
> AMIRA
>
>
> 2013/12/31 <numpy-discussion-request at scipy.org>
>
> > Send NumPy-Discussion mailing list submissions to
> > numpy-discussion at scipy.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > or, via email, send a message with subject or body 'help' to
> > numpy-discussion-request at scipy.org
> >
> > You can reach the person managing the list at
> > numpy-discussion-owner at scipy.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of NumPy-Discussion digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Loading large NIfTI file -> MemoryError (Amira Chekir)
> > 2. Re: Loading large NIfTI file -> MemoryError (Julian Taylor)
> > 3. Re: proposal: min, max of complex should give warning (Cera,
> > Tim)
> > 4. Re: proposal: min, max of complex should give warning
> > (Neal Becker)
> > 5. Re: proposal: min, max of complex should give warning
> > (Ralf Gommers)
> > 6. Re: proposal: min, max of complex should give warning
> > (Neal Becker)
> > 7. ANN: NumPy 1.7.2 release (Julian Taylor)
> > 8. Re: ANN: NumPy 1.7.2 release (Charles R Harris)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Tue, 31 Dec 2013 14:13:57 +0100
> > From: Amira Chekir <chekir.amira at gmail.com>
> > Subject: [Numpy-discussion] Loading large NIfTI file -> MemoryError
> > To: numpy-discussion at scipy.org
> > Message-ID:
> > <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg=
> > EQ29Zw at mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hello together,
> >
> > I try to load a (large) NIfTI file (DMRI from Human Connectome Project,
> > about 1 GB) with NiBabel.
> >
> > import nibabel as nib
> > img = nib.load("dmri.nii.gz")
> > data = img.get_data()
> >
> > The program crashes during "img.get_data()" with an "MemoryError"
> (having 4
> > GB of RAM in my machine).
> >
> > Any suggestions?
> >
> > Best regards,
> > AMIRA
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> >
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/b13969b3/attachment-0001.html
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Tue, 31 Dec 2013 14:29:42 +0100
> > From: Julian Taylor <jtaylor.debian at googlemail.com>
> > Subject: Re: [Numpy-discussion] Loading large NIfTI file ->
> > MemoryError
> > To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> > Message-ID: <52C2C6C6.6070002 at googlemail.com>
> > Content-Type: text/plain; charset=ISO-8859-1
> >
> > On 31.12.2013 14:13, Amira Chekir wrote:
> > > Hello together,
> > >
> > > I try to load a (large) NIfTI file (DMRI from Human Connectome
> Project,
> > > about 1 GB) with NiBabel.
> > >
> > > import nibabel as nib
> > > img = nib.load("dmri.nii.gz")
> > > data = img.get_data()
> > >
> > > The program crashes during "img.get_data()" with an "MemoryError"
> > > (having 4 GB of RAM in my machine).
> > >
> > > Any suggestions?
> >
> > are you using a 64 bit operating system?
> > which version of numpy?
> >
> > assuming nibabel uses np.load under the hood you could try it with numpy
> > 1.8 which reduces excess memory usage when loading compressed files.
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Tue, 31 Dec 2013 08:51:52 -0500
> > From: "Cera, Tim" <tim at cerazone.net>
> > Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> > give warning
> > To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> > Message-ID:
> > <
> > CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg at mail.gmail.com>
> > Content-Type: text/plain; charset=ISO-8859-1
> >
> > I don't work with complex numbers, but just sampling what others do:
> >
> >
> > Python: no ordering, results in TypeError
> >
> > Matlab: sorts by magnitude
> > http://www.mathworks.com/help/matlab/ref/sort.html
> >
> > R: sorts first by real, then by imaginary
> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> >
> > Numpy: sorts first by real, then by imaginary (the documentation link
> > below calls this sort 'lexicographical' which I don't think is
> > correct)
> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> >
> >
> > I would think that the Matlab sort might be more useful, but easy
> > enough by using the absolute value.
> >
> > I think what Numpy does is normal enough to not justify a warning, but
> > leave this to others because as I pointed out in the beginning I don't
> > work with complex numbers.
> >
> > Kindest regards,
> > Tim
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Tue, 31 Dec 2013 10:52:47 -0500
> > From: Neal Becker <ndbecker2 at gmail.com>
> > Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> > give warning
> > To: numpy-discussion at scipy.org
> > Message-ID: <l9up83$549$1 at ger.gmane.org>
> > Content-Type: text/plain; charset="ISO-8859-1"
> >
> > Cera, Tim wrote:
> >
> > > I don't work with complex numbers, but just sampling what others do:
> > >
> > >
> > > Python: no ordering, results in TypeError
> > >
> > > Matlab: sorts by magnitude
> > > http://www.mathworks.com/help/matlab/ref/sort.html
> > >
> > > R: sorts first by real, then by imaginary
> > > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> > >
> > > Numpy: sorts first by real, then by imaginary (the documentation link
> > > below calls this sort 'lexicographical' which I don't think is
> > > correct)
> > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> > >
> > >
> > > I would think that the Matlab sort might be more useful, but easy
> > > enough by using the absolute value.
> > >
> > > I think what Numpy does is normal enough to not justify a warning, but
> > > leave this to others because as I pointed out in the beginning I don't
> > > work with complex numbers.
> > >
> > > Kindest regards,
> > > Tim
> >
> > But I'm not proposing to change numpy's result, which I'm sure would
> raise
> > many
> > objections. I'm just asking to give a warning, because I think in most
> > cases
> > this is actually a mistake on the user's part. Just like the warning
> > currently
> > given when complex data are truncated to real part.
> >
> >
> >
> > ------------------------------
> >
> > Message: 5
> > Date: Tue, 31 Dec 2013 17:24:05 +0100
> > From: Ralf Gommers <ralf.gommers at gmail.com>
> > Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> > give warning
> > To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> > Message-ID:
> > <
> > CABL7CQh9Fc0Uh36W9p16mzAR-oYjJ7_k7rU_Dwq+eZND6YrbDA at mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com>
> wrote:
> >
> > > Cera, Tim wrote:
> > >
> > > > I don't work with complex numbers, but just sampling what others do:
> > > >
> > > >
> > > > Python: no ordering, results in TypeError
> > > >
> > > > Matlab: sorts by magnitude
> > > > http://www.mathworks.com/help/matlab/ref/sort.html
> > > >
> > > > R: sorts first by real, then by imaginary
> > > > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> > > >
> > > > Numpy: sorts first by real, then by imaginary (the documentation link
> > > > below calls this sort 'lexicographical' which I don't think is
> > > > correct)
> > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> > > >
> > > >
> > > > I would think that the Matlab sort might be more useful, but easy
> > > > enough by using the absolute value.
> > > >
> > > > I think what Numpy does is normal enough to not justify a warning,
> but
> > > > leave this to others because as I pointed out in the beginning I
> don't
> > > > work with complex numbers.
> > > >
> > > > Kindest regards,
> > > > Tim
> > >
> > > But I'm not proposing to change numpy's result, which I'm sure would
> > raise
> > > many
> > > objections. I'm just asking to give a warning, because I think in most
> > > cases
> > > this is actually a mistake on the user's part. Just like the warning
> > > currently
> > > given when complex data are truncated to real part.
> > >
> >
> > Keep in mind that warnings can be highly annoying. If you're a user who
> > uses this functionality regularly (and you know what you're doing), then
> > you're going to be very unhappy to have to wrap each function call in:
> > olderr = np.seterr(all='ignore')
> > max(...)
> > np.seterr(**olderr)
> > or in:
> > with warnings.catch_warnings():
> > warnings.filterwarnings('ignore', ...)
> > max(...)
> >
> > The actual behavior isn't documented now it looks like, so that should be
> > done. In the Notes section of max/min probably.
> >
> > As for your proposal, it would be good to know if adding a warning would
> > actually catch any bugs. For the truncation warning it caught several in
> > scipy and other libs IIRC.
> >
> > Ralf
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> >
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/add729d8/attachment-0001.html
> >
> > ------------------------------
> >
> > Message: 6
> > Date: Tue, 31 Dec 2013 11:45:08 -0500
> > From: Neal Becker <ndbecker2 at gmail.com>
> > Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> > give warning
> > To: numpy-discussion at scipy.org
> > Message-ID: <l9usa8$7tr$1 at ger.gmane.org>
> > Content-Type: text/plain; charset="ISO-8859-1"
> >
> > Ralf Gommers wrote:
> >
> > > On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com>
> > wrote:
> > >
> > >> Cera, Tim wrote:
> > >>
> > >> > I don't work with complex numbers, but just sampling what others do:
> > >> >
> > >> >
> > >> > Python: no ordering, results in TypeError
> > >> >
> > >> > Matlab: sorts by magnitude
> > >> > http://www.mathworks.com/help/matlab/ref/sort.html
> > >> >
> > >> > R: sorts first by real, then by imaginary
> > >> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> > >> >
> > >> > Numpy: sorts first by real, then by imaginary (the documentation
> link
> > >> > below calls this sort 'lexicographical' which I don't think is
> > >> > correct)
> > >> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> > >> >
> > >> >
> > >> > I would think that the Matlab sort might be more useful, but easy
> > >> > enough by using the absolute value.
> > >> >
> > >> > I think what Numpy does is normal enough to not justify a warning,
> but
> > >> > leave this to others because as I pointed out in the beginning I
> don't
> > >> > work with complex numbers.
> > >> >
> > >> > Kindest regards,
> > >> > Tim
> > >>
> > >> But I'm not proposing to change numpy's result, which I'm sure would
> > raise
> > >> many
> > >> objections. I'm just asking to give a warning, because I think in
> most
> > >> cases
> > >> this is actually a mistake on the user's part. Just like the warning
> > >> currently
> > >> given when complex data are truncated to real part.
> > >>
> > >
> > > Keep in mind that warnings can be highly annoying. If you're a user who
> > > uses this functionality regularly (and you know what you're doing),
> then
> > > you're going to be very unhappy to have to wrap each function call in:
> > > olderr = np.seterr(all='ignore')
> > > max(...)
> > > np.seterr(**olderr)
> > > or in:
> > > with warnings.catch_warnings():
> > > warnings.filterwarnings('ignore', ...)
> > > max(...)
> > >
> > > The actual behavior isn't documented now it looks like, so that should
> be
> > > done. In the Notes section of max/min probably.
> > >
> > > As for your proposal, it would be good to know if adding a warning
> would
> > > actually catch any bugs. For the truncation warning it caught several
> in
> > > scipy and other libs IIRC.
> > >
> > > Ralf
> >
> > I tripped over it yesterday, which is what prompted my suggestion.
> >
> >
> >
> > ------------------------------
> >
> > Message: 7
> > Date: Tue, 31 Dec 2013 17:57:18 +0100
> > From: Julian Taylor <jtaylor.debian at googlemail.com>
> > Subject: [Numpy-discussion] ANN: NumPy 1.7.2 release
> > To: Discussion of Numerical Python <numpy-discussion at scipy.org>,
> > SciPy Users List <scipy-user at scipy.org>, SciPy Developers
> > List
> > <scipy-dev at scipy.org>
> > Message-ID: <52C2F76E.9010509 at googlemail.com>
> > Content-Type: text/plain; charset=ISO-8859-1
> >
> > Hello,
> >
> > I'm happy to announce the of Numpy 1.7.2.
> > This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3.
> >
> > More than 42 issues were fixed, the most important issues are listed in
> > the release notes:
> > https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst
> >
> > Compared to the last release candidate four additional minor issues have
> > been fixed and compatibility with python 3.4b1 improved.
> >
> > Source tarballs, installers and release notes can be found at
> > https://sourceforge.net/projects/numpy/files/NumPy/1.7.2
> >
> > Cheers,
> > Julian Taylor
> >
> >
> > ------------------------------
> >
> > Message: 8
> > Date: Tue, 31 Dec 2013 10:47:44 -0700
> > From: Charles R Harris <charlesr.harris at gmail.com>
> > Subject: Re: [Numpy-discussion] ANN: NumPy 1.7.2 release
> > To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> > Message-ID:
> > <CAB6mnxJBWn+FM25=
> > abrqm4DNRG7f6-1keU_hPd253O64d0-Yhw at mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > On Tue, Dec 31, 2013 at 9:57 AM, Julian Taylor <
> > jtaylor.debian at googlemail.com> wrote:
> >
> > > Hello,
> > >
> > > I'm happy to announce the of Numpy 1.7.2.
> > > This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 -
> 3.3.
> > >
> > > More than 42 issues were fixed, the most important issues are listed in
> > > the release notes:
> > > https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst
> > >
> > > Compared to the last release candidate four additional minor issues
> have
> > > been fixed and compatibility with python 3.4b1 improved.
> > >
> > > Source tarballs, installers and release notes can be found at
> > > https://sourceforge.net/projects/numpy/files/NumPy/1.7.2
> > >
> > >
> > Congrats on the release.
> >
> > Chuck
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> >
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/946abcb9/attachment.html
> >
> > ------------------------------
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > End of NumPy-Discussion Digest, Vol 87, Issue 35
> > ************************************************
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140101/279def51/attachment.html
>
> ------------------------------
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> End of NumPy-Discussion Digest, Vol 88, Issue 1
> ***********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140101/c62292f1/attachment.html>
More information about the NumPy-Discussion
mailing list