Mailman 3 July 2009 - NumPy-Discussion

Re: [Numpy-discussion] String manipulation
by David Goldsmith 28 Jul '09

28 Jul '09

Hi, Chris. Look at this, _I'm_ answering one of _your_ questions (correctly, I hope): --- On Tue, 7/21/09, Christopher Barker <Chris.Barker(a)noaa.gov> wrote: > I don't see why: > > np.array('a string', dtype='S1') > > results in a length (1,) array, for instance. > > Actually, I think I do -- numpy is treating the string as a > single > scalar, rather than a sequence of characters, and doing its > best to > convert that scaler to a length one string. However, I Well, as for why "[it's doing] its best to convert that scalar to a length one string," that's because you used dtype='S1' instead of dtype='S8': >>> np.array('a string', dtype='S1') array('a', dtype='|S1') >>> np.array('a string', dtype='S8') array('a string', dtype='|S8') but as for shape, I can't reproduce your result at all: >>> np.array('a string', dtype='S1').shape () >>> np.array('a string', dtype='S8').shape () and as for size, yes, in both cases it is treating the entire string as a single "element": >>> np.array('a string', dtype='S1').size 1 >>> np.array('a string', dtype='S8').size 1 > don't know if > there is a compelling reason why it should do that -- in > other contexts, > Python generally treats strings as a sequence of > characters. I see your point (global consistency) but personally, IMO, if you want that kind of string behavior, work in the Python namesapace: I like (though I've yet to actually use it) options (if you want to process a single string array element in the "standard" way, extract it from the array first w/ an index - it's only a few more characters of code). :-) DG > -Chris > > > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R > (206) 526-6959 voice > 7600 Sand Point Way NE (206) > 526-6329 fax > Seattle, WA 98115 > (206) 526-6317 main > reception > > Chris.Barker(a)noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

4 6

Installed C libraries and using core math library
by David Cournapeau 27 Jul '09

27 Jul '09

Hi, I have recently integrated my work on numpy.distutils to build so called "installable" C libraries, that is pure C libraries which can be installed and reused by 3rd parties. The general documentation is in the distutils section: http://docs.scipy.org/doc/numpy/reference/distutils.html#building-installab… Example to reuse the math library is given in the core math library section in the C api chapter: http://docs.scipy.org/doc/numpy/reference/c-api.coremath.html#linking-again… cheers, David

1 0

Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB
by Kim Hansen 27 Jul '09

27 Jul '09

OS. Win XP SP3, 32 bits Python: 2.5.4 Numpy: 1.3.0 I have am having some major problems converting a 750 MB recarray into a 850 MB recarray To save RAM I would like to use a read-only and a writeable memap for the two recarrays during the conversion. So I do something like: import os from stat import ST_SIZE import numpy as np ... records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype, mode="r").view(np.recarray) result = np.memmap(result_path, dtype = result_dtype, mode = "w+", shape=(records,)) The code manages to create the toconvert memmap (750 MB), but when trying to create the second memap object I get File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) WindowsError: [Error 8] Not enough storage is available to process this command By tracing before and after, I can see the file size is zero before calling mmap.mmap and has the expected 850 MB size after the WindowsError has been thrown somewhere inside mmap.mmap. There is 26 GB of free disc space, so the error message seems wrong? If I comment out the creation of the first memmap, I can successfully create the result memmap, so the error seems to be related to the accumulated size of all mmap.mmaps generated. I have other cases with somewhat smaller files to convert where the transition is OK. It seem like I begin to get these problems when the accumulated size of memmaps exceeds 1GB I am surprised by this, as http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#memory-mapped… mentions there are upper bounds to the size when using versions of python before 2.5. >From this I had the impression that there was no size limit as long as you were using ver. >=2.5 (as I am) Is it due to the 32 bit OS I am using? Is there anything I can do to resolve the problem? Best wishes, Kim

7 18

Why does assert_array_almost_equal sometimes raise ValueError instead of AssertionError ?
by David Cournapeau 27 Jul '09

27 Jul '09

Hi, In some cases, some of the testing functions assert_array_* raise a ValueError instead of AssertionError: >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.nan]), np.array([1, 2, 3])) # raises ValueError >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.inf]), np.array([1, 2, 3])) # raises AssertionError This seems at least inconsistent - is there a rationale or should this be considered as a bug ? cheers, David

2 1

Re: [Numpy-discussion] fft issues
by David Goldsmith 27 Jul '09

27 Jul '09

--- On Sun, 7/26/09, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > 3. For the hfft/ihfft functions, I am unsure what this > means: "a signal whose spectrum has Hermitian > symmetry". Can someone give a one-liner for checking > this condition, and give an example where signals like this > may occur? I think I can handle this one too. DG

1 0

fft issues
by Ralf Gommers 26 Jul '09

26 Jul '09

Hi, I came across a few issues in the fft module: 1. fftshift and ifftshift do not accept an integer for the `axes` argument. I opened a ticket, #1182, and attached a patch with test that fixes this. 2. The functions refft, irefft, refft2, irefft2, refftn and irefftn have been deprecated for at least three years. Isn't it time to remove them? 3. For the hfft/ihfft functions, I am unsure what this means: "a signal whose spectrum has Hermitian symmetry". Can someone give a one-liner for checking this condition, and give an example where signals like this may occur? Cheers, Ralf

1 0

ValueError: cannot convert float to NaN to integer
by Keith Goodman 25 Jul '09

25 Jul '09

On Ubuntu 9.04, python 2.6.2, numpy 1.2.1 this gives a ValueError: x = np.array([1,2,3]) x[0] = np.nan ValueError: cannot convert float to NaN to integer But on Debian squeeze, python 2.5.4, numpy 1.2.1 the assignment works (well, the float nan is convert to the int 0): x[0] = np.nan x array([0,1,2]) The numpy version numbers are the same. Is this a python issue? BTW, is there a way to determine the svn revision number of numpy when installed from binary?

2 2

Re: [Numpy-discussion] String to integer array of ASCII values
by David Goldsmith 24 Jul '09

24 Jul '09

--- On Thu, 7/23/09, Peter <numpy-discussion(a)maubp.freeserve.co.uk> wrote: > I should have guessed that one. Why isn't numpy.fromstring > listed with the > other entries in the "From existing data" section here? Maybe because it got listed in the I/O category and wasn't to be double listed? > http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html > > This looks like a simple improvement to the > documentation... Yup, that it is; corrected here: http://docs.scipy.org/numpy/docs/numpy-docs/reference/routines.array-creati… But not yet showing up at your link. :-( DG

2 1

pdf for multivariate normal function?
by per freem 24 Jul '09

24 Jul '09

hi all, i'm trying to find the function for the pdf of a multivariate normal pdf. i know that the function "multivariate_normal" can be used to sample from the multivariate normal distribution, but i just want to get the pdf for a given vector of means and a covariance matrix. is there a function to do this? thank you.

4 3

Odd nosetest error in test_umath.py
by Matthew Brett 24 Jul '09

24 Jul '09

Hi, I just got an email about a broken buildbot build. Looking at the logs, I see this: ====================================================================== FAIL: Test bug in reduceat with structured arrays copied for speed. ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/python-support/python2.5/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/numpybb/Buildbot/numpy/b14/numpy-install/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 818, in test_reduceat assert_array_almost_equal(h1, h2) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 537, in assert_array_almost_equal File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 395, in assert_array_compare AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 701.1171875, 801.1171875, 1001.1171875, 7501.1171875], dtype=float32) y: array([ 700., 800., 1000., 7500.], dtype=float32) I'm posting because I saw this bug before on one of my own machines. It is an extremely strange error that seemed to occur in the following circumstances: 1) Running via nosetests 2) With one or more other tests in the same file as the test above, where the test was a TestCase class 3) With nose 0.10.3, 0.10.4, but not 0.11 The error given seemed to be reliable, but differed in the numbers given for different versions of the test file (but stayed the same for any given version when the test was repeated) Consider this example on my own machine: mb312@fedora11:~$ nosetests sp/numpy/core/tests/test_umath.py ... (mismatch 100.0%) x: array([ 800., 900., 1100., 7600.], dtype=float32) y: array([ 700., 800., 1000., 7500.], dtype=float32) ---------------------------------------------------------------------- Ran 71 tests in 0.064s FAILED (errors=1, failures=1) (where the error is a SkipTest). Now, just running the failing test via nose: mb312@fedora11:~$ nosetests sp/numpy/core/tests/test_umath.py:test_reduceat . ---------------------------------------------------------------------- Ran 1 test in 0.001s OK I'm flagging it in case y'all run into it, and in case someone knows what it means, and whether it is a problem. It does not occur, as I mentioned, for nose 0.11 Matthew

2 2