Mailman 3 May 2014 - NumPy-Discussion

Can dtype be set universally?
by Dinesh Vadhia 22 May '14

22 May '14

In a 64-bit environment, is it possible to universally set the dtype to 32-bit for all ints, floats etc. to avoid setting the dtype individually for each array object and calculations?

2 1

NumPy C API question
by Stefan Seefeld 22 May '14

22 May '14

Hello, I would like to expose an existing (C++) object as a NumPy array to Python. Right now I'm using PyArray_New, passing the pointer to my object's storage. It now happens that the storage point of my object may change over its lifetime, so I'd like to change the pointer that is used in the PyArrayObject. Is there any API to do this ? (I'd like to avoid allocating a new PyArrayObject, as that is presumably a costly operation.) If not, may I access (i.e., change) the "data" member of the array object, or would I risk corrupting the application state doing that ? Many thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

2 4

Re: [Numpy-discussion] Easter Egg or what I am missing here?
by Siegfried Gonzi 22 May '14

22 May '14

On 22/05/2014 00:37, numpy-discussion-request(a)scipy.org wrote: > Message: 4 Date: Wed, 21 May 2014 18:32:30 -0400 From: Warren > Weckesser <warren.weckesser(a)gmail.com> Subject: Re: [Numpy-discussion] > Easter Egg or what I am missing here? To: Discussion of Numerical > Python <numpy-discussion(a)scipy.org> Message-ID: > <CAGzF1udkdAP+YD2SQy9cCA6Rm4ZzfyjJzFv0TiEEsv_DRiVvew(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 On 5/21/14, Siegfried Gonzi > <siegfried.gonzi(a)ed.ac.uk> wrote: >> >Please would anyone tell me the following is an undocumented bug >> >otherwise I will lose faith in everything: >> > >> >== >> >import numpy as np >> > >> > >> >years = [2004,2005,2006,2007] >> > >> >dates = [20040501,20050601,20060801,20071001] >> > >> >for x in years: >> > >> > print 'year ',x >> > >> > xy = np.array([x*1.0e-4 for x in dates]).astype(np.int) >> > >> > print 'year ',x >> >== >> > >> >Or is this a recipe to blow up a power plant? >> > > This is a "wart" of Python 2.x. The dummy variable used in a list > comprehension remains defined with its final value in the enclosing > scope. For example, this is Python 2.7: > >>>> >>>x = 100 >>>> >>>w = [x*x for x in range(4)] >>>> >>>x > 3 > > > This behavior has been changed in Python 3. Here's the same sequence > in Python 3.4: > >>>> >>>x = 100 >>>> >>>w = [x*x for x in range(4)] >>>> >>>x > 100 > > > Guido van Rossum gives a summary of this issue near the end of this > blog:http://python-history.blogspot.com/2010/06/from-list-comprehensions-to… > > Warren > > > [I still do not know how to properly use the reply function here. I apologise.] Hi all and thanks to all the respondes. I think I would have expected my code to be behaving like you said version 3.4 will do. I would never have thought 'x' is being changed during execution. I took me nearly 2 hours in my code to figure out what was going on (it was a lenghty piece of code an not so easy to spot). Siegfried -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

Find Daily max - create lists using date and add hourly data to that list for the day
by questions anon 22 May '14

22 May '14

I have hourly 2D temperature data in a monthly netcdf and I would like to find the daily maximum temperature. The shape of the netcdf is (744, 106, 193) I would like to use the year-month-day as a new list name (i.e. 2009-03-01, 2009-03-02....2009-03-31) and then add each of the hours worth of temperature data to each corresponding list. Therefore each new list should contain 24 hours worth of data and the shape should be (24,106,193) . This is the part I cannot seem to get to work. I am using datetime and then groupby to group by date but I am not sure how to use the output to make a new list name and then add the data for that day into that list. see below and attached for my latest attempt. Any feedback will be greatly appreciated. from netCDF4 import Dataset import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.basemap import Basemap from netcdftime import utime from datetime import datetime as dt import os import gc from numpy import * import pytz from itertools import groupby MainFolder=r"/DATA/2009/03" dailydate=[] alltime=[] lists={} ncvariablename='T_SFC' for (path, dirs, files) in os.walk(MainFolder): for ncfile in files: print ncfile fileext='.nc' if ncfile.endswith(ncvariablename+'.nc'): print "dealing with ncfiles:", path+ncfile ncfile=os.path.join(path,ncfile) ncfile=Dataset(ncfile, 'r+', 'NETCDF4') variable=ncfile.variables[ncvariablename][:,:,:] TIME=ncfile.variables['time'][:] ncfile.close() for temp, time in zip((variable[:]),(TIME[:])): cdftime=utime('seconds since 1970-01-01 00:00:00') ncfiletime=cdftime.num2date(time) timestr=str(ncfiletime) utc_dt = dt.strptime(timestr, '%Y-%m-%d %H:%M:%S') au_tz = pytz.timezone('Australia/Sydney') local_dt = utc_dt.replace(tzinfo=pytz.utc).astimezone(au_tz) alltime.append(local_dt) for k, g in groupby(alltime, key=lambda d: d.date()): kstrp_local=k.strftime('%Y-%m-%d_%H') klocal_date=k.strftime('%Y-%m-%d') dailydate.append(klocal_date) for n in dailydate: lists[n]=[] lists[n].append(temp) big_array=np.ma.concatenate(lists[n]) DailyTemp=big_array.max(axis=0)

2 2

Easter Egg or what I am missing here?
by Siegfried Gonzi 21 May '14

21 May '14

Please would anyone tell me the following is an undocumented bug otherwise I will lose faith in everything: == import numpy as np years = [2004,2005,2006,2007] dates = [20040501,20050601,20060801,20071001] for x in years: print 'year ',x xy = np.array([x*1.0e-4 for x in dates]).astype(np.int) print 'year ',x == Or is this a recipe to blow up a power plant? Thanks, Siegfried -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

4 3

Fancy Indexing of Structured Arrays is Slow
by Dave Hirschfeld 21 May '14

21 May '14

As can be seen from the code below (or in the notebook linked beneath) fancy indexing of a structured array is twice as slow as indexing both fields independently - making it 4x slower? I found that fancy indexing was a bottleneck in my application so I was hoping to reduce the overhead by combining the arrays into a structured array and only doing one indexing operation. Unfortunately that doubled the time that it took! Is there any reason for this? If not, I'm happy to open an enhancement issue on GitHub - just let me know. Thanks, Dave In [32]: nrows, ncols = 365, 10000 In [33]: items = np.rec.fromarrays(randn(2,nrows, ncols), names= ['widgets','gadgets']) In [34]: row_idx = randint(0, nrows, ncols) ...: col_idx = np.arange(ncols) In [35]: %timeit filtered_items = items[row_idx, col_idx] 100 loops, best of 3: 3.45 ms per loop In [36]: %%timeit ...: widgets = items['widgets'][row_idx, col_idx] ...: gadgets = items['gadgets'][row_idx, col_idx] ...: 1000 loops, best of 3: 1.57 ms per loop http://nbviewer.ipython.org/urls/gist.githubusercontent.com/dhirschfeld/98b9 970fb68adf23dfea/raw/10c0f968ea1489f0a24da80d3af30de7106848ac/Slow%20Structu red%20Array%20Indexing.ipynb https://gist.github.com/dhirschfeld/98b9970fb68adf23dfea

3 7

Inverse function of numpy.polyval()
by Yuxiang Wang 20 May '14

20 May '14

Dear all, I was wondering is there a convenient inverse function of np.polyval(), where I give the y value and it solves for x? I know one way I could do this is: import numpy as np # Set up the question p = np.array([1, 1, -10]) y = 100 # Solve p_temp = p p_temp[-1] -= y x = np.roots(p_temp) However my guess is most would agree on that this code has low readability. Any suggestions? Thanks! -Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj(a)virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/

2 1

ANN: PyViennaCL 1.0.3 -- very easy GPGPU linear algebra
by Toby St Clere Smithe 19 May '14

19 May '14

Hello everybody, I am pleased to announce the 1.0.3 release of PyViennaCL! This release fixes a number of important bugs, and improves performance on nVidia Kepler GPUs. The ChangeLog is below, and the associated ViennaCL version is 1.5.2. About PyViennaCL ================ *PyViennaCL* aims to make fast, powerful GPGPU and heterogeneous scientific computing really transparently easy, especially for users already using NumPy for representing matrices. PyViennaCL does this by harnessing the `ViennaCL <http://viennacl.sourceforge.net/>`_ linear algebra and numerical computation library for GPGPU and heterogeneous systems, thereby making available to Python programmers ViennaCL’s fast *OpenCL* and *CUDA* algorithms. PyViennaCL does this in a way that is idiomatic and compatible with the Python community’s most popular scientific packages, *NumPy* and *SciPy*. PyViennaCL exposes the following functionality: * sparse (compressed, co-ordinate, ELL, and hybrid) and dense (row-major and column-major) matrices, vectors and scalars on your compute device using OpenCL; * standard arithmetic operations and mathematical functions; * fast matrix products for sparse and dense matrices, and inner and outer products for vectors; * direct solvers for dense triangular systems; * iterative solvers for sparse and dense systems, using the BiCGStab, CG, and GMRES algorithms; * iterative algorithms for eigenvalue estimation problems. PyViennaCL has also been designed for straightforward use in the context of NumPy and SciPy: PyViennaCL objects can be constructed using NumPy arrays, and arithmetic operations and comparisons in PyViennaCL are type-agnostic. See the following link for documentation and example code: http://viennacl.sourceforge.net/pyviennacl/doc/ Get PyViennaCL ============== PyViennaCL is easily installed from PyPI. If you are on Windows, there are binaries for Python versions 2.7, 3.2, 3.3, and 3.4. If you are on Mac OS X and want to provide binaries, then please get in touch! Otherwise, the installation process will build PyViennaCL from source, which can take a while. If you are on Debian or Ubuntu, binaries are available in Debian testing and unstable, and Ubuntu utopic. Just run:: apt-get install python-pyviennacl python3-pyviennacl To install PyViennaCL from PyPI, make sure you've got a recent version of the *pip* package manager, and run:: pip install pyviennacl Bugs and support ================ If you find a problem in PyViennaCL, then please report it at https://github.com/viennacl/pyviennacl-dev/issues ChangeLog ========= 2014-05-15 Toby St Clere Smithe <pyviennacl(a)tsmithe.net> * Release 1.0.3. * Update external/viennacl-dev to version 1.5.2. [91b7589a8fccc92927306e0ae3e061d85ac1ae93] This contains two important fixes: one for a build failure on Windows (PyViennaCL issue #17) relating to the re-enabling of the Lanczos algorithm in 1.0.2, and one for an issue relating to missing support for matrix transposition in the ViennaCL scheduler (PyViennaCL issue #19, ViennaCL issue #73). This release is also benefitial for performance on nVidia Kepler GPUs, increasing the performance of matrix-matrix multiplications to 600 GFLOPs in single precision on a GeForce GTX 680. * Fix bug when using integers in matrix and vector index key [dbb1911fd788e66475f5717c1692be49d083a506] * Fix slicing of dense matrices (issue #18). [9c745710ebc2a1066c7074b6c5de61b227017cc6] * Enable test for matrix transposition [9e951103b883a3848aa2115df3edce73d347c09b] * Add non-square matrix-vector product test [21dd29cd10ebe02a96ee23c20ee55401bc6c874f] 2014-05-06 Toby St Clere Smithe <pyviennacl(a)tsmithe.net> * Release 1.0.2. * Re-enable Lanczos algorithm for eigenvalues (issue #11). [cbfb41fca3fb1f3db42fd7b3ccb8332b701d1e20] * Enable eigenvalue computations for compressed and coordinate matrices. [8ecee3b200a92ae99b72653a823c1f60e62f75dd] * Fix matrix-vector product for non-square matrices (issue #13). [bf3aa2bf91339df72b6f7561afaf8b12aad57cda] * Link against rt on Linux (issue #12). [d5784b62b353ebbfd78fe1335fd96971b5089f53] Best regards, -- Toby St Clere Smithe http://tsmithe.net

5 8

Win64 Build Fails
by David Hirschfeld 19 May '14

19 May '14

I'm trying to build numpy on win64 with msvc9 dynamically linked to mkl_rt. It seems to be finding the libraries fine but the build fails with the following error numpy\core\src\npymath\npy_math_private.h(171) : fatal error C1083: Cannot open include file: 'complex.h': No such file or directory Am I missing something from my site.cfg or otherwise doing something dumb? Thanks, Dave

1 0

building 32 bit numpy on 64 bit linux
by David Jones 19 May '14

19 May '14

I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message: "Broken toolchain: cannot link a simple C program" It get's the compile flags right, but not the linker: C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c' gcc: _configtest.c gcc -pthread _configtest.o -o _configtest _configtest.o: could not read symbols: File in wrong format collect2: ld returned 1 exit status I'm bulding it using a 32bit python build that I compiled on the same system. I tried: OPT="-m32" FOPT="-m32" python setup.py build and setarch x86_64 -B python setup.py build But with the same results. Someone worked around this by altering ccompiler.py, but I'm trying to find a cleaner solution. See:http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-v… Is there a standard way of doing this? Regards, David J.

3 13