Mailman 3 December 2009 - NumPy-Discussion

Compact way of performing array math with specified result type?
by Russell E. Owen Feb. 8, 2010

Feb. 8, 2010

I often find myself doing simple math on sequences of numbers (which might or might not be numpy arrays) where I want the result (and thus the inputs) coerced to a particular data type. I'd like to be able to say: numpy.divide(seq1, seq2, dtype=float) but ufuncs don't allow on to specify a result type. So I do this instead: numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float) Is there a more compact solution (without having to create the result array first and supply it as an argument)? -- Russell

4 3

MKL with 64bit crashes
by Kashyap Ashwin Jan. 21, 2010

Jan. 21, 2010

Hello, I compiled numpy-1.3.0 from sources on Ubuntu-hardy, x86-64 (Intel) with MKL. This is my site.cfg: [mkl] # library_dirs = /opt/intel/mkl/10.0.1.014/lib/32/ library_dirs = /opt/intel/mkl/10.2.2.025/lib/em64t include_dirs = /opt/intel/mkl/10.2.2.025/include lapack_libs = mkl_lapack #mkl_libs = mkl_core, guide, mkl_gf_ilp64, mkl_def, mkl_gnu_thread, iomp5, mkl_vml_mc3 mkl_libs = guide, mkl_core, mkl_gnu_thread, iomp5, mkl_gf_ilp64, mkl_mc3, mkl_def In [2]: numpy.test() Running unit tests for numpy NumPy version 1.3.0 NumPy is installed in /opt/********************* Python version 2.5.2 (r252:60911, Jul 22 2009, 15:33:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] nose version 0.11.0 .......... MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV .. MKL ERROR: Parameter 4 was incorrect on entry to DGESV FSegmentation fault I am using gcc: gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4) Anyone having the same issues? Do I have the mkl_libs correctly configured (this seems like a black art!) Thanks, Ashwin

5 6

Repeated dot products
by T J Jan. 7, 2010

Jan. 7, 2010

Hi, Suppose I have an array of shape: (n, k, k). In this case, I have n k-by-k matrices. My goal is to compute the product of a (potentially large) user-specified selection (with replacement) of these matrices. For example, x = [0,1,2,1,3,3,2,1,3,2,1,5,3,2,3,5,2,5,3,2,1,3,5,6] says that I want to take the 0th matrix and dot it with the 1st matrix and dot that product with the 2nd matrix and dot that product with the 1st matrix again and so on... Essentially, I am looking for efficient ways of doing this. It seems like what I *need* is for dot to be a ufunc with a reduce() operator. Then I would construct an array of the matrices, as specified by the input x. For now, I am using a python loop and this is unbearable. >>> prod = np.eye(k) >>> for i in x: ... prod = dot(prod, matrices[i]) ... Is there a better way?

5 4

Announcing toydist, improving distribution and packaging situation
by David Cournapeau Jan. 4, 2010

Jan. 4, 2010

(warning, long post) Hi there, As some of you already know, the packaging and distributions of scientific python packages have been a constant source of frustration. Open source is about making it easy for anyone to use software how they see fit, and I think python packaging infrastructure has not been very successfull for people not intimately familiar with python. A few weeks ago, after Guido visited Berkeley and was told how those issues were still there for the scientific community, he wrote an email asking whether current efforts on distutils-sig will be enough (see http://aspn.activestate.com/ASPN/Mail/Message/distutils-sig/3775972). Several of us have been participating to this discussion, but I feel like the divide between current efforts on distutils-sig and us (the SciPy community) is not getting smaller. At best, their efforts will be more work for us to track the new distribute fork, and more likely, it will be all for nothing as it won't solve any deep issue. To be honest, most of what is considered on distutils-sig sounds like anti-goals to me. Instead of keeping up with the frustrating process of "improving" distutils, I think we have enough smart people and manpower in the scientific community to go with our own solution. I am convinced it is doable because R or haskell, with a much smaller community than python, managed to pull out something with is miles ahead compared to pypi. The SciPy community is hopefully big enough so that a SciPy-specific solution may reach critical mass. Ideally, I wish we had something with the following capabilities: - easy to understand tools - http-based package repository ala CRAN, which would be easy to mirror and backup (through rsync-like tools) - decoupling the building, packaging and distribution of code and data - reliable install/uninstall/query of what is installed locally - facilities for building windows/max os x binaries - making the life of OS vendors (Linux, *BSD, etc...) easier The packaging part ============== Speaking is easy, so I started coding part of this toolset, called toydist (temporary name), which I presented at Scipy India a few days ago: http://github.com/cournape/toydist/ Toydist is more or less a rip off of cabal (http://www.haskell.org/cabal/), and consist of three parts: - a core which builds a package description from a declarative file similar to cabal files. The file is almost purely declarative, and can be parsed so that no arbitrary code is executed, thus making it easy to sandbox packages builds (e.g. on a build farm). - a set of command line tools to configure, build, install, build installers (egg only for now) etc... from the declarative file - backward compatibility tools: a tool to convert existing setup.py to the new format has been written, and a tool to use distutils through the new format for backward compatibility with complex distutils extensions should be relatively easy. The core idea is to make the format just rich enough to describe most packages out there, but simple enough so interfacing it with external tools is possible and reliable. As a regular contributor to scons, I am all too aware that a build tool is a very complex beast to get right, and repeating their efforts does not make sense. Typically, I envision that complex packages such as numpy, scipy or matplotlib would use make/waf/scons for the build - in a sense, toydist is written so that writing something like numscons would be easier. OTOH, most if not all scikits should be buildable from a purely declarative file. To give you a feel of the format, here is a snippet for the grin package from Robert K. (automatically converted): Name: grin Version: 1.1.1 Summary: A grep program configured the way I like it. Description: ==== grin ==== I wrote grin to help me search directories full of source code. The venerable GNU grep_ and find_ are great tools, but they fall just a little short for my normal use cases. <snip> License: BSD Platforms: UNKNOWN Classifiers: License :: OSI Approved :: BSD License, Development Status :: 5 - Production/Stable, Environment :: Console, Intended Audience :: Developers, Operating System :: OS Independent, Programming Language :: Python, Topic :: Utilities, ExtraSourceFiles: README.txt, setup.cfg, setup.py, Library: InstallDepends: argparse, Modules: grin, Executable: grin module: grin function: grin_main Executable: grind module: grin function: grind_main Although still very much experimental at this point, toydist already makes some things much easier than with distutils/setuptools: - path customization for any target can be done easily: you can easily add an option in the file so that configure --mynewdir=value works and is accessible at every step. - making packages FHS compliant is not a PITA anymore, and the scheme can be adapted to any OS, be it traditional FHS-like unix, mac os x, windows, etc... - All the options are accessible at every step (no more distutils commands nonsense) - data files can finally be handled correctly and consistently, instead of the 5 or 6 magics methods currently available in distutils/setuptools/numpy.distutils - building eggs does not involve setuptools anymore - not much coupling between package description and build infrastructure (building extensions is actually done through distutils ATM). Repository ======== The goal here is to have something like CRAN (http://cran.r-project.org/web/views/), ideally with a build farm so that whenever anyone submits a package to our repository, it would automatically be checked, and built for windows/mac os x and maybe a few major linux distributions. One could investigate the build service from open suse to that end (http://en.opensuse.org/Build_Service), which is based on xen VM to build installers in a reproducible way. Installed package db =============== I believe that the current open source enstaller package from Enthought can be a good starting point. It is based on eggs, but eggs are only used as a distribution format (eggs are never installed as eggs AFAIK). You can easily remove packages, query installed versions, etc... Since toydist produces eggs, interoperation between toydist and enstaller should not be too difficult. What's next ? ========== At this point, I would like to ask for help and comments, in particular: - Does all this make sense, or hopelessly intractable ? - Besides the points I have mentioned, what else do you think is needed ? - There has already been some work for the scikits webportal, but I think we should bypass pypi entirely (the current philosophy of not enforcing consistent metadata does not make much sense to me, and is at the opposite of most other similar system out there). - I think a build farm for at least windows packages would be a killer feature, and enough incentive to push some people to use our new infrastructure. It would be good to have a windows guy familiar with windows sandboxing/virtualization to do something there. The people working on the opensuse build service have started working on windows support - I think being able to automatically convert most of scientific packages is a significant feature, and needs to be more robust - so anyone is welcomed to try converting existing setup.py with toydist (see toydist readme). thanks, David

11 42

Empty strings not empty?
by Matthew Brett Jan. 1, 2010

Jan. 1, 2010

Hi, I was surprised by this - should I have been? In [35]: e = np.array(['a']) In [36]: e.shape Out[36]: (1,) In [37]: e.size Out[37]: 1 In [38]: e.tostring() Out[38]: 'a' In [39]: f = np.array(['a']) In [40]: f.shape == e.shape Out[40]: True In [41]: f.size == e.size Out[41]: True In [42]: f.tostring() Out[42]: 'a' In [43]: z = np.array(['\x00']) In [44]: z.shape Out[44]: (1,) In [45]: z.size Out[45]: 1 In [46]: z Out[46]: array([''], dtype='|S1') That is, an empty string array seems to be the same as a string array with a single 0 byte, including having shape (1,) and size 1... Best, Matthew

5 14

linalg.py det of transpose
by paul Dec. 30, 2009

Dec. 30, 2009

In numpy/linalg/linalg.py, the function det(a) does a fastCopyAndTranspose of a before calling lapack. Is that necessary as det(a.T) = det(a)?

2 1

Re: [Numpy-discussion] Complex slicing and take
by Eric Emsellem Dec. 30, 2009

Dec. 30, 2009

Thanks! will try that and see how the performance varies depending on the size of my arrays. thanks again! Eric > Constructing big intermediate arrays, might not improve performance > compared to a loop > >>>> >>> np.arange(30).reshape(6,5) > array([[ 0, 1, 2, 3, 4], > [ 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14], > [15, 16, 17, 18, 19], > [20, 21, 22, 23, 24], > [25, 26, 27, 28, 29]]) > >>>> >>> np.arange(30).reshape(6,5)[np.array([[1,2,2,1]]).T,np.arange(0,3)+np.array([[0,1,2,1]]).T] > array([[ 5, 6, 7], > [11, 12, 13], > [12, 13, 14], > [ 6, 7, 8]]) > >>>> >>> np.arange(30).reshape(6,5)[np.array([[1,2,2,1]]).T,np.arange(0,3)+np.array([[0,1,2,1]]).T].sum(0) > array([34, 38, 42]) > > Josef

1 0

Re: [Numpy-discussion] Complex slicing and take
by Eric Emsellem Dec. 30, 2009

Dec. 30, 2009

Hi thanks for the tips. Unfortunately this is not what I am after. >> > ? import numpy as num >> > ? startarray = random((1000,100)) >> > ? take_sample = [1,2,5,6,1,2] >> > ? temp = num.take(startarray,take_sample,axis=1) > Would it help to make temp a 1000x4 array instead of 1000x6? Could you > do that by changing take_sample to [1,2,5,6] and multiplying columns 1 > and 2 by a factor of 2? That would slow down the construction of temp > but speed up the addition (and slicing?) in the loop below. No it wouldn't help unfortunately, because the second instance of "1,2" would have different shifts. So I cannot just count the number of occurrence of each line. From the initial 2D array, 1D lines could be extracted several times, with each time a different shift. >> > ? shift = [10,20,34,-10,22,-20] >> > ? result = num.zeros(900) ?# shorter than initial because of the shift >> > ? for i in range(len(shift)) : >> > ? ? ?result += temp[100+shift[i]:-100+shift[1]] > This looks fast to me. The slicing doesn't make a copy nor does the > addition. I've read that cython does fast indexing but I don't know if > that applies to slicing as well. I assume that shift[1] is a typo and > should be shift[i]. (yes of course the shift[1] should be shift[i]) Well this may be fast, but not fast enough. And also, starting from my 2D startarray again, it looks odd that I cannot do something like: startarray = random((1000,100)) take_sample = [1,2,5,6,1,2] shift = [10,20,34,-10,22,-20] result = num.sum(num.take(startarray,take_sample,axis=1)[100+shift:100-shift]) but of course this is nonsense because I cannot address the data this way (with "shift"). In fact I realise now that my question is simpler: how do I extract and sum 1d lines from a 2D array if I want first each line to be "shifted". So starting again now, I want a quick way to write: startarray = random((1000,6)) shift = [10,20,34,-10,22,-20] result = num.zeros(1000, dtype=float) for i in len(shift) : result += startarray[100+shift[i]:900+shift[i]] Can I write this directly with some numpy indexing without the loop in python? thanks for any tip. Eric

3 2

LA improvements (was: dot function or dot notation, matrices, arrays?)
by David Goldsmith Dec. 30, 2009

Dec. 30, 2009

Starting a new thread for this. On Tue, Dec 22, 2009 at 7:13 PM, Anne Archibald <peridot.faceted(a)gmail.com> wrote: > I think we have one major lacuna: vectorized linear algebra. If I have > to solve a whole whack of four-dimensional linear systems, right now I > need to either write a python loop and use linear algebra on them one > by one, or implement my own linear algebra. It's a frustrating lacuna, > because all the machinery is there: generalized ufuncs and LAPACK > wrappers. Somebody just needs to glue them together. I've even tried > making a start on it, but numpy's ufunc machinery and generic type > system is just too much of a pain for me to make any progress as is. Please be more specific: what (which aspects) have been "too much of a pain"? (I ask out of ignorance, not out of challenging your opinion/experience.) > I think if someone wanted to start building a low-level Again, please be more specific: what do you mean by this? (I know generally what is meant by "low level," but I'd like you to spell out a little more fully what you mean by this in this context.) > generalized > ufunc library interface to LAPACK, that would be a big improvement in > numpy/scipy's linear algebra. Pretty much everything else strikes me > as a question of notation. (Not to trivialize it: good notation makes > a tremendous difference.) Thanks, Anne. DG > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

3 8

Complex slicing and take
by Eric Emsellem Dec. 29, 2009

Dec. 29, 2009

Hi (sorry if you receive this twice, but I did not see the first post appear) I have a nagging problem which I think could be solved nicely with numpy indexing but I cannot find the solution, except by invoking a stupid loop. I would like to do this with some numpy item. Problem ======== I have a 2D array which is let's say 1000 x 100. This represents 100 1D spectra, each one containing 1000 datapoints. For example: startarray = random((1000,100)) Assuming I have a list of indices, for example: ind = [1,2,5,6,1,2] and a corresponding list of integer shifts let's say between -100 and 100: shift = [10,20,34,-10,22,-20] I would like to do the following sum result = stararray[100+shift[0]:-100+shift[0],ind[0]] + stararray[100+shift[1]:-100+shift[0],ind[0]] + ... Basically: I would like to sum some extracted 1D line after they have been shifted by some amount. Note that I would like to be able to use each line several times (with different shifts). At the moment, I am using "take" to extract from this array some of these spectra, so let's start from scratch here: import numpy as num startarray = random((1000,100)) take_sample = [1,2,5,6,1,2] temp = num.take(startarray,take_sample,axis=1) # and to do the sum after shifting I need a loop shift = [10,20,34,-10,22,-20] result = num.zeros(900) # shorter than initial because of the shift for i in range(len(shift)) : result += temp[100+shift[i]:-100+shift[1]] Is there a way to do this without invoking a loop? Is there also a FAST solution which makes the shifts at the same time than the "take" (that would then prevent the use of the "temp" array)? Of course the arrays I am using are much BIGGER than these, so I really need an efficient way to do all this. thanks! Eric -- -------------------------------------------------------------------------- Ce message a été envoyé depuis le webmail IMP (Internet Messaging Program)

2 1