Mailman 3 December 2007 - NumPy-Discussion

any better way to normalise a matrix
by devnew＠gmail.com 28 Dec '07

28 Dec '07

in my code i am trying to normalise a matrix as below mymatrix=matrix(..# items are of double type..can be negative values....) numrows,numcols=mymatrix.shape for i in range(numrows): temp=mymatrix[i].max() for j in range(numcols): mymatrix[i,j]=abs(mymatrix[i,j]/temp) # i am using abs() to make neg vals positive this way the code takes too much time to run for a case of numrows=25, and numcols=8100 etc.. being a beginner in numpy and python ,i would like to know if this can be done more efficiently..can anyone advise? thanx dn

3 6

Scipy server down...
by Fernando Perez 27 Dec '07

27 Dec '07

Any chance it might be brought back up? (I'm trying to commit the weave cleanup work Min did at the sprint...) Thanks, f

3 3

SciPy Sprint results
by Travis E. Oliphant 27 Dec '07

27 Dec '07

Hi all, We had a great Sprint at Berkeley over last weekend. Jarrod deserves a huge hand for organizing it and Fernando should be also congradulated for making the Sprint a productive communication session with a lot of different people. Going forward, there will be a relatively informal SciPy board whose purpose is to keep SciPy (and NumPy) moving forward. Currently, this board consists of (alphabetically) Eric Jones Robert Kern Jarrod Millman Travis Oliphant Our goal is to clean up SciPy and get it ready for 1.0 release over the next year or so (which will need lots of help from the community). If anybody else is interested in serving on this board, just send me email. As part of this goal, we will be having regular "sprints" as well virtual "bug-days" and "doc-days" where people who want to participate using IRC can join in and coordinate efforts. There will be at least one bug-day or doc-day every month over the next year (on the last Friday of the month). The first one is a "doc-day" which will be held Friday on December 28, 2007 (getting started on New Year's resolutions early). This doc-day will be virtual where anyone with an internet connection can join in on the scipy channel on irc.freenode.net. At least one board member will be available at each "doc-day" or "bug-day" (even if we have to recruit board members to make it happen :-) ) The recent Sprint was very helpful. Jarrod is putting together some material from the Sprint. But, I wanted to provide over-view information for those who may be interested in what happend. Summary: A lot of great discussion took place (and some fine actual coding by a few) which resulted in the following plans: Schedule ------------ * NumPy 1.0.5 in mid January * SciPy 0.7.0 in mid March to April * NumPy 1.1 by August 2008 (may slip a bit depending on what is wanted to be included) The plans below are for NumPy 1.0.5 and SciPy 0.7.0 unless otherwise noted. IO ---- * scipy.io will be gutted and what functionality remains will be placed in numpy.io. * scipy.io will be a place for file readers and writers for various data formats (data, audio, video, images, matlab, excel, etc.) * NumPy will get a standard binary file format (.npy/.npz) for arrays/groups_of_arrays. * NumPy will be trying to incorporate some of matplotlib's csv2rec and rec2csv functionality. * Pickling arrays will be discouraged (you will still be able to do it, we will just try to not make it seem that it is the "default" way to save arrays). Testing --------- * scipy unit-testing will be "nose-compliant" and therefore nose will be required to run the SciPy tests. * NumPy will still use the current testing framework but will support SciPy's desire to be nose-compliant. NumPy 1.1 tests may move to just being "nose-compliant" * The goal is to make tests easier for contributors to write. Weave --------- * weave will not move into NumPy yet, but possibly at NumPy 1.1, there could be a separate package containing all the "wrapping" support code for NumPy in a more unified fashion (if somebody is interested in this, it is a great time to jump in). Sandbox ----------- * the scipy sandbox is disappearing (except for user playgrounds) and useful code in it will be placed in other areas. Python versions -------------------- * SciPy 0.7.0 will require Python 2.4 (we can now use decorators for SciPy). * NumPy will still be useful with Python 2.3 until at least 1.1 Other discussions ---------------------- * numpy-scons will be a separate package for now for building extensions with scons (we need experience to figure out what to do with it). * fixes to repr for numpy float scalars were put in place * Thanks to Rob Falck scipy.optimize grew slsqp (sequential least-squares programming) method (allows for equality and inequality constraints). The code by Dieter Kraft was wrapped. * We will be working to coordinate efforts with William Stein (of SAGE fame) in the future. Sage developers will be coming to Austin at the end of February to do some cooperative sprinting. * Brian Granger is working on a parallel version of NumPy that is very interesting. Deprecation approaches ------------------------------- Functions in SciPy that are disappearing will be "deprecated" with appendages to the docstring to explain how to do it differently. The deprecation will issue a warning when the function is run. In the next version, the function will disappear. Once SciPy hits 1.0, the deprecation paradigm will be a bit more conservative. A lot of fabulous things are happening with SciPy. It is an exciting time to be a part of it. There are a lot of ways to jump in and participate so feel free. If there is something you think needs addressing, then please share it. We may have a simple PEP process in the future, but for now the rule is basically "working code." Best regards, -Travis O.

3 2

Re: [Numpy-discussion] non uniform random number generator
by Nadav Horesh 27 Dec '07

27 Dec '07

Lot of nonuniform distributions, see numpy.random module. Nadav. -----Original Message----- From: numpy-discussion-bounces(a)scipy.org on behalf of dikshie Sent: Thu 27-Dec-07 17:45 To: numpy-discussion(a)scipy.org Subject: [Numpy-discussion] non uniform random number generator hi, does numpy support non uniform random number generator ? regards, -dikshie- _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

1 0

non uniform random number generator
by dikshie 27 Dec '07

27 Dec '07

hi, does numpy support non uniform random number generator ? regards, -dikshie-

2 1

matrix multipln takes too much time
by devnew＠gmail.com 25 Dec '07

25 Dec '07

hi i am doing some maths calculations involving matrices of double values using numpy.matrix , java code for this is something like int items=25; int sample=5; int totalcols=8100; double[][]dblarrayone=new double[items][totalcols]; double[][]dblarraytwo=new double[items][totalcols]; //their elements are set elsewhere before calculation double[][] resultarray = new double[items][sample]; for(int i=0;i<items;i++){ for(int j=0;j<sample;j++){ double tval=0.0; for(int p=0;p<totalcols;p++) tval+=dblarrayone[j][p] * dblarraytwo[i][p]; resultarray[i][j]=Math.abs(tval); } } so I wanted to do the same in python..(may be this code is not in the recommended way..) i am storing the filled matrices and other values as instance variable of a class and access them by self.whatever... self.items=25 self.sample=5 self.totalcols=8100 #self.matrixone,self.matrixtwo are numply matix objects with already filled elements #but for testing i filled it with zeros self.matrixone=matrix(zeros((items,totalcols))) self.matrixtwo=matrix(zeros((items,totalcols))) resultmatrix=matrix(zeros((self.items,self.sample))) for i in range(self.items): for j in range(self.sample): tval=0.0 for p in range(self.totalcols): tval +=self.matrixone[ j , p ] * self.matrixtwo[ i , p ] resultmatrix[ i, j ]=abs(tval) here I found that while the java code takes barely 110 milliseconds to execute the code ,the python code takes something like 53 secs to execute !!..I am baffled by this ..can anyone advise me how i can improve this? (i want to code in python so I can't use c,c++ , java) dn

3 3

distutils_scons_command branch: 2nd step towards scons support in numpy
by David Cournapeau 25 Dec '07

25 Dec '07

Hi, I have started publishing the distutils_scons_command branch in numpy svn repository. This implements all the distutils infrastructure necessary to call scons, by using the scons distutils command. In more details: - distutils setup files can ask for a different name for setup file: if the top setup file calls the Configuration __init__ function with setup_name argument = foo.py, it will recursively call foo.py instead of setup.py. This is useful to have several build configuration in parrallel (so that we can include setup.py files using the current build system, and setupscons.py files which use scons instead). - the scons distutils command is implemented. For every scons script file registered through the add_sconscript method, it will call scons. If at least one scons script is registered, it will need the numscons package. If no scons scripts files are registered, the command does nothing. Once those code changes are integrated in numpy trunk, the last step for a working scons solution will be the release of numscons, which implements all the functionalities to use scons script files easily within numpy code. The plan is to be able to optionally build numpy 1.0.5 using scons. cheers, David

1 0

gfortran/g77+f2py vs gcc+Cython speed comparison
by Ondrej Certik 24 Dec '07

24 Dec '07

Hi, I needed to write 2D Ising model simulation into my school and I decided to compare the two possible solutions how to do it, so I of course wrote it in Python, then rewrote it in Fortran + f2py, and also Cython. What is better? Read below. :) But for the impatient, I am going to use Cython, reasons below. CCing to Cython, numpy (f2py is discussed there), and sage-devel (there are people there who could be interested in these kinds of comparisons). The code is available at: http://hg.sharesource.org/isingmodel/ How to play with that - just do this (after installing Mercurial): $ hg clone http://hg.sharesource.org/isingmodel/ [...] $ cd isingmodel $ hg up db7dd01cdc26 # just to be sure that we are talking about the same revision / code $ make [...] $ time python simulate.py [...] real 0m2.026s user 0m1.988s sys 0m0.020s This runs Cython code. Then apply this patch to run fortran code instead: $ hg di diff -r db7dd01cdc26 simulate.py --- a/simulate.py Sun Dec 23 02:23:30 2007 +0100 +++ b/simulate.py Sun Dec 23 02:24:33 2007 +0100 @@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps J=1 #coupling constant k=1 #Boltzman constant - #from mcising import mc - from pyising import mc + from mcising import mc + #from pyising import mc B = D1(A) mc(B, dim, steps, temp, H, mu, J, k) return D2(B) And then again (and apply the patch below, otherwise it might not work for you): $ time python simulate.py [...] real 0m3.600s user 0m3.528s sys 0m0.052s So it's a lot slower. We are comparing many things here - wrappers, my fortran coding skills vs Cython C code generation and gcc vs gfortran. So I wrote to numpy mailinglist. First Travis (the author of numpy) suggested: " My experience with similar kinds of comparisons is that gnu fortran compilers are not very good, especially on 2-d problems. Try using a different fortran compiler to see if speeds improve. " Then Pearu (the author of f2py) suggested: " Though the problem is 2D, your implementations are essentially 1D. If you would treat the array A as 2D array (and avoid calling subroutine p) then you would gain some 7% speed up in Fortran. When using -DF2PY_REPORT_ATEXIT for f2py then a summary of timings will be printed out about how much time was spent in Fortran code and how much in the interface. In the given case I get (nsteps=50000): Overall time spent in ... (a) wrapped (Fortran/C) functions : 1962 msec (b) f2py interface, 60 calls : 0 msec (c) call-back (Python) functions : 0 msec (d) f2py call-back interface, 0 calls : 0 msec (e) wrapped (Fortran/C) functions (acctual) : 1962 msec that is, most of the time is spent in Fortran function and no time in wrapper. The conclusion is that the cause of the difference in timings is not in f2py or cpython generated interfaces but in Fortran and C codes and/or compilers. Some idiom used in Fortran code is just slower than in C.. For example, in C code you are doing calculations using float precision but in Fortran you are forcing double precision. HTH, Pearu PS: Here follows a setup.py file that I used to build the extension modules instead of the Makefile: #file: setup.py def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration config = Configuration('',parent_package,top_path) config.add_extension('mcising', sources=['mcising.f'], define_macros = [('F2PY_REPORT_ATEXIT',1)] ) #config.add_extension('pyising', sources=['pyising.pyx']) return config from numpy.distutils.core import setup setup(configuration = configuration) " and then quickly added " When using g77 compiler instead of gfortran, I get a speed up 4.8 times. Btw, a line in a if statement of the fortran code should read `A(p(i,j,N)) = - A(p(i,j,N))`. " (btw I have no idea how it could work for me without the A(p(i,j,N)) = - A(p(i,j,N)) fix, quite embarassing) So then we discussed it on #sympy IRC: * Now talking on #sympy <ondrej> hi pearu, thanks a lot for testing it! <ondrej> 4.8 speedup, jesus christ. so the gfortran sucks a lot <pearu> hey ondrej * ondrej is trying g77 <pearu> gortran has advantage of being Fortran 90 compiler <ondrej> g77 is depricated in Debian and dead upstram (imho) <pearu> yes but I guess those guys who are maintaing g77/gfortran, those do not crunch numbers much;) <pearu> g77 has a long development history and is well optimized <ondrej> I fear the fortran is a bad investment for new projects <ondrej> I know g77 is well optimized, but it's dead <pearu> gfortran is a write from scratch and is too young <ondrej> do you think many people still write fortran? I use it just because g77 was faster than gcc <pearu> g77 is certainly not dead, scientist use it a lot because of its speed <ondrej> btw`A(p(i,j,N)) = - A(p(i,j,N))`. <ondrej> means my fortran code was broken <ondrej> doesn't it? <pearu> some think that fortran is a dead language, some use it a lot because lots of code is written in fortran over several decades <ondrej> yes <pearu> yes, it was, I got segfaults because of that <ondrej> I don't know what to think myself <pearu> it depends on application <pearu> if it is a research app then use fortran because of speed <ondrej> but as you can see, you need to use g77 <ondrej> and not gfortran <ondrej> (but all Debian is moving to use gfortran) <pearu> I use gfortran only for f90 code <ondrej> hm <pearu> I think Debian is wrong in short term, may be in future gfortran will be faster, but not at the moment <ondrej> hm, that's bad <ondrej> but is someone developing g77? <pearu> no, I don't think so <ondrej> debian moved to gfortran, because there are problems with g77 being umaintained <pearu> but it is a complete Fortran 77 compiler and a very good one <ondrej> hm. I think when one wants speed, one should use some comercial compilers <pearu> probably because new architectures are developed and g77 is not updated to use their features <ondrej> and when one wants robustness, it should use a free compiler that is maintinaed (gfortran) <pearu> commercial compilers does not mean fast compliers, in general <ondrej> I didn't try intel, so I don't know how it compares to g77 <pearu> using intel restricts one to be on Intel platform <ondrej> the g77 doesn't support -fdefault-real-8 for example <pearu> AMD is daster in floating point that most scientist use <pearu> why is this flag important, one can always write real*8 <ondrej> indeed, g77 is 1.8s and gcc 2.0s on my comp <ondrej> as to real*8 - I thought a good strategy is to write real and let the compiler choose the precision <pearu> try to use float in fortran as you do in C <ondrej> ok <pearu> this will mess up f2py-ing as f2py assumes that real is real*4 <ondrej> yes - <ondrej> real*4 is float, isn't it? <pearu> yes <ondrej> so I'll just use real*4 everywhere <pearu> yep <ondrej> ok <ondrej> btw, I was writing a python script for automatically converting real*4 to real*8 and back <ondrej> but it's freaking hard to parse fortran <ondrej> (I needed to also work with "real", etc) <ondrej> I only managed it to work in simple cases <pearu> just do re.sub <ondrej> it's not that easy <ondrej> there are constructs like real(something) <ondrej> that shouldn't be converted <ondrej> but real T should be converted <ondrej> etc. <ondrej> so using real*4 with g77 slowed the code done from 1.8s to 1.9s <pearu> re.sub should be able to handle these, use cb argument, fo instance <ondrej> ok <pearu> you are using random numbers, could this make also a difference <ondrej> I think it does <ondrej> (I had to use glibc fast random to speed C up) <pearu> a, ok, that also explains things <ondrej> shit, the gfortran is really slow <ondrej> 3.5s on my comp <ondrej> no matter which precision <pearu> try to disable random to see if loops are faster in C or fortran <ondrej> ok <ondrej> so gfortran is now 1.3s <ondrej> and g77 1.04s * ondrej is trying C <ondrej> C is 1.05s <pearu> so gfortran random is slow <ondrej> yes <ondrej> g77 random is fast <ondrej> C random is slower than g77 <ondrej> it depends on the quality of the random generator as well <ondrej> there are huge differences in that <pearu> and so gfortran itsel is not too bad <pearu> :) <ondrej> but as you can see - there is no point for me to use g77 if I can achieve the same speed with C <ondrej> and gfortran is slower than both <pearu> yep <ondrej> ok, going to write this to the list <pearu> can you try ifc <ondrej> I'll also send it to the Cython guys, they'll like the result. :) <ondrej> ifc = intel fortran compiler? <pearu> yes <ondrej> I like opensource and Debian <ondrej> I think this clearly shows, that fortran is dead for me. I think gcc is pretty good too. <pearu> note that these results are not thanks to pyrex /cpython but thanks to compilers:) <pearu> i like to be fair;) <ondrej> (yes, it's just compilers comparison, not f2py vs Cython/pyrex) <pearu> yep <ondrej> but there are two ways a user can choose - either cython+gcc, or f2py+fortran <pearu> next time we shoul probably have this discussion inscipy irc <ondrej> and if he wants to use the most recent default free compilers, at least I will choose cython+gcc <pearu> there are more ways, weave, etc etc <ondrej> I am going to paste this to my email about that <pearu> ok <ondrej> (if you are ok with that) <pearu> ok with me So, what do you think of that? Ondrej

2 3

matlab to numpy help
by Gabriel J.L. Beckers 23 Dec '07

23 Dec '07

I found a matlab script that I want to translate into numpy, but have difficulties with understanding indexing in matlab. I haven't used matlab very much and I was hoping that someone could help with the following: It says: Uns = ones(1,m); ... and then later Xijm(:,Uns); m is a positive integer. The problem is that Xijm should be a 1-dimensional array (I think), so what is Uns doing in the second statement? Is this some way to expand Xijm into a second dimension? What would be a numpy equivalent? Gabriel

3 2

setting items in a matrix
by devnew＠gmail.com 23 Dec '07

23 Dec '07

hi i am a beginner with numpy and python,so pardon me if this doubt seems silly i want to create a matrix with say 3 rows and 5 columns..and then set the values of each item in it .for this i did something like below myarray=zeros((3,5)) #then set the items for row in range(3): for col in range(5): myarray[row][col]=999999.9999 mymatrix=matrix(myarray) is this the way to do the matrix creation and value setting? is the use of zeros() unnecessary? i am in the early learning stage so your reply wd help me much dn

4 3