Mailman 3 November 2011 - SciPy-User

Cython vs Vectorized Numpy vs MATLAB
by Jaidev Deshpande 18 Nov '11

18 Nov '11

Hi, I have two questions. 1. Why does the 'spline.m' function in MATLAB perform much faster than the same interpolation in NumPy? (In MATLAB the first function call takes time but the subsequent calls are much faster.) 2. I wrote a cubic spline interpolation algorithm with NumPy and I vectorized it. Is it surprising that the Cython compiled version of the same function is no faster? I guess that shouldn't happen, as parts of the code would be compiled into C. Although, please note that I used Cython on the vectorized code *as it is*, without adding static types. I know, stupid thing to do, but shouldn't it have given me *some *speed-up? In my problem, for the same data MATLAB takes ............................................................. 0.038 seconds Vectorized NumPy takes................................................. 1.0342 second The above, Cythonized ................................................... 0.997 seconds Functions in the scipy.interpolate package take.................. well above 1 second (I've tried out almost everything scipy has to offer) Since MATLAB is the fastest, I am trying to get Python to be atleast as fast if not faster. Is that even possible? How? Thanks

6 6

kdtree, custom distance function
by Oleksandr Huziy 18 Nov '11

18 Nov '11

Hello, I am trying to use scipy.spatial.kdtree to interpolate data from a lat/lon grid to a set of points (also with lat/lon coordinates). Is it possible to specify a custom distance function for the kdtree that should be used for querying? Also is there a function that computes distance on a sphere in scipy.spatial? thanks -- Oleksandr Huziy UQAM

3 2

match extent of arrays
by questions anon 16 Nov '11

16 Nov '11

Hello All, I have an array that I created from a shapefile using gdal.RasterizeLayer and then ReadAsArray(). I would like the array I created from a shapefile to match my array from a netcdf file. I am not sure how I go about this. The extents of the shapefile/raster/array are: x_min, x_max, y_min, y_max 140.962408758 149.974994992 -39.1366533667 -33.9813898583 The extents and size for my array from netcdf files are: 139.8 max longitude: 150.0 min latitude -39.2 max latitude: -33.6 LAT size 106 LON size 193 Any ideas on how I might achieve this? The overal goal is to then use the array from a shapefile as a mask for my netcdf files. Thanks

1 1

scipy import problem in apache-mod_wsgi environment
by Myunghwa Hwang 16 Nov '11

16 Nov '11

Hello, list! First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question. I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application. Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 is also manually installed from one of default linux repository. To be specific, it is installed from source code together with httpd-dev. 3. Python -- Python 2.7.2 is also installed from source code across all nodes in the cluster. Its source code is downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file. When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. 1. ######################## 2. starting linalg.__init__ 3. pre __init__.__doc__ 4. pre __init__.__version__ 5. pre __init__.misc 6. pre __init__.basic 7. ####################### 8. Starting basic 9. pre basic.flinalg 10. pre basic.lapack 11. pre basic.misc 12. pre basic.scipy.linalg 13. pre basic.decomp_svd 14. pre __init__.decomp 15. ################ 16. starting decomp 17. pre decomp.array et al. 18. pre decomp.calc_lwork 19. pre decomp.LinAlgError 20. pre decomp.get_lapack_funcs 21. pre decomp.get_blas_funcs 22. #################### 23. Starting blas 24. pre blas.scipy.linalg.fblas 25. pre blas.scipy.linalg.cblas 26. pre __init__.decomp_lu 27. pre __init__.decomp_cholesky 28. pre __init__.decomp_qr 29. ################# 30. Starting special_matrices 31. pre special_matrices.math 32. pre special_matrices.np 33. pre __init__.decomp_svd 34. pre __init__.decomp_schur 35. ################## 36. starting schur... 37. pre decomp_schur.misc 38. pre decomp_schur.LinAlgError 39. pre decomp_schur.get_lapack_funcs 40. pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 41. schur testing 42. pre __init__.matfuncs 43. ##################### 44. Starting matfuncs 45. pre matfuncs. asarray et al 46. pre matfuncs.matrix 47. pre matfuncs.np 48. pre matfuncs.misc 49. pre matfuncs.basic 50. pre matfuncs.special_matrices 51. pre matfuncs.decomp 52. pre matfuncs.decomp_svd 53. pre matfuncs.decomp_schur 54. pre __init__.blas 55. pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup. If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me. Thanks for reading this post. I will look forward to yo -- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4(a)gmail.com or Myunghwa.Hwang(a)asu.edu

5 10

MODIS data and true-color plotting
by Gökhan Sever 16 Nov '11

16 Nov '11

Hello groups, I have two questions about working with MODIS data. 1-) Is there any light Pythonic HDF-EOS wrapper to handle HDF-EOS data other than PyNIO [http://www.pyngl.ucar.edu/Nio.shtml] Although, I have managed to install that package from its source, it took me many hours to figure out all the installation quirks. Something simpler to build and mainly for HDFEOS data?? 2-) Another similar question: Has anybody attempted to create true-color MODIS images (like the ones shown at [ http://rapidfire.sci.gsfc.nasa.gov/realtime/]) in Python? So far, I have seen one clear tutorial [ftp://ftp.ssec.wisc.edu/pub/IMAPP/MODIS/TrueColor/] to create natural color images, but uses ms2gt [ http://nsidc.org/data/modis/ms2gt/], NDVI and IDL. Except the reflectance correction via NDVI, ms2gt and IDL parts seem to be implemented in Python. Till now, I have some progress combining GOES imagery with aircraft data. My next task is to combine MODIS data with aircraft and radar data. I would be happy to get some guidance and code support if there is any previous work been done using Python. Thanks.

3 11

[SciPy-user] scipy.test() curious failure
by jajabinker 16 Nov '11

16 Nov '11

HI ALL! Hope you can help me! After i run scipy.test() i get the following output: http://pastebin.com/eQwnKDy4 Thats that error. Has any1 seen it? I have installed numpy no errors while running numpy.test(). In case you spot a bug in my atlas. Here is my configure But the compilation was clean. ../configure -b 64 -Fa alg -fPIC --prefix=/home/$USER/numpyscipy/atlas --with-netlib-lapack=/home/$USER/numpyscipy/lapack-3.4.0/liblapack.a THANKS! -- View this message in context: http://old.nabble.com/scipy.test%28%29-curious-failure-tp32852072p32852072.… Sent from the Scipy-User mailing list archive at Nabble.com.

2 2

how to generate very small random number
by franck kalala 16 Nov '11

16 Nov '11

Hello Folk I would like to generate very small random number in scipy. the command random() generate random number in (0,1). I am doing a simulation that involve very small numbers. for example I am doing somthing like this: >>> import random >>> a = random.random() >>> if a < 10**-6: do something But the 'do domething' is never executed, because 10**-6 is very small, and in many cases the random number generate is large, I would like then to generate very small number of order of 10**-6 for example, Any idea on how to do that? Cheers F

2 1

ANN: Spyder v2.1.2
by Pierre Raybaut 16 Nov '11

16 Nov '11

Hi all, On the behalf of Spyder's development team (http://code.google.com/p/spyderlib/people/list), I'm pleased to announce that Spyder v2.1.2 has been released and is available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://code.google.com/p/spyderlib/ As this is mostly a maintenance release, a lot of bugs were fixed and some minor features were added: http://code.google.com/p/spyderlib/wiki/ChangeLog Spyder is a free, open-source (MIT license) interactive development environment for the Python language with advanced editing, interactive testing, debugging and introspection features. Originally designed to provide MATLAB-like features (integrated help, interactive console, variable explorer with GUI-based editors for dictionaries, NumPy arrays, ...), it is strongly oriented towards scientific computing and software development. Thanks to the `spyderlib` library, Spyder also provides powerful ready-to-use widgets: embedded Python console (example: http://packages.python.org/guiqwt/_images/sift3.png), NumPy array editor (example: http://packages.python.org/guiqwt/_images/sift2.png), dictionary editor, source code editor, etc. Description of key features with tasty screenshots can be found at: http://code.google.com/p/spyderlib/wiki/Features On Windows platforms, Spyder is also available as a stand-alone executable (don't forget to disable UAC on Vista/7). This all-in-one portable version is still experimental (for example, it does not embed sphinx -- meaning no rich text mode for the object inspector) but it should provide a working version of Spyder for Windows platforms without having to install anything else (except Python 2.x itself, of course). Don't forget to follow Spyder updates/news: * on the project website: http://code.google.com/p/spyderlib/ * and on our official blog: http://spyder-ide.blogspot.com/ Last, but not least, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment! (http://code.google.com/p/spyderlib/wiki/NoteForContributors) Enjoy! -Pierre

1 0

scipy.misc.lena returns wrong array type?
by Bill Janssen 15 Nov '11

15 Nov '11

I've been trying to figure out why scipy.misc.lena() return an array of 8-bit grayscale values but with the dtype as int32. Seems to me it should be an array of uint8? As an added bonus, then this would then work with the PIL .fromarray() method, which it currently doesn't, because PIL thinks that arrays of int32 are RGB or RGBA images).

1 0

Segmentation fault in scipy linkage function with large data set
by Uri Laserson 15 Nov '11

15 Nov '11

Hi all, I am trying to cluster a data set with almost 50,000 objects using hierarchical clustering. I generate a distance matrix like so: Y = pdist( unique_seqs, vdj.clusteringcore.levenshtein ) and then try to perform the linkage like so: Z = sp.cluster.hierarchy.linkage(Y,method=linkage) The distance matrix is computed fine (albeit after 10 hours or so), and the segfault occurs in the `linkage` function. However, I run the same script on many other inputs that are smaller, and it finishes successfully. This one largest input is giving me problems. You can see the memory usage as a function input size here: https://picasaweb.google.com/lh/photo/KjPHcosMKxrehK22tslr4A?feat=directlink and the CPU time here: https://picasaweb.google.com/lh/photo/ygS_njM80Olja04pRRP2vw?feat=directlink Each point is one execution of the script with a different set of input sequences. The vertical blue line shows the size of the current input, which is causing the segfaults. Does anyone have any ideas/suggestions as to what the problem is here. When I searched for other possible solutions, I found my own reporting on the same problem in the past: http://projects.scipy.org/scipy/ticket/967 However in that case, I was able to reduce the input size so that I don't segfault. I am running these on a large linux cluster running python 2.7.1 using numpy 1.5.0b1 and scipy 0.8.0. According to the cluster administrators, the process did *not* make any sudden large requests for resources that were unmet. Debugging here is especially hard because it take 10 hours to get to the segfault...sigh... Thanks! Uri ....................................................................................... Uri Laserson Graduate Student | Biomedical Engineering | Church Lab Harvard-MIT Division of Health Sciences and Technology M +1 617 910 0447 laserson(a)mit.edu http://web.mit.edu/laserson/www/

1 0