Mailman 3 October 2011 - NumPy-Discussion

Large numbers into float128
by Matthew Brett 30 Oct '11

30 Oct '11

Hi, Can anyone think of a good way to set a float128 value to an arbitrarily large number? As in v = int_to_float128(some_value) ? I'm trying things like v = np.float128(2**64+2) but, because (in other threads) the float128 seems to be going through float64 on assignment, this loses precision, so although 2**64+2 is representable in float128, in fact I get: In [35]: np.float128(2**64+2) Out[35]: 18446744073709551616.0 In [36]: 2**64+2 Out[36]: 18446744073709551618L So - can anyone think of another way to assign values to float128 that will keep the precision? Thanks a lot, Matthew

4 5

Conditional random variables
by Ted To 29 Oct '11

29 Oct '11

Hi, Is there an easy way to make random draws from a conditional random variable? E.g., draw a random variable, x conditional on x>=\bar x. Thank you, Ted To

3 6

ANN: PyTables 2.3.1 released
by Antonio Valentino 29 Oct '11

29 Oct '11

=========================== Announcing PyTables 2.3.1 =========================== We are happy to announce PyTables 2.3.1. This is a bugfix release. Upgrading is recommended for users that are running PyTables in production environments. What's new ========== This release includes a small number of changes. It only fixes a couple of bugs that are considered serious even if they should not impact a large number of users: - :issue:`113` caused installation of PyTables 2.3 to fail on hosts with multiple python versions installed. - :issue:`111` prevented to read scalar datasets of UnImplemented types. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/@VERSION@ For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

1 0

NA masks in the next numpy release?
by Nathaniel Smith 28 Oct '11

28 Oct '11

Hi all, I was surprised today to notice that Mark's NA mask support appears to have been merged into numpy master and is described in the draft release notes[1]. My surprise is because merging it to mainline without any discussion on the list seems to contradict what what Travis wrote in July, that it was being developed as an experiment and explicitly *not* intended to be merged without further discussion: "Basically, because there is not consensus and in fact a strong and reasonable opposition to specific points, Mark's NEP as proposed cannot be accepted in its entirety right now. However, I believe an implementation of his NEP is useful and will be instructive in resolving the issues and so I have instructed him to spend Enthought time on the implementation. Any changes that need to be made to the API before it is accepted into a released form of NumPy can still be made even after most of the implementation is completed as far as I understand it."[2] Can anyone explain what the plan is here? Is the idea to continue the discussion and rework the API while it is in master, delaying the next release for as long as it takes to achieve consensus? Or is there some mysterious git thing going on where "master" is actually an experimental branch and the real mainline development is happening somewhere else? Or something else I'm not thinking of? Please help me understand. -- Nathaniel [1] https://github.com/numpy/numpy/blob/master/doc/release/2.0.0-notes.rst [2] http://mail.scipy.org/pipermail/numpy-discussion/2011-July/057579.html

16 68

skip lines at the end of file with loadtxt
by Massimo Di Stefano 28 Oct '11

28 Oct '11

i'm tring to generate an array reading a txt file from internet. my target is to use python instead of matlab, to replace this steps in matlab : url=['http://www.cdc.noaa.gov/Correlation/amon.us.long.data']; urlwrite(url,'file.txt'); i'm using this code : urllib.urlretrieve('http://www.cdc.noaa.gov/Correlation/amon.us.long.data', 'file.txt') a = np.loadtxt('file.txt', skiprows=1) but it fails becouse of the txt description at the end of the file, do you know if exist a way to skip the X lines at the end, something like "skipmultiplerows='1,-4'" (to skip the first and the last 4 rows in the file) or i have to use some sort of string manipulation (readlines?) instead ? Thanks! --Massimo

3 3

[ANN] SciPy India 2011 Abstracts due November 2nd
by Jarrod Millman 28 Oct '11

28 Oct '11

========================== SciPy 2011 Call for Papers ========================== The third `SciPy India Conference <http://scipy.in>`_ will be held from December 4th through the 7th at the `Indian Institute of Technology, Bombay (IITB) <http://www.iitb.ac.in/>`_ in Mumbai, Maharashtra India. At this conference, novel applications and breakthroughs made in the pursuit of science using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is followed by two days of tutorials and a code sprint, during which community experts provide training on several scientific Python packages. We invite you to take part by submitting a talk abstract on the conference website at: http://scipy.in Talk/Paper Submission ========================== We solicit talks and accompanying papers (either formal academic or magazine-style articles) that discuss topics regarding scientific computing using Python, including applications, teaching, development and research. We welcome contributions from academia as well as industry. Important Dates ========================== November 2, 2011, Wednesday: Abstracts Due November 7, 2011, Monday: Schedule announced November 28, 2011, Monday: Proceedings paper submission due December 4-5, 2011, Sunday-Monday: Conference December 6-7 2011, Tuesday-Wednesday: Tutorials/Sprints Organizers ========================== * Jarrod Millman, Neuroscience Institute, UC Berkeley, USA (Conference Co-Chair) * Prabhu Ramachandran, Department of Aerospace Engineering, IIT Bombay, India (Conference Co-Chair) * FOSSEE Team

1 0

float128 / longdouble on PPC - is it broken?
by Matthew Brett 26 Oct '11

26 Oct '11

Hi, I just ran into this on a PPC machine: In [1]: import numpy as np In [2]: np.__version__ Out[2]: '2.0.0.dev-4daf949' In [3]: res = np.longdouble(2)**64 In [4]: res Out[4]: 18446744073709551616.0 In [5]: 2**64 Out[5]: 18446744073709551616L In [6]: res-1 Out[6]: 36893488147419103231.0 Same for numpy 1.4.1. I don't have a SPARC to test on but I believe it's the same double-double type? See you, Matthew

6 20

numpy.matrix subclassing
by David Voong 25 Oct '11

25 Oct '11

Hi guys, I have a question regarding subclassing of the numpy.matrix class. I read through the wiki page, http://docs.scipy.org/doc/numpy/user/basics.subclassing.html and tried to subclass numpy.matrix, I find that if I override the __finalize_array__ method I have problems using the sum method and get the following error, Traceback (most recent call last): File "test.py", line 61, in <module> print (a * b).sum() File "/afs/ cern.ch/user/d/dvoong/programs/lib/python2.6/site-packages/numpy/matrixlib/defmatrix.py", line 435, in sum return N.ndarray.sum(self, axis, dtype, out)._align(axis) File "/afs/ cern.ch/user/d/dvoong/programs/lib/python2.6/site-packages/numpy/matrixlib/defmatrix.py", line 370, in _align return self[0,0] File "/afs/ cern.ch/user/d/dvoong/programs/lib/python2.6/site-packages/numpy/matrixlib/defmatrix.py", line 305, in __getitem__ out = N.ndarray.__getitem__(self, index) IndexError: 0-d arrays can only use a single () or a list of newaxes (and a single ...) as an index Can anyone help? It's my first time on this mailing list so apologies if this is not the right place to discuss this. Regards David Voong

2 1

neighborhood iterator speed
by Nadav Horesh 25 Oct '11

25 Oct '11

I am trying to replace an old code (biliteral filter) that rely on ndimage.generic_filter with the neighborhood iterator. In the old code, the generic_filter generates a contiguous copy of the neighborhood, thus the (cython) code could use C loop to iterate over the neighbourhood copy. In the new code version the PyArrayNeighborhoodIter_Next must be called to retrieve every neighbourhood item. The results of rough benchmarking to compare bilateral filtering on a 1000x1000 array: Old code (ndimage.generic_filter): 16.5 sec New code (neighborhood iteration): 60.5 sec New code with PyArrayNeighborhoodIter_Next omitted: 1.5 sec * The last benchmark is not "real" since the omitted call is a must. It just demonstrates the iterator overhead. * I assune the main overhead in the old code is the python function callback process. There are instructions in the manual how to wrap a C code for a faster callback, but I rather use the neighbourhood iterator as I consider it as more generic. If the PyArrayNeighborhoodIter_Reset could (optionally) copy the relevant data (as the generic_filter does) it would provide a major speed up in many cases. Nadav

2 8

libmkl_lapack error in numpy
by akshar bhosale 24 Oct '11

24 Oct '11

i am getting following error. python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], numpy.complex128).T.I.H' MKL FATAL ERROR: Cannot load libmkl_lapack.so have installed numpy 1.6.0 with python 2.6. i have intel cluster toolkit installed on my system. (11/069 version and mlk=10.3). i have machine having intel xeon processor and rhel 5.2 x86_64 platform. i am trying with intel compilers. if i do python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], numpy.complex128).T.I.H' python: symbol lookup error: /opt/intel/Compiler/11.0/069/ mkl/lib/em64/libmkl_lapack.so: undefined symbol: mkl_lapack_zgeqrf my site.cfg is : #################### [mkl] mkl_libs =3D mkl_def, mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_mc - Ignored: lapack_libs =3D mkl_lapack95_lp64 library_dirs =3D /opt/intel/Compiler/11.0/069/ mkl/lib/em64t:/opt/intel/Compiler/11.0/069/lib/intel64/ include_dirs =3D /opt/intel/Compiler/11.0/069/mkl/include:/opt/intel/Compiler/11.0/069/inclu= de/ #################### and intelcompiler.py is : ############################ from distutils.unixccompiler import UnixCCompiler from numpy.distutils.exec_command import find_executable import sys class IntelCCompiler(UnixCCompiler): """ A modified Intel compiler compatible with an gcc built Python.""" compiler_type =3D 'intel' cc_exe =3D 'icc' cc_args =3D 'fPIC' def __init__ (self, verbose=3D0, dry_run=3D0, force=3D0): sys.exit(0) UnixCCompiler.__init__ (self, verbose,dry_run, force) self.cc_exe =3D 'icc -fPIC ' compiler =3D self.cc_exe self.set_executables(compiler=3Dcompiler, compiler_so=3Dcompiler, compiler_cxx=3Dcompiler, linker_exe=3Dcompiler, linker_so=3Dcompiler + ' -shared -lstdc++') class IntelItaniumCCompiler(IntelCCompiler): compiler_type =3D 'intele' # On Itanium, the Intel Compiler used to be called ecc, let's search fo= r # it (now it's also icc, so ecc is last in the search). for cc_exe in map(find_executable,['icc','ecc']): if cc_exe: break class IntelEM64TCCompiler(UnixCCompiler): """ A modified Intel x86_64 compiler compatible with a 64bit gcc built Python. """ compiler_type =3D 'intelem' cc_exe =3D 'icc -m64 -fPIC' cc_args =3D "-fPIC -openmp" def __init__ (self, verbose=3D0, dry_run=3D0, force=3D0): UnixCCompiler.__init__ (self, verbose,dry_run, force) self.cc_exe =3D 'icc -m64 -fPIC -openmp ' compiler =3D self.cc_exe self.set_executables(compiler=3Dcompiler, compiler_so=3Dcompiler, compiler_cxx=3Dcompiler, linker_exe=3Dcompiler, linker_so=3Dcompiler + ' -shared -lstdc++') ########################## LD_LIBRARY_PATH is : ######################### /opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/scalasca-1.3.3/lib:/opt/PBS= /lib:/opt/intel/mpi/lib64:/opt/maui/lib:/opt/jdk1.6.0_23/lib:/opt/intel/Com= piler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em6= 4t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/l= ib:/opt/intel/Compiler/11.0/069/lib/intel64:/opt/intel/Compiler/11.0/069/ip= p/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Com= piler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Com= piler/11.0/069/lib/intel64:/usr/local/lib ######################### - Done. ---------- Forwarded message ---------- From: akshar bhosale <akshar.bhosale(a)gmail.com> To: numpy-discussion-request(a)scipy.org Date: Sun, 23 Oct 2011 11:19:27 +0530 Subject: libmkl_lapack error Hi, i am getting following error. python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], numpy.complex128).T.I.H' MKL FATAL ERROR: Cannot load libmkl_lapack.so have installed numpy 1.6.0 with python 2.6. i have intel cluster toolkit installed on my system. (11/069 version and mkl 10.3). i have machine having intel xeon processor and rhel 5.2 x86_64 platform. i am trying with intel compilers. if i do python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], numpy.complex128).T.I.H' python: symbol lookup error: /opt/intel/Compiler/11.0/069/mkl/lib/em64/libmkl_lapack.so: undefined symbol: mkl_lapack_zgeqrf my site.cfg is : #################### [mkl] mkl_libs = mkl_def, mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_mc lapack_libs = mkl_lapack95_lp64 library_dirs = /opt/intel/Compiler/11.0/069/ mkl/lib/em64t:/opt/intel/Compiler/11.0/069/lib/intel64/ include_dirs = /opt/intel/Compiler/11.0/069/mkl/include:/opt/intel/Compiler/11.0/069/include/ #################### and intelcompiler.py is : ############################ from distutils.unixccompiler import UnixCCompiler from numpy.distutils.exec_command import find_executable import sys class IntelCCompiler(UnixCCompiler): """ A modified Intel compiler compatible with an gcc built Python.""" compiler_type = 'intel' cc_exe = 'icc' cc_args = 'fPIC' def __init__ (self, verbose=0, dry_run=0, force=0): sys.exit(0) UnixCCompiler.__init__ (self, verbose,dry_run, force) self.cc_exe = 'icc -fPIC ' compiler = self.cc_exe self.set_executables(compiler=compiler, compiler_so=compiler, compiler_cxx=compiler, linker_exe=compiler, linker_so=compiler + ' -shared -lstdc++') class IntelItaniumCCompiler(IntelCCompiler): compiler_type = 'intele' # On Itanium, the Intel Compiler used to be called ecc, let's search for # it (now it's also icc, so ecc is last in the search). for cc_exe in map(find_executable,['icc','ecc']): if cc_exe: break class IntelEM64TCCompiler(UnixCCompiler): """ A modified Intel x86_64 compiler compatible with a 64bit gcc built Python. """ compiler_type = 'intelem' cc_exe = 'icc -m64 -fPIC' cc_args = "-fPIC -openmp" def __init__ (self, verbose=0, dry_run=0, force=0): UnixCCompiler.__init__ (self, verbose,dry_run, force) self.cc_exe = 'icc -m64 -fPIC -openmp ' compiler = self.cc_exe self.set_executables(compiler=compiler, compiler_so=compiler, compiler_cxx=compiler, linker_exe=compiler, linker_so=compiler + ' -shared -lstdc++') ########################## LD_LIBRARY_PATH is : ######################### /opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/scalasca-1.3.3/lib:/opt/PBS/lib:/opt/intel/mpi/lib64:/opt/maui/lib:/opt/jdk1.6.0_23/lib:/opt/intel/Compiler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/069/lib/intel64:/opt/intel/Compiler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/069/lib/intel64:/usr/local/lib #########################

1 2