Mailman 3 October 2014 - NumPy-Discussion

Re: [Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt
by Saullo Castro 27 Oct '14

27 Oct '14

I agree with @Daniele's point, storing huge arrays in text files migh indicate a bad process.... but once these functions can be improved, why not? Unless this turns to be a burden to change. Regarding the estimation of the array size, I don't see a big performance loss when the file iterator is exhausting once more in order to estimate the number of rows and pre-allocate the proper arrays to avoid using list of lists. The hardest part seems to be dealing with arrays of strings (perhaps easily solved with dtype=object) and structured arrays. Cheers, Saullo 2014-10-26 18:00 GMT+01:00 <numpy-discussion-request(a)scipy.org>: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Memory efficient alternative for np.loadtxt and > np.genfromtxt (Daniele Nicolodi) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 26 Oct 2014 17:42:32 +0100 > From: Daniele Nicolodi <daniele(a)grinta.net> > Subject: Re: [Numpy-discussion] Memory efficient alternative for > np.loadtxt and np.genfromtxt > To: numpy-discussion(a)scipy.org > Message-ID: <544D2478.8020504(a)grinta.net> > Content-Type: text/plain; charset=windows-1252 > > On 26/10/14 09:46, Saullo Castro wrote: > > I would like to start working on a memory efficient alternative for > > np.loadtxt and np.genfromtxt that uses arrays instead of lists to store > > the data while the file iterator is exhausted. > > ... > > > I would be glad if you could share your experience on this matter. > > I'm of the opinion that if your workflow requires you to regularly load > large arrays from text files, something else needs to be fixed rather > than the numpy speed and memory usage in reading data from text files. > > There are a number of data formats that are interoperable and allow to > store data much more efficiently. hdf5 is one natural choice, maybe with > the blosc compressor. > > Cheers, > Daniele > > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 97, Issue 57 > ************************************************ >

1 0

npy_log2 undefined on Linux
by Matthew Brett 26 Oct '14

26 Oct '14

Hi, We (dipy developers) have a hit a new problem trying to use the ``npy_log`` C function in our code. Specifically, on Linux, but not on Mac or Windows, we are getting errors of form: ImportError: /path/to/extension/distances.cpython-34m.so: undefined symbol: npy_log2 when compiling something like: <eg_log.pyx> import numpy as np cimport numpy as cnp cdef extern from "numpy/npy_math.h" nogil: double npy_log(double x) def use_log(double val): return npy_log(val) </eg_log.pyx> See : https://github.com/matthew-brett/mincy/tree/npy_log_example for a self-contained example that replicates the failure with ``make``. I guess this means that the code referred to by ``npy_log`` is not on the ordinary runtime path on Linux? What should I do next to debug? Thanks a lot, Matthew

2 4

segfault in np.arange
by Dave Hirschfeld 24 Oct '14

24 Oct '14

Hi, I accidentally passed a pandas DatetimeIndex to `np.arange` which caused it to segfault. It's a pretty dumb thing to do but I don't think it should cause a segfault! Python 2.7.5 |Continuum Analytics, Inc.| (default, Jul 1 2013, 12:37:52) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. #>>> import faulthandler #>>> faulthandler.enable() #>>> import numpy as np #>>> np.__version__ '1.9.0' #>>> import pandas as pd #>>> pd.__version__ '0.14.1' #>>> np.arange(pd.DatetimeIndex([])) Fatal Python error: Segmentation fault Current thread 0x00001d18: File "<stdin>", line 1 in <module> The exception dialog which pops up contains the following information: Unhandled exception at 0x000000000255EB49 (multiarray.pyd) in python.exe: 0xC0000005: Access violation reading location 0x0000000000000008. Thanks, Dave

2 2

Re: [Numpy-discussion] successful Windows x64 build with acml-5.3.1 ifort64 static libs passes tests
by Mark Mikofski 24 Oct '14

24 Oct '14

AMD acml libraries do not include the CBLAS implementation. To build CBLAS against the libacml.lib requires the Intel Fortran compiler. There is a note in the mailing list here http://mail.scipy.org/pipermail/numpy-discussion/2006-February/018379.html This detail should probably be added to the new wiki, but in general it means that the AMD acml is not a drop in library

1 0

successful Windows x64 build with acml-5.3.1 ifort64 static libs passes tests
by Mark Mikofski 24 Oct '14

24 Oct '14

Both the /MT and /MD versions of the acml static libraries build, and pass all tests, except that for me at least there's a weird glitch in the build process. I have to purposefully add something so that config fails. EG: In the previous site.cfg, I did not realize that I had mistyped the vector math libraries as libsvml_disp.lib instead of svml_disp.lib. In fact you don't even need those libraries. Here I use the /MT version of the static libs, but add a dummy lib, "_". Config says NOT AVAILABLE for everything, but the build and install work fine. [blas] blas_libs = libacml, libifcoremt, libifport, libirc, libircmt, libmmt, svml_disp, svml_dispmt, _ library_dirs = C:\AMD\acml5.3.1\ifort64\lib\MT include_dirs = C:\AMD\acml5.3.1\ifort64\include [lapack] lapack_libs = libacml, libifcoremt, libifport, libirc, libircmt, libmmt, svml_disp, svml_dispmt, _ library_dirs = C:\AMD\acml5.3.1\ifort64\lib\MT include_dirs = C:\AMD\acml5.3.1\ifort64\include If I try to run the build without the dummy ("_"), then I get a traceback that says that lapack libs with appended underscores are unresolved. I think in the static library and the acml header file the c-wrappers remove the fortran underscore. Also I am using the Intel fortran built libs which I think use ALL CAPS not trailing underscores. I think this is the trailing underscore import issue, covered in this article: http://developer.amd.com/community/blog/2009/06/29/removing-c-wrapper-funct… Adding the dummy lib somehow sets the NO_APPEND_FORTRAN macro. Also it seems like the [amd] section in site.cfg does nothing. Also it doesn't matter if I change the library names to remove the lib- prefix, or if I rename the lib and header from acml to amd, makes no difference. THe only reference to AMD/ACML is http://wiki.scipy.org/Installing_SciPy/Windows but this site seems out of date, and in contradiction to other building scipy site that supports MKL out of the box.

1 1

Re: [Numpy-discussion] MKL not available as separate download since 10/16/2014
by Mark Mikofski 23 Oct '14

23 Oct '14

Michael, Thanks for your response. The Intel Math Kernel Library package with just the static and dynamic libraries for blas, lapack (similar to AMD's acml self extracting zip file) is what I think many Windows users were depending on to build Python's numerical and scientific libraries (numpy and scipy). If it could be made available that would be very welcomed. Currently the only remaining option is AMD's acml here: http://developer.amd.com/tools-and-sdks/cpu-development/amd-core-math-libra… which passes all tests. Currently it is unfortunate that the Intel MKL website has instructions for building numpy and scipy here: https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkland numpy/scipy has them here: http://www.scipy.org/scipylib/building/windows.html#building-numpy-with-the… but you can no longer download the MKL files. Thanks for your help! --Mark

1 0

Fwd: [julia-dev] OpenBLAS machines
by Fernando Perez 23 Oct '14

23 Oct '14

Not sure if folks saw this one... Might be interesting to contact this team from the python side of things. Cheers f ---------- Forwarded message ---------- From: Zhang Xianyi <traits.zhang(a)gmail.com> Date: Sun, Oct 19, 2014 at 12:50 AM Subject: [julia-dev] OpenBLAS machines To: Julia Dev <julia-dev(a)googlegroups.com> Hi Julia developers, The OpenBLAS team deployed the following machines to support developing and testing the project. We like to share the machines with Julia developers. The interested developer can apply the remote access by sending the mail to OpenBLAS-dev mailing list <https://groups.google.com/forum/#!forum/openblas-dev> or traits.zhang(a)gmail.com. - 2-way Intel Xeon E5-2680 (2.7GHz, Intel Sandy Bridge), 32GB memory, 1TB hard disk, Ubuntu 14.04 64-bit. - 1-way Intel Core i7-4770 (3.4GHz, Intel Haswell), 8GB memory, 512GB hard disk, Ubuntu 14.04 64-bit. - 1-way AMD FX-8320 (3.5GHz, AMD Piledriver), 8GB memory, 512GB hard disk, Ubuntu 14.04 64-bit. - Odroid-U2 board <http://www.hardkernel.com/main/products/prdt_info.php?g_code=G135341370451>. 1-way Exynos4412 Prime 1.7Ghz ARM Cortex-A9 Quad Cores, 2GB memory, 16GB SD card, Ubuntu 14.04 32-bit. The AMD HSA Apu (Kaveri) machine and Intel KNC MIC are coming soon. Xianyi -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

successful Windows x64 build with acml-5.3.1 ifort64 static libs passes tests
by Mark Mikofski 23 Oct '14

23 Oct '14

* Either windows sdk 7 or ms visual studio 2008 is required, however Fortran compilers are not.* Download acml5.3.1-ifort64.exe from http://developer.amd.com/tools-and-sdks/cpu-development/amd-core-math-libra… and it extracts to c:\AMD\acml5.3.1 * NOTE: There is also a newer acml-6.0.6.17-ifort64 update but it only includes dynamic libraries which depend on msvcr110 (visual studio 2013). * NOTE: There are several other optional libs incl. ifort64-mt, ifort64-fma4, ifort64-mp and ifort64-fma4-mp* System info: Windows-7 (64bit), python-2.7.6, numpy-1.9.0, virtualenv-1.11.5, nose-1.3.4* create site.cfg with: [amd]amd_libs = libacml, libifcoremd, libifportmd, libirc, libmmd, libsvml_disp, libsvml_dispmdlibrary_dirs = C:\AMD\acml5.3.1\ifort64\libinclude_dirs = C:\AMD\acml5.3.1\ifort64\include[blas]blas_libs = libacml, libifcoremd, libifportmd, libirc, libmmd, libsvml_disp, libsvml_dispmdlibrary_dirs = C:\AMD\acml5.3.1\ifort64\libinclude_dirs = C:\AMD\acml5.3.1\ifort64\include[lapack]lapack_libs = libacml, libifcoremd, libifportmd, libirc, libmmd, libsvml_disp, libsvml_dispmdlibrary_dirs = C:\AMD\acml5.3.1\ifort64\libinclude_dirs = C:\AMD\acml5.3.1\ifort64\include * activate virtualenv and inside extracted numpy-1.9.0 folder find setup.py and run it with the following commands (venv) numpy-1.9.0/numpy-1.9.0 $python setup.py config --compiler=msvc * start python, import numpy and run numpy.test()Ran 5458 tests in 29.346s OK (KNOWNFAIL=10, SKIP=18)<nose.result.TextTestResult run=5458 errors=0 failures=0> “As I breath in, I calm my body, as I breath out, I smile” - Thich_Nhat_Hanh

1 0

Bug in 1.9?
by Neil Girdhar 23 Oct '14

23 Oct '14

Hello, Is this desired behaviour or a regression or a bug? http://stackoverflow.com/questions/26497656/how-do-i-align-a-numpy-record-a… Thanks, Neil

3 3

numpy log and exceptions
by Daniel Hyams 22 Oct '14

22 Oct '14

I would have thought that this snippet would raise an exception: import numpy numpy.seterr(all='raise') a = numpy.array([1.0,0.0,-1.0]) b = numpy.log(a) I get as a result (in b): [0, -Inf, NaN] It's basically the same issue as: http://numpy-discussion.10968.n7.nabble.com/numpy-log-does-not-raise- exceptions-td5854.html Except that I have explicitly set the error flags to raise exceptions. It works fine for sqrt(), but not for log(). I've checked numpy 1.4.0 and 1.7.1 and both have the same behavior. Is there a way to force the log (and log10) function to raise an exception on invalid input?

4 5