Mailman 3 August 2010 - NumPy-Discussion

NumPy segfault when running tests (2.7, 1.4.1)
by David Warde-Farley Aug. 5, 2010

Aug. 5, 2010

This was using the Intel MKL/icc to compile NumPy and also icc to compile Python on a shared cluster, but I don't think that's relevant given where the segmentation fault occurs... gpc-f104n084-$ gdb python GNU gdb Fedora (6.8-27.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... (gdb) (gdb) run Starting program: /home/dwf/pkg/bin/python [Thread debugging using libthread_db enabled] Python 2.7 (r27:82500, Aug 3 2010, 15:33:51) [GCC Intel(R) C++ gcc 4.1 mode] on linux2 Type "help", "copyright", "credits" or "license" for more information. [New Thread 0x2b76149111a0 (LWP 23034)] >>> import numpy numpy>>> numpy.test() Running unit tests for numpy NumPy version 1.4.1 NumPy is installed in /home/dwf/pkg/lib/python2.7/site-packages/numpy Python version 2.7 (r27:82500, Aug 3 2010, 15:33:51) [GCC Intel(R) C++ gcc 4.1 mode] nose version 0.11.4 ................................................................................................................................... Program received signal SIGSEGV, Segmentation fault. 0x0000000000500916 in PyErr_WarnEx (category=0x11eb6c54, text=0x5f90c0 "PyOS_ascii_strtod and PyOS_ascii_atof are deprecated. Use PyOS_string_to_double instead.", stack_level=0) at Python/_warnings.c:719 719 res = do_warn(message, category, stack_level); (gdb)

2 2

deprecations for 1.5
by Ralf Gommers Aug. 5, 2010

Aug. 5, 2010

Hi all, I'm looking at what needs to be deprecated/removed and have a few questions: 1. ma.core has some deprecations without version info: make_mask: flag keyword MaskedArray.flag MaskedArray.raw_data allclose: fillvalue keyword Should these be removed, and if not can someone provide a version number for when that does need to happen? 2. correlate: old_behavior keyword should be removed. Is it correct that that means (besides changes in core/numeric.py) removing multiarray.correlate code in multiarraymodule.c and renaming correlate2 to correlate there? 3. Anything else that needs to be removed/deprecated besides what is noted in the 1.4.0 release notes and ticket 1431? Thanks, Ralf

2 1

update on the transition to git/github
by Jarrod Millman Aug. 5, 2010

Aug. 5, 2010

Hello all, On May 26th, I sent an email titled "curious about how people would feel about moving to github." While there were a few concerns raised, everyone was generally positive and were mainly concerned that this transition would need to be done carefully with clear workflow instructions and an clearly marked master repository. Since then David Cournapeau has done a lot of work looking into how to make the transition go as smoothly as possible. He has written a script to convert the svn repository to a git repository. And I've registered a numpy organization account on github. Over the next week, David, Stefan, and I will set things up so that everyone can see how things will work after the transition to git. David will convert the current trunk to a git repository and put it on the github site. We will write up instructions on how to use git and the github site. Everyone who wants to can get a github account and test out the workflow. At that point everyone can provide feedback and we can decide if we are ready to move forward. If we are ready to move forward, we will set up a date for the transition. On that date, we would turn off the old subversion account and create a new git repository which would from the point forward be the new master branch. If the transition to git and github for numpy goes smoothly, we will turn our attention to scipy. During the SciPy conference in Austin, we had a Birds-of-a-Feather to discuss the transition to git and github. [[ Here is a picture of the git/github BOF (several people joined the discussion later including Travis Oliphant and Fernando Perez): http://www.flickr.com/photos/irees/4750650877/sizes/l/in/set-72157624272131… ]] At the end of the discussion there was a general consensus that it was time to make the move. Several questions and concerns were raised: 1. Since there are many possible workflows, it is important to clearly document are proposed workflow. This document should provide simple cut-and-paste commands necessary to get developers up and running with git. 2. The question was raised about how to handle bug reports. It was pointed out that while our current trac bug report system isn't perfect, it does work and people are used to it. We decided to keep our existing trac instance and integrate it with the github site. Potentially moving from trac to another system like redmine was brought up, but most people felt it was better to only change one thing at a time (and besides that no one volunteered to do the work necessary to move to a new bug tracking system). 3. Since Ralf Gommers is in the middle of making a release, did he want us to delay any transition preparation until he was finished. This is Ralf's response when Stefan van der Walt contacted him: "Thanks for asking! For me the sooner the better, I do everything with git and haven't touched svn since I discovered Mercurial while writing my thesis (and that feels like a long long time ago)." When Stefan contacted Ralf, Ralf raised the following additional concern: 4. "The two things I haven't seen a good solution for are the svn/externals in scipy which pulls in doc/sphinxext from numpy, and the vendor branch." In response, Stefan asked whether submodules would provide a solution. David Cournapeau responded to Stefan stating "submodules are very awkward to use." Then David added in response to Ralf's original query: "For vendor, it will be a separate repo, and there is no need for synchronization, so that's easy to deal with. For the sphinx extension, I would just merge with the subtree strategy from time to time from a separate repository. "That's what I do for bento + yaku: yaku has its own repo, and when I update the copy in bento, I just use git merge -s subtree --no-squash, so everything is updated in one commit. "The big advantage is that there is no need to even be aware of the second repo for users (git clone will bring everything), and there is no chance of screwing things up: http://book.git-scm.com/5_advanced_branching_and_merging.html" Best, Jarrod

6 6

Quick array value assignment based on common values
by PHobson＠Geosyntec.com Aug. 5, 2010

Aug. 5, 2010

Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul

4 8

distutils issue - python 3.1 on windows
by Ralf Gommers Aug. 4, 2010

Aug. 4, 2010

Hi, I'm trying to get building to work with Python 3.1 under Wine on OS X. The first thing you run into is a python distutils problem, which is fixed by replacing line 379 of cygwinccompiler.py with result = RE_VERSION.search(str(out_string)) The next thing I run into is a numpy.distutils issue (complete build output below email): File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\mingw32ccompiler.py", line 177, in link func(*args[:func.__func__.__code__.co_argcount]) AttributeError: 'function' object has no attribute '__func__' What I think that line is for is to test what kind of arguments can be passed to 'func', but I'm really not sure. A comment would have been helpful, that must be the most obscure line of Python code I've ever seen:) Does anyone have any idea what causes this error? Thanks, Ralf <... 2to3 output ...> F2PY Version 1 Running from numpy source directory.numpy\core\code_generators\generate_umath.py:117: DeprecationWarning: string.maketrans is deprecated, use bytes.maketrans instead bytes(string.ascii_uppercase, "ascii")) blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in C:\Python31\lib libraries mkl,vml,guide not found in C:\ libraries mkl,vml,guide not found in C:\Python31\libs NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Python31\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Python31\libs NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in C:\Python31\lib libraries f77blas,cblas,atlas not found in C:\ libraries f77blas,cblas,atlas not found in C:\Python31\libs NOT AVAILABLE Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\system_info.py:1399: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: FOUND: libraries = ['blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in C:\Python31\lib libraries mkl,vml,guide not found in C:\ libraries mkl,vml,guide not found in C:\Python31\libs NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Python31\lib libraries lapack_atlas not found in C:\Python31\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Python31\libs libraries lapack_atlas not found in C:\Python31\libs <class 'numpy.distutils.system_info.atlas_threads_info'> NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in C:\Python31\lib libraries lapack_atlas not found in C:\Python31\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries f77blas,cblas,atlas not found in C:\Python31\libs libraries lapack_atlas not found in C:\Python31\libs <class 'numpy.distutils.system_info.atlas_info'> NOT AVAILABLE Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\system_info.py:1306: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 could not resolve pattern in '': '*.txt' non-existing path in '': 'COMPATIBILITY' non-existing path in '': 'site.cfg.example' running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build\src.win32-3.1 creating build\src.win32-3.1\numpy creating build\src.win32-3.1\numpy\distutils building library "npymath" sources gcc -dumpversion ld -v dllwrap --version customize GnuFCompiler Found executable C:\MinGW\bin\g77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\MinGW\bin\g77.exe customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Python31\include -IC:\Python31\PC -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Python31\include -IC:\Python31\PC -c _configtest.c -o _configtest.o Found executable C:\MinGW\bin\gcc.exe Traceback (most recent call last): File "setup.py", line 210, in <module> setup_package() File "setup.py", line 203, in setup_package configuration=configuration ) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python31\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Python31\lib\distutils\dist.py", line 919, in run_commands self.run_command(cmd) File "C:\Python31\lib\distutils\dist.py", line 938, in run_command cmd_obj.run() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python31\lib\distutils\command\build.py", line 128, in run self.run_command(cmd_name) File "C:\Python31\lib\distutils\cmd.py", line 315, in run_command self.distribution.run_command(command) File "C:\Python31\lib\distutils\dist.py", line 938, in run_command cmd_obj.run() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 678, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "C:\Python31\lib\distutils\command\config.py", line 248, in try_link libraries, library_dirs, lang) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\config.py", line 149, in _link libraries, library_dirs, lang)) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\config.py", line 89, in _wrap_method ret = mth(*((self,)+args)) File "C:\Python31\lib\distutils\command\config.py", line 145, in _link target_lang=lang) File "C:\Python31\lib\distutils\ccompiler.py", line 808, in link_executable debug, extra_preargs, extra_postargs, None, target_lang) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\mingw32ccompiler.py", line 177, in link func(*args[:func.__func__.__code__.co_argcount]) AttributeError: 'function' object has no attribute '__func__' Captured Task Output: --------------------- ---> pavement.bdist_superpack Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", line 164, in _run_task return do_task() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", line 161, in do_task return func(**kw) File "pavement.py", line 213, in bdist_superpack bdist_wininst_arch(pyver, 'nosse') File "pavement.py", line 196, in bdist_wininst_arch _bdist_wininst(pyver, SITECFG[arch]) File "pavement.py", line 266, in _bdist_wininst subprocess.check_call(cmd, env=cfg_env) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 488, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['wine', '/Users/rgommers/.wine/drive_c/Python31/python.exe', 'setup.py', 'build', '-c', 'mingw32', 'bdist_wininst']' returned non-zero exit status 1

4 7

ndarray.resize that preserve view content ?
by Antoine Dechaume Aug. 4, 2010

Aug. 4, 2010

Hi, given A=empty([10,10]), I would like to keep A[:5,:5] as a contiguous memory segment. How to do it efficiently? Thanks.

3 7

Python 3.2 crashes numpy with undefined symbol PyCObject_AsVoidPtr
by Bruce Southey Aug. 4, 2010

Aug. 4, 2010

Hi, Since I was testing the 1.5 beta, I also tested the alpha release of Python 3.2 on Linux 64-bit (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)). While my other Python versions passed the tests (once I copied the necessary tools directory for Python3.1), Python3.2 has problems so I created ticket 1571: http://projects.scipy.org/numpy/ticket/1571 The minor change is that string.maketrans function, depreciated in Python 3.1, has been removed. http://docs.python.org/dev/py3k/whatsnew/3.2.html However the more serious problem is that both 1.5 beta and the current svn version crashed: $ python3.2 -c "import numpy; numpy.test()" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.2/site-packages/numpy/__init__.py", line 136, in <module> from . import add_newdocs File "/usr/local/lib/python3.2/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python3.2/site-packages/numpy/lib/__init__.py", line 4, in <module> from .type_check import * File "/usr/local/lib/python3.2/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/usr/local/lib/python3.2/site-packages/numpy/core/__init__.py", line 5, in <module> from . import multiarray ImportError: /usr/local/lib/python3.2/site-packages/numpy/core/multiarray.so: undefined symbol: PyCObject_AsVoidPtr This warning that does not appears in Python3.1: In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:28: numpy/core/src/multiarray/convert_datatype.c: In function ‘PyArray_GetCastFunc’: numpy/core/src/multiarray/convert_datatype.c:106: warning: implicit declaration of function ‘PyCObject_AsVoidPtr’ numpy/core/src/multiarray/convert_datatype.c:106: warning: assignment makes pointer from integer without a cast Bruce

1 1

Tabular: Importing data from file
by Robert Faryabi Aug. 3, 2010

Aug. 3, 2010

Hi there; I'm using Tabular Package for manipulating tab-delimited data. There is a small problem that I cannot get my head around it. When I construct my tabarray from file, the blank fields are replaced by "nan". Does any one knows how to just keep them as empty string (ie. ' ')? Thanks,

2 1

Re: [Numpy-discussion] Numexpr: erfc function
by Francesc Alted Aug. 3, 2010

Aug. 3, 2010

Hi Till, 2010/8/3, Till Stensitzki <mail.till(a)gmx.de>: > Hello, > I am sorry for contacting you directly, i tried but i didn't found > something like a numexpr mailinglist. > Fist of all, thanks for this modul, it really helps to speed up some of > my fitting rounties! > Now for my qustion: Is there any way to add the complementary error > function (erfc in scipy and c) to the supported functions? Yes, it is possible to add more functions to Numexpr indeed. Just start looking at how support for some POSIX functions have been added in file ``missing_posix_functions.inc``, and try to replicate the support code for, for example, `log1p()`. It is not complicated, just time-consuming. If you end with the suport, just send me the patch and I'll add it to the repo. Regarding the mailing list, I don't really think Numexpr would need one for now. I suppose that discussion for Numexpr is welcome in NumPy/SciPy list (and the PyTables list can certainly also be used). Luck! -- Francesc Alted

1 0

Tensor-Like Outer Product Question
by matt_in_nyc Aug. 3, 2010

Aug. 3, 2010

I am trying to perform the following operation: X is an m by n matrix, and I want to store outer products of the form Y[i] = numpy.outer(X[i,:], X[i,:]), leading to the relation Y[i,j,k] = X[i,j]*X[i,k] for i = 0,...,m-1; j,k = 0,...,n-1. I am trying to think of how to do this using tensordot, but so far I am finding no inspiration. Some far, my only solution has been to loop over i Y = numpy.empty([m,n,n]) for i in range(m): Y[i] = numpy.outer(X[i,:], X[i,:]) but this is fairly slow as for my dataset, m is of order 10^7 or 10^8 and n is around 20. Any help on how to vectorize/tensorize this operation to avoid the for loop would be much appreciated. -- View this message in context: http://old.nabble.com/Tensor-Like-Outer-Product-Question-tp29339226p2933922… Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 1