Mailman 3 January 2013 - NumPy-Discussion

Insights / lessons learned from NumPy design
by Mike Anderson Jan. 14, 2013

Jan. 14, 2013

Hello all, In the Clojure community there has been some discussion about creating a common matrix maths library / API. Currently there are a few different fledgeling matrix libraries in Clojure, so it seemed like a worthwhile effort to unify them and have a common base on which to build on. NumPy has been something of an inspiration for this, so I though I'd ask here to see what lessons have been learned. We're thinking of a matrix library with roughly the following design (subject to change!) - Support for multi-dimensional matrices (but with fast paths for 1D vectors and 2D matrices as the common cases) - Immutability by default, i.e. matrix operations are pure functions that create new matrices. There could be a "backdoor" option to mutate matrices, but that would be unidiomatic in Clojure - Support for 64-bit double precision floats only (this is the standard float type in Clojure) - Ability to support multiple different back-end matrix implementations (JBLAS, Colt, EJML, Vectorz, javax.vecmath etc.) - A full range of matrix operations. Operations would be delegated to back end implementations where they are supported, otherwise generic implementations could be used. Any thoughts on this topic based on the NumPy experience? In particular would be very interesting to know: - Features in NumPy which proved to be redundant / not worth the effort - Features that you wish had been designed in at the start - Design decisions that turned out to be a particularly big mistake / success Would love to hear your insights, any ideas+advice greatly appreciated! Mike.

7 18

How many build systems do we need?
by Charles R Harris Jan. 13, 2013

Jan. 13, 2013

Hi All, In the continuing proposal for cleanups, note that we currently support three (3!) build systems, distutils, scons, and bento. That's a bit much to maintain when contemplating changes, and scons and bento both have external dependencies. Can we dispense with any of these? Thoughts? Chuck

4 11

return index of maximum value in an array easily?
by Chao YUE Jan. 12, 2013

Jan. 12, 2013

Dear all, Are we going to consider returning the index of maximum value in an array easily without calling np.argmax and np.unravel_index consecutively? I saw few posts in mailing archive and stackover flow on this, when I tried to return the index of maximum value of 2d array. It seems that I am not the first to be confused by this. http://stackoverflow.com/questions/11377028/getting-index-of-numpy-ndarray http://old.nabble.com/maximum-value-and-corresponding-index-td24834930.html http://stackoverflow.com/questions/5469286/how-to-get-the-index-of-a-maximu… http://stackoverflow.com/questions/4150542/determine-index-of-highest-value… cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 4

Building Numpy 1.6.2 for Python 3.3 on Windows
by klo Jan. 11, 2013

Jan. 11, 2013

Hi, I run `python3 setup.py config` and then python3 setup.py build --compiler=mingw32 but it picks that I have MSVC 10 and complains about manifests. Why, or even better, how to compile with available MinGW compilers? Here is log: ======================================== C:\src\numpy-1.6.2>python3 setup.py --compiler=mingw32 Converting to Python3 via 2to3... Running from numpy source directory.usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: option --compiler not recognized C:\src\numpy-1.6.2>python3 setup.py build --compiler=mingw32 Converting to Python3 via 2to3... F2PY Version 2 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_rt'] library_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/lib/ia32'] include_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/include'] define_macros = [('SCIPY_MKL_H', None)] FOUND: libraries = ['mkl_rt'] library_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/lib/ia32'] include_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/include'] define_macros = [('SCIPY_MKL_H', None)] non-existing path in 'numpy\\lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_rt'] library_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/lib/ia32'] include_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/include'] define_macros = [('SCIPY_MKL_H', None)] FOUND: libraries = ['mkl_rt'] library_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/lib/ia32'] include_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/include'] define_macros = [('SCIPY_MKL_H', None)] FOUND: libraries = ['mkl_rt'] library_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/lib/ia32'] include_dirs = ['C:/Progra~1/Intel/Compos~1/mkl/include'] define_macros = [('SCIPY_MKL_H', None)] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources building library "npymath" sources customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelVisualFCompiler Could not locate executable ifort Could not locate executable ifl customize AbsoftFCompiler Could not locate executable f90 customize CompaqVisualFCompiler Could not locate executable DF customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Found executable C:\MinGW\bin\gfortran.exe Found executable C:\MinGW\bin\gfortran.exe Running from numpy source directory.customize Gnu95FCompiler customize Gnu95FCompiler using config Traceback (most recent call last): File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 399, in msvc_manifest_xml fullver = _MSVCRVER_TO_FULLVER[str(maj * 10 + min)] KeyError: '100' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "c:\python33\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\python33\lib\distutils\dist.py", line 917, in run_commands self.run_command(cmd) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "c:\python33\lib\distutils\command\build.py", line 126, in run self.run_command(cmd_name) File "c:\python33\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 694, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "c:\python33\lib\distutils\command\config.py", line 246, in try_link libraries, library_dirs, lang) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\command\config.py", line 146, in _link generate_manifest(self) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 484, in generate_manifest manxml = msvc_manifest_xml(ma, mi) File "C:\src\numpy-1.6.2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 402, in msvc_manifest_xml % (maj, min)) ValueError: Version 10,0 of MSVCRT not supported yet ========================================

4 7

Which Python to use for Mac binaries
by Ondřej Čertík Jan. 11, 2013

Jan. 11, 2013

Hi, Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons: MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], } So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine. Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org? Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6. Also, what is the meaning of the following piece of code in pavement.py: def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build") if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) In particular, the last line gets executed and it then fails with: paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch: diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg") @task def simple_dmg(): and then things work. So an obvious question is --- why do we need to fiddle with LDFLAGS and paths to the exact Python version? Here is a proposed simpler version of the build_mpkg() function: def _build_mpkg(pyver): sh("python setupegg.py bdist_mpkg") Thanks for any tips. Ondrej

5 18

Remove support for numeric and numarray in 1.8
by Charles R Harris Jan. 10, 2013

Jan. 10, 2013

Thoughts? Chuck

5 17

int and long issues
by Mads Ipsen Jan. 10, 2013

Jan. 10, 2013

Hi, I find this to be a little strange: x = numpy.arange(10) isinstance(x[0],int) gives True y = numpy.where(x < 5)[0] isinstance(y[0],int) gives False isinstance(y[0],long) gives True Specs: Python 2.7.2, numpy-1.6.1, Win7, 64 bit Best regards, Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | Gåsebæksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen(a)gmail.com | +----------------------+------------------------------+

2 3

Linear least squares
by Till Stensitz Jan. 10, 2013

Jan. 10, 2013

Hi, i did some profiling and testing of my data-fitting code. One of its core parts is doing some linear least squares, until now i used np.linalg.lstsq. Most of time the size a is (250, 7) and of b is (250, 800). Today i compared it to using pinv manually, to my surprise, it is much faster. I taught, both are svd based? Too check another computer i also run my test on wakari: https://www.wakari.io/nb/tillsten/linear_least_squares Also using scipy.linalg instead of np.linalg is slower for both cases. My numpy and scipy are both from C. Gohlkes website. If my result is valid in general, maybe the lstsq function should be changed or a hint should be added to the documentation. greetings Till

5 6

ANN: first previews of WinPython for Python 3 32/64bit
by Pierre Raybaut Jan. 10, 2013

Jan. 10, 2013

Hi all, I'm pleased to announce that the first previews of WinPython for Python 3 32bit and 64bit are available (WinPython v3.3.0.0alpha1): http://code.google.com/p/winpython/ This first release based on Python 3 required to migrate the following libraries which were only available for Python 2: * formlayout 1.0.12 * guidata 1.6.0dev1 * guiqwt 2.3.0dev1 * Spyder 2.1.14dev Please note that these libraries are still development release. [Special thanks to Christoph Gohlke for patching and building a version of PyQwt compatible with Python 3.3] WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel), and adding shortcuts to Windows Start menu. Enjoy! -Pierre

2 1

ANN: NumPy 1.7.0rc1 release
by Ondřej Čertík Jan. 9, 2013

Jan. 9, 2013

Hi, I'm pleased to announce the availability of the first release candidate of NumPy 1.7.0rc1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/ We have fixed all issues known to us since the 1.7.0b2 release. The only remaining issue is a documentation improvement: https://github.com/numpy/numpy/issues/561 Please test this release and report any issues on the numpy-discussion mailing list. If there are no more problems, we'll release the final version soon. I'll wait at least a week and please write me an email if you need more time for testing. I would like to thank Sebastian Berg, Ralf Gommers, Han Genuit, Nathaniel J. Smith, Jay Bourque, Gael Varoquaux, Mark Wiebe, Matthew Brett, Skipper Seabold, Peter Cock, Charles Harris, Frederic, Gabriel, Luis Pedro Coelho, Pauli Virtanen, Travis E. Oliphant and cgohlke for sending patches and fixes for this release since 1.7.0b2. Cheers, Ondrej P.S. Source code is uploaded to sourceforge, and I'll upload the rest of the Windows and Mac binaries in a few hours as they finish building.

6 7