Mailman 3 June 2012 - NumPy-Discussion

"Symbol table not found" compiling numpy from git repository on Windows
by John Salvatier 18 Jul '12

18 Jul '12

Hello, I'm trying to compile numpy on Windows 7 using the command: "python setup.py config --compiler=mingw32 build" but I get an error about a symbol table not found. Anyone know how to work around this or what to look into? building library "npymath" sources Building msvcr library: "C:\Python26\libs\libmsvcr90.a" (from C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll) objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python26\lib\distutils\command\build.py", line 134, in run self.run_command(cmd_name) File "C:\Python26\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 646, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "C:\Python26\lib\distutils\command\config.py", line 257, in try_link self._check_compiler() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\config.py", line 45, in _check_compiler old_config._check_compiler(self) File "C:\Python26\lib\distutils\command\config.py", line 107, in _check_compiler dry_run=self.dry_run, force=1) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\ccompiler.py", line 560, in new_compiler compiler = klass(None, dry_run, force) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 94, in __init__ msvcr_success = build_msvcr_library() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 362, in build_msvcr_library generate_def(dll_file, def_file) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 282, in generate_def raise ValueError("Symbol table not found") ValueError: Symbol table not found Thank you, John

4 5

Would a patch with a function for incrementing an array with advanced indexing be accepted?
by John Salvatier 17 Jul '12

17 Jul '12

Hello, If you increment an array using advanced indexing and have repeated indexes, the array doesn't get repeatedly incremented, http://comments.gmane.org/gmane.comp.python.numeric.general/50291. I wrote a C function that does incrementing with repeated indexes correctly. The branch is here (https://github.com/jsalvatier/numpy see the last two commits). Would a patch with a cleaned up version of a function like this be accepted into numpy? I'm not experienced writing numpy C code so I'm sure it still needs improvement. If you compile and install that branch, you can test the code using: from numpy import * from numpy.core.multiarray import index_increment a = arange(12).reshape((3,4)) index = ([1,1,2,0], [0,0,2,3]) vals = [50,50., 30.,16] b = index_increment(a, index, vals) print b """ should print out: [[ 0. 1. 2. 19.] [ 104. 5. 6. 7.] [ 8. 9. 40. 11.]] """ Cheers, John

3 11

Matrix rank default tolerance - is it too low?
by Matthew Brett 15 Jul '12

15 Jul '12

Hi, I noticed that numpy.linalg.matrix_rank sometimes gives full rank for matrices that are numerically rank deficient: If I repeatedly make random matrices, then set the first column to be equal to the sum of the second and third columns: def make_deficient(): X = np.random.normal(size=(40, 10)) deficient_X = X.copy() deficient_X[:, 0] = deficient_X[:, 1] + deficient_X[:, 2] return deficient_X then the current numpy.linalg.matrix_rank algorithm returns full rank (10) in about 8 percent of cases (see appended script). I think this is a tolerance problem. The ``matrix_rank`` algorithm does this by default: S = spl.svd(M, compute_uv=False) tol = S.max() * np.finfo(S.dtype).eps return np.sum(S > tol) I guess we'd we want the lowest tolerance that nearly always or always identifies numerically rank deficient matrices. I suppose one way of looking at whether the tolerance is in the right range is to compare the calculated tolerance (``tol``) to the minimum singular value (``S.min()``) because S.min() in our case should be very small and indicate the rank deficiency. The mean value of tol / S.min() for the current algorithm, across many iterations, is about 2.8. We might hope this value would be higher than 1, but not much higher, otherwise we might be rejecting too many columns. Our current algorithm for tolerance is the same as the 2-norm of M * eps. We're citing Golub and Van Loan for this, but now I look at our copy (p 261, last para) - they seem to be suggesting using u * |M| where u = (p 61, section 2.4.2) eps / 2. (see [1]). I think the Golub and Van Loan suggestion corresponds to: tol = np.linalg.norm(M, np.inf) * np.finfo(M.dtype).eps / 2 This tolerance gives full rank for these rank-deficient matrices in about 39 percent of cases (tol / S.min() ratio of 1.7) We see on p 56 (section 2.3.2) that: m, n = M.shape 1 / sqrt(n) . |M|_{inf} <= |M|_2 So we can get an upper bound on |M|_{inf} with |M|_2 * sqrt(n). Setting: tol = S.max() * np.finfo(M.dtype).eps / 2 * np.sqrt(n) gives about 0.5 percent error (tol / S.min() of 4.4) Using the Mathworks threshold [2]: tol = S.max() * np.finfo(M.dtype).eps * max((m, n)) There are no false negatives (0 percent rank 10), but tol / S.min() is around 110 - so conservative, in this case. So - summary - I'm worrying our current threshold is too small, letting through many rank-deficient matrices without detection. I may have misread Golub and Van Loan, but maybe we aren't doing what they suggest. Maybe what we could use is either the MATLAB threshold or something like: tol = S.max() * np.finfo(M.dtype).eps * np.sqrt(n) - so 2 * the upper bound for the inf norm = 2 * |M|_2 * sqrt(n) . This gives 0 percent misses and tol / S.min() of 8.7. What do y'all think? Best, Matthew [1] http://matthew-brett.github.com/pydagogue/floating_error.html#machine-epsil… [2] http://www.mathworks.com/help/techdoc/ref/rank.html Output from script: Percent undetected current: 9.8, tol / S.min(): 2.762 Percent undetected inf norm: 39.1, tol / S.min(): 1.667 Percent undetected upper bound inf norm: 0.5, tol / S.min(): 4.367 Percent undetected upper bound inf norm * 2: 0.0, tol / S.min(): 8.734 Percent undetected MATLAB: 0.0, tol / S.min(): 110.477 <script> import numpy as np import scipy.linalg as npl M = 40 N = 10 def make_deficient(): X = np.random.normal(size=(M, N)) deficient_X = X.copy() if M > N: # Make a column deficient deficient_X[:, 0] = deficient_X[:, 1] + deficient_X[:, 2] else: # Make a row deficient deficient_X[0] = deficient_X[1] + deficient_X[2] return deficient_X matrices = [] ranks = [] ranks_inf = [] ranks_ub_inf = [] ranks_ub2_inf = [] ranks_mlab = [] tols = np.zeros((1000, 6)) for i in range(1000): m = make_deficient() matrices.append(m) # The SVD tolerances S = npl.svd(m, compute_uv=False) S0 = S.max() # u in Golub and Van Loan == numpy eps / 2 eps = np.finfo(m.dtype).eps u = eps / 2 # Current numpy matrix_rank algorithm ranks.append(np.linalg.matrix_rank(m)) # Which is the same as: tol_s0 = S0 * eps # ranks.append(np.linalg.matrix_rank(m, tol=tol_s0)) # Golub and Van Loan suggestion tol_inf = npl.norm(m, np.inf) * u ranks_inf.append(np.linalg.matrix_rank(m, tol=tol_inf)) # Upper bound of |X|_{inf} tol_ub_inf = tol_s0 * np.sqrt(N) / 2 ranks_ub_inf.append(np.linalg.matrix_rank(m, tol=tol_ub_inf)) # Times 2 fudge tol_ub2_inf = tol_s0 * np.sqrt(N) ranks_ub2_inf.append(np.linalg.matrix_rank(m, tol=tol_ub2_inf)) # MATLAB algorithm tol_mlab = tol_s0 * max(m.shape) ranks_mlab.append(np.linalg.matrix_rank(m, tol=tol_mlab)) # Collect tols tols[i] = tol_s0, tol_inf, tol_ub_inf, tol_ub2_inf, tol_mlab, S.min() rel_tols = tols / tols[:, -1][:, None] fmt = 'Percent undetected %s: %3.1f, tol / S.min(): %2.3f' max_rank = min(M, N) for name, ranks, mrt in zip( ('current', 'inf norm', 'upper bound inf norm', 'upper bound inf norm * 2', 'MATLAB'), (ranks, ranks_inf, ranks_ub_inf, ranks_ub2_inf, ranks_mlab), rel_tols.mean(axis=0)[:5]): pcnt = np.sum(np.array(ranks) == max_rank) / 1000. * 100 print fmt % (name, pcnt, mrt) </script>

5 18

Preferring gfortran over g77 on OS X and other distributions?
by Aron Ahmadia 06 Jul '12

06 Jul '12

f2py, by default, seems to prefer g77 (no longer maintained, deprecated, speedy, doesn't support Fortran 90 or Fortran 95) over gfortran (maintained, slower, Fortran 90 and Fortran 95 support). This causes problems when we try to compile Fortran 90 extensions using f2py on platforms where both g77 and gfortran are installed without manually switching the compiler's flags. It is a very minor edit to the fcompiler/__init__.py file to prefer gfortran over g77 on OS X, and I can think of almost no reason not to do so, since the Vectorize framework (OS X tuned LAPACK/BLAS) appears to be ABI compatible with gfortran. I am not sure what the situation is on the distributions that numpy is trying to support, but my feeling is that g77 should not be preferred when gfortran is available. Regards, Aron

2 5

PyArray_FILLWBYTE dangerous doc
by Frédéric Bastien 06 Jul '12

06 Jul '12

Hi, The doc of PyArray_FILLWBYTE here http://docs.scipy.org/doc/numpy/reference/c-api.array.html is this PyArray_FILLWBYTE(PyObject* obj, int val) Fill the array pointed to by obj —which must be a (subclass of) bigndarray—with the contents of val (evaluated as a byte). In the code, what it does is call memset: numpy/core/include/numpy/ndarrayobject.h #define PyArray_FILLWBYTE(obj, val) memset(PyArray_DATA(obj), val, \ PyArray_NBYTES(obj)) This make it ignore completely the strides! So the easy fix would be to update the doc, the real fix is to test the contiguity before calling memset, if not contiguous, call something else appropriate. Fred

1 1

Combined versus separate build
by Nathaniel Smith 02 Jul '12

02 Jul '12

Currently the numpy build system(s) support two ways of building numpy: either by compiling a giant concatenated C file, or by the more conventional route of first compiling each .c file to a .o file, and then linking those together. I gather from comments in the source code that the former is the traditional method, and the latter is the newer "experimental" approach. It's easy to break one of these builds without breaking the other (I just did this with the NA branch, and David had to clean up after me), and I don't see what value we really get from having both options -- it seems to just double the size of the test matrix without adding value. Now that the separate build seems to be fully supported, maybe it's time to finish the "experiment" and pick one approach to support going forward? I guess the arguments for each would be: - The monolithic build in principle allows for some extra intra-procedural optimization. I won't believe this until I see benchmarks, though; numpy doesn't have a lot of tiny inline-able function calls or anything like that. - The separate build is probably more convenient for developers, allowing faster rebuilds. Numpy builds fast enough for me that I'm not too worried about which approach we use, but it definitely seems worthwhile to reduce the number of configurations we have to support one way or the other. -N

3 13

Bug in pickling an ndarray?
by Daniel Hyams 01 Jul '12

01 Jul '12

I am having trouble pickling (and then unpickling) an ndarray. Upon unpickling, the "base" attribute of the ndarray is set to some very strange string ("base" was None when the ndarray was pickled, so it should remain None). I have tried on various platforms and versions of numpy, with inconclusive results: # tested: Linux (Suse 11.1), numpy 1.5.1 BUG # Linux (Suse 11,0), numpy 1.6.1 OK # Linux (Mint Debian), numpy 1.6.1 BUG # Linux (Mint Debian), numpy 1.6.2 BUG # OSX (Snow Leopard), numpy 1.5.1rc1 BUG # OSX (Snow Leopard), numpy 1.6.2 BUG # Windows 7, numpy 1.4.1 OK I have attached a script below that can be used to check for the problem; I suppose that this is a bug report, unless I'm doing something terribly wrong or my expectations for the base attribute are off. ---------------- cut here --------------------------------- # this little demo shows a problem with the base attribute of an ndarray, when # pickling. Before pickling, dset.base is None, but after pickling, it is some # strange string. import cPickle as pickle import numpy print numpy.__version__ #import pickle dset = numpy.ones((2,2)) print "BEFORE PICKLING" print dset print "base = ",dset.base print dset.flags # pickle. s = pickle.dumps(dset) # now unpickle. dset = pickle.loads(s) print "AFTER PICKLING AND THEN IMMEDIATELY UNPICKLING" print dset print "base = ",dset.base print dset.flags -- Daniel Hyams dhyams(a)gmail.com

4 5

Meta: help, devel and stackoverflow
by srean 30 Jun '12

30 Jun '12

Hi List, this has been brought up several times, and the response has been generally positive but it has fallen through the cracks. So here are a few repeat requests. Am keeping it terse just for brevity i) Split the list into [devel] and [help] and as was mentioned recently [rant/flame]: some request for help get drowned out during active development related discussions and simple help requests pollutes more urgent development related matters. ii) Stackoverflow like site for help as well as for proposals. The silent majority has been referred to a few times recently. I suspect there does exist many lurkers on the list who do prefer one discussed solution over the other but for various reasons do not break out of their lurk mode to send a mail saying "I prefer this solution". Such an interface will also help in keeping track of the level of support as compared to mails that are larges hunks of quoted text with a line or two stating ones preference or seconding a proposal. One thing I have learned from traffic accidents is that if one asks for a help of the assembled crowd, no one knows how to respond. On the other hand if you say "hey there in a blue shirt could you get some water" you get instant results. So pardon me for taking the presumptuous liberty to request Travis to please set it up or delegate. Splitting the lists shouldn't be hard work, setting up overflow might be more work in comparison. Best -- srean

16 53

NumPy 1.7 release delays
by Travis Oliphant 30 Jun '12

30 Jun '12

Hey all, After some more investigation, I'm not optimistic that we will be able to get a 1.7 release out before SciPy. I would like to get a beta release out by SciPy (or even an rc1 release). But, given the number of code changes and differences between 1.5.x and 1.7, I think we will need an extended beta release stage for 1.7 that will allow as many users as possible to try out the new code base and report back any regressions or backward incompatibilities that need to be fixed before the final release. The fundamental rule I think we have is that "code depending on NumPy that worked with 1.5.x should continue to work with 1.7 without alterations required by the user" This does not mean we can't add new APIs or deprecate old APIs --- but I think that we do have to be much more careful about when deprecated APIs become unavailable. There is a lot of code that assumes the current API. Both code that is in released packages and code that is in "unreleased packages" which we are not even aware of. I don't want to finalize the 1.7 release until we get enough feedback from end-users about the impact of all the changes. This will likely take a longer beta-release period than usual: certainly not until after SciPy where we will make a concerted effort to get people to try the new 1.7 beta and report back on the impact on their code-base. Ondrej is helping out on this effort which I really appreciate. Other people who have time to help with the release effort --- especially in fixing regressions will be greatly appreciated. We are also using this time to 1) setup Continuous Integration services for NumPy using both Jenkins and Travis-CI and 2) migrate the issue tracker to github. Ondrej is heading up #1 and Ray Jones is heading up #2. Please coordinate with them if you'd like to help out on any of those areas. Thanks, -Travis

14 24

[ANN] IPython 0.13 is officially out!
by Fernando Perez 30 Jun '12

30 Jun '12

Hi all, on behalf of the IPython development team, and just in time for the imminent Debian freeze and SciPy 2012, I'm thrilled to announce, after an intense 6 months of work, the official release of IPython 0.13. This version contains several major new features, as well as a large amount of bug and regression fixes. The previous version (0.12) was released on December 19 2011, so in this development cycle we had: - ~6 months of work. - 373 pull requests merged. - 742 issues closed (non-pull requests). - contributions from 62 authors. - 1760 commits. - a diff of 114226 lines. This means that we closed a total of 1115 issues over 6 months, for a rate of almost 200 issues closed per month and almost 300 commits per month. We are very grateful to all of you who have contributed so enthusiastically to the project and have had the patience of pushing your contributions through our often lengthy review process. We've also welcomed several new members to the core IPython development group: Jörgen Stenarson (@jstenar - this really was an omission as Jörgen has been our Windows expert for a long time) and Matthias Bussonier (@Carreau), who has been very active on all fronts of the project. *Highlights* There is too much new work to write up here, so we refer you to our full What's New document (http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html) for the full details. But the main highlights of this release are: * Brand new UI for the notebook, with major usability improvements (real menus, toolbar, and much more) * Manage all your parallel cluster configurations from the notebook with push-button simplicity (cluster start/stop with one button). * Cell magics: commands prefixed with %% apply to an entire cell. We ship with many cell magics by default, including timing, profiling, running cells under bash, Perl and Ruby as well as magics to interface seamlessly with Cython, R and Octave. * The IPython.parallel tools have received many fixes, optimizations, and a number of API improvements to make writing, profiling and debugging parallel codes with IPython much easier. * We have unified our interactive kernels (the basic ipython object you know and love) with the engines running in parallel, so that you can now use all IPython special tricks in parallel too. And you can connect a console or qtconsole to any parallel engine for direct, interactive execution, plotting and debugging in a cluster. *Downloads* Download links and instructions are at: http://ipython.org/download.html And IPython is also on PyPI: http://pypi.python.org/pypi/ipython Those contain a built version of the HTML docs; if you want pure source downloads with no docs, those are available on github: Tarball: https://github.com/ipython/ipython/tarball/rel-0.13 Zipball: https://github.com/ipython/ipython/zipball/rel-0.13 Please see our release notes for the full details on everything about this release: http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html As usual, if you find any other problem, please file a ticket --or even better, a pull request fixing it-- on our github issues site (https://github.com/ipython/ipython/issues). Many thanks to all who contributed! Fernando, on behalf of the IPython development team. http://ipython.org

1 0