Mailman 3 July 2012 - NumPy-Discussion

Synonym standards
by Colin J. Williams July 27, 2012

July 27, 2012

7 9

numpy.dot: why second-to-last index
by Christoph Groth July 27, 2012

July 27, 2012

Dear numpy historians, When multiplying two arrays with numpy.dot, the summation is made over the last index of the first argument, and over the *second-to-last* index of the second argument. I wonder why the convention has been chosen like this? The only reason I can think of is that this allows to use GEMM as a building block also for the >2d case. Is this the motivation? However, the actual implementation of numpy.dot uses GEMM only in the 2d x 2d case... Summation over the last index of the first argument and the first index of the second would seem a more obvious choice.

1 0

Bug in numpy.mean() revisited
by Tom Aldcroft July 27, 2012

July 27, 2012

There was a thread in January discussing the non-obvious behavior of numpy.mean() for large arrays of float32 values [1]. This issue is nicely discussed at the end of the numpy.mean() documentation [2] with an example: >>> a = np.zeros((2, 512*512), dtype=np.float32) >>> a[0, :] = 1.0 >>> a[1, :] = 0.1 >>> np.mean(a) 0.546875 >From the docs and previous discussion it seems there is no technical difficulty in choosing a different (higher precision) type for the accumulator using the dtype arg, and in fact this is done automatically for int values. My question is whether there would be any support for doing something more than documenting this behavior. I suspect very few people ever make it below the fold for the np.mean() documentation. Taking the mean of large arrays of float32 values is a *very* common use case and giving the wrong answer with default inputs is really disturbing. I recently had to rebuild a complex science data archive because of corrupted mean values. Possible ideas to stimulate discussion: 1. Always use float64 to accumulate float types that are 64 bits or less. Are there serious performance impacts to automatically using float64 to accumulate float32 arrays? I appreciate this would likely introduce unwanted regressions (sometimes suddenly getting the right answer is a bad thing). So could this be considered for numpy 2.0? 2. Might there be a way to emit a warning if the number of values and the max accumulated value [3] are such that the estimated fractional error is above some tolerance? I'm not even sure if this is a good idea or if there will be howls from the community as their codes start warning about inaccurate mean values. Better idea along this line?? Cheers, Tom [1]: http://mail.scipy.org/pipermail/numpy-discussion/2012-January/059960.html [2]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html [3]: Using the max accumulated value during accumulation instead of the final accumulated value seems like the right thing for estimating precision loss. But this would affect performance so maybe just using the final value would catch many cases.

4 3

Warnings and errors while building numpy 1.6.2 on 64 bit system
by Prakash Joshi July 27, 2012

July 27, 2012

Hi All, I am building numpy 1.6.2 on 64 bit Mac OSX/CentOS and I found many warnings and errors of similar kind, however the numpy passes all the tests. Please let me know if I can consider this build as good? llvm-gcc-4.2: _configtest.c _configtest.c:5: error: size of array Œtest_array¹ is negative _configtest.c:1:20: error: endian.h: No such file or directory _configtest.c:7: error: ŒSIZEOF_LONGDOUBLE¹ undeclared (first use in this function) _configtest.c:1: warning: conflicting types for built-in function Œexp¹ numpy/core/src/npymath/ieee754.c.src:260: warning: implicit conversion shortens 64-bit value into a 32-bit value _configtest.c:5: warning: function declaration isn¹t a prototype _configtest.c:4: warning: conflicting types for built-in function Œfabs¹ _configtest.c:8: warning: control reaches end of non-void function numpy/core/src/umath/loops.c.src:1403: warning: ŒFLOAT_ldexp_long¹ defined but not used numpy/core/src/private/lowlevel_strided_loops.h:36: warning: ŒPyArray_FreeStridedTransferData¹ declared Œstatic¹ but never defined numpy/lib/src/_compiled_base.c:120: warning: assignment from incompatible pointer type Best Regards, Prakash Joshi

2 1

Re: [Numpy-discussion] Synonym standards
by Colin J. Williams July 26, 2012

July 26, 2012

Sent from my BlackBerry® PlayBook™ www.blackberry.com ------------------------------ *From:* "Benjamin Root" <ben.root(a)ou.edu> *To:* "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> *Sent:* 26 July 2012 16:57 *Subject:* Re: [Numpy-discussion] Synonym standards On Thu, Jul 26, 2012 at 4:45 PM, Colin J. Williams <fn681(a)ncf.ca> wrote: > It seems that these standards have been adopted, which is good: > > The following import conventions are used throughout the NumPy source and > documentation: > > import numpy as np > import matplotlib as mpl > import matplotlib.pyplot as plt > > Source: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt > > Is there some similar standard for PyLab? > > Thanks, > > Colin W. > > Colin, Typically, with pylab mode of matplotlib, you do: from pylab import * This is essentially equivalent to: from numpy import * from matplotlib.pyplot import * Note that the pylab "module" is actually a part of matplotlib and is a shortcut to provide an environment that is very familiar to Matlab users. Converts are then encouraged to use the imports you mentioned in order to properly utilize python namespaces. I hope that helps! Ben Root

1 0

afte run python setup.py build - error: expected specifier-qualifier-list before ‘complex’
by Carlos Adean July 26, 2012

July 26, 2012

Hi all, After run the command below (numpy 1.6.2): python setup.py build I receive the following message error: gcc: build/src.linux-x86_64-2.5/numpy/core/src/npymath/ieee754.c In file included from numpy/core/src/npymath/ieee754.c.src:8: numpy/core/src/npymath/npy_math_private.h:452: error: expected specifier-qualifier-list before ‘complex’ numpy/core/src/npymath/npy_math_private.h:457: error: expected specifier-qualifier-list before ‘complex’ numpy/core/src/npymath/npy_math_private.h:462: error: expected specifier-qualifier-list before ‘complex’ In file included from numpy/core/src/npymath/ieee754.c.src:8: numpy/core/src/npymath/npy_math_private.h:452: error: expected specifier-qualifier-list before ‘complex’ numpy/core/src/npymath/npy_math_private.h:457: error: expected specifier-qualifier-list before ‘complex’ numpy/core/src/npymath/npy_math_private.h:462: error: expected specifier-qualifier-list before ‘complex’ error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/tawala/packages/python-2.5.4-2/include/python2.5 -Ibuild/src.linux-x86_64-2.5/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-2.5/numpy/core/src/umath -c build/src.linux-x86_64-2.5/numpy/core/src/npymath/ieee754.c -o build/temp.linux-x86_64-2.5/build/src.linux-x86_64-2.5/numpy/core/src/npymath/ieee754.o" failed with exit status 1 CentOS 5.8 python version 2.5.4 numpy 1.6.2 any ideia to solve this? thanks -- -- Carlos Adean IT Team linea.gov.br

1 0

Fwd: Github notifications and trac-to-github migration
by Fernando Perez July 26, 2012

July 26, 2012

Forwarding Jordi's message on the trac migration, he's having issues sending it directly. ---------- Forwarded message ---------- From: Jordi Torrents <jordi.t21(a)gmail.com> Date: 2012/7/26 Subject: Re: [Numpy-discussion] Github notifications and trac-to-github migration To: Aric Hagberg <aric.hagberg(a)gmail.com> Cc: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Hi, Sorry for the delay in reporting feedback on the migration. I was planning to do it when I came back to Barcelona from Scipy but had some domestic problems (broken pipe, partially flooded flat) that got in the way. 2012/7/26 Aric Hagberg <aric.hagberg(a)gmail.com>: > On Wed, Jul 25, 2012 at 3:51 AM, Thouis (Ray) Jones <thouis(a)gmail.com> wrote: >> On Wed, Jul 25, 2012 at 6:53 AM, Fernando Perez <fperez.net(a)gmail.com> wrote: >>> Hi Thouis, >>> >>> On Tue, Jul 24, 2012 at 2:46 PM, Thouis (Ray) Jones <thouis(a)gmail.com> wrote: >>>> I would estimate I'm between a fourth and halfway through the >>>> implementation of the trac-to-github-issues migration code. The work >>>> lives in at https://github.com/thouis/numpy-trac-migration >>> >>> mmh, I would have thought you're farther ahead... Aric Hagberg >>> (@hagberg) and Jordi Torrents (@jtorrents) of NetworkX fame last >>> weekend completed the trac2github migration for nx, and he said he'd >>> only had to make a few improvements to your code. >>> >>> I'm cc'ing Aric here so he can give us more details, but based on the >>> fact that they were able to successfully migrate nx completely to GH, >>> I would have imagined you'd be much, much closer for numpy/scipy. >> >> Perhaps my estimate was low. I hadn't done any work with creating >> issues on github (only extracting them from Trac into a form that maps >> onto github issues), but I expect the PyGithub library >> (https://github.com/jacquev6/PyGithub) helps make the rest of the work >> easier. Glad to hear it helped. >> >>> Their migration looks pretty solid, including all old comments and >>> attachments being correctly linked, cf this one: >>> >>> https://github.com/networkx/networkx/issues/693 >> >> Based on that issue, it looks like I wasn't careful enough in temporal >> ordering of comments, not that it's that critical. >> >> Aric, is the code you ended up using available somewhere? > > We obviously didn't get all of the details quite right when we > migrated the Trac tickets to Github issues. We gave ourselves a day > or so at the SciPy sprints to do it and made a best effort. We would > have never been able to accomplish what we did without the code Ray > wrote. Really there wasn't much more to add and we are happy to share > what we wrote (though it is a hack). I've cc'd Jordi who wrote the > extra code. As Aric says, Ray's code was a life saver for us. We tried several other scripts for the migration before knowing about Ray's code, and all of them failed badly. The only important part missing in Ray's code was the method push() of the class issue in issue.py. We didn't migrate milestones nor labels, so more work is needed to do that. Here is how we implemented the push method: def push(self, repo): github_issue = repo.create_issue(title=self.github.title, body=self.github.body)#, #assignee=self.github.assignee, #milestone=self.github.milestone, #labels=self.github.labels) try: for comment in self.github.comments: github_issue.create_comment(comment) except: print("!!! Error in ticket %s" % self.trac.id) finally: if self.github.state == "closed": github_issue.edit(state='closed') Some formatting in the comments was problematic (more on that below). In our case this affected approximately 5% of the tickets. Github returned an ugly HTTP 500 (Internal Server Error) and all comments coming after a problematic one were lost. A more careful handling of individual comments would have prevented the loss of subsequent comments. Then the move_issues.py code was simply: import trac from ghissues import gh_repo repo = gh_repo() # Manual auth here for issue in trac.issues('data/trac.db'): print("processing issue %s" % issue.trac.id) issue.githubify() issue.push(repo) > Briefly here are some of the issues we encountered - Jordi can > probably add more. > > 1) We made and applied a mapping from the changeset hashes in our old > repository (Mercurial) to the Git changeset hashes. This mostly > worked. We used a regular expression to match the mercurial hash for commits in trac comments and a map generated by hg-git. It turns out that the regular expression was not general enough and we missed some hashes. The quick and dirty code that we used is: m = re.compile('\[(.*)/networkx\]') def load_hg_map(fname='git-mapfile'): hg_map = {} f = open(fname,'r') for row in f: hg_map[row.split(" ")[1].strip()] = row.split(" ")[0].strip() f.close() return hg_map hg_map = load_hg_map() def map_hg(hg_hash): if hg_hash in hg_map: return hg_map[hg_hash] else: return hg_hash def t2g_markup(s): h = m.search(s) if h: hh = h.group(1) s = s.replace(hh,map_hg(hh)) return s.replace('{{{', "'''").replace('}}}', "'''") > 2) We didn't make a mapping between the Git issue numbers and the Trac > issue numbers so many of the cross references were wrong. I recommend > doing that. > > 3) You are right that many messages will get sent out so considering > the impact of that is worthwhile. > > 4) Some of the tickets/comments (maybe 50 of the approx 800 tickets > we converted) had some formatting that broke during conversion. Jordi > might have some thoughts on how to fix that. As I said above, the problematic comments triggered an ugly HTTP 500 (Internal Server Error) from Github. We didn't spend time trying to debug and fix that. Most of the comments that failed in the migration had python code in their code using trac syntax ({{{#!python .... }}}). However not all comments with python code failed, my feeling is that the problematic parts were the prints that used the percent sign (%) but I'm not sure about that. Also, the edited trac comments also triggered a HTTP 500 error, see for instance: https://networkx.lanl.gov/trac/ticket/609#comment:26 The list of almost all issues with problematic comments is (to look at them https://networkx.lanl.gov/trac/ticket/{id} ): 231, 255, 273, 282, 283, 301, 310, 314, 346, 348, 362, 401, 431, 450, 466, 477, 494, 501, 539, 563, 574, 583, 598, 609, 623, 628, 632, 637, 643, 676, 704, 707, 713, 740 Ray, thank you very much for your work, without your code our migration would have been a lot more painful and slow. Salut!

1 0

np.unique for one bi-dimensional array
by Giuseppe Amatulli July 25, 2012

July 25, 2012

Hi, would like to identify unique pairs of numbers in two arrays o in one bi-dimensional array, and count the observation a_clean=array([4,4,5,4,4,4]) b_clean=array([3,5,4,4,3,4]) and obtain (4,3,2) (4,5,1) (5,4,1) (4,4,2) I solved with tow loops but off course there will be a fast solution Any idea? what about using np.unique for one bi-dimensional array? In bash I usually unique command thanks in advance Giuseppe -- Giuseppe Amatulli Web: www.spatial-ecology.net

4 4

Test failures on Windows XP 64-bit
by Ondřej Čertík July 25, 2012

July 25, 2012

Hi, I've created a Windows installer of the maintenance/1.7.x branch (commit f93774d), the binary is available here (created in Wine on linux): https://github.com/certik/numpy-vendor/blob/69555c40dff5ae9f2d27d827f283bc6… and when I install it on the Windows XP 64-bit machine, here are the test results: https://gist.github.com/3173696 There are four failures --- three of them are related to the same exception: OSError: Failed to use '_localtime64_s' to convert to a local time and the fourth one seems like a regular bug in the test_iterator.test_iter_array_cast test. Are these known failures? Note that I can reproduce the _localtime64_s failures in Wine, but not the test_iterator.test_iter_array_cast one. Ondrej

2 1

Segfault in mingw in test_arrayprint.TestComplexArray
by Ondřej Čertík July 25, 2012

July 25, 2012

Hi, I managed to compile NumPy in MinGW under Wine in Ubuntu 11.10 and here is a full log of the tests: https://gist.github.com/3135607 It fails at the test test_str (test_arrayprint.TestComplexArray) with a segfault like this: test_str (test_arrayprint.TestComplexArray) ... wine: Unhandled page fault on read access to 0x00000000 at address (nil) (thread 0009), starting debugger... Unhandled exception: page fault on read access to 0x00000000 in 32-bit code (0x00000000). Register dump: CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b EIP:00000000 ESP:0041c230 EBP:00000000 EFLAGS:00010202( R- -- I - - - ) EAX:00000000 EBX:1e00807f ECX:00000000 EDX:0041c208 ESI:00f46de0 EDI:00000000 ... See the gist for the full log. Any ideas? I downloaded Python from python.org, is it supposed to work with numpy compiled using mingw? Ondrej

3 16