From rjd4+numpy at cam.ac.uk Tue Apr 1 10:31:50 2014 From: rjd4+numpy at cam.ac.uk (Bob Dowling) Date: Tue, 01 Apr 2014 15:31:50 +0100 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: <533ACD5B.1030301@cam.ac.uk> References: <533ACD5B.1030301@cam.ac.uk> Message-ID: <533ACDD6.8020403@cam.ac.uk> Versions: >>> sys.version '3.3.2 (default, Mar 5 2014, 08:21:05) \n[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)]' >>> numpy.__version__ '1.8.0' Problem: I'm trying to unpick the shape requirements of numpy.linalg.solve(). The help text says: solve(a, b) - a : (..., M, M) array_like Coefficient matrix. b : {(..., M,), (..., M, K)}, array_like Ordinate or "dependent variable" values. It's the requirements on "b" that are giving me grief. My read of the help text is that "b" must have a shape with either its final axis or its penultimate axis equal to M in size. Which axis the matrix contraction is along depends on the size of the final axis of "b". So, according to my reading, if "b" has shape (6,3) then the first choice, "(..., M,)", is invoked but if "a" has shape (3,3) and "b" has shape (3,6) then the second choice, "(..., M, K)", is invoked. I find this weird, but I've dealt with (much) weirder. However, this is not what I see. When "b" has shape (3,6) everything goes as expected. When "b" has shape (6,3) I get an error message that 6 is not equal to 3: > ValueError: solve: Operand 1 has a mismatch in its core dimension 0, > with gufunc signature (m,m),(m,n)->(m,n) (size 6 is different from 3) Obviously my reading is incorrect. Can somebody elucidate for me exactly what the requirements are on the shape of "b"? Example code: import numpy import numpy.linalg # Works. M = numpy.array([ [1.0, 1.0/2.0, 1.0/3.0], [1.0/2.0, 1.0/3.0, 1.0/4.0], [1.0/3.0, 1.0/4.0, 1.0/5.0] ] ) yy1 = numpy.array([ [1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0] ]) print(yy1.shape) xx1 = numpy.linalg.solve(M, yy1) print(xx1) # Works too. yy2 = numpy.array([ [1.0, 0.0, 0.0, 1.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0, 0.0, 1.0] ]) print(yy2.shape) xx2 = numpy.linalg.solve(M, yy2) print(xx2) # Fails. yy3 = numpy.array([ [1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0], [1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0] ]) print(yy3.shape) xx3 = numpy.linalg.solve(M, yy3) print(xx3) From sebastian at sipsolutions.net Tue Apr 1 10:57:21 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 01 Apr 2014 16:57:21 +0200 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: <533ACDD6.8020403@cam.ac.uk> References: <533ACD5B.1030301@cam.ac.uk> <533ACDD6.8020403@cam.ac.uk> Message-ID: <1396364241.4352.15.camel@sebastian-t440> On Di, 2014-04-01 at 15:31 +0100, Bob Dowling wrote: > Versions: > > >>> sys.version > '3.3.2 (default, Mar 5 2014, 08:21:05) \n[GCC 4.8.2 20131212 (Red Hat > 4.8.2-7)]' > > >>> numpy.__version__ > '1.8.0' > > > > Problem: > > I'm trying to unpick the shape requirements of numpy.linalg.solve(). > The help text says: > > solve(a, b) - > a : (..., M, M) array_like > Coefficient matrix. > b : {(..., M,), (..., M, K)}, array_like > Ordinate or "dependent variable" values. > > It's the requirements on "b" that are giving me grief. My read of the > help text is that "b" must have a shape with either its final axis or > its penultimate axis equal to M in size. Which axis the matrix > contraction is along depends on the size of the final axis of "b". > > > So, according to my reading, if "b" has shape (6,3) then the first > choice, "(..., M,)", is invoked but if "a" has shape (3,3) and "b" has > shape (3,6) then the second choice, "(..., M, K)", is invoked. I find > this weird, but I've dealt with (much) weirder. > I bet the documentation needs some more info there (if you have time, please write a pull request). If you look at the code (that part is just python code), you will see what really happens. If `a` has exactly one dimension more then `b`, the first case is used. Otherwise (..., M, K) is used instead. To make sure you always get the expected result, it may be best to make sure that the number of broadcasting (...) dimensions of `a` and `b` are identical (I am not sure if you expect this to be the case or not). The shape itself does not matter, only the (relative) number of dimensions does for the decision which of the two signatures is used. In other words, since you do not use `...` your examples always use the (M, K) logic. - Sebastian > > However, this is not what I see. When "b" has shape (3,6) everything > goes as expected. When "b" has shape (6,3) I get an error message that > 6 is not equal to 3: > > > ValueError: solve: Operand 1 has a mismatch in its core dimension 0, > > with gufunc signature (m,m),(m,n)->(m,n) (size 6 is different from 3) > > > > Obviously my reading is incorrect. Can somebody elucidate for me > exactly what the requirements are on the shape of "b"? > > > > Example code: > > import numpy > import numpy.linalg > > # Works. > M = numpy.array([ > [1.0, 1.0/2.0, 1.0/3.0], > [1.0/2.0, 1.0/3.0, 1.0/4.0], > [1.0/3.0, 1.0/4.0, 1.0/5.0] > ] ) > > yy1 = numpy.array([ > [1.0, 0.0, 0.0], > [0.0, 1.0, 0.0], > [0.0, 0.0, 1.0] > ]) > print(yy1.shape) > xx1 = numpy.linalg.solve(M, yy1) > print(xx1) > > # Works too. > yy2 = numpy.array([ > [1.0, 0.0, 0.0, 1.0, 0.0, 0.0], > [0.0, 1.0, 0.0, 0.0, 1.0, 0.0], > [0.0, 0.0, 1.0, 0.0, 0.0, 1.0] > ]) > print(yy2.shape) > xx2 = numpy.linalg.solve(M, yy2) > print(xx2) > > # Fails. > yy3 = numpy.array([ > [1.0, 0.0, 0.0], > [0.0, 1.0, 0.0], > [0.0, 0.0, 1.0], > [1.0, 0.0, 0.0], > [0.0, 1.0, 0.0], > [0.0, 0.0, 1.0] > ]) > print(yy3.shape) > xx3 = numpy.linalg.solve(M, yy3) > print(xx3) > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Apr 1 11:13:26 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 1 Apr 2014 09:13:26 -0600 Subject: [Numpy-discussion] Resolving the associativity/precedence debate for @ In-Reply-To: References: Message-ID: On Mon, Mar 24, 2014 at 6:33 PM, Nathaniel Smith wrote: > On Mon, Mar 24, 2014 at 11:58 PM, Charles R Harris > wrote: > > On Mon, Mar 24, 2014 at 5:56 PM, Nathaniel Smith wrote: > >> > >> On Sat, Mar 22, 2014 at 6:13 PM, Nathaniel Smith wrote: > >> > After 88 emails we don't have a conclusion in the other thread (see > >> > [1] for background). But we have to come to some conclusion or another > >> > if we want @ to exist :-). So I'll summarize where the discussion > >> > stands and let's see if we can find some way to resolve this. > >> > >> Response in this thread so far seems (AFAICT) to have pretty much > >> converged on same-left. > >> > >> If you think that this would be terrible and there is some compelling > >> argument against it, then please speak up! Otherwise, if no-one > >> objects, then I'll go ahead in the next few days and put same-left > >> into the PEP. > > > > > > I think we should take a close look at broadcasting before deciding on > the > > precedence. > > Can you elaborate? Like what, concretely, do you think we need to do now? > > Mostly I like to think of the '@' operators like commas in a function call where each argument gets evaluated before the matrix multiplications take place, so that would put it of lower precedence than '*', but still higher than '+, -' . However, since most matrix expressions seem to be small it may not matter much and the same result could be gotten with parenthesis. But I do think it would make it easier to read and parse matrix expressions as the '@' would serve as a natural divider. So 'A @ B*v' would be equivalent to 'A @ (B*v)' and not '(A @B)*v'. Hmm, now that I stare at it, it may actually be easier to simply read left to right and use parenthesis when needed. So put me down as neutral at this point and maybe trending towards equal precedence. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 1 11:25:01 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 16:25:01 +0100 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: <1396364241.4352.15.camel@sebastian-t440> References: <533ACD5B.1030301@cam.ac.uk> <533ACDD6.8020403@cam.ac.uk> <1396364241.4352.15.camel@sebastian-t440> Message-ID: On Tue, Apr 1, 2014 at 3:57 PM, Sebastian Berg wrote: > If `a` has exactly one dimension more then `b`, the first case is used. > Otherwise (..., M, K) is used instead. To make sure you always get the > expected result, it may be best to make sure that the number of > broadcasting (...) dimensions of `a` and `b` are identical (I am not > sure if you expect this to be the case or not). The shape itself does > not matter, only the (relative) number of dimensions does for the > decision which of the two signatures is used. Oh, really? This seems really unfortunate -- AFAICT it makes it impossible to write a generic broadcasting matrix-solve or vector-solve :-/ (except by explicitly checking shapes and prepending ones by hand, more or less doing the broadcasting manually). Surely it would be better to use PEP 467 style broadcasting, where the only special case is if `b` has exactly 1 dimension? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Tue Apr 1 12:04:16 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 1 Apr 2014 09:04:16 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Mon, Mar 31, 2014 at 3:09 PM, Matthew Brett wrote: > I am hopelessly lost here, but it looks as though Python extension > modules get loaded via > > hDLL = LoadLibraryEx(pathname, NULL, > LOAD_WITH_ALTERED_SEARCH_PATH); > > See: > http://hg.python.org/cpython/file/3a1db0d2747e/Python/dynload_win.c#l195 > > I think this means that the first directory on the search path is > indeed the path containing the extension module: > > > http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx#alternate_search_order_for_desktop_applications yup -- that seems to be what it says... So I'm guessing that it would not work putting DLLs into the 'DLLs' > directory - unless the extension modules went in there too. and yet there is a bunch of stuff there, so something is going on...It looks like my Windows box is down at the moment, but I _think_ there are a bunch of dependency dlls in there -- and not the extensions themselves. But I'm way out of my depth, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Apr 1 12:10:34 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 1 Apr 2014 09:10:34 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: On Mon, Mar 31, 2014 at 7:19 PM, Nathaniel Smith wrote: > > The difference is that datetime.datetime doesn't provide any iso string > parsing. > > Sure it does. datetime.strptime, with the %z modifier in particular. > that's not ISO parsing, that's parsing according to a user-defined format string, which can be used for ISO parsing, but the user is in control of how that's done. And I see this: "For a naive object, the %z and %Z format codes are replaced by empty strings." though I'm not entirely sure what that means -- probably only for writing. > The use case I'm imagining is for folks with ISO strings with a Z on the > end -- they'll need to deal with pre-parsing the strings to strip off the > Z, when it wouldn't change the result. > > > > Maybe this is an argument for "UTC always" rather than "naive"? > > Probably it is, but that approach seems a lot harder to extend to proper > tz support later, plus being more likely to cause trouble for pandas's > proper tz support now. > I was originally advocating for naive to begin with ;-) Someone else pushed for UTC -- I thought it was you! (but I guess not) It seems this committee of two has come to a consensus on naive -- and you're probably right, raise an exception if there is a time zone specifier. -CHB > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Apr 1 12:18:21 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Tue, 1 Apr 2014 12:18:21 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker wrote: > "For a naive object, the %z and %Z format codes are replaced by empty > strings." > > though I'm not entirely sure what that means -- probably only for writing. > That's right: >>> from datetime import * >>> datetime.now().strftime('%z') '' >>> datetime.now(timezone.utc).strftime('%z') '+0000' -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Apr 1 12:22:02 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Tue, 1 Apr 2014 12:22:02 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker wrote: > It seems this committee of two has come to a consensus on naive -- and > you're probably right, raise an exception if there is a time zone specifier. Count me as +1 on naive, but consider converting garbage (including strings with trailing Z) to NaT. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 1 13:12:24 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 18:12:24 +0100 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: On Tue, Apr 1, 2014 at 5:22 PM, Alexander Belopolsky wrote: > > On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker wrote: >> >> It seems this committee of two has come to a consensus on naive -- and >> you're probably right, raise an exception if there is a time zone specifier. > > > Count me as +1 on naive, but consider converting garbage (including strings > with trailing Z) to NaT. That's not how we handle other types, e.g.: In [5]: a = np.zeros(1, dtype=float) In [6]: a[0] = "garbage" ValueError: could not convert string to float: garbage (Cf, "Errors should never pass silently".) Any reason why datetime64 should be different? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Tue Apr 1 13:26:53 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Apr 2014 10:26:53 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, On Tue, Apr 1, 2014 at 9:04 AM, Chris Barker wrote: > On Mon, Mar 31, 2014 at 3:09 PM, Matthew Brett > wrote: >> >> I am hopelessly lost here, but it looks as though Python extension >> modules get loaded via >> >> hDLL = LoadLibraryEx(pathname, NULL, >> LOAD_WITH_ALTERED_SEARCH_PATH); >> >> See: >> http://hg.python.org/cpython/file/3a1db0d2747e/Python/dynload_win.c#l195 >> >> I think this means that the first directory on the search path is >> indeed the path containing the extension module: >> >> >> http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx#alternate_search_order_for_desktop_applications > > > yup -- that seems to be what it says... > >> So I'm guessing that it would not work putting DLLs into the 'DLLs' >> directory - unless the extension modules went in there too. > > > and yet there is a bunch of stuff there, so something is going on...It looks > like my Windows box is down at the moment, but I _think_ there are a bunch > of dependency dlls in there -- and not the extensions themselves. I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL loaded via: hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); will in turn (by default) search for its dependent DLLs in their own directory. Or maybe in the directory of the first DLL to be loaded with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the documentation. Looking forward to doing my tax return after this. But - anyway - that means that any extensions in the DLLs directory will get their dependencies from the DLLs directory, but that is only true for extensions in that directory. Cheers, Matthew From njs at pobox.com Tue Apr 1 13:43:37 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 18:43:37 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett wrote: > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL loaded via: > > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); > > will in turn (by default) search for its dependent DLLs in their own > directory. Or maybe in the directory of the first DLL to be loaded > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the > documentation. Looking forward to doing my tax return after this. > > But - anyway - that means that any extensions in the DLLs directory > will get their dependencies from the DLLs directory, but that is only > true for extensions in that directory. So in conclusion, if we just drop our compiled dependencies next to the compiled module files then we're good, even on older Windows versions? That sounds much simpler than previous discussions, but good news if it's true... -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Tue Apr 1 13:53:46 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Apr 2014 10:53:46 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, On Tue, Apr 1, 2014 at 10:43 AM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett wrote: >> I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL loaded via: >> >> hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> >> will in turn (by default) search for its dependent DLLs in their own >> directory. Or maybe in the directory of the first DLL to be loaded >> with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> documentation. Looking forward to doing my tax return after this. >> >> But - anyway - that means that any extensions in the DLLs directory >> will get their dependencies from the DLLs directory, but that is only >> true for extensions in that directory. > > So in conclusion, if we just drop our compiled dependencies next to > the compiled module files then we're good, even on older Windows > versions? That sounds much simpler than previous discussions, but good > news if it's true... I think that's right, but as you can see, I am not sure. It might explain why Carl Kleffner found that he could drop libopenblas.dll in numpy/core and it just worked [1]. Well, if all the extensions using blas / lapack are in fact in numpy/core. Christoph - have you tried doing the same with MKL? Cheers, Matthew [1] http://numpy-discussion.10968.n7.nabble.com/Default-builds-of-OpenBLAS-development-branch-are-now-fork-safe-td36523.html From smudkavi at uwaterloo.ca Tue Apr 1 14:13:52 2014 From: smudkavi at uwaterloo.ca (Sankarshan Mudkavi) Date: Tue, 1 Apr 2014 14:13:52 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: I agree with that interpretation of naive as well. I'll change the proposal to reflect that. So any modifier should raise an error then? (At the risk of breaking people's code.) The only question is, should we consider accepting the modifier and disregard it with a warning, letting the user know that this is only for temporary compatibility purposes? As of now, it's not clear to me which of those options is better. Cheers, Sankarshan On Apr 1, 2014, at 1:12 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 5:22 PM, Alexander Belopolsky wrote: >> >> On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker wrote: >>> >>> It seems this committee of two has come to a consensus on naive -- and >>> you're probably right, raise an exception if there is a time zone specifier. >> >> >> Count me as +1 on naive, but consider converting garbage (including strings >> with trailing Z) to NaT. > > That's not how we handle other types, e.g.: > > In [5]: a = np.zeros(1, dtype=float) > > In [6]: a[0] = "garbage" > ValueError: could not convert string to float: garbage > > (Cf, "Errors should never pass silently".) Any reason why datetime64 > should be different? > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From matthew.brett at gmail.com Tue Apr 1 14:26:33 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Apr 2014 11:26:33 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, I just noticed this C reference implementation of blas: https://github.com/rljames/coblas No lapack, no benchmarks, but tests, and BSD. I wonder if it is possible to craft a Frankenlibrary from OpenBLAS and reference implementations to avoid broken parts of OpenBLAS? Cheers, Matthew From Thomas.Haslwanter at fh-linz.at Tue Apr 1 14:27:01 2014 From: Thomas.Haslwanter at fh-linz.at (Haslwanter Thomas) Date: Tue, 1 Apr 2014 20:27:01 +0200 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value Message-ID: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> While most other Python applications (scipy, pandas) use for the calculation of the standard deviation the default "ddof=1" (i.e. they calculate the sample standard deviation), the Numpy implementation uses the default "ddof=0". Personally I cannot think of many applications where it would be desired to calculate the standard deviation with ddof=0. In addition, I feel that there should be consistency between standard modules such as numpy, scipy, and pandas. I am wondering if there is a good reason to stick to "ddof=0" as the default for "std", or if others would agree with my suggestion to change the default to "ddof=1"? Thomas --- Prof. (FH) PD Dr. Thomas Haslwanter School of Applied Health and Social Sciences University of Applied Sciences Upper Austria FH O? Studienbetriebs GmbH Garnisonstra?e 21 4020 Linz/Austria Tel.: +43 (0)5 0804 -52170 Fax: +43 (0)5 0804 -52171 E-Mail: Thomas.Haslwanter at fh-linz.at Web: me-research.fh-linz.at or work.thaslwanter.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Apr 1 14:50:56 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Tue, 1 Apr 2014 14:50:56 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: On Tue, Apr 1, 2014 at 1:12 PM, Nathaniel Smith wrote: > In [6]: a[0] = "garbage" > ValueError: could not convert string to float: garbage > > (Cf, "Errors should never pass silently".) Any reason why datetime64 > should be different? > datetime64 is different because it has NaT support from the start. NaN support for floats seems to be an afterthought if not an accident of implementation. And it looks like some errors do pass silently: >>> a[0] = "1" # not a TypeError But I withdraw my suggestion. The closer datetime64 behavior is to numeric types the better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Tue Apr 1 15:59:52 2014 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 1 Apr 2014 15:59:52 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> Message-ID: Because np.mean() is ddof=0? (I mean effectively, not that it actually has a parameter for that) There is consistency within the library, and I certainly wouldn't want to have NaN all of the sudden coming from my calls to mean() that I apply to an arbitrary non-empty array of values that happened to have only one value. So, if we can't change the default for mean, then it only makes sense to keep np.std() consistent with np.mean(). My 2 cents... Ben Root On Tue, Apr 1, 2014 at 2:27 PM, Haslwanter Thomas < Thomas.Haslwanter at fh-linz.at> wrote: > While most other Python applications (scipy, pandas) use for the > calculation of the standard deviation the default "ddof=1" (i.e. they > calculate the sample standard deviation), the Numpy implementation uses the > default "ddof=0". > > Personally I cannot think of many applications where it would be desired > to calculate the standard deviation with ddof=0. In addition, I feel that > there should be consistency between standard modules such as numpy, scipy, > and pandas. > > > > I am wondering if there is a good reason to stick to "ddof=0" as the > default for "std", or if others would agree with my suggestion to change > the default to "ddof=1"? > > > > Thomas > > > > --- > Prof. (FH) PD Dr. Thomas Haslwanter > School of Applied Health and Social Sciences > > *University of Applied Sciences* *Upper Austria* > *FH O? Studienbetriebs GmbH* > Garnisonstra?e 21 > 4020 Linz/Austria > Tel.: +43 (0)5 0804 -52170 > Fax: +43 (0)5 0804 -52171 > E-Mail: Thomas.Haslwanter at fh-linz.at > Web: me-research.fh-linz.at > or work.thaslwanter.at > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Apr 1 16:02:11 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 1 Apr 2014 20:02:11 +0000 (UTC) Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> Message-ID: <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Haslwanter Thomas wrote: > Personally I cannot think of many applications where it would be desired > to calculate the standard deviation with ddof=0. In addition, I feel that > there should be consistency between standard modules such as numpy, scipy, and pandas. ddof=0 is the maxiumum likelihood estimate. It is also needed in Bayesian estimation. If you are not eatimating from a sample, but rather calculating for the whole population, you always want ddof=0. What does Matlab do by default? (Yes, it is a retorical question.) > I am wondering if there is a good reason to stick to "ddof=0" as the > default for "std", or if others would agree with my suggestion to change > the default to "ddof=1"? It is a bad idea to suddenly break everyone's code. Sturla From hoogendoorn.eelco at gmail.com Tue Apr 1 16:06:02 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Tue, 1 Apr 2014 22:06:02 +0200 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: I agree; breaking code over this would be ridiculous. Also, I prefer the zero default, despite the mean/std combo probably being more common. On Tue, Apr 1, 2014 at 10:02 PM, Sturla Molden wrote: > Haslwanter Thomas wrote: > > > Personally I cannot think of many applications where it would be desired > > to calculate the standard deviation with ddof=0. In addition, I feel that > > there should be consistency between standard modules such as numpy, > scipy, and pandas. > > ddof=0 is the maxiumum likelihood estimate. It is also needed in Bayesian > estimation. > > If you are not eatimating from a sample, but rather calculating for the > whole population, you always want ddof=0. > > What does Matlab do by default? (Yes, it is a retorical question.) > > > > I am wondering if there is a good reason to stick to "ddof=0" as the > > default for "std", or if others would agree with my suggestion to change > > the default to "ddof=1"? > > It is a bad idea to suddenly break everyone's code. > > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 1 16:08:49 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 21:08:49 +0100 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden wrote: > Haslwanter Thomas wrote: > >> Personally I cannot think of many applications where it would be desired >> to calculate the standard deviation with ddof=0. In addition, I feel that >> there should be consistency between standard modules such as numpy, scipy, and pandas. > > ddof=0 is the maxiumum likelihood estimate. It is also needed in Bayesian > estimation. It's true, but the counter-arguments are also strong. And regardless of whether ddof=1 or ddof=0 is better, surely the same one is better for both numpy and scipy. > If you are not eatimating from a sample, but rather calculating for the > whole population, you always want ddof=0. > > What does Matlab do by default? (Yes, it is a retorical question.) R (which is probably a more relevant comparison) does do ddof=1 by default. >> I am wondering if there is a good reason to stick to "ddof=0" as the >> default for "std", or if others would agree with my suggestion to change >> the default to "ddof=1"? > > It is a bad idea to suddenly break everyone's code. It would be a disruptive transition, but OTOH having inconsistencies like this guarantees the ongoing creation of new broken code. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From rjd4+numpy at cam.ac.uk Tue Apr 1 16:39:29 2014 From: rjd4+numpy at cam.ac.uk (Bob Dowling) Date: Tue, 01 Apr 2014 21:39:29 +0100 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: References: <533ACD5B.1030301@cam.ac.uk> <533ACDD6.8020403@cam.ac.uk> <1396364241.4352.15.camel@sebastian-t440> Message-ID: <533B2401.3080000@cam.ac.uk> On 04/01/2014 04:25 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 3:57 PM, Sebastian Berg > wrote: >> If `a` has exactly one dimension more then `b`, the first case is used. >> Otherwise (..., M, K) is used instead. To make sure you always get the >> expected result, it may be best to make sure that the number of >> broadcasting (...) dimensions of `a` and `b` are identical (I am not >> sure if you expect this to be the case or not). The shape itself does >> not matter, only the (relative) number of dimensions does for the >> decision which of the two signatures is used. > Oh, really? This seems really unfortunate It also seems quite counter-intuitive. It means that an array "a" of shape (3,3) will behave radically differently to one of shape (1,3,3). But thank you for the explanation. From sebastian at sipsolutions.net Tue Apr 1 16:50:36 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 01 Apr 2014 22:50:36 +0200 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: References: <533ACD5B.1030301@cam.ac.uk> <533ACDD6.8020403@cam.ac.uk> <1396364241.4352.15.camel@sebastian-t440> Message-ID: <1396385436.22698.2.camel@sebastian-t440> On Di, 2014-04-01 at 16:25 +0100, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 3:57 PM, Sebastian Berg > wrote: > > If `a` has exactly one dimension more then `b`, the first case is used. > > Otherwise (..., M, K) is used instead. To make sure you always get the > > expected result, it may be best to make sure that the number of > > broadcasting (...) dimensions of `a` and `b` are identical (I am not > > sure if you expect this to be the case or not). The shape itself does > > not matter, only the (relative) number of dimensions does for the > > decision which of the two signatures is used. > Since b is a system of equations if it is 2-dim, I think it basically doesn't make sense to have a (M, K) shaped b anyway, since you could use a (K, M) shaped b with broadcasting logic (though I guess that is slower unless you add extra logic). - Sebastian > Oh, really? This seems really unfortunate -- AFAICT it makes it > impossible to write a generic broadcasting matrix-solve or > vector-solve :-/ (except by explicitly checking shapes and prepending > ones by hand, more or less doing the broadcasting manually). Surely it > would be better to use PEP 467 style broadcasting, where the only > special case is if `b` has exactly 1 dimension? > > -n > From ralf.gommers at gmail.com Tue Apr 1 16:51:43 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 1 Apr 2014 22:51:43 +0200 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 10:08 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden > wrote: > > Haslwanter Thomas wrote: > > > >> Personally I cannot think of many applications where it would be desired > >> to calculate the standard deviation with ddof=0. In addition, I feel > that > >> there should be consistency between standard modules such as numpy, > scipy, and pandas. > > > > ddof=0 is the maxiumum likelihood estimate. It is also needed in Bayesian > > estimation. > > It's true, but the counter-arguments are also strong. And regardless > of whether ddof=1 or ddof=0 is better, surely the same one is better > for both numpy and scipy. > If we could still choose here without any costs, obviously that's true. This particular ship sailed a long time ago though. By the way, there isn't even a `scipy.stats.std`, so we're comparing with differently named functions (nanstd for example). > > If you are not eatimating from a sample, but rather calculating for the > > whole population, you always want ddof=0. > > > > What does Matlab do by default? (Yes, it is a retorical question.) > > R (which is probably a more relevant comparison) does do ddof=1 by default. > > >> I am wondering if there is a good reason to stick to "ddof=0" as the > >> default for "std", or if others would agree with my suggestion to change > >> the default to "ddof=1"? > > > > It is a bad idea to suddenly break everyone's code. > > It would be a disruptive transition, but OTOH having inconsistencies > like this guarantees the ongoing creation of new broken code. > Not much of an argument to change return values for a so heavily used function. Ralf > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 1 16:54:45 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 1 Apr 2014 14:54:45 -0600 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 2:08 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden > wrote: > > Haslwanter Thomas wrote: > > > >> Personally I cannot think of many applications where it would be desired > >> to calculate the standard deviation with ddof=0. In addition, I feel > that > >> there should be consistency between standard modules such as numpy, > scipy, and pandas. > > > > ddof=0 is the maxiumum likelihood estimate. It is also needed in Bayesian > > estimation. > > It's true, but the counter-arguments are also strong. And regardless > of whether ddof=1 or ddof=0 is better, surely the same one is better > for both numpy and scipy. > > > If you are not eatimating from a sample, but rather calculating for the > > whole population, you always want ddof=0. > > > > What does Matlab do by default? (Yes, it is a retorical question.) > > R (which is probably a more relevant comparison) does do ddof=1 by default. > > >> I am wondering if there is a good reason to stick to "ddof=0" as the > >> default for "std", or if others would agree with my suggestion to change > >> the default to "ddof=1"? > > > > It is a bad idea to suddenly break everyone's code. > > It would be a disruptive transition, but OTOH having inconsistencies > like this guarantees the ongoing creation of new broken code. > > This topic comes up regularly. The original choice was made for numpy 1.0b1 by Travis, see this later thread.At this point it is probably best to leave it alone. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 1 17:11:19 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 22:11:19 +0100 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 9:51 PM, Ralf Gommers wrote: > > > > On Tue, Apr 1, 2014 at 10:08 PM, Nathaniel Smith wrote: >> >> On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden >> wrote: >> > Haslwanter Thomas wrote: >> > >> >> Personally I cannot think of many applications where it would be >> >> desired >> >> to calculate the standard deviation with ddof=0. In addition, I feel >> >> that >> >> there should be consistency between standard modules such as numpy, >> >> scipy, and pandas. >> > >> > ddof=0 is the maxiumum likelihood estimate. It is also needed in >> > Bayesian >> > estimation. >> >> It's true, but the counter-arguments are also strong. And regardless >> of whether ddof=1 or ddof=0 is better, surely the same one is better >> for both numpy and scipy. > > If we could still choose here without any costs, obviously that's true. This > particular ship sailed a long time ago though. By the way, there isn't even > a `scipy.stats.std`, so we're comparing with differently named functions > (nanstd for example). Presumably nanstd is a lot less heavily used than std, and presumably people expect 'nanstd' to be a 'nan' version of 'std' -- what do you think of changing nanstd to ddof=0 to match numpy? (With appropriate FutureWarning transition, etc.) -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From argriffi at ncsu.edu Tue Apr 1 17:11:59 2014 From: argriffi at ncsu.edu (alex) Date: Tue, 1 Apr 2014 17:11:59 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 4:54 PM, Charles R Harris wrote: > > > > On Tue, Apr 1, 2014 at 2:08 PM, Nathaniel Smith wrote: >> >> On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden >> wrote: >> > Haslwanter Thomas wrote: >> > >> >> Personally I cannot think of many applications where it would be >> >> desired >> >> to calculate the standard deviation with ddof=0. In addition, I feel >> >> that >> >> there should be consistency between standard modules such as numpy, >> >> scipy, and pandas. >> > >> > ddof=0 is the maxiumum likelihood estimate. It is also needed in >> > Bayesian >> > estimation. >> >> It's true, but the counter-arguments are also strong. And regardless >> of whether ddof=1 or ddof=0 is better, surely the same one is better >> for both numpy and scipy. >> >> > If you are not eatimating from a sample, but rather calculating for the >> > whole population, you always want ddof=0. >> > >> > What does Matlab do by default? (Yes, it is a retorical question.) >> >> R (which is probably a more relevant comparison) does do ddof=1 by >> default. >> >> >> I am wondering if there is a good reason to stick to "ddof=0" as the >> >> default for "std", or if others would agree with my suggestion to >> >> change >> >> the default to "ddof=1"? >> > >> > It is a bad idea to suddenly break everyone's code. >> >> It would be a disruptive transition, but OTOH having inconsistencies >> like this guarantees the ongoing creation of new broken code. >> > > This topic comes up regularly. The original choice was made for numpy 1.0b1 > by Travis, see this later thread. At this point it is probably best to leave > it alone. I don't have any opinion about this debate, but I love the justification in that thread "Any surprise that is created by the different default should be mitigated by the fact that it's an opportunity to learn something about what you are doing." This masterpiece of rhetoric will surely help me win many internet arguments in the future! From njs at pobox.com Tue Apr 1 17:15:40 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Apr 2014 22:15:40 +0100 Subject: [Numpy-discussion] Confused by spec of numpy.linalg.solve In-Reply-To: <1396385436.22698.2.camel@sebastian-t440> References: <533ACD5B.1030301@cam.ac.uk> <533ACDD6.8020403@cam.ac.uk> <1396364241.4352.15.camel@sebastian-t440> <1396385436.22698.2.camel@sebastian-t440> Message-ID: On Tue, Apr 1, 2014 at 9:50 PM, Sebastian Berg wrote: > On Di, 2014-04-01 at 16:25 +0100, Nathaniel Smith wrote: >> On Tue, Apr 1, 2014 at 3:57 PM, Sebastian Berg >> wrote: >> > If `a` has exactly one dimension more then `b`, the first case is used. >> > Otherwise (..., M, K) is used instead. To make sure you always get the >> > expected result, it may be best to make sure that the number of >> > broadcasting (...) dimensions of `a` and `b` are identical (I am not >> > sure if you expect this to be the case or not). The shape itself does >> > not matter, only the (relative) number of dimensions does for the >> > decision which of the two signatures is used. >> > > Since b is a system of equations if it is 2-dim, I think it basically > doesn't make sense to have a (M, K) shaped b anyway, since you could use > a (K, M) shaped b with broadcasting logic (though I guess that is slower > unless you add extra logic). Not sure I'm following your point exactly, but the argument for having (M, M) `a` and (M, K) `b` is that solve(a, b) is the same as dot(inv(a), b), which obviously accepts 2d `a` and `b`... -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From josef.pktd at gmail.com Tue Apr 1 17:18:13 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 1 Apr 2014 17:18:13 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 1, 2014 at 5:11 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 9:51 PM, Ralf Gommers wrote: >> >> >> >> On Tue, Apr 1, 2014 at 10:08 PM, Nathaniel Smith wrote: >>> >>> On Tue, Apr 1, 2014 at 9:02 PM, Sturla Molden >>> wrote: >>> > Haslwanter Thomas wrote: >>> > >>> >> Personally I cannot think of many applications where it would be >>> >> desired >>> >> to calculate the standard deviation with ddof=0. In addition, I feel >>> >> that >>> >> there should be consistency between standard modules such as numpy, >>> >> scipy, and pandas. >>> > >>> > ddof=0 is the maxiumum likelihood estimate. It is also needed in >>> > Bayesian >>> > estimation. >>> >>> It's true, but the counter-arguments are also strong. And regardless >>> of whether ddof=1 or ddof=0 is better, surely the same one is better >>> for both numpy and scipy. >> >> If we could still choose here without any costs, obviously that's true. This >> particular ship sailed a long time ago though. By the way, there isn't even >> a `scipy.stats.std`, so we're comparing with differently named functions >> (nanstd for example). > > Presumably nanstd is a lot less heavily used than std, and presumably > people expect 'nanstd' to be a 'nan' version of 'std' -- what do you > think of changing nanstd to ddof=0 to match numpy? (With appropriate > FutureWarning transition, etc.) numpy is numpy, a numerical library scipy.stats is stats and behaves differently. (axis=0) nanstd in scipy.stats will hopefully also go away soon, so I don't think it's worth changing there either. pandas came later and thought ddof=1 is worth more than consistency. I don't think ddof defaults's are worth jumping through deprecation hoops. (bias in cov, corrcoef is "non-standard" ddof) Josef > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Tue Apr 1 18:58:44 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 1 Apr 2014 23:58:44 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett > wrote: > > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL > loaded via: > > > > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); > > > > will in turn (by default) search for its dependent DLLs in their own > > directory. Or maybe in the directory of the first DLL to be loaded > > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the > > documentation. Looking forward to doing my tax return after this. > > > > But - anyway - that means that any extensions in the DLLs directory > > will get their dependencies from the DLLs directory, but that is only > > true for extensions in that directory. > > So in conclusion, if we just drop our compiled dependencies next to > the compiled module files then we're good, even on older Windows > versions? That sounds much simpler than previous discussions, but good > news if it's true... > That does not work very well in my experience: - numpy has extension modules in multiple directories, so we would need to copy the dlls in multiple subdirectories - copying dlls means that windows will load that dll multiple times, with all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen serious issues when doing something similar for hdf5 dll and pytables/h5py). David > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 1 19:36:04 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 2 Apr 2014 00:36:04 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau wrote: > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett >> wrote: >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL >> > loaded via: >> > >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> > >> > will in turn (by default) search for its dependent DLLs in their own >> > directory. Or maybe in the directory of the first DLL to be loaded >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> > documentation. Looking forward to doing my tax return after this. >> > >> > But - anyway - that means that any extensions in the DLLs directory >> > will get their dependencies from the DLLs directory, but that is only >> > true for extensions in that directory. >> >> So in conclusion, if we just drop our compiled dependencies next to >> the compiled module files then we're good, even on older Windows >> versions? That sounds much simpler than previous discussions, but good >> news if it's true... > > > That does not work very well in my experience: > > - numpy has extension modules in multiple directories, so we would need to > copy the dlls in multiple subdirectories > - copying dlls means that windows will load that dll multiple times, with > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen > serious issues when doing something similar for hdf5 dll and pytables/h5py). We could just ship all numpy's extension modules in the same directory if we wanted. It would be pretty easy to stick some code at the top of numpy/__init__.py to load them from numpy/all_dlls/ and then slot them into the appropriate places in the package namespace. Of course scipy and numpy will still both have to ship BLAS etc., and so I guess it will get loaded at least twice in *any* binary install system. I'm not sure why this would be a problem (Windows, unlike Unix, carefully separates DLL namespaces, right?), but if it is a problem then it's a very fundamental one for any binaries we ship. Do the binaries we ship now have this problem? Or are we currently managing to statically link everything? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From cournape at gmail.com Tue Apr 1 19:46:20 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 2 Apr 2014 00:46:20 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith wrote: > On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau > wrote: > > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: > >> > >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett > >> wrote: > >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL > >> > loaded via: > >> > > >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); > >> > > >> > will in turn (by default) search for its dependent DLLs in their own > >> > directory. Or maybe in the directory of the first DLL to be loaded > >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the > >> > documentation. Looking forward to doing my tax return after this. > >> > > >> > But - anyway - that means that any extensions in the DLLs directory > >> > will get their dependencies from the DLLs directory, but that is only > >> > true for extensions in that directory. > >> > >> So in conclusion, if we just drop our compiled dependencies next to > >> the compiled module files then we're good, even on older Windows > >> versions? That sounds much simpler than previous discussions, but good > >> news if it's true... > > > > > > That does not work very well in my experience: > > > > - numpy has extension modules in multiple directories, so we would > need to > > copy the dlls in multiple subdirectories > > - copying dlls means that windows will load that dll multiple times, > with > > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen > > serious issues when doing something similar for hdf5 dll and > pytables/h5py). > > We could just ship all numpy's extension modules in the same directory > if we wanted. It would be pretty easy to stick some code at the top of > numpy/__init__.py to load them from numpy/all_dlls/ and then slot them > into the appropriate places in the package namespace. > > Of course scipy and numpy will still both have to ship BLAS etc., and > so I guess it will get loaded at least twice in *any* binary install > system. I'm not sure why this would be a problem (Windows, unlike > Unix, carefully separates DLL namespaces, right?) It does not really matter here. For pure blas/lapack, that may be ok because the functions are "stateless", but I would not count on it either. The cleanest solution I can think of is to have 'privately shared DLL', but that would AFAIK require patching python, so not really an option. , but if it is a > problem then it's a very fundamental one for any binaries we ship. > > Do the binaries we ship now have this problem? Or are we currently > managing to statically link everything? > We currently statically link everything. The main challenge is that 'new' (>= 4) versions of mingw don't easily allow statically linking all the mingw-related dependencies. While the options are there, everytime I tried to do it with an official build of mingw, I had some weird, very hard to track crashes. The other alternative that has been suggested is to build one own's toolchain where everything is static by default. I am not sure why that works, and that brings the risk of depending on a toolchain that we can't really maintain. David > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Apr 2 02:52:28 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Apr 2014 23:52:28 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, On Tue, Apr 1, 2014 at 4:46 PM, David Cournapeau wrote: > > > > On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith wrote: >> >> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau >> wrote: >> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: >> >> >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett >> >> wrote: >> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL >> >> > loaded via: >> >> > >> >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> >> > >> >> > will in turn (by default) search for its dependent DLLs in their own >> >> > directory. Or maybe in the directory of the first DLL to be loaded >> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> >> > documentation. Looking forward to doing my tax return after this. >> >> > >> >> > But - anyway - that means that any extensions in the DLLs directory >> >> > will get their dependencies from the DLLs directory, but that is only >> >> > true for extensions in that directory. >> >> >> >> So in conclusion, if we just drop our compiled dependencies next to >> >> the compiled module files then we're good, even on older Windows >> >> versions? That sounds much simpler than previous discussions, but good >> >> news if it's true... >> > >> > >> > That does not work very well in my experience: >> > >> > - numpy has extension modules in multiple directories, so we would >> > need to >> > copy the dlls in multiple subdirectories >> > - copying dlls means that windows will load that dll multiple times, >> > with >> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen >> > serious issues when doing something similar for hdf5 dll and >> > pytables/h5py). >> >> We could just ship all numpy's extension modules in the same directory >> if we wanted. It would be pretty easy to stick some code at the top of >> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them >> into the appropriate places in the package namespace. >> >> Of course scipy and numpy will still both have to ship BLAS etc., and >> so I guess it will get loaded at least twice in *any* binary install >> system. I'm not sure why this would be a problem (Windows, unlike >> Unix, carefully separates DLL namespaces, right?) > > > It does not really matter here. For pure blas/lapack, that may be ok because > the functions are "stateless", but I would not count on it either. > > The cleanest solution I can think of is to have 'privately shared DLL', but > that would AFAIK require patching python, so not really an option. David - do you know anything about private assemblies [1]? Might they work for our problem? How about AddDllDirectory [2]? Cheers, Matthew [1] http://msdn.microsoft.com/en-us/library/windows/desktop/ff951638(v=vs.85).aspx [2] http://msdn.microsoft.com/en-us/library/windows/desktop/hh310513(v=vs.85).asp From chris.barker at noaa.gov Wed Apr 2 10:44:29 2014 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 2 Apr 2014 07:44:29 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: <-6637307485460825292@unknownmsgid> On Apr 1, 2014, at 4:36 PM, Nathaniel Smith wrote: > We could just ship all numpy's extension modules in the same directory > if we wanted. It would be pretty easy to stick some code at the top of > numpy/__init__.py to load them from numpy/all_dlls/ and then slot them > into the appropriate places in the package namespace. I like this, and it could play well with wheels and virtualenv. (which reminds me -- looking at what Anaconda does may be instructive) > Of course scipy and numpy will still both have to ship BLAS etc., and > so I guess it will get loaded at least twice in *any* binary install Since scipy has a strict dependency on numpy, and the same people are maintaining the builds, couldn't scipy use the libs provided by numpy? ( maybe adding more) -Chris From random.seed at web.de Wed Apr 2 15:47:38 2014 From: random.seed at web.de (mbyt) Date: Wed, 2 Apr 2014 21:47:38 +0200 Subject: [Numpy-discussion] structured array assignments Message-ID: <20140402194736.GB2529@staufen> Hello, I am writing due to an issue in structured array assignments. Let's consider the following example: In [31]: x = np.empty(1, dtype=[('field', 'i4', 10)]) In [32]: type(x[0]) Out[32]: numpy.void In [33]: x[0] = np.ones(10, dtype='i4').view('V40') In [34]: x Out[34]: array([([1, 1, 1, 1, 1, 1, 1, 1, 1, 1],)], dtype=[('field', ' Be sure and get your proposals in for PyData Silicon Valley 2014 by April 6! The event brings together scientists, analysts, developers, engineers, architects and others from the Python data science community to discuss new techniques and tools for management, analytics and visualization of data. Presentation content can be at a novice, intermediate or advanced level. Talks will run 30-40 min and hands-on tutorial and workshop sessions will run 90 min. or 3 hours. If you are interested in presenting a talk, tutorial or sponsor workshop on meeting the challenges of big data using Python, please submit a SHORT abstract and bio. If you don't have time to create an abstract, feel free to submit a presentation outline. To see the type of topics presented at previous PyData events, please look at our past conference sites at pydata.org or check out the videos on vimeo.com/pydata. Submit Your Proposal Registration will be complimentary for speakers. Any questions can be directed to admin at pydata.org. Conference dates are May 2-4. Proposal deadline has been extended through April 6, 2014. --- Leah Silen leah at numfocus.org leah at pydata.org 512.222.5449 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Thu Apr 3 08:56:39 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 03 Apr 2014 14:56:39 +0200 Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe In-Reply-To: References: <1225660970414595360.835902sturla.molden-gmail.com@news.gmane.org> <53331DBF.6020504@googlemail.com> Message-ID: <533D5A87.1040907@googlemail.com> FYI, binaries linking openblas should add this patch in some way: https://github.com/numpy/numpy/pull/4580 Cliffs: linking OpenBLAS prevents parallelization via threading or multiprocessing. just wasted a bunch of time figuring that out ... (though its well documented in numerous stackoverflow questions, too bad none of them reached us) On 04/01/2014 02:59 AM, Matthew Brett wrote: > Hi, > > On Wed, Mar 26, 2014 at 11:34 AM, Julian Taylor > wrote: >> On 26.03.2014 16:27, Olivier Grisel wrote: >>> Hi Carl, >>> >>> I installed Python 2.7.6 64 bits on a windows server instance from >>> rackspace cloud and then ran get-pip.py and then could successfully >>> install the numpy and scipy wheel packages from your google drive >>> folder. I tested dot products and scipy.linalg.svd and they work as >>> expected. >>> >> >>> >>> Would it make sense to embed the blas and lapack header files as part >>> of this numpy wheel and make numpy.distutils.system_info return the >>> lib and include folder pointing to the embedded libopenblas.dll and >>> header files so has to make third party libraries directly buildable >>> against those? >>> >> >> as for using openblas by default in binary builds, no. >> pthread openblas build is now fork safe which is great but it is still >> not reliable enough for a default. >> E.g. the current latest release 0.2.8 still has one crash bug on >> dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3]. >> git head has the former four fixed bug still has wrong results for cgemv. > > I noticed the Carl was only getting three test failures on scipy - are > these related? > > ====================================================================== > FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 4)) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "D:\devel\py27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\devel\py27\lib\site-packages\scipy\linalg\tests\test_decomp.py", > line 642, in eigenhproblem_general > assert_array_almost_equal(diag2_, ones(diag2_.shape[0]), DIGITS[dtype]) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 811, in assert_array_almost_equal > header=('Arrays are not almost equal to %d decimals' % decimal)) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 4 decimals > > (mismatch 100.0%) > x: array([ 0., 0., 0.], dtype=float32) > y: array([ 1., 1., 1.]) > > ====================================================================== > FAIL: Tests for the minimize wrapper. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "D:\devel\py27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\devel\py27\lib\site-packages\scipy\optimize\tests\test_optimize.py", > line 435, in test_minimize > self.test_powell(True) > File "D:\devel\py27\lib\site-packages\scipy\optimize\tests\test_optimize.py", > line 209, in test_powell > atol=1e-14, rtol=1e-7) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 1181, in assert_allclose > verbose=verbose, header=header) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-07, atol=1e-14 > > (mismatch 100.0%) > x: array([[ 0.75077639, -0.44156936, 0.47100962], > [ 0.75077639, -0.44156936, 0.48052496], > [ 1.50155279, -0.88313872, 0.95153458],... > y: array([[ 0.72949016, -0.44156936, 0.47100962], > [ 0.72949016, -0.44156936, 0.48052496], > [ 1.45898031, -0.88313872, 0.95153458],... > > ====================================================================== > FAIL: Powell (direction set) optimization routine > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "D:\devel\py27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\devel\py27\lib\site-packages\scipy\optimize\tests\test_optimize.py", > line 209, in test_powell > atol=1e-14, rtol=1e-7) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 1181, in assert_allclose > verbose=verbose, header=header) > File "D:\devel\py27\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-07, atol=1e-14 > > (mismatch 100.0%) > x: array([[ 0.75077639, -0.44156936, 0.47100962], > [ 0.75077639, -0.44156936, 0.48052496], > [ 1.50155279, -0.88313872, 0.95153458],... > y: array([[ 0.72949016, -0.44156936, 0.47100962], > [ 0.72949016, -0.44156936, 0.48052496], > [ 1.45898031, -0.88313872, 0.95153458],... > > ---------------------------------------------------------------------- > Ran 8940 tests in 143.892s > >> Openblas is great if you do not have the patience to build ATLAS and >> only use a restricted set of functionality and platforms you can easily >> test. > > I don't think it's possible to build ATLAS on Windows 64-bit at the > moment, and it would take a lot of work to make it build, and Clint W > has said he does not want to invest much time maintaining the Windows > build, so unless something changes, I think ATLAS is not a viable > option - for 64 bits at least. > From cournape at gmail.com Thu Apr 3 08:58:25 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 3 Apr 2014 13:58:25 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: On Wed, Apr 2, 2014 at 7:52 AM, Matthew Brett wrote: > Hi, > > On Tue, Apr 1, 2014 at 4:46 PM, David Cournapeau > wrote: > > > > > > > > On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith wrote: > >> > >> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau > >> wrote: > >> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith > wrote: > >> >> > >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett < > matthew.brett at gmail.com> > >> >> wrote: > >> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a > DLL > >> >> > loaded via: > >> >> > > >> >> > hDLL = LoadLibraryEx(pathname, NULL, > LOAD_WITH_ALTERED_SEARCH_PATH); > >> >> > > >> >> > will in turn (by default) search for its dependent DLLs in their > own > >> >> > directory. Or maybe in the directory of the first DLL to be > loaded > >> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the > >> >> > documentation. Looking forward to doing my tax return after this. > >> >> > > >> >> > But - anyway - that means that any extensions in the DLLs directory > >> >> > will get their dependencies from the DLLs directory, but that is > only > >> >> > true for extensions in that directory. > >> >> > >> >> So in conclusion, if we just drop our compiled dependencies next to > >> >> the compiled module files then we're good, even on older Windows > >> >> versions? That sounds much simpler than previous discussions, but > good > >> >> news if it's true... > >> > > >> > > >> > That does not work very well in my experience: > >> > > >> > - numpy has extension modules in multiple directories, so we would > >> > need to > >> > copy the dlls in multiple subdirectories > >> > - copying dlls means that windows will load that dll multiple times, > >> > with > >> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've > seen > >> > serious issues when doing something similar for hdf5 dll and > >> > pytables/h5py). > >> > >> We could just ship all numpy's extension modules in the same directory > >> if we wanted. It would be pretty easy to stick some code at the top of > >> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them > >> into the appropriate places in the package namespace. > >> > >> Of course scipy and numpy will still both have to ship BLAS etc., and > >> so I guess it will get loaded at least twice in *any* binary install > >> system. I'm not sure why this would be a problem (Windows, unlike > >> Unix, carefully separates DLL namespaces, right?) > > > > > > It does not really matter here. For pure blas/lapack, that may be ok > because > > the functions are "stateless", but I would not count on it either. > > > > The cleanest solution I can think of is to have 'privately shared DLL', > but > > that would AFAIK require patching python, so not really an option. > > David - do you know anything about private assemblies [1]? > I never managed to make that work properly. > Might they work for our problem? How about AddDllDirectory [2]? > I don't think it is appropriate to use those functions in a C extension module (as it impacts the whole process). David > > Cheers, > > Matthew > > [1] > http://msdn.microsoft.com/en-us/library/windows/desktop/ff951638(v=vs.85).aspx > [2] > http://msdn.microsoft.com/en-us/library/windows/desktop/hh310513(v=vs.85).asp > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Wed Apr 2 09:13:47 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 2 Apr 2014 13:13:47 +0000 (UTC) Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: <133925612418137143.361157sturla.molden-gmail.com@news.gmane.org> alex wrote: > I don't have any opinion about this debate, but I love the > justification in that thread "Any surprise that is created by the > different default should be mitigated by the fact that it's an > opportunity to learn something about what you are doing." That is so true. Sturla From sturla.molden at gmail.com Wed Apr 2 10:06:08 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 2 Apr 2014 14:06:08 +0000 (UTC) Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> Message-ID: <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> wrote: > pandas came later and thought ddof=1 is worth more than consistency. Pandas is a data analysis package. NumPy is a numerical array package. I think ddof=1 is justified for Pandas, for consistency with statistical software (SPSS et al.) For NumPy, there are many computational tasks where the Bessel correction is not wanted, so providing a uncorrected result is the correct thing to do. NumPy should be a low-level array library that does very little magic. Those who need the Bessel correction can multiply with sqrt(n/float(n-1)) or specify ddof. Bu that belongs in the docs. Sturla P.S. Personally I am not convinced "unbiased" is ever a valid argument, as the biased estimator has smaller error. This is from experience in marksmanship: I'd rather shoot a tight series with small systematic error than scatter my bullets wildly but "unbiased" on the target. It is the total error that counts. The series with smallest total error gets the best score. It is better to shoot two series and calibrate the sight in between than use a calibration-free sight that don't allow us to aim. That's why I think classical statistics got this one wrong. Unbiased is never a virtue, but the smallest error is. Thus, if we are to repeat an experiment, we should calibrate our estimator just like a marksman calibrates his sight. But the aim should always be calibrated to give the smallest error, not an unbiased scatter. Noone in their right mind would claim a shotgun is more precise than a rifle because it has smaller bias. But that is what applying the Bessel correction implies. From olivier.grisel at ensta.org Thu Apr 3 09:59:54 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Thu, 3 Apr 2014 15:59:54 +0200 Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe In-Reply-To: <533D5A87.1040907@googlemail.com> References: <1225660970414595360.835902sturla.molden-gmail.com@news.gmane.org> <53331DBF.6020504@googlemail.com> <533D5A87.1040907@googlemail.com> Message-ID: 2014-04-03 14:56 GMT+02:00 Julian Taylor : > FYI, binaries linking openblas should add this patch in some way: > https://github.com/numpy/numpy/pull/4580 > > Cliffs: linking OpenBLAS prevents parallelization via threading or > multiprocessing. > > just wasted a bunch of time figuring that out ... (though its well > documented in numerous stackoverflow questions, too bad none of them > reached us) You mean because of the default CPU affinity stuff in the default OpenBLAS? If we ship OpenBLAS with a windows binary of numpy / scipy we can compile OpenBLAS with the NO_AFFINITY=1 flag to avoid the issue. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From ndbecker2 at gmail.com Thu Apr 3 09:35:26 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 03 Apr 2014 09:35:26 -0400 Subject: [Numpy-discussion] mtrand normal sigma >= 0 too restrictive Message-ID: Traceback (most recent call last): File "./test_inroute_frame.py", line 1694, in run_line (sys.argv) File "./test_inroute_frame.py", line 1690, in run_line return run (opt, cmdline) File "./test_inroute_frame.py", line 1115, in run burst.tr (xbits, freq=freqs[i]+burst.freq_offset, tau=burst.time_offset, phase=burst.phase) File "/home/nbecker/hn-inroute-fixed/transmitter.py", line 191, in __call__ self.channel_out, self.complex_channel_gain = self.channel (mix_out) File "./test_inroute_frame.py", line 105, in __call__ ampl = 10**(0.05*self.pwr_gen()) File "./test_inroute_frame.py", line 148, in __call__ pwr = self.gen() File "./test_inroute_frame.py", line 124, in __call__ x = self.gen() File "/home/nbecker/sigproc.ndarray/normal.py", line 11, in __call__ return self.rs.normal (self.mean, self.std, size) File "mtrand.pyx", line 1479, in mtrand.RandomState.normal (numpy/random/mtrand/mtrand.c:9359) ValueError: scale <= 0 I believe this restriction is too restrictive, and should be scale < 0 There is nothing wrong with scale == 0 as far as I know. It's a convenient way to turn off the noise in my simulation. From josef.pktd at gmail.com Thu Apr 3 13:47:29 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 3 Apr 2014 13:47:29 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> Message-ID: On Wed, Apr 2, 2014 at 10:06 AM, Sturla Molden wrote: > wrote: > >> pandas came later and thought ddof=1 is worth more than consistency. > > Pandas is a data analysis package. NumPy is a numerical array package. > > I think ddof=1 is justified for Pandas, for consistency with statistical > software (SPSS et al.) > > For NumPy, there are many computational tasks where the Bessel correction > is not wanted, so providing a uncorrected result is the correct thing to > do. NumPy should be a low-level array library that does very little magic. > > Those who need the Bessel correction can multiply with sqrt(n/float(n-1)) > or specify ddof. Bu that belongs in the docs. > > > Sturla > > P.S. Personally I am not convinced "unbiased" is ever a valid argument, as > the biased estimator has smaller error. This is from experience in > marksmanship: I'd rather shoot a tight series with small systematic error > than scatter my bullets wildly but "unbiased" on the target. It is the > total error that counts. The series with smallest total error gets the best > score. It is better to shoot two series and calibrate the sight in between > than use a calibration-free sight that don't allow us to aim. calibration == bias correction ? That's why I > think classical statistics got this one wrong. Unbiased is never a virtue, > but the smallest error is. Thus, if we are to repeat an experiment, we > should calibrate our estimator just like a marksman calibrates his sight. > But the aim should always be calibrated to give the smallest error, not an > unbiased scatter. Noone in their right mind would claim a shotgun is more > precise than a rifle because it has smaller bias. But that is what applying > the Bessel correction implies. https://www.youtube.com/watch?v=i4xcEZZDW_I I spent several days trying to figure out what Stata is doing for small sample corrections to reduce the bias of the rejection interval with "uncorrected" variance estimates. Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From mrbago at gmail.com Thu Apr 3 14:21:19 2014 From: mrbago at gmail.com (Bago) Date: Thu, 3 Apr 2014 11:21:19 -0700 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> Message-ID: > Sturla > > P.S. Personally I am not convinced "unbiased" is ever a valid argument, as > the biased estimator has smaller error. This is from experience in > marksmanship: I'd rather shoot a tight series with small systematic error > than scatter my bullets wildly but "unbiased" on the target. It is the > total error that counts. The series with smallest total error gets the best > score. It is better to shoot two series and calibrate the sight in between > than use a calibration-free sight that don't allow us to aim. That's why I > think classical statistics got this one wrong. Unbiased is never a virtue, > but the smallest error is. Thus, if we are to repeat an experiment, we > should calibrate our estimator just like a marksman calibrates his sight. > But the aim should always be calibrated to give the smallest error, not an > unbiased scatter. Noone in their right mind would claim a shotgun is more > precise than a rifle because it has smaller bias. But that is what applying > the Bessel correction implies. > > I agree with the point, and what makes it even worse is that ddof=1 does not even produce an unbiased standard deviation estimate. I produces an unbiased variance estimate but the sqrt of this variance estimate is a biased standard deviation estimate, http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation. Bago -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 3 15:31:25 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 3 Apr 2014 15:31:25 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> Message-ID: On Thu, Apr 3, 2014 at 2:21 PM, Bago wrote: > > > >> >> Sturla >> >> P.S. Personally I am not convinced "unbiased" is ever a valid argument, as >> the biased estimator has smaller error. This is from experience in >> marksmanship: I'd rather shoot a tight series with small systematic error >> than scatter my bullets wildly but "unbiased" on the target. It is the >> total error that counts. The series with smallest total error gets the >> best >> score. It is better to shoot two series and calibrate the sight in between >> than use a calibration-free sight that don't allow us to aim. That's why I >> think classical statistics got this one wrong. Unbiased is never a virtue, >> but the smallest error is. Thus, if we are to repeat an experiment, we >> should calibrate our estimator just like a marksman calibrates his sight. >> But the aim should always be calibrated to give the smallest error, not an >> unbiased scatter. Noone in their right mind would claim a shotgun is more >> precise than a rifle because it has smaller bias. But that is what >> applying >> the Bessel correction implies. >> > > I agree with the point, and what makes it even worse is that ddof=1 does not > even produce an unbiased standard deviation estimate. I produces an unbiased > variance estimate but the sqrt of this variance estimate is a biased > standard deviation estimate, > http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation. But ddof=1 still produces a smaller bias than ddof=0 I think the main point in stats is that without ddof, the variance will be too small and t-test or similar will be liberal in small samples, or confidence intervals will be too short. (for statisticians that prefer to have tests that maintain their level and prefer to err on the "conservative" side.) Josef > > Bago > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at gmail.com Thu Apr 3 16:14:24 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 3 Apr 2014 22:14:24 +0200 Subject: [Numpy-discussion] ANN: Scipy 0.14.0 release candidate 1 Message-ID: Hi, I'm pleased to announce the availability of the first release candidate of Scipy 0.14.0. Please try this RC and report any issues on the scipy-dev mailing list. A significant number of fixes for scipy.sparse went in after the beta release, so users of that module may want to test this release carefully. Source tarballs, binaries and the full release notes can be found at https://sourceforge.net/projects/scipy/files/scipy/0.14.0rc1/. The final release will follow in one week if no new issues are found. A big thank you to everyone who contributed to this release! Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Fri Apr 4 08:50:08 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 4 Apr 2014 14:50:08 +0200 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> Message-ID: On 2 April 2014 16:06, Sturla Molden wrote: > wrote: > > > pandas came later and thought ddof=1 is worth more than consistency. > > Pandas is a data analysis package. NumPy is a numerical array package. > > I think ddof=1 is justified for Pandas, for consistency with statistical > software (SPSS et al.) > > For NumPy, there are many computational tasks where the Bessel correction > is not wanted, so providing a uncorrected result is the correct thing to > do. NumPy should be a low-level array library that does very little magic. All this discussion reminds me of the book "Numerical Recipes": "if the difference between N and N - 1 ever matters to you, then you are probably up to no good anyway -- e.g., trying to substantiate a questionable hypothesis with marginal data." For any reasonably sized data set, it is a correction in the second significant figure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 4 09:04:05 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Apr 2014 09:04:05 -0400 Subject: [Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value In-Reply-To: References: <1CFD8CBC30E0454B9DFAADEB36AD1739B05CB24A4F@LNZEXCHANGE001.linz.fhooe.at> <153135729418075082.303066sturla.molden-gmail.com@news.gmane.org> <1951378129418137831.781579sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Apr 4, 2014 at 8:50 AM, Da?id wrote: > > On 2 April 2014 16:06, Sturla Molden wrote: >> >> wrote: >> >> > pandas came later and thought ddof=1 is worth more than consistency. >> >> Pandas is a data analysis package. NumPy is a numerical array package. >> >> I think ddof=1 is justified for Pandas, for consistency with statistical >> software (SPSS et al.) >> >> For NumPy, there are many computational tasks where the Bessel correction >> is not wanted, so providing a uncorrected result is the correct thing to >> do. NumPy should be a low-level array library that does very little magic. > > > All this discussion reminds me of the book "Numerical Recipes": > > "if the difference between N and N ? 1 ever matters to you, then you > are probably up to no good anyway ? e.g., trying to substantiate a > questionable > hypothesis with marginal data." > > For any reasonably sized data set, it is a correction in the second > significant figure. I fully agree, but sometimes you don't have much choice. `big data` == `statistics with negative degrees of freedom` ? or maybe `machine learning` == `statistics with negative degrees of freedom` ? Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cmkleffner at gmail.com Fri Apr 4 11:19:05 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 4 Apr 2014 17:19:05 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: I'ts time for me to come back to the discussion after a longer break. some personal history: I was looking for a 64bit mingw more than a year ago (unrelated to python) for Fortran development and tried out quite some mingw toolchain variants based on the mingw-w64 project. In a nutshell: the most appropriate and best documentated solution are the toolchains provided by the mingw builds project IMHO. The next step was the idea to use this toolchain for compiling python extensions (C and Fortran) and then to try out compiling numpy and scipy with OpenBLAS. Despite the fact, that a mingw-w64 based toolchain is rock solid today the following possible issues should be considered for Python development: (1) deploy problem: mingw runtime DLLs can not be found at runtime Solution: use flags for static linking or use a dedicated 'static' GCC toolchain for compiling and linking. Both solutions should work. (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since GCC4.6, MSVC uses 4 bytes Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64 X86_64 is not affected. This issue is the major cause for segment faults on 32bit systems. (3) Import library problem: numpy distutils does not play well with mingw-w64 Solution: create a Python import library with the mingw-w64 tools. Use a patched numpy distutils. A detailed description can be found here: http://article.gmane.org/gmane.comp.python.numeric.general/56749 . (4) linking against the correct msvcrXXX Version. Solution: create a 'specs' file (howto see http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes necessary informations. (5) manifest resources Solution: extend the GCC toolchain with the Manifest resource files and ensure linkage with the help of the 'specs' file. (6) Blas Lapack for numpy scipy There is no silver bullet! A trade-off between licence acceptance, performance and stability remains to be found. OpenBLAS on Win32 seems to be quite stable. Some OpenBLAS issues on Win64 can be adressed with a single threaded version of that library. On my google drive: https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharingI provide the necessary parts to try the procedures described at http://article.gmane.org/gmane.comp.python.numeric.general/56749 and http://article.gmane.org/gmane.comp.python.numeric.general/56767 Up to now I didn't find time to put a comprehensive description on the Web and to update all that stuff (MSVCR100 support for the toolchain still missing), so I add my incomplete, not yet published mingw-w64 FAQ at the end of my longish E-Mail for further discussions. Carl --- my personal mingw-w64 FAQ ========================= what is mingw-w64 ----------------- mingw-w64 is a fork of the mingw32 project - http://sourceforge.net/apps/trac/mingw-w64/wiki/History why choose mingw-w64 over mingw ------------------------------- - 32 AND 64bit support - large file support - winpthread pthreads implementation, MIT licenced. - cross compiler toolchains availabe for Linux official mingw-w64 releases --------------------------- source releases of the mingw-64 repository - http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/ official mingw-w64 GCC toolchains --------------------------------- 'recommened' builds are available from the mingw-builds project http://mingw-w64.sourceforge.net/download.php#mingw-builds for example - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/ - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/ These are common combinations of exception and thread models. You can find other combinations as well. Exception handling affects C++ development. Don't ever link object code with different types of exception and/or thread handling! threads concerning the question 'where to find mingw-w64 builds' - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700 - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484 how to build a mingw-w64 based GCC toolchain on Windows ------------------------------------------------------- "mingw-builds" is a set of scripts and patches for compiling the GCC toolchain under Windows with the help of msys2 POSIX enviroment - https://github.com/niXman/mingw-builds/tree/develop recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64 sf.net (4) what is msys2 ------------- msys2 is the successor of msys. Msys2 is necessary as enviroment for the mingw build process on Windows. - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ where to get precompiled mingw-w64 compiled libraries ----------------------------------------------------- recent mingw-w64 based tools and library packages together with sources and patches are available from archlinux as well as from the msys2 maintainers. - http://sourceforge.net/projects/mingw-w64-archlinux/files/ (i686: Sjlj | x86_64: SEH) - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/ (i686: Dwarf | x86_64: SEH) what is a static GCC toolchain ------------------------------ GCC as well as the all neccessary libraries for a toolchain can be compiled with "-disabled-shared". This is supported by the mingw-builds scripts as an option. All the necessary object code from the GCC runtimes will be statically linked into the binaries. As a consequence the binary size will be increased in comparison to the standard toolchains. The advantage is, that there will be no dependancy to external GCC runtime libraries, so the deployment of python extensions is greatly improved. Using such a toolchain is more reliable than using -static-XXX flags. However, exception heavy C++ programms (i.e. QT) should be compiled with shared runtimes to avoid problems with exceptions handling over DLL boundaries. For building typically Python extensions a customized static GCC toolchain is the best compromise IMHO. customizations over standard mingw-builds releases -------------------------------------------------- - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf exceptions) and 64 bit (posix threads, SEH exceptions) - statically build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0 - languages: C, C++, gfortran, LTO - customized 'specs' file for MSVCR90 linkage and manifest support (MSVCR100 linkage coming soon) - additional ftime64 patch to allow winpthreads and OpenMP to work with MSVCR90 linkage - openblas-2.9rc1 with windows thread support (OpenMP disabled) included 2014-04-02 1:46 GMT+02:00 David Cournapeau : > > > > On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith wrote: > >> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau >> wrote: >> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: >> >> >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett > > >> >> wrote: >> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL >> >> > loaded via: >> >> > >> >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> >> > >> >> > will in turn (by default) search for its dependent DLLs in their own >> >> > directory. Or maybe in the directory of the first DLL to be loaded >> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> >> > documentation. Looking forward to doing my tax return after this. >> >> > >> >> > But - anyway - that means that any extensions in the DLLs directory >> >> > will get their dependencies from the DLLs directory, but that is only >> >> > true for extensions in that directory. >> >> >> >> So in conclusion, if we just drop our compiled dependencies next to >> >> the compiled module files then we're good, even on older Windows >> >> versions? That sounds much simpler than previous discussions, but good >> >> news if it's true... >> > >> > >> > That does not work very well in my experience: >> > >> > - numpy has extension modules in multiple directories, so we would >> need to >> > copy the dlls in multiple subdirectories >> > - copying dlls means that windows will load that dll multiple times, >> with >> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen >> > serious issues when doing something similar for hdf5 dll and >> pytables/h5py). >> >> We could just ship all numpy's extension modules in the same directory >> if we wanted. It would be pretty easy to stick some code at the top of >> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them >> into the appropriate places in the package namespace. >> >> Of course scipy and numpy will still both have to ship BLAS etc., and >> so I guess it will get loaded at least twice in *any* binary install >> system. I'm not sure why this would be a problem (Windows, unlike >> Unix, carefully separates DLL namespaces, right?) > > > It does not really matter here. For pure blas/lapack, that may be ok > because the functions are "stateless", but I would not count on it either. > > The cleanest solution I can think of is to have 'privately shared DLL', > but that would AFAIK require patching python, so not really an option. > > > , but if it is a >> problem then it's a very fundamental one for any binaries we ship. >> >> Do the binaries we ship now have this problem? Or are we currently >> managing to statically link everything? >> > > We currently statically link everything. The main challenge is that 'new' > (>= 4) versions of mingw don't easily allow statically linking all the > mingw-related dependencies. While the options are there, everytime I tried > to do it with an official build of mingw, I had some weird, very hard to > track crashes. The other alternative that has been suggested is to build > one own's toolchain where everything is static by default. I am not sure > why that works, and that brings the risk of depending on a toolchain that > we can't really maintain. > > David > >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 4 14:12:06 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Apr 2014 12:12:06 -0600 Subject: [Numpy-discussion] Where to put versionadded Message-ID: Hi All, Currently there are several placements of the '.. versionadded::' directive and I'd like to settle on a proper style for consistency. There are two occasions on which it is used, first, when a new function or class is added and second, when a new keyword is added to an existing function or method. The options are as follows. *New Function* 1) Originally, the directive was added in the notes section. Notes ----- .. versionadded:: 1.5.0 2) Alternatively, it is placed after the extended summary. blah, blah ..versionadded:: 1.5.0 Between these two, I vote for 2) because the version is easily found when reading the documentation either in a terminal or rendered into HTML. *New Parameter* 1) It is placed before the parameter description newoption : int, optional .. versionadded:: 1.5.0 blah. 2) It is placed after the parameter description. newoption : int, optional blah. .. versionadded:: 1.5.0 Both of these render correctly, but the first is more compact while the second puts the version after the description where it doesn't interrupt the reading. I'm tending towards 1) on account of its compactness. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 4 14:48:06 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Apr 2014 14:48:06 -0400 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris wrote: > Hi All, > > Currently there are several placements of the '.. versionadded::' directive > and I'd like to settle > on a proper style for consistency. There are two occasions on which it is > used, first, when a new function or class is added and second, when a new > keyword is added to an existing function or method. The options are as > follows. > > New Function > > 1) Originally, the directive was added in the notes section. > > Notes > ----- > .. versionadded:: 1.5.0 > > 2) Alternatively, it is placed after the extended summary. > > blah, blah > > ..versionadded:: 1.5.0 > > Between these two, I vote for 2) because the version is easily found when > reading the documentation either in a terminal or rendered into HTML. > > New Parameter > > 1) It is placed before the parameter description > > newoption : int, optional > .. versionadded:: 1.5.0 > blah. > > 2) It is placed after the parameter description. > > newoption : int, optional > blah. > > .. versionadded:: 1.5.0 > > Both of these render correctly, but the first is more compact while the > second puts the version > after the description where it doesn't interrupt the reading. I'm tending > towards 1) on account of its compactness. > > Thoughts? I'm in favor of putting them only in the Notes section. Most of the time they are not "crucial" information and it's distracting. I usually only look for them when I'm working explicitly across several numpy versions. like in python: versionadded 2.1 is only interesting for historians. Josef > > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Fri Apr 4 14:54:47 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Apr 2014 12:54:47 -0600 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: On Fri, Apr 4, 2014 at 12:48 PM, wrote: > On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris > wrote: > > Hi All, > > > > Currently there are several placements of the '.. versionadded::' > directive > > and I'd like to settle > > on a proper style for consistency. There are two occasions on which it is > > used, first, when a new function or class is added and second, when a new > > keyword is added to an existing function or method. The options are as > > follows. > > > > New Function > > > > 1) Originally, the directive was added in the notes section. > > > > Notes > > ----- > > .. versionadded:: 1.5.0 > > > > 2) Alternatively, it is placed after the extended summary. > > > > blah, blah > > > > ..versionadded:: 1.5.0 > > > > Between these two, I vote for 2) because the version is easily found when > > reading the documentation either in a terminal or rendered into HTML. > > > > New Parameter > > > > 1) It is placed before the parameter description > > > > newoption : int, optional > > .. versionadded:: 1.5.0 > > blah. > > > > 2) It is placed after the parameter description. > > > > newoption : int, optional > > blah. > > > > .. versionadded:: 1.5.0 > > > > Both of these render correctly, but the first is more compact while the > > second puts the version > > after the description where it doesn't interrupt the reading. I'm tending > > towards 1) on account of its compactness. > > > > Thoughts? > > I'm in favor of putting them only in the Notes section. > > Most of the time they are not "crucial" information and it's > distracting. I usually only look for them when I'm working explicitly > across several numpy versions. > > like in python: versionadded 2.1 is only interesting for historians. > I find the opposite to be true. Because numpy needs maintain compatibility with a number python versions, I often check the python documentation to see in which version a function was added. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Apr 4 15:07:28 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Apr 2014 12:07:28 -0700 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: Hi, On Fri, Apr 4, 2014 at 11:54 AM, Charles R Harris wrote: > > > > On Fri, Apr 4, 2014 at 12:48 PM, wrote: >> >> On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris >> wrote: >> > Hi All, >> > >> > Currently there are several placements of the '.. versionadded::' >> > directive >> > and I'd like to settle >> > on a proper style for consistency. There are two occasions on which it >> > is >> > used, first, when a new function or class is added and second, when a >> > new >> > keyword is added to an existing function or method. The options are as >> > follows. >> > >> > New Function >> > >> > 1) Originally, the directive was added in the notes section. >> > >> > Notes >> > ----- >> > .. versionadded:: 1.5.0 >> > >> > 2) Alternatively, it is placed after the extended summary. >> > >> > blah, blah >> > >> > ..versionadded:: 1.5.0 >> > >> > Between these two, I vote for 2) because the version is easily found >> > when >> > reading the documentation either in a terminal or rendered into HTML. >> > >> > New Parameter >> > >> > 1) It is placed before the parameter description >> > >> > newoption : int, optional >> > .. versionadded:: 1.5.0 >> > blah. >> > >> > 2) It is placed after the parameter description. >> > >> > newoption : int, optional >> > blah. >> > >> > .. versionadded:: 1.5.0 >> > >> > Both of these render correctly, but the first is more compact while the >> > second puts the version >> > after the description where it doesn't interrupt the reading. I'm >> > tending >> > towards 1) on account of its compactness. >> > >> > Thoughts? >> >> I'm in favor of putting them only in the Notes section. >> >> Most of the time they are not "crucial" information and it's >> distracting. I usually only look for them when I'm working explicitly >> across several numpy versions. >> >> like in python: versionadded 2.1 is only interesting for historians. > > > I find the opposite to be true. Because numpy needs maintain compatibility > with a number python versions, I often check the python documentation to see > in which version a function was added. I agree; versionadded 2.1 is not likely interesting but versionadded 2.7 is very interesting. Cheers, Matthew From cmkleffner at gmail.com Fri Apr 4 15:09:01 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 4 Apr 2014 21:09:01 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: I'ts time for me to come back to the discussion after a longer break. some personal history: I was looking for a 64bit mingw more than a year ago (unrelated to python) for Fortran development and tried out quite some mingw toolchain variants based on the mingw-w64 project. In a nutshell: the most appropriate and best documentated solution are the toolchains provided by the mingw builds project IMHO. The next step was the idea to use this toolchain for compiling python extensions (C and Fortran) and then to try out compiling numpy and scipy with OpenBLAS. Despite the fact, that a mingw-w64 based toolchain is rock solid today the following possible issues should be considered for Python development: (1) deploy problem: mingw runtime DLLs can not be found at runtime Solution: use flags for static linking or use a dedicated 'static' GCC toolchain for compiling and linking. Both solutions should work. (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since GCC4.6, MSVC uses 4 bytes Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64 X86_64 is not affected. This issue is the major cause for segment faults on 32bit systems. (3) Import library problem: numpy distutils does not play well with mingw-w64 Solution: create a Python import library with the mingw-w64 tools. Use a patched numpy distutils. A detailed description can be found here: http://article.gmane.org/gmane.comp.python.numeric.general/56749 . (4) linking against the correct msvcrXXX Version. Solution: create a 'specs' file (howto see http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes necessary informations. (5) manifest resources Solution: extend the GCC toolchain with the Manifest resource files and ensure linkage with the help of the 'specs' file. (6) Blas Lapack for numpy scipy There is no silver bullet! A trade-off between licence acceptance, performance and stability remains to be found. OpenBLAS on Win32 seems to be quite stable. Some OpenBLAS issues on Win64 can be adressed with a single threaded version of that library. On my google drive: https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharingI provide the necessary parts to try the procedures described at http://article.gmane.org/gmane.comp.python.numeric.general/56749 and http://article.gmane.org/gmane.comp.python.numeric.general/56767 Up to now I didn't find time to put a comprehensive description on the Web and to update all that stuff (MSVCR100 support for the toolchain still missing), so I add my incomplete, not yet published mingw-w64 FAQ at the end of my longish E-Mail for further discussions. Carl --- my personal mingw-w64 FAQ ========================= what is mingw-w64 ----------------- mingw-w64 is a fork of the mingw32 project - http://sourceforge.net/apps/trac/mingw-w64/wiki/History why choose mingw-w64 over mingw ------------------------------- - 32 AND 64bit support - large file support - winpthread pthreads implementation, MIT licenced. - cross compiler toolchains availabe for Linux official mingw-w64 releases --------------------------- source releases of the mingw-64 repository - http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/ official mingw-w64 GCC toolchains --------------------------------- 'recommened' builds are available from the mingw-builds project http://mingw-w64.sourceforge.net/download.php#mingw-builds for example - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/ - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/ These are common combinations of exception and thread models. You can find other combinations as well. Exception handling affects C++ development. Don't ever link object code with different types of exception and/or thread handling! threads concerning the question 'where to find mingw-w64 builds' - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700 - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484 how to build a mingw-w64 based GCC toolchain on Windows ------------------------------------------------------- "mingw-builds" is a set of scripts and patches for compiling the GCC toolchain under Windows with the help of msys2 POSIX enviroment - https://github.com/niXman/mingw-builds/tree/develop recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64 sf.net (4) what is msys2 ------------- msys2 is the successor of msys. Msys2 is necessary as enviroment for the mingw build process on Windows. - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ where to get precompiled mingw-w64 compiled libraries ----------------------------------------------------- recent mingw-w64 based tools and library packages together with sources and patches are available from archlinux as well as from the msys2 maintainers. - http://sourceforge.net/projects/mingw-w64-archlinux/files/ (i686: Sjlj | x86_64: SEH) - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/ (i686: Dwarf | x86_64: SEH) what is a static GCC toolchain ------------------------------ GCC as well as the all neccessary libraries for a toolchain can be compiled with "-disabled-shared". This is supported by the mingw-builds scripts as an option. All the necessary object code from the GCC runtimes will be statically linked into the binaries. As a consequence the binary size will be increased in comparison to the standard toolchains. The advantage is, that there will be no dependancy to external GCC runtime libraries, so the deployment of python extensions is greatly improved. Using such a toolchain is more reliable than using -static-XXX flags. However, exception heavy C++ programms (i.e. QT) should be compiled with shared runtimes to avoid problems with exceptions handling over DLL boundaries. For building typically Python extensions a customized static GCC toolchain is the best compromise IMHO. customizations over standard mingw-builds releases -------------------------------------------------- - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf exceptions) and 64 bit (posix threads, SEH exceptions) - statically build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0 - languages: C, C++, gfortran, LTO - customized 'specs' file for MSVCR90 linkage and manifest support (MSVCR100 linkage coming soon) - additional ftime64 patch to allow winpthreads and OpenMP to work with MSVCR90 linkage - openblas-2.9rc1 with windows thread support (OpenMP disabled) included 2014-04-02 1:46 GMT+02:00 David Cournapeau : > > > > On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith wrote: > >> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau >> wrote: >> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith wrote: >> >> >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett > > >> >> wrote: >> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL >> >> > loaded via: >> >> > >> >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> >> > >> >> > will in turn (by default) search for its dependent DLLs in their own >> >> > directory. Or maybe in the directory of the first DLL to be loaded >> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> >> > documentation. Looking forward to doing my tax return after this. >> >> > >> >> > But - anyway - that means that any extensions in the DLLs directory >> >> > will get their dependencies from the DLLs directory, but that is only >> >> > true for extensions in that directory. >> >> >> >> So in conclusion, if we just drop our compiled dependencies next to >> >> the compiled module files then we're good, even on older Windows >> >> versions? That sounds much simpler than previous discussions, but good >> >> news if it's true... >> > >> > >> > That does not work very well in my experience: >> > >> > - numpy has extension modules in multiple directories, so we would >> need to >> > copy the dlls in multiple subdirectories >> > - copying dlls means that windows will load that dll multiple times, >> with >> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen >> > serious issues when doing something similar for hdf5 dll and >> pytables/h5py). >> >> We could just ship all numpy's extension modules in the same directory >> if we wanted. It would be pretty easy to stick some code at the top of >> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them >> into the appropriate places in the package namespace. >> >> Of course scipy and numpy will still both have to ship BLAS etc., and >> so I guess it will get loaded at least twice in *any* binary install >> system. I'm not sure why this would be a problem (Windows, unlike >> Unix, carefully separates DLL namespaces, right?) > > > It does not really matter here. For pure blas/lapack, that may be ok > because the functions are "stateless", but I would not count on it either. > > The cleanest solution I can think of is to have 'privately shared DLL', > but that would AFAIK require patching python, so not really an option. > > > , but if it is a >> problem then it's a very fundamental one for any binaries we ship. >> >> Do the binaries we ship now have this problem? Or are we currently >> managing to statically link everything? >> > > We currently statically link everything. The main challenge is that 'new' > (>= 4) versions of mingw don't easily allow statically linking all the > mingw-related dependencies. While the options are there, everytime I tried > to do it with an official build of mingw, I had some weird, very hard to > track crashes. The other alternative that has been suggested is to build > one own's toolchain where everything is static by default. I am not sure > why that works, and that brings the risk of depending on a toolchain that > we can't really maintain. > > David > >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 4 15:33:45 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Apr 2014 15:33:45 -0400 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: On Fri, Apr 4, 2014 at 3:07 PM, Matthew Brett wrote: > Hi, > > On Fri, Apr 4, 2014 at 11:54 AM, Charles R Harris > wrote: >> >> >> >> On Fri, Apr 4, 2014 at 12:48 PM, wrote: >>> >>> On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris >>> wrote: >>> > Hi All, >>> > >>> > Currently there are several placements of the '.. versionadded::' >>> > directive >>> > and I'd like to settle >>> > on a proper style for consistency. There are two occasions on which it >>> > is >>> > used, first, when a new function or class is added and second, when a >>> > new >>> > keyword is added to an existing function or method. The options are as >>> > follows. >>> > >>> > New Function >>> > >>> > 1) Originally, the directive was added in the notes section. >>> > >>> > Notes >>> > ----- >>> > .. versionadded:: 1.5.0 >>> > >>> > 2) Alternatively, it is placed after the extended summary. >>> > >>> > blah, blah >>> > >>> > ..versionadded:: 1.5.0 >>> > >>> > Between these two, I vote for 2) because the version is easily found >>> > when >>> > reading the documentation either in a terminal or rendered into HTML. >>> > >>> > New Parameter >>> > >>> > 1) It is placed before the parameter description >>> > >>> > newoption : int, optional >>> > .. versionadded:: 1.5.0 >>> > blah. >>> > >>> > 2) It is placed after the parameter description. >>> > >>> > newoption : int, optional >>> > blah. >>> > >>> > .. versionadded:: 1.5.0 >>> > >>> > Both of these render correctly, but the first is more compact while the >>> > second puts the version >>> > after the description where it doesn't interrupt the reading. I'm >>> > tending >>> > towards 1) on account of its compactness. >>> > >>> > Thoughts? >>> >>> I'm in favor of putting them only in the Notes section. >>> >>> Most of the time they are not "crucial" information and it's >>> distracting. I usually only look for them when I'm working explicitly >>> across several numpy versions. >>> >>> like in python: versionadded 2.1 is only interesting for historians. >> >> >> I find the opposite to be true. Because numpy needs maintain compatibility >> with a number python versions, I often check the python documentation to see >> in which version a function was added. > > I agree; versionadded 2.1 is not likely interesting but versionadded > 2.7 is very interesting. That's true, but it's a mess for maintainers because we support now 5 python versions. numpy doesn't have a long history of versionadded yet, I didn't find anything for 1.3 in a quick search. statsmodels has now numpy 1.6 as minimum requirement and I'm interested in the features that become available with a minimum version increase. Once I know what I'm allowed to use, I only care about the "real" documentation, "How does einsum really work?". But as a numpy user, I was never really interested in the information that arraysetops where enhanced and renamed in numpy 1.x (x=?<4), or that take was added in 0.y, ... Even the first part of polynomial is already in 1.4 (It might just make me feel old if I remember when it was changed.) versionadded is not very distracting in the html rendering, so I'm just +0.1 on Notes. Josef > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Fri Apr 4 15:56:08 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Apr 2014 12:56:08 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, On Fri, Apr 4, 2014 at 8:19 AM, Carl Kleffner wrote: > > I'ts time for me to come back to the discussion after a longer break. > > some personal history: I was looking for a 64bit mingw more than a year ago > (unrelated to python) for Fortran development and tried out quite some mingw > toolchain variants based on the mingw-w64 project. In a nutshell: the most > appropriate and best documentated solution are the toolchains provided by > the mingw builds project IMHO. > > The next step was the idea to use this toolchain for compiling python > extensions (C and Fortran) and then to try out compiling numpy and scipy > with OpenBLAS. > > Despite the fact, that a mingw-w64 based toolchain is rock solid today the > following possible issues should be considered for Python development: > > (1) deploy problem: mingw runtime DLLs can not be found at runtime > Solution: use flags for static linking or use a dedicated 'static' GCC > toolchain for compiling and linking. Both solutions should work. > > (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since > GCC4.6, MSVC uses 4 bytes > Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64 > X86_64 is not affected. This issue is the major cause for segment faults on > 32bit systems. > > (3) Import library problem: numpy distutils does not play well with > mingw-w64 > Solution: create a Python import library with the mingw-w64 tools. Use a > patched numpy distutils. A detailed description can be found here: > http://article.gmane.org/gmane.comp.python.numeric.general/56749 . > > (4) linking against the correct msvcrXXX Version. > Solution: create a 'specs' file (howto see > http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes > necessary informations. > > (5) manifest resources > Solution: extend the GCC toolchain with the Manifest resource files and > ensure linkage with the help of the 'specs' file. > > (6) Blas Lapack for numpy scipy > There is no silver bullet! A trade-off between licence acceptance, > performance and stability remains to be found. OpenBLAS on Win32 seems to be > quite stable. Some OpenBLAS issues on Win64 can be adressed with a single > threaded version of that library. > > On my google drive: > https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharing > I provide the necessary parts to try the procedures described at > http://article.gmane.org/gmane.comp.python.numeric.general/56749 and > http://article.gmane.org/gmane.comp.python.numeric.general/56767 > > Up to now I didn't find time to put a comprehensive description on the Web > and to update all that stuff (MSVCR100 support for the toolchain still > missing), so I add my incomplete, not yet published mingw-w64 FAQ at the end > of my longish E-Mail for further discussions. > > Carl > > --- > > my personal mingw-w64 FAQ > ========================= > > what is mingw-w64 > ----------------- > > mingw-w64 is a fork of the mingw32 project > - http://sourceforge.net/apps/trac/mingw-w64/wiki/History > > why choose mingw-w64 over mingw > ------------------------------- > > - 32 AND 64bit support > - large file support > - winpthread pthreads implementation, MIT licenced. > - cross compiler toolchains availabe for Linux > > official mingw-w64 releases > --------------------------- > > source releases of the mingw-64 repository > - > http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/ > > official mingw-w64 GCC toolchains > --------------------------------- > > 'recommened' builds are available from the mingw-builds project > http://mingw-w64.sourceforge.net/download.php#mingw-builds for example > - > http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/ > - > http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/ > > These are common combinations of exception and thread models. You can find > other combinations as well. Exception handling affects C++ development. > Don't ever link object code with different types of exception and/or thread > handling! > > threads concerning the question 'where to find mingw-w64 builds' > - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700 > - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484 > > how to build a mingw-w64 based GCC toolchain on Windows > ------------------------------------------------------- > > "mingw-builds" is a set of scripts and patches for compiling the GCC > toolchain under Windows with the help of msys2 POSIX enviroment > - https://github.com/niXman/mingw-builds/tree/develop > recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64 > sf.net (4) > > what is msys2 > ------------- > > msys2 is the successor of msys. Msys2 is necessary as enviroment for the > mingw build process on Windows. > - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ > > where to get precompiled mingw-w64 compiled libraries > ----------------------------------------------------- > > recent mingw-w64 based tools and library packages together with sources and > patches are available from archlinux as well as from the msys2 maintainers. > - http://sourceforge.net/projects/mingw-w64-archlinux/files/ (i686: Sjlj | > x86_64: SEH) > - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/ (i686: Dwarf | > x86_64: SEH) > > what is a static GCC toolchain > ------------------------------ > > GCC as well as the all neccessary libraries for a toolchain can be compiled > with "-disabled-shared". This is supported by the mingw-builds scripts as an > option. All the necessary object code from the GCC runtimes will be > statically linked into the binaries. As a consequence the binary size will > be increased in comparison to the standard toolchains. The advantage is, > that there will be no dependancy to external GCC runtime libraries, so the > deployment of python extensions is greatly improved. Using such a toolchain > is more reliable than using -static-XXX flags. > However, exception heavy C++ programms (i.e. QT) should be compiled with > shared runtimes to avoid problems with exceptions handling over DLL > boundaries. > For building typically Python extensions a customized static GCC toolchain > is the best compromise IMHO. > > customizations over standard mingw-builds releases > -------------------------------------------------- > > - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf > exceptions) and 64 bit (posix threads, SEH exceptions) > - statically build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0 > - languages: C, C++, gfortran, LTO > - customized 'specs' file for MSVCR90 linkage and manifest support (MSVCR100 > linkage coming soon) > - additional ftime64 patch to allow winpthreads and OpenMP to work with > MSVCR90 linkage > - openblas-2.9rc1 with windows thread support (OpenMP disabled) included Thanks very much for this. Would you consider putting this up as a numpy wiki page? https://github.com/numpy/numpy/wiki I think it would be very valuable... Cheers, Matthew From josef.pktd at gmail.com Fri Apr 4 16:08:08 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Apr 2014 16:08:08 -0400 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: On Fri, Apr 4, 2014 at 3:33 PM, wrote: > On Fri, Apr 4, 2014 at 3:07 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Apr 4, 2014 at 11:54 AM, Charles R Harris >> wrote: >>> >>> >>> >>> On Fri, Apr 4, 2014 at 12:48 PM, wrote: >>>> >>>> On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris >>>> wrote: >>>> > Hi All, >>>> > >>>> > Currently there are several placements of the '.. versionadded::' >>>> > directive >>>> > and I'd like to settle >>>> > on a proper style for consistency. There are two occasions on which it >>>> > is >>>> > used, first, when a new function or class is added and second, when a >>>> > new >>>> > keyword is added to an existing function or method. The options are as >>>> > follows. >>>> > >>>> > New Function >>>> > >>>> > 1) Originally, the directive was added in the notes section. >>>> > >>>> > Notes >>>> > ----- >>>> > .. versionadded:: 1.5.0 >>>> > >>>> > 2) Alternatively, it is placed after the extended summary. >>>> > >>>> > blah, blah >>>> > >>>> > ..versionadded:: 1.5.0 >>>> > >>>> > Between these two, I vote for 2) because the version is easily found >>>> > when >>>> > reading the documentation either in a terminal or rendered into HTML. >>>> > >>>> > New Parameter >>>> > >>>> > 1) It is placed before the parameter description >>>> > >>>> > newoption : int, optional >>>> > .. versionadded:: 1.5.0 >>>> > blah. >>>> > >>>> > 2) It is placed after the parameter description. >>>> > >>>> > newoption : int, optional >>>> > blah. >>>> > >>>> > .. versionadded:: 1.5.0 >>>> > >>>> > Both of these render correctly, but the first is more compact while the >>>> > second puts the version >>>> > after the description where it doesn't interrupt the reading. I'm >>>> > tending >>>> > towards 1) on account of its compactness. >>>> > >>>> > Thoughts? >>>> >>>> I'm in favor of putting them only in the Notes section. >>>> >>>> Most of the time they are not "crucial" information and it's >>>> distracting. I usually only look for them when I'm working explicitly >>>> across several numpy versions. >>>> >>>> like in python: versionadded 2.1 is only interesting for historians. since I like history: AFAICS: arraysetops was changed in 1.4 histogram was added in 0.4 corrcoef was added in 0.9.2 numpy 0.9.2 is 8 years old python 2.1 has soon it's 13th anniversary Josef >>> >>> >>> I find the opposite to be true. Because numpy needs maintain compatibility >>> with a number python versions, I often check the python documentation to see >>> in which version a function was added. >> >> I agree; versionadded 2.1 is not likely interesting but versionadded >> 2.7 is very interesting. > > That's true, but it's a mess for maintainers because we support now 5 > python versions. > > numpy doesn't have a long history of versionadded yet, I didn't find > anything for 1.3 in a quick search. > statsmodels has now numpy 1.6 as minimum requirement and I'm > interested in the features that become available with a minimum > version increase. > Once I know what I'm allowed to use, I only care about the "real" > documentation, "How does einsum really work?". > > But as a numpy user, I was never really interested in the information > that arraysetops where enhanced and renamed in numpy 1.x (x=?<4), or > that take was added in 0.y, ... Even the first part of polynomial is > already in 1.4 > (It might just make me feel old if I remember when it was changed.) > > versionadded is not very distracting in the html rendering, so I'm > just +0.1 on Notes. > > Josef > >> >> Cheers, >> >> Matthew >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion From davidmenhur at gmail.com Fri Apr 4 18:37:21 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sat, 5 Apr 2014 00:37:21 +0200 Subject: [Numpy-discussion] Where to put versionadded In-Reply-To: References: Message-ID: On Apr 4, 2014 8:54 PM, "Charles R Harris" wrote: > > > > > On Fri, Apr 4, 2014 at 12:48 PM, wrote: >> >> On Fri, Apr 4, 2014 at 2:12 PM, Charles R Harris >> wrote: >> > Hi All, >> > >> > Currently there are several placements of the '.. versionadded::' directive >> > and I'd like to settle >> > on a proper style for consistency. There are two occasions on which it is >> > used, first, when a new function or class is added and second, when a new >> > keyword is added to an existing function or method. The options are as >> > follows. >> > >> > New Function >> > >> > 1) Originally, the directive was added in the notes section. >> > >> > Notes >> > ----- >> > .. versionadded:: 1.5.0 >> > >> > 2) Alternatively, it is placed after the extended summary. >> > >> > blah, blah >> > >> > ..versionadded:: 1.5.0 >> > >> > Between these two, I vote for 2) because the version is easily found when >> > reading the documentation either in a terminal or rendered into HTML. >> > >> > New Parameter >> > >> > 1) It is placed before the parameter description >> > >> > newoption : int, optional >> > .. versionadded:: 1.5.0 >> > blah. >> > >> > 2) It is placed after the parameter description. >> > >> > newoption : int, optional >> > blah. >> > >> > .. versionadded:: 1.5.0 >> > >> > Both of these render correctly, but the first is more compact while the >> > second puts the version >> > after the description where it doesn't interrupt the reading. I'm tending >> > towards 1) on account of its compactness. >> > >> > Thoughts? >> >> I'm in favor of putting them only in the Notes section. >> >> Most of the time they are not "crucial" information and it's >> distracting. I usually only look for them when I'm working explicitly >> across several numpy versions. >> >> like in python: versionadded 2.1 is only interesting for historians. > > > I find the opposite to be true. Because numpy needs maintain compatibility with a number python versions, I often check the python documentation to see in which version a function was added. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > My user perspective: I am developing a tool whose main use is to run on my computer, so I prefer to run the newest and sweetest version of the libraries, and I this report the minimum versions. But it would be good if I could grep my code, see what numpy functions are being used and infer a probable minimum version required. If other libraries follow similar conventions, and one does not do metaprogramming or other fancy things, it is relatively easy to get automatically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Sat Apr 5 17:53:34 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sat, 5 Apr 2014 23:53:34 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.8.1 release In-Reply-To: References: <5332135C.7040903@googlemail.com> Message-ID: Hi, I will add a a wiki page ASAP. BTW: I copied my tools (gcc toolchain, numpy, scipy wheels) from my google drive to bitbucket: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads Regards Carl 2014-04-04 21:56 GMT+02:00 Matthew Brett : > Hi, > > On Fri, Apr 4, 2014 at 8:19 AM, Carl Kleffner > wrote: > > > > I'ts time for me to come back to the discussion after a longer break. > > > > some personal history: I was looking for a 64bit mingw more than a year > ago > > (unrelated to python) for Fortran development and tried out quite some > mingw > > toolchain variants based on the mingw-w64 project. In a nutshell: the > most > > appropriate and best documentated solution are the toolchains provided by > > the mingw builds project IMHO. > > > > The next step was the idea to use this toolchain for compiling python > > extensions (C and Fortran) and then to try out compiling numpy and scipy > > with OpenBLAS. > > > > Despite the fact, that a mingw-w64 based toolchain is rock solid today > the > > following possible issues should be considered for Python development: > > > > (1) deploy problem: mingw runtime DLLs can not be found at runtime > > Solution: use flags for static linking or use a dedicated 'static' GCC > > toolchain for compiling and linking. Both solutions should work. > > > > (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes > since > > GCC4.6, MSVC uses 4 bytes > > Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64 > > X86_64 is not affected. This issue is the major cause for segment faults > on > > 32bit systems. > > > > (3) Import library problem: numpy distutils does not play well with > > mingw-w64 > > Solution: create a Python import library with the mingw-w64 tools. Use a > > patched numpy distutils. A detailed description can be found here: > > http://article.gmane.org/gmane.comp.python.numeric.general/56749 . > > > > (4) linking against the correct msvcrXXX Version. > > Solution: create a 'specs' file (howto see > > http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes > > necessary informations. > > > > (5) manifest resources > > Solution: extend the GCC toolchain with the Manifest resource files and > > ensure linkage with the help of the 'specs' file. > > > > (6) Blas Lapack for numpy scipy > > There is no silver bullet! A trade-off between licence acceptance, > > performance and stability remains to be found. OpenBLAS on Win32 seems > to be > > quite stable. Some OpenBLAS issues on Win64 can be adressed with a single > > threaded version of that library. > > > > On my google drive: > > > https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharing > > I provide the necessary parts to try the procedures described at > > http://article.gmane.org/gmane.comp.python.numeric.general/56749 and > > http://article.gmane.org/gmane.comp.python.numeric.general/56767 > > > > Up to now I didn't find time to put a comprehensive description on the > Web > > and to update all that stuff (MSVCR100 support for the toolchain still > > missing), so I add my incomplete, not yet published mingw-w64 FAQ at the > end > > of my longish E-Mail for further discussions. > > > > Carl > > > > --- > > > > my personal mingw-w64 FAQ > > ========================= > > > > what is mingw-w64 > > ----------------- > > > > mingw-w64 is a fork of the mingw32 project > > - http://sourceforge.net/apps/trac/mingw-w64/wiki/History > > > > why choose mingw-w64 over mingw > > ------------------------------- > > > > - 32 AND 64bit support > > - large file support > > - winpthread pthreads implementation, MIT licenced. > > - cross compiler toolchains availabe for Linux > > > > official mingw-w64 releases > > --------------------------- > > > > source releases of the mingw-64 repository > > - > > > http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/ > > > > official mingw-w64 GCC toolchains > > --------------------------------- > > > > 'recommened' builds are available from the mingw-builds project > > http://mingw-w64.sourceforge.net/download.php#mingw-builds for example > > - > > > http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/ > > - > > > http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/ > > > > These are common combinations of exception and thread models. You can > find > > other combinations as well. Exception handling affects C++ development. > > Don't ever link object code with different types of exception and/or > thread > > handling! > > > > threads concerning the question 'where to find mingw-w64 builds' > > - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700 > > - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484 > > > > how to build a mingw-w64 based GCC toolchain on Windows > > ------------------------------------------------------- > > > > "mingw-builds" is a set of scripts and patches for compiling the GCC > > toolchain under Windows with the help of msys2 POSIX enviroment > > - https://github.com/niXman/mingw-builds/tree/develop > > recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64 > > sf.net (4) > > > > what is msys2 > > ------------- > > > > msys2 is the successor of msys. Msys2 is necessary as enviroment for the > > mingw build process on Windows. > > - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ > > > > where to get precompiled mingw-w64 compiled libraries > > ----------------------------------------------------- > > > > recent mingw-w64 based tools and library packages together with sources > and > > patches are available from archlinux as well as from the msys2 > maintainers. > > - http://sourceforge.net/projects/mingw-w64-archlinux/files/ (i686: > Sjlj | > > x86_64: SEH) > > - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/ (i686: > Dwarf | > > x86_64: SEH) > > > > what is a static GCC toolchain > > ------------------------------ > > > > GCC as well as the all neccessary libraries for a toolchain can be > compiled > > with "-disabled-shared". This is supported by the mingw-builds scripts > as an > > option. All the necessary object code from the GCC runtimes will be > > statically linked into the binaries. As a consequence the binary size > will > > be increased in comparison to the standard toolchains. The advantage is, > > that there will be no dependancy to external GCC runtime libraries, so > the > > deployment of python extensions is greatly improved. Using such a > toolchain > > is more reliable than using -static-XXX flags. > > However, exception heavy C++ programms (i.e. QT) should be compiled with > > shared runtimes to avoid problems with exceptions handling over DLL > > boundaries. > > For building typically Python extensions a customized static GCC > toolchain > > is the best compromise IMHO. > > > > customizations over standard mingw-builds releases > > -------------------------------------------------- > > > > - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf > > exceptions) and 64 bit (posix threads, SEH exceptions) > > - statically build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0 > > - languages: C, C++, gfortran, LTO > > - customized 'specs' file for MSVCR90 linkage and manifest support > (MSVCR100 > > linkage coming soon) > > - additional ftime64 patch to allow winpthreads and OpenMP to work with > > MSVCR90 linkage > > - openblas-2.9rc1 with windows thread support (OpenMP disabled) included > > Thanks very much for this. > > Would you consider putting this up as a numpy wiki page? > > https://github.com/numpy/numpy/wiki > > I think it would be very valuable... > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Apr 6 03:18:04 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 6 Apr 2014 00:18:04 -0700 Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe In-Reply-To: <53331DBF.6020504@googlemail.com> References: <1225660970414595360.835902sturla.molden-gmail.com@news.gmane.org> <53331DBF.6020504@googlemail.com> Message-ID: Hi, On Wed, Mar 26, 2014 at 11:34 AM, Julian Taylor wrote: > On 26.03.2014 16:27, Olivier Grisel wrote: >> Hi Carl, >> >> I installed Python 2.7.6 64 bits on a windows server instance from >> rackspace cloud and then ran get-pip.py and then could successfully >> install the numpy and scipy wheel packages from your google drive >> folder. I tested dot products and scipy.linalg.svd and they work as >> expected. >> > >> >> Would it make sense to embed the blas and lapack header files as part >> of this numpy wheel and make numpy.distutils.system_info return the >> lib and include folder pointing to the embedded libopenblas.dll and >> header files so has to make third party libraries directly buildable >> against those? >> > > as for using openblas by default in binary builds, no. > pthread openblas build is now fork safe which is great but it is still > not reliable enough for a default. > E.g. the current latest release 0.2.8 still has one crash bug on > dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3]. > git head has the former four fixed bug still has wrong results for cgemv. > The not so old 0.2.8 also fixed whole bunch more crashes and wrong > result issues (crashes on QR, uninitialized data use in dgemm, ...). > None of the fixes received unit tests, so I am somewhat pessimistic that > it will improve, especially as the maintainer is dissertating (is that > the right word?) and most of the code is assembler code only few people > can write (it is simply not required anymore, we have code generators > and intrinsics for that). > > Openblas is great if you do not have the patience to build ATLAS and > only use a restricted set of functionality and platforms you can easily > test. > Currently it is in my opinion not suitable for a general purpose library > like numpy. > > I don't have any objections to adding get_info("openblas") if that does > not work yet. Patches welcome. Julian - do you have any opinion on using gotoBLAS instead of OpenBLAS for the Windows binaries? Cheers, Matthew From faltet at gmail.com Sun Apr 6 06:51:16 2014 From: faltet at gmail.com (Francesc Alted) Date: Sun, 06 Apr 2014 12:51:16 +0200 Subject: [Numpy-discussion] ANN: numexpr 2.4 RC1 Message-ID: <534131A4.7020104@gmail.com> =========================== Announcing Numexpr 2.4 RC1 =========================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== A new `contains()` function has been added for detecting substrings in strings. Thanks to Marcin Krol. Also, there is a new version of setup.py that allows better management of the NumPy dependency during pip installs. Thanks to Aleks Bunin. This is the first release candidate before 2.4 final would be out, so please give it a go and report back any problems you may have. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted From sturla.molden at gmail.com Sun Apr 6 14:47:21 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 6 Apr 2014 18:47:21 +0000 (UTC) Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe References: <53331DBF.6020504@googlemail.com> Message-ID: <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Julian - do you have any opinion on using gotoBLAS instead of OpenBLAS > for the Windows binaries? That is basically OpenBLAS too, except with more bugs and no AVX support. Sturla From matthew.brett at gmail.com Sun Apr 6 14:59:35 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 6 Apr 2014 11:59:35 -0700 Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe In-Reply-To: <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> References: <53331DBF.6020504@googlemail.com> <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Sun, Apr 6, 2014 at 11:47 AM, Sturla Molden wrote: > Matthew Brett wrote: > >> Julian - do you have any opinion on using gotoBLAS instead of OpenBLAS >> for the Windows binaries? > > That is basically OpenBLAS too, except with more bugs and no AVX support. I know that OpenBLAS is a fork of gotoBLAS2, but you said in another thread that gotoBLAS2 was 'rock-solid'. If the bugs in gotoBLAS2 do not in practice arise for default Windows builds, then it could be a good option until OpenBLAS is more mature. Put another way - does anyone know what bugs in gotoBLAS2 do arise for Windows / Intel builds? Cheers, Matthew From cmkleffner at gmail.com Sun Apr 6 15:47:43 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sun, 6 Apr 2014 21:47:43 +0200 Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe In-Reply-To: References: <53331DBF.6020504@googlemail.com> <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> Message-ID: MKL BLAS LAPACK has issues as well: http://software.intel.com/en-us/articles/intel-mkl-110-bug-fixes . In case of OpenBLAS or GOTOBLAS what precisly is the problem you identify as showstopper? Regards Carl 2014-04-06 20:59 GMT+02:00 Matthew Brett : > Hi, > > On Sun, Apr 6, 2014 at 11:47 AM, Sturla Molden > wrote: > > Matthew Brett wrote: > > > >> Julian - do you have any opinion on using gotoBLAS instead of OpenBLAS > >> for the Windows binaries? > > > > That is basically OpenBLAS too, except with more bugs and no AVX support. > > I know that OpenBLAS is a fork of gotoBLAS2, but you said in another > thread that gotoBLAS2 was 'rock-solid'. If the bugs in gotoBLAS2 do > not in practice arise for default Windows builds, then it could be a > good option until OpenBLAS is more mature. > > Put another way - does anyone know what bugs in gotoBLAS2 do arise for > Windows / Intel builds? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sun Apr 6 16:05:10 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 06 Apr 2014 22:05:10 +0200 Subject: [Numpy-discussion] numpy git master requiring cython for build Message-ID: <5341B376.6090100@googlemail.com> hi, numpy.random is largely built from a cython file. Up to know numpy git included generated c sources for this one file. It is troublesome to have merge 20k line changes for one line bugfixes, so it is proposed to remove the generated sources from the master branch in this PR: https://github.com/numpy/numpy/pull/4589 Like in SciPy the sources will not be contained in git anymore but built by the regular setup.py build when required. Release source tarballs (sdist) will continue contain the generated sources so users of stable release are not required to have cython installed. If you have any objects to this please speak up soon. Cheers, Julian From cgohlke at uci.edu Sun Apr 6 16:13:55 2014 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 06 Apr 2014 13:13:55 -0700 Subject: [Numpy-discussion] ANN: numexpr 2.4 RC1 In-Reply-To: <534131A4.7020104@gmail.com> References: <534131A4.7020104@gmail.com> Message-ID: <5341B583.1000500@uci.edu> On 4/6/2014 3:51 AM, Francesc Alted wrote: > =========================== > Announcing Numexpr 2.4 RC1 > =========================== > > Numexpr is a fast numerical expression evaluator for NumPy. With it, > expressions that operate on arrays (like "3*a+4*b") are accelerated > and use less memory than doing the same calculation in Python. > > It wears multi-threaded capabilities, as well as support for Intel's > MKL (Math Kernel Library), which allows an extremely fast evaluation > of transcendental functions (sin, cos, tan, exp, log...) while > squeezing the last drop of performance out of your multi-core > processors. Look here for a some benchmarks of numexpr using MKL: > > https://github.com/pydata/numexpr/wiki/NumexprMKL > > Its only dependency is NumPy (MKL is optional), so it works well as an > easy-to-deploy, easy-to-use, computational engine for projects that > don't want to adopt other solutions requiring more heavy dependencies. > > What's new > ========== > > A new `contains()` function has been added for detecting substrings in > strings. Thanks to Marcin Krol. > > Also, there is a new version of setup.py that allows better management > of the NumPy dependency during pip installs. Thanks to Aleks Bunin. > > This is the first release candidate before 2.4 final would be out, > so please give it a go and report back any problems you may have. > > In case you want to know more in detail what has changed in this > version, see: > > https://github.com/pydata/numexpr/wiki/Release-Notes > > or have a look at RELEASE_NOTES.txt in the tarball. > > Where I can find Numexpr? > ========================= > > The project is hosted at GitHub in: > > https://github.com/pydata/numexpr > > You can get the packages from PyPI as well (but not for RC releases): > > http://pypi.python.org/pypi/numexpr > > Share your experience > ===================== > > Let us know of any bugs, suggestions, gripes, kudos, etc. you may > have. > > > Enjoy data! > Hi Francesc, some minor issues building with msvc/MKL on Windows: * the `numexpr\win32` folder is missing in the PyPI package numexpr-2.4-rc1.tar.gz * `numexpr\interpreter.cpp` uses POSIX syslog.h, which is not available with msvc9/10 * `numexpr\interpreter.cpp` uses the C99 fminl function, which is not available with msvc9/10 I'll submit a PR later for the latter two. Some tests fail on Python 3 (attached). Christoph -------------- next part -------------- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Numexpr version: 2.4-rc1 NumPy version: 1.8.1 Python version: 3.3.5 (v3.3.5:62cf4e77f785, Mar 9 2014, 10:35:05) [MSC v.1600 64 bit (AMD64)] AMD/Intel CPU? True VML available? True VML/MKL version: Intel(R) Math Kernel Library Version 11.1.2 Product Build 20140122 for Intel(R) 64 architecture appli cations Number of threads used by default: 8 (out of 8 detected cores) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ........EEEEEEEEE........EEEEEEEEE...................................................................................... ====================================================================== ERROR: test_str_contains_basic0 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 735, in evaluate compiled_ex = _numexpr_cache[numexpr_key] KeyError: ('contains("abc", "ab")', (('optimization', 'aggressive'), ('truediv', False)), ()) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 166, in test_str_contains_basic0 res = evaluate('contains("abc", "ab")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 738, in evaluate NumExpr(ex, signature, **context) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 554, in NumExpr precompile(ex, signature, context) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 511, in precompile constants_order, constants = getConstants(ast) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 294, in getConstants for a in constants_order] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 294, in for a in constants_order] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 284, in convertConstantToKind return kind_to_type[kind](x) TypeError: string argument without an encoding ====================================================================== ERROR: test_str_contains_basic1 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 171, in test_str_contains_basic1 res = evaluate('contains(haystack, "ab")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_basic2 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 176, in test_str_contains_basic2 res = evaluate('contains("abcd", haystack)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_basic3 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 182, in test_str_contains_basic3 res = evaluate('contains(haystacks, needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str224 ====================================================================== ERROR: test_str_contains_basic4 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 188, in test_str_contains_basic4 res = evaluate('contains("test abc here", needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str160 ====================================================================== ERROR: test_str_contains_basic5 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 193, in test_str_contains_basic5 res = evaluate('contains("test abc here", needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str160 ====================================================================== ERROR: test_str_contains_listproduct (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 246, in test_str_contains_listproduct res = [bool(x) for x in evaluate('contains(b, a)')] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str192 ====================================================================== ERROR: test_str_contains_withemptystr1 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 251, in test_str_contains_withemptystr1 res = evaluate('contains("abcd", withemptystr)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_withemptystr2 (numexpr.tests.test_numexpr.test_numexpr) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 256, in test_str_contains_withemptystr2 res = evaluate('contains(withemptystr, "")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_basic0 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 735, in evaluate compiled_ex = _numexpr_cache[numexpr_key] KeyError: ('contains("abc", "ab")', (('optimization', 'aggressive'), ('truediv', False)), ()) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 166, in test_str_contains_basic0 res = evaluate('contains("abc", "ab")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 738, in evaluate NumExpr(ex, signature, **context) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 554, in NumExpr precompile(ex, signature, context) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 511, in precompile constants_order, constants = getConstants(ast) File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 294, in getConstants for a in constants_order] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 294, in for a in constants_order] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 284, in convertConstantToKind return kind_to_type[kind](x) TypeError: string argument without an encoding ====================================================================== ERROR: test_str_contains_basic1 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 171, in test_str_contains_basic1 res = evaluate('contains(haystack, "ab")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_basic2 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 176, in test_str_contains_basic2 res = evaluate('contains("abcd", haystack)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_basic3 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 182, in test_str_contains_basic3 res = evaluate('contains(haystacks, needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str224 ====================================================================== ERROR: test_str_contains_basic4 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 188, in test_str_contains_basic4 res = evaluate('contains("test abc here", needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str160 ====================================================================== ERROR: test_str_contains_basic5 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 193, in test_str_contains_basic5 res = evaluate('contains("test abc here", needles)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str160 ====================================================================== ERROR: test_str_contains_listproduct (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 246, in test_str_contains_listproduct res = [bool(x) for x in evaluate('contains(b, a)')] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str192 ====================================================================== ERROR: test_str_contains_withemptystr1 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 251, in test_str_contains_withemptystr1 res = evaluate('contains("abcd", withemptystr)') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ====================================================================== ERROR: test_str_contains_withemptystr2 (numexpr.tests.test_numexpr.test_numexpr2) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numexpr\tests\test_numexpr.py", line 256, in test_str_contains_withemptystr2 res = evaluate('contains(withemptystr, "")') File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in evaluate signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 730, in signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)] File "X:\Python33\lib\site-packages\numexpr\necompiler.py", line 629, in getType raise ValueError("unkown type %s" % a.dtype.name) ValueError: unkown type str96 ---------------------------------------------------------------------- Ran 5431 tests in 5.938s FAILED (errors=18) From njs at pobox.com Sun Apr 6 17:01:27 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 Apr 2014 22:01:27 +0100 Subject: [Numpy-discussion] Resolving the associativity/precedence debate for @ In-Reply-To: References: Message-ID: On Tue, Apr 1, 2014 at 4:13 PM, Charles R Harris wrote: > > > > On Mon, Mar 24, 2014 at 6:33 PM, Nathaniel Smith wrote: >> >> On Mon, Mar 24, 2014 at 11:58 PM, Charles R Harris >> wrote: >> > On Mon, Mar 24, 2014 at 5:56 PM, Nathaniel Smith wrote: >> >> >> >> On Sat, Mar 22, 2014 at 6:13 PM, Nathaniel Smith wrote: >> >> > After 88 emails we don't have a conclusion in the other thread (see >> >> > [1] for background). But we have to come to some conclusion or >> >> > another >> >> > if we want @ to exist :-). So I'll summarize where the discussion >> >> > stands and let's see if we can find some way to resolve this. >> >> >> >> Response in this thread so far seems (AFAICT) to have pretty much >> >> converged on same-left. >> >> >> >> If you think that this would be terrible and there is some compelling >> >> argument against it, then please speak up! Otherwise, if no-one >> >> objects, then I'll go ahead in the next few days and put same-left >> >> into the PEP. >> > >> > >> > I think we should take a close look at broadcasting before deciding on >> > the >> > precedence. >> >> Can you elaborate? Like what, concretely, do you think we need to do now? >> > > Mostly I like to think of the '@' operators like commas in a function call > where each argument gets evaluated before the matrix multiplications take > place, so that would put it of lower precedence than '*', but still higher > than '+, -' . However, since most matrix expressions seem to be small it may > not matter much and the same result could be gotten with parenthesis. But I > do think it would make it easier to read and parse matrix expressions as the > '@' would serve as a natural divider. So 'A @ B*v' would be equivalent to 'A > @ (B*v)' and not '(A @B)*v'. > > Hmm, now that I stare at it, it may actually be easier to simply read left > to right and use parenthesis when needed. So put me down as neutral at this > point and maybe trending towards equal precedence. Okay, I'm going to let that be the last word on this (unless someone wants a laster word, but no-one seems too eager at this point). We have to pick something, we seem to have mostly converged on left-associativity, and the bulk of the opinion on left-associativity is clearly for same-left, so I've updated the PEP to include @ parsing as same-left: https://github.com/njsmith/numpy/blob/matmul-pep/doc/neps/return-of-revenge-of-matmul-pep.rst#precedence-and-associativity Thanks everyone for your feedback, both here and throughout the process in general -- I know this has been a long discussion to slog through! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From lists at onerussian.com Sun Apr 6 17:43:10 2014 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sun, 6 Apr 2014 17:43:10 -0400 Subject: [Numpy-discussion] match RNG numbers with R? Message-ID: <20140406214310.GA8748@onerussian.com> Hi NumPy gurus, We wanted to test some of our code by comparing to results of R implementation which provides bootstrapped results. R, Python std library, numpy all have Mersenne Twister RNG implementation. But all of them generate different numbers. This issue was previously discussed in https://github.com/numpy/numpy/issues/4530 : In Python, and numpy generated numbers are based on using 53 bits of two 32 bit random integers generated by the algorithm (see below). Upon my brief inspection, original 32bit numbers are nohow available for access neither in NumPy nor in Python stdlib implementation. I wonder if I have missed something and there is an easy way (i.e. without reimplementing core algorithm, or RPy'ing numbers from R) to generate random numbers in Python to match the ones in R? Excerpt from http://nbviewer.ipython.org/url/www.onerussian.com/tmp/random_randomness.ipynb # R %R RNGkind("Mersenne-Twister"); set.seed(1); sample(0:9, 10, replace=T) array([2, 3, 5, 9, 2, 8, 9, 6, 6, 0], dtype=int32) # stock Python random.seed(1); [random.randint(0, 10) for i in range(10)] [1, 9, 8, 2, 5, 4, 7, 8, 1, 0] # numpy rng = nprandom.RandomState(1); [rng.randint(0, 10) for i in range(10)] [5, 8, 9, 5, 0, 0, 1, 7, 6, 9] -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From sturla.molden at gmail.com Sun Apr 6 18:38:15 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 6 Apr 2014 22:38:15 +0000 (UTC) Subject: [Numpy-discussion] match RNG numbers with R? References: <20140406214310.GA8748@onerussian.com> Message-ID: <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> Yaroslav Halchenko wrote: > R, Python std library, numpy all have Mersenne Twister RNG implementation. But > all of them generate different numbers. This issue was previously discussed in > https://github.com/numpy/numpy/issues/4530 : In Python, and numpy generated > numbers are based on using 53 bits of two 32 bit random integers generated by > the algorithm (see below). Upon my brief inspection, original 32bit numbers > are nohow available for access neither in NumPy nor in Python stdlib > implementation. NumPy uses the Randomkit library. The source is here: https://github.com/numpy/numpy/tree/master/numpy/random/mtrand It very easy to modify to produce whatever result you want. You can build it separately from NumPy. RandomState.bytes and np.random.bytes gives you the original random bytes in little endian order. It basically calles rk_fill in Randomkit and then unpacks the 32 bit random integer into four bytes. Just view it as dtype=' <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> Message-ID: <745980293418516756.489639sturla.molden-gmail.com@news.gmane.org> > a = np.random.bytes(4*n).view(dtype='> 5).astype(np.int32) b = (np.random.bytes(4*n).view(dtype='> 6).astype(np.int32) r = (a * 67108864.0 + b) / 9007199254740992.0 Sturla From sturla.molden at gmail.com Sun Apr 6 19:07:42 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 6 Apr 2014 23:07:42 +0000 (UTC) Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe References: <53331DBF.6020504@googlemail.com> <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> Message-ID: <42123531418517869.946140sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Put another way - does anyone know what bugs in gotoBLAS2 do arise for > Windows / Intel builds? http://www.openblas.net/Changelog.txt There are some bug fixes for x86_64 here. GotoBLAS (and GotoBLAS2) were the de facto BLAS on many HPC systems, and are well proven. But there is no software without bugs. I don't think there is a reason to prefer GotoBLAS2 to OpenBLAS. But you could of course just cherry-pick all bugfixes for x86 and amd64 and leave out the rest of the changes. Differences between OpenBLAS and GotoBLAS2 for MIPS does not really matter for Windows... Sturla From sturla.molden at gmail.com Sun Apr 6 19:28:09 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 6 Apr 2014 23:28:09 +0000 (UTC) Subject: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe References: <53331DBF.6020504@googlemail.com> <1222733898418502779.416136sturla.molden-gmail.com@news.gmane.org> Message-ID: <101887869418518963.899279sturla.molden-gmail.com@news.gmane.org> Carl Kleffner wrote: > MKL BLAS LAPACK has issues as well: > href="http://software.intel.com/en-us/articles/intel-mkl-110-bug-fixes">http://software.intel.com/en-us/articles/intel-mkl-110-bug-fixes > . > In case of OpenBLAS or GOTOBLAS what precisly is the problem you identify > as showstopper? For example: https://github.com/xianyi/OpenBLAS/issues/340 However, the main problem is the quality of the projects: - GotoBLAS2 is abandonware. After Goto went to Intel, all development has ceased. Any bugs will not be fixed. - GotoBLAS2 uses OpenMP on Posix. GOMP is not fork-safe (though not an issue on Windows). - OpenBLAS looks a bit like a one-man student project. Right now Zhang Xianyi is writing his dissertation, so development has stopped. - The SIMD code is written in inline assembly instead of using compiler intrinsics for SIMD ops. This makes it hard to contribute as it is insufficient just to know C. - AT&T syntax in the inline assembly prevents us from building with MSVC. Sturla From lists at onerussian.com Sun Apr 6 21:59:42 2014 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sun, 6 Apr 2014 21:59:42 -0400 Subject: [Numpy-discussion] match RNG numbers with R? In-Reply-To: <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> Message-ID: <20140407015942.GB8748@onerussian.com> On Sun, 06 Apr 2014, Sturla Molden wrote: > > R, Python std library, numpy all have Mersenne Twister RNG implementation. But > > all of them generate different numbers. This issue was previously discussed in > > https://github.com/numpy/numpy/issues/4530 : In Python, and numpy generated > > numbers are based on using 53 bits of two 32 bit random integers generated by > > the algorithm (see below). Upon my brief inspection, original 32bit numbers > > are nohow available for access neither in NumPy nor in Python stdlib > > implementation. > NumPy uses the Randomkit library. The source is here: > https://github.com/numpy/numpy/tree/master/numpy/random/mtrand > It very easy to modify to produce whatever result you want. You can build > it separately from NumPy. > RandomState.bytes and np.random.bytes gives you the original random bytes > in little endian order. It basically calles rk_fill in Randomkit and then > unpacks the 32 bit random integer into four bytes. Just view it as > dtype=' a = np.random.bytes(4*n).view(dtype='> np.random.seed(1); [int(np.asscalar(np.fromstring(np.random.bytes(4), dtype=' References: <534131A4.7020104@gmail.com> Message-ID: <5342416A.8090104@gmail.com> =========================== Announcing Numexpr 2.4 RC2 =========================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== A new `contains()` function has been added for detecting substrings in strings. Only plain strings (bytes) are supported for now (see ticket #142). Thanks to Marcin Krol. Also, there is a new version of setup.py that allows better management of the NumPy dependency during pip installs. Thanks to Aleks Bunin. Windows related bugs have been addressed and (hopefully) squashed. Thanks to Christoph Gohlke. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! From sturla.molden at gmail.com Mon Apr 7 09:32:11 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 7 Apr 2014 13:32:11 +0000 (UTC) Subject: [Numpy-discussion] match RNG numbers with R? References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> <20140407015942.GB8748@onerussian.com> Message-ID: <1641724813418566627.752250sturla.molden-gmail.com@news.gmane.org> Yaroslav Halchenko wrote: > so I would assume that the devil is indeed in R post-processing and would look > into it (if/when get a chance). I tried to look into the R source code. It's the worst mess I have ever seen. I couldn't even find their Mersenne twister. Sturla From lists at onerussian.com Mon Apr 7 10:06:39 2014 From: lists at onerussian.com (Yaroslav Halchenko) Date: Mon, 7 Apr 2014 10:06:39 -0400 Subject: [Numpy-discussion] match RNG numbers with R? In-Reply-To: <1641724813418566627.752250sturla.molden-gmail.com@news.gmane.org> References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> <20140407015942.GB8748@onerussian.com> <1641724813418566627.752250sturla.molden-gmail.com@news.gmane.org> Message-ID: <20140407140639.GC8748@onerussian.com> On Mon, 07 Apr 2014, Sturla Molden wrote: > > so I would assume that the devil is indeed in R post-processing and would look > > into it (if/when get a chance). > I tried to look into the R source code. It's the worst mess I have ever > seen. I couldn't even find their Mersenne twister. it is in src/main/RNG.c (ack is nice ;) )... from visual inspection looks "matching" -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From davidmenhur at gmail.com Mon Apr 7 10:16:30 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Mon, 7 Apr 2014 16:16:30 +0200 Subject: [Numpy-discussion] match RNG numbers with R? In-Reply-To: <20140407015942.GB8748@onerussian.com> References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> <20140407015942.GB8748@onerussian.com> Message-ID: On Apr 7, 2014 3:59 AM, "Yaroslav Halchenko" wrote: > so I would assume that the devil is indeed in R post-processing and would look > into it (if/when get a chance). The devil here is the pigeon and the holes problem. Mersenne Twister generates random integers in a certain range. The output is guaranteed to be deterministic, uniform, and reproducible. But when you want to cast those m possible input in n possible outputs, you need to do magic (or maths) to keep the new distribution truly uniform. Simply getting random bytes and viewing them as ints will give you low quality random numbers. The algorithm that casts MT output to a random integer is probably what is different, and perhaps you could find it documented somewhere. As a side note, C++11 includes in the standard a MT RNG guaranteed to be the same across implementations, but there is no promise about the distributions -not even across versions of the same implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 7 10:26:07 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2014 15:26:07 +0100 Subject: [Numpy-discussion] match RNG numbers with R? In-Reply-To: References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> <20140407015942.GB8748@onerussian.com> Message-ID: On Mon, Apr 7, 2014 at 3:16 PM, Da?id wrote: > > On Apr 7, 2014 3:59 AM, "Yaroslav Halchenko" wrote: >> so I would assume that the devil is indeed in R post-processing and would >> look >> into it (if/when get a chance). > > The devil here is the pigeon and the holes problem. Mersenne Twister > generates random integers in a certain range. The output is guaranteed to be > deterministic, uniform, and reproducible. > > But when you want to cast those m possible input in n possible outputs, you > need to do magic (or maths) to keep the new distribution truly uniform. > Simply getting random bytes and viewing them as ints will give you low > quality random numbers. > > The algorithm that casts MT output to a random integer is probably what is > different, and perhaps you could find it documented somewhere. This is ours: https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/randomkit.c#L259 -- Robert Kern From bjodah at gmail.com Mon Apr 7 11:12:05 2014 From: bjodah at gmail.com (=?ISO-8859-1?Q?Bj=F6rn_Dahlgren?=) Date: Mon, 7 Apr 2014 17:12:05 +0200 Subject: [Numpy-discussion] Proposed new function for joining arrays: np.interleave Message-ID: Hello, Interleaving arrays is something I need to do every now and then, and by the looks of stackoverflow so do others: http://stackoverflow.com/questions/12861314/interleave-rows-of-two-numpy-arrays-in-python http://stackoverflow.com/questions/5347065/interweaving-two-numpy-arrays I think the code needed for the general n dimensional case with m number of arrays is non-trivial enough for it to be useful to provide such a function in numpy, so I took the liberty to make a Pull Request with my implementation. This would be my first contribution to NumPy so I apologize if something is not adhering to your practices. Jaime has already pointed out that I should email this list (I hope I managed to email the list in the right way - never used a mailing list before) for you to notice the pull request. He also pointed out some improvements of my proposed implementation (not forcing consistent dtype), but before I go on and make changes I might need to ask you guys if this is really something you are interested in including? Best regards, /Bj?rn Dahlgren -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Apr 7 11:46:50 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 7 Apr 2014 15:46:50 +0000 (UTC) Subject: [Numpy-discussion] match RNG numbers with R? References: <20140406214310.GA8748@onerussian.com> <1964748114418515532.436453sturla.molden-gmail.com@news.gmane.org> <20140407015942.GB8748@onerussian.com> <1641724813418566627.752250sturla.molden-gmail.com@news.gmane.org> <20140407140639.GC8748@onerussian.com> Message-ID: <1446802429418576747.248077sturla.molden-gmail.com@news.gmane.org> Yaroslav Halchenko wrote: > it is in src/main/RNG.c (ack is nice ;) )... from visual inspection looks "matching" I see... It's a rather vanilla Mersenne Twister, and it just use 32 bits of randomness. An signed int32 is multiplied by 2.3283064365386963e-10 to scale it to [0,1). Then they also have a function called "fixup" that prevents 0 or 1 from being returned. If the generated random value x <= 0.0 they return 0.5*2.328306437080797e-10, and if (1.0-x)<=0.0 they return 1.0-0.5*2.328306437080797e-10. http://svn.r-project.org/R/branches/R-3-1-branch/src/main/RNG.c Also note that the algorithm used to set the seed is important. If you set a seed equal 1 in NumPy, that might not mean the same in R. And then of course there is the algorithm used to sample integers. Sturla From robert.kern at gmail.com Mon Apr 7 15:36:17 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2014 20:36:17 +0100 Subject: [Numpy-discussion] mtrand normal sigma >= 0 too restrictive In-Reply-To: References: Message-ID: On Thu, Apr 3, 2014 at 2:35 PM, Neal Becker wrote: > Traceback (most recent call last): > File "./test_inroute_frame.py", line 1694, in > run_line (sys.argv) > File "./test_inroute_frame.py", line 1690, in run_line > return run (opt, cmdline) > File "./test_inroute_frame.py", line 1115, in run > burst.tr (xbits, freq=freqs[i]+burst.freq_offset, tau=burst.time_offset, > phase=burst.phase) > File "/home/nbecker/hn-inroute-fixed/transmitter.py", line 191, in __call__ > self.channel_out, self.complex_channel_gain = self.channel (mix_out) > File "./test_inroute_frame.py", line 105, in __call__ > ampl = 10**(0.05*self.pwr_gen()) > File "./test_inroute_frame.py", line 148, in __call__ > pwr = self.gen() > File "./test_inroute_frame.py", line 124, in __call__ > x = self.gen() > File "/home/nbecker/sigproc.ndarray/normal.py", line 11, in __call__ > return self.rs.normal (self.mean, self.std, size) > File "mtrand.pyx", line 1479, in mtrand.RandomState.normal > (numpy/random/mtrand/mtrand.c:9359) > ValueError: scale <= 0 > > I believe this restriction is too restrictive, and should be > scale < 0 > > There is nothing wrong with scale == 0 as far as I know. It's a convenient way > to turn off the noise in my simulation. That argument rings a bell, so I think I considered it at the time. I *think* that I ended up deciding that in general this approach would only work for normally-distributed noise, so one *ought* to write their code in the more general way (different `if:` branches) anyways to turn off noise. Thus, excluding scale=0 would help catch some errors at the expense of making some others do some work to do "the right thing" instead of the expedient thing. That was nearly a decade ago, and I don't hang so many consequences on those kinds of principles anymore. scale=0 is formally valid, so I think it's reasonable to allow. Considering I specifically implemented multivariate_normal() to allow non-negative semidefinite covariance matrices instead of restricting it to positive semidefinite covariance matrices, this is an odd inconsistency. -- Robert Kern From njs at pobox.com Mon Apr 7 19:23:56 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 8 Apr 2014 00:23:56 +0100 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed Message-ID: Hey all, Guido just formally accepted PEP 465: https://mail.python.org/pipermail/python-dev/2014-April/133819.html http://legacy.python.org/dev/peps/pep-0465/#implementation-details Yay. The next step is to implement it, in CPython and in numpy. I have time to advise on this, but not to do it myself, so, any volunteers? Ever wanted to hack on the interpreter itself, with BDFL guarantee your patch will be accepted (if correct)? The todo list for CPython is here: http://legacy.python.org/dev/peps/pep-0465/#implementation-details There's one open question which is where the type slots should be added. I'd just add them to PyNumberMethods and then if someone objects during patch review it can be changed. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From fperez.net at gmail.com Mon Apr 7 19:51:03 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Apr 2014 16:51:03 -0700 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: On Mon, Apr 7, 2014 at 4:23 PM, Nathaniel Smith wrote: > Hey all, > > Guido just formally accepted PEP 465: > https://mail.python.org/pipermail/python-dev/2014-April/133819.html > http://legacy.python.org/dev/peps/pep-0465/#implementation-details Congratulations!! Getting a PEP through is very hard work, and you did a remarkable job. I suspect this will make in the eyes of many of us, python 3.5 a compelling target like no other 3.x version had been before. Many thanks for all your patience steering this through, as well as to everyone who contributed. The level and quality of the discussion was pretty impressive. Cheers, f -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Apr 7 22:53:07 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Mon, 7 Apr 2014 22:53:07 -0400 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: I took liberty and reposted this as an "ENH" issue on the Python bug tracker. http://bugs.python.org/issue21176 On Mon, Apr 7, 2014 at 7:23 PM, Nathaniel Smith wrote: > Guido just formally accepted PEP 465: > https://mail.python.org/pipermail/python-dev/2014-April/133819.html > http://legacy.python.org/dev/peps/pep-0465/#implementation-details > > Yay. > > The next step is to implement it, in CPython and in numpy. I have time > to advise on this, but not to do it myself, so, any volunteers? Ever > wanted to hack on the interpreter itself, with BDFL guarantee your > patch will be accepted (if correct)? > > The todo list for CPython is here: > http://legacy.python.org/dev/peps/pep-0465/#implementation-details > There's one open question which is where the type slots should be > added. I'd just add them to PyNumberMethods and then if someone > objects during patch review it can be changed. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Apr 8 09:38:06 2014 From: travis at continuum.io (Travis Oliphant) Date: Tue, 8 Apr 2014 08:38:06 -0500 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: Congratulations! This is definitely a big step for array-computing with Python. Working with the Python devs to implement a PEP can be a tremendous opportunity to increase your programming awareness and ability --- as well as make some good friends. This is a great way to get involved with both Python and the NumPy community and have a big impact. If you are in a position to devote several hours a week to the task, then you won't find a better opportunity to contribute. Best, -Travis On Apr 7, 2014 6:24 PM, "Nathaniel Smith" wrote: > Hey all, > > Guido just formally accepted PEP 465: > https://mail.python.org/pipermail/python-dev/2014-April/133819.html > http://legacy.python.org/dev/peps/pep-0465/#implementation-details > > Yay. > > The next step is to implement it, in CPython and in numpy. I have time > to advise on this, but not to do it myself, so, any volunteers? Ever > wanted to hack on the interpreter itself, with BDFL guarantee your > patch will be accepted (if correct)? > > The todo list for CPython is here: > http://legacy.python.org/dev/peps/pep-0465/#implementation-details > There's one open question which is where the type slots should be > added. I'd just add them to PyNumberMethods and then if someone > objects during patch review it can be changed. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Apr 8 13:29:13 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Tue, 8 Apr 2014 13:29:13 -0400 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: Benjamin Peterson has posted a complete patch implementing the @ operator for Python 3.5: http://bugs.python.org/file34762/mat-mult5.patch Now we should implement matmul in numpy: https://github.com/numpy/numpy/issues/4464 -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 8 14:14:13 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 8 Apr 2014 19:14:13 +0100 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: On Tue, Apr 8, 2014 at 12:51 AM, Fernando Perez wrote: > On Mon, Apr 7, 2014 at 4:23 PM, Nathaniel Smith wrote: >> >> Hey all, >> >> Guido just formally accepted PEP 465: >> https://mail.python.org/pipermail/python-dev/2014-April/133819.html >> http://legacy.python.org/dev/peps/pep-0465/#implementation-details > > > Congratulations!! Getting a PEP through is very hard work, and you did a > remarkable job. Thank you! Though I suspect that the most important part of my contribution may have just been my high tolerance for writing emails ;-). > Many thanks for all your patience steering this through, as well as to everyone who contributed. The level and quality of the discussion was pretty impressive. Indeed -- this never would have happened without the contributions of many, many people, both during the conversations around the PEP this month, and over the previous years that set the stage for this to be possible. Thank you, everyone! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Wed Apr 9 16:46:58 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 9 Apr 2014 13:46:58 -0700 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: On Tue, Apr 8, 2014 at 11:14 AM, Nathaniel Smith wrote: > Thank you! Though I suspect that the most important part of my > contribution may have just been my high tolerance for writing emails > ;-). no -- it's your high tolerance for _reading_ emails... Far too many of us have a high tolerance for writing them! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Thu Apr 10 04:43:44 2014 From: faltet at gmail.com (Francesc Alted) Date: Thu, 10 Apr 2014 10:43:44 +0200 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: Message-ID: <534659C0.4070809@gmail.com> On 4/9/14, 10:46 PM, Chris Barker wrote: > On Tue, Apr 8, 2014 at 11:14 AM, Nathaniel Smith > wrote: > > Thank you! Though I suspect that the most important part of my > contribution may have just been my high tolerance for writing emails > ;-). > > > no -- it's your high tolerance for _reading_ emails... > > Far too many of us have a high tolerance for writing them! Ha ha, very true! -- Francesc Alted From josef.pktd at gmail.com Thu Apr 10 09:11:35 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 10 Apr 2014 09:11:35 -0400 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: <534659C0.4070809@gmail.com> References: <534659C0.4070809@gmail.com> Message-ID: On Thu, Apr 10, 2014 at 4:43 AM, Francesc Alted wrote: > On 4/9/14, 10:46 PM, Chris Barker wrote: >> On Tue, Apr 8, 2014 at 11:14 AM, Nathaniel Smith > > wrote: >> >> Thank you! Though I suspect that the most important part of my >> contribution may have just been my high tolerance for writing emails >> ;-). >> >> >> no -- it's your high tolerance for _reading_ emails... >> >> Far too many of us have a high tolerance for writing them! > > Ha ha, very true! I'm trying, but it's hard. Sometimes avoiding the `send` button helps. I still have a collection of draft replies. Josef "I'm not going to send this reply" -- oops, did it again > > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From projetmbc at gmail.com Thu Apr 10 09:27:13 2014 From: projetmbc at gmail.com (Christophe Bal) Date: Thu, 10 Apr 2014 15:27:13 +0200 Subject: [Numpy-discussion] PEP 465 has been accepted / volunteers needed In-Reply-To: References: <534659C0.4070809@gmail.com> Message-ID: Hello, I really think that add a new operator @ is not a good thing. Sorry I just want to make a joke... ;-) Christophe BAL 2014-04-10 15:11 GMT+02:00 : > On Thu, Apr 10, 2014 at 4:43 AM, Francesc Alted wrote: > > On 4/9/14, 10:46 PM, Chris Barker wrote: > >> On Tue, Apr 8, 2014 at 11:14 AM, Nathaniel Smith >> > wrote: > >> > >> Thank you! Though I suspect that the most important part of my > >> contribution may have just been my high tolerance for writing emails > >> ;-). > >> > >> > >> no -- it's your high tolerance for _reading_ emails... > >> > >> Far too many of us have a high tolerance for writing them! > > > > Ha ha, very true! > > I'm trying, but it's hard. > Sometimes avoiding the `send` button helps. I still have a collection > of draft replies. > > Josef > "I'm not going to send this reply" -- oops, did it again > > > > > -- > > Francesc Alted > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 10 15:45:49 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 10 Apr 2014 20:45:49 +0100 Subject: [Numpy-discussion] Fwd: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication Message-ID: Hey all, Given the sometimes rocky history of collaboration between numerical Python and core Python, I thought it might be helpful to flag this posting for broader distribution -- it gives one perspective on how the core devs see things. (And is certainly consistent with my experience around PEP 465.) (Nick is, among other things, core Python's packaging czar, and previously on record [1] as wishing for more feedback on how the current energy around python packaging could take our needs into account.) -n [1] https://ncoghlan_devs-python-notes.readthedocs.org/en/latest/pep_ideas/core_packaging_api.html#a-long-caveat-on-this-essay ---------- Forwarded message ---------- From: Nick Coghlan Date: Tue, Apr 8, 2014 at 1:32 PM Subject: Re: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication To: Bj?rn Lindqvist Cc: Sturla Molden , "python-dev at python.org" On 8 April 2014 21:24, Bj?rn Lindqvist wrote: > 2014-04-08 12:23 GMT+02:00 Sturla Molden : >> Bj?rn Lindqvist wrote: >> >>> import numpy as np >>> from numpy.linalg import inv, solve >>> >>> # Using dot function: >>> S = np.dot((np.dot(H, beta) - r).T, >>> np.dot(inv(np.dot(np.dot(H, V), H.T)), np.dot(H, beta) - r)) >>> >>> # Using dot method: >>> S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r) >>> >>> Don't keep your reader hanging! Tell us what the magical variables H, >>> beta, r and V are. And why import solve when you aren't using it? >>> Curious readers that aren't very good at matrix math, like me, should >>> still be able to follow your logic. Even if it is just random data, >>> it's better than nothing! >> >> Perhaps. But you don't need to know matrix multiplication to see that those >> expressions are not readable. And by extension, you can still imagine that >> bugs can easily hide in unreadable code. >> >> Matrix multiplications are used extensively in anything from engineering to >> statistics to computer graphics (2D and 3D). This operator will be a good >> thing for a lot of us. > > All I ask for is to be able to see that with my own eyes. Maybe there > is some drastic improvement I can invent to make the algorithm much > more readable? Then the PEP:s motivation falls down. I don't want to > have to believe that the code the pep author came up with is the most > optimal. I want to prove that for myself. Note that the relationship between the CPython core development team and the Numeric Python community is based heavily on trust - we don't expect them to teach us to become numeric experts, we just expect them to explain themselves well enough to persuade us that a core language or interpreter change is the only realistic way to achieve a particular goal. This does occasionally result in quirky patterns of feature adoption, as things like extended slicing, full rich comparison support, ellipsis support, rich buffer API support, and now matrix multiplication support, were added for the numeric community's benefit without necessarily offering any immediately obvious benefit for those not using the numeric Python stack - it was only later that they became pervasively adopted throughout the standard library (with matmul, for example, a follow on proposal to allow tuples, lists and arrays to handle vector dot products may make sense). This particular problem has been kicking around long enough, and is sufficiently familiar to several of us, that what's in the PEP already presents a compelling rationale for the *folks that need to be convinced* (which is primarily Guido, but if enough of the other core devs think something is a questionable idea, we can often talk him out of it - that's not the case here though). Attempting to condense that preceding 10+ years of history *into the PEP itself* wouldn't be a good return on investment - the links to the earlier PEPs are there, as are the links to these discussion threads. Cheers, Nick. P.S. We've been relatively successful in getting a similar trust based dynamic off the ground for the packaging and distribution community over the past year or so. The next big challenge in trust based delegation for the core development team will likely be around a Python 3.5+ only WSGI 2.0 (that can assume the Python 3 text model, the restoration of binary interpolation, the availability of asyncio, etc), but most of the likely principals there are still burned out from the WSGI 1.1 debate and the Python 3 transition in general :( > > > -- > mvh/best regards Bj?rn Lindqvist > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Thu Apr 10 22:44:30 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 10 Apr 2014 19:44:30 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows Message-ID: Hi, I've been working on a general wiki page on building numerical stuff on Windows: https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows I'm hoping to let Microsoft know what problems we're having, and seeing whether we numericists can share some work - us and R and Julia and Octave and so on. Feedback / edits very - very - welcome, Cheers, matthew From sturla.molden at gmail.com Thu Apr 10 23:10:56 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 03:10:56 +0000 (UTC) Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows References: Message-ID: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Hi, > > I've been working on a general wiki page on building numerical stuff on Windows: > > https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows > > I'm hoping to let Microsoft know what problems we're having, and > seeing whether we numericists can share some work - us and R and Julia > and Octave and so on. > > Feedback / edits very - very - welcome, Is seems Microsoft is working on an "accelerate" framework on their own. https://ampblas.codeplex.com https://amplapack.codeplex.com https://ampfft.codeplex.com https://amprng.codeplex.com https://ampalgorithms.codeplex.com It seems to be written in C++ and require VS2012 to build, and possibly DirectX11 to run. Apache license, from what I can see. Sturla P.S. GotoBLAS2 is spelled with capital G and a trailing 2. From matthew.brett at gmail.com Thu Apr 10 23:46:03 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 10 Apr 2014 20:46:03 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Thu, Apr 10, 2014 at 8:10 PM, Sturla Molden wrote: > Matthew Brett wrote: >> Hi, >> >> I've been working on a general wiki page on building numerical stuff on Windows: >> >> https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows >> >> I'm hoping to let Microsoft know what problems we're having, and >> seeing whether we numericists can share some work - us and R and Julia >> and Octave and so on. >> >> Feedback / edits very - very - welcome, > > > Is seems Microsoft is working on an "accelerate" framework on their own. > > https://ampblas.codeplex.com > https://amplapack.codeplex.com > > https://ampfft.codeplex.com > https://amprng.codeplex.com > https://ampalgorithms.codeplex.com > > It seems to be written in C++ and require VS2012 to build, and possibly > DirectX11 to run. For ampblas : https://ampblas.codeplex.com/SourceControl/latest#readme.txt """ This library contains an adaptation of the legacy cblas interface to BLAS for C++ AMP. At this point almost all interfaces are not implemented. One exception is the ampblas_saxpy and ampblas_daxpy which serve as a template for the implementation of other routines. """ Last commit appears to be October 2012 : https://ampblas.codeplex.com/SourceControl/list/changesets Cheers, Matthew From sandeep at techaddict.me Fri Apr 11 05:04:24 2014 From: sandeep at techaddict.me (techaddict) Date: Fri, 11 Apr 2014 02:04:24 -0700 (PDT) Subject: [Numpy-discussion] numpy.copyto alternative for previous versions than 1.7.0 ? Message-ID: <1397207064114-37282.post@n7.nabble.com> Is there a alternative way to mimic the same behaviour like numpy.copyto in previous versions of numpy ? -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/numpy-copyto-alternative-for-previous-versions-than-1-7-0-tp37282.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sandeep at techaddict.me Fri Apr 11 05:36:33 2014 From: sandeep at techaddict.me (techaddict) Date: Fri, 11 Apr 2014 02:36:33 -0700 (PDT) Subject: [Numpy-discussion] numpy.copyto alternative for previous versions than 1.7.0 ? In-Reply-To: <1397207064114-37282.post@n7.nabble.com> References: <1397207064114-37282.post@n7.nabble.com> Message-ID: <1397208993564-37283.post@n7.nabble.com> Like how do i convert these to previous versions ? copyto(ndarray(shape=[length], buffer=ba, offset=16, dtype="float64"), v) and copyto(ndarray(shape=[rows, cols], buffer=ba, offset=24, dtype="float64", order='C'), m) -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/numpy-copyto-alternative-for-previous-versions-than-1-7-0-tp37282p37283.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sebastian at sipsolutions.net Fri Apr 11 06:05:22 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Apr 2014 12:05:22 +0200 Subject: [Numpy-discussion] numpy.copyto alternative for previous versions than 1.7.0 ? In-Reply-To: <1397208993564-37283.post@n7.nabble.com> References: <1397207064114-37282.post@n7.nabble.com> <1397208993564-37283.post@n7.nabble.com> Message-ID: <1397210722.3217.3.camel@sebastian-t440> On Fr, 2014-04-11 at 02:36 -0700, techaddict wrote: > Like how do i convert these to previous versions ? > > copyto(ndarray(shape=[length], buffer=ba, offset=16, dtype="float64"), v) > and > copyto(ndarray(shape=[rows, cols], buffer=ba, offset=24, dtype="float64", > order='C'), m) > > First thing that comes to mind is using plain indexing which on versions with copyto basically ends up calling copyto anyway if I remember correctly (and v is an array, otherwise logic may be slightly more complex for indexing). So just using: arr_view = ndarray(shape=[length], buffer=ba, offset=16, dtype="float64") arr_view[...] = v should do the trick. - Sebastian > > -- > View this message in context: http://numpy-discussion.10968.n7.nabble.com/numpy-copyto-alternative-for-previous-versions-than-1-7-0-tp37282p37283.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cmkleffner at gmail.com Fri Apr 11 07:21:30 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 11 Apr 2014 13:21:30 +0200 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, a small correction: a recent octave for windows is here: http://mxeoctave.osuv.de see http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38124 ... Binary of octave 3.8.0 on windows is now prepared in voluntary contribution by Markus Bergholz. a discussion about OpenBLAS on the octave maintainer list: http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 Regards Carl 2014-04-11 5:46 GMT+02:00 Matthew Brett : > Hi, > > On Thu, Apr 10, 2014 at 8:10 PM, Sturla Molden > wrote: > > Matthew Brett wrote: > >> Hi, > >> > >> I've been working on a general wiki page on building numerical stuff on > Windows: > >> > >> https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows > >> > >> I'm hoping to let Microsoft know what problems we're having, and > >> seeing whether we numericists can share some work - us and R and Julia > >> and Octave and so on. > >> > >> Feedback / edits very - very - welcome, > > > > > > Is seems Microsoft is working on an "accelerate" framework on their own. > > > > https://ampblas.codeplex.com > > https://amplapack.codeplex.com > > > > https://ampfft.codeplex.com > > https://amprng.codeplex.com > > https://ampalgorithms.codeplex.com > > > > It seems to be written in C++ and require VS2012 to build, and possibly > > DirectX11 to run. > > For ampblas : https://ampblas.codeplex.com/SourceControl/latest#readme.txt > > """ > This library contains an adaptation of the legacy cblas interface to BLAS > for > C++ AMP. At this point almost all interfaces are not implemented. One > exception is the ampblas_saxpy and ampblas_daxpy which serve as a > template for the > implementation of other routines. > """ > > Last commit appears to be October 2012 : > https://ampblas.codeplex.com/SourceControl/list/changesets > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Apr 11 08:31:00 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 12:31:00 +0000 (UTC) Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: <1149085960418912055.947697sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > """ > This library contains an adaptation of the legacy cblas interface to BLAS for > C++ AMP. At this point almost all interfaces are not implemented. One > exception is the ampblas_saxpy and ampblas_daxpy which serve as a > template for the > implementation of other routines. > """ Right, so they gave up. By the way, it seems we have forgotten an important Fortran compiler for Windows: Portland Group. pgroup.com Sturla From njs at pobox.com Fri Apr 11 12:03:02 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Apr 2014 17:03:02 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) Message-ID: On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner wrote: > a discussion about OpenBLAS on the octave maintainer list: > http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 I'm getting the impression that OpenBLAS is being both a tantalizing opportunity and a practical thorn-in-the-side for everyone -- Python, Octave, Julia, R. How crazy would it be to get together an organized effort to fix this problem -- "OpenBLAS for everyone"? E.g., by collecting patches to fix the bits we don't like (like unhelpful build system defaults), applying more systematic QA, etc. Ideally this could be done upstream, but if upstream is MIA or disagrees about OpenBLAS's goals, then it could be maintained as a kind of "OpenBLAS++" that merges regularly from upstream (compare to [1][2][3] for successful projects handled in this way). If hardware for testing is a problem, then I suspect NumFOCUS would be overjoyed to throw a few kilodollars at buying one instance of each widely-distributed microarchitecture released in the last few years as a test farm... I think the goal is pretty clear: a modern optionally-multithreaded BLAS under a BSD-like license with a priority on correctness, out-of-the-box functionality (like runtime configurability and feature detection), speed, and portability, in that order. I unfortunately don't have the skills to actually lead such an effort (I've never written a line of asm in my life...), but surely our collective communities have people who do? -n [1] http://www.openssh.com/portable.html [2] http://www.eglibc.org/mission (a "friendly fork" of glibc holding stuff that Ulrich Drepper got cranky about, which eventually was merged back) [3] https://en.wikipedia.org/wiki/Go-oo -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From sturla.molden at gmail.com Fri Apr 11 12:53:30 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 16:53:30 +0000 (UTC) Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) References: Message-ID: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> Nathaniel Smith wrote: > I unfortunately don't have the skills to actually lead such an effort > (I've never written a line of asm in my life...), but surely our > collective communities have people who do? The assembly part in OpenBLAS/GotoBLAS is the major problem. Not just that it's AT&T syntax (i.e. it requires MinGW to build on Windows), but also that it sopports a wide range of processors. We just need a fast BLAS we can use on Windows binary wheels (and possibly Mac OS X). There is no need to support anything else than x86 and AMD64 architectures. So in theory one could throw out all assembly and rewrite the kernels with compiler intrinsics for various SIMD architectures. Or one just rely on the compiler to autovectorize. Just program the code so it is easily vectorized. If we manually unroll loops properly, and make sure the compiler is hinted about memory alignment and pointer aliasing, the compiler will know what to do. There is already a reference BLAS implementation at Netlib, which we could translate to C and optimize for SIMD. Then we need a fast threadpool. I have one I can donate, or we could use libxdispatch (a port of Apple's libdispatch, aka GCD, to Windows as Linux.) Even Intel could not make their TBB perform better than libdispatch. And that's about what we need. Or we could start with OpenBLAS and throw away everything we don't need. Making a totally new BLAS might seem like a crazy idea, but it might be the best solution in the long run. Sturla From sturla.molden at gmail.com Fri Apr 11 13:05:02 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 17:05:02 +0000 (UTC) Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> Message-ID: <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Sturla Molden wrote: > Making a totally new BLAS might seem like a crazy idea, but it might be the > best solution in the long run. To see if this can be done, I'll try to re-implement cblas_dgemm and then benchmark against MKL, Accelerate and OpenBLAS. If I can get the performance better than 75% of their speed, without any assembly or dark magic, just plain C99 compiled with Intel icc, that would be sufficient for binary wheels on Windows I think. Sturla From matthew.brett at gmail.com Fri Apr 11 13:39:46 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 10:39:46 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: <1149085960418912055.947697sturla.molden-gmail.com@news.gmane.org> References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> <1149085960418912055.947697sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Apr 11, 2014 at 5:31 AM, Sturla Molden wrote: > Matthew Brett wrote: > >> """ >> This library contains an adaptation of the legacy cblas interface to BLAS for >> C++ AMP. At this point almost all interfaces are not implemented. One >> exception is the ampblas_saxpy and ampblas_daxpy which serve as a >> template for the >> implementation of other routines. >> """ > > Right, so they gave up. > > By the way, it seems we have forgotten an important Fortran compiler for > Windows: Portland Group. > > pgroup.com Thanks for reminding me, I've put that in. Man, they have an awful license, making it quite useless for open-source: http://www.pgroup.com/doc/LICENSE.txt Cheers, Matthew From sturla.molden at gmail.com Fri Apr 11 13:49:34 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 17:49:34 +0000 (UTC) Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> <1149085960418912055.947697sturla.molden-gmail.com@news.gmane.org> Message-ID: <179780396418930938.883111sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Man, they have an awful license, making it quite useless for > open-source: http://www.pgroup.com/doc/LICENSE.txt Awful, and insanely expensive. :-( And if you look at ACML, you will find that the MSVC compatible version is built with the PG compiler. (There is an Intel ifort version too, but the PG version is the only one that actually works.) So if you want ACML, beware that it is tainted with a PG license on the runtime libraries. Sturla From matthew.brett at gmail.com Fri Apr 11 14:10:21 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 11:10:21 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Apr 11, 2014 at 4:21 AM, Carl Kleffner wrote: > Hi, > > a small correction: a recent octave for windows is here: > http://mxeoctave.osuv.de > > see http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38124 ... > Binary of octave 3.8.0 on windows is now prepared in voluntary contribution > by Markus Bergholz. > > a discussion about OpenBLAS on the octave maintainer list: > http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 Thanks - I've put those corrections in as well. Can you edit the page by the way? I'm assuming y'all can... Matthew From aron at ahmadia.net Fri Apr 11 14:26:43 2014 From: aron at ahmadia.net (Aron Ahmadia) Date: Fri, 11 Apr 2014 14:26:43 -0400 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: Thanks Matthew for putting this page together. The OpenBLAS guys have been accepting/merging pull requests (their GitHub tree shows 26 contributors and no open pull requests), and I know that several people from the Python and Julia community have gotten pull requests merged. I modified your comments in the Wiki slightly, feel free to revert if inappopriate. A On Fri, Apr 11, 2014 at 2:10 PM, Matthew Brett wrote: > Hi, > > On Fri, Apr 11, 2014 at 4:21 AM, Carl Kleffner > wrote: > > Hi, > > > > a small correction: a recent octave for windows is here: > > http://mxeoctave.osuv.de > > > > see http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38124 ... > > Binary of octave 3.8.0 on windows is now prepared in voluntary > contribution > > by Markus Bergholz. > > > > a discussion about OpenBLAS on the octave maintainer list: > > http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 > > Thanks - I've put those corrections in as well. > > Can you edit the page by the way? I'm assuming y'all can... > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Apr 11 14:29:46 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 11 Apr 2014 20:29:46 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: Message-ID: <5348349A.5010800@googlemail.com> On 11.04.2014 18:03, Nathaniel Smith wrote: > On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner wrote: >> a discussion about OpenBLAS on the octave maintainer list: >> http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 > > I'm getting the impression that OpenBLAS is being both a tantalizing > opportunity and a practical thorn-in-the-side for everyone -- Python, > Octave, Julia, R. > > How crazy would it be to get together an organized effort to fix this > problem -- "OpenBLAS for everyone"? E.g., by collecting patches to fix > the bits we don't like (like unhelpful build system defaults), > applying more systematic QA, etc. Ideally this could be done upstream, > but if upstream is MIA or disagrees about OpenBLAS's goals, then it > could be maintained as a kind of "OpenBLAS++" that merges regularly > from upstream (compare to [1][2][3] for successful projects handled in > this way). If hardware for testing is a problem, then I suspect > NumFOCUS would be overjoyed to throw a few kilodollars at buying one > instance of each widely-distributed microarchitecture released in the > last few years as a test farm... > x86 cpus are backward compatible with almost all instructions they ever introduced, so one machine with the latest instruction set supported is sufficient to test almost everything. For that the runtime kernel selection must be tuneable via the environment so you can use kernels intended for older cpus. The larger issue is finding a good and thorough testsuite that wasn't written 30 years ago and thus does covers problem sizes larger than a few megabytes. These are the problem sizes are that often crashed openblas in the past. Isn't there a kind of comprehensive BLAS verification testsuite which all BLAS implementations should test against and contribute to available somewhere? E.g. like the POSIX compliance testsuite. From matthew.brett at gmail.com Fri Apr 11 14:32:03 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 11:32:03 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: Message-ID: Hi, On Fri, Apr 11, 2014 at 9:03 AM, Nathaniel Smith wrote: > On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner wrote: >> a discussion about OpenBLAS on the octave maintainer list: >> http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 > > I'm getting the impression that OpenBLAS is being both a tantalizing > opportunity and a practical thorn-in-the-side for everyone -- Python, > Octave, Julia, R. > > How crazy would it be to get together an organized effort to fix this > problem -- "OpenBLAS for everyone"? E.g., by collecting patches to fix > the bits we don't like (like unhelpful build system defaults), > applying more systematic QA, etc. Ideally this could be done upstream, > but if upstream is MIA or disagrees about OpenBLAS's goals, then it > could be maintained as a kind of "OpenBLAS++" that merges regularly > from upstream (compare to [1][2][3] for successful projects handled in > this way). If hardware for testing is a problem, then I suspect > NumFOCUS would be overjoyed to throw a few kilodollars at buying one > instance of each widely-distributed microarchitecture released in the > last few years as a test farm... > > I think the goal is pretty clear: a modern optionally-multithreaded > BLAS under a BSD-like license with a priority on correctness, > out-of-the-box functionality (like runtime configurability and feature > detection), speed, and portability, in that order. It sounds like a joint conversation with R, Julia, Octave team at least would be useful here, Anyone volunteer for starting that conversation? Cheers, Matthew From jtaylor.debian at googlemail.com Fri Apr 11 14:38:12 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 11 Apr 2014 20:38:12 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: <53483694.2090207@googlemail.com> On 11.04.2014 19:05, Sturla Molden wrote: > Sturla Molden wrote: > >> Making a totally new BLAS might seem like a crazy idea, but it might be the >> best solution in the long run. > > To see if this can be done, I'll try to re-implement cblas_dgemm and then > benchmark against MKL, Accelerate and OpenBLAS. If I can get the > performance better than 75% of their speed, without any assembly or dark > magic, just plain C99 compiled with Intel icc, that would be sufficient for > binary wheels on Windows I think. > hi, if you can, also give gcc with graphite a try. Its loop transformations should give you similar results as manual blocking if the compiler is able to understand the loop, see http://gcc.gnu.org/gcc-4.4/changes.html -floop-strip-mine -floop-block -floop-interchange + a couple options to tune the parameters you may need gcc-4.8 for it to work properly on not compile time fixed loop iteration counts. So far i know clang/llvm also has graphite integration. Cheers, Julian From matthew.brett at gmail.com Fri Apr 11 14:34:31 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 11:34:31 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Apr 11, 2014 at 11:26 AM, Aron Ahmadia wrote: > Thanks Matthew for putting this page together. > > The OpenBLAS guys have been accepting/merging pull requests (their GitHub > tree shows 26 contributors and no open pull requests), and I know that > several people from the Python and Julia community have gotten pull requests > merged. I modified your comments in the Wiki slightly, feel free to revert > if inappopriate. Excellent - thanks for that, Matthew From njs at pobox.com Fri Apr 11 14:47:17 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Apr 2014 19:47:17 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Apr 11, 2014 at 6:05 PM, Sturla Molden wrote: > Sturla Molden wrote: > >> Making a totally new BLAS might seem like a crazy idea, but it might be the >> best solution in the long run. > > To see if this can be done, I'll try to re-implement cblas_dgemm and then > benchmark against MKL, Accelerate and OpenBLAS. If I can get the > performance better than 75% of their speed, without any assembly or dark > magic, just plain C99 compiled with Intel icc, that would be sufficient for > binary wheels on Windows I think. Sounds like a worthwhile experiment! My suspicion is that it we'll be better off starting with something that is almost good enough (OpenBLAS) and then incrementally improving it to meet our needs, rather than starting from scratch -- there's a *long* way to get from dgemm to a fully supported BLAS project -- but no matter what it'll generate useful data, and possibly some useful code that could either be the basis of something new or integrated into whatever we do end up doing. Also, while Windows is maybe in the worst shape, all platforms would seriously benefit from the existence of a reliable speed-competitive binary-distribution-compatible BLAS that doesn't break fork(). -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From jtaylor.debian at googlemail.com Fri Apr 11 14:53:53 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 11 Apr 2014 20:53:53 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: Message-ID: <53483A41.3050201@googlemail.com> On 11.04.2014 18:03, Nathaniel Smith wrote: > On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner wrote: >> a discussion about OpenBLAS on the octave maintainer list: >> http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 > > I'm getting the impression that OpenBLAS is being both a tantalizing > opportunity and a practical thorn-in-the-side for everyone -- Python, > Octave, Julia, R. > does anyone have experience with BLIS? https://code.google.com/p/blis/ https://github.com/flame/blis from the description it looks interesting and its BSD licensed. though windows support is experimental according to the FAQ. From matthew.brett at gmail.com Fri Apr 11 15:18:52 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 12:18:52 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: <179780396418930938.883111sturla.molden-gmail.com@news.gmane.org> References: <1799056478418878273.401911sturla.molden-gmail.com@news.gmane.org> <1149085960418912055.947697sturla.molden-gmail.com@news.gmane.org> <179780396418930938.883111sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Apr 11, 2014 at 10:49 AM, Sturla Molden wrote: > Matthew Brett wrote: > >> Man, they have an awful license, making it quite useless for >> open-source: http://www.pgroup.com/doc/LICENSE.txt > > Awful, and insanely expensive. :-( > > And if you look at ACML, you will find that the MSVC compatible version is > built with the PG compiler. (There is an Intel ifort version too, but the > PG version is the only one that actually works.) So if you want ACML, > beware that it is tainted with a PG license on the runtime libraries. The ACML license does not include those terms, unless I missed them: http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/ACML_June_24_2010_v2.pdf I assume that AMD negotiated to release themselves from full terms of the PG license, but if anyone knows differently, please say... Cheers, Matthew From njs at pobox.com Fri Apr 11 15:34:18 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Apr 2014 20:34:18 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <53483A41.3050201@googlemail.com> References: <53483A41.3050201@googlemail.com> Message-ID: On Fri, Apr 11, 2014 at 7:53 PM, Julian Taylor wrote: > On 11.04.2014 18:03, Nathaniel Smith wrote: >> On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner wrote: >>> a discussion about OpenBLAS on the octave maintainer list: >>> http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 >> >> I'm getting the impression that OpenBLAS is being both a tantalizing >> opportunity and a practical thorn-in-the-side for everyone -- Python, >> Octave, Julia, R. >> > > does anyone have experience with BLIS? > https://code.google.com/p/blis/ > https://github.com/flame/blis Also: Does BLIS automatically detect my hardware? Not yet. For now, BLIS requires the user/developer to manually specify an existing configuration that corresponds to the hardware for which to build a BLIS library. So for now, BLIS is mostly a developer's tool? Yes. In order to achieve high performance, BLIS requires that hand-coded kernels and micro-kernels be written and referenced in a valid BLIS configuration. These components are usually written by developers and then included within BLIS for use by others. If high performance is not important, then you can always build the reference implementation on any hardware platform. The reference implementation does not contain any machine-specific code and thus should be very portable. Does BLIS support multithreading? BLIS does not yet implement multithreaded versions of its operations. However, BLIS can very easily be made thread-safe so that you can call BLIS from threads[...] Can I build BLIS as a shared library? The BLIS build system is not yet capable of outputting a shared library. [...] https://code.google.com/p/blis/wiki/FAQ -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Fri Apr 11 17:11:27 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 14:11:27 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Apr 11, 2014 at 10:05 AM, Sturla Molden wrote: > Sturla Molden wrote: > >> Making a totally new BLAS might seem like a crazy idea, but it might be the >> best solution in the long run. > > To see if this can be done, I'll try to re-implement cblas_dgemm and then > benchmark against MKL, Accelerate and OpenBLAS. If I can get the > performance better than 75% of their speed, without any assembly or dark > magic, just plain C99 compiled with Intel icc, that would be sufficient for > binary wheels on Windows I think. Did you check out the Intel license though? http://software.intel.com/sites/default/files/managed/95/23/Intel_SW_Dev_Products__EULA.pdf D. DISTRIBUTION: Distribution of the Redistributables is also subject to the following limitations: You (i) shall be solely responsible to your customers for any update or support obligation or other liability which may arise from the distribution, (ii) shall not make any statement that your product is "certified", or that its performance is guaranteed, by Intel, (iii) shall not use Intel's name or trademarks to market your product without written permission, (iv) shall use a license agreement that prohibits disassembly and reverse engineering of the Redistributables, (v) shall indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from your distribution of any product. Are you sure that you can redistribute object code statically linked against icc runtimes? Cheers, Matthew From sturla.molden at gmail.com Fri Apr 11 17:58:39 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 11 Apr 2014 23:58:39 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 11/04/14 23:11, Matthew Brett wrote: > Are you sure that you can redistribute object code statically linked > against icc runtimes? I am not a lawyer... From matthew.brett at gmail.com Fri Apr 11 18:01:37 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 11 Apr 2014 15:01:37 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Apr 11, 2014 at 2:58 PM, Sturla Molden wrote: > On 11/04/14 23:11, Matthew Brett wrote: > >> Are you sure that you can redistribute object code statically linked >> against icc runtimes? > > I am not a lawyer... No - sure - but it would be frustrating if you found yourself optimizing with a compiler that is useless for subsequent open-source builds. Best, Matthew From sturla.molden at gmail.com Fri Apr 11 18:10:30 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 00:10:30 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 12/04/14 00:01, Matthew Brett wrote: > No - sure - but it would be frustrating if you found yourself > optimizing with a compiler that is useless for subsequent open-source > builds. No, I think MSVC or gcc 4.8/4.9 will work too. It's just that I happen to have icc and clang on this computer :) Sturla From sturla.molden at gmail.com Fri Apr 11 18:15:56 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 00:15:56 +0200 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: Message-ID: On 11/04/14 04:44, Matthew Brett wrote: > I've been working on a general wiki page on building numerical stuff on Windows: > > https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows I am worried that the conclusion will be that there is no viable BLAS alternative on Windows... Sturla From smudkavi at uwaterloo.ca Fri Apr 11 18:25:46 2014 From: smudkavi at uwaterloo.ca (Sankarshan Mudkavi) Date: Fri, 11 Apr 2014 18:25:46 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> Message-ID: <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> So is the consensus that we don't accept any tags at all (not even temporarily)? Would that break too much existing code? Cheers, Sankarshan On Apr 1, 2014, at 2:50 PM, Alexander Belopolsky wrote: > > On Tue, Apr 1, 2014 at 1:12 PM, Nathaniel Smith wrote: > In [6]: a[0] = "garbage" > ValueError: could not convert string to float: garbage > > (Cf, "Errors should never pass silently".) Any reason why datetime64 > should be different? > > datetime64 is different because it has NaT support from the start. NaN support for floats seems to be an afterthought if not an accident of implementation. > > And it looks like some errors do pass silently: > > >>> a[0] = "1" > # not a TypeError > > But I withdraw my suggestion. The closer datetime64 behavior is to numeric types the better. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From sturla.molden at gmail.com Fri Apr 11 18:26:35 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 00:26:35 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 11/04/14 20:47, Nathaniel Smith wrote: > Also, while Windows is maybe in the worst shape, all platforms would > seriously benefit from the existence of a reliable speed-competitive > binary-distribution-compatible BLAS that doesn't break fork(). Windows is worst off, yes. I don't think fork breakage by Accelerate is a big problem on Mac OS X. Apple has made clear that only POSIX APIs are fork safe. And actually this is now recognized as an error in multiprocessing and fixed in Python 3.4: multiprocessing.set_start_method('spawn') On Linux the distributions will usually ship with prebuilt ATLAS. Sturla From njs at pobox.com Fri Apr 11 18:28:59 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Apr 2014 23:28:59 +0100 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Fri, Apr 11, 2014 at 11:25 PM, Sankarshan Mudkavi wrote: > So is the consensus that we don't accept any tags at all (not even > temporarily)? Would that break too much existing code? Well, we don't know. If anyone has any ideas on how to figure it out then they should speak up :-). Barring any brilliant suggestions though, I suggest we just go ahead with disallowing all timezone tags for now. We can always change our mind as we get closer to the release and people start experimenting with the new code. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Fri Apr 11 18:39:39 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Apr 2014 23:39:39 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Apr 11, 2014 at 11:26 PM, Sturla Molden wrote: > On 11/04/14 20:47, Nathaniel Smith wrote: > >> Also, while Windows is maybe in the worst shape, all platforms would >> seriously benefit from the existence of a reliable speed-competitive >> binary-distribution-compatible BLAS that doesn't break fork(). > > Windows is worst off, yes. > > I don't think fork breakage by Accelerate is a big problem on Mac OS X. > Apple has made clear that only POSIX APIs are fork safe. And actually > this is now recognized as an error in multiprocessing and fixed in > Python 3.4: > > multiprocessing.set_start_method('spawn') I don't really care whether it's *documented* that BLAS and fork are incompatible. I care whether it *works*, because it is useful functionality :-). The spawn mode is fine and all, but (a) the presence of something in 3.4 helps only a minority of users, (b) "spawn" is not a full replacement for fork; with large read-mostly data sets it can be a *huge* win to load them into the parent process and then let them be COW-inherited by forked children. ATM the only other way to work with a data set that's larger than memory-divided-by-numcpus is to explicitly set up shared memory, and this is *really* hard for anything more complicated than a single flat array. > On Linux the distributions will usually ship with prebuilt ATLAS. And it's generally recommended that everyone rebuild their own ATLAS anyway. I can do it, but I'd much rather be able to install a BLAS library that just worked. (Presumably this is a large part of why scipy-stack distributors prefer MKL over ATLAS.) If it comes down to it then of course I'd rather have a Windows-only BLAS than no BLAS at all. I just don't think we should be setting our sights so low at this point. The marginal cost of portability doesn't seem high. Besides, even Windows users will benefit more from having a standard cross-platform BLAS that everyone uses -- it would mean lots more people familiar with the library's quirks, better testing, etc. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From charlesr.harris at gmail.com Fri Apr 11 18:56:05 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Apr 2014 16:56:05 -0600 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Fri, Apr 11, 2014 at 4:25 PM, Sankarshan Mudkavi wrote: > So is the consensus that we don't accept any tags at all (not even > temporarily)? Would that break too much existing code? > > Cheers, > Sankarshan > > On Apr 1, 2014, at 2:50 PM, Alexander Belopolsky wrote: > > > On Tue, Apr 1, 2014 at 1:12 PM, Nathaniel Smith wrote: > >> In [6]: a[0] = "garbage" >> ValueError: could not convert string to float: garbage >> >> (Cf, "Errors should never pass silently".) Any reason why datetime64 >> should be different? >> > > datetime64 is different because it has NaT support from the start. NaN > support for floats seems to be an afterthought if not an accident of > implementation. > > And it looks like some errors do pass silently: > > >>> a[0] = "1" > # not a TypeError > > But I withdraw my suggestion. The closer datetime64 behavior is to > numeric types the better. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > Are we in a position to start looking at implementation? If so, it would be useful to have a collection of test cases, i.e., typical uses with specified results. That should also cover conversion from/(to?) datetime.datetime. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Apr 11 19:07:02 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 01:07:02 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 12/04/14 00:39, Nathaniel Smith wrote: > The spawn mode is fine and all, but (a) the presence of something in > 3.4 helps only a minority of users, (b) "spawn" is not a full > replacement for fork; It basically does the same as on Windows. If you want portability to Windows, you must abide by these restrictions anyway. > with large read-mostly data sets it can be a > *huge* win to load them into the parent process and then let them be > COW-inherited by forked children. The thing is that Python reference counts breaks COW fork. This has been discussed several times on the Python-dev list. What happens is that as soon as the child process updates a refcount, the OS copies the page. And because of how Python behaves, this copying of COW-marked pages quickly gets excessive. Effectively the performance of os.fork in Python will close to a non-COW fork. A suggested solution is to move the refcount out of the PyObject struct, and perhaps keep them in a dedicated heap. But doing so will be unfriendly to cache. > ATM the only other way to work with > a data set that's larger than memory-divided-by-numcpus is to > explicitly set up shared memory, and this is *really* hard for > anything more complicated than a single flat array. Not difficult. You just go to my GitHub site and grab the code ;) (I have some problems running it on my MBP though, not sure why, but it used to work on Linux and Windows, and possibly still does.) https://github.com/sturlamolden/sharedmem-numpy Sturla From njs at pobox.com Fri Apr 11 19:16:04 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 12 Apr 2014 00:16:04 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Sat, Apr 12, 2014 at 12:07 AM, Sturla Molden wrote: > On 12/04/14 00:39, Nathaniel Smith wrote: > >> The spawn mode is fine and all, but (a) the presence of something in >> 3.4 helps only a minority of users, (b) "spawn" is not a full >> replacement for fork; > > It basically does the same as on Windows. If you want portability to > Windows, you must abide by these restrictions anyway. Yes, but "sorry Unix guys, we've decided to take away this nice feature from you because it doesn't work on Windows" is a really terrible argument. If it can't be made to work, then fine, but fork safety is just not *that* much to ask. >> with large read-mostly data sets it can be a >> *huge* win to load them into the parent process and then let them be >> COW-inherited by forked children. > > The thing is that Python reference counts breaks COW fork. This has been > discussed several times on the Python-dev list. What happens is that as > soon as the child process updates a refcount, the OS copies the page. > And because of how Python behaves, this copying of COW-marked pages > quickly gets excessive. Effectively the performance of os.fork in Python > will close to a non-COW fork. A suggested solution is to move the > refcount out of the PyObject struct, and perhaps keep them in a > dedicated heap. But doing so will be unfriendly to cache. Yes, it's limited, but again this is not a reason to break it in the cases where it *does* work. The case where I ran into this was loading a big language model using SRILM: http://www.speech.sri.com/projects/srilm/ https://github.com/njsmith/pysrilm This produces a single Python object that references an opaque, tens-of-gigabytes mess of C++ objects. For this case explicit shared mem is useless, but fork worked brilliantly. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Fri Apr 11 19:19:26 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 12 Apr 2014 00:19:26 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <5348349A.5010800@googlemail.com> References: <5348349A.5010800@googlemail.com> Message-ID: On Fri, Apr 11, 2014 at 7:29 PM, Julian Taylor wrote: > x86 cpus are backward compatible with almost all instructions they ever > introduced, so one machine with the latest instruction set supported is > sufficient to test almost everything. > For that the runtime kernel selection must be tuneable via the > environment so you can use kernels intended for older cpus. Overriding runtime kernel selection sounds like a good bite-sized feature that could be added to OpenBLAS... > The larger issue is finding a good and thorough testsuite that wasn't > written 30 years ago and thus does covers problem sizes larger than a > few megabytes. These are the problem sizes are that often crashed > openblas in the past. > Isn't there a kind of comprehensive BLAS verification testsuite which > all BLAS implementations should test against and contribute to available > somewhere? > E.g. like the POSIX compliance testsuite. I doubt it! Someone could make a good start on one in an afternoon though. (Only a start, but half a test suite is heck of a lot better than nothing.) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From sturla.molden at gmail.com Fri Apr 11 19:32:24 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 01:32:24 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 12/04/14 01:07, Sturla Molden wrote: >> ATM the only other way to work with >> a data set that's larger than memory-divided-by-numcpus is to >> explicitly set up shared memory, and this is *really* hard for >> anything more complicated than a single flat array. > > > Not difficult. You just go to my GitHub site and grab the code ;) > > (I have some problems running it on my MBP though, not sure why, but it > used to work on Linux and Windows, and possibly still does.) > > https://github.com/sturlamolden/sharedmem-numpy Hmm, today it works fine on my MBP too... Good. :) import multiprocessing as mp import numpy as np import sharedmem as shm def proc(qin, qout): print("grabbing array from queue") a = qin.get() print(a) print("putting array in queue") b = shm.zeros(10) print(b) qout.put(b) print("waiting for array to be updated by another process") a = qin.get() print(b) if __name__ == "__main__": qin = mp.Queue() qout = mp.Queue() p = mp.Process(target=proc, args=(qin,qout)) p.start() a = shm.zeros(4) qin.put(a) b = qout.get() b[:] = range(10) qin.put(None) p.join() sturla$ python example.py grabbing array from queue [ 0. 0. 0. 0.] putting array in queue [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] waiting for array to be updated by another process [ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] Sturla From njs at pobox.com Fri Apr 11 19:42:03 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 12 Apr 2014 00:42:03 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <5348349A.5010800@googlemail.com> Message-ID: Okay, I started taking notes here: https://github.com/numpy/numpy/wiki/BLAS-desiderata Please add as appropriate... -n On Sat, Apr 12, 2014 at 12:19 AM, Nathaniel Smith wrote: > On Fri, Apr 11, 2014 at 7:29 PM, Julian Taylor > wrote: >> x86 cpus are backward compatible with almost all instructions they ever >> introduced, so one machine with the latest instruction set supported is >> sufficient to test almost everything. >> For that the runtime kernel selection must be tuneable via the >> environment so you can use kernels intended for older cpus. > > Overriding runtime kernel selection sounds like a good bite-sized > feature that could be added to OpenBLAS... > >> The larger issue is finding a good and thorough testsuite that wasn't >> written 30 years ago and thus does covers problem sizes larger than a >> few megabytes. These are the problem sizes are that often crashed >> openblas in the past. >> Isn't there a kind of comprehensive BLAS verification testsuite which >> all BLAS implementations should test against and contribute to available >> somewhere? >> E.g. like the POSIX compliance testsuite. > > I doubt it! Someone could make a good start on one in an afternoon > though. (Only a start, but half a test suite is heck of a lot better > than nothing.) > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From shoyer at gmail.com Fri Apr 11 19:58:22 2014 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 11 Apr 2014 16:58:22 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris wrote: > Are we in a position to start looking at implementation? If so, it would > be useful to have a collection of test cases, i.e., typical uses with > specified results. That should also cover conversion from/(to?) > datetime.datetime. > Indeed, my personal wish-list for np.datetime64 is centered much more on robust conversion to/from native date objects, including comparison. Here are some of my particular points of frustration (apologies for the thread jacking!): - NaT should have similar behavior to NaN when used for comparisons (i.e., comparisons should always be False). - You can't compare a datetime object to a datetime64 object. - datetime64 objects with high precision (e.g., ns) can't compare to datetime objects. Pandas has a very nice wrapper around datetime64 arrays that solves most of these issues, but it would be nice to get much of that functionality in core numpy, since I don't always want to store my values in a 1-dimensional array + hash-table (the pandas Index): http://pandas.pydata.org/pandas-docs/stable/timeseries.html Here's code which reproduces all of the above: from numpy import datetime64 from datetime import datetime print np.datetime64('NaT') < np.datetime64('2011-01-01') # this should not to true print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises exception print np.datetime64('2011-01-01T00:00', 'ns') > datetime(2010, 1, 1) # another exception print np.datetime64('2011-01-01T00:00') > datetime(2010, 1, 1) # finally something works! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Fri Apr 11 20:24:09 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Fri, 11 Apr 2014 20:24:09 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Fri, Apr 11, 2014 at 7:58 PM, Stephan Hoyer wrote: > print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises exception This is somewhat consistent with >>> from datetime import * >>> datetime(2010, 1, 1) < date(2010, 1, 1) Traceback (most recent call last): File "", line 1, in TypeError: can't compare datetime.datetime to datetime.date but I would expect date(2010, 1, 1) < np.datetime64('2011-01-01') to return False. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Sat Apr 12 03:24:04 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sat, 12 Apr 2014 09:24:04 +0200 Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows In-Reply-To: References: Message-ID: I wonder: how hard would it be to create a more 21th-century oriented BLAS, relying more on code generation tools, and perhaps LLVM/JITting? Wouldn't we get ten times the portability with one-tenth the lines of code? Or is there too much dark magic going on in BLAS for such an approach to come close enough to hand-tuned performance? On Sat, Apr 12, 2014 at 12:15 AM, Sturla Molden wrote: > On 11/04/14 04:44, Matthew Brett wrote: > > > I've been working on a general wiki page on building numerical stuff on > Windows: > > > > https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows > > I am worried that the conclusion will be that there is no viable BLAS > alternative on Windows... > > > Sturla > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sat Apr 12 07:12:32 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 12 Apr 2014 11:12:32 +0000 (UTC) Subject: [Numpy-discussion] Wiki page for building numerical stuff on Windows References: Message-ID: <1252932894418993249.227226sturla.molden-gmail.com@news.gmane.org> Eelco Hoogendoorn wrote: > I wonder: how hard would it be to create a more 21th-century oriented BLAS, > relying more on code generation tools, and perhaps LLVM/JITting? > > Wouldn't we get ten times the portability with one-tenth the lines of code? > Or is there too much dark magic going on in BLAS for such an approach to > come close enough to hand-tuned performance? The "dark magic" in OpenBLAS is mostly to place prefetch instructions strategically, to make sure hierarchical memory is used optimally. This is very hard for the compiler to get correctly, because it doesn't know matrix algebra like we do. The reason prefetching is needed, is because when two matrices are multiplied, one of them will have strided memory access. On the other hand, putting in other SIMD instructions than _mm_prefetch is something a compiler might be able to vectorize without a lot of help today. Sturla From alan.isaac at gmail.com Sat Apr 12 10:02:05 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 12 Apr 2014 10:02:05 -0400 Subject: [Numpy-discussion] boolean operations on boolean arrays Message-ID: <5349475D.2050604@gmail.com> This is a very basic question. Suppose `a` and `b` are boolean arrays with the same shape. Are there any considerations besides convenience in choosing between: a&b a*b logical_and(a,b) a|b a+b logical_or(a,b) ~a True-a logical_not(a) I somewhat expect the last column to be slowest as well as least convenient, since it is built to first convert non-booleans to booleans. Are there other differences? Also, is this made clear anywhere in the docs? Thanks, Alan Isaac From charlesr.harris at gmail.com Sat Apr 12 11:52:55 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 12 Apr 2014 09:52:55 -0600 Subject: [Numpy-discussion] boolean operations on boolean arrays In-Reply-To: <5349475D.2050604@gmail.com> References: <5349475D.2050604@gmail.com> Message-ID: On Sat, Apr 12, 2014 at 8:02 AM, Alan G Isaac wrote: > This is a very basic question. > Suppose `a` and `b` are boolean arrays with the same shape. > Are there any considerations besides convenience in choosing > between: > > a&b a*b logical_and(a,b) > a|b a+b logical_or(a,b) > ~a True-a logical_not(a) > > I somewhat expect the last column to be slowest > as well as least convenient, since it is built to > first convert non-booleans to booleans. > Are there other differences? > Also, is this made clear anywhere in the docs? > > The Python operators for ndarrays can be dynamically set, but they are initialized by the ufunc module (don't ask) as follows. #define BOOL_invert BOOL_logical_not #define BOOL_negative BOOL_logical_not #define BOOL_add BOOL_logical_or #define BOOL_bitwise_and BOOL_logical_and #define BOOL_bitwise_or BOOL_logical_or #define BOOL_bitwise_xor BOOL_logical_xor #define BOOL_multiply BOOL_logical_and #define BOOL_subtract BOOL_logical_xor So they are all equivalent to logical_* functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dineshbvadhia at hotmail.com Sat Apr 12 12:15:12 2014 From: dineshbvadhia at hotmail.com (Dinesh Vadhia) Date: Sat, 12 Apr 2014 09:15:12 -0700 Subject: [Numpy-discussion] The BLAS problem In-Reply-To: References: <5348349A.5010800@googlemail.com> Message-ID: Agree that OpenBLAS is the most favorable route instead of starting from scratch. Btw, why is sparse BLAS not included as I've always been under the impression that scipy sparse supports BLAS - no? From ndarray at mac.com Sat Apr 12 12:58:34 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sat, 12 Apr 2014 12:58:34 -0400 Subject: [Numpy-discussion] boolean operations on boolean arrays In-Reply-To: <5349475D.2050604@gmail.com> References: <5349475D.2050604@gmail.com> Message-ID: On Sat, Apr 12, 2014 at 10:02 AM, Alan G Isaac wrote: > Are there any considerations besides convenience in choosing > between: > > a&b a*b logical_and(a,b) > a|b a+b logical_or(a,b) > ~a True-a logical_not(a) > Boolean "-" is being deprecated: https://github.com/numpy/numpy/pull/4105 The choice between &| and *+ is best dictated by what is more natural in your problem domain and how your functions should treat non-boolean types. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Sat Apr 12 14:34:30 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sat, 12 Apr 2014 20:34:30 +0200 Subject: [Numpy-discussion] Wiki page for building numerical stuff onWindows Message-ID: <53498762.84d50e0a.638a.ffffec65@mx.google.com> BLIS seems like a nice project as well. I like the arbitrary striding; BLAS lacking this has always annoyed me. -----Original Message----- From: "Sturla Molden" Sent: ?12-?4-?2014 13:12 To: "numpy-discussion at scipy.org" Subject: Re: [Numpy-discussion] Wiki page for building numerical stuff onWindows Eelco Hoogendoorn wrote: > I wonder: how hard would it be to create a more 21th-century oriented BLAS, > relying more on code generation tools, and perhaps LLVM/JITting? > > Wouldn't we get ten times the portability with one-tenth the lines of code? > Or is there too much dark magic going on in BLAS for such an approach to > come close enough to hand-tuned performance? The "dark magic" in OpenBLAS is mostly to place prefetch instructions strategically, to make sure hierarchical memory is used optimally. This is very hard for the compiler to get correctly, because it doesn't know matrix algebra like we do. The reason prefetching is needed, is because when two matrices are multiplied, one of them will have strided memory access. On the other hand, putting in other SIMD instructions than _mm_prefetch is something a compiler might be able to vectorize without a lot of help today. Sturla _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Sat Apr 12 16:47:19 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 12 Apr 2014 16:47:19 -0400 Subject: [Numpy-discussion] index partition Message-ID: <5349A657.7080801@gmail.com> From a 1d array, I want two arrays of indexes: the first for elements that satisfy a criterion, and the second for elements that do not. Naturally there are many ways to do this. Is there a preferred way? As a simple example, suppose for array `a` I want np.flatnonzero(a>0) and np.flatnonzero(a<=0). Can I get them both in one go? Thanks, Alan Isaac From ndarray at mac.com Sat Apr 12 17:01:42 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sat, 12 Apr 2014 17:01:42 -0400 Subject: [Numpy-discussion] index partition In-Reply-To: <5349A657.7080801@gmail.com> References: <5349A657.7080801@gmail.com> Message-ID: On Sat, Apr 12, 2014 at 4:47 PM, Alan G Isaac wrote: > As a simple example, suppose for array `a` I want > np.flatnonzero(a>0) and np.flatnonzero(a<=0). > Can I get them both in one go? > I don't think you can do better than x = a > 0 p, q = np.flatnonzero(x), np.flatnonzero(~x) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Apr 12 17:03:45 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 12 Apr 2014 23:03:45 +0200 Subject: [Numpy-discussion] index partition In-Reply-To: <5349A657.7080801@gmail.com> References: <5349A657.7080801@gmail.com> Message-ID: <1397336625.2433.1.camel@sebastian-t440> On Sa, 2014-04-12 at 16:47 -0400, Alan G Isaac wrote: > From a 1d array, I want two arrays of indexes: > the first for elements that satisfy a criterion, > and the second for elements that do not. Naturally > there are many ways to do this. Is there a preferred way? > > As a simple example, suppose for array `a` I want > np.flatnonzero(a>0) and np.flatnonzero(a<=0). > Can I get them both in one go? > Might be missing something, but I don't think there is a way to do it in one go. The result is irregularly structured and there are few functions like nonzero which give something like that. - Sebastian > Thanks, > Alan Isaac > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndarray at mac.com Sat Apr 12 17:20:59 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sat, 12 Apr 2014 17:20:59 -0400 Subject: [Numpy-discussion] index partition In-Reply-To: <1397336625.2433.1.camel@sebastian-t440> References: <5349A657.7080801@gmail.com> <1397336625.2433.1.camel@sebastian-t440> Message-ID: On Sat, Apr 12, 2014 at 5:03 PM, Sebastian Berg wrote: > > As a simple example, suppose for array `a` I want > > np.flatnonzero(a>0) and np.flatnonzero(a<=0). > > Can I get them both in one go? > > > > Might be missing something, but I don't think there is a way to do it in > one go. The result is irregularly structured and there are few functions > like nonzero which give something like that. The "set routines" [1] are in this category and may help you deal with partitions, but I would recommend using boolean arrays instead. If you commonly deal with both a subset and a complement, set representation does not give you a memory advantage over a boolean mask. [1] http://docs.scipy.org/doc/numpy/reference/routines.set.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Sun Apr 13 14:44:30 2014 From: faltet at gmail.com (Francesc Alted) Date: Sun, 13 Apr 2014 20:44:30 +0200 Subject: [Numpy-discussion] ANN: numexpr 2.4 is out Message-ID: <534ADB0E.1010300@gmail.com> ======================== Announcing Numexpr 2.4 ======================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== A new `contains()` function has been added for detecting substrings in strings. Only plain strings (bytes) are supported for now (see ticket #142). Thanks to Marcin Krol. You can have a glimpse on how `contains()` works in this notebook: http://nbviewer.ipython.org/gist/FrancescAlted/10595974 where it can be seen that this can make substring searches more than 10x faster than with regular Python. You can find the source for the notebook here: https://github.com/FrancescAlted/ngrams Also, there is a new version of setup.py that allows better management of the NumPy dependency during pip installs. Thanks to Aleks Bunin. Windows related bugs have been addressed and (hopefully) squashed. Thanks to Christoph Gohlke. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted From matt at pagan.io Mon Apr 14 00:26:44 2014 From: matt at pagan.io (matt at pagan.io) Date: Sun, 13 Apr 2014 23:26:44 -0500 Subject: [Numpy-discussion] Writing successful tests Message-ID: Greetings, I'm working on an additional function for numpy/lib/twodim_base.py. I'm trying to add some tests for the new function, and numpy/lib/tests/test_twodim_base.py seems like the right place for them. My problem is travis-ci tells me my tests are no good. The error message I get on my local machine is: ValueError: no such test method in : runTest (TestElementary is the class I made to put my tests in). This error makes me think I need a to put in a function like def __init__(self, methodName="runTest"): or maybe def runTest(self): somewhere. I don't see functions like this associated with any of the other classes in numpy/lib/tests/test_twodim_base.py though. Any hints on how to proceed? -- Matt Pagan matt at pagan.io PGP: 0xE9284418E360583C From njs at pobox.com Mon Apr 14 00:31:07 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 14 Apr 2014 06:31:07 +0200 Subject: [Numpy-discussion] Writing successful tests In-Reply-To: References: Message-ID: On Mon, Apr 14, 2014 at 6:26 AM, wrote: > Greetings, > > I'm working on an additional function for numpy/lib/twodim_base.py. I'm > trying to add some tests for the new function, and > numpy/lib/tests/test_twodim_base.py seems like the right place for > them. > My problem is travis-ci tells me my tests are no good. Hard to say without seeing the code, but are you subclassing TestCase? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From robert.kern at gmail.com Mon Apr 14 05:27:27 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Apr 2014 10:27:27 +0100 Subject: [Numpy-discussion] Writing successful tests In-Reply-To: References: Message-ID: On Mon, Apr 14, 2014 at 5:26 AM, wrote: > Greetings, > > I'm working on an additional function for numpy/lib/twodim_base.py. I'm > trying to add some tests for the new function, and > numpy/lib/tests/test_twodim_base.py seems like the right place for > them. > My problem is travis-ci tells me my tests are no good. > > The error message I get on my local machine is: > > ValueError: no such test method in : > runTest Please copy-and-paste the whole error message. Can you point to your code somewhere and the travis-ci results? How are you running the test suite? `numpy.test()` is the right way to do so. Do not use `unittest.main()` or `python -m unittest ...` to run the numpy test suite. > (TestElementary is the class I made to put my tests in). > > This error makes me think I need a to put in a function like > > def __init__(self, methodName="runTest"): > > or maybe > > def runTest(self): > > somewhere. I don't see functions like this associated with any of the > other classes in numpy/lib/tests/test_twodim_base.py though. You would never do so, no. numpy uses the `nose` test runner which does not need these. If you were using the standard unittest test collector, you would subclass `TestElementary` from `unittest.TestCase`, and that would provide the correct `__init__()` and `runTest()` methods. -- Robert Kern From warren.weckesser at gmail.com Mon Apr 14 07:09:11 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 14 Apr 2014 07:09:11 -0400 Subject: [Numpy-discussion] assert_equal(-0.0, 0.0) fails. Message-ID: The test function numpy.testing.assert_equal fails when comparing -0.0 and 0.0: In [16]: np.testing.assert_equal(-0.0, 0.0) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) in () ----> 1 np.testing.assert_equal(-0.0, 0.0) /Users/warren/anaconda/lib/python2.7/site-packages/numpy/testing/utils.pyc in assert_equal(actual, desired, err_msg, verbose) 309 elif desired == 0 and actual == 0: 310 if not signbit(desired) == signbit(actual): --> 311 raise AssertionError(msg) 312 # If TypeError or ValueError raised while using isnan and co, just handle 313 # as before AssertionError: Items are not equal: ACTUAL: -0.0 DESIRED: 0.0 There is code that checks for this specific case, so this is intentional. But this is not consistent with how negative zeros in arrays are compared: In [22]: np.testing.assert_equal(np.array(-0.0), np.array(0.0)) # PASS In [23]: a = np.array([-0.0]) In [24]: b = np.array([0.0]) In [25]: np.testing.assert_array_equal(a, b) # PASS Is there a reason the values are considered equal in an array, but not when compared as scalars? Warren From alan.isaac at gmail.com Mon Apr 14 12:17:35 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Mon, 14 Apr 2014 12:17:35 -0400 Subject: [Numpy-discussion] index partition In-Reply-To: References: <5349A657.7080801@gmail.com> <1397336625.2433.1.camel@sebastian-t440> Message-ID: <534C0A1F.4020100@gmail.com> On 4/12/2014 5:20 PM, Alexander Belopolsky wrote: > The "set routines" [1] are in this category and may help > you deal with partitions, but I would recommend using > boolean arrays instead. If you commonly deal with both > a subset and a complement, set representation does not > give you a memory advantage over a boolean mask. I take it that by a lack of a memory advantage you mean because boolean arrays are 8 bit representations. That makes sense. I find it rather more convenient to use boolean arrays, but I wonder if arrays of indexes might have other advantages (which would suggest using the set operations instead). In particular, might a[boolean_array] be slower that a[indexes]? (I'm just asking, not suggesting.) Thanks! Alan From matthew.brett at gmail.com Mon Apr 14 14:59:31 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 11:59:31 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit Message-ID: Hi, With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for Windows 64-bit, and latest stable ATLAS. It works fine, apart from the following test failure: ====================================================================== FAIL: test_special (test_umath.TestExpm1) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", line 329, in test_special assert_equal(ncu.expm1(-0.), -0.) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 311, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 0.0 DESIRED: -0.0 Has anyone seen this? Is it in fact necessary that expm1(-0.) return -0 instead of 0? Thanks a lot, Matthew From chris.barker at noaa.gov Mon Apr 14 14:59:37 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 14 Apr 2014 11:59:37 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer wrote: > On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Are we in a position to start looking at implementation? If so, it would >> be useful to have a collection of test cases, i.e., typical uses with >> specified results. That should also cover conversion from/(to?) >> datetime.datetime. >> > yup -- tests are always good! Indeed, my personal wish-list for np.datetime64 is centered much more on > robust conversion to/from native date objects, including comparison. > A good use case. > Here are some of my particular points of frustration (apologies for the > thread jacking!): > - NaT should have similar behavior to NaN when used for comparisons (i.e., > comparisons should always be False). > make sense. > - You can't compare a datetime object to a datetime64 object. > that would be nice to have. > - datetime64 objects with high precision (e.g., ns) can't compare to > datetime objects. > That's a problem, but how do you think it should be handled? My thought is that it should round to microseconds, and then compare -- kind of like comparing float32 and float64... > Pandas has a very nice wrapper around datetime64 arrays that solves most > of these issues, but it would be nice to get much of that functionality in > core numpy, > yes -- it would -- but learning from pandas is certainly a good idea. > > from numpy import datetime64 > from datetime import datetime > > print np.datetime64('NaT') < np.datetime64('2011-01-01') # this should not > to true > print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises exception > print np.datetime64('2011-01-01T00:00', 'ns') > datetime(2010, 1, 1) # > another exception > print np.datetime64('2011-01-01T00:00') > datetime(2010, 1, 1) # finally > something works! > > now to get them into proper unit tests.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Apr 14 15:12:48 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 14 Apr 2014 15:12:48 -0400 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: Message-ID: On Mon, Apr 14, 2014 at 2:59 PM, Matthew Brett wrote: > Hi, > > With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for > Windows 64-bit, and latest stable ATLAS. > > It works fine, apart from the following test failure: > > ====================================================================== > FAIL: test_special (test_umath.TestExpm1) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", > line 329, in test_special > assert_equal(ncu.expm1(-0.), -0.) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 311, in assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > ACTUAL: 0.0 > DESIRED: -0.0 > > Has anyone seen this? Is it in fact necessary that expm1(-0.) return > -0 instead of 0? > > What a cowinky dink. This moring I ran into this issue in a scipy pull request (https://github.com/scipy/scipy/pull/3547), and I asked about this comparison failing on the mailing list a few hours ago. In the pull request, the modified function returns -0.0 where it used to return 0.0, and the test for the value 0.0 failed. My work-around was to use `assert_array_equal` instead of `assert_equal`. The array comparison functions treat the values -0.0 and 0.0 as equal. `assert_equal` has code that checks for signed zeros, and fails if they are not the same sign. Warren Thanks a lot, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Apr 14 15:25:33 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 12:25:33 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: Message-ID: Hi, On Mon, Apr 14, 2014 at 12:12 PM, Warren Weckesser wrote: > > On Mon, Apr 14, 2014 at 2:59 PM, Matthew Brett > wrote: >> >> Hi, >> >> With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for >> Windows 64-bit, and latest stable ATLAS. >> >> It works fine, apart from the following test failure: >> >> ====================================================================== >> FAIL: test_special (test_umath.TestExpm1) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", >> line 329, in test_special >> assert_equal(ncu.expm1(-0.), -0.) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 311, in assert_equal >> raise AssertionError(msg) >> AssertionError: >> Items are not equal: >> ACTUAL: 0.0 >> DESIRED: -0.0 >> >> Has anyone seen this? Is it in fact necessary that expm1(-0.) return >> -0 instead of 0? >> > > > What a cowinky dink. This moring I ran into this issue in a scipy pull > request (https://github.com/scipy/scipy/pull/3547), and I asked about this > comparison failing on the mailing list a few hours ago. In the pull > request, the modified function returns -0.0 where it used to return 0.0, and > the test for the value 0.0 failed. My work-around was to use > `assert_array_equal` instead of `assert_equal`. The array comparison > functions treat the values -0.0 and 0.0 as equal. `assert_equal` has code > that checks for signed zeros, and fails if they are not the same sign. Yes, sorry, I should have mentioned that I saw your post too. I'd live to use that workaround, but it looks like the teste against -0 is intentional, and I was hoping for help understanding. The relevant two lines of the test are: assert_equal(ncu.expm1(0.), 0.) assert_equal(ncu.expm1(-0.), -0.) Julian - I think this one is for you - as the author of these lines: f53ab41a numpy/core/tests/test_umath.py (Julian Taylor 2014-03-02 02:55:30 +0100 329) assert_equal(ncu.expm1(-0.), -0.) Can you say why you wanted -0 specifically? Any clue as to why ATLAS 64 bit may give 0 instead? Cheers, Matthew From ewm at redtetrahedron.org Mon Apr 14 15:34:24 2014 From: ewm at redtetrahedron.org (Eric Moore) Date: Mon, 14 Apr 2014 15:34:24 -0400 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: Message-ID: On Monday, April 14, 2014, Matthew Brett wrote: > Hi, > > On Mon, Apr 14, 2014 at 12:12 PM, Warren Weckesser > > wrote: > > > > On Mon, Apr 14, 2014 at 2:59 PM, Matthew Brett > > > > wrote: > >> > >> Hi, > >> > >> With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for > >> Windows 64-bit, and latest stable ATLAS. > >> > >> It works fine, apart from the following test failure: > >> > >> ====================================================================== > >> FAIL: test_special (test_umath.TestExpm1) > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", > >> line 329, in test_special > >> assert_equal(ncu.expm1(-0.), -0.) > >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > >> 311, in assert_equal > >> raise AssertionError(msg) > >> AssertionError: > >> Items are not equal: > >> ACTUAL: 0.0 > >> DESIRED: -0.0 > >> > >> Has anyone seen this? Is it in fact necessary that expm1(-0.) return > >> -0 instead of 0? > >> > > > > > > What a cowinky dink. This moring I ran into this issue in a scipy pull > > request (https://github.com/scipy/scipy/pull/3547), and I asked about > this > > comparison failing on the mailing list a few hours ago. In the pull > > request, the modified function returns -0.0 where it used to return 0.0, > and > > the test for the value 0.0 failed. My work-around was to use > > `assert_array_equal` instead of `assert_equal`. The array comparison > > functions treat the values -0.0 and 0.0 as equal. `assert_equal` has > code > > that checks for signed zeros, and fails if they are not the same sign. > > Yes, sorry, I should have mentioned that I saw your post too. I'd > live to use that workaround, but it looks like the teste against -0 is > intentional, and I was hoping for help understanding. > > The relevant two lines of the test are: > > assert_equal(ncu.expm1(0.), 0.) > assert_equal(ncu.expm1(-0.), -0.) > > Julian - I think this one is for you - as the author of these lines: > > f53ab41a numpy/core/tests/test_umath.py (Julian Taylor > 2014-03-02 02:55:30 +0100 329) assert_equal(ncu.expm1(-0.), > -0.) > > Can you say why you wanted -0 specifically? Any clue as to why ATLAS > 64 bit may give 0 instead? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I think this is a real bug in the version of exp1m in core/src/npymath/npy_math.c.src that's being used since your windows build couldn't find a system version of exp1m to use. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Apr 14 16:05:48 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 14 Apr 2014 22:05:48 +0200 Subject: [Numpy-discussion] assert_equal(-0.0, 0.0) fails. In-Reply-To: References: Message-ID: On Mon, Apr 14, 2014 at 1:09 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > The test function numpy.testing.assert_equal fails when comparing -0.0 and > 0.0: > > In [16]: np.testing.assert_equal(-0.0, 0.0) > --------------------------------------------------------------------------- > AssertionError Traceback (most recent call last) > in () > ----> 1 np.testing.assert_equal(-0.0, 0.0) > > /Users/warren/anaconda/lib/python2.7/site-packages/numpy/testing/utils.pyc > in assert_equal(actual, desired, err_msg, verbose) > 309 elif desired == 0 and actual == 0: > 310 if not signbit(desired) == signbit(actual): > --> 311 raise AssertionError(msg) > 312 # If TypeError or ValueError raised while using isnan and > co, just handle > 313 # as before > > AssertionError: > Items are not equal: > ACTUAL: -0.0 > DESIRED: 0.0 > > There is code that checks for this specific case, so this is > intentional. But this is not consistent with how negative zeros in > arrays are compared: > > In [22]: np.testing.assert_equal(np.array(-0.0), np.array(0.0)) # PASS > > In [23]: a = np.array([-0.0]) > > In [24]: b = np.array([0.0]) > > In [25]: np.testing.assert_array_equal(a, b) # PASS > > > Is there a reason the values are considered equal in an array, but not > when compared as scalars? > Unlikely to be intentional. I expect this was a fix to assert_equal that wasn't synced to assert_array_equal. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Mon Apr 14 16:43:17 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Mon, 14 Apr 2014 22:43:17 +0200 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: Message-ID: Hi, mingw has expm1. Is this function suitable? Regards Carl 2014-04-14 21:34 GMT+02:00 Eric Moore : > > > On Monday, April 14, 2014, Matthew Brett wrote: > >> Hi, >> >> On Mon, Apr 14, 2014 at 12:12 PM, Warren Weckesser >> wrote: >> > >> > On Mon, Apr 14, 2014 at 2:59 PM, Matthew Brett > > >> > wrote: >> >> >> >> Hi, >> >> >> >> With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for >> >> Windows 64-bit, and latest stable ATLAS. >> >> >> >> It works fine, apart from the following test failure: >> >> >> >> ====================================================================== >> >> FAIL: test_special (test_umath.TestExpm1) >> >> ---------------------------------------------------------------------- >> >> Traceback (most recent call last): >> >> File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", >> >> line 329, in test_special >> >> assert_equal(ncu.expm1(-0.), -0.) >> >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> >> 311, in assert_equal >> >> raise AssertionError(msg) >> >> AssertionError: >> >> Items are not equal: >> >> ACTUAL: 0.0 >> >> DESIRED: -0.0 >> >> >> >> Has anyone seen this? Is it in fact necessary that expm1(-0.) return >> >> -0 instead of 0? >> >> >> > >> > >> > What a cowinky dink. This moring I ran into this issue in a scipy pull >> > request (https://github.com/scipy/scipy/pull/3547), and I asked about >> this >> > comparison failing on the mailing list a few hours ago. In the pull >> > request, the modified function returns -0.0 where it used to return >> 0.0, and >> > the test for the value 0.0 failed. My work-around was to use >> > `assert_array_equal` instead of `assert_equal`. The array comparison >> > functions treat the values -0.0 and 0.0 as equal. `assert_equal` has >> code >> > that checks for signed zeros, and fails if they are not the same sign. >> >> Yes, sorry, I should have mentioned that I saw your post too. I'd >> live to use that workaround, but it looks like the teste against -0 is >> intentional, and I was hoping for help understanding. >> >> The relevant two lines of the test are: >> >> assert_equal(ncu.expm1(0.), 0.) >> assert_equal(ncu.expm1(-0.), -0.) >> >> Julian - I think this one is for you - as the author of these lines: >> >> f53ab41a numpy/core/tests/test_umath.py (Julian Taylor >> 2014-03-02 02:55:30 +0100 329) assert_equal(ncu.expm1(-0.), >> -0.) >> >> Can you say why you wanted -0 specifically? Any clue as to why ATLAS >> 64 bit may give 0 instead? >> >> Cheers, >> >> Matthew >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > I think this is a real bug in the version of exp1m in > core/src/npymath/npy_math.c.src that's being used since your windows build > couldn't find a system version of exp1m to use. > > Eric > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Apr 14 17:02:28 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 14 Apr 2014 23:02:28 +0200 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: Message-ID: <534C4CE4.1030407@googlemail.com> The official numpy mingw binaries do not have all these math issues. Only the VC builds do. As mingw is fine the functions must be somewhere in the windows API but no-one has contributed a fix for the VC builds to numpy yet. On 14.04.2014 22:43, Carl Kleffner wrote: > Hi, > > mingw has expm1. Is this function suitable? > > Regards > > Carl > > > > > 2014-04-14 21:34 GMT+02:00 Eric Moore >: > > > > On Monday, April 14, 2014, Matthew Brett > wrote: > > Hi, > > On Mon, Apr 14, 2014 at 12:12 PM, Warren Weckesser > wrote: > > > > On Mon, Apr 14, 2014 at 2:59 PM, Matthew Brett > > > wrote: > >> > >> Hi, > >> > >> With Carl Kleffner, I am trying to build a numpy 1.8.1 wheel for > >> Windows 64-bit, and latest stable ATLAS. > >> > >> It works fine, apart from the following test failure: > >> > >> > ====================================================================== > >> FAIL: test_special (test_umath.TestExpm1) > >> > ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", > >> line 329, in test_special > >> assert_equal(ncu.expm1(-0.), -0.) > >> File > "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > >> 311, in assert_equal > >> raise AssertionError(msg) > >> AssertionError: > >> Items are not equal: > >> ACTUAL: 0.0 > >> DESIRED: -0.0 > >> > >> Has anyone seen this? Is it in fact necessary that > expm1(-0.) return > >> -0 instead of 0? > >> > > > > > > What a cowinky dink. This moring I ran into this issue in a > scipy pull > > request (https://github.com/scipy/scipy/pull/3547), and I > asked about this > > comparison failing on the mailing list a few hours ago. In > the pull > > request, the modified function returns -0.0 where it used to > return 0.0, and > > the test for the value 0.0 failed. My work-around was to use > > `assert_array_equal` instead of `assert_equal`. The array > comparison > > functions treat the values -0.0 and 0.0 as equal. > `assert_equal` has code > > that checks for signed zeros, and fails if they are not the > same sign. > > Yes, sorry, I should have mentioned that I saw your post too. I'd > live to use that workaround, but it looks like the teste against > -0 is > intentional, and I was hoping for help understanding. > > The relevant two lines of the test are: > > assert_equal(ncu.expm1(0.), 0.) > assert_equal(ncu.expm1(-0.), -0.) > > Julian - I think this one is for you - as the author of these lines: > > f53ab41a numpy/core/tests/test_umath.py (Julian Taylor > 2014-03-02 02:55:30 +0100 329) assert_equal(ncu.expm1(-0.), > -0.) > > Can you say why you wanted -0 specifically? Any clue as to why > ATLAS > 64 bit may give 0 instead? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > I think this is a real bug in the version of exp1m in > core/src/npymath/npy_math.c.src that's being used since your windows > build couldn't find a system version of exp1m to use. > > Eric > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthew.brett at gmail.com Mon Apr 14 17:38:18 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 14:38:18 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: <534C4CE4.1030407@googlemail.com> References: <534C4CE4.1030407@googlemail.com> Message-ID: Hi, On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor wrote: > The official numpy mingw binaries do not have all these math issues. > Only the VC builds do. > As mingw is fine the functions must be somewhere in the windows API but > no-one has contributed a fix for the VC builds to numpy yet. I'm building with mingw-w64. It looks like this works as expected from this test: #include #include int main() { double z; z = expm1(-0.0); printf("Result %f\n", z); } (prints -0). as does this (modified from core/src/npymath/npy_math.c.src): #include double npy_expm1(double x) { if (isinf(x) && x > 0) { return x; } else { const double u = exp(x); if (u == 1.0) { return x; } else if (u - 1.0 == -1.0) { return -1; } else { return (u - 1.0) * x/log(u); } } } int main() { double z; z = npy_expm1(-0.0); printf("Result %f\n", z); } Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come from? Cheers, Matthew From charlesr.harris at gmail.com Mon Apr 14 17:58:41 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 14 Apr 2014 15:58:41 -0600 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: On Mon, Apr 14, 2014 at 3:38 PM, Matthew Brett wrote: > Hi, > > On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor > wrote: > > The official numpy mingw binaries do not have all these math issues. > > Only the VC builds do. > > As mingw is fine the functions must be somewhere in the windows API but > > no-one has contributed a fix for the VC builds to numpy yet. > > I'm building with mingw-w64. > > It looks like this works as expected from this test: > > #include > #include > > int main() { > double z; > z = expm1(-0.0); > printf("Result %f\n", z); > } > > (prints -0). > > as does this (modified from core/src/npymath/npy_math.c.src): > > #include > > double npy_expm1(double x) > { > if (isinf(x) && x > 0) { > return x; > } > else { > const double u = exp(x); > > if (u == 1.0) { > return x; > } else if (u - 1.0 == -1.0) { > return -1; > } else { > return (u - 1.0) * x/log(u); > } > } > } > > int main() { > double z; > z = npy_expm1(-0.0); > printf("Result %f\n", z); > } > > Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come from? > > Remember all configuration output at the beginning of the build? One of those tries to link 'expm1{f, , l}' and sets the value of 'HAVE_EXPM1{F,,L}' accordingly. You can find the result in the `config.h` file in the build directory. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 14 18:33:36 2014 From: cournape at gmail.com (David Cournapeau) Date: Mon, 14 Apr 2014 23:33:36 +0100 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: <534C4CE4.1030407@googlemail.com> References: <534C4CE4.1030407@googlemail.com> Message-ID: On Mon, Apr 14, 2014 at 10:02 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > The official numpy mingw binaries do not have all these math issues. > Only the VC builds do. > As mingw is fine the functions must be somewhere in the windows API but > no-one has contributed a fix for the VC builds to numpy yet. > Mingw generally has their own implementation, and do not use the C runtime from MS for math functions: https://github.com/mirror/mingw-w64/tree/master/trunk/mingw-w64-crt/math David -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Apr 14 18:40:11 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 15:40:11 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: Hi, On Mon, Apr 14, 2014 at 2:58 PM, Charles R Harris wrote: > > > > On Mon, Apr 14, 2014 at 3:38 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor >> wrote: >> > The official numpy mingw binaries do not have all these math issues. >> > Only the VC builds do. >> > As mingw is fine the functions must be somewhere in the windows API but >> > no-one has contributed a fix for the VC builds to numpy yet. >> >> I'm building with mingw-w64. >> >> It looks like this works as expected from this test: >> >> #include >> #include >> >> int main() { >> double z; >> z = expm1(-0.0); >> printf("Result %f\n", z); >> } >> >> (prints -0). >> >> as does this (modified from core/src/npymath/npy_math.c.src): >> >> #include >> >> double npy_expm1(double x) >> { >> if (isinf(x) && x > 0) { >> return x; >> } >> else { >> const double u = exp(x); >> >> if (u == 1.0) { >> return x; >> } else if (u - 1.0 == -1.0) { >> return -1; >> } else { >> return (u - 1.0) * x/log(u); >> } >> } >> } >> >> int main() { >> double z; >> z = npy_expm1(-0.0); >> printf("Result %f\n", z); >> } >> >> Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come from? >> > > Remember all configuration output at the beginning of the build? One of > those tries to link 'expm1{f, , l}' and sets the value of 'HAVE_EXPM1{F,,L}' > accordingly. You can find the result in the `config.h` file in the build > directory. Ah - thanks - that got me going - somewhere. I noticed that HAVE_EXPM1 is not defined in any of OSX, my Windows build, Debian config.h - is that expected? Cheers, Matthew From charlesr.harris at gmail.com Mon Apr 14 18:55:08 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 14 Apr 2014 16:55:08 -0600 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: On Mon, Apr 14, 2014 at 4:40 PM, Matthew Brett wrote: > Hi, > > On Mon, Apr 14, 2014 at 2:58 PM, Charles R Harris > wrote: > > > > > > > > On Mon, Apr 14, 2014 at 3:38 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor > >> wrote: > >> > The official numpy mingw binaries do not have all these math issues. > >> > Only the VC builds do. > >> > As mingw is fine the functions must be somewhere in the windows API > but > >> > no-one has contributed a fix for the VC builds to numpy yet. > >> > >> I'm building with mingw-w64. > >> > >> It looks like this works as expected from this test: > >> > >> #include > >> #include > >> > >> int main() { > >> double z; > >> z = expm1(-0.0); > >> printf("Result %f\n", z); > >> } > >> > >> (prints -0). > >> > >> as does this (modified from core/src/npymath/npy_math.c.src): > >> > >> #include > >> > >> double npy_expm1(double x) > >> { > >> if (isinf(x) && x > 0) { > >> return x; > >> } > >> else { > >> const double u = exp(x); > >> > >> if (u == 1.0) { > >> return x; > >> } else if (u - 1.0 == -1.0) { > >> return -1; > >> } else { > >> return (u - 1.0) * x/log(u); > >> } > >> } > >> } > >> > >> int main() { > >> double z; > >> z = npy_expm1(-0.0); > >> printf("Result %f\n", z); > >> } > >> > >> Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come > from? > >> > > > > Remember all configuration output at the beginning of the build? One of > > those tries to link 'expm1{f, , l}' and sets the value of > 'HAVE_EXPM1{F,,L}' > > accordingly. You can find the result in the `config.h` file in the build > > directory. > > Ah - thanks - that got me going - somewhere. > > I noticed that HAVE_EXPM1 is not defined in any of OSX, my Windows > build, Debian config.h - is that expected? > It's defined for me, gcc version 4.8.2 on fedora 20. I'm surprised that Debian doesn't show it, what compiler version does Debian provide? We could always borrow the boost version, although it uses the Kahan sum for the series. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at pagan.io Mon Apr 14 19:08:20 2014 From: matt at pagan.io (Matt Pagan) Date: Mon, 14 Apr 2014 23:08:20 +0000 Subject: [Numpy-discussion] Writing successful tests In-Reply-To: References: Message-ID: <534C6A64.5010309@pagan.io> Robert Kern: > On Mon, Apr 14, 2014 at 5:26 AM, wrote: >> Greetings, >> >> I'm working on an additional function for numpy/lib/twodim_base.py. I'm >> trying to add some tests for the new function, and >> numpy/lib/tests/test_twodim_base.py seems like the right place for >> them. >> My problem is travis-ci tells me my tests are no good. >> >> The error message I get on my local machine is: >> >> ValueError: no such test method in : >> runTest > > Please copy-and-paste the whole error message. Can you point to your > code somewhere and the travis-ci results? How are you running the test > suite? `numpy.test()` is the right way to do so. Do not use > `unittest.main()` or `python -m unittest ...` to run the numpy test > suite. > Thanks for telling me the correct way to run tests. This info got me a more useful error message, which I used to make this commit to my local fork: https://github.com/mttp/numpy/commit/480c51b17c608a2cbcca65b8a633dbe1b1654070 Seems to fix the problem. -- Matt Pagan matt at pagan.io PGP: 0xE9284418E360583C From matthew.brett at gmail.com Mon Apr 14 20:01:20 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 17:01:20 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: Hi, On Mon, Apr 14, 2014 at 3:55 PM, Charles R Harris wrote: > > > > On Mon, Apr 14, 2014 at 4:40 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Mon, Apr 14, 2014 at 2:58 PM, Charles R Harris >> wrote: >> > >> > >> > >> > On Mon, Apr 14, 2014 at 3:38 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor >> >> wrote: >> >> > The official numpy mingw binaries do not have all these math issues. >> >> > Only the VC builds do. >> >> > As mingw is fine the functions must be somewhere in the windows API >> >> > but >> >> > no-one has contributed a fix for the VC builds to numpy yet. >> >> >> >> I'm building with mingw-w64. >> >> >> >> It looks like this works as expected from this test: >> >> >> >> #include >> >> #include >> >> >> >> int main() { >> >> double z; >> >> z = expm1(-0.0); >> >> printf("Result %f\n", z); >> >> } >> >> >> >> (prints -0). >> >> >> >> as does this (modified from core/src/npymath/npy_math.c.src): >> >> >> >> #include >> >> >> >> double npy_expm1(double x) >> >> { >> >> if (isinf(x) && x > 0) { >> >> return x; >> >> } >> >> else { >> >> const double u = exp(x); >> >> >> >> if (u == 1.0) { >> >> return x; >> >> } else if (u - 1.0 == -1.0) { >> >> return -1; >> >> } else { >> >> return (u - 1.0) * x/log(u); >> >> } >> >> } >> >> } >> >> >> >> int main() { >> >> double z; >> >> z = npy_expm1(-0.0); >> >> printf("Result %f\n", z); >> >> } >> >> >> >> Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come >> >> from? >> >> >> > >> > Remember all configuration output at the beginning of the build? One of >> > those tries to link 'expm1{f, , l}' and sets the value of >> > 'HAVE_EXPM1{F,,L}' >> > accordingly. You can find the result in the `config.h` file in the build >> > directory. >> >> Ah - thanks - that got me going - somewhere. >> >> I noticed that HAVE_EXPM1 is not defined in any of OSX, my Windows >> build, Debian config.h - is that expected? > > > It's defined for me, gcc version 4.8.2 on fedora 20. I'm surprised that > Debian doesn't show it, what compiler version does Debian provide? Yes, sorry - I had an old version of numpy on Debian. It is defined for 1.8.1 It's not defined on OSX because it is defined in pyconfig.h (via Python.h). I was wrong, it is defined in config.h for Windows, I'm not sure how I missed that. Now I'm trying to understand why it's giving different results from the tiny test function above, Cheers, Matthew From matthew.brett at gmail.com Mon Apr 14 22:30:05 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Apr 2014 19:30:05 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: On Mon, Apr 14, 2014 at 5:01 PM, Matthew Brett wrote: > Hi, > > On Mon, Apr 14, 2014 at 3:55 PM, Charles R Harris > wrote: >> >> >> >> On Mon, Apr 14, 2014 at 4:40 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> On Mon, Apr 14, 2014 at 2:58 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > >>> > On Mon, Apr 14, 2014 at 3:38 PM, Matthew Brett >>> > wrote: >>> >> >>> >> Hi, >>> >> >>> >> On Mon, Apr 14, 2014 at 2:02 PM, Julian Taylor >>> >> wrote: >>> >> > The official numpy mingw binaries do not have all these math issues. >>> >> > Only the VC builds do. >>> >> > As mingw is fine the functions must be somewhere in the windows API >>> >> > but >>> >> > no-one has contributed a fix for the VC builds to numpy yet. >>> >> >>> >> I'm building with mingw-w64. >>> >> >>> >> It looks like this works as expected from this test: >>> >> >>> >> #include >>> >> #include >>> >> >>> >> int main() { >>> >> double z; >>> >> z = expm1(-0.0); >>> >> printf("Result %f\n", z); >>> >> } >>> >> >>> >> (prints -0). >>> >> >>> >> as does this (modified from core/src/npymath/npy_math.c.src): >>> >> >>> >> #include >>> >> >>> >> double npy_expm1(double x) >>> >> { >>> >> if (isinf(x) && x > 0) { >>> >> return x; >>> >> } >>> >> else { >>> >> const double u = exp(x); >>> >> >>> >> if (u == 1.0) { >>> >> return x; >>> >> } else if (u - 1.0 == -1.0) { >>> >> return -1; >>> >> } else { >>> >> return (u - 1.0) * x/log(u); >>> >> } >>> >> } >>> >> } >>> >> >>> >> int main() { >>> >> double z; >>> >> z = npy_expm1(-0.0); >>> >> printf("Result %f\n", z); >>> >> } >>> >> >>> >> Sorry for my ignorance, but where does the `HAVE_EXPM1` symbol come >>> >> from? >>> >> >>> > >>> > Remember all configuration output at the beginning of the build? One of >>> > those tries to link 'expm1{f, , l}' and sets the value of >>> > 'HAVE_EXPM1{F,,L}' >>> > accordingly. You can find the result in the `config.h` file in the build >>> > directory. >>> >>> Ah - thanks - that got me going - somewhere. >>> >>> I noticed that HAVE_EXPM1 is not defined in any of OSX, my Windows >>> build, Debian config.h - is that expected? >> >> >> It's defined for me, gcc version 4.8.2 on fedora 20. I'm surprised that >> Debian doesn't show it, what compiler version does Debian provide? > > Yes, sorry - I had an old version of numpy on Debian. It is defined for 1.8.1 > > It's not defined on OSX because it is defined in pyconfig.h (via Python.h). > > I was wrong, it is defined in config.h for Windows, I'm not sure how I > missed that. Now I'm trying to understand why it's giving different > results from the tiny test function above, It looks as though mingw-w64 is at fault, and I was confused (still am) because of the different behavior with double and a constant: #include #include int main() { double z, i = -0.0; printf("With double %f=%f, with constant=%f\n", i, expm1(i), expm1(-0.)); } gives: With double -0.000000=0.000000, with constant=-0.000000 That was ugly to track down. What is the right way to work round this (using the numpy version instead of the system version I suppose)? Cheers, Matthew From jtaylor.debian at googlemail.com Tue Apr 15 03:34:57 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 15 Apr 2014 09:34:57 +0200 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: On Tue, Apr 15, 2014 at 4:30 AM, Matthew Brett wrote: > > It looks as though mingw-w64 is at fault, and I was confused (still > am) because of the different behavior with double and a constant: > > #include > #include > > int main() { > double z, i = -0.0; > printf("With double %f=%f, with constant=%f\n", > i, expm1(i), expm1(-0.)); > } > > gives: > > With double -0.000000=0.000000, with constant=-0.000000 > > That was ugly to track down. > > What is the right way to work round this (using the numpy version > instead of the system version I suppose)? > The right way is to file a bug at mingw and get it fixed at the source. Additionally as this time npymath seems to be better (thats 3 bugs in npymath vs 1 in mingw on my scoreboard) one could use the mingw preprocessor define instead of HAVE_EXP1M to select this function from npymath. From davidmenhur at gmail.com Tue Apr 15 04:34:48 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Tue, 15 Apr 2014 10:34:48 +0200 Subject: [Numpy-discussion] index partition In-Reply-To: <534C0A1F.4020100@gmail.com> References: <5349A657.7080801@gmail.com> <1397336625.2433.1.camel@sebastian-t440> <534C0A1F.4020100@gmail.com> Message-ID: On 14 April 2014 18:17, Alan G Isaac wrote: > I find it rather more convenient to use boolean arrays, > but I wonder if arrays of indexes might have other > advantages (which would suggest using the set operations > instead). In particular, might a[boolean_array] be slower > that a[indexes]? (I'm just asking, not suggesting.) Indexing is generally faster, but convert from boolean to indexes gets more expensive: In [2]: arr =np.random.random(1000) In [3]: mask = arr>0.7 In [4]: mask.sum() Out[4]: 290 In [5]: %timeit arr[mask] 100000 loops, best of 3: 4.01 ?s per loop In [6]: %%timeit ...: wh = np.where(mask) ...: arr[wh] ...: 100000 loops, best of 3: 6.47 ?s per loop In [8]: wh = np.where(mask) In [9]: %timeit arr[wh] 100000 loops, best of 3: 2.57 ?s per loop In [10]: %timeit np.where(mask) 100000 loops, best of 3: 3.89 ?s per loop In [14]: np.all(arr[wh] == arr[mask]) Out[14]: True If you want to apply the same mask to several arrays, it is then worth (performance-wise) to do it. /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 15 05:48:14 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 15 Apr 2014 11:48:14 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed Message-ID: Hey all, The well known memory_profiler module [1] is super-useful, but has a fundamental limitation, which is the only way it can track allocations is by constantly polling the OS for the size of the total process address space. This is a crude and unreliable way of making measurements. In Python 3.4, there's a new "allocation hooks" infrastructure, that allows one to precisely track the lifetime and size of every allocation [2]. So this is pretty awesome, and we can expect there will be more tools growing up around this interface. But unfortunately, this system is useless for numpy right now, because numpy does not use the Python memory allocation interface; this means that numpy data is "invisible" to tracemalloc and related tools. Why doesn't numpy use the Python memory allocation interface? Because numpy needs calloc(), but Python doesn't expose calloc(), only malloc()/realloc()/free(). Good news, though! python-dev is in favor of adding calloc() to the core allocation interfaces, which will let numpy join the party. See python-dev thread: https://mail.python.org/pipermail/python-dev/2014-April/133985.html It would be especially nice if we could get this into 3.5, since it seems likely that lots of numpy users will be switching to 3.5 when it comes out, and having a good memory tracing infrastructure there waiting for them make it even more awesome. Anyone interested in picking this up? http://bugs.python.org/issue21233 -n [1] https://pypi.python.org/pypi/memory_profiler [2] https://docs.python.org/3.4/library/tracemalloc.html -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From jtaylor.debian at googlemail.com Tue Apr 15 06:06:13 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 15 Apr 2014 12:06:13 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: > Good news, though! python-dev is in favor of adding calloc() to the > core allocation interfaces, which will let numpy join the party. See > python-dev thread: > https://mail.python.org/pipermail/python-dev/2014-April/133985.html > > It would be especially nice if we could get this into 3.5, since it > seems likely that lots of numpy users will be switching to 3.5 when it > comes out, and having a good memory tracing infrastructure there > waiting for them make it even more awesome. > > Anyone interested in picking this up? > http://bugs.python.org/issue21233 > Hi, I think it would be a better idea to instead of API functions for one different type of allocator we get access to use the python hooks directly with whatever allocator we want to use. This would allow as to for example use aligned memory allocators which might be relevant for the new cpu instruction sets with up to 64 byte wide registers (it would be great if someone with avx512 hardware could provide some benchmarks for unaligned, 16 byte aligned and 64 byte aligned memory to judge if this is actually required) On the other hand memory tracing is a debugging feature and you might not care about performance, so we could use the python allocators in debug mode and the aligned unhooked allocators in normal mode? Cheers, Julian From njs at pobox.com Tue Apr 15 09:07:25 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 15 Apr 2014 15:07:25 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor wrote: >> Good news, though! python-dev is in favor of adding calloc() to the >> core allocation interfaces, which will let numpy join the party. See >> python-dev thread: >> https://mail.python.org/pipermail/python-dev/2014-April/133985.html >> >> It would be especially nice if we could get this into 3.5, since it >> seems likely that lots of numpy users will be switching to 3.5 when it >> comes out, and having a good memory tracing infrastructure there >> waiting for them make it even more awesome. >> >> Anyone interested in picking this up? >> http://bugs.python.org/issue21233 > > Hi, > I think it would be a better idea to instead of API functions for one > different type of allocator we get access to use the python hooks > directly with whatever allocator we want to use. Unfortunately, that's not how the API works. The way that third-party tracers register a 'hook' is by providing a new implementation of malloc/free/etc. So there's no general way to say "please pretend to have done a malloc". I guess we could potentially request the addition of fake_malloc/fake_free functions. > This would allow as to for example use aligned memory allocators which > might be relevant for the new cpu instruction sets with up to 64 byte > wide registers I think we might have had this conversation before, but I don't remember how it went... did you have some explanation about how this could matter in principle? We have to write code to handle unaligned (or imperfectly aligned) arrays regardless, so aligned allocation doesn't affect maintainability. And regarding speed, I can't see how an extra instruction here or there could make a big difference on small arrays, since the effect should be overwhelmed by interpreter overhead and memory stalls (not much time for prefetch to come into play on small arrays), but OTOH large arrays are usually page-aligned in practice, and if not then any extra start-up overhead will be amortized out by their size. > (it would be great if someone with avx512 hardware > could provide some benchmarks for unaligned, 16 byte aligned and 64 > byte aligned memory to judge if this is actually required) That would be great, but given that the processors won't be in general distribution until next year, do we have any idea of how to find such a person? :-) > On the other hand memory tracing is a debugging feature and you might > not care about performance, so we could use the python allocators in > debug mode and the aligned unhooked allocators in normal mode? I don't think we can change which memory allocator is in use once we have any live ndarray objects in memory (because eventually we will need to free them, and we'll need to know which allocator was used for which). But a major use case for this is stuff like [1]: %memit my_func() and it would be really user-unfriendly to force a restart every time you want to use %memit. -n [1] http://scikit-learn.org/stable/developers/performance.html#memory-usage-profiling -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From jtaylor.debian at googlemail.com Tue Apr 15 10:08:45 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 15 Apr 2014 16:08:45 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On Tue, Apr 15, 2014 at 3:07 PM, Nathaniel Smith wrote: > On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor > wrote: >>> Good news, though! python-dev is in favor of adding calloc() to the >>> core allocation interfaces, which will let numpy join the party. See >>> python-dev thread: >>> https://mail.python.org/pipermail/python-dev/2014-April/133985.html >>> >>> It would be especially nice if we could get this into 3.5, since it >>> seems likely that lots of numpy users will be switching to 3.5 when it >>> comes out, and having a good memory tracing infrastructure there >>> waiting for them make it even more awesome. >>> >>> Anyone interested in picking this up? >>> http://bugs.python.org/issue21233 >> >> Hi, >> I think it would be a better idea to instead of API functions for one >> different type of allocator we get access to use the python hooks >> directly with whatever allocator we want to use. > > Unfortunately, that's not how the API works. The way that third-party > tracers register a 'hook' is by providing a new implementation of > malloc/free/etc. So there's no general way to say "please pretend to > have done a malloc". > > I guess we could potentially request the addition of > fake_malloc/fake_free functions. Unfortunate, looking at the pep it seems either you have a custom allocator or you have tracing but not both (unless you trace yourself). This seems like quite a limitation. Maybe it would have been more flexible if instead python provided three functions: PyMem_RegisterAlloc(size); PyMem_RegisterReAlloc(size); PyMem_RegisterFree(size); + possibly nogil variantes These functions call into registered tracing functions (registered e.g. by tracemalloc.start()) or do nothing. Our allocator (and pythons) then just always calls these functions and continues doing its stuff. > >> This would allow as to for example use aligned memory allocators which >> might be relevant for the new cpu instruction sets with up to 64 byte >> wide registers > > I think we might have had this conversation before, but I don't > remember how it went... did you have some explanation about how this > could matter in principle? We have to write code to handle unaligned > (or imperfectly aligned) arrays regardless, so aligned allocation > doesn't affect maintainability. And regarding speed, I can't see how > an extra instruction here or there could make a big difference on > small arrays, since the effect should be overwhelmed by interpreter > overhead and memory stalls (not much time for prefetch to come into > play on small arrays), but OTOH large arrays are usually page-aligned > in practice, and if not then any extra start-up overhead will be > amortized out by their size. yes we already had this conversation :) if you have two or more arrays not aligned the same way you can only align one of them via peeling, the others will always have to be accessed unaligned. But it probably does not matter anymore with newer cpus, I should probably just throw out my old core2 where it does :) From njs at pobox.com Tue Apr 15 12:39:11 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 15 Apr 2014 18:39:11 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On Tue, Apr 15, 2014 at 4:08 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On Tue, Apr 15, 2014 at 3:07 PM, Nathaniel Smith wrote: >> On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor >> wrote: >>>> Good news, though! python-dev is in favor of adding calloc() to the >>>> core allocation interfaces, which will let numpy join the party. See >>>> python-dev thread: >>>> https://mail.python.org/pipermail/python-dev/2014-April/133985.html >>>> >>>> It would be especially nice if we could get this into 3.5, since it >>>> seems likely that lots of numpy users will be switching to 3.5 when it >>>> comes out, and having a good memory tracing infrastructure there >>>> waiting for them make it even more awesome. >>>> >>>> Anyone interested in picking this up? >>>> http://bugs.python.org/issue21233 >>> >>> Hi, >>> I think it would be a better idea to instead of API functions for one >>> different type of allocator we get access to use the python hooks >>> directly with whatever allocator we want to use. >> >> Unfortunately, that's not how the API works. The way that third-party >> tracers register a 'hook' is by providing a new implementation of >> malloc/free/etc. So there's no general way to say "please pretend to >> have done a malloc". >> >> I guess we could potentially request the addition of >> fake_malloc/fake_free functions. > > Unfortunate, looking at the pep it seems either you have a custom > allocator or you have tracing but not both (unless you trace > yourself). > This seems like quite a limitation. I don't think this is right - notice the PyMem_GetAllocator function, which lets you grab the old allocator. This means you can write a tracing "allocator" which just does its tracing and then delegates to the old allocator. (And looking at _tracemalloc.c this does seem to be how it works.) This means that any full allocator replacement has to be enabled first before any tracing allocator is enabled, but that's okay, because a full allocator has to be inserted *very* early in any case (like, before any allocations have happened) and can never be removed, so this doesn't seem so bad. OTOH I don't think they've really thought about the case of stacking multiple tracing allocators. tracemalloc.stop() just unconditionally resets the allocator to whatever it was when tracemalloc.start() was called, and there's no guidelines on how to handle the lifetime of the ctx pointer. I'm not sure these issues cause any problems in practice though. > Maybe it would have been more flexible if instead python provided > three functions: > > PyMem_RegisterAlloc(size); > PyMem_RegisterReAlloc(size); > PyMem_RegisterFree(size); > + possibly nogil variantes > These functions call into registered tracing functions (registered > e.g. by tracemalloc.start()) or do nothing. > > Our allocator (and pythons) then just always calls these functions and > continues doing its stuff. You'd need to add some void* arguments as well -- tracemalloc actually tracks every allocation independently, so you can do things like ask "which line of code was responsible for allocating the largest portion of the memory that is still in use". And unfortunately once you add these arguments the resulting signatures don't quite match regular malloc/realloc/free (you have to pass a void* into malloc instead of receiving one), so we can't just define a PYMEM_NULL domain. (Or rather, we could, but then it would have to return an opaque void* used only for memory tracking, and we'd have to keep track of this alongside every allocation we did, and that would suck.) >>> This would allow as to for example use aligned memory allocators which >>> might be relevant for the new cpu instruction sets with up to 64 byte >>> wide registers >> >> I think we might have had this conversation before, but I don't >> remember how it went... did you have some explanation about how this >> could matter in principle? We have to write code to handle unaligned >> (or imperfectly aligned) arrays regardless, so aligned allocation >> doesn't affect maintainability. And regarding speed, I can't see how >> an extra instruction here or there could make a big difference on >> small arrays, since the effect should be overwhelmed by interpreter >> overhead and memory stalls (not much time for prefetch to come into >> play on small arrays), but OTOH large arrays are usually page-aligned >> in practice, and if not then any extra start-up overhead will be >> amortized out by their size. > > yes we already had this conversation :) > if you have two or more arrays not aligned the same way you can only > align one of them via peeling, the others will always have to be > accessed unaligned. > But it probably does not matter anymore with newer cpus, I should > probably just throw out my old core2 where it does :) Oh right! Yes, that makes sense, sorry :-) On the one hand it would be nice to actually know whether posix_memalign is important, before making api decisions on this basis. OTOH we've made it this far without, and apparently the processors for which it might or might not matter won't be out for some time, so we could revisit things then I guess... Anyone know how picky ARM NEON is about alignment? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Apr 15 14:10:52 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 15 Apr 2014 14:10:52 -0400 Subject: [Numpy-discussion] Wiki page for building numerical stuff onWindows In-Reply-To: <53498762.84d50e0a.638a.ffffec65@mx.google.com> References: <53498762.84d50e0a.638a.ffffec65@mx.google.com> Message-ID: Quick question on this: has anyone looked at the Open Watcom compiler? There was a lighting talk at Pycon about a guy who's been working on getting Python itself to build on Windows with this compiler. I don't know if it might help in all this, if nothing else it mitght be good to have. This was the fellow who presented (and his Twitter handle makes me think he might be interested in this conversation too, I didn't have a chance to catch him, unfortunately): https://twitter.com/fortranjeff Cheers, f -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Tue Apr 15 14:14:14 2014 From: aron at ahmadia.net (Aron Ahmadia) Date: Tue, 15 Apr 2014 14:14:14 -0400 Subject: [Numpy-discussion] Wiki page for building numerical stuff onWindows In-Reply-To: References: <53498762.84d50e0a.638a.ffffec65@mx.google.com> Message-ID: Fernando is referring to this: http://lightningpython.org/ I have to admit that as a Pythonist supporting Windows builds, Cygwin and/or MinGW have always been the most appealing because so much of the rest of the numerical computing infrastructure is primarily supported on UNIXy platforms. A On Tue, Apr 15, 2014 at 2:10 PM, Fernando Perez wrote: > > Quick question on this: has anyone looked at the Open Watcom compiler? > There was a lighting talk at Pycon about a guy who's been working on > getting Python itself to build on Windows with this compiler. I don't know > if it might help in all this, if nothing else it mitght be good to have. > This was the fellow who presented (and his Twitter handle makes me think he > might be interested in this conversation too, I didn't have a chance to > catch him, unfortunately): > > https://twitter.com/fortranjeff > > Cheers, > > f > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Apr 15 14:22:12 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Apr 2014 11:22:12 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff onWindows In-Reply-To: References: <53498762.84d50e0a.638a.ffffec65@mx.google.com> Message-ID: Hi, On Tue, Apr 15, 2014 at 11:10 AM, Fernando Perez wrote: > > Quick question on this: has anyone looked at the Open Watcom compiler? There > was a lighting talk at Pycon about a guy who's been working on getting > Python itself to build on Windows with this compiler. I don't know if it > might help in all this, if nothing else it mitght be good to have. This was > the fellow who presented (and his Twitter handle makes me think he might be > interested in this conversation too, I didn't have a chance to catch him, > unfortunately): I didn't list openwatcom in the Fortran section because it doesn't do 64-bit - as far as I can see: http://www.openwatcom.org/index.php/About_Open_Watcom That's a real killer, because it seems that about 90 percent of Windows users are on 64-bit systems, and I suppose some large proportion of those will download the Python 64-bit installer and then wonder why we don't support it for numpy. Cheers, Matthew From matthew.brett at gmail.com Tue Apr 15 14:26:09 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Apr 2014 11:26:09 -0700 Subject: [Numpy-discussion] Wiki page for building numerical stuff onWindows In-Reply-To: References: <53498762.84d50e0a.638a.ffffec65@mx.google.com> Message-ID: Hi, On Tue, Apr 15, 2014 at 11:14 AM, Aron Ahmadia wrote: > Fernando is referring to this: http://lightningpython.org/ > > I have to admit that as a Pythonist supporting Windows builds, Cygwin and/or > MinGW have always been the most appealing because so much of the rest of the > numerical computing infrastructure is primarily supported on UNIXy > platforms. Yes, absolutely. Given we have so few developers on Windows, the releases are likely going to be largely supported by Unixey people. Likely then, that a Unixey Windows build will get much more care than a Windows-specific one. I tried to say this very briefly on the page https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows#introduction but it needs a bit more explanation and detail. Please feel free to edit, it can certainly be improved. Cheers, Matthew From jtaylor.debian at googlemail.com Tue Apr 15 14:27:52 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 15 Apr 2014 20:27:52 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: <534D7A28.4040404@googlemail.com> On 15.04.2014 18:39, Nathaniel Smith wrote: > On Tue, Apr 15, 2014 at 4:08 PM, Julian Taylor > > > wrote: >> On Tue, Apr 15, 2014 at 3:07 PM, Nathaniel Smith > wrote: >>> On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor >>> > wrote: >>>>> Good news, though! python-dev is in favor of adding calloc() to the >>>>> core allocation interfaces, which will let numpy join the party. See >>>>> python-dev thread: >>>>> https://mail.python.org/pipermail/python-dev/2014-April/133985.html >>>>> >>>>> It would be especially nice if we could get this into 3.5, since it >>>>> seems likely that lots of numpy users will be switching to 3.5 when it >>>>> comes out, and having a good memory tracing infrastructure there >>>>> waiting for them make it even more awesome. >>>>> >>>>> Anyone interested in picking this up? >>>>> http://bugs.python.org/issue21233 >>>> >>>> Hi, >>>> I think it would be a better idea to instead of API functions for one >>>> different type of allocator we get access to use the python hooks >>>> directly with whatever allocator we want to use. >>> >>> Unfortunately, that's not how the API works. The way that third-party >>> tracers register a 'hook' is by providing a new implementation of >>> malloc/free/etc. So there's no general way to say "please pretend to >>> have done a malloc". >>> >>> I guess we could potentially request the addition of >>> fake_malloc/fake_free functions. >> >> Unfortunate, looking at the pep it seems either you have a custom >> allocator or you have tracing but not both (unless you trace >> yourself). >> This seems like quite a limitation. > > I don't think this is right - notice the PyMem_GetAllocator function, > which lets you grab the old allocator. This means you can write a > tracing "allocator" which just does its tracing and then delegates to > the old allocator. (And looking at _tracemalloc.c this does seem to be > how it works.) This means that any full allocator replacement has to be > enabled first before any tracing allocator is enabled, but that's okay, > because a full allocator has to be inserted *very* early in any case > (like, before any allocations have happened) and can never be removed, > so this doesn't seem so bad. > > OTOH I don't think they've really thought about the case of stacking > multiple tracing allocators. tracemalloc.stop() just unconditionally > resets the allocator to whatever it was when tracemalloc.start() was > called, and there's no guidelines on how to handle the lifetime of the > ctx pointer. I'm not sure these issues cause any problems in practice > though. > >> Maybe it would have been more flexible if instead python provided >> three functions: >> >> PyMem_RegisterAlloc(size); >> PyMem_RegisterReAlloc(size); >> PyMem_RegisterFree(size); >> + possibly nogil variantes >> These functions call into registered tracing functions (registered >> e.g. by tracemalloc.start()) or do nothing. >> >> Our allocator (and pythons) then just always calls these functions and >> continues doing its stuff. > > You'd need to add some void* arguments as well -- tracemalloc actually > tracks every allocation independently, so you can do things like ask > "which line of code was responsible for allocating the largest portion > of the memory that is still in use". > > And unfortunately once you add these arguments the resulting signatures > don't quite match regular malloc/realloc/free (you have to pass a void* > into malloc instead of receiving one), so we can't just define a > PYMEM_NULL domain. (Or rather, we could, but then it would have to > return an opaque void* used only for memory tracking, and we'd have to > keep track of this alongside every allocation we did, and that would suck.) > the tracing could register the context at the same time it registers its tracing functions and then retrieve it from python when they need it. I assume you currently can only have one custom allocator per interpreter instance so a global context for tracing should not be a big issue. From jtaylor.debian at googlemail.com Tue Apr 15 14:49:28 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 15 Apr 2014 20:49:28 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: <534D7F38.4050003@googlemail.com> On 15.04.2014 18:39, Nathaniel Smith wrote: > On Tue, Apr 15, 2014 at 4:08 PM, Julian Taylor > > > wrote: >> On Tue, Apr 15, 2014 at 3:07 PM, Nathaniel Smith > wrote: >>> On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor >>> > wrote: >>>>> Good news, though! python-dev is in favor of adding calloc() to the >>>>> core allocation interfaces, which will let numpy join the party. See >>>>> python-dev thread: >>>>> https://mail.python.org/pipermail/python-dev/2014-April/133985.html >>>>> >>>>> It would be especially nice if we could get this into 3.5, since it >>>>> seems likely that lots of numpy users will be switching to 3.5 when it >>>>> comes out, and having a good memory tracing infrastructure there >>>>> waiting for them make it even more awesome. >>>>> >>>>> Anyone interested in picking this up? >>>>> http://bugs.python.org/issue21233 >>>> >>>> Hi, >>>> I think it would be a better idea to instead of API functions for one >>>> different type of allocator we get access to use the python hooks >>>> directly with whatever allocator we want to use. >>> >>> Unfortunately, that's not how the API works. The way that third-party >>> tracers register a 'hook' is by providing a new implementation of >>> malloc/free/etc. So there's no general way to say "please pretend to >>> have done a malloc". >>> >>> I guess we could potentially request the addition of >>> fake_malloc/fake_free functions. >> >> Unfortunate, looking at the pep it seems either you have a custom >> allocator or you have tracing but not both (unless you trace >> yourself). >> This seems like quite a limitation. > > I don't think this is right - notice the PyMem_GetAllocator function, > which lets you grab the old allocator. This means you can write a > tracing "allocator" which just does its tracing and then delegates to > the old allocator. (And looking at _tracemalloc.c this does seem to be > how it works.) This means that any full allocator replacement has to be > enabled first before any tracing allocator is enabled, but that's okay, > because a full allocator has to be inserted *very* early in any case > (like, before any allocations have happened) and can never be removed, > so this doesn't seem so bad. > Hm right, so we don't need python to provide calloc, we just rely on tracemalloc using our custom allocator after its done its tracing. But in this approach on loading numpy we would replace the allocator of all modules imported in the same process and any other module that changes the allocator would also change the allocator numpy uses. That sounds dangerous. From ralf.gommers at gmail.com Tue Apr 15 15:03:17 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 15 Apr 2014 21:03:17 +0200 Subject: [Numpy-discussion] EuroSciPy 2014 abstract submission deadline extended Message-ID: Hi, In response to a number of requests, the organizing committee of the EuroSciPy 2014 conference has extended the deadline for abstract submission by 12 days, to Sunday April 27th 2014, 23:59:59 UTC. Up to then, new abstracts may be submitted on https://www.euroscipy.org/2014/ . We are very much looking forward to your submissions to the conference. EuroSciPy 2014 is the annual European conference for scientists using Python. It will be held August 27-30 2014 in Cambridge, UK. This year promises to be an exciting event again. The number of abstract submissions is already above the level of the 2013 edition, and the first keynote has been announced: Steven G. Johnson (MIT), Crossing Language Barriers with Julia, Scipy, and IPython Questions regarding abstract submission may be addressed to euroscipy -org at python.org Best regards, Ralf Gommers (Program Chair) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Tue Apr 15 17:30:02 2014 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Tue, 15 Apr 2014 17:30:02 -0400 Subject: [Numpy-discussion] Universal functions and introspection Message-ID: Hi, Unfortunately, numpy ufuncs are incompatible with the inspect module. While ufunc.nin and ufunc.nout provide some information on the signature, I don't see how to retrieve argument names. What is the preferred/best way to get argument names of a numpy ufuncs? Best, Sylvain -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Tue Apr 15 18:00:56 2014 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Tue, 15 Apr 2014 18:00:56 -0400 Subject: [Numpy-discussion] Generalized universal functions returning higher-dimensional arrays. Message-ID: Hello all, numpy.vectorize is a very convenient way to build universal function from scalar functions, and handle broadcasting... A use case that is not handled is the generalization to functions already returning nd array: For example: If foo(x,y) is a scalar function returning a array of dim k, it would be convenient to create a vectorized version of it, such that if the broadcasted shape of x and y has dimension d, then foo(x,y) returns a (d+k)-dimensional array. For example: # Example 1 x = array([1,2,3]) y = array([4,5,6]) # foo(x,y)[i] is equal to foo(x[i], y[i]), a vector of dim k, # Example 2 x = array([1,2,3]) y = array([4,5,6]) # foo(x,y[:, newaxis])[i,j] is equal to foo(x[i], y[j]), a vector of dim k, Does any one know if there is already a way to do this? Best, Sylvain -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Apr 16 02:55:52 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 16 Apr 2014 08:55:52 +0200 Subject: [Numpy-discussion] Universal functions and introspection In-Reply-To: References: Message-ID: I don't think the names exist currently except as mnemonics in the docstring. Patches to improve 'inspect' support would be welcome of course. -n On Tue, Apr 15, 2014 at 11:30 PM, Sylvain Corlay wrote: > Hi, > Unfortunately, numpy ufuncs are incompatible with the inspect module. > While ufunc.nin and ufunc.nout provide some information on the signature, I > don't see how to retrieve argument names. What is the preferred/best way to > get argument names of a numpy ufuncs? > Best, > Sylvain > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From rhattersley at gmail.com Wed Apr 16 06:03:20 2014 From: rhattersley at gmail.com (R Hattersley) Date: Wed, 16 Apr 2014 11:03:20 +0100 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On 15 April 2014 10:48, Nathaniel Smith wrote: > Anyone interested in picking this up? > http://bugs.python.org/issue21233 > Yes, I'll give it a go. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhattersley at gmail.com Wed Apr 16 11:17:54 2014 From: rhattersley at gmail.com (R Hattersley) Date: Wed, 16 Apr 2014 16:17:54 +0100 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: For some reason the Python issue 21223 didn't show any activity until I logged in to post my patch. At which point I saw that haypo had already submitted pretty much exactly the same patch. *sigh* That was pretty much a waste of time then. :-| -------------- next part -------------- An HTML attachment was scrubbed... URL: From onefire.myself at gmail.com Wed Apr 16 14:26:37 2014 From: onefire.myself at gmail.com (onefire) Date: Wed, 16 Apr 2014 14:26:37 -0400 Subject: [Numpy-discussion] About the npz format Message-ID: Hi all, I have been playing with the idea of using Numpy's binary format as a lightweight alternative to HDF5 (which I believe is the "right" way to do if one does not have a problem with the dependency). I am pretty happy with the npy format, but the npz format seems to be broken as far as performance is concerned (or I am missing obvious!). The following ipython session illustrates the issue: ln [1]: import numpy as np In [2]: x = np.linspace(1, 10, 50000000) In [3]: %time np.save("x.npy", x) CPU times: user 40 ms, sys: 230 ms, total: 270 ms Wall time: 488 ms In [4]: %time np.savez("x.npz", data = x) CPU times: user 657 ms, sys: 707 ms, total: 1.36 s Wall time: 7.7 s I can inspect the files to verify that they contain the same data, and I can change the example, but this seems to always hold (I am running Arch Linux, but I've done the test on other machines too): for bigger arrays, the npz format seems to add an unbelievable amount of overhead. Looking at Numpy's code, it looks like the real work is being done by Python's zipfile module, and I suspect that all the extra time is spent computing the crc32. Am I correct in my assumption (I am not familiar with zipfile's internals)? Or perhaps I am doing something really dumb and there is an easy way to speed things up? Assuming that I am correct, my next question is: why compute the crc32 at all? I mean, I know that it is part of what defines a "zip file", but is it really necessary for a npz file to be a (compliant) zip file? If, for example, I open the resulting npz file with a hex editor, and insert a bogus crc32, np.load will happily load the file anyway (Gnome's Archive Manager will do the same) To me this suggests that the fact that npz files are zip files is not that important. . Perhaps, people think that the ability to browse arrays and extract individual ones like they would do with a regular zip file is really important, but reading the little documentation that I found, I got the impression that npz files are zip files just because this was the easiest way to have multiple arrays in the same file. But my main point is: it should be fairly simple to make npz files much more efficient with simple changes like not computing checksums (or using a different algorithm like adler32). Let me know what you think about this. I've searched around the internet, and on places like Stackoverflow, it seems that the standard answer is: you are doing it wrong, forget Numpy's format and start using hdf5! Please do not give that answer. Like I said in the beginning, I am well aware of hdf5 and I use it on my "production code" (on C++). But I believe that there should be a lightweight alternative (right now, to use hdf5 I need to have installed the C library, the C++ wrappers, and the h5py library to play with the data using Python, that is a bit too heavy for my needs). I really like Numpy's format (if anything, it makes me feel better knowing that it is so easy to reverse engineer it, while the hdf5 format is very complicated), but the (apparent) poor performance of npz files if a deal breaker. Gilberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaylenexue at yahoo.com Wed Apr 16 15:23:31 2014 From: jaylenexue at yahoo.com (jaylene) Date: Wed, 16 Apr 2014 12:23:31 -0700 (PDT) Subject: [Numpy-discussion] ImportError: /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString Message-ID: <1397676211880-37372.post@n7.nabble.com> Hi, I rebuilt the python with -enabled_shared to resolve the following error: /usr/bin/ld: /usr/local/lib/libpython2.7.a(cobject.o): relocation R_X86_64_32S against `PyCObject_Type' can not be used when making a shared object; recompile with -fPIC /usr/local/lib/libpython2.7.a: could not read symbols: Bad value However, there is a new error with this rebuilt python "ImportError: /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString". Does anyone know how to fix this error? Thanks! -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/ImportError-usr-local-lib-python2-7-site-packages-numpy-1-8-0-py2-7-linux-x86-64-egg-numpy-core-multg-tp37372.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From fperez.net at gmail.com Wed Apr 16 15:50:12 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 16 Apr 2014 12:50:12 -0700 Subject: [Numpy-discussion] Universal functions and introspection In-Reply-To: References: Message-ID: Does argument clinic work with python2 or is it python3 only? http://legacy.python.org/dev/peps/pep-0436/ f On Tue, Apr 15, 2014 at 11:55 PM, Nathaniel Smith wrote: > I don't think the names exist currently except as mnemonics in the > docstring. Patches to improve 'inspect' support would be welcome of > course. > > -n > > On Tue, Apr 15, 2014 at 11:30 PM, Sylvain Corlay > wrote: > > Hi, > > Unfortunately, numpy ufuncs are incompatible with the inspect module. > > While ufunc.nin and ufunc.nout provide some information on the > signature, I > > don't see how to retrieve argument names. What is the preferred/best way > to > > get argument names of a numpy ufuncs? > > Best, > > Sylvain > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Apr 16 16:48:53 2014 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 16 Apr 2014 16:48:53 -0400 Subject: [Numpy-discussion] ImportError: /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString In-Reply-To: <1397676211880-37372.post@n7.nabble.com> References: <1397676211880-37372.post@n7.nabble.com> Message-ID: Did you rebuild numpy after rebuilding python? Note that if that works, then you will likely need to recompile the entire scipy stack (matplotlib, scipy, etc.). Ben Root On Wed, Apr 16, 2014 at 3:23 PM, jaylene wrote: > Hi, > > I rebuilt the python with -enabled_shared to resolve the following error: > /usr/bin/ld: /usr/local/lib/libpython2.7.a(cobject.o): relocation > R_X86_64_32S against `PyCObject_Type' can not be used when making a > shared object; recompile with -fPIC > /usr/local/lib/libpython2.7.a: could not read symbols: Bad value > > However, there is a new error with this rebuilt python "ImportError: > > /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: > undefined symbol: PyUnicodeUCS2_AsASCIIString". > > Does anyone know how to fix this error? > > Thanks! > > > > -- > View this message in context: > http://numpy-discussion.10968.n7.nabble.com/ImportError-usr-local-lib-python2-7-site-packages-numpy-1-8-0-py2-7-linux-x86-64-egg-numpy-core-multg-tp37372.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From valentin at haenel.co Wed Apr 16 16:57:30 2014 From: valentin at haenel.co (Valentin Haenel) Date: Wed, 16 Apr 2014 22:57:30 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: <20140416205730.GF2085@kudu.in-berlin.de> Hi Gilberto, * onefire [2014-04-16]: > I have been playing with the idea of using Numpy's binary format as a > lightweight alternative to HDF5 (which I believe is the "right" way to do > if one does not have a problem with the dependency). > > I am pretty happy with the npy format, but the npz format seems to be > broken as far as performance is concerned (or I am missing obvious!). The > following ipython session illustrates the issue: > > ln [1]: import numpy as np > > In [2]: x = np.linspace(1, 10, 50000000) > > In [3]: %time np.save("x.npy", x) > CPU times: user 40 ms, sys: 230 ms, total: 270 ms > Wall time: 488 ms > > In [4]: %time np.savez("x.npz", data = x) > CPU times: user 657 ms, sys: 707 ms, total: 1.36 s > Wall time: 7.7 s If it just serialization speed, You may want to look at Bloscpack: https://github.com/Blosc/Bloscpack Which only has blosc/python-blosc and Numpy as a dependency. You can use it on Numpy arrays like so: https://github.com/Blosc/Bloscpack#numpy (thats instructions for master you are looking at) And it can certainly be faster than NPZ and sometimes faster than NPY -- depending of course on your system and the type of data -- and also more lightweight than HDF5. I wrote an article about it with some benchmarks, also vs NPY/NPZ here: https://github.com/euroscipy/euroscipy_proceedings/tree/master/papers/23_haenel Since it is not yet officially published, you can find a compiled PDF draft I just made at: http://fldmp.zetatech.org/haenel_bloscpack_euroscipy2013_ac25c19cb6.pdf Perhaps it is interesting for you. > I can inspect the files to verify that they contain the same data, and I > can change the example, but this seems to always hold (I am running Arch > Linux, but I've done the test on other machines too): for bigger arrays, > the npz format seems to add an unbelievable amount of overhead. You mean time or space wise? In my experience NPZ is fairly slow but can yield some good compression rations, depending on the LZ-complexity of the input data. In fact, AFAIK, NPZ uses the DEFLATE algorithm as implemented by ZLIB which is fairly slow and not optimized for compression decompression speed. FYI: if you really want ZLIB, Blosc also supports using it internally, which is nice. > Looking at Numpy's code, it looks like the real work is being done by > Python's zipfile module, and I suspect that all the extra time is spent > computing the crc32. Am I correct in my assumption (I am not familiar with > zipfile's internals)? Or perhaps I am doing something really dumb and there > is an easy way to speed things up? I am guessing here, but a checksum *should* be fairly fast. I would guess it is at least in part due to use of DEFLATE. > Assuming that I am correct, my next question is: why compute the crc32 at > all? I mean, I know that it is part of what defines a "zip file", but is it > really necessary for a npz file to be a (compliant) zip file? If, for > example, I open the resulting npz file with a hex editor, and insert a > bogus crc32, np.load will happily load the file anyway (Gnome's Archive > Manager will do the same) To me this suggests that the fact that npz files > are zip files is not that important. . Well, the good news here is that Bloscpack supports adding checksums to secure the integrity of the compressed data. You can choose between many, including CRC32, ADLER32 and even sha512. > Perhaps, people think that the ability to browse arrays and extract > individual ones like they would do with a regular zip file is really > important, but reading the little documentation that I found, I got the > impression that npz files are zip files just because this was the easiest > way to have multiple arrays in the same file. But my main point is: it > should be fairly simple to make npz files much more efficient with simple > changes like not computing checksums (or using a different algorithm like > adler32) Ah, so you want to store multiple arrays in a single file. I must disappoint you there, Bloscpack doesn't support that right now. Although it is in principle possible to achieve this. > Let me know what you think about this. I've searched around the internet, > and on places like Stackoverflow, it seems that the standard answer is: you > are doing it wrong, forget Numpy's format and start using hdf5! Please do > not give that answer. Like I said in the beginning, I am well aware of > hdf5 and I use it on my "production code" (on C++). But I believe that > there should be a lightweight alternative (right now, to use hdf5 I need to > have installed the C library, the C++ wrappers, and the h5py library to > play with the data using Python, that is a bit too heavy for my needs). I > really like Numpy's format (if anything, it makes me feel better knowing > that it is > so easy to reverse engineer it, while the hdf5 format is very complicated), > but the (apparent) poor performance of npz files if a deal breaker. Well, I hope that Bloscpack is lightweight enough for you. As I said the only dependency is blosc/python-blosc which can be compiled using a C compiler (C++ if you want all the additional codecs) and the Python headers. Hope it helps and let me know what you think! V- From ndarray at mac.com Wed Apr 16 17:03:02 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Wed, 16 Apr 2014 17:03:02 -0400 Subject: [Numpy-discussion] Universal functions and introspection In-Reply-To: References: Message-ID: On Wed, Apr 16, 2014 at 3:50 PM, Fernando Perez wrote: > Does argument clinic work with python2 or is it python3 only? > > http://legacy.python.org/dev/peps/pep-0436/ > > It is python3 only, but is should not be hard to adapt it to generate 2/3 compatible code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Apr 16 17:03:55 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 16 Apr 2014 23:03:55 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: crc32 extremely fast, and I think zip might use adler32 instead which is even faster. OTOH compression is incredibly slow, unless you're using one of the 'just a little bit of compression' formats like blosc or lzo1. If your npz files are compressed then this is certainly the culprit. The zip format supports storing files without compression. Maybe what you want is an option to use this with .npz? -n On 16 Apr 2014 20:26, "onefire" wrote: > Hi all, > > I have been playing with the idea of using Numpy's binary format as a > lightweight alternative to HDF5 (which I believe is the "right" way to do > if one does not have a problem with the dependency). > > I am pretty happy with the npy format, but the npz format seems to be > broken as far as performance is concerned (or I am missing obvious!). The > following ipython session illustrates the issue: > > ln [1]: import numpy as np > > In [2]: x = np.linspace(1, 10, 50000000) > > In [3]: %time np.save("x.npy", x) > CPU times: user 40 ms, sys: 230 ms, total: 270 ms > Wall time: 488 ms > > In [4]: %time np.savez("x.npz", data = x) > CPU times: user 657 ms, sys: 707 ms, total: 1.36 s > Wall time: 7.7 s > > I can inspect the files to verify that they contain the same data, and I > can change the example, but this seems to always hold (I am running Arch > Linux, but I've done the test on other machines too): for bigger arrays, > the npz format seems to add an unbelievable amount of overhead. > > Looking at Numpy's code, it looks like the real work is being done by > Python's zipfile module, and I suspect that all the extra time is spent > computing the crc32. Am I correct in my assumption (I am not familiar with > zipfile's internals)? Or perhaps I am doing something really dumb and there > is an easy way to speed things up? > > Assuming that I am correct, my next question is: why compute the crc32 at > all? I mean, I know that it is part of what defines a "zip file", but is it > really necessary for a npz file to be a (compliant) zip file? If, for > example, I open the resulting npz file with a hex editor, and insert a > bogus crc32, np.load will happily load the file anyway (Gnome's Archive > Manager will do the same) To me this suggests that the fact that npz files > are zip files is not that important. . > > Perhaps, people think that the ability to browse arrays and extract > individual ones like they would do with a regular zip file is really > important, but reading the little documentation that I found, I got the > impression that npz files are zip files just because this was the easiest > way to have multiple arrays in the same file. But my main point is: it > should be fairly simple to make npz files much more efficient with simple > changes like not computing checksums (or using a different algorithm like > adler32). > > Let me know what you think about this. I've searched around the internet, > and on places like Stackoverflow, it seems that the standard answer is: you > are doing it wrong, forget Numpy's format and start using hdf5! Please do > not give that answer. Like I said in the beginning, I am well aware of > hdf5 and I use it on my "production code" (on C++). But I believe that > there should be a lightweight alternative (right now, to use hdf5 I need to > have installed the C library, the C++ wrappers, and the h5py library to > play with the data using Python, that is a bit too heavy for my needs). I > really like Numpy's format (if anything, it makes me feel better knowing > that it is > so easy to reverse engineer it, while the hdf5 format is very > complicated), but the (apparent) poor performance of npz files if a deal > breaker. > > Gilberto > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Wed Apr 16 17:39:07 2014 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Wed, 16 Apr 2014 17:39:07 -0400 Subject: [Numpy-discussion] Universal functions and introspection In-Reply-To: References: Message-ID: Besides, the result of numpy.vectorize don't have 'nin' and 'nout' members. Cheers, On Wed, Apr 16, 2014 at 2:55 AM, Nathaniel Smith wrote: > I don't think the names exist currently except as mnemonics in the > docstring. Patches to improve 'inspect' support would be welcome of > course. > > -n > > On Tue, Apr 15, 2014 at 11:30 PM, Sylvain Corlay > wrote: > > Hi, > > Unfortunately, numpy ufuncs are incompatible with the inspect module. > > While ufunc.nin and ufunc.nout provide some information on the > signature, I > > don't see how to retrieve argument names. What is the preferred/best way > to > > get argument names of a numpy ufuncs? > > Best, > > Sylvain > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From onefire.myself at gmail.com Wed Apr 16 20:57:36 2014 From: onefire.myself at gmail.com (onefire) Date: Wed, 16 Apr 2014 20:57:36 -0400 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: Valentin Haenel, Bloscpack definitely looks interesting but I need to take a careful look first. I will let you know if I like it. Thanks for the suggestion! I think you and Nathaniel Smith misunderstood my questions (my fault, since I did not explain myself well!). First, Numpy's savez will not do any compression by default. It will simply store the npy file normally. The documentation suggests so and I can open the resulting file to confirm it. Also, if you run the commands that I specified in my previous post, you can see that the resulting files have sizes 400000080 (x.npy) and 400000194 (x.npz). The npy header takes 80 bytes (it actually needs less than that, but it is padded to be divisible by 16). The npz file that saves the same array takes 114 extra bytes (for the zip file metadata), so the space overhead is pretty small. What I cannot understand is why savez takes more than 10 times longer than saving the data to a npy file. The only reason that I could come up with was the computation of the crc32. BUT it might be more than this... This afternoon I found out about this Julia package ( https://github.com/fhs/NPZ.jl) to manipulate Numpy files. I did a few tests and it seems to work correctly. It becomes interesting when I do the npy-npz comparison using Julia. Here is the code that I used: using NPZ function write_npy(x) tic() npzwrite("data.npy", x) toc() end function write_npz(x) tic() npzwrite("data.npz", (ASCIIString => Any)["data" => x]) toc() end x = linspace(1, 10, 50000000) write_npy(x) # this prints: elapsed time: 0.417742163 seconds write_npz(x) # this prints: elapsed time: 0.882226675 seconds The Julia timings (tested with Julia 0.3) are closer to what I would expect. Notice that the time to save the npy file is very similar to the one that I got with Numpy's save function (see my previous post), but the "npz overhead" only adds half a second. So now I think there are two things going on: 1) It is wasteful to compute the crc32. At a minimum I would like to either have the option to choose a different, faster checksum (like adler32) or to turn that off (I prefer the second option, because if I am worried about the integrity of the data, I will likely compute the sha512sum of the entire file anyway). 2) The Python implementation is inefficient (to be honest, I just found out about the Julia package and I cannot guarantee anything about its quality, but if I compute a crc32 from 0.5 GB of data from C code, it does takes less than a second!). My guess is that the problem is in the zip module, but like I said before, I do not know the details of what it is doing. Let me know what you think. Gilberto On Wed, Apr 16, 2014 at 5:03 PM, Nathaniel Smith wrote: > crc32 extremely fast, and I think zip might use adler32 instead which is > even faster. OTOH compression is incredibly slow, unless you're using one > of the 'just a little bit of compression' formats like blosc or lzo1. If > your npz files are compressed then this is certainly the culprit. > > The zip format supports storing files without compression. Maybe what you > want is an option to use this with .npz? > > -n > On 16 Apr 2014 20:26, "onefire" wrote: > >> Hi all, >> >> I have been playing with the idea of using Numpy's binary format as a >> lightweight alternative to HDF5 (which I believe is the "right" way to do >> if one does not have a problem with the dependency). >> >> I am pretty happy with the npy format, but the npz format seems to be >> broken as far as performance is concerned (or I am missing obvious!). The >> following ipython session illustrates the issue: >> >> ln [1]: import numpy as np >> >> In [2]: x = np.linspace(1, 10, 50000000) >> >> In [3]: %time np.save("x.npy", x) >> CPU times: user 40 ms, sys: 230 ms, total: 270 ms >> Wall time: 488 ms >> >> In [4]: %time np.savez("x.npz", data = x) >> CPU times: user 657 ms, sys: 707 ms, total: 1.36 s >> Wall time: 7.7 s >> >> I can inspect the files to verify that they contain the same data, and I >> can change the example, but this seems to always hold (I am running Arch >> Linux, but I've done the test on other machines too): for bigger arrays, >> the npz format seems to add an unbelievable amount of overhead. >> >> Looking at Numpy's code, it looks like the real work is being done by >> Python's zipfile module, and I suspect that all the extra time is spent >> computing the crc32. Am I correct in my assumption (I am not familiar with >> zipfile's internals)? Or perhaps I am doing something really dumb and there >> is an easy way to speed things up? >> >> Assuming that I am correct, my next question is: why compute the crc32 at >> all? I mean, I know that it is part of what defines a "zip file", but is it >> really necessary for a npz file to be a (compliant) zip file? If, for >> example, I open the resulting npz file with a hex editor, and insert a >> bogus crc32, np.load will happily load the file anyway (Gnome's Archive >> Manager will do the same) To me this suggests that the fact that npz files >> are zip files is not that important. . >> >> Perhaps, people think that the ability to browse arrays and extract >> individual ones like they would do with a regular zip file is really >> important, but reading the little documentation that I found, I got the >> impression that npz files are zip files just because this was the easiest >> way to have multiple arrays in the same file. But my main point is: it >> should be fairly simple to make npz files much more efficient with simple >> changes like not computing checksums (or using a different algorithm like >> adler32). >> >> Let me know what you think about this. I've searched around the internet, >> and on places like Stackoverflow, it seems that the standard answer is: you >> are doing it wrong, forget Numpy's format and start using hdf5! Please do >> not give that answer. Like I said in the beginning, I am well aware of >> hdf5 and I use it on my "production code" (on C++). But I believe that >> there should be a lightweight alternative (right now, to use hdf5 I need to >> have installed the C library, the C++ wrappers, and the h5py library to >> play with the data using Python, that is a bit too heavy for my needs). I >> really like Numpy's format (if anything, it makes me feel better knowing >> that it is >> so easy to reverse engineer it, while the hdf5 format is very >> complicated), but the (apparent) poor performance of npz files if a deal >> breaker. >> >> Gilberto >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpalao.python at gmail.com Thu Apr 17 05:17:37 2014 From: dpalao.python at gmail.com (David Palao) Date: Thu, 17 Apr 2014 11:17:37 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: 2014-04-16 20:26 GMT+02:00 onefire : > Hi all, > > I have been playing with the idea of using Numpy's binary format as a > lightweight alternative to HDF5 (which I believe is the "right" way to do if > one does not have a problem with the dependency). > > I am pretty happy with the npy format, but the npz format seems to be broken > as far as performance is concerned (or I am missing obvious!). The following > ipython session illustrates the issue: > > ln [1]: import numpy as np > > In [2]: x = np.linspace(1, 10, 50000000) > > In [3]: %time np.save("x.npy", x) > CPU times: user 40 ms, sys: 230 ms, total: 270 ms > Wall time: 488 ms > > In [4]: %time np.savez("x.npz", data = x) > CPU times: user 657 ms, sys: 707 ms, total: 1.36 s > Wall time: 7.7 s > Hi, In my case (python-2.7.3, numpy-1.6.1): In [23]: %time save("xx.npy", x) CPU times: user 0.00 s, sys: 0.23 s, total: 0.23 s Wall time: 4.07 s In [24]: %time savez("xx.npz", data = x) CPU times: user 0.42 s, sys: 0.61 s, total: 1.02 s Wall time: 4.26 s In my case I don't see the "unbelievable amount of overhead" of the npz thing. Best From njs at pobox.com Thu Apr 17 05:23:07 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Apr 2014 11:23:07 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: On 17 Apr 2014 01:57, "onefire" wrote: > > What I cannot understand is why savez takes more than 10 times longer than saving the data to a npy file. The only reason that I could come up with was the computation of the crc32. We can all make guesses but the solution is just to profile it :-). %prun in ipython (and then if you need more granularity installing line_profiler is useful). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 17 09:09:57 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Apr 2014 14:09:57 +0100 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On Wed, Apr 16, 2014 at 4:17 PM, R Hattersley wrote: > For some reason the Python issue 21223 didn't show any activity until I > logged in to post my patch. At which point I saw that haypo had already > submitted pretty much exactly the same patch. *sigh* That was pretty much a > waste of time then. :-| Oh, that sucks :-(. I knew that there was a patch posted there, but I was travelling yesterday when you posted :-/. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From jaylenexue at yahoo.com Thu Apr 17 09:30:46 2014 From: jaylenexue at yahoo.com (jaylene) Date: Thu, 17 Apr 2014 06:30:46 -0700 (PDT) Subject: [Numpy-discussion] ImportError: /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString In-Reply-To: References: <1397676211880-37372.post@n7.nabble.com> Message-ID: <1397741446795-37383.post@n7.nabble.com> No. I didn't rebuild numpy after rebuilding python. I searched online about this error. It said that this error might be caused by building python with USC-4. Is there a way to check if the python was built with USC-4 or USC-2? Will rebuilding python with USC-2 work? I'm really reluctant to recompile the entire scipy stack. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/ImportError-usr-local-lib-python2-7-site-packages-numpy-1-8-0-py2-7-linux-x86-64-egg-numpy-core-multg-tp37372p37383.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From aron at ahmadia.net Thu Apr 17 10:08:22 2014 From: aron at ahmadia.net (Aron Ahmadia) Date: Thu, 17 Apr 2014 10:08:22 -0400 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: > On the one hand it would be nice to actually know whether posix_memalign is important, before making api decisions on this basis. FWIW: On the lightweight IBM cores that the extremely popular BlueGene machines were based on, accessing unaligned memory raised system faults. The default behavior of these machines was to terminate the program if more than 1000 such errors occurred on a given process, and an environment variable allowed you to terminate the program if *any* unaligned memory access occurred. This is because unaligned memory accesses were 15x (or more) slower than aligned memory access. The newer /Q chips seem to be a little more forgiving of this, but I think one can in general expect allocated memory alignment to be an important performance technique for future high performance computing architectures. A On Thu, Apr 17, 2014 at 9:09 AM, Nathaniel Smith wrote: > On Wed, Apr 16, 2014 at 4:17 PM, R Hattersley > wrote: > > For some reason the Python issue 21223 didn't show any activity until I > > logged in to post my patch. At which point I saw that haypo had already > > submitted pretty much exactly the same patch. *sigh* That was pretty > much a > > waste of time then. :-| > > Oh, that sucks :-(. I knew that there was a patch posted there, but I > was travelling yesterday when you posted :-/. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 17 10:18:18 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Apr 2014 15:18:18 +0100 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: On 17 Apr 2014 15:09, "Aron Ahmadia" wrote: > > > On the one hand it would be nice to actually know whether posix_memalign is important, before making api decisions on this basis. > > FWIW: On the lightweight IBM cores that the extremely popular BlueGene machines were based on, accessing unaligned memory raised system faults. The default behavior of these machines was to terminate the program if more than 1000 such errors occurred on a given process, and an environment variable allowed you to terminate the program if *any* unaligned memory access occurred. This is because unaligned memory accesses were 15x (or more) slower than aligned memory access. > > The newer /Q chips seem to be a little more forgiving of this, but I think one can in general expect allocated memory alignment to be an important performance technique for future high performance computing architectures. Right, this much is true on lots of architectures, and so malloc is careful to always return values with sufficient alignment (e.g. 8 bytes) to make sure that any standard operation can succeed. The question here is whether it will be important to have *even more* alignment than malloc gives us by default. A 16 or 32 byte wide SIMD instruction might prefer that data have 16 or 32 byte alignment, even if normal memory access for the types being operated on only requires 4 or 8 byte alignment. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Thu Apr 17 10:26:37 2014 From: aron at ahmadia.net (Aron Ahmadia) Date: Thu, 17 Apr 2014 10:26:37 -0400 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: Hmnn, I wasn't being clear :) The default malloc on BlueGene/Q only returns 8 byte alignment, but the SIMD units need 32-byte alignment for loads, stores, and operations or performance suffers. On the /P the required alignment was 16-bytes, but malloc only gave you 8, and trying to perform vectorized loads/stores generated alignment exceptions on unaligned memory. See https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q and https://computing.llnl.gov/tutorials/bgp/BGP-usage.Walkup.pdf (slides 14 for overview, 15 for the effective performance difference between the unaligned/aligned code) for some notes on this. A On Thu, Apr 17, 2014 at 10:18 AM, Nathaniel Smith wrote: > On 17 Apr 2014 15:09, "Aron Ahmadia" wrote: > > > > > On the one hand it would be nice to actually know whether > posix_memalign is important, before making api decisions on this basis. > > > > FWIW: On the lightweight IBM cores that the extremely popular BlueGene > machines were based on, accessing unaligned memory raised system faults. > The default behavior of these machines was to terminate the program if > more than 1000 such errors occurred on a given process, and an environment > variable allowed you to terminate the program if *any* unaligned memory > access occurred. This is because unaligned memory accesses were 15x (or > more) slower than aligned memory access. > > > > The newer /Q chips seem to be a little more forgiving of this, but I > think one can in general expect allocated memory alignment to be an > important performance technique for future high performance computing > architectures. > > Right, this much is true on lots of architectures, and so malloc is > careful to always return values with sufficient alignment (e.g. 8 bytes) to > make sure that any standard operation can succeed. > > The question here is whether it will be important to have *even more* > alignment than malloc gives us by default. A 16 or 32 byte wide SIMD > instruction might prefer that data have 16 or 32 byte alignment, even if > normal memory access for the types being operated on only requires 4 or 8 > byte alignment. > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Thu Apr 17 11:39:22 2014 From: bryanv at continuum.io (Bryan Van de Ven) Date: Thu, 17 Apr 2014 10:39:22 -0500 Subject: [Numpy-discussion] ANN: Bokeh 0.4.4 released Message-ID: <5A50EDB3-4624-43B9-874B-6332242AF579@continuum.io> I am happy to announce the release of Bokeh version 0.4.4! Bokeh is a Python library for visualizing large and realtime datasets on the web. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity to thin clients. Bokeh includes its own Javascript library (BokehJS) that implements a reactive scenegraph representation of the plot, and renders efficiently to HTML5 Canvas. Bokeh works well with IPython Notebook, but can generate standalone graphics that embed into regular HTML. If you are a Matplotlib user, you can just use %bokeh magic to start interacting with your plots in the notebook immediately! Check out the full documentation, interactive gallery, and tutorial at http://bokeh.pydata.org If you are using Anaconda, you can install with conda: conda install bokeh Alternatively, you can install with pip: pip install bokeh We are still working on some bigger features but want to get new fixes and functionality out to users as soon as we can. Some notable features of this release are: * Additional Matplotlib, ggplot, and Seaborn compatibility (styling, more examples) * TravisCI testing integration at https://travis-ci.org/ContinuumIO/bokeh * Tool enhancements, constrained pan/zoom, more hover glyphs * Server remote data and downsampling examples * Initial work for Bokeh "app" concept Also, we've also made lots of little bug fixes and enhancements - see the CHANGELOG for full details. BokehJS is also available by CDN for use in standalone javascript applications: http://cdn.pydata.org/bokeh-0.4.4.js http://cdn.pydata.org/bokeh-0.4.4.css http://cdn.pydata.org/bokeh-0.4.4.min.js http://cdn.pydata.org/bokeh-0.4.4.min.css Some examples of BokehJS use can be found on the Bokeh JSFiddle page: http://jsfiddle.net/user/bokeh/fiddles/ The release of Bokeh 0.5 is planned for early May. Some notable features we plan to include are: * Abstract Rendering for semantically meaningful downsampling of large datasets * Better grid-based layout system, using Cassowary.js * More MPL/Seaborn/ggplot.py compatibility and examples, using MPLExporter * Additional tools, improved interactions, and better plot frame * Touch support Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/continuumio/bokeh Questions can be directed to the Bokeh mailing list: bokeh at continuum.io If you have interest in helping to develop Bokeh, please get involved! Special thanks to recent contributors: Amy Troschinetz and Gerald Dalley Bryan Van de Ven Continuum Analytics http://continuum.io From faltet at gmail.com Thu Apr 17 12:06:56 2014 From: faltet at gmail.com (Francesc Alted) Date: Thu, 17 Apr 2014 18:06:56 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: References: Message-ID: <534FFC20.6010106@gmail.com> Uh, 15x slower for unaligned access is quite a lot. But Intel (and AMD) arquitectures are much more tolerant in this aspect (and improving). For example, with a Xeon(R) CPU E5-2670 (2 years old) I get: In [1]: import numpy as np In [2]: shape = (10000, 10000) In [3]: x_aligned = np.zeros(shape, dtype=[('x',np.float64),('y',np.int64)])['x'] In [4]: x_unaligned = np.zeros(shape, dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] In [5]: %timeit res = x_aligned ** 2 1 loops, best of 3: 289 ms per loop In [6]: %timeit res = x_unaligned ** 2 1 loops, best of 3: 664 ms per loop so the added cost in this case is just a bit more than 2x. But you can also alleviate this overhead if you do a copy that fits in cache prior to do computations. numexpr does this: https://github.com/pydata/numexpr/blob/master/numexpr/interp_body.cpp#L203 and the results are pretty good: In [8]: import numexpr as ne In [9]: %timeit res = ne.evaluate('x_aligned ** 2') 10 loops, best of 3: 133 ms per loop In [10]: %timeit res = ne.evaluate('x_unaligned ** 2') 10 loops, best of 3: 134 ms per loop i.e. there is not a significant difference between aligned and unaligned access to data. I wonder if the same technique could be applied to NumPy. Francesc El 17/04/14 16:26, Aron Ahmadia ha escrit: > Hmnn, I wasn't being clear :) > > The default malloc on BlueGene/Q only returns 8 byte alignment, but > the SIMD units need 32-byte alignment for loads, stores, and > operations or performance suffers. On the /P the required alignment > was 16-bytes, but malloc only gave you 8, and trying to perform > vectorized loads/stores generated alignment exceptions on unaligned > memory. > > See https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q and > https://computing.llnl.gov/tutorials/bgp/BGP-usage.Walkup.pdf (slides > 14 for overview, 15 for the effective performance difference between > the unaligned/aligned code) for some notes on this. > > A > > > > > On Thu, Apr 17, 2014 at 10:18 AM, Nathaniel Smith > wrote: > > On 17 Apr 2014 15:09, "Aron Ahmadia" > wrote: > > > > > On the one hand it would be nice to actually know whether > posix_memalign is important, before making api decisions on this > basis. > > > > FWIW: On the lightweight IBM cores that the extremely popular > BlueGene machines were based on, accessing unaligned memory raised > system faults. The default behavior of these machines was to > terminate the program if more than 1000 such errors occurred on a > given process, and an environment variable allowed you to > terminate the program if *any* unaligned memory access occurred. > This is because unaligned memory accesses were 15x (or more) > slower than aligned memory access. > > > > The newer /Q chips seem to be a little more forgiving of this, > but I think one can in general expect allocated memory alignment > to be an important performance technique for future high > performance computing architectures. > > Right, this much is true on lots of architectures, and so malloc > is careful to always return values with sufficient alignment (e.g. > 8 bytes) to make sure that any standard operation can succeed. > > The question here is whether it will be important to have *even > more* alignment than malloc gives us by default. A 16 or 32 byte > wide SIMD instruction might prefer that data have 16 or 32 byte > alignment, even if normal memory access for the types being > operated on only requires 4 or 8 byte alignment. > > -n > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Thu Apr 17 12:32:42 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 17 Apr 2014 12:32:42 -0400 Subject: [Numpy-discussion] min depth to nonzero in 3d array Message-ID: <5350022A.4060008@gmail.com> Given an array A of shape m x n x n (i.e., a stack of square matrices), I want an n x n array that gives the minimum "depth" to a nonzero element. E.g., the 0,0 element of the result is np.flatnonzero(A[:,0,0])[0] Can this be vectorized? (Assuming a nonzero element exists is ok, but dealing nicely with its absence is even better.) Thanks, Alan Isaac From jtaylor.debian at googlemail.com Thu Apr 17 13:28:10 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 17 Apr 2014 19:28:10 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: <534FFC20.6010106@gmail.com> References: <534FFC20.6010106@gmail.com> Message-ID: <53500F2A.8070105@googlemail.com> On 17.04.2014 18:06, Francesc Alted wrote: > > In [4]: x_unaligned = np.zeros(shape, > dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] on arrays of this size you won't see alignment issues you are dominated by memory bandwidth. If at all you will only see it if the data fits into the cache. Its also about unaligned to simd vectors not unaligned to basic types. But it doesn't matter anymore on modern x86 cpus. I guess for array data cache line splits should also not be a big concern. Aligned allocators are not the only allocator which might be useful in numpy. Modern CPUs also support larger pages than 4K (huge pages up to 1GB in size) which reduces TLB cache misses. Memory of this type typically needs to be allocated with special mmap flags, though newer kernel versions can now also provide this memory to transparent anonymous pages (normal non-file mmaps). > > In [8]: import numexpr as ne > > In [9]: %timeit res = ne.evaluate('x_aligned ** 2') > 10 loops, best of 3: 133 ms per loop > > In [10]: %timeit res = ne.evaluate('x_unaligned ** 2') > 10 loops, best of 3: 134 ms per loop > > i.e. there is not a significant difference between aligned and unaligned > access to data. > > I wonder if the same technique could be applied to NumPy. you already can do so with relatively simple means: http://nbviewer.ipython.org/gist/anonymous/10942132 If you change the blocking function to get a function as input and use inplace operations numpy can even beat numexpr (though I used the numexpr Ubuntu package which might not be compiled optimally) This type of transformation can probably be applied on the AST quite easily. > > Francesc > > > El 17/04/14 16:26, Aron Ahmadia ha escrit: >> Hmnn, I wasn't being clear :) >> >> The default malloc on BlueGene/Q only returns 8 byte alignment, but >> the SIMD units need 32-byte alignment for loads, stores, and >> operations or performance suffers. On the /P the required alignment >> was 16-bytes, but malloc only gave you 8, and trying to perform >> vectorized loads/stores generated alignment exceptions on unaligned >> memory. >> >> See https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q and >> https://computing.llnl.gov/tutorials/bgp/BGP-usage.Walkup.pdf (slides >> 14 for overview, 15 for the effective performance difference between >> the unaligned/aligned code) for some notes on this. >> >> A >> >> >> >> >> On Thu, Apr 17, 2014 at 10:18 AM, Nathaniel Smith > > wrote: >> >> On 17 Apr 2014 15:09, "Aron Ahmadia" > > wrote: >> > >> > > On the one hand it would be nice to actually know whether >> posix_memalign is important, before making api decisions on this >> basis. >> > >> > FWIW: On the lightweight IBM cores that the extremely popular >> BlueGene machines were based on, accessing unaligned memory raised >> system faults. The default behavior of these machines was to >> terminate the program if more than 1000 such errors occurred on a >> given process, and an environment variable allowed you to >> terminate the program if *any* unaligned memory access occurred. >> This is because unaligned memory accesses were 15x (or more) >> slower than aligned memory access. >> > >> > The newer /Q chips seem to be a little more forgiving of this, >> but I think one can in general expect allocated memory alignment >> to be an important performance technique for future high >> performance computing architectures. >> >> Right, this much is true on lots of architectures, and so malloc >> is careful to always return values with sufficient alignment (e.g. >> 8 bytes) to make sure that any standard operation can succeed. >> >> The question here is whether it will be important to have *even >> more* alignment than malloc gives us by default. A 16 or 32 byte >> wide SIMD instruction might prefer that data have 16 or 32 byte >> alignment, even if normal memory access for the types being >> operated on only requires 4 or 8 byte alignment. >> >> -n >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > -- > Francesc Alted > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From shoyer at gmail.com Thu Apr 17 13:45:59 2014 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 17 Apr 2014 10:45:59 -0700 Subject: [Numpy-discussion] min depth to nonzero in 3d array In-Reply-To: <5350022A.4060008@gmail.com> References: <5350022A.4060008@gmail.com> Message-ID: Hi Alan, You can abuse np.argmax to calculate the first nonzero element in a vectorized manner: import numpy as np A = (2 * np.random.rand(100, 50, 50)).astype(int) Compare: np.argmax(A != 0, axis=0) np.array([[np.flatnonzero(A[:,i,j])[0] for j in range(50)] for i in range(50)]) You'll also want to check for all zero arrays with np.all: np.all(A == 0, axis=0) Cheers, Stephan On Thu, Apr 17, 2014 at 9:32 AM, Alan G Isaac wrote: > Given an array A of shape m x n x n > (i.e., a stack of square matrices), > I want an n x n array that gives the > minimum "depth" to a nonzero element. > E.g., the 0,0 element of the result is > np.flatnonzero(A[:,0,0])[0] > Can this be vectorized? > (Assuming a nonzero element exists is ok, > but dealing nicely with its absence is even better.) > > Thanks, > Alan Isaac > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Thu Apr 17 13:49:13 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Thu, 17 Apr 2014 19:49:13 +0200 Subject: [Numpy-discussion] min depth to nonzero in 3d array In-Reply-To: References: <5350022A.4060008@gmail.com> Message-ID: I agree; argmax would the best option here; though I would hardly call it abuse. It seems perfectly readable and idiomatic to me. Though the != comparison requires an extra pass over the array, that's the kind of tradeoff you make in using numpy. On Thu, Apr 17, 2014 at 7:45 PM, Stephan Hoyer wrote: > Hi Alan, > > You can abuse np.argmax to calculate the first nonzero element in a > vectorized manner: > > import numpy as np > A = (2 * np.random.rand(100, 50, 50)).astype(int) > > Compare: > > np.argmax(A != 0, axis=0) > np.array([[np.flatnonzero(A[:,i,j])[0] for j in range(50)] for i in > range(50)]) > > You'll also want to check for all zero arrays with np.all: > > np.all(A == 0, axis=0) > > Cheers, > Stephan > > > On Thu, Apr 17, 2014 at 9:32 AM, Alan G Isaac wrote: > >> Given an array A of shape m x n x n >> (i.e., a stack of square matrices), >> I want an n x n array that gives the >> minimum "depth" to a nonzero element. >> E.g., the 0,0 element of the result is >> np.flatnonzero(A[:,0,0])[0] >> Can this be vectorized? >> (Assuming a nonzero element exists is ok, >> but dealing nicely with its absence is even better.) >> >> Thanks, >> Alan Isaac >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Thu Apr 17 14:30:06 2014 From: faltet at gmail.com (Francesc Alted) Date: Thu, 17 Apr 2014 20:30:06 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: <53500F2A.8070105@googlemail.com> References: <534FFC20.6010106@gmail.com> <53500F2A.8070105@googlemail.com> Message-ID: <53501DAE.8040007@gmail.com> El 17/04/14 19:28, Julian Taylor ha escrit: > On 17.04.2014 18:06, Francesc Alted wrote: > >> In [4]: x_unaligned = np.zeros(shape, >> dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] > on arrays of this size you won't see alignment issues you are dominated > by memory bandwidth. If at all you will only see it if the data fits > into the cache. > Its also about unaligned to simd vectors not unaligned to basic types. > But it doesn't matter anymore on modern x86 cpus. I guess for array data > cache line splits should also not be a big concern. Yes, that was my point, that in x86 CPUs this is not such a big problem. But still a factor of 2 is significant, even for CPU-intensive tasks. For example, computing sin() is affected similarly (sin() is using SIMD, right?): In [6]: shape = (10000, 10000) In [7]: x_aligned = np.zeros(shape, dtype=[('x',np.float64),('y',np.int64)])['x'] In [8]: x_unaligned = np.zeros(shape, dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] In [9]: %timeit res = np.sin(x_aligned) 1 loops, best of 3: 654 ms per loop In [10]: %timeit res = np.sin(x_unaligned) 1 loops, best of 3: 1.08 s per loop and again, numexpr can deal with that pretty well (using 8 physical cores here): In [6]: %timeit res = ne.evaluate('sin(x_aligned)') 10 loops, best of 3: 149 ms per loop In [7]: %timeit res = ne.evaluate('sin(x_unaligned)') 10 loops, best of 3: 151 ms per loop > Aligned allocators are not the only allocator which might be useful in > numpy. Modern CPUs also support larger pages than 4K (huge pages up to > 1GB in size) which reduces TLB cache misses. Memory of this type > typically needs to be allocated with special mmap flags, though newer > kernel versions can now also provide this memory to transparent > anonymous pages (normal non-file mmaps). That's interesting. In which scenarios do you think that could improve performance? >> In [8]: import numexpr as ne >> >> In [9]: %timeit res = ne.evaluate('x_aligned ** 2') >> 10 loops, best of 3: 133 ms per loop >> >> In [10]: %timeit res = ne.evaluate('x_unaligned ** 2') >> 10 loops, best of 3: 134 ms per loop >> >> i.e. there is not a significant difference between aligned and unaligned >> access to data. >> >> I wonder if the same technique could be applied to NumPy. > > you already can do so with relatively simple means: > http://nbviewer.ipython.org/gist/anonymous/10942132 > > If you change the blocking function to get a function as input and use > inplace operations numpy can even beat numexpr (though I used the > numexpr Ubuntu package which might not be compiled optimally) > This type of transformation can probably be applied on the AST quite easily. That's smart. Yeah, I don't see a reason why numexpr would be performing badly on Ubuntu. But I am not getting your performance for blocked_thread on my AMI linux vbox: http://nbviewer.ipython.org/gist/anonymous/11000524 oh well, threads are always tricky beasts. By the way, apparently the optimal block size for my machine is something like 1 MB, not 128 KB, although the difference is not big: http://nbviewer.ipython.org/gist/anonymous/11002751 (thanks to Stefan Van der Walt for the script). -- Francesc Alted From jtaylor.debian at googlemail.com Thu Apr 17 15:19:58 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 17 Apr 2014 21:19:58 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: <53501DAE.8040007@gmail.com> References: <534FFC20.6010106@gmail.com> <53500F2A.8070105@googlemail.com> <53501DAE.8040007@gmail.com> Message-ID: <5350295E.5040204@googlemail.com> On 17.04.2014 20:30, Francesc Alted wrote: > El 17/04/14 19:28, Julian Taylor ha escrit: >> On 17.04.2014 18:06, Francesc Alted wrote: >> >>> In [4]: x_unaligned = np.zeros(shape, >>> dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] >> on arrays of this size you won't see alignment issues you are dominated >> by memory bandwidth. If at all you will only see it if the data fits >> into the cache. >> Its also about unaligned to simd vectors not unaligned to basic types. >> But it doesn't matter anymore on modern x86 cpus. I guess for array data >> cache line splits should also not be a big concern. > > Yes, that was my point, that in x86 CPUs this is not such a big > problem. But still a factor of 2 is significant, even for CPU-intensive > tasks. For example, computing sin() is affected similarly (sin() is > using SIMD, right?): > > In [6]: shape = (10000, 10000) > > In [7]: x_aligned = np.zeros(shape, > dtype=[('x',np.float64),('y',np.int64)])['x'] > > In [8]: x_unaligned = np.zeros(shape, > dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] > > In [9]: %timeit res = np.sin(x_aligned) > 1 loops, best of 3: 654 ms per loop > > In [10]: %timeit res = np.sin(x_unaligned) > 1 loops, best of 3: 1.08 s per loop > > and again, numexpr can deal with that pretty well (using 8 physical > cores here): > > In [6]: %timeit res = ne.evaluate('sin(x_aligned)') > 10 loops, best of 3: 149 ms per loop > > In [7]: %timeit res = ne.evaluate('sin(x_unaligned)') > 10 loops, best of 3: 151 ms per loop in this case the unaligned triggers a strided memcpy calling loop to copy the data into a aligned buffer which is terrible for performance, even compared to the expensive sin call. numexpr handles this well as it allows the compiler to replace the memcpy with inline assembly (a mov instruction). We could fix that in numpy, though I don't consider it very important, you usually always have base type aligned memory. (sin is not a SIMD using function unless you use a vector math library not supported by numpy directly yet) > > >> Aligned allocators are not the only allocator which might be useful in >> numpy. Modern CPUs also support larger pages than 4K (huge pages up to >> 1GB in size) which reduces TLB cache misses. Memory of this type >> typically needs to be allocated with special mmap flags, though newer >> kernel versions can now also provide this memory to transparent >> anonymous pages (normal non-file mmaps). > > That's interesting. In which scenarios do you think that could improve > performance? it might improve all numpy operations dealing with big arrays. big arrays trigger many large temporaries meaning glibc uses mmap meaning lots of moving of address space between the kernel and userspace. but I haven't benchmarked it, so it could also be completely irrelevant. Also memory fragments really fast, so after a few hours of operation you often can't allocate any huge pages anymore, so you need to reserve space for them which requires special setup of machines. Another possibility for special allocators are numa allocators that ensure you get memory local to a specific compute node regardless of the system numa policy. But again its probably not very important as python has poor thread scalability anyway, these are just examples for keeping flexibility of our allocators in numpy and not binding us to what python does. > >>> In [8]: import numexpr as ne >>> >>> In [9]: %timeit res = ne.evaluate('x_aligned ** 2') >>> 10 loops, best of 3: 133 ms per loop >>> >>> In [10]: %timeit res = ne.evaluate('x_unaligned ** 2') >>> 10 loops, best of 3: 134 ms per loop >>> >>> i.e. there is not a significant difference between aligned and unaligned >>> access to data. >>> >>> I wonder if the same technique could be applied to NumPy. >> >> you already can do so with relatively simple means: >> http://nbviewer.ipython.org/gist/anonymous/10942132 >> >> If you change the blocking function to get a function as input and use >> inplace operations numpy can even beat numexpr (though I used the >> numexpr Ubuntu package which might not be compiled optimally) >> This type of transformation can probably be applied on the AST quite easily. > > That's smart. Yeah, I don't see a reason why numexpr would be > performing badly on Ubuntu. But I am not getting your performance for > blocked_thread on my AMI linux vbox: > > http://nbviewer.ipython.org/gist/anonymous/11000524 my numexpr amd64 package does not make use of SIMD e.g. sqrt which is vectorized in numpy: numexpr: 1.30 ? 4638: sqrtss (%r14),%xmm0 0.01 ? ucomis %xmm0,%xmm0 0.00 ? ? jp 11ec4 4.14 ? 4646: movss %xmm0,(%r15,%r12,1) ? add %rbp,%r14 ? add $0x4,%r12 (unrolled a couple times) vs numpy: 83.25 ?190: sqrtps (%rbx,%r12,4),%xmm0 0.52 ? movaps %xmm0,0x0(%rbp,%r12,4) 14.63 ? add $0x4,%r12 1.60 ? cmp %rdx,%r12 ? ? jb 190 (note the ps vs ss suffix, packed vs scalar) From onefire.myself at gmail.com Thu Apr 17 15:30:59 2014 From: onefire.myself at gmail.com (onefire) Date: Thu, 17 Apr 2014 15:30:59 -0400 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: Hi Nathaniel, Thanks for the suggestion. I did profile the program before, just not using Python. But following your suggestion, I used %prun. Here's (part of) the output (when I use savez): 195503 function calls in 4.466 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 2 2.284 1.142 2.284 1.142 {method 'close' of '_io.BufferedWriter' objects} 1 0.918 0.918 0.918 0.918 {built-in method remove} 48841 0.568 0.000 0.568 0.000 {method 'write' of '_io.BufferedWriter' objects} 48829 0.379 0.000 0.379 0.000 {built-in method crc32} 48830 0.148 0.000 0.148 0.000 {method 'read' of '_io.BufferedReader' objects} 1 0.090 0.090 0.993 0.993 zipfile.py:1315(write) 1 0.072 0.072 0.072 0.072 {method 'tostring' of 'numpy.ndarray' objects} 48848 0.005 0.000 0.005 0.000 {built-in method len} 1 0.001 0.001 0.270 0.270 format.py:362(write_array) 3 0.000 0.000 0.000 0.000 {built-in method open} 1 0.000 0.000 4.466 4.466 npyio.py:560(_savez) 2 0.000 0.000 0.000 0.000 zipfile.py:1459(close) 1 0.000 0.000 4.466 4.466 {built-in method exec} Here's the output when I use save to save to a npy file: 39 function calls in 0.266 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 4 0.196 0.049 0.196 0.049 {method 'write' of '_io.BufferedWriter' objects} 1 0.069 0.069 0.069 0.069 {method 'tostring' of 'numpy.ndarray' objects} 1 0.001 0.001 0.266 0.266 format.py:362(write_array) 1 0.000 0.000 0.000 0.000 {built-in method open} 1 0.000 0.000 0.266 0.266 npyio.py:406(save) 1 0.000 0.000 0.000 0.000 format.py:261(write_array_header_1_0) 1 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedWriter' objects} 1 0.000 0.000 0.266 0.266 {built-in method exec} 1 0.000 0.000 0.000 0.000 format.py:154(magic) 1 0.000 0.000 0.000 0.000 format.py:233(header_data_from_array_1_0) 1 0.000 0.000 0.266 0.266 :1() 1 0.000 0.000 0.000 0.000 numeric.py:462(asanyarray) 1 0.000 0.000 0.000 0.000 py3k.py:28(asbytes) The calls to close and the built-in method remove seem to be the responsible for the inefficiency of the Numpy implementation (compared to the Julia package that I mentioned before). This was tested using Python 3.4 and Numpy 1.8.1. However if I do the tests with Python 3.3.5 and Numpy 1.8.0, savez becomes much faster, so I think there is something wrong with this combination Python 3.4/Numpy 1.8.1. Also, if I use Python 2.4 and Numpy 1.2 (from my school's cluster) I get that np.save takes about 3.5 seconds and np.savez takes about 7 seconds, so all these timings seem to be hugely dependent on the system/version (maybe this explain David Palao's results?). However, they all point out that a significant amount of time is spent computing the crc32. Notice that prun reports that it takes 0.379 second to compute the crc32 of an array that takes 0.2 seconds to save to a npy file. I believe this is too much! And it get worse if you try to save bigger arrays. On Thu, Apr 17, 2014 at 5:23 AM, Nathaniel Smith wrote: > On 17 Apr 2014 01:57, "onefire" wrote: > > > > What I cannot understand is why savez takes more than 10 times longer > than saving the data to a npy file. The only reason that I could come up > with was the computation of the crc32. > > We can all make guesses but the solution is just to profile it :-). %prun > in ipython (and then if you need more granularity installing line_profiler > is useful). > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Thu Apr 17 15:51:32 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 17 Apr 2014 21:51:32 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: <535030C4.9020700@googlemail.com> On 17.04.2014 21:30, onefire wrote: > Hi Nathaniel, > > Thanks for the suggestion. I did profile the program before, just not > using Python. one problem of npz is that the zipfile module does not support streaming data in (or if it does now we aren't using it). So numpy writes the file uncompressed to disk and then zips it which is horrible for performance and disk usage. It would be nice if we could add support for different compression modules like gzip or xz which allow streaming data directly into a file without an intermediate. From valentin at haenel.co Thu Apr 17 16:01:04 2014 From: valentin at haenel.co (Valentin Haenel) Date: Thu, 17 Apr 2014 22:01:04 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: Message-ID: <20140417200104.GA22624@kudu.in-berlin.de> Hi again, * David Palao [2014-04-17]: > 2014-04-16 20:26 GMT+02:00 onefire : > > Hi all, > > > > I have been playing with the idea of using Numpy's binary format as a > > lightweight alternative to HDF5 (which I believe is the "right" way to do if > > one does not have a problem with the dependency). > > > > I am pretty happy with the npy format, but the npz format seems to be broken > > as far as performance is concerned (or I am missing obvious!). The following > > ipython session illustrates the issue: > > > > ln [1]: import numpy as np > > > > In [2]: x = np.linspace(1, 10, 50000000) > > > > In [3]: %time np.save("x.npy", x) > > CPU times: user 40 ms, sys: 230 ms, total: 270 ms > > Wall time: 488 ms > > > > In [4]: %time np.savez("x.npz", data = x) > > CPU times: user 657 ms, sys: 707 ms, total: 1.36 s > > Wall time: 7.7 s > > > > Hi, > In my case (python-2.7.3, numpy-1.6.1): > > In [23]: %time save("xx.npy", x) > CPU times: user 0.00 s, sys: 0.23 s, total: 0.23 s > Wall time: 4.07 s > > In [24]: %time savez("xx.npz", data = x) > CPU times: user 0.42 s, sys: 0.61 s, total: 1.02 s > Wall time: 4.26 s > > In my case I don't see the "unbelievable amount of overhead" of the npz thing. When profiling IO operations, there are many factors that can influence measurements. In my experience on Linux: these may include: the filesystem cache, the cpu govenor, the system load, power saving features, the type of hard drive and how it is connected, any powersaving features (e.g. laptop-mode tools) and any cron-jobs that might be running (e.g. updating locate DB). So for example when measuring the time it takes to write something to disk on Linux, I always at least include a call to ``sync`` which will ensure that all kernel filesystem buffers will be written to disk. Even then, you may still have a lot of variability. As part of bloscpack.sysutil I have wrapped this to be available from Python (needs root though). So, to re-rurn the benchmarks, doing each one twice: In [1]: import numpy as np In [2]: import bloscpack.sysutil as bps In [3]: x = np.linspace(1, 10, 50000000) In [4]: %time np.save("x.npy", x) CPU times: user 12 ms, sys: 356 ms, total: 368 ms Wall time: 1.41 s In [5]: %time np.save("x.npy", x) CPU times: user 0 ns, sys: 368 ms, total: 368 ms Wall time: 811 ms In [6]: %time np.savez("x.npz", data = x) CPU times: user 540 ms, sys: 864 ms, total: 1.4 s Wall time: 4.74 s In [7]: %time np.savez("x.npz", data = x) CPU times: user 580 ms, sys: 808 ms, total: 1.39 s Wall time: 9.47 s In [8]: bps.sync() In [9]: %time np.save("x.npy", x) ; bps.sync() CPU times: user 0 ns, sys: 368 ms, total: 368 ms Wall time: 2.2 s In [10]: %time np.save("x.npy", x) ; bps.sync() CPU times: user 0 ns, sys: 356 ms, total: 356 ms Wall time: 2.16 s In [11]: bps.sync() In [12]: %time np.savez("x.npz", x) ; bps.sync() CPU times: user 564 ms, sys: 816 ms, total: 1.38 s Wall time: 8.21 s In [13]: %time np.savez("x.npz", x) ; bps.sync() CPU times: user 588 ms, sys: 772 ms, total: 1.36 s Wall time: 6.83 s As you can see, even when using ``sync`` the values might vary, so in addition it might be worth using %timeit, which will at least run it three times and select the best one in its default setting: In [14]: %timeit np.save("x.npy", x) 1 loops, best of 3: 2.4 s per loop In [15]: %timeit np.savez("x.npz", x) 1 loops, best of 3: 7.1 s per loop In [16]: %timeit np.save("x.npy", x) ; bps.sync() 1 loops, best of 3: 3.11 s per loop In [17]: %timeit np.savez("x.npz", x) ; bps.sync() 1 loops, best of 3: 7.36 s per loop So, anyway, given these readings, I would tend to support the claim that there is something slowing down writing when using plain NPZ w/o compression. FYI: when reading, the kernel keeps files that were recently read in the filesystem buffers and so when measuring reads, I tend to drop those caches using ``drop_caches()`` from bloscpack.sysutil (which wraps using the linux proc fs). best, V- From valentin at haenel.co Thu Apr 17 16:26:35 2014 From: valentin at haenel.co (Valentin Haenel) Date: Thu, 17 Apr 2014 22:26:35 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <535030C4.9020700@googlemail.com> References: <535030C4.9020700@googlemail.com> Message-ID: <20140417202635.GB22624@kudu.in-berlin.de> Hi, * Julian Taylor [2014-04-17]: > On 17.04.2014 21:30, onefire wrote: > > Hi Nathaniel, > > > > Thanks for the suggestion. I did profile the program before, just not > > using Python. > > one problem of npz is that the zipfile module does not support streaming > data in (or if it does now we aren't using it). > So numpy writes the file uncompressed to disk and then zips it which is > horrible for performance and disk usage. As a workaround may also be possible to write the temporary NPY files to cStringIO instances and then use ``ZipFile.writestr`` with the ``getvalue()`` of the cStringIO object. However that approach may require some memory. In python 2.7, for each array: one copy inside the cStringIO instance and then another copy of when calling getvalue on the cString, I believe. best, V- From valentin at haenel.co Thu Apr 17 16:56:27 2014 From: valentin at haenel.co (Valentin Haenel) Date: Thu, 17 Apr 2014 22:56:27 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417202635.GB22624@kudu.in-berlin.de> References: <535030C4.9020700@googlemail.com> <20140417202635.GB22624@kudu.in-berlin.de> Message-ID: <20140417205627.GA4192@kudu.in-berlin.de> * Valentin Haenel [2014-04-17]: > Hi, > > * Julian Taylor [2014-04-17]: > > On 17.04.2014 21:30, onefire wrote: > > > Hi Nathaniel, > > > > > > Thanks for the suggestion. I did profile the program before, just not > > > using Python. > > > > one problem of npz is that the zipfile module does not support streaming > > data in (or if it does now we aren't using it). > > So numpy writes the file uncompressed to disk and then zips it which is > > horrible for performance and disk usage. > > As a workaround may also be possible to write the temporary NPY files to > cStringIO instances and then use ``ZipFile.writestr`` with the > ``getvalue()`` of the cStringIO object. However that approach may > require some memory. In python 2.7, for each array: one copy inside the > cStringIO instance and then another copy of when calling getvalue on the > cString, I believe. There is a proof-of-concept implementation here: https://github.com/esc/numpy/compare/feature;npz_no_temp_file Here are the timings, again using ``sync()`` from bloscpack (but it's just a ``os.system('sync')``, in case you want to run your own benchmarks): In [1]: import numpy as np In [2]: import bloscpack.sysutil as bps In [3]: x = np.linspace(1, 10, 50000000) In [4]: %timeit np.save("x.npy", x) ; bps.sync() 1 loops, best of 3: 1.93 s per loop In [5]: %timeit np.savez("x.npz", x) ; bps.sync() 1 loops, best of 3: 7.88 s per loop In [6]: %timeit np._savez_no_temp("x.npy", [x], {}, False) ; bps.sync() 1 loops, best of 3: 3.22 s per loop Not too bad, but still slower than plain NPY, memory copies would be my guess. V- PS: Running Python 2.7.6 :: Anaconda 1.9.2 (64-bit) and Numpy master From valentin at haenel.co Thu Apr 17 17:18:09 2014 From: valentin at haenel.co (Valentin Haenel) Date: Thu, 17 Apr 2014 23:18:09 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417205627.GA4192@kudu.in-berlin.de> References: <535030C4.9020700@googlemail.com> <20140417202635.GB22624@kudu.in-berlin.de> <20140417205627.GA4192@kudu.in-berlin.de> Message-ID: <20140417211809.GB6950@kudu.in-berlin.de> * Valentin Haenel [2014-04-17]: > * Valentin Haenel [2014-04-17]: > > Hi, > > > > * Julian Taylor [2014-04-17]: > > > On 17.04.2014 21:30, onefire wrote: > > > > Hi Nathaniel, > > > > > > > > Thanks for the suggestion. I did profile the program before, just not > > > > using Python. > > > > > > one problem of npz is that the zipfile module does not support streaming > > > data in (or if it does now we aren't using it). > > > So numpy writes the file uncompressed to disk and then zips it which is > > > horrible for performance and disk usage. > > > > As a workaround may also be possible to write the temporary NPY files to > > cStringIO instances and then use ``ZipFile.writestr`` with the > > ``getvalue()`` of the cStringIO object. However that approach may > > require some memory. In python 2.7, for each array: one copy inside the > > cStringIO instance and then another copy of when calling getvalue on the > > cString, I believe. > > There is a proof-of-concept implementation here: > > https://github.com/esc/numpy/compare/feature;npz_no_temp_file > > Here are the timings, again using ``sync()`` from bloscpack (but it's > just a ``os.system('sync')``, in case you want to run your own > benchmarks): > > In [1]: import numpy as np > > In [2]: import bloscpack.sysutil as bps > > In [3]: x = np.linspace(1, 10, 50000000) > > In [4]: %timeit np.save("x.npy", x) ; bps.sync() > 1 loops, best of 3: 1.93 s per loop > > In [5]: %timeit np.savez("x.npz", x) ; bps.sync() > 1 loops, best of 3: 7.88 s per loop > > In [6]: %timeit np._savez_no_temp("x.npy", [x], {}, False) ; bps.sync() > 1 loops, best of 3: 3.22 s per loop > > Not too bad, but still slower than plain NPY, memory copies would be my > guess. > PS: Running Python 2.7.6 :: Anaconda 1.9.2 (64-bit) and Numpy master Also, in cae you were wondering, here is the profiler output: In [2]: %prun -l 10 np._savez_no_temp("x.npy", [x], {}, False) 943 function calls (917 primitive calls) in 1.139 seconds Ordered by: internal time List reduced from 99 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.386 0.386 0.386 0.386 {zlib.crc32} 8 0.234 0.029 0.234 0.029 {method 'write' of 'file' objects} 27 0.162 0.006 0.162 0.006 {method 'write' of 'cStringIO.StringO' objects} 1 0.158 0.158 0.158 0.158 {method 'getvalue' of 'cStringIO.StringO' objects} 1 0.091 0.091 0.091 0.091 {method 'close' of 'file' objects} 24 0.064 0.003 0.064 0.003 {method 'tobytes' of 'numpy.ndarray' objects} 1 0.022 0.022 1.119 1.119 npyio.py:608(_savez_no_temp) 1 0.019 0.019 1.139 1.139 :1() 1 0.002 0.002 0.227 0.227 format.py:362(write_array) 1 0.001 0.001 0.001 0.001 zipfile.py:433(_GenerateCRCTable) V- From valentin at haenel.co Thu Apr 17 17:35:37 2014 From: valentin at haenel.co (Valentin Haenel) Date: Thu, 17 Apr 2014 23:35:37 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417211809.GB6950@kudu.in-berlin.de> References: <535030C4.9020700@googlemail.com> <20140417202635.GB22624@kudu.in-berlin.de> <20140417205627.GA4192@kudu.in-berlin.de> <20140417211809.GB6950@kudu.in-berlin.de> Message-ID: <20140417213537.GC6950@kudu.in-berlin.de> Hi, * Valentin Haenel [2014-04-17]: > * Valentin Haenel [2014-04-17]: > > * Valentin Haenel [2014-04-17]: > > > Hi, > > > > > > * Julian Taylor [2014-04-17]: > > > > On 17.04.2014 21:30, onefire wrote: > > > > > Hi Nathaniel, > > > > > > > > > > Thanks for the suggestion. I did profile the program before, just not > > > > > using Python. > > > > > > > > one problem of npz is that the zipfile module does not support streaming > > > > data in (or if it does now we aren't using it). > > > > So numpy writes the file uncompressed to disk and then zips it which is > > > > horrible for performance and disk usage. > > > > > > As a workaround may also be possible to write the temporary NPY files to > > > cStringIO instances and then use ``ZipFile.writestr`` with the > > > ``getvalue()`` of the cStringIO object. However that approach may > > > require some memory. In python 2.7, for each array: one copy inside the > > > cStringIO instance and then another copy of when calling getvalue on the > > > cString, I believe. > > > > There is a proof-of-concept implementation here: > > > > https://github.com/esc/numpy/compare/feature;npz_no_temp_file > > > > Here are the timings, again using ``sync()`` from bloscpack (but it's > > just a ``os.system('sync')``, in case you want to run your own > > benchmarks): > > > > In [1]: import numpy as np > > > > In [2]: import bloscpack.sysutil as bps > > > > In [3]: x = np.linspace(1, 10, 50000000) > > > > In [4]: %timeit np.save("x.npy", x) ; bps.sync() > > 1 loops, best of 3: 1.93 s per loop > > > > In [5]: %timeit np.savez("x.npz", x) ; bps.sync() > > 1 loops, best of 3: 7.88 s per loop > > > > In [6]: %timeit np._savez_no_temp("x.npy", [x], {}, False) ; bps.sync() > > 1 loops, best of 3: 3.22 s per loop > > > > Not too bad, but still slower than plain NPY, memory copies would be my > > guess. > > > PS: Running Python 2.7.6 :: Anaconda 1.9.2 (64-bit) and Numpy master > > Also, in cae you were wondering, here is the profiler output: > > In [2]: %prun -l 10 np._savez_no_temp("x.npy", [x], {}, False) > 943 function calls (917 primitive calls) in 1.139 seconds > > Ordered by: internal time > List reduced from 99 to 10 due to restriction <10> > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.386 0.386 0.386 0.386 {zlib.crc32} > 8 0.234 0.029 0.234 0.029 {method 'write' of 'file' objects} > 27 0.162 0.006 0.162 0.006 {method 'write' of 'cStringIO.StringO' objects} > 1 0.158 0.158 0.158 0.158 {method 'getvalue' of 'cStringIO.StringO' objects} > 1 0.091 0.091 0.091 0.091 {method 'close' of 'file' objects} > 24 0.064 0.003 0.064 0.003 {method 'tobytes' of 'numpy.ndarray' objects} > 1 0.022 0.022 1.119 1.119 npyio.py:608(_savez_no_temp) > 1 0.019 0.019 1.139 1.139 :1() > 1 0.002 0.002 0.227 0.227 format.py:362(write_array) > 1 0.001 0.001 0.001 0.001 zipfile.py:433(_GenerateCRCTable) And, to shed some more light on this, the kernprofiler (line-by-line) output (of a slightly modified version): zsh? cat mp.py import numpy as np x = np.linspace(1, 10, 50000000) np._savez_no_temp("x.npy", [x], {}, False) zsh? ./kernprof.py -v -l mp.py Wrote profile results to mp.py.lprof Timer unit: 1e-06 s File: numpy/lib/npyio.py Function: _savez_no_temp at line 608 Total time: 1.16438 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 608 @profile 609 def _savez_no_temp(file, args, kwds, compress): 610 # Import is postponed to here since zipfile depends on gzip, an optional 611 # component of the so-called standard library. 612 1 5655 5655.0 0.5 import zipfile 613 614 1 6 6.0 0.0 from cStringIO import StringIO 615 616 1 2 2.0 0.0 if isinstance(file, basestring): 617 1 2 2.0 0.0 if not file.endswith('.npz'): 618 1 1 1.0 0.0 file = file + '.npz' 619 620 1 1 1.0 0.0 namedict = kwds 621 2 4 2.0 0.0 for i, val in enumerate(args): 622 1 6 6.0 0.0 key = 'arr_%d' % i 623 1 1 1.0 0.0 if key in namedict.keys(): 624 raise ValueError( 625 "Cannot use un-named variables and keyword %s" % key) 626 1 1 1.0 0.0 namedict[key] = val 627 628 1 0 0.0 0.0 if compress: 629 compression = zipfile.ZIP_DEFLATED 630 else: 631 1 1 1.0 0.0 compression = zipfile.ZIP_STORED 632 633 1 42734 42734.0 3.7 zipf = zipfile_factory(file, mode="w", compression=compression) 634 # reusable memory buffer 635 1 5 5.0 0.0 sio = StringIO() 636 2 10 5.0 0.0 for key, val in namedict.items(): 637 1 3 3.0 0.0 fname = key + '.npy' 638 1 4 4.0 0.0 sio.seek(0) # reset buffer 639 1 219843 219843.0 18.9 format.write_array(sio, np.asanyarray(val)) 640 1 156962 156962.0 13.5 array_bytes = sio.getvalue(True) 641 1 625162 625162.0 53.7 zipf.writestr(fname, array_bytes) 642 643 1 113977 113977.0 9.8 zipf.close() So it would appear that >50% of the time is spent in the zipfile.writestr. V- From valentin at haenel.co Thu Apr 17 18:45:53 2014 From: valentin at haenel.co (Valentin Haenel) Date: Fri, 18 Apr 2014 00:45:53 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417200104.GA22624@kudu.in-berlin.de> References: <20140417200104.GA22624@kudu.in-berlin.de> Message-ID: <20140417224553.GA2361@kudu.in-berlin.de> Hello, * Valentin Haenel [2014-04-17]: > As part of bloscpack.sysutil I have wrapped this to be available from > Python (needs root though). So, to re-rurn the benchmarks, doing each > one twice: Actually, I just realized, that doing a ``sync`` doesn't require root. my bad, V- From onefire.myself at gmail.com Thu Apr 17 20:09:59 2014 From: onefire.myself at gmail.com (onefire) Date: Thu, 17 Apr 2014 20:09:59 -0400 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417224553.GA2361@kudu.in-berlin.de> References: <20140417200104.GA22624@kudu.in-berlin.de> <20140417224553.GA2361@kudu.in-berlin.de> Message-ID: Interesting! Using sync() as you suggested makes every write slower, and it decreases the time difference between save and savez, so maybe I was observing the 10 times difference because the file system buffers were being flushed immediately after a call to savez, but not right after a call to np.save. I think your workaround might help, but a better solution would be to not use Python's zipfile module at all. This would make it possible to, say, let the user choose the checksum algorithm or to turn that off. Or maybe the compression stuff makes this route too complicated to be worth the trouble? (after all, the zip format is not that hard to understand) Gilberto On Thu, Apr 17, 2014 at 6:45 PM, Valentin Haenel wrote: > Hello, > > * Valentin Haenel [2014-04-17]: > > As part of bloscpack.sysutil I have wrapped this to be available from > > Python (needs root though). So, to re-rurn the benchmarks, doing each > > one twice: > > Actually, I just realized, that doing a ``sync`` doesn't require root. > > my bad, > > V- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From onefire.myself at gmail.com Thu Apr 17 20:12:43 2014 From: onefire.myself at gmail.com (onefire) Date: Thu, 17 Apr 2014 20:12:43 -0400 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: <20140417200104.GA22624@kudu.in-berlin.de> <20140417224553.GA2361@kudu.in-berlin.de> Message-ID: I found this github issue (https://github.com/numpy/numpy/pull/3465) where someone mentions the idea of forking the zip library. Gilberto On Thu, Apr 17, 2014 at 8:09 PM, onefire wrote: > Interesting! Using sync() as you suggested makes every write slower, and > it decreases the time difference between save and savez, > so maybe I was observing the 10 times difference because the file system > buffers were being flushed immediately after a call to savez, but not right > after a call to np.save. > > I think your workaround might help, but a better solution would be to not > use Python's zipfile module at all. This would make it possible to, say, > let the user choose the checksum algorithm or to turn that off. > Or maybe the compression stuff makes this route too complicated to be > worth the trouble? (after all, the zip format is not that hard to > understand) > > Gilberto > > > > On Thu, Apr 17, 2014 at 6:45 PM, Valentin Haenel wrote: > >> Hello, >> >> * Valentin Haenel [2014-04-17]: >> > As part of bloscpack.sysutil I have wrapped this to be available from >> > Python (needs root though). So, to re-rurn the benchmarks, doing each >> > one twice: >> >> Actually, I just realized, that doing a ``sync`` doesn't require root. >> >> my bad, >> >> V- >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From valentin at haenel.co Fri Apr 18 06:16:32 2014 From: valentin at haenel.co (Valentin Haenel) Date: Fri, 18 Apr 2014 12:16:32 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: <20140417200104.GA22624@kudu.in-berlin.de> <20140417224553.GA2361@kudu.in-berlin.de> Message-ID: <20140418101632.GB2361@kudu.in-berlin.de> Hi Gilberto, * onefire [2014-04-18]: > Interesting! Using sync() as you suggested makes every write slower, and > it decreases the time difference between save and savez, > so maybe I was observing the 10 times difference because the file system > buffers were being flushed immediately after a call to savez, but not right > after a call to np.save. I am happy that you found my suggestion useful! Given that the current savez implementation first writes temporary arrays to disk and then copies them from their temporary location to the zipfile, one might argue that this is what causes the buffers to be flushed, since it does more IO than the save implementation. Then again I don't really now the gory details of the how the filesystem buffers behave and how they can be configured. best, V- From valentin at haenel.co Fri Apr 18 07:01:09 2014 From: valentin at haenel.co (Valentin Haenel) Date: Fri, 18 Apr 2014 13:01:09 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: References: <20140417200104.GA22624@kudu.in-berlin.de> <20140417224553.GA2361@kudu.in-berlin.de> Message-ID: <20140418110109.GA26611@kudu.in-berlin.de> Hi again, * onefire [2014-04-18]: > I think your workaround might help, but a better solution would be to not > use Python's zipfile module at all. This would make it possible to, say, > let the user choose the checksum algorithm or to turn that off. > Or maybe the compression stuff makes this route too complicated to be worth > the trouble? (after all, the zip format is not that hard to understand) Just to give you an idea of what my aforementioned Bloscpack library can do in the case of linspace: In [1]: import numpy as np In [2]: import bloscpack as bp In [3]: import bloscpack.sysutil as bps In [4]: x = np.linspace(1, 10, 50000000) In [5]: %timeit np.save("x.npy", x) ; bps.sync() 1 loops, best of 3: 2.12 s per loop In [6]: %timeit bp.pack_ndarray_file(x, 'x.blp') ; bps.sync() 1 loops, best of 3: 627 ms per loop In [7]: %timeit -n 3 -r 3 np.save("x.npy", x) ; bps.sync() 3 loops, best of 3: 1.92 s per loop In [8]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x.blp') ; bps.sync() 3 loops, best of 3: 564 ms per loop In [9]: ls -lah x.npy x.blp -rw-r--r-- 1 root root 49M Apr 18 12:53 x.blp -rw-r--r-- 1 root root 382M Apr 18 12:52 x.npy However, this is a bit of special case, since Blosc does extremely well -- both speed and size wise -- on the linspace data, your milage may vary. best, V- From faltet at gmail.com Fri Apr 18 07:39:22 2014 From: faltet at gmail.com (Francesc Alted) Date: Fri, 18 Apr 2014 13:39:22 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: <5350295E.5040204@googlemail.com> References: <534FFC20.6010106@gmail.com> <53500F2A.8070105@googlemail.com> <53501DAE.8040007@gmail.com> <5350295E.5040204@googlemail.com> Message-ID: <53510EEA.7040405@gmail.com> El 17/04/14 21:19, Julian Taylor ha escrit: > On 17.04.2014 20:30, Francesc Alted wrote: >> El 17/04/14 19:28, Julian Taylor ha escrit: >>> On 17.04.2014 18:06, Francesc Alted wrote: >>> >>>> In [4]: x_unaligned = np.zeros(shape, >>>> dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] >>> on arrays of this size you won't see alignment issues you are dominated >>> by memory bandwidth. If at all you will only see it if the data fits >>> into the cache. >>> Its also about unaligned to simd vectors not unaligned to basic types. >>> But it doesn't matter anymore on modern x86 cpus. I guess for array data >>> cache line splits should also not be a big concern. >> Yes, that was my point, that in x86 CPUs this is not such a big >> problem. But still a factor of 2 is significant, even for CPU-intensive >> tasks. For example, computing sin() is affected similarly (sin() is >> using SIMD, right?): >> >> In [6]: shape = (10000, 10000) >> >> In [7]: x_aligned = np.zeros(shape, >> dtype=[('x',np.float64),('y',np.int64)])['x'] >> >> In [8]: x_unaligned = np.zeros(shape, >> dtype=[('y1',np.int8),('x',np.float64),('y2',np.int8,(7,))])['x'] >> >> In [9]: %timeit res = np.sin(x_aligned) >> 1 loops, best of 3: 654 ms per loop >> >> In [10]: %timeit res = np.sin(x_unaligned) >> 1 loops, best of 3: 1.08 s per loop >> >> and again, numexpr can deal with that pretty well (using 8 physical >> cores here): >> >> In [6]: %timeit res = ne.evaluate('sin(x_aligned)') >> 10 loops, best of 3: 149 ms per loop >> >> In [7]: %timeit res = ne.evaluate('sin(x_unaligned)') >> 10 loops, best of 3: 151 ms per loop > in this case the unaligned triggers a strided memcpy calling loop to > copy the data into a aligned buffer which is terrible for performance, > even compared to the expensive sin call. > numexpr handles this well as it allows the compiler to replace the > memcpy with inline assembly (a mov instruction). > We could fix that in numpy, though I don't consider it very important, > you usually always have base type aligned memory. Well, that *could* be important for evaluating conditions in structured arrays, as it is pretty easy to get unaligned 'columns'. But apparently this does not affect very much to numpy: In [23]: na_aligned = np.fromiter((("", i, i*2) for i in xrange(N)), dtype="S16,i4,i8") In [24]: na_unaligned = np.fromiter((("", i, i*2) for i in xrange(N)), dtype="S15,i4,i8") In [25]: %time sum((r['f1'] for r in na_aligned[na_aligned['f2'] > 10])) CPU times: user 10.2 s, sys: 93 ms, total: 10.3 s Wall time: 10.3 s Out[25]: 49999994999985 In [26]: %time sum((r['f1'] for r in na_unaligned[na_unaligned['f2'] > 10])) CPU times: user 10.2 s, sys: 82 ms, total: 10.3 s Wall time: 10.3 s Out[26]: 49999994999985 probably because the bottleneck is in another place. So yeah, probably not worth to worry about that. > (sin is not a SIMD using function unless you use a vector math library > not supported by numpy directly yet) Ah, so MKL is making use of SIMD for computing the sin(), but not in general. But you later said that numpy's sqrt *is* making use of SIMD. I wonder why. > >> >>> Aligned allocators are not the only allocator which might be useful in >>> numpy. Modern CPUs also support larger pages than 4K (huge pages up to >>> 1GB in size) which reduces TLB cache misses. Memory of this type >>> typically needs to be allocated with special mmap flags, though newer >>> kernel versions can now also provide this memory to transparent >>> anonymous pages (normal non-file mmaps). >> That's interesting. In which scenarios do you think that could improve >> performance? > it might improve all numpy operations dealing with big arrays. > big arrays trigger many large temporaries meaning glibc uses mmap > meaning lots of moving of address space between the kernel and userspace. > but I haven't benchmarked it, so it could also be completely irrelevant. I was curious about this and apparently the speedups that typically bring large page caches is around 5%: http://stackoverflow.com/questions/14275170/performance-degradation-with-large-pages not a big deal, but it is something. > > Also memory fragments really fast, so after a few hours of operation you > often can't allocate any huge pages anymore, so you need to reserve > space for them which requires special setup of machines. > > Another possibility for special allocators are numa allocators that > ensure you get memory local to a specific compute node regardless of the > system numa policy. > But again its probably not very important as python has poor thread > scalability anyway, these are just examples for keeping flexibility of > our allocators in numpy and not binding us to what python does. Agreed. > That's smart. Yeah, I don't see a reason why numexpr would be > performing badly on Ubuntu. But I am not getting your performance for > blocked_thread on my AMI linux vbox: > > http://nbviewer.ipython.org/gist/anonymous/11000524 > > my numexpr amd64 package does not make use of SIMD e.g. sqrt which is > vectorized in numpy: > > numexpr: > 1.30 ? 4638: sqrtss (%r14),%xmm0 > 0.01 ? ucomis %xmm0,%xmm0 > 0.00 ? ? jp 11ec4 > 4.14 ? 4646: movss %xmm0,(%r15,%r12,1) > ? add %rbp,%r14 > ? add $0x4,%r12 > (unrolled a couple times) > > vs numpy: > 83.25 ?190: sqrtps (%rbx,%r12,4),%xmm0 > 0.52 ? movaps %xmm0,0x0(%rbp,%r12,4) > 14.63 ? add $0x4,%r12 > 1.60 ? cmp %rdx,%r12 > ? ? jb 190 > > (note the ps vs ss suffix, packed vs scalar) Yup, I can reproduce that: In [4]: a = np.random.rand(int(1e8)) In [5]: %timeit np.sqrt(a) 1 loops, best of 3: 558 ms per loop In [6]: %timeit ne.evaluate('sqrt(a)') 1 loops, best of 3: 249 ms per loop In [7]: ne.set_num_threads(1) Out[7]: 8 In [8]: %timeit ne.evaluate('sqrt(a)') 1 loops, best of 3: 924 ms per loop So, yes, the non-SIMD version of sqrt in numexpr is performing quite more slowly than the SIMD one in NumPy. Of course, a numexpr compiled with MKL support can achieve similar performance than numpy in single thread mode: In [4]: %timeit ne.evaluate('sqrt(a)') 1 loops, best of 3: 191 ms per loop In [5]: ne.set_num_threads(1) Out[5]: 8 In [6]: %timeit ne.evaluate('sqrt(a)') 1 loops, best of 3: 565 ms per loop So, sqrt in numpy has barely the same speed than the one in MKL. Again, I wonder why :) -- Francesc Alted From faltet at gmail.com Fri Apr 18 08:03:00 2014 From: faltet at gmail.com (Francesc Alted) Date: Fri, 18 Apr 2014 14:03:00 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140418110109.GA26611@kudu.in-berlin.de> References: <20140417200104.GA22624@kudu.in-berlin.de> <20140417224553.GA2361@kudu.in-berlin.de> <20140418110109.GA26611@kudu.in-berlin.de> Message-ID: <53511474.2050003@gmail.com> El 18/04/14 13:01, Valentin Haenel ha escrit: > Hi again, > > * onefire [2014-04-18]: >> I think your workaround might help, but a better solution would be to not >> use Python's zipfile module at all. This would make it possible to, say, >> let the user choose the checksum algorithm or to turn that off. >> Or maybe the compression stuff makes this route too complicated to be worth >> the trouble? (after all, the zip format is not that hard to understand) > Just to give you an idea of what my aforementioned Bloscpack library can > do in the case of linspace: > > In [1]: import numpy as np > > In [2]: import bloscpack as bp > > In [3]: import bloscpack.sysutil as bps > > In [4]: x = np.linspace(1, 10, 50000000) > > In [5]: %timeit np.save("x.npy", x) ; bps.sync() > 1 loops, best of 3: 2.12 s per loop > > In [6]: %timeit bp.pack_ndarray_file(x, 'x.blp') ; bps.sync() > 1 loops, best of 3: 627 ms per loop > > In [7]: %timeit -n 3 -r 3 np.save("x.npy", x) ; bps.sync() > 3 loops, best of 3: 1.92 s per loop > > In [8]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x.blp') ; bps.sync() > 3 loops, best of 3: 564 ms per loop > > In [9]: ls -lah x.npy x.blp > -rw-r--r-- 1 root root 49M Apr 18 12:53 x.blp > -rw-r--r-- 1 root root 382M Apr 18 12:52 x.npy > > However, this is a bit of special case, since Blosc does extremely well > -- both speed and size wise -- on the linspace data, your milage may > vary. Exactly, and besides, Blosc can use different codes inside it. Just for completeness, here it is a small benchmark of what you can expect from them (my laptop does not have a SSD, so my figures are a bit slow compared with Valentin's): In [50]: %timeit -n 3 -r 3 np.save("x.npy", x) ; bps.sync() 3 loops, best of 3: 5.7 s per loop In [51]: cargs = bp.args.DEFAULT_BLOSC_ARGS In [52]: cargs['cname'] = 'blosclz' In [53]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x-blosclz.blp', blosc_args=cargs) ; bps.sync() 3 loops, best of 3: 1.12 s per loop In [54]: cargs['cname'] = 'lz4' In [55]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x-lz4.blp', blosc_args=cargs) ; bps.sync() 3 loops, best of 3: 985 ms per loop In [56]: cargs['cname'] = 'lz4hc' In [57]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x-lz4hc.blp', blosc_args=cargs) ; bps.sync() 3 loops, best of 3: 1.95 s per loop In [58]: cargs['cname'] = 'snappy' In [59]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x-snappy.blp', blosc_args=cargs) ; bps.sync() 3 loops, best of 3: 1.11 s per loop In [60]: cargs['cname'] = 'zlib' In [61]: %timeit -n 3 -r 3 bp.pack_ndarray_file(x, 'x-zlib.blp', blosc_args=cargs) ; bps.sync() 3 loops, best of 3: 3.12 s per loop so all the codecs can make the storage go faster than a pure np.save(), and most specially blosclz, lz4 and snappy. However, lz4hc and zlib achieve the best compression ratios: In [62]: ls -lht x*.* -rw-r--r-- 1 faltet users 7,0M 18 abr 13:49 x-zlib.blp -rw-r--r-- 1 faltet users 54M 18 abr 13:48 x-snappy.blp -rw-r--r-- 1 faltet users 7,0M 18 abr 13:48 x-lz4hc.blp -rw-r--r-- 1 faltet users 48M 18 abr 13:47 x-lz4.blp -rw-r--r-- 1 faltet users 49M 18 abr 13:47 x-blosclz.blp -rw-r--r-- 1 faltet users 382M 18 abr 13:42 x.npy But again, we are talking about a specially nice compression case. -- Francesc Alted From matti.picus at gmail.com Fri Apr 18 09:55:34 2014 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 18 Apr 2014 16:55:34 +0300 Subject: [Numpy-discussion] mtrand and cffi Message-ID: <53512ED6.6010701@gmail.com> Hi. I am part of the pypy team. I recently rewrote mtrand.pyx as mtrand.py that uses cffi, in order to use it with pypy. mtrand.pyx uses cython to compile a python extension module, where my approach was to compile the randomkit code as a shared object, and to use that directly via cffi. If this approach interests anyone, the code is on the cffi-random fork of numpy at bitbucket.org/pypy/numpy, which is a partial implementation of pure-python numpy for pypy. Comments are welcome Matti Picus From valentin at haenel.co Fri Apr 18 12:29:27 2014 From: valentin at haenel.co (Valentin Haenel) Date: Fri, 18 Apr 2014 18:29:27 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140417205627.GA4192@kudu.in-berlin.de> References: <535030C4.9020700@googlemail.com> <20140417202635.GB22624@kudu.in-berlin.de> <20140417205627.GA4192@kudu.in-berlin.de> Message-ID: <20140418162927.GB1837@kudu.in-berlin.de> Hi, * Valentin Haenel [2014-04-17]: > * Valentin Haenel [2014-04-17]: > > * Julian Taylor [2014-04-17]: > > > On 17.04.2014 21:30, onefire wrote: > > > > Thanks for the suggestion. I did profile the program before, just not > > > > using Python. > > > > > > one problem of npz is that the zipfile module does not support streaming > > > data in (or if it does now we aren't using it). > > > So numpy writes the file uncompressed to disk and then zips it which is > > > horrible for performance and disk usage. > > > > As a workaround may also be possible to write the temporary NPY files to > > cStringIO instances and then use ``ZipFile.writestr`` with the > > ``getvalue()`` of the cStringIO object. However that approach may > > require some memory. In python 2.7, for each array: one copy inside the > > cStringIO instance and then another copy of when calling getvalue on the > > cString, I believe. > > There is a proof-of-concept implementation here: > > https://github.com/esc/numpy/compare/feature;npz_no_temp_file Anybody interested in me fixing this up (unit tests, API, etc..) for inclusion? V- From jtaylor.debian at googlemail.com Fri Apr 18 13:20:33 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 18 Apr 2014 19:20:33 +0200 Subject: [Numpy-discussion] About the npz format In-Reply-To: <20140418162927.GB1837@kudu.in-berlin.de> References: <535030C4.9020700@googlemail.com> <20140417202635.GB22624@kudu.in-berlin.de> <20140417205627.GA4192@kudu.in-berlin.de> <20140418162927.GB1837@kudu.in-berlin.de> Message-ID: <53515EE1.4080101@googlemail.com> On 18.04.2014 18:29, Valentin Haenel wrote: > Hi, > > * Valentin Haenel [2014-04-17]: >> * Valentin Haenel [2014-04-17]: >>> * Julian Taylor [2014-04-17]: >>>> On 17.04.2014 21:30, onefire wrote: >>>>> Thanks for the suggestion. I did profile the program before, just not >>>>> using Python. >>>> >>>> one problem of npz is that the zipfile module does not support streaming >>>> data in (or if it does now we aren't using it). >>>> So numpy writes the file uncompressed to disk and then zips it which is >>>> horrible for performance and disk usage. >>> >>> As a workaround may also be possible to write the temporary NPY files to >>> cStringIO instances and then use ``ZipFile.writestr`` with the >>> ``getvalue()`` of the cStringIO object. However that approach may >>> require some memory. In python 2.7, for each array: one copy inside the >>> cStringIO instance and then another copy of when calling getvalue on the >>> cString, I believe. >> >> There is a proof-of-concept implementation here: >> >> https://github.com/esc/numpy/compare/feature;npz_no_temp_file > > Anybody interested in me fixing this up (unit tests, API, etc..) for > inclusion? > I wonder if it would be better to instead use a fifo to avoid the memory doubling. Windows probably hasn't got them (exposed via python) but one can slap a platform check in front. attached a proof of concept without proper error handling (which is unfortunately the tricky part) -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-use-a-pipe-for-savez.patch Type: text/x-diff Size: 1652 bytes Desc: not available URL: From jaylenexue at yahoo.com Fri Apr 18 14:43:39 2014 From: jaylenexue at yahoo.com (jaylene) Date: Fri, 18 Apr 2014 11:43:39 -0700 (PDT) Subject: [Numpy-discussion] ImportError: /usr/local/lib/python2.7/site-packages/numpy-1.8.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString In-Reply-To: <1397676211880-37372.post@n7.nabble.com> References: <1397676211880-37372.post@n7.nabble.com> Message-ID: <1397846619009-37412.post@n7.nabble.com> Resolved by installing anaconda. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/ImportError-usr-local-lib-python2-7-site-packages-numpy-1-8-0-py2-7-linux-x86-64-egg-numpy-core-multg-tp37372p37412.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From smudkavi at uwaterloo.ca Sat Apr 19 00:17:19 2014 From: smudkavi at uwaterloo.ca (Sankarshan Mudkavi) Date: Sat, 19 Apr 2014 00:17:19 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: <4F898E39-6A6D-4E85-A00C-3113CA9C4A99@uwaterloo.ca> I think we'll be ready to start implementation once I get the conversion to datetime.datetime on the proposal with some decent examples. It would also be great to have opinions on what test cases should be used, so please speak up if you feel you have anything to say about that. Cheers, Sankarshan On Apr 14, 2014, at 2:59 PM, Chris Barker wrote: > On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer wrote: > On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris wrote: > Are we in a position to start looking at implementation? If so, it would be useful to have a collection of test cases, i.e., typical uses with specified results. That should also cover conversion from/(to?) datetime.datetime. > > yup -- tests are always good! > > Indeed, my personal wish-list for np.datetime64 is centered much more on robust conversion to/from native date objects, including comparison. > > A good use case. > > Here are some of my particular points of frustration (apologies for the thread jacking!): > - NaT should have similar behavior to NaN when used for comparisons (i.e., comparisons should always be False). > > make sense. > > - You can't compare a datetime object to a datetime64 object. > > that would be nice to have. > > - datetime64 objects with high precision (e.g., ns) can't compare to datetime objects. > > That's a problem, but how do you think it should be handled? My thought is that it should round to microseconds, and then compare -- kind of like comparing float32 and float64... > > Pandas has a very nice wrapper around datetime64 arrays that solves most of these issues, but it would be nice to get much of that functionality in core numpy, > > yes -- it would -- but learning from pandas is certainly a good idea. > > from numpy import datetime64 > from datetime import datetime > > print np.datetime64('NaT') < np.datetime64('2011-01-01') # this should not to true > print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises exception > print np.datetime64('2011-01-01T00:00', 'ns') > datetime(2010, 1, 1) # another exception > print np.datetime64('2011-01-01T00:00') > datetime(2010, 1, 1) # finally something works! > > > now to get them into proper unit tests.... > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From shoyer at gmail.com Sat Apr 19 00:50:38 2014 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 18 Apr 2014 21:50:38 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: On Mon, Apr 14, 2014 at 11:59 AM, Chris Barker wrote: > > - datetime64 objects with high precision (e.g., ns) can't compare to >> datetime objects. >> > > That's a problem, but how do you think it should be handled? My thought is > that it should round to microseconds, and then compare -- kind of like > comparing float32 and float64... > I agree -- if the ns matter, you shouldn't be using datetime.datetime objects. Similarly, it's currently not possible to convert high precision datetime64 objects into datetimes. Worse, this doesn't even raise an error! >>> from datetime import datetime >>> import numpy as np >>> np.datetime64('2000-01-01T00:00:00Z', 'us').astype(datetime) datetime.datetime(2000, 1, 1, 0, 0) >>> np.datetime64('2000-01-01T00:00:00Z', 'ns').astype(datetime) 946684800000000000L Other inconsistent behavior: >>> np.datetime64('2000', 'M') numpy.datetime64('2000-01') >>> np.datetime64('2000', 'D') numpy.datetime64('2000-01-01') >>> np.datetime64('2000', 's') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 np.datetime64('2000', 's') TypeError: Cannot parse "2000" as unit 's' using casting rule 'same_kind' More broadly, my recommendation is to look through the unit tests for pandas' datetIme handling: https://github.com/pydata/pandas/tree/master/pandas/tseries/tests Not everything is relevant but you might find some test cases you could borrow wholesale. Pandas is BSD licensed, so you may even be able to copy them directly into numpy. Best, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Sat Apr 19 02:13:58 2014 From: faltet at gmail.com (Francesc Alted) Date: Sat, 19 Apr 2014 08:13:58 +0200 Subject: [Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed In-Reply-To: <53510EEA.7040405@gmail.com> References: <534FFC20.6010106@gmail.com> <53500F2A.8070105@googlemail.com> <53501DAE.8040007@gmail.com> <5350295E.5040204@googlemail.com> <53510EEA.7040405@gmail.com> Message-ID: <53521426.2030604@gmail.com> El 18/04/14 13:39, Francesc Alted ha escrit: > So, sqrt in numpy has barely the same speed than the one in MKL. > Again, I wonder why :) So by peeking into the code I have seen that you implemented sqrt using SSE2 intrinsics. Cool! -- Francesc Alted From lists at hilboll.de Sat Apr 19 03:03:06 2014 From: lists at hilboll.de (Andreas Hilboll) Date: Sat, 19 Apr 2014 09:03:06 +0200 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> Message-ID: <53521FAA.4030004@hilboll.de> On 14.04.2014 20:59, Chris Barker wrote: > On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer > wrote: > > On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris > > wrote: > > Are we in a position to start looking at implementation? If so, > it would be useful to have a collection of test cases, i.e., > typical uses with specified results. That should also cover > conversion from/(to?) datetime.datetime. > > > yup -- tests are always good! > > Indeed, my personal wish-list for np.datetime64 is centered much > more on robust conversion to/from native date objects, including > comparison. > > > A good use case. > > > Here are some of my particular points of frustration (apologies for > the thread jacking!): > - NaT should have similar behavior to NaN when used for comparisons > (i.e., comparisons should always be False). > > > make sense. > > > - You can't compare a datetime object to a datetime64 object. > > > that would be nice to have. > > > - datetime64 objects with high precision (e.g., ns) can't compare to > datetime objects. > > > That's a problem, but how do you think it should be handled? My thought > is that it should round to microseconds, and then compare -- kind of > like comparing float32 and float64... > > > Pandas has a very nice wrapper around datetime64 arrays that solves > most of these issues, but it would be nice to get much of that > functionality in core numpy, > > > yes -- it would -- but learning from pandas is certainly a good idea. > > > from numpy import datetime64 > from datetime import datetime > > print np.datetime64('NaT') < np.datetime64('2011-01-01') # this > should not to true > print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises > exception > print np.datetime64('2011-01-01T00:00', 'ns') > datetime(2010, 1, 1) > # another exception > print np.datetime64('2011-01-01T00:00') > datetime(2010, 1, 1) # > finally something works! > > > now to get them into proper unit tests.... As one further suggestion, I think it would be nice if doing arithmetic using np.datetime64 and datetime.timedelta objects would work: np.datetime64(2011,1,1) + datetime.timedelta(1) == np.datetime64(2011,1,2) And of course, but this is probably in the loop anyways, np.asarray([list_of_datetime.datetime_objects]) should work as expected. -- Andreas. From alan.isaac at gmail.com Sun Apr 20 10:55:35 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 20 Apr 2014 10:55:35 -0400 Subject: [Numpy-discussion] numerical gradient, Jacobian, and Hessian Message-ID: <5353DFE7.5070007@gmail.com> Awhile back there were good signs that SciPy would end up with a `diff` module: https://github.com/scipy/scipy/issues/2035 Is this still moving forward? It would certainly be nice for SciPy to have intuitive numerical gradients, Jacobians, and Hessians. The last two are I think missing altogether. The first exists as scipy.optimize.approx_fprime. `approx_fprime` seems to work fine, but I suggest it has the following drawbacks: - it is hard to find (e.g., try doing a Google search on "scipy gradient" or "scipy numerical gradient" - related, it is in the wrong location (scipy.optimize) - the signature is odd: (x,f,dx) instead of (f,x,dx) (This matters for ease of recall and for teaching.) In any case, as I understand it, the author's of numdifftools http://code.google.com/p/numdifftools/ expressed willingness to have their code moved into SciPy. This seems like an excellent way forward. There was talk of making this a summer of code project, but that seems to have sputtered. Alan Isaac From siegfried.gonzi at ed.ac.uk Sun Apr 20 11:25:18 2014 From: siegfried.gonzi at ed.ac.uk (Siegfried Gonzi) Date: Sun, 20 Apr 2014 16:25:18 +0100 Subject: [Numpy-discussion] string replace Message-ID: <5353E6DE.30800@ed.ac.uk> Hi all I know this is not numpy related but a colleague insists the following is supposed to work. But it doesn't: == line_left = './erfo/restart.ST010.EN0001-EN0090.YYYYMMDDhh' enafix = 'ST000.EN0001-EN0092' line_left = line_left.replace('STYYY.ENXXXX-ENXXXX', enafix) print 'line_left',line_left == [right answer would be: ./erfo/restart.ST000.EN0001-EN0092.YYYYMMDDhh' ] I think it works in Fortran but not in Python. What would be the easiest way to replacing this kind of pattern in a variable lenght string? Thanks, Siegfried -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From robert.kern at gmail.com Sun Apr 20 11:58:53 2014 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 20 Apr 2014 16:58:53 +0100 Subject: [Numpy-discussion] string replace In-Reply-To: <5353E6DE.30800@ed.ac.uk> References: <5353E6DE.30800@ed.ac.uk> Message-ID: On Sun, Apr 20, 2014 at 4:25 PM, Siegfried Gonzi wrote: > Hi all > > I know this is not numpy related but a colleague insists the following > is supposed to work. But it doesn't: > > == > line_left = './erfo/restart.ST010.EN0001-EN0090.YYYYMMDDhh' > enafix = 'ST000.EN0001-EN0092' > line_left = line_left.replace('STYYY.ENXXXX-ENXXXX', enafix) > print 'line_left',line_left > == > > [right answer would be: ./erfo/restart.ST000.EN0001-EN0092.YYYYMMDDhh' ] > > I think it works in Fortran but not in Python. What would be the easiest > way to replacing this kind of pattern in a variable lenght string? Your colleague is incorrect. There is nothing in Python that would take a pattern like that (with Xs and Ys as the wildcards) and do a replacement. The `str.replace()` method only does exact replacements. https://docs.python.org/2/library/stdtypes.html#str.replace You need to use regular expressions here. https://docs.python.org/2/library/re.html ``` import re line_left = './erfo/restart.ST010.EN0001-EN0090.YYYYMMDDhh' enafix = 'ST000.EN0001-EN0092' line_left = re.sub(r'ST\d\d\d\.EN\d\d\d\d-EN\d\d\d\d', enafix, line_left) print 'line_left',line_left ``` I will leave the reading about constructing correct regular expressions to you, but if you try my example code, please note that every character is precisely important. -- Robert Kern From hoogendoorn.eelco at gmail.com Mon Apr 21 03:13:33 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Mon, 21 Apr 2014 09:13:33 +0200 Subject: [Numpy-discussion] numerical gradient, Jacobian, and Hessian In-Reply-To: <5353DFE7.5070007@gmail.com> References: <5353DFE7.5070007@gmail.com> Message-ID: I was going to suggest numdifftools; its a very capable package in my experience. Indeed it would be nice to have it integrated into scipy. Also, in case trying to calculate a numerical gradient is a case of 'the math getting too bothersome' rather than no closed form gradient actually existing: Theano may be your best bet; I have very good experiences with it as well. As far as I can tell, it is actually the only tensor/ndarray aware differentiator out there (maple and mathematica don't appear to support this) On Sun, Apr 20, 2014 at 4:55 PM, Alan G Isaac wrote: > Awhile back there were good signs that SciPy > would end up with a `diff` module: > https://github.com/scipy/scipy/issues/2035 > Is this still moving forward? > > It would certainly be nice for SciPy to have intuitive > numerical gradients, Jacobians, and Hessians. The last > two are I think missing altogether. The first exists > as scipy.optimize.approx_fprime. > > `approx_fprime` seems to work fine, but I suggest it > has the following drawbacks: > - it is hard to find (e.g., try doing a Google search > on "scipy gradient" or "scipy numerical gradient" > - related, it is in the wrong location (scipy.optimize) > - the signature is odd: (x,f,dx) instead of (f,x,dx) > (This matters for ease of recall and for teaching.) > > In any case, as I understand it, the author's of numdifftools > http://code.google.com/p/numdifftools/ > expressed willingness to have their code moved into SciPy. > This seems like an excellent way forward. > There was talk of making this a summer of code project, > but that seems to have sputtered. > > Alan Isaac > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Mon Apr 21 03:25:57 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Mon, 21 Apr 2014 09:25:57 +0200 Subject: [Numpy-discussion] string replace In-Reply-To: <5353E6DE.30800@ed.ac.uk> References: <5353E6DE.30800@ed.ac.uk> Message-ID: Indeed this isn't numpy, and I don't see how your collegues opinions have bearing on that issue; but anyway.. There isn't a 'python' way to do this, the best method involves some form of parsing library. Undoubtly there is a one-line regex to do this kind of thing, but regexes are themselves the antithesis of python. Here is how id do it, using pyparsing: from pyparsing import * line_left = './erfo/restart.ST010.EN0001-EN0090.YYYYMMDDhh' n1 = Word(nums,exact=3).setParseAction(replaceWith('000')) n2 = Word(nums,exact=4).setParseAction(replaceWith('0001')) n3 = Word(nums,exact=4).setParseAction(replaceWith('0092')) pattern = Literal('ST')+ n1 +'.EN'+n2+'-EN'+n3 print pattern.transformString(line_left) On Sun, Apr 20, 2014 at 5:25 PM, Siegfried Gonzi wrote: > Hi all > > I know this is not numpy related but a colleague insists the following > is supposed to work. But it doesn't: > > == > line_left = './erfo/restart.ST010.EN0001-EN0090.YYYYMMDDhh' > enafix = 'ST000.EN0001-EN0092' > line_left = line_left.replace('STYYY.ENXXXX-ENXXXX', enafix) > print 'line_left',line_left > == > > [right answer would be: ./erfo/restart.ST000.EN0001-EN0092.YYYYMMDDhh' ] > > I think it works in Fortran but not in Python. What would be the easiest > way to replacing this kind of pattern in a variable lenght string? > > > Thanks, > Siegfried > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Apr 21 05:55:05 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 21 Apr 2014 11:55:05 +0200 Subject: [Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda In-Reply-To: References: Message-ID: On Thu, Mar 27, 2014 at 10:11 PM, Alex Goodman wrote: > Hi Robert, > > That did the trick, thanks! > > Alex > > > On Thu, Mar 27, 2014 at 3:02 PM, Robert Kern wrote: > >> On Thu, Mar 27, 2014 at 8:50 PM, David Cournapeau >> wrote: >> > >> > On Thu, Mar 27, 2014 at 8:30 PM, Alex Goodman < >> alex.goodman at colostate.edu> >> > wrote: >> >> >> >> Hi all, >> >> >> >> I have used f2py in the past on a Linux machine with virtually no >> issues. >> >> However on my Mac, I get the following error when importing an f2py >> >> generated extension: >> >> >> >> Fatal Python error: PyThreadState_Get: no current thread >> >> Abort trap: 6 >> >> >> >> After doing some research I found out that the extension is linked to >> the >> >> wrong python installation: >> >> otool -L add.so >> >> add.so: >> >> ./add.so (compatibility version 0.0.0, current version 0.0.0) >> >> /System/Library/Frameworks/Python.framework/Versions/2.7/Python >> >> (compatibility version 2.7.0, current version 2.7.2) >> >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version >> >> 169.3.0) >> >> >> >> This seems odd because I am using the f2py executable included in >> Anaconda >> >> 1.9.1. I can easily fix this problem by manually using >> install_name_tool >> >> -change on the extension to link the correct library location, but >> this is >> >> really cumbersome. Is there an alternative solution, such as an >> additional >> >> command-line argument when invoking f2py? >> > >> > >> > This sounds like an issue specific to Anaconda, and you may get better >> > support on the Anaconda support ML. >> >> I think it's our bug. numpy.distutils adds an explicit `-framework >> Python` in the Intel Fortran link line. We should be just be using >> `-undefined dynamic_lookup`. >> >> >> https://github.com/numpy/numpy/blob/master/numpy/distutils/fcompiler/intel.py#L71 >> >> Alex, can you edit that file to remove the '-Wl,-framework,Python' >> from that list and try building again? >> > Submitted a PR for this: https://github.com/numpy/numpy/pull/4630. Thanks Robert. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From argriffi at ncsu.edu Mon Apr 21 11:29:54 2014 From: argriffi at ncsu.edu (alex) Date: Mon, 21 Apr 2014 11:29:54 -0400 Subject: [Numpy-discussion] numerical gradient, Jacobian, and Hessian In-Reply-To: References: <5353DFE7.5070007@gmail.com> Message-ID: On Mon, Apr 21, 2014 at 3:13 AM, Eelco Hoogendoorn wrote: > As far as I can tell, [Theano] is actually the only tensor/ndarray aware differentiator out there And AlgoPy, a tensor/ndarray aware arbitrary order automatic differentiator (https://pythonhosted.org/algopy/) From ralf.gommers at gmail.com Mon Apr 21 16:25:48 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 21 Apr 2014 22:25:48 +0200 Subject: [Numpy-discussion] Proposed new function for joining arrays: np.interleave In-Reply-To: References: Message-ID: On Mon, Apr 7, 2014 at 5:12 PM, Bj?rn Dahlgren wrote: > Hello, > > Interleaving arrays is something I need to do every now and then, and by > the looks of stackoverflow so do others: > > > http://stackoverflow.com/questions/12861314/interleave-rows-of-two-numpy-arrays-in-python > http://stackoverflow.com/questions/5347065/interweaving-two-numpy-arrays > > I think the code needed for the general n dimensional case with m number > of arrays > is non-trivial enough for it to be useful to provide such a function in > numpy, so I took the liberty to > make a Pull Request with my > implementation. This would be my first contribution to NumPy so I apologize > if something is not adhering to your practices. > It looks pretty good to me for a first contribution. And I'm fine with adding this function, should see a reasonable amount of usage. Cheers, Ralf > Jaime has already pointed out that I should email this list (I hope I > managed to email the list in > the right way - never used a mailing list before) for you to notice the > pull request. He also pointed > out some improvements of my proposed implementation (not forcing > consistent dtype), but before > I go on and make changes I might need to ask you guys if this is really > something you are interested in including? > > Best regards, > /Bj?rn Dahlgren > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.collette at gmail.com Tue Apr 22 09:25:49 2014 From: andrew.collette at gmail.com (Andrew Collette) Date: Tue, 22 Apr 2014 07:25:49 -0600 Subject: [Numpy-discussion] ANN: HDF5 for Python 2.3.0 Message-ID: Announcing HDF5 for Python (h5py) 2.3.0 ======================================= The h5py team is happy to announce the availability of h5py 2.3.0 (final). Thanks to everyone who provided beta feedback! What's h5py? ------------ The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Changes ------- This release introduces some important new features, including: * Support for arbitrary vlen data * Improved exception messages * Improved setuptools support * Multiple additions to the low-level API * Improved support for MPI features * Single-step build for HDF5 on Windows Major fixes since beta: * LZF compression crash on Win64 * Unhelpful error message relating to chunked storage * Import error for IPython completer on certain platforms A complete description of changes is available online: http://docs.h5py.org/en/latest/whatsnew/2.3.html Where to get it --------------- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org Acknowledgements ---------------- The h5py package relies on third-party testing and contributions. For the 2.3 release, thanks especially to: * Martin Teichmann * Florian Rathgerber * Pierre de Buyl * Thomas Caswell * Andy Salnikov * Darren Dale * Robert David Grant * Toon Verstraelen * Many others who contributed bug reports and testing From matthieu.brucher at gmail.com Tue Apr 22 09:33:31 2014 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 22 Apr 2014 14:33:31 +0100 Subject: [Numpy-discussion] [Hdf-forum] ANN: HDF5 for Python 2.3.0 In-Reply-To: References: Message-ID: Good work! Small question : do you now have the interface to set alignment? Cheers, Matthieu 2014-04-22 14:25 GMT+01:00 Andrew Collette : > Announcing HDF5 for Python (h5py) 2.3.0 > ======================================= > > The h5py team is happy to announce the availability of h5py 2.3.0 (final). > Thanks to everyone who provided beta feedback! > > What's h5py? > ------------ > > The h5py package is a Pythonic interface to the HDF5 binary data format. > > It lets you store huge amounts of numerical data, and easily manipulate > that data from NumPy. For example, you can slice into multi-terabyte > datasets stored on disk, as if they were real NumPy arrays. Thousands of > datasets can be stored in a single file, categorized and tagged however > you want. > > Changes > ------- > > This release introduces some important new features, including: > > * Support for arbitrary vlen data > * Improved exception messages > * Improved setuptools support > * Multiple additions to the low-level API > * Improved support for MPI features > * Single-step build for HDF5 on Windows > > Major fixes since beta: > > * LZF compression crash on Win64 > * Unhelpful error message relating to chunked storage > * Import error for IPython completer on certain platforms > > A complete description of changes is available online: > > http://docs.h5py.org/en/latest/whatsnew/2.3.html > > Where to get it > --------------- > > Downloads, documentation, and more are available at the h5py website: > > http://www.h5py.org > > Acknowledgements > ---------------- > > The h5py package relies on third-party testing and contributions. For the > 2.3 release, thanks especially to: > > * Martin Teichmann > * Florian Rathgerber > * Pierre de Buyl > * Thomas Caswell > * Andy Salnikov > * Darren Dale > * Robert David Grant > * Toon Verstraelen > * Many others who contributed bug reports and testing > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum at lists.hdfgroup.org > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ From andrew.collette at gmail.com Tue Apr 22 10:45:12 2014 From: andrew.collette at gmail.com (Andrew Collette) Date: Tue, 22 Apr 2014 08:45:12 -0600 Subject: [Numpy-discussion] [Hdf-forum] ANN: HDF5 for Python 2.3.0 In-Reply-To: References: Message-ID: Hi, > Good work! > Small question : do you now have the interface to set alignment? Unfortunately this didn't make it in to 2.3. Pull requests are welcome for this and other MPI features! Andrew From matthieu.brucher at gmail.com Tue Apr 22 14:40:52 2014 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 22 Apr 2014 19:40:52 +0100 Subject: [Numpy-discussion] [Hdf-forum] ANN: HDF5 for Python 2.3.0 In-Reply-To: References: Message-ID: OK, I may end up doing it, as it can be quite interesting! Cheers, Matthieu 2014-04-22 15:45 GMT+01:00 Andrew Collette : > Hi, > >> Good work! >> Small question : do you now have the interface to set alignment? > > Unfortunately this didn't make it in to 2.3. Pull requests are > welcome for this and other MPI features! > > Andrew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ From charlesr.harris at gmail.com Tue Apr 22 17:35:20 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Apr 2014 15:35:20 -0600 Subject: [Numpy-discussion] 1.9.x branch Message-ID: Hi All, I'd like to branch 1.9.x at the end of the month. There are a couple of reasons for the timing. First, we have a lot of new stuff in the development branch. Second, there is work ongoing in masked arrays that I'd like to keep out of the release so that it has more time to settle. Third, it's past time ;) There are currently a number of 1.9.0 blockers, which can be seen here. *Datetime timezone handling broken in 1.7.x* I don't think there is time to get this done for 1.9.0 and it needs to be pushed off to 1.10.0. *Return multiple field selection as ro view * I have a branch for this, but because the returned type differs from a copy by alignment spacing there was a test failure. Merging that branch might cause some incompatibilities. *Object array creation new conversion to int* This one needs a decision. Julian, Sebastian, thoughts? *Median of np.matrix is broken* Not sure what the status of this one is. *1.8 deprecations: Follow-up ticket * Things that might should be removed. *ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 * I believe this one was fixed. For general problems reading/writing big files on OS X, I believe they were fixed in Maverick and I'm inclined to recommend an OS upgrade rather than work to chunk all the io. Deprecate NPY_CHAR This one is waiting on a fix from Pearu to make f2py use numpy strings. I think we will need to do this ourselves if we want to carry through the deprecation. In any case it probably needs to be pushed off to 1.10. 1.7 deprecations: Follow-up ticketSome of these changes have been made, ro diagonal view for instance, some still remain. Additions, updates, and thoughts welcome. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Apr 23 01:22:36 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 22 Apr 2014 22:22:36 -0700 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? Message-ID: Hi, I'm exploring Mingw-w64 for numpy building, and I've found it gives a slightly different answer for 'exp' than - say - gcc on OSX. The difference is of the order of the eps value for the output number (2 * eps for a result of ~2.0). Is accuracy somewhere specified for C functions like exp? Or is accuracy left as an implementation detail for the C library author? Cheers, Matthew From njs at pobox.com Wed Apr 23 04:43:35 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Apr 2014 09:43:35 +0100 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett wrote: > Hi, > > I'm exploring Mingw-w64 for numpy building, and I've found it gives a > slightly different answer for 'exp' than - say - gcc on OSX. > > The difference is of the order of the eps value for the output number > (2 * eps for a result of ~2.0). > > Is accuracy somewhere specified for C functions like exp? Or is > accuracy left as an implementation detail for the C library author? C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point operations ... and of the library functions in and that return floating point results is implemenetation defined. The implementation may state that the accuracy is unknown." (This last sentence is basically saying that with regard to some higher up clauses that required all conforming implementations to document this stuff, saying "eh, who knows" counts as documenting it. Hooray for standards!) Presumably the accuracy in this case is a function of the C library anyway, not the compiler? Numpy has its own implementations for a bunch of the math functions, and it's been unclear in the past whether numpy or the libc implementations were better in any particular case. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From cournape at gmail.com Wed Apr 23 05:01:26 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 23 Apr 2014 10:01:26 +0100 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: On Wed, Apr 23, 2014 at 9:43 AM, Nathaniel Smith wrote: > On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett > wrote: > > Hi, > > > > I'm exploring Mingw-w64 for numpy building, and I've found it gives a > > slightly different answer for 'exp' than - say - gcc on OSX. > > > > The difference is of the order of the eps value for the output number > > (2 * eps for a result of ~2.0). > > > > Is accuracy somewhere specified for C functions like exp? Or is > > accuracy left as an implementation detail for the C library author? > > C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point > operations ... and of the library functions in and > that return floating point results is implemenetation > defined. The implementation may state that the accuracy is unknown." > (This last sentence is basically saying that with regard to some > higher up clauses that required all conforming implementations to > document this stuff, saying "eh, who knows" counts as documenting it. > Hooray for standards!) > > Presumably the accuracy in this case is a function of the C library > anyway, not the compiler? Numpy has its own implementations for a > bunch of the math functions, and it's been unclear in the past whether > numpy or the libc implementations were better in any particular case. > In the case of MS runtime, at least 9 (as shipped in VS 2008), our implementation is likely to be better (most of the code was taken from the sun math library when the license allowed it). David > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Apr 23 05:32:03 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 23 Apr 2014 11:32:03 +0200 Subject: [Numpy-discussion] 1.9.x branch In-Reply-To: References: Message-ID: <1398245523.4282.18.camel@sebastian-t440> On Di, 2014-04-22 at 15:35 -0600, Charles R Harris wrote: > Hi All, > > > I'd like to branch 1.9.x at the end of the month. There are a couple > of reasons for the timing. First, we have a lot of new stuff in the > development branch. Second, there is work ongoing in masked arrays > that I'd like to keep out of the release so that it has more time to > settle. Third, it's past time ;) Sounds good. > There are currently a number of 1.9.0 blockers, which can be seen > here. > > Datetime timezone handling broken in 1.7.x > > I don't think there is time to get this done for 1.9.0 and it needs to > be pushed off to 1.10.0. > > Return multiple field selection as ro view > > I have a branch for this, but because the returned type differs from a > copy by alignment spacing there was a test failure. Merging that > branch might cause some incompatibilities. > I am a bit worried here that comparisons might make trouble. > Object array creation new conversion to int > > > This one needs a decision. Julian, Sebastian, thoughts? > Maybe for all to consider this is about what happens for object arrays if you do things like: # Array cast to object array (np.array(arr) would be identical): a = np.arange(10).astype(object) # Array passed into new array creation (not just *one* array): b = np.array([np.arange(10)], dtype=object) # Numerical array is assigned to object array: c = np.empty(10, dtype=object) c[...] = np.arange(10) Before this change, the result was: type(a[0]) is int type(b[0,0]) is np.int_ # Note the numpy type type(c[0]) is int After this change, they are all `int`. Though note that the numpy type is preserved for example for long double. On the one hand preserving the numpy type might be nice, but on the other hand we don't care much about the dtypes of scalars and in practice the python types are probably more often wanted. Since I just realized that things are safe (float128 does not cast to float after all), I changed my mind and am tempted to keep the new behaviour. That is, if it does not create any problems (there was some issue in scipy, not sure how bad). - Sebastian > Median of np.matrix is broken( > > > Not sure what the status of this one is. > > 1.8 deprecations: Follow-up ticket > > > Things that might should be removed. > > ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 > > > I believe this one was fixed. For general problems reading/writing big > files on OS X, I believe they were fixed in Maverick and I'm inclined > to recommend an OS upgrade rather than work to chunk all the io. > > Deprecate NPY_CHAR > This one is waiting on a fix from Pearu to make f2py use numpy > strings. I think we will need to do this ourselves if we want to carry > through the deprecation. In any case it probably needs to be pushed > off to 1.10. > > 1.7 deprecations: Follow-up ticket > Some of these changes have been made, ro diagonal view for instance, > some still remain. > > > > Additions, updates, and thoughts welcome. > > > Chuck > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From josef.pktd at gmail.com Wed Apr 23 10:11:22 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 23 Apr 2014 10:11:22 -0400 Subject: [Numpy-discussion] 1.9.x branch In-Reply-To: <1398245523.4282.18.camel@sebastian-t440> References: <1398245523.4282.18.camel@sebastian-t440> Message-ID: On Wed, Apr 23, 2014 at 5:32 AM, Sebastian Berg wrote: > On Di, 2014-04-22 at 15:35 -0600, Charles R Harris wrote: >> Hi All, >> >> >> I'd like to branch 1.9.x at the end of the month. There are a couple >> of reasons for the timing. First, we have a lot of new stuff in the >> development branch. Second, there is work ongoing in masked arrays >> that I'd like to keep out of the release so that it has more time to >> settle. Third, it's past time ;) > > Sounds good. > >> There are currently a number of 1.9.0 blockers, which can be seen >> here. >> >> Datetime timezone handling broken in 1.7.x >> >> I don't think there is time to get this done for 1.9.0 and it needs to >> be pushed off to 1.10.0. >> >> Return multiple field selection as ro view >> >> I have a branch for this, but because the returned type differs from a >> copy by alignment spacing there was a test failure. Merging that >> branch might cause some incompatibilities. >> > > I am a bit worried here that comparisons might make trouble. > >> Object array creation new conversion to int >> >> >> This one needs a decision. Julian, Sebastian, thoughts? >> > > Maybe for all to consider this is about what happens for object arrays > if you do things like: > > # Array cast to object array (np.array(arr) would be identical): > a = np.arange(10).astype(object) > # Array passed into new array creation (not just *one* array): > b = np.array([np.arange(10)], dtype=object) > # Numerical array is assigned to object array: > c = np.empty(10, dtype=object) > c[...] = np.arange(10) > > Before this change, the result was: > type(a[0]) is int > type(b[0,0]) is np.int_ # Note the numpy type > type(c[0]) is int > > After this change, they are all `int`. Though note that the numpy type > is preserved for example for long double. On the one hand preserving the > numpy type might be nice, but on the other hand we don't care much about > the dtypes of scalars and in practice the python types are probably more > often wanted. what if I don't like python? >>> np.int_(0)**(-1) inf >>> 0**-1 Traceback (most recent call last): File "", line 1, in 0**-1 ZeroDivisionError: 0.0 cannot be raised to a negative power >>> type(np.arange(5)[0]) >>> np.arange(5)[0]**-1 inf >>> type(np.arange(5)[0].item()) >>> np.arange(5)[0].item()**-1 Traceback (most recent call last): File "", line 1, in np.arange(5)[0].item()**-1 ZeroDivisionError: 0.0 cannot be raised to a negative power >>> np.__version__ '1.6.1' I remember struggling through this (avoiding python operations) quite a bit in my early bugfixes to scipy.stats.distributions. (IIRC I ended up avoiding most ints.) Josef > > Since I just realized that things are safe (float128 does not cast to > float after all), I changed my mind and am tempted to keep the new > behaviour. That is, if it does not create any problems (there was some > issue in scipy, not sure how bad). > > - Sebastian > >> Median of np.matrix is broken( >> >> >> Not sure what the status of this one is. >> >> 1.8 deprecations: Follow-up ticket >> >> >> Things that might should be removed. >> >> ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 >> >> >> I believe this one was fixed. For general problems reading/writing big >> files on OS X, I believe they were fixed in Maverick and I'm inclined >> to recommend an OS upgrade rather than work to chunk all the io. >> >> Deprecate NPY_CHAR >> This one is waiting on a fix from Pearu to make f2py use numpy >> strings. I think we will need to do this ourselves if we want to carry >> through the deprecation. In any case it probably needs to be pushed >> off to 1.10. >> >> 1.7 deprecations: Follow-up ticket >> Some of these changes have been made, ro diagonal view for instance, >> some still remain. >> >> >> >> Additions, updates, and thoughts welcome. >> >> >> Chuck >> >> >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ndbecker2 at gmail.com Wed Apr 23 11:01:43 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 23 Apr 2014 11:01:43 -0400 Subject: [Numpy-discussion] numerical gradient, Jacobian, and Hessian References: <5353DFE7.5070007@gmail.com> Message-ID: alex wrote: > On Mon, Apr 21, 2014 at 3:13 AM, Eelco Hoogendoorn > wrote: >> As far as I can tell, [Theano] is actually the only tensor/ndarray aware >> differentiator out there > > And AlgoPy, a tensor/ndarray aware arbitrary order automatic > differentiator (https://pythonhosted.org/algopy/) I noticed julia seems to have a package From ksmith at enthought.com Wed Apr 23 13:58:21 2014 From: ksmith at enthought.com (Kurt Smith) Date: Wed, 23 Apr 2014 12:58:21 -0500 Subject: [Numpy-discussion] ANN: DistArray 0.2 -- first public development release Message-ID: GitHub repo: https://github.com/enthought/distarray Documentation: http://distarray.readthedocs.org License: Three-clause BSD Python versions: 2.7 and 3.3 OS support: *nix and Mac OS X DistArray aims to bring the strengths of NumPy to data-parallel high-performance computing. It provides distributed multi-dimensional NumPy-like arrays and distributed ufuncs, distributed IO capabilities, and can integrate with external distributed libraries, like Trilinos. DistArray works with NumPy and builds on top of it in a flexible and natural way. Brian Granger started DistArray as a NASA-funded SBIR project in 2008. Enthought picked it up as part of a DOE Phase II SBIR [0] to provide a generally useful distributed array package. It builds on IPython, IPython.parallel, NumPy, MPI, and interfaces with the Trilinos suite of distributed HPC solvers (via PyTrilinos) [1]. Distarray: - has a client-engine (or master-worker) process design -- data resides on the worker processes, commands are initiated from master; - allows full control over what is executed on the worker processes and integrates transparently with the master process; - allows direct communication between workers bypassing the master process for scalability; - integrates with IPython.parallel for interactive creation and exploration of distributed data; - supports distributed ufuncs (currently without broadcasting); - builds on and leverages MPI via MPI4Py in a transparent and user-friendly way; - supports NumPy-like structured multidimensional arrays; - has basic support for unstructured arrays; - supports user-controllable array distributions across workers (block, cyclic, block-cyclic, and unstructured) on a per-axis basis; - has a straightforward API to control how an array is distributed; - has basic plotting support for visualization of array distributions; - separates the array?s distribution from the array?s data -- useful for slicing, reductions, redistribution, broadcasting, all of which will be implemented in coming releases; - implements distributed random arrays; - supports `.npy`-like flat-file IO and hdf5 parallel IO (via h5py); leverages MPI-based IO parallelism in an easy-to-use and transparent way; and - supports the distributed array protocol[2], which allows independently developed parallel libraries to share distributed arrays without copying, analogous to the PEP-3118 new buffer protocol. This is the first public development release. DistArray is not ready for real-world use, but we want to get input from the larger scientific-Python community to help drive its development. The API is changing rapidly and we are adding many new features on a fast timescale. For that reason, DistArray is currently implemented in pure Python for maximal flexibility. Performance improvements are coming. The 0.2 release's goals are to provide the components necessary to support upcoming features that are non-trivial to implement in a distributed environment. Planned features for upcoming releases: - Distributed reductions - Distributed slicing - Distributed broadcasting - Distributed fancy indexing - Re-distribution methods - Integration with Trilinos [1] and other packages [3] that are compatible with the distributed array protocol [2] - Lazy evaluation and deferred computation for latency hiding - Out-of-core computations - Extensive examples, tutorials, documentation - Support for distributed sorting and other non-trivial distributed algorithms - MPI-only communication for non-interactive deployment on clusters and supercomputers - End-user control over communication, temporary array creation, and other performance aspects of distributed computations [0] http://www.sbir.gov/sbirsearch/detail/410257 [1] http://trilinos.org/ [2] http://distributed-array-protocol.readthedocs.org/en/rel-0.10.0/ [3] http://www.mcs.anl.gov/petsc/ -- Kurt W. Smith, Ph.D. Enthought, Inc. | 512.536.1057 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Apr 23 14:59:23 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Apr 2014 11:59:23 -0700 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: Hi, On Wed, Apr 23, 2014 at 1:43 AM, Nathaniel Smith wrote: > On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett wrote: >> Hi, >> >> I'm exploring Mingw-w64 for numpy building, and I've found it gives a >> slightly different answer for 'exp' than - say - gcc on OSX. >> >> The difference is of the order of the eps value for the output number >> (2 * eps for a result of ~2.0). >> >> Is accuracy somewhere specified for C functions like exp? Or is >> accuracy left as an implementation detail for the C library author? > > C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point > operations ... and of the library functions in and > that return floating point results is implemenetation > defined. The implementation may state that the accuracy is unknown." > (This last sentence is basically saying that with regard to some > higher up clauses that required all conforming implementations to > document this stuff, saying "eh, who knows" counts as documenting it. > Hooray for standards!) > > Presumably the accuracy in this case is a function of the C library > anyway, not the compiler? Mingw-w64 implementation is in assembly: http://sourceforge.net/p/mingw-w64/code/HEAD/tree/trunk/mingw-w64-crt/math/exp.def.h > Numpy has its own implementations for a > bunch of the math functions, and it's been unclear in the past whether > numpy or the libc implementations were better in any particular case. I only investigated this particular value, in which case it looked as though the OSX value was closer to the exact value (via sympy.mpmath) - by ~1 unit-at-the-last-place. This was causing a divergence in the powell optimization path and therefore a single scipy test failure. I haven't investigated further - was wondering what investigation I should do, more than running the numpy / scipy test suites. Cheers, Matthew From matthew.brett at gmail.com Wed Apr 23 15:25:34 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Apr 2014 12:25:34 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: Hi, On Tue, Apr 15, 2014 at 12:34 AM, Julian Taylor wrote: > On Tue, Apr 15, 2014 at 4:30 AM, Matthew Brett wrote: >> >> It looks as though mingw-w64 is at fault, and I was confused (still >> am) because of the different behavior with double and a constant: >> >> #include >> #include >> >> int main() { >> double z, i = -0.0; >> printf("With double %f=%f, with constant=%f\n", >> i, expm1(i), expm1(-0.)); >> } >> >> gives: >> >> With double -0.000000=0.000000, with constant=-0.000000 >> >> That was ugly to track down. >> >> What is the right way to work round this (using the numpy version >> instead of the system version I suppose)? >> > > The right way is to file a bug at mingw and get it fixed at the source. http://sourceforge.net/p/mingw-w64/code/6594/ > Additionally as this time npymath seems to be better (thats 3 bugs in > npymath vs 1 in mingw on my scoreboard) one could use the mingw > preprocessor define instead of HAVE_EXP1M to select this function from > npymath. Sorry to be slow - could you unpack what you mean in this paragraph? Is there a way to force use of the numpy version of the function using environment variables or similar? Cheers, Matthew From chris.barker at noaa.gov Wed Apr 23 17:14:04 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Apr 2014 14:14:04 -0700 Subject: [Numpy-discussion] 1.9.x branch In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 2:35 PM, Charles R Harris wrote: > *Datetime timezone handling broken in 1.7.x* > > I don't think there is time to get this done for 1.9.0 and it needs to be > pushed off to 1.10.0. > > * * > Darn! that's what we said for 1.8. However, Sankarshan Mudkavi has written up a NEP, and this is really a matter of ripping code out, rather than adding much. Someone familiar with the code should be able to whip this out pretty easily (I think we've abandoned for now any hope of "proper TZ handling) ( https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst ) Datetime64 is really pretty broken this way -- the sooner it gets cleaned up the better. https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst The trick is that as simple as it may be, someone still needs to do it. I, for one, am not familiar with the code, and have pathetic C skills anyway, so it would not be an efficient use of my time to try to do it. but I will go and edit the NEP and write test cases! -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Wed Apr 23 17:27:03 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 23 Apr 2014 23:27:03 +0200 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: References: <534C4CE4.1030407@googlemail.com> Message-ID: <53583027.4000103@googlemail.com> On 23.04.2014 21:25, Matthew Brett wrote: > Hi, > > On Tue, Apr 15, 2014 at 12:34 AM, Julian Taylor > wrote: >> On Tue, Apr 15, 2014 at 4:30 AM, Matthew Brett wrote: >>> >>> It looks as though mingw-w64 is at fault, and I was confused (still >>> am) because of the different behavior with double and a constant: >>> >>> #include >>> #include >>> >>> int main() { >>> double z, i = -0.0; >>> printf("With double %f=%f, with constant=%f\n", >>> i, expm1(i), expm1(-0.)); >>> } >>> >>> gives: >>> >>> With double -0.000000=0.000000, with constant=-0.000000 >>> >>> That was ugly to track down. >>> >>> What is the right way to work round this (using the numpy version >>> instead of the system version I suppose)? >>> >> >> The right way is to file a bug at mingw and get it fixed at the source. > > http://sourceforge.net/p/mingw-w64/code/6594/ great thanks for reporting it. > >> Additionally as this time npymath seems to be better (thats 3 bugs in >> npymath vs 1 in mingw on my scoreboard) one could use the mingw >> preprocessor define instead of HAVE_EXP1M to select this function from >> npymath. > > Sorry to be slow - could you unpack what you mean in this paragraph? > Is there a way to force use of the numpy version of the function using > environment variables or similar? > I mean something along the line of attached patch, maybe you can try it. If it works I can file a numpy PR. -------------- next part -------------- A non-text attachment was scrubbed... Name: mingw.patch Type: text/x-diff Size: 1578 bytes Desc: not available URL: From chris.barker at noaa.gov Wed Apr 23 17:58:45 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Apr 2014 14:58:45 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> Message-ID: On Wed, Mar 19, 2014 at 7:07 PM, Sankarshan Mudkavi wrote: > > I've written a rather rudimentary NEP, (lacking in technical details which > I will hopefully add after some further discussion and receiving > clarification/help on this thread). > > Please let me know how to proceed and what you think should be added to > the current proposal (attached to this mail). > > Here is a rendered version of the same: > > https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst > I've done a bit of copy-editing, and added some more from this discussion. See the pull request on gitHub. There are a fair number of rough edges, but I think we have a consensus among the small group of folks that participated in this discussion anyway, so now "all" we need is someone to actually fix the code. If someone steps up, then we should also go in and add a bunch of unit tests, as discussed in this thread. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From smudkavi at uwaterloo.ca Wed Apr 23 23:23:14 2014 From: smudkavi at uwaterloo.ca (Sankarshan Mudkavi) Date: Wed, 23 Apr 2014 23:23:14 -0400 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> Message-ID: <5629C066-E1DA-4293-94AC-6ACC7F1AC87A@uwaterloo.ca> Thank you very much, I will incorporate it! I've been quite busy for the past few weeks but I should be much freer after next week and can pick up on this (fixing the code and actually implement things). Cheers, Sankarshan On Apr 23, 2014, at 5:58 PM, Chris Barker wrote: > On Wed, Mar 19, 2014 at 7:07 PM, Sankarshan Mudkavi wrote: > > I've written a rather rudimentary NEP, (lacking in technical details which I will hopefully add after some further discussion and receiving clarification/help on this thread). > > Please let me know how to proceed and what you think should be added to the current proposal (attached to this mail). > > Here is a rendered version of the same: > https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst > > I've done a bit of copy-editing, and added some more from this discussion. See the pull request on gitHub. > > There are a fair number of rough edges, but I think we have a consensus among the small group of folks that participated in this discussion anyway, so now "all" we need is someone to actually fix the code. > > If someone steps up, then we should also go in and add a bunch of unit tests, as discussed in this thread. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From matthew.brett at gmail.com Thu Apr 24 01:54:20 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Apr 2014 22:54:20 -0700 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: <53583027.4000103@googlemail.com> References: <534C4CE4.1030407@googlemail.com> <53583027.4000103@googlemail.com> Message-ID: Hi, On Wed, Apr 23, 2014 at 2:27 PM, Julian Taylor wrote: > On 23.04.2014 21:25, Matthew Brett wrote: >> Hi, >> >> On Tue, Apr 15, 2014 at 12:34 AM, Julian Taylor >> wrote: >>> On Tue, Apr 15, 2014 at 4:30 AM, Matthew Brett wrote: >>>> >>>> It looks as though mingw-w64 is at fault, and I was confused (still >>>> am) because of the different behavior with double and a constant: >>>> >>>> #include >>>> #include >>>> >>>> int main() { >>>> double z, i = -0.0; >>>> printf("With double %f=%f, with constant=%f\n", >>>> i, expm1(i), expm1(-0.)); >>>> } >>>> >>>> gives: >>>> >>>> With double -0.000000=0.000000, with constant=-0.000000 >>>> >>>> That was ugly to track down. >>>> >>>> What is the right way to work round this (using the numpy version >>>> instead of the system version I suppose)? >>>> >>> >>> The right way is to file a bug at mingw and get it fixed at the source. >> >> http://sourceforge.net/p/mingw-w64/code/6594/ > > great thanks for reporting it. > >> >>> Additionally as this time npymath seems to be better (thats 3 bugs in >>> npymath vs 1 in mingw on my scoreboard) one could use the mingw >>> preprocessor define instead of HAVE_EXP1M to select this function from >>> npymath. >> >> Sorry to be slow - could you unpack what you mean in this paragraph? >> Is there a way to force use of the numpy version of the function using >> environment variables or similar? >> > > I mean something along the line of attached patch, maybe you can try it. > If it works I can file a numpy PR. Thanks for the patch - I see what you mean now. I wonder if there should be some mechanism by which different compilers can set these? What was the issue with MSVC for example - does it need something like this too? Cheers, Matthew From chris.barker at noaa.gov Thu Apr 24 12:26:35 2014 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 24 Apr 2014 09:26:35 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <5629C066-E1DA-4293-94AC-6ACC7F1AC87A@uwaterloo.ca> References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <5629C066-E1DA-4293-94AC-6ACC7F1AC87A@uwaterloo.ca> Message-ID: <3491667248167119497@unknownmsgid> On Apr 23, 2014, at 8:23 PM, Sankarshan Mudkavi wrote: I've been quite busy for the past few weeks but I should be much freer after next week and can pick up on this (fixing the code and actually implement things). wonderful! Thanks. Chris Cheers, Sankarshan On Apr 23, 2014, at 5:58 PM, Chris Barker wrote: On Wed, Mar 19, 2014 at 7:07 PM, Sankarshan Mudkavi wrote: > > I've written a rather rudimentary NEP, (lacking in technical details which > I will hopefully add after some further discussion and receiving > clarification/help on this thread). > > Please let me know how to proceed and what you think should be added to > the current proposal (attached to this mail). > > Here is a rendered version of the same: > > https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst > I've done a bit of copy-editing, and added some more from this discussion. See the pull request on gitHub. There are a fair number of rough edges, but I think we have a consensus among the small group of folks that participated in this discussion anyway, so now "all" we need is someone to actually fix the code. If someone steps up, then we should also go in and add a bunch of unit tests, as discussed in this thread. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 24 13:07:09 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2014 11:07:09 -0600 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <3491667248167119497@unknownmsgid> References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <5629C066-E1DA-4293-94AC-6ACC7F1AC87A@uwaterloo.ca> <3491667248167119497@unknownmsgid> Message-ID: On Thu, Apr 24, 2014 at 10:26 AM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > On Apr 23, 2014, at 8:23 PM, Sankarshan Mudkavi > wrote: > > I've been quite busy for the past few weeks but I should be much freer > after next week and can pick up on this (fixing the code and actually > implement things). > > > wonderful! Thanks. > > Might want to take a look at the datetime proposalfor blaze. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Thu Apr 24 16:22:47 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Thu, 24 Apr 2014 22:22:47 +0200 Subject: [Numpy-discussion] Test error with ATLAS, Windows 64 bit In-Reply-To: <53583027.4000103@googlemail.com> References: <534C4CE4.1030407@googlemail.com> <53583027.4000103@googlemail.com> Message-ID: Hi Julian, to distinguish between mingw32 and mingw-w64 we need something like this: #include if !defined(HAVE_EXPM1) || defined(__MINGW64_VERSION_MAJOR) instead of if !defined(HAVE_EXPM1) || defined(__MINGW32__) the latter is true for both: mingw32 and mingw-w64. I guess the mingw32 implementation of expm1 is again different. stdlib.h has to be included to define __MINGW64_VERSION_MAJOR 2014-04-23 23:27 GMT+02:00 Julian Taylor : > On 23.04.2014 21:25, Matthew Brett wrote: > > Hi, > > > > On Tue, Apr 15, 2014 at 12:34 AM, Julian Taylor > > wrote: > >> On Tue, Apr 15, 2014 at 4:30 AM, Matthew Brett > wrote: > >>> > >>> It looks as though mingw-w64 is at fault, and I was confused (still > >>> am) because of the different behavior with double and a constant: > >>> > >>> #include > >>> #include > >>> > >>> int main() { > >>> double z, i = -0.0; > >>> printf("With double %f=%f, with constant=%f\n", > >>> i, expm1(i), expm1(-0.)); > >>> } > >>> > >>> gives: > >>> > >>> With double -0.000000=0.000000, with constant=-0.000000 > >>> > >>> That was ugly to track down. > >>> > >>> What is the right way to work round this (using the numpy version > >>> instead of the system version I suppose)? > >>> > >> > >> The right way is to file a bug at mingw and get it fixed at the source. > > > > http://sourceforge.net/p/mingw-w64/code/6594/ > > great thanks for reporting it. > > > > >> Additionally as this time npymath seems to be better (thats 3 bugs in > >> npymath vs 1 in mingw on my scoreboard) one could use the mingw > >> preprocessor define instead of HAVE_EXP1M to select this function from > >> npymath. > > > > Sorry to be slow - could you unpack what you mean in this paragraph? > > Is there a way to force use of the numpy version of the function using > > environment variables or similar? > > > > I mean something along the line of attached patch, maybe you can try it. > If it works I can file a numpy PR. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Apr 24 17:56:47 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 24 Apr 2014 14:56:47 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing Message-ID: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cheers, Matthew From jtaylor.debian at googlemail.com Thu Apr 24 18:35:39 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 25 Apr 2014 00:35:39 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: <535991BB.7000102@googlemail.com> On 24.04.2014 23:56, Matthew Brett wrote: > Hi, > > Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > (main author of ATLAS), I've built 64-bit windows numpy and scipy > wheels for testing. > > The build uses Carl's custom mingw-w64 build with static linking. > > There are two harmless test failures on scipy (being discussed on the > list at the moment) - tests otherwise clean. > This is great news, thanks for working on this. Have you already documented the procedure used to create the wheels? I would like to be able to reproduce the builds. Is it possible to add this toolchain and build procedure to the vagrant/fabric based numpy release virtual machine setup? The current version doing linux + win32 builds is available here: https://github.com/juliantaylor/numpy-vendor The windows builds are done in a linux guest using wine. Wine also seems to support win64. The mac build procedure would also needs updating. Cheers, Julian From charlesr.harris at gmail.com Thu Apr 24 19:00:58 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2014 17:00:58 -0600 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: > Hi, > > Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > (main author of ATLAS), I've built 64-bit windows numpy and scipy > wheels for testing. > > The build uses Carl's custom mingw-w64 build with static linking. > > There are two harmless test failures on scipy (being discussed on the > list at the moment) - tests otherwise clean. > > Wheels are here: > > > https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl > > https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl > > You can test with: > > pip install -U pip # to upgrade pip to latest > pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy > > Please do send feedback. > > ATLAS binary here: > > > https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 > > Many thanks for Carl in particular for doing all the hard work, > > Cool. After all these long years... Now all we need is a box running tests for CI. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 24 19:06:18 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Apr 2014 00:06:18 +0100 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 12:00 AM, Charles R Harris wrote: > Cool. After all these long years... Now all we need is a box running tests > for CI. There is http://www.appveyor.com/ though I haven't tried doing anything with it yet... (yes it says ".NET" at the top, but then at the bottom it says that this is a lie and it doesn't care what kind of project you have) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From josef.pktd at gmail.com Thu Apr 24 19:08:29 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 19:08:29 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris wrote: > > Hi Matthew, > > On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: > >> Hi, >> >> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >> (main author of ATLAS), I've built 64-bit windows numpy and scipy >> wheels for testing. >> >> The build uses Carl's custom mingw-w64 build with static linking. >> >> There are two harmless test failures on scipy (being discussed on the >> list at the moment) - tests otherwise clean. >> >> Wheels are here: >> >> >> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >> >> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >> >> You can test with: >> >> pip install -U pip # to upgrade pip to latest >> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy >> >> Please do send feedback. >> >> ATLAS binary here: >> >> >> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >> >> Many thanks for Carl in particular for doing all the hard work, >> >> > Cool. After all these long years... Now all we need is a box running tests > for CI. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I get two test failures with numpy Josef >>> np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 ====================================================================== FAIL: test_iterator.test_iter_broadcasting_errors ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line 657, in test_iter_broadcasting_errors '(2)->(2,newaxis)') % msg) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError: Message "operands could not be broadcast together with remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and requested shape (4,3)" doesn't contain remapped operand shape(2)->(2,newaxis) ====================================================================== FAIL: test_iterator.test_iter_array_cast ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 ---------------------------------------------------------------------- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 24 19:20:30 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2014 17:20:30 -0600 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 5:08 PM, wrote: > > > > On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> Hi Matthew, >> >> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: >> >>> Hi, >>> >>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>> wheels for testing. >>> >>> The build uses Carl's custom mingw-w64 build with static linking. >>> >>> There are two harmless test failures on scipy (being discussed on the >>> list at the moment) - tests otherwise clean. >>> >>> Wheels are here: >>> >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>> >>> You can test with: >>> >>> pip install -U pip # to upgrade pip to latest >>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy >>> scipy >>> >>> Please do send feedback. >>> >>> ATLAS binary here: >>> >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>> >>> Many thanks for Carl in particular for doing all the hard work, >>> >>> >> Cool. After all these long years... Now all we need is a box running >> tests for CI. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > I get two test failures with numpy > > Josef > > >>> np.test() > Running unit tests for numpy > NumPy version 1.8.1 > NumPy is installed in C:\Python27\lib\site-packages\numpy > Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit > (AMD64)] > nose version 1.1.2 > > ====================================================================== > FAIL: test_iterator.test_iter_broadcasting_errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 657, in test_iter_broadcasting_errors > '(2)->(2,newaxis)') % msg) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, in > assert_ > raise AssertionError(msg) > AssertionError: Message "operands could not be broadcast together with > remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and > requested shape (4,3)" doesn't contain remapped operand > shape(2)->(2,newaxis) > > ====================================================================== > FAIL: test_iterator.test_iter_array_cast > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 836, in test_iter_array_cast > assert_equal(i.operands[0].strides, (-96,8,-32)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, > in assert_equal > assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), > verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, > in assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > item=0 > > ACTUAL: 96L > DESIRED: -96 > > ---------------------------------------------------------------------- > Ran 4828 tests in 46.306s > > FAILED (KNOWNFAIL=10, SKIP=8, failures=2) > > > Strange. That second one looks familiar, at least the "-96" part. Wonder why this doesn't show up with the MKL builds. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 24 19:20:37 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 19:20:37 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 7:08 PM, wrote: > > > > On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> Hi Matthew, >> >> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: >> >>> Hi, >>> >>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>> wheels for testing. >>> >>> The build uses Carl's custom mingw-w64 build with static linking. >>> >>> There are two harmless test failures on scipy (being discussed on the >>> list at the moment) - tests otherwise clean. >>> >>> Wheels are here: >>> >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>> >>> You can test with: >>> >>> pip install -U pip # to upgrade pip to latest >>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy >>> scipy >>> >>> Please do send feedback. >>> >>> ATLAS binary here: >>> >>> >>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>> >>> Many thanks for Carl in particular for doing all the hard work, >>> >>> >> Cool. After all these long years... Now all we need is a box running >> tests for CI. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > I get two test failures with numpy > scipy looks good, just two powell trace failures Josef > > Josef > > >>> np.test() > Running unit tests for numpy > NumPy version 1.8.1 > NumPy is installed in C:\Python27\lib\site-packages\numpy > Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit > (AMD64)] > nose version 1.1.2 > > ====================================================================== > FAIL: test_iterator.test_iter_broadcasting_errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 657, in test_iter_broadcasting_errors > '(2)->(2,newaxis)') % msg) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, in > assert_ > raise AssertionError(msg) > AssertionError: Message "operands could not be broadcast together with > remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and > requested shape (4,3)" doesn't contain remapped operand > shape(2)->(2,newaxis) > > ====================================================================== > FAIL: test_iterator.test_iter_array_cast > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 836, in test_iter_array_cast > assert_equal(i.operands[0].strides, (-96,8,-32)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, > in assert_equal > assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), > verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, > in assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > item=0 > > ACTUAL: 96L > DESIRED: -96 > > ---------------------------------------------------------------------- > Ran 4828 tests in 46.306s > > FAILED (KNOWNFAIL=10, SKIP=8, failures=2) > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 24 19:29:38 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 19:29:38 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris wrote: > > > > On Thu, Apr 24, 2014 at 5:08 PM, wrote: > >> >> >> >> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> Hi Matthew, >>> >>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: >>> >>>> Hi, >>>> >>>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>>> wheels for testing. >>>> >>>> The build uses Carl's custom mingw-w64 build with static linking. >>>> >>>> There are two harmless test failures on scipy (being discussed on the >>>> list at the moment) - tests otherwise clean. >>>> >>>> Wheels are here: >>>> >>>> >>>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>>> >>>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>>> >>>> You can test with: >>>> >>>> pip install -U pip # to upgrade pip to latest >>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy >>>> scipy >>>> >>>> Please do send feedback. >>>> >>>> ATLAS binary here: >>>> >>>> >>>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>>> >>>> Many thanks for Carl in particular for doing all the hard work, >>>> >>>> >>> Cool. After all these long years... Now all we need is a box running >>> tests for CI. >>> >>> Chuck >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> I get two test failures with numpy >> >> Josef >> >> >>> np.test() >> Running unit tests for numpy >> NumPy version 1.8.1 >> NumPy is installed in C:\Python27\lib\site-packages\numpy >> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit >> (AMD64)] >> nose version 1.1.2 >> >> ====================================================================== >> FAIL: test_iterator.test_iter_broadcasting_errors >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest >> self.test(*self.arg) >> File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >> line 657, in test_iter_broadcasting_errors >> '(2)->(2,newaxis)') % msg) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, >> in assert_ >> raise AssertionError(msg) >> AssertionError: Message "operands could not be broadcast together with >> remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and >> requested shape (4,3)" doesn't contain remapped operand >> shape(2)->(2,newaxis) >> >> ====================================================================== >> FAIL: test_iterator.test_iter_array_cast >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest >> self.test(*self.arg) >> File "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >> line 836, in test_iter_array_cast >> assert_equal(i.operands[0].strides, (-96,8,-32)) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, >> in assert_equal >> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), >> verbose) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, >> in assert_equal >> raise AssertionError(msg) >> AssertionError: >> Items are not equal: >> item=0 >> >> ACTUAL: 96L >> DESIRED: -96 >> >> ---------------------------------------------------------------------- >> Ran 4828 tests in 46.306s >> >> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) >> >> >> > Strange. That second one looks familiar, at least the "-96" part. Wonder > why this doesn't show up with the MKL builds. > ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) so pip also seems to be reusing leftover files. all clear. Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 24 19:47:20 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 19:47:20 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: > > > OT: Oh, I hate pip and packages that require numpy E:\tmp>C:\Python27\python C:\Python27\Scripts\pip-script.py install -U patsy Downloading/unpacking patsy Running setup.py (path:c:\users\josef\appdata\local\temp\pip_build_josef\patsy\setup.py) egg_info for package patsy no previously-included directories found matching 'doc\_build' Downloading/unpacking numpy from https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678(from patsy) .... Found existing installation: numpy 1.6.2 Uninstalling numpy: Successfully uninstalled numpy Running setup.py install for numpy ... ... C:\Python27\lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'define_macros' warnings.warn(msg) error: Unable to find vcvarsall.bat ---------------------------------------- Rolling back uninstall of numpy Cleaning up... ------------------------- user error ? I have a stale numpy-1.6.2-py2.7.egg-info file in site-packages put that's a nice new feature of pip "Rolling back uninstall of numpy" numpy is still here Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Apr 24 19:58:15 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 24 Apr 2014 16:58:15 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: References: <79374FB2-205D-4B76-ADB2-F9895D3A2DF4@uwaterloo.ca> <5629C066-E1DA-4293-94AC-6ACC7F1AC87A@uwaterloo.ca> <3491667248167119497@unknownmsgid> Message-ID: On Thu, Apr 24, 2014 at 10:07 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > Might want to take a look at the datetime proposalfor blaze. > oh man! not again!. Oh well, that is a decidedly different proposal -- maybe better, I don't know. But it's different enough that I think we should pretty much ignore it for now, and still do a few fixes to make the current datetime64 usable. Maybe as that gets mature, we could adopt it, or something like it, to numpy. Or maybe we'll all be using Blaze then anyway ;-) But thanks for the ping... -CHB > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 24 20:26:56 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 20:26:56 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 7:29 PM, wrote: > > > > On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Thu, Apr 24, 2014 at 5:08 PM, wrote: >> >>> >>> >>> >>> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> >>>> Hi Matthew, >>>> >>>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett >>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >>>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>>>> wheels for testing. >>>>> >>>>> The build uses Carl's custom mingw-w64 build with static linking. >>>>> >>>>> There are two harmless test failures on scipy (being discussed on the >>>>> list at the moment) - tests otherwise clean. >>>>> >>>>> Wheels are here: >>>>> >>>>> >>>>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>>>> >>>>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>>>> >>>>> You can test with: >>>>> >>>>> pip install -U pip # to upgrade pip to latest >>>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy >>>>> scipy >>>>> >>>>> Please do send feedback. >>>>> >>>>> ATLAS binary here: >>>>> >>>>> >>>>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>>>> >>>>> Many thanks for Carl in particular for doing all the hard work, >>>>> >>>>> >>>> Cool. After all these long years... Now all we need is a box running >>>> tests for CI. >>>> >>>> Chuck >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> I get two test failures with numpy >>> >>> Josef >>> >>> >>> np.test() >>> Running unit tests for numpy >>> NumPy version 1.8.1 >>> NumPy is installed in C:\Python27\lib\site-packages\numpy >>> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit >>> (AMD64)] >>> nose version 1.1.2 >>> >>> ====================================================================== >>> FAIL: test_iterator.test_iter_broadcasting_errors >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest >>> self.test(*self.arg) >>> File >>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line >>> 657, in test_iter_broadcasting_errors >>> '(2)->(2,newaxis)') % msg) >>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, >>> in assert_ >>> raise AssertionError(msg) >>> AssertionError: Message "operands could not be broadcast together with >>> remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and >>> requested shape (4,3)" doesn't contain remapped operand >>> shape(2)->(2,newaxis) >>> >>> ====================================================================== >>> FAIL: test_iterator.test_iter_array_cast >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest >>> self.test(*self.arg) >>> File >>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line >>> 836, in test_iter_array_cast >>> assert_equal(i.operands[0].strides, (-96,8,-32)) >>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, >>> in assert_equal >>> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), >>> verbose) >>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, >>> in assert_equal >>> raise AssertionError(msg) >>> AssertionError: >>> Items are not equal: >>> item=0 >>> >>> ACTUAL: 96L >>> DESIRED: -96 >>> >>> ---------------------------------------------------------------------- >>> Ran 4828 tests in 46.306s >>> >>> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) >>> >>> >>> >> Strange. That second one looks familiar, at least the "-96" part. Wonder >> why this doesn't show up with the MKL builds. >> > > ok tried again, this time deleting the old numpy directories before > installing > > Ran 4760 tests in 42.124s > > OK (KNOWNFAIL=10, SKIP=8) > > > > so pip also seems to be reusing leftover files. > > all clear. > Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ Josef > > Josef > > > > >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 24 20:33:46 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 24 Apr 2014 20:33:46 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris wrote: > > Hi Matthew, > > On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett wrote: > >> Hi, >> >> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >> (main author of ATLAS), I've built 64-bit windows numpy and scipy >> wheels for testing. >> >> The build uses Carl's custom mingw-w64 build with static linking. >> >> There are two harmless test failures on scipy (being discussed on the >> list at the moment) - tests otherwise clean. >> >> Wheels are here: >> >> >> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >> >> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >> >> You can test with: >> >> pip install -U pip # to upgrade pip to latest >> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy >> >> Please do send feedback. >> >> ATLAS binary here: >> >> >> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >> >> Many thanks for Carl in particular for doing all the hard work, >> >> > Cool. After all these long years... Now all we need is a box running tests > for CI. > Very good news, after 3 years interruption I might be able to build scipy again, and switch now to a 64bit development version. Thanks for pushing for this, and doing all the hard work. Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Apr 25 01:21:15 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 24 Apr 2014 22:21:15 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, On Thu, Apr 24, 2014 at 5:26 PM, wrote: > > > > On Thu, Apr 24, 2014 at 7:29 PM, wrote: >> >> >> >> >> On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris >> wrote: >>> >>> >>> >>> >>> On Thu, Apr 24, 2014 at 5:08 PM, wrote: >>>> >>>> >>>> >>>> >>>> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris >>>> wrote: >>>>> >>>>> >>>>> Hi Matthew, >>>>> >>>>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >>>>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>>>>> wheels for testing. >>>>>> >>>>>> The build uses Carl's custom mingw-w64 build with static linking. >>>>>> >>>>>> There are two harmless test failures on scipy (being discussed on the >>>>>> list at the moment) - tests otherwise clean. >>>>>> >>>>>> Wheels are here: >>>>>> >>>>>> >>>>>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>>>>> >>>>>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>>>>> >>>>>> You can test with: >>>>>> >>>>>> pip install -U pip # to upgrade pip to latest >>>>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy >>>>>> scipy >>>>>> >>>>>> Please do send feedback. >>>>>> >>>>>> ATLAS binary here: >>>>>> >>>>>> >>>>>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>>>>> >>>>>> Many thanks for Carl in particular for doing all the hard work, >>>>>> >>>>> >>>>> Cool. After all these long years... Now all we need is a box running >>>>> tests for CI. >>>>> >>>>> Chuck >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>> >>>> I get two test failures with numpy >>>> >>>> Josef >>>> >>>> >>> np.test() >>>> Running unit tests for numpy >>>> NumPy version 1.8.1 >>>> NumPy is installed in C:\Python27\lib\site-packages\numpy >>>> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit >>>> (AMD64)] >>>> nose version 1.1.2 >>>> >>>> ====================================================================== >>>> FAIL: test_iterator.test_iter_broadcasting_errors >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >>>> runTest >>>> self.test(*self.arg) >>>> File >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line 657, >>>> in test_iter_broadcasting_errors >>>> '(2)->(2,newaxis)') % msg) >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 44, >>>> in assert_ >>>> raise AssertionError(msg) >>>> AssertionError: Message "operands could not be broadcast together with >>>> remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) and >>>> requested shape (4,3)" doesn't contain remapped operand >>>> shape(2)->(2,newaxis) >>>> >>>> ====================================================================== >>>> FAIL: test_iterator.test_iter_array_cast >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >>>> runTest >>>> self.test(*self.arg) >>>> File >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", line 836, >>>> in test_iter_array_cast >>>> assert_equal(i.operands[0].strides, (-96,8,-32)) >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 255, >>>> in assert_equal >>>> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), >>>> verbose) >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 317, >>>> in assert_equal >>>> raise AssertionError(msg) >>>> AssertionError: >>>> Items are not equal: >>>> item=0 >>>> >>>> ACTUAL: 96L >>>> DESIRED: -96 >>>> >>>> ---------------------------------------------------------------------- >>>> Ran 4828 tests in 46.306s >>>> >>>> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) >>>> >>>> >>> >>> Strange. That second one looks familiar, at least the "-96" part. Wonder >>> why this doesn't show up with the MKL builds. >> >> >> ok tried again, this time deleting the old numpy directories before >> installing >> >> Ran 4760 tests in 42.124s >> >> OK (KNOWNFAIL=10, SKIP=8) >> >> >> >> so pip also seems to be reusing leftover files. >> >> all clear. > > > Running the statsmodels test suite, I get a failure in > test_discrete.TestProbitCG where fmin_cg converges to something that differs > in the 3rd decimal. > > I usually only test the 32-bit version, so I don't know if this is specific > to this scipy version, but we haven't seen this in a long time. > I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ That's interesting, you saw also we're getting failures on the tests for powell optimization because of small unit-at-last-place differences in the exp function in mingw-w64. Is there any chance you can track down where the optimization path is diverging and why? It's just that - if this is also the exp function maybe we can see if the error is exceeding reasonable bounds and then feed back to mingw-w64 and fall back to the numpy default implementation in the meantime. Cheers, Matthew From josef.pktd at gmail.com Sat Apr 26 10:10:47 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 10:10:47 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett wrote: > Hi, > > On Thu, Apr 24, 2014 at 5:26 PM, wrote: > > > > > > > > On Thu, Apr 24, 2014 at 7:29 PM, wrote: > >> > >> > >> > >> > >> On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris > >> wrote: > >>> > >>> > >>> > >>> > >>> On Thu, Apr 24, 2014 at 5:08 PM, wrote: > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris > >>>> wrote: > >>>>> > >>>>> > >>>>> Hi Matthew, > >>>>> > >>>>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett > >>>>> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > >>>>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy > >>>>>> wheels for testing. > >>>>>> > >>>>>> The build uses Carl's custom mingw-w64 build with static linking. > >>>>>> > >>>>>> There are two harmless test failures on scipy (being discussed on > the > >>>>>> list at the moment) - tests otherwise clean. > >>>>>> > >>>>>> Wheels are here: > >>>>>> > >>>>>> > >>>>>> > https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl > >>>>>> > >>>>>> > https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl > >>>>>> > >>>>>> You can test with: > >>>>>> > >>>>>> pip install -U pip # to upgrade pip to latest > >>>>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy > >>>>>> scipy > >>>>>> > >>>>>> Please do send feedback. > >>>>>> > >>>>>> ATLAS binary here: > >>>>>> > >>>>>> > >>>>>> > https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 > >>>>>> > >>>>>> Many thanks for Carl in particular for doing all the hard work, > >>>>>> > >>>>> > >>>>> Cool. After all these long years... Now all we need is a box running > >>>>> tests for CI. > >>>>> > >>>>> Chuck > >>>>> > >>>>> _______________________________________________ > >>>>> NumPy-Discussion mailing list > >>>>> NumPy-Discussion at scipy.org > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>>> > >>>> > >>>> I get two test failures with numpy > >>>> > >>>> Josef > >>>> > >>>> >>> np.test() > >>>> Running unit tests for numpy > >>>> NumPy version 1.8.1 > >>>> NumPy is installed in C:\Python27\lib\site-packages\numpy > >>>> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 > bit > >>>> (AMD64)] > >>>> nose version 1.1.2 > >>>> > >>>> ====================================================================== > >>>> FAIL: test_iterator.test_iter_broadcasting_errors > >>>> ---------------------------------------------------------------------- > >>>> Traceback (most recent call last): > >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in > >>>> runTest > >>>> self.test(*self.arg) > >>>> File > >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 657, > >>>> in test_iter_broadcasting_errors > >>>> '(2)->(2,newaxis)') % msg) > >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 44, > >>>> in assert_ > >>>> raise AssertionError(msg) > >>>> AssertionError: Message "operands could not be broadcast together with > >>>> remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) > and > >>>> requested shape (4,3)" doesn't contain remapped operand > >>>> shape(2)->(2,newaxis) > >>>> > >>>> ====================================================================== > >>>> FAIL: test_iterator.test_iter_array_cast > >>>> ---------------------------------------------------------------------- > >>>> Traceback (most recent call last): > >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in > >>>> runTest > >>>> self.test(*self.arg) > >>>> File > >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", > line 836, > >>>> in test_iter_array_cast > >>>> assert_equal(i.operands[0].strides, (-96,8,-32)) > >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 255, > >>>> in assert_equal > >>>> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), > >>>> verbose) > >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 317, > >>>> in assert_equal > >>>> raise AssertionError(msg) > >>>> AssertionError: > >>>> Items are not equal: > >>>> item=0 > >>>> > >>>> ACTUAL: 96L > >>>> DESIRED: -96 > >>>> > >>>> ---------------------------------------------------------------------- > >>>> Ran 4828 tests in 46.306s > >>>> > >>>> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) > >>>> > >>>> > >>> > >>> Strange. That second one looks familiar, at least the "-96" part. > Wonder > >>> why this doesn't show up with the MKL builds. > >> > >> > >> ok tried again, this time deleting the old numpy directories before > >> installing > >> > >> Ran 4760 tests in 42.124s > >> > >> OK (KNOWNFAIL=10, SKIP=8) > >> > >> > >> > >> so pip also seems to be reusing leftover files. > >> > >> all clear. > > > > > > Running the statsmodels test suite, I get a failure in > > test_discrete.TestProbitCG where fmin_cg converges to something that > differs > > in the 3rd decimal. > > > > I usually only test the 32-bit version, so I don't know if this is > specific > > to this scipy version, but we haven't seen this in a long time. > > I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ > > That's interesting, you saw also we're getting failures on the tests > for powell optimization because of small unit-at-last-place > differences in the exp function in mingw-w64. Is there any chance you > can track down where the optimization path is diverging and why? > It's just that - if this is also the exp function maybe we can see if > the error is exceeding reasonable bounds and then feed back to > mingw-w64 and fall back to the numpy default implementation in the > meantime. > I'm a bit doubtful it's exp, the probit model is based on the normal distribution and has an exp only in the gradient via norm._pdf, the objective function uses norm._cdf. I can look into it. However: We don't use fmin_cg for anything by default, it's part of "testing all supported scipy optimizers" and we had problems with it before on various machines https://github.com/statsmodels/statsmodels/issues/109 The test was completely disabled on Windows for a while, and I might have to turn some screws again. I'm fighting with more serious problems with fmin_slsqp and fmin_bfgs, which we really need to use. If minor precision issues matter, then the code is not "robust" and should be fixed. compared to precision issues. I'm fighting more with the large scale properties of exp. https://github.com/scipy/scipy/issues/3581 Neverthless, I would really like to know why I'm running into many platform differences and problems with scipy.optimize. Cheers, Josef > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 26 10:20:28 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 10:20:28 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Sat, Apr 26, 2014 at 10:10 AM, wrote: > > > > On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett wrote: > >> Hi, >> >> On Thu, Apr 24, 2014 at 5:26 PM, wrote: >> > >> > >> > >> > On Thu, Apr 24, 2014 at 7:29 PM, wrote: >> >> >> >> >> >> >> >> >> >> On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris >> >> wrote: >> >>> >> >>> >> >>> >> >>> >> >>> On Thu, Apr 24, 2014 at 5:08 PM, wrote: >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris >> >>>> wrote: >> >>>>> >> >>>>> >> >>>>> Hi Matthew, >> >>>>> >> >>>>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett >> >>>>> wrote: >> >>>>>> >> >>>>>> Hi, >> >>>>>> >> >>>>>> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley >> >>>>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >> >>>>>> wheels for testing. >> >>>>>> >> >>>>>> The build uses Carl's custom mingw-w64 build with static linking. >> >>>>>> >> >>>>>> There are two harmless test failures on scipy (being discussed on >> the >> >>>>>> list at the moment) - tests otherwise clean. >> >>>>>> >> >>>>>> Wheels are here: >> >>>>>> >> >>>>>> >> >>>>>> >> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >> >>>>>> >> >>>>>> >> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >> >>>>>> >> >>>>>> You can test with: >> >>>>>> >> >>>>>> pip install -U pip # to upgrade pip to latest >> >>>>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installersnumpy >> >>>>>> scipy >> >>>>>> >> >>>>>> Please do send feedback. >> >>>>>> >> >>>>>> ATLAS binary here: >> >>>>>> >> >>>>>> >> >>>>>> >> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >> >>>>>> >> >>>>>> Many thanks for Carl in particular for doing all the hard work, >> >>>>>> >> >>>>> >> >>>>> Cool. After all these long years... Now all we need is a box running >> >>>>> tests for CI. >> >>>>> >> >>>>> Chuck >> >>>>> >> >>>>> _______________________________________________ >> >>>>> NumPy-Discussion mailing list >> >>>>> NumPy-Discussion at scipy.org >> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >>>>> >> >>>> >> >>>> I get two test failures with numpy >> >>>> >> >>>> Josef >> >>>> >> >>>> >>> np.test() >> >>>> Running unit tests for numpy >> >>>> NumPy version 1.8.1 >> >>>> NumPy is installed in C:\Python27\lib\site-packages\numpy >> >>>> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 >> bit >> >>>> (AMD64)] >> >>>> nose version 1.1.2 >> >>>> >> >>>> >> ====================================================================== >> >>>> FAIL: test_iterator.test_iter_broadcasting_errors >> >>>> >> ---------------------------------------------------------------------- >> >>>> Traceback (most recent call last): >> >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >> >>>> runTest >> >>>> self.test(*self.arg) >> >>>> File >> >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >> line 657, >> >>>> in test_iter_broadcasting_errors >> >>>> '(2)->(2,newaxis)') % msg) >> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 44, >> >>>> in assert_ >> >>>> raise AssertionError(msg) >> >>>> AssertionError: Message "operands could not be broadcast together >> with >> >>>> remapped shapes [original->remapped]: (2,3)->(2,3) (2,)->(2,newaxis) >> and >> >>>> requested shape (4,3)" doesn't contain remapped operand >> >>>> shape(2)->(2,newaxis) >> >>>> >> >>>> >> ====================================================================== >> >>>> FAIL: test_iterator.test_iter_array_cast >> >>>> >> ---------------------------------------------------------------------- >> >>>> Traceback (most recent call last): >> >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >> >>>> runTest >> >>>> self.test(*self.arg) >> >>>> File >> >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >> line 836, >> >>>> in test_iter_array_cast >> >>>> assert_equal(i.operands[0].strides, (-96,8,-32)) >> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 255, >> >>>> in assert_equal >> >>>> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), >> >>>> verbose) >> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 317, >> >>>> in assert_equal >> >>>> raise AssertionError(msg) >> >>>> AssertionError: >> >>>> Items are not equal: >> >>>> item=0 >> >>>> >> >>>> ACTUAL: 96L >> >>>> DESIRED: -96 >> >>>> >> >>>> >> ---------------------------------------------------------------------- >> >>>> Ran 4828 tests in 46.306s >> >>>> >> >>>> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) >> >>>> >> >>>> >> >>> >> >>> Strange. That second one looks familiar, at least the "-96" part. >> Wonder >> >>> why this doesn't show up with the MKL builds. >> >> >> >> >> >> ok tried again, this time deleting the old numpy directories before >> >> installing >> >> >> >> Ran 4760 tests in 42.124s >> >> >> >> OK (KNOWNFAIL=10, SKIP=8) >> >> >> >> >> >> >> >> so pip also seems to be reusing leftover files. >> >> >> >> all clear. >> > >> > >> > Running the statsmodels test suite, I get a failure in >> > test_discrete.TestProbitCG where fmin_cg converges to something that >> differs >> > in the 3rd decimal. >> > >> > I usually only test the 32-bit version, so I don't know if this is >> specific >> > to this scipy version, but we haven't seen this in a long time. >> > I used our nightly binaries >> http://statsmodels.sourceforge.net/binaries/ >> >> That's interesting, you saw also we're getting failures on the tests >> for powell optimization because of small unit-at-last-place >> differences in the exp function in mingw-w64. Is there any chance you >> can track down where the optimization path is diverging and why? >> It's just that - if this is also the exp function maybe we can see if >> the error is exceeding reasonable bounds and then feed back to >> mingw-w64 and fall back to the numpy default implementation in the >> meantime. >> > > I'm a bit doubtful it's exp, the probit model is based on the normal > distribution and has an exp only in the gradient via norm._pdf, the > objective function uses norm._cdf. > > I can look into it. > > However: > We don't use fmin_cg for anything by default, it's part of "testing all > supported scipy optimizers" and we had problems with it before on various > machines https://github.com/statsmodels/statsmodels/issues/109 > The test was completely disabled on Windows for a while, and I might have > to turn some screws again. > > I'm fighting with more serious problems with fmin_slsqp and fmin_bfgs, > which we really need to use. > > If minor precision issues matter, then the code is not "robust" and should > be fixed. > > compared to precision issues. I'm fighting more with the large scale > properties of exp. > https://github.com/scipy/scipy/issues/3581 > > > Neverthless, > I would really like to know why I'm running into many platform differences > and problems with scipy.optimize. > To avoid giving a wrong impression: Scipy.optimize works in general very well for statsmodels, we use it heavily and we have a large set of test cases for it. It's just the last 5% or so of cases where I spend a considerable amount of time figuring out how to get around convergence problems, which are sometimes platform specific and sometimes not. Josef > > Cheers, > > Josef > > > >> >> Cheers, >> >> Matthew >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 26 12:01:45 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 12:01:45 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Sat, Apr 26, 2014 at 10:20 AM, wrote: > > > > On Sat, Apr 26, 2014 at 10:10 AM, wrote: > >> >> >> >> On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett wrote: >> >>> Hi, >>> >>> On Thu, Apr 24, 2014 at 5:26 PM, wrote: >>> > >>> > >>> > >>> > On Thu, Apr 24, 2014 at 7:29 PM, wrote: >>> >> >>> >> >>> >> >>> >> >>> >> On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris >>> >> wrote: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Apr 24, 2014 at 5:08 PM, wrote: >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris >>> >>>> wrote: >>> >>>>> >>> >>>>> >>> >>>>> Hi Matthew, >>> >>>>> >>> >>>>> On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett >>> >>>>> wrote: >>> >>>>>> >>> >>>>>> Hi, >>> >>>>>> >>> >>>>>> Thanks to Cark Kleffner's toolchain and some help from Clint >>> Whaley >>> >>>>>> (main author of ATLAS), I've built 64-bit windows numpy and scipy >>> >>>>>> wheels for testing. >>> >>>>>> >>> >>>>>> The build uses Carl's custom mingw-w64 build with static linking. >>> >>>>>> >>> >>>>>> There are two harmless test failures on scipy (being discussed on >>> the >>> >>>>>> list at the moment) - tests otherwise clean. >>> >>>>>> >>> >>>>>> Wheels are here: >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl >>> >>>>>> >>> >>>>>> >>> https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl >>> >>>>>> >>> >>>>>> You can test with: >>> >>>>>> >>> >>>>>> pip install -U pip # to upgrade pip to latest >>> >>>>>> pip install -f https://nipy.bic.berkeley.edu/scipy_installersnumpy >>> >>>>>> scipy >>> >>>>>> >>> >>>>>> Please do send feedback. >>> >>>>>> >>> >>>>>> ATLAS binary here: >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 >>> >>>>>> >>> >>>>>> Many thanks for Carl in particular for doing all the hard work, >>> >>>>>> >>> >>>>> >>> >>>>> Cool. After all these long years... Now all we need is a box >>> running >>> >>>>> tests for CI. >>> >>>>> >>> >>>>> Chuck >>> >>>>> >>> >>>>> _______________________________________________ >>> >>>>> NumPy-Discussion mailing list >>> >>>>> NumPy-Discussion at scipy.org >>> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>>>> >>> >>>> >>> >>>> I get two test failures with numpy >>> >>>> >>> >>>> Josef >>> >>>> >>> >>>> >>> np.test() >>> >>>> Running unit tests for numpy >>> >>>> NumPy version 1.8.1 >>> >>>> NumPy is installed in C:\Python27\lib\site-packages\numpy >>> >>>> Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 >>> 64 bit >>> >>>> (AMD64)] >>> >>>> nose version 1.1.2 >>> >>>> >>> >>>> >>> ====================================================================== >>> >>>> FAIL: test_iterator.test_iter_broadcasting_errors >>> >>>> >>> ---------------------------------------------------------------------- >>> >>>> Traceback (most recent call last): >>> >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >>> >>>> runTest >>> >>>> self.test(*self.arg) >>> >>>> File >>> >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >>> line 657, >>> >>>> in test_iter_broadcasting_errors >>> >>>> '(2)->(2,newaxis)') % msg) >>> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >>> 44, >>> >>>> in assert_ >>> >>>> raise AssertionError(msg) >>> >>>> AssertionError: Message "operands could not be broadcast together >>> with >>> >>>> remapped shapes [original->remapped]: (2,3)->(2,3) >>> (2,)->(2,newaxis) and >>> >>>> requested shape (4,3)" doesn't contain remapped operand >>> >>>> shape(2)->(2,newaxis) >>> >>>> >>> >>>> >>> ====================================================================== >>> >>>> FAIL: test_iterator.test_iter_array_cast >>> >>>> >>> ---------------------------------------------------------------------- >>> >>>> Traceback (most recent call last): >>> >>>> File "C:\Python27\lib\site-packages\nose\case.py", line 197, in >>> >>>> runTest >>> >>>> self.test(*self.arg) >>> >>>> File >>> >>>> "C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py", >>> line 836, >>> >>>> in test_iter_array_cast >>> >>>> assert_equal(i.operands[0].strides, (-96,8,-32)) >>> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >>> 255, >>> >>>> in assert_equal >>> >>>> assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, >>> err_msg), >>> >>>> verbose) >>> >>>> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >>> 317, >>> >>>> in assert_equal >>> >>>> raise AssertionError(msg) >>> >>>> AssertionError: >>> >>>> Items are not equal: >>> >>>> item=0 >>> >>>> >>> >>>> ACTUAL: 96L >>> >>>> DESIRED: -96 >>> >>>> >>> >>>> >>> ---------------------------------------------------------------------- >>> >>>> Ran 4828 tests in 46.306s >>> >>>> >>> >>>> FAILED (KNOWNFAIL=10, SKIP=8, failures=2) >>> >>>> >>> >>>> >>> >>> >>> >>> Strange. That second one looks familiar, at least the "-96" part. >>> Wonder >>> >>> why this doesn't show up with the MKL builds. >>> >> >>> >> >>> >> ok tried again, this time deleting the old numpy directories before >>> >> installing >>> >> >>> >> Ran 4760 tests in 42.124s >>> >> >>> >> OK (KNOWNFAIL=10, SKIP=8) >>> >> >>> >> >>> >> >>> >> so pip also seems to be reusing leftover files. >>> >> >>> >> all clear. >>> > >>> > >>> > Running the statsmodels test suite, I get a failure in >>> > test_discrete.TestProbitCG where fmin_cg converges to something that >>> differs >>> > in the 3rd decimal. >>> > >>> > I usually only test the 32-bit version, so I don't know if this is >>> specific >>> > to this scipy version, but we haven't seen this in a long time. >>> > I used our nightly binaries >>> http://statsmodels.sourceforge.net/binaries/ >>> >>> That's interesting, you saw also we're getting failures on the tests >>> for powell optimization because of small unit-at-last-place >>> differences in the exp function in mingw-w64. Is there any chance you >>> can track down where the optimization path is diverging and why? >>> It's just that - if this is also the exp function maybe we can see if >>> the error is exceeding reasonable bounds and then feed back to >>> mingw-w64 and fall back to the numpy default implementation in the >>> meantime. >>> >> >> I'm a bit doubtful it's exp, the probit model is based on the normal >> distribution and has an exp only in the gradient via norm._pdf, the >> objective function uses norm._cdf. >> >> I can look into it. >> > with 32 bit official binaries MingW 32 Warning: Desired error not necessarily achieved due to precision loss. Current function value: 0.400588 Iterations: 75 Function evaluations: 213 Gradient evaluations: 201 relative and absolute deviation from "desired" [ -1.26257296e-05 -4.77535711e-05 -9.93794940e-06 -1.78815725e-05] [ -2.05270407e-05 -2.47024202e-06 -1.41748189e-05 1.33259208e-04] with your wheels, after increasing maxiter in the test case Optimization terminated successfully. Current function value: 0.400588 Iterations: 766 Function evaluations: 1591 Gradient evaluations: 1591 relative and absolute deviation from "desired" [ -1.57311713e-07 -4.25324806e-08 -3.01557919e-08 -1.19794357e-07] [ -2.55758996e-07 -2.20016050e-09 -4.30121820e-08 8.92745931e-07] So actually the 64 bit wheel has the better final result, and just needs more iterations to get close enough to what we had required in the unit tests. The trace of the 64bit version seems to slow down in the movement, but then doesn't run into the "precision loss" >From visual comparison, after the 20 iteration the parameters start to slowly diverge in the 5th decimal. attached is a script that replicates the testcase Thanks Josef > >> However: >> We don't use fmin_cg for anything by default, it's part of "testing all >> supported scipy optimizers" and we had problems with it before on various >> machines https://github.com/statsmodels/statsmodels/issues/109 >> The test was completely disabled on Windows for a while, and I might have >> to turn some screws again. >> >> I'm fighting with more serious problems with fmin_slsqp and fmin_bfgs, >> which we really need to use. >> >> If minor precision issues matter, then the code is not "robust" and >> should be fixed. >> >> compared to precision issues. I'm fighting more with the large scale >> properties of exp. >> https://github.com/scipy/scipy/issues/3581 >> >> >> Neverthless, >> I would really like to know why I'm running into many platform >> differences and problems with scipy.optimize. >> > > To avoid giving a wrong impression: > > Scipy.optimize works in general very well for statsmodels, we use it > heavily and we have a large set of test cases for it. > It's just the last 5% or so of cases where I spend a considerable amount > of time figuring out how to get around convergence problems, which are > sometimes platform specific and sometimes not. > > Josef > > > >> >> Cheers, >> >> Josef >> >> >> >>> >>> Cheers, >>> >>> Matthew >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- # -*- coding: utf-8 -*- """ Created on Sat Apr 26 11:21:07 2014 Author: josef """ import numpy as np from numpy.testing import assert_almost_equal import statsmodels.api as sm from statsmodels.discrete.tests.results.results_discrete import Spector hist = [] def callb(*args): hist.append(args[0]) data = sm.datasets.spector.load() data.exog = sm.add_constant(data.exog, prepend=False) res2 = Spector() res2.probit() cls_res2 = res2 cls_res1 = sm.Probit(data.endog, data.exog).fit(method="cg", disp=1, maxiter=1000, gtol=1e-08, callback=callb) print cls_res1.params print cls_res2.params print cls_res1.params / cls_res2.params - 1 print cls_res1.params - cls_res2.params print cls_res1.mle_retvals import pandas res_ = pandas.DataFrame(np.asarray(hist)) print res_.to_string() #res_.to_csv('trace_probitcg_32.csv') #res_.to_csv('trace_probitcg_64wheel.csv') DECIMAL_4 = 4 assert_almost_equal(cls_res1.params, cls_res2.params, DECIMAL_4) From sturla.molden at gmail.com Fri Apr 25 01:57:39 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 25 Apr 2014 05:57:39 +0000 (UTC) Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing References: Message-ID: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > (main author of ATLAS), I've built 64-bit windows numpy and scipy > wheels for testing. Thanks for your great effort to solve this mess. By Murphy's law, I do not have access to a Windows computer on which to test now. :-( This approach worries me a bit though: Will we have to maintain a fork of MinGW-w64 for building NumPy and SciPy? Should this toolset be distributed along with NumPy and SciPy on Windows? I presume it is needed to build C and Cython extensions? On the positive side: Does this mean we finally can use gfortran on Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy now? Or is Mac OS X a blocker? Sturla From pav at iki.fi Sat Apr 26 13:01:29 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Apr 2014 20:01:29 +0300 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> Message-ID: 25.04.2014 08:57, Sturla Molden kirjoitti: [clip] > On the positive side: Does this mean we finally can use gfortran on > Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy > now? Or is Mac OS X a blocker? Yes, Windows is the only platform on which Fortran was problematic. OSX is somewhat saner in this respect. -- Pauli Virtanen From sebastian at sipsolutions.net Sat Apr 26 13:12:10 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 26 Apr 2014 19:12:10 +0200 Subject: [Numpy-discussion] 1.9.x branch In-Reply-To: References: <1398245523.4282.18.camel@sebastian-t440> Message-ID: <1398532330.7311.42.camel@sebastian-t440> On Mi, 2014-04-23 at 10:11 -0400, josef.pktd at gmail.com wrote: > On Wed, Apr 23, 2014 at 5:32 AM, Sebastian Berg > wrote: > > On Di, 2014-04-22 at 15:35 -0600, Charles R Harris wrote: > >> Hi All, > >> > >> > >> I'd like to branch 1.9.x at the end of the month. There are a couple > >> of reasons for the timing. First, we have a lot of new stuff in the > >> development branch. Second, there is work ongoing in masked arrays > >> that I'd like to keep out of the release so that it has more time to > >> settle. Third, it's past time ;) > > > > Sounds good. > > > >> There are currently a number of 1.9.0 blockers, which can be seen > >> here. > >> > >> Datetime timezone handling broken in 1.7.x > >> > >> I don't think there is time to get this done for 1.9.0 and it needs to > >> be pushed off to 1.10.0. > >> > >> Return multiple field selection as ro view > >> > >> I have a branch for this, but because the returned type differs from a > >> copy by alignment spacing there was a test failure. Merging that > >> branch might cause some incompatibilities. > >> > > > > I am a bit worried here that comparisons might make trouble. > > > >> Object array creation new conversion to int > >> > >> > >> This one needs a decision. Julian, Sebastian, thoughts? > >> > > > > Maybe for all to consider this is about what happens for object arrays > > if you do things like: > > > > # Array cast to object array (np.array(arr) would be identical): > > a = np.arange(10).astype(object) > > # Array passed into new array creation (not just *one* array): > > b = np.array([np.arange(10)], dtype=object) > > # Numerical array is assigned to object array: > > c = np.empty(10, dtype=object) > > c[...] = np.arange(10) > > > > Before this change, the result was: > > type(a[0]) is int > > type(b[0,0]) is np.int_ # Note the numpy type > > type(c[0]) is int > > > > After this change, they are all `int`. Though note that the numpy type > > is preserved for example for long double. On the one hand preserving the > > numpy type might be nice, but on the other hand we don't care much about > > the dtypes of scalars and in practice the python types are probably more > > often wanted. > > what if I don't like python? > Fair point :). I also think it is more consistent if we use the numpy types (and the user has to cast to the python type explicitly). Could argue that if someone casts to object, they might like python objects, but if you don't want them that is tricky, too. There is the option that the numerical array -> object array cast always returns an array of numpy scalars. Making it consistent (opposite to what is currently in numpy master). This would mean that `numerical_arr.astype(object)` would give an array of numpy scalars always. Getting python scalars would only be possible using `arr.item()` (or `tolist`). I guess that change is less likely to cause problems, because the numpy types can do more things normally though they are slower. So a (still a bit unsure) +1 from me for making numeric -> object casts return arrays of numpy scalars unless we have reason to expect that to cause problems. Not sure how easy that would be to change, it wasn't a one line change when I tried, so there is something slightly tricky to clear out, but probably not too hard either. - Sebastian > >>> np.int_(0)**(-1) > inf > >>> 0**-1 > Traceback (most recent call last): > File "", line 1, in > 0**-1 > ZeroDivisionError: 0.0 cannot be raised to a negative power > > > >>> type(np.arange(5)[0]) > > >>> np.arange(5)[0]**-1 > inf > > >>> type(np.arange(5)[0].item()) > > >>> np.arange(5)[0].item()**-1 > Traceback (most recent call last): > File "", line 1, in > np.arange(5)[0].item()**-1 > ZeroDivisionError: 0.0 cannot be raised to a negative power > > >>> np.__version__ > '1.6.1' > > > I remember struggling through this (avoiding python operations) quite > a bit in my early bugfixes to scipy.stats.distributions. > > (IIRC I ended up avoiding most ints.) > > Josef > > > > > Since I just realized that things are safe (float128 does not cast to > > float after all), I changed my mind and am tempted to keep the new > > behaviour. That is, if it does not create any problems (there was some > > issue in scipy, not sure how bad). > > > > - Sebastian > > > >> Median of np.matrix is broken( > >> > >> > >> Not sure what the status of this one is. > >> > >> 1.8 deprecations: Follow-up ticket > >> > >> > >> Things that might should be removed. > >> > >> ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 > >> > >> > >> I believe this one was fixed. For general problems reading/writing big > >> files on OS X, I believe they were fixed in Maverick and I'm inclined > >> to recommend an OS upgrade rather than work to chunk all the io. > >> > >> Deprecate NPY_CHAR > >> This one is waiting on a fix from Pearu to make f2py use numpy > >> strings. I think we will need to do this ourselves if we want to carry > >> through the deprecation. In any case it probably needs to be pushed > >> off to 1.10. > >> > >> 1.7 deprecations: Follow-up ticket > >> Some of these changes have been made, ro diagonal view for instance, > >> some still remain. > >> > >> > >> > >> Additions, updates, and thoughts welcome. > >> > >> > >> Chuck > >> > >> > >> > >> > >> > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Sat Apr 26 13:36:26 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 13:36:26 -0400 Subject: [Numpy-discussion] 1.9.x branch In-Reply-To: <1398532330.7311.42.camel@sebastian-t440> References: <1398245523.4282.18.camel@sebastian-t440> <1398532330.7311.42.camel@sebastian-t440> Message-ID: On Sat, Apr 26, 2014 at 1:12 PM, Sebastian Berg wrote: > On Mi, 2014-04-23 at 10:11 -0400, josef.pktd at gmail.com wrote: > > On Wed, Apr 23, 2014 at 5:32 AM, Sebastian Berg > > wrote: > > > On Di, 2014-04-22 at 15:35 -0600, Charles R Harris wrote: > > >> Hi All, > > >> > > >> > > >> I'd like to branch 1.9.x at the end of the month. There are a couple > > >> of reasons for the timing. First, we have a lot of new stuff in the > > >> development branch. Second, there is work ongoing in masked arrays > > >> that I'd like to keep out of the release so that it has more time to > > >> settle. Third, it's past time ;) > > > > > > Sounds good. > > > > > >> There are currently a number of 1.9.0 blockers, which can be seen > > >> here. > > >> > > >> Datetime timezone handling broken in 1.7.x > > >> > > >> I don't think there is time to get this done for 1.9.0 and it needs to > > >> be pushed off to 1.10.0. > > >> > > >> Return multiple field selection as ro view > > >> > > >> I have a branch for this, but because the returned type differs from a > > >> copy by alignment spacing there was a test failure. Merging that > > >> branch might cause some incompatibilities. > > >> > > > > > > I am a bit worried here that comparisons might make trouble. > > > > > >> Object array creation new conversion to int > > >> > > >> > > >> This one needs a decision. Julian, Sebastian, thoughts? > > >> > > > > > > Maybe for all to consider this is about what happens for object arrays > > > if you do things like: > > > > > > # Array cast to object array (np.array(arr) would be identical): > > > a = np.arange(10).astype(object) > > > # Array passed into new array creation (not just *one* array): > > > b = np.array([np.arange(10)], dtype=object) > > > # Numerical array is assigned to object array: > > > c = np.empty(10, dtype=object) > > > c[...] = np.arange(10) > > > > > > Before this change, the result was: > > > type(a[0]) is int > > > type(b[0,0]) is np.int_ # Note the numpy type > > > type(c[0]) is int > > > > > > After this change, they are all `int`. Though note that the numpy type > > > is preserved for example for long double. On the one hand preserving > the > > > numpy type might be nice, but on the other hand we don't care much > about > > > the dtypes of scalars and in practice the python types are probably > more > > > often wanted. > > > > what if I don't like python? > > > > Fair point :). I also think it is more consistent if we use the numpy > types (and the user has to cast to the python type explicitly). Could > argue that if someone casts to object, they might like python objects, > but if you don't want them that is tricky, too. > > There is the option that the numerical array -> object array cast always > returns an array of numpy scalars. Making it consistent (opposite to > what is currently in numpy master). > > This would mean that `numerical_arr.astype(object)` would give an array > of numpy scalars always. Getting python scalars would only be possible > using `arr.item()` (or `tolist`). I guess that change is less likely to > cause problems, because the numpy types can do more things normally > though they are slower. > > So a (still a bit unsure) +1 from me for making numeric -> object casts > return arrays of numpy scalars unless we have reason to expect that to > cause problems. Not sure how easy that would be to change, it wasn't a > one line change when I tried, so there is something slightly tricky to > clear out, but probably not too hard either. > More general background question. Why is there casting involved in object arrays? I thought object arrays will just take and return whatever you put in. If I use python ints, I might want python ints. If I use numpy int_s, I might want numpy ints. b1 = np.array([np.arange(10)], dtype=object) versus b2 = np.array([list(range(10))], dtype=object) >>> b1 = np.array([np.arange(10)], dtype=object) >>> type(b1[0,2]) >>> type(b1[0][2]) >>> >>> b2 = np.array([list(range(10))], dtype=object) >>> type(b2[0,2]) >>> type(b2[0][2]) another version >>> type(np.array(np.arange(10).tolist(), dtype=object)[2]) >>> type(np.array(list(np.arange(10)), dtype=object)[2]) Josef > > - Sebastian > > > >>> np.int_(0)**(-1) > > inf > > >>> 0**-1 > > Traceback (most recent call last): > > File "", line 1, in > > 0**-1 > > ZeroDivisionError: 0.0 cannot be raised to a negative power > > > > > > >>> type(np.arange(5)[0]) > > > > >>> np.arange(5)[0]**-1 > > inf > > > > >>> type(np.arange(5)[0].item()) > > > > >>> np.arange(5)[0].item()**-1 > > Traceback (most recent call last): > > File "", line 1, in > > np.arange(5)[0].item()**-1 > > ZeroDivisionError: 0.0 cannot be raised to a negative power > > > > >>> np.__version__ > > '1.6.1' > > > > > > I remember struggling through this (avoiding python operations) quite > > a bit in my early bugfixes to scipy.stats.distributions. > > > > (IIRC I ended up avoiding most ints.) > > > > Josef > > > > > > > > Since I just realized that things are safe (float128 does not cast to > > > float after all), I changed my mind and am tempted to keep the new > > > behaviour. That is, if it does not create any problems (there was some > > > issue in scipy, not sure how bad). > > > > > > - Sebastian > > > > > >> Median of np.matrix is broken( > > >> > > >> > > >> Not sure what the status of this one is. > > >> > > >> 1.8 deprecations: Follow-up ticket > > >> > > >> > > >> Things that might should be removed. > > >> > > >> ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 > > >> > > >> > > >> I believe this one was fixed. For general problems reading/writing big > > >> files on OS X, I believe they were fixed in Maverick and I'm inclined > > >> to recommend an OS upgrade rather than work to chunk all the io. > > >> > > >> Deprecate NPY_CHAR > > >> This one is waiting on a fix from Pearu to make f2py use numpy > > >> strings. I think we will need to do this ourselves if we want to carry > > >> through the deprecation. In any case it probably needs to be pushed > > >> off to 1.10. > > >> > > >> 1.7 deprecations: Follow-up ticket > > >> Some of these changes have been made, ro diagonal view for instance, > > >> some still remain. > > >> > > >> > > >> > > >> Additions, updates, and thoughts welcome. > > >> > > >> > > >> Chuck > > >> > > >> > > >> > > >> > > >> > > >> > > >> _______________________________________________ > > >> NumPy-Discussion mailing list > > >> NumPy-Discussion at scipy.org > > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Sat Apr 26 14:10:08 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sat, 26 Apr 2014 20:10:08 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, basically the toolchain was created with a local fork of the "mingw-builds" build process along with some addons and patches. It is NOT a mingw-w64 fork. BTW: there are numerous mingw-w64 based toolchains out there, most of them build without any information about the build process and patches they used. As long as the "mingw-builds" maintainers continue working on their project, maintaining usuable toolchain for Python development on Windows should be feasible. More details are given here: http://article.gmane.org/gmane.comp.python.numeric.general/57446 Regards Carl 2014-04-25 7:57 GMT+02:00 Sturla Molden : > Matthew Brett wrote: > > > Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > > (main author of ATLAS), I've built 64-bit windows numpy and scipy > > wheels for testing. > > Thanks for your great effort to solve this mess. > > By Murphy's law, I do not have access to a Windows computer on which to > test now. :-( > > This approach worries me a bit though: Will we have to maintain a fork of > MinGW-w64 for building NumPy and SciPy? Should this toolset be distributed > along with NumPy and SciPy on Windows? I presume it is needed to build C > and Cython extensions? > > On the positive side: Does this mean we finally can use gfortran on > Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy > now? Or is Mac OS X a blocker? > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Apr 26 18:37:25 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 26 Apr 2014 15:37:25 -0700 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: Hi, On Wed, Apr 23, 2014 at 11:59 AM, Matthew Brett wrote: > Hi, > > On Wed, Apr 23, 2014 at 1:43 AM, Nathaniel Smith wrote: >> On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett wrote: >>> Hi, >>> >>> I'm exploring Mingw-w64 for numpy building, and I've found it gives a >>> slightly different answer for 'exp' than - say - gcc on OSX. >>> >>> The difference is of the order of the eps value for the output number >>> (2 * eps for a result of ~2.0). >>> >>> Is accuracy somewhere specified for C functions like exp? Or is >>> accuracy left as an implementation detail for the C library author? >> >> C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point >> operations ... and of the library functions in and >> that return floating point results is implemenetation >> defined. The implementation may state that the accuracy is unknown." >> (This last sentence is basically saying that with regard to some >> higher up clauses that required all conforming implementations to >> document this stuff, saying "eh, who knows" counts as documenting it. >> Hooray for standards!) >> >> Presumably the accuracy in this case is a function of the C library >> anyway, not the compiler? > > Mingw-w64 implementation is in assembly: > > http://sourceforge.net/p/mingw-w64/code/HEAD/tree/trunk/mingw-w64-crt/math/exp.def.h > >> Numpy has its own implementations for a >> bunch of the math functions, and it's been unclear in the past whether >> numpy or the libc implementations were better in any particular case. > > I only investigated this particular value, in which case it looked as > though the OSX value was closer to the exact value (via sympy.mpmath) > - by ~1 unit-at-the-last-place. This was causing a divergence in the > powell optimization path and therefore a single scipy test failure. I > haven't investigated further - was wondering what investigation I > should do, more than running the numpy / scipy test suites. Investigating further, with this script: https://gist.github.com/matthew-brett/11301221 The following are tests of np.exp accuracy for input values between 0 and 10, for numpy 1.8.1. If np.exp(x) performs perfectly, it will return the nearest floating point value to the exact value of exp(x). If it does, this scores a zero for error in the tables below. If 'proportion of zeros' is 1 - then np.exp performs perfectly for all tested values of exp (as is the case for linux here). OSX 10.9 Proportion of zeros: 0.99789 Sum of error: 2.15021267458e-09 Sum of squared error: 2.47149370032e-14 Max / min error: 5.96046447754e-08 -2.98023223877e-08 Sum of squared relative error: 5.22456992025e-30 Max / min relative error: 2.19700100681e-16 -2.2098803255e-16 eps: 2.22044604925e-16 Proportion of relative err >= eps: 0.0 Debian Jessie / Sid Proportion of zeros: 1.0 Sum of error: 0.0 Sum of squared error: 0.0 Max / min error: 0.0 0.0 Sum of squared relative error: 0.0 Max / min relative error: 0.0 0.0 eps: 2.22044604925e-16 Proportion of relative err >= eps: 0.0 Mingw-w64 Windows 7 Proportion of zeros: 0.82089 Sum of error: 8.08415331122e-07 Sum of squared error: 2.90045099615e-12 Max / min error: 5.96046447754e-08 -5.96046447754e-08 Sum of squared relative error: 4.18466468175e-28 Max / min relative error: 2.22041308226e-16 -2.22042100773e-16 eps: 2.22044604925e-16 Proportion of relative err >= eps: 0.0 Take-home : exp implementation for mingw-w64 is exactly (floating point) correct 82% of the time, and one unit-at-the-last-place off for the rest [1]. OSX is off by 1 ULP only 0.2% of the time. Is mingw-w64 accurate enough? Do we have any policy on this? Cheers, Matthew [1] http://matthew-brett.github.io/pydagogue/floating_error.html From ndarray at mac.com Sat Apr 26 19:28:37 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sat, 26 Apr 2014 19:28:37 -0400 Subject: [Numpy-discussion] Proposed new function for joining arrays: np.interleave In-Reply-To: References: Message-ID: On Mon, Apr 7, 2014 at 11:12 AM, Bj?rn Dahlgren wrote: > I think the code needed for the general n dimensional case with m number > of arrays > is non-trivial enough for it to be useful to provide such a function in > numpy > As of version 1.8.1, I count 571 public names in numpy namespace: >>> len([x for x in dir(numpy) if not x.startswith('_')]) 571 Rather than adding 572nd name, we should investigate why it is "non-trivial enough" to express this using existing functions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 26 20:05:20 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 20:05:20 -0400 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: On Sat, Apr 26, 2014 at 6:37 PM, Matthew Brett wrote: > Hi, > > On Wed, Apr 23, 2014 at 11:59 AM, Matthew Brett > wrote: > > Hi, > > > > On Wed, Apr 23, 2014 at 1:43 AM, Nathaniel Smith wrote: > >> On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett > wrote: > >>> Hi, > >>> > >>> I'm exploring Mingw-w64 for numpy building, and I've found it gives a > >>> slightly different answer for 'exp' than - say - gcc on OSX. > >>> > >>> The difference is of the order of the eps value for the output number > >>> (2 * eps for a result of ~2.0). > >>> > >>> Is accuracy somewhere specified for C functions like exp? Or is > >>> accuracy left as an implementation detail for the C library author? > >> > >> C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point > >> operations ... and of the library functions in and > >> that return floating point results is implemenetation > >> defined. The implementation may state that the accuracy is unknown." > >> (This last sentence is basically saying that with regard to some > >> higher up clauses that required all conforming implementations to > >> document this stuff, saying "eh, who knows" counts as documenting it. > >> Hooray for standards!) > >> > >> Presumably the accuracy in this case is a function of the C library > >> anyway, not the compiler? > > > > Mingw-w64 implementation is in assembly: > > > > > http://sourceforge.net/p/mingw-w64/code/HEAD/tree/trunk/mingw-w64-crt/math/exp.def.h > > > >> Numpy has its own implementations for a > >> bunch of the math functions, and it's been unclear in the past whether > >> numpy or the libc implementations were better in any particular case. > > > > I only investigated this particular value, in which case it looked as > > though the OSX value was closer to the exact value (via sympy.mpmath) > > - by ~1 unit-at-the-last-place. This was causing a divergence in the > > powell optimization path and therefore a single scipy test failure. I > > haven't investigated further - was wondering what investigation I > > should do, more than running the numpy / scipy test suites. > > Investigating further, with this script: > > https://gist.github.com/matthew-brett/11301221 > > The following are tests of np.exp accuracy for input values between 0 > and 10, for numpy 1.8.1. > > If np.exp(x) performs perfectly, it will return the nearest floating > point value to the exact value of exp(x). If it does, this scores a > zero for error in the tables below. If 'proportion of zeros' is 1 - > then np.exp performs perfectly for all tested values of exp (as is the > case for linux here). > > OSX 10.9 > > Proportion of zeros: 0.99789 > Sum of error: 2.15021267458e-09 > Sum of squared error: 2.47149370032e-14 > Max / min error: 5.96046447754e-08 -2.98023223877e-08 > Sum of squared relative error: 5.22456992025e-30 > Max / min relative error: 2.19700100681e-16 -2.2098803255e-16 > eps: 2.22044604925e-16 > Proportion of relative err >= eps: 0.0 > > Debian Jessie / Sid > > Proportion of zeros: 1.0 > Sum of error: 0.0 > Sum of squared error: 0.0 > Max / min error: 0.0 0.0 > Sum of squared relative error: 0.0 > Max / min relative error: 0.0 0.0 > eps: 2.22044604925e-16 > Proportion of relative err >= eps: 0.0 > > Mingw-w64 Windows 7 > > Proportion of zeros: 0.82089 > Sum of error: 8.08415331122e-07 > Sum of squared error: 2.90045099615e-12 > Max / min error: 5.96046447754e-08 -5.96046447754e-08 > Sum of squared relative error: 4.18466468175e-28 > Max / min relative error: 2.22041308226e-16 -2.22042100773e-16 > eps: 2.22044604925e-16 > Proportion of relative err >= eps: 0.0 > > Take-home : exp implementation for mingw-w64 is exactly (floating > point) correct 82% of the time, and one unit-at-the-last-place off for > the rest [1]. OSX is off by 1 ULP only 0.2% of the time. > Windows 64 with MKL \WinPython-64bit-3.3.2.2\python-3.3.2.amd64>python "E:\Josef\eclipsegworkspace\statsmodels-git\local_scripts\local_scripts\try_exp_error.py" Proportion of zeros: 0.99793 Sum of error: -2.10546855506e-07 Sum of squared error: 3.33304327526e-14 Max / min error: 5.96046447754e-08 -5.96046447754e-08 Sum of squared relative error: 4.98420694339e-30 Max / min relative error: 2.20881302691e-16 -2.18321571939e-16 eps: 2.22044604925e-16 Proportion of relative err >= eps: 0.0 Windows 32 bit python with official MingW binaries Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32 Proportion of zeros: 0.99464 Sum of error: -3.91621083118e-07 Sum of squared error: 9.2239247812e-14 Max / min error: 5.96046447754e-08 -5.96046447754e-08 Sum of squared relative error: 1.3334972729e-29 Max / min relative error: 2.21593462148e-16 -2.2098803255e-16 eps: 2.22044604925e-16 Proportion of relative err >= eps: 0.0 > > Is mingw-w64 accurate enough? Do we have any policy on this? > I wouldn't worry about a missing or an extra eps in our applications, but the competition is more accurate. Josef > > Cheers, > > Matthew > > [1] http://matthew-brett.github.io/pydagogue/floating_error.html > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 26 20:31:39 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 20:31:39 -0400 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: On Sat, Apr 26, 2014 at 8:05 PM, wrote: > > > > On Sat, Apr 26, 2014 at 6:37 PM, Matthew Brett wrote: > >> Hi, >> >> On Wed, Apr 23, 2014 at 11:59 AM, Matthew Brett >> wrote: >> > Hi, >> > >> > On Wed, Apr 23, 2014 at 1:43 AM, Nathaniel Smith wrote: >> >> On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett < >> matthew.brett at gmail.com> wrote: >> >>> Hi, >> >>> >> >>> I'm exploring Mingw-w64 for numpy building, and I've found it gives a >> >>> slightly different answer for 'exp' than - say - gcc on OSX. >> >>> >> >>> The difference is of the order of the eps value for the output number >> >>> (2 * eps for a result of ~2.0). >> >>> >> >>> Is accuracy somewhere specified for C functions like exp? Or is >> >>> accuracy left as an implementation detail for the C library author? >> >> >> >> C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point >> >> operations ... and of the library functions in and >> >> that return floating point results is implemenetation >> >> defined. The implementation may state that the accuracy is unknown." >> >> (This last sentence is basically saying that with regard to some >> >> higher up clauses that required all conforming implementations to >> >> document this stuff, saying "eh, who knows" counts as documenting it. >> >> Hooray for standards!) >> >> >> >> Presumably the accuracy in this case is a function of the C library >> >> anyway, not the compiler? >> > >> > Mingw-w64 implementation is in assembly: >> > >> > >> http://sourceforge.net/p/mingw-w64/code/HEAD/tree/trunk/mingw-w64-crt/math/exp.def.h >> > >> >> Numpy has its own implementations for a >> >> bunch of the math functions, and it's been unclear in the past whether >> >> numpy or the libc implementations were better in any particular case. >> > >> > I only investigated this particular value, in which case it looked as >> > though the OSX value was closer to the exact value (via sympy.mpmath) >> > - by ~1 unit-at-the-last-place. This was causing a divergence in the >> > powell optimization path and therefore a single scipy test failure. I >> > haven't investigated further - was wondering what investigation I >> > should do, more than running the numpy / scipy test suites. >> >> Investigating further, with this script: >> >> https://gist.github.com/matthew-brett/11301221 >> >> The following are tests of np.exp accuracy for input values between 0 >> and 10, for numpy 1.8.1. >> >> If np.exp(x) performs perfectly, it will return the nearest floating >> point value to the exact value of exp(x). If it does, this scores a >> zero for error in the tables below. If 'proportion of zeros' is 1 - >> then np.exp performs perfectly for all tested values of exp (as is the >> case for linux here). >> >> OSX 10.9 >> >> Proportion of zeros: 0.99789 >> Sum of error: 2.15021267458e-09 >> Sum of squared error: 2.47149370032e-14 >> Max / min error: 5.96046447754e-08 -2.98023223877e-08 >> Sum of squared relative error: 5.22456992025e-30 >> Max / min relative error: 2.19700100681e-16 -2.2098803255e-16 >> eps: 2.22044604925e-16 >> Proportion of relative err >= eps: 0.0 >> >> Debian Jessie / Sid >> >> Proportion of zeros: 1.0 >> Sum of error: 0.0 >> Sum of squared error: 0.0 >> Max / min error: 0.0 0.0 >> Sum of squared relative error: 0.0 >> Max / min relative error: 0.0 0.0 >> eps: 2.22044604925e-16 >> Proportion of relative err >= eps: 0.0 >> >> Mingw-w64 Windows 7 >> >> Proportion of zeros: 0.82089 >> Sum of error: 8.08415331122e-07 >> Sum of squared error: 2.90045099615e-12 >> Max / min error: 5.96046447754e-08 -5.96046447754e-08 >> Sum of squared relative error: 4.18466468175e-28 >> Max / min relative error: 2.22041308226e-16 -2.22042100773e-16 >> eps: 2.22044604925e-16 >> Proportion of relative err >= eps: 0.0 >> >> Take-home : exp implementation for mingw-w64 is exactly (floating >> point) correct 82% of the time, and one unit-at-the-last-place off for >> the rest [1]. OSX is off by 1 ULP only 0.2% of the time. >> > > > Windows 64 with MKL > > \WinPython-64bit-3.3.2.2\python-3.3.2.amd64>python > "E:\Josef\eclipsegworkspace\statsmodels-git\local_scripts\local_scripts\try_exp_error.py" > Proportion of zeros: 0.99793 > Sum of error: -2.10546855506e-07 > Sum of squared error: 3.33304327526e-14 > Max / min error: 5.96046447754e-08 -5.96046447754e-08 > Sum of squared relative error: 4.98420694339e-30 > Max / min relative error: 2.20881302691e-16 -2.18321571939e-16 > eps: 2.22044604925e-16 > Proportion of relative err >= eps: 0.0 > > > Windows 32 bit python with official MingW binaries > > Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit > (Intel)] on win32 > > Proportion of zeros: 0.99464 > Sum of error: -3.91621083118e-07 > Sum of squared error: 9.2239247812e-14 > Max / min error: 5.96046447754e-08 -5.96046447754e-08 > Sum of squared relative error: 1.3334972729e-29 > Max / min relative error: 2.21593462148e-16 -2.2098803255e-16 > eps: 2.22044604925e-16 > Proportion of relative err >= eps: 0.0 > > > >> >> Is mingw-w64 accurate enough? Do we have any policy on this? >> > > I wouldn't worry about a missing or an extra eps in our applications, but > the competition is more accurate. > Just for comparison, I increased `until` to 300 the proportion of zeros and relative error stays about the same for both MKL and your wheels absolute error are huge, the following is MKL Sum of error: -3.78802736366e+112 Sum of squared error: 1.51136754049e+225 Max / min error: 4.80981520952e+111 -3.84785216762e+112 (I looked a lot at the behavior of exp in the hundreds recently :( As illustration why I don't care about one **relative** eps >>> np.finfo(np.double).eps + 10 == 10 True https://github.com/scipy/scipy/pull/3547 and many others Josef > > Josef > > >> >> Cheers, >> >> Matthew >> >> [1] http://matthew-brett.github.io/pydagogue/floating_error.html >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 26 22:03:05 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 26 Apr 2014 22:03:05 -0400 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: On Sat, Apr 26, 2014 at 8:31 PM, wrote: > > > > On Sat, Apr 26, 2014 at 8:05 PM, wrote: > >> >> >> >> On Sat, Apr 26, 2014 at 6:37 PM, Matthew Brett wrote: >> >>> Hi, >>> >>> On Wed, Apr 23, 2014 at 11:59 AM, Matthew Brett >>> wrote: >>> > Hi, >>> > >>> > On Wed, Apr 23, 2014 at 1:43 AM, Nathaniel Smith >>> wrote: >>> >> On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett < >>> matthew.brett at gmail.com> wrote: >>> >>> Hi, >>> >>> >>> >>> I'm exploring Mingw-w64 for numpy building, and I've found it gives a >>> >>> slightly different answer for 'exp' than - say - gcc on OSX. >>> >>> >>> >>> The difference is of the order of the eps value for the output number >>> >>> (2 * eps for a result of ~2.0). >>> >>> >>> >>> Is accuracy somewhere specified for C functions like exp? Or is >>> >>> accuracy left as an implementation detail for the C library author? >>> >> >>> >> C99 says (sec 5.2.4.2.2) that "The accuracy of the floating point >>> >> operations ... and of the library functions in and >>> >> that return floating point results is implemenetation >>> >> defined. The implementation may state that the accuracy is unknown." >>> >> (This last sentence is basically saying that with regard to some >>> >> higher up clauses that required all conforming implementations to >>> >> document this stuff, saying "eh, who knows" counts as documenting it. >>> >> Hooray for standards!) >>> >> >>> >> Presumably the accuracy in this case is a function of the C library >>> >> anyway, not the compiler? >>> > >>> > Mingw-w64 implementation is in assembly: >>> > >>> > >>> http://sourceforge.net/p/mingw-w64/code/HEAD/tree/trunk/mingw-w64-crt/math/exp.def.h >>> > >>> >> Numpy has its own implementations for a >>> >> bunch of the math functions, and it's been unclear in the past whether >>> >> numpy or the libc implementations were better in any particular case. >>> > >>> > I only investigated this particular value, in which case it looked as >>> > though the OSX value was closer to the exact value (via sympy.mpmath) >>> > - by ~1 unit-at-the-last-place. This was causing a divergence in the >>> > powell optimization path and therefore a single scipy test failure. I >>> > haven't investigated further - was wondering what investigation I >>> > should do, more than running the numpy / scipy test suites. >>> >>> Investigating further, with this script: >>> >>> https://gist.github.com/matthew-brett/11301221 >>> >>> The following are tests of np.exp accuracy for input values between 0 >>> and 10, for numpy 1.8.1. >>> >>> If np.exp(x) performs perfectly, it will return the nearest floating >>> point value to the exact value of exp(x). If it does, this scores a >>> zero for error in the tables below. If 'proportion of zeros' is 1 - >>> then np.exp performs perfectly for all tested values of exp (as is the >>> case for linux here). >>> >>> OSX 10.9 >>> >>> Proportion of zeros: 0.99789 >>> Sum of error: 2.15021267458e-09 >>> Sum of squared error: 2.47149370032e-14 >>> Max / min error: 5.96046447754e-08 -2.98023223877e-08 >>> Sum of squared relative error: 5.22456992025e-30 >>> Max / min relative error: 2.19700100681e-16 -2.2098803255e-16 >>> eps: 2.22044604925e-16 >>> Proportion of relative err >= eps: 0.0 >>> >>> Debian Jessie / Sid >>> >>> Proportion of zeros: 1.0 >>> Sum of error: 0.0 >>> Sum of squared error: 0.0 >>> Max / min error: 0.0 0.0 >>> Sum of squared relative error: 0.0 >>> Max / min relative error: 0.0 0.0 >>> eps: 2.22044604925e-16 >>> Proportion of relative err >= eps: 0.0 >>> >>> Mingw-w64 Windows 7 >>> >>> Proportion of zeros: 0.82089 >>> Sum of error: 8.08415331122e-07 >>> Sum of squared error: 2.90045099615e-12 >>> Max / min error: 5.96046447754e-08 -5.96046447754e-08 >>> Sum of squared relative error: 4.18466468175e-28 >>> Max / min relative error: 2.22041308226e-16 -2.22042100773e-16 >>> eps: 2.22044604925e-16 >>> Proportion of relative err >= eps: 0.0 >>> >>> Take-home : exp implementation for mingw-w64 is exactly (floating >>> point) correct 82% of the time, and one unit-at-the-last-place off for >>> the rest [1]. OSX is off by 1 ULP only 0.2% of the time. >>> >> >> >> Windows 64 with MKL >> >> \WinPython-64bit-3.3.2.2\python-3.3.2.amd64>python >> "E:\Josef\eclipsegworkspace\statsmodels-git\local_scripts\local_scripts\try_exp_error.py" >> Proportion of zeros: 0.99793 >> Sum of error: -2.10546855506e-07 >> Sum of squared error: 3.33304327526e-14 >> Max / min error: 5.96046447754e-08 -5.96046447754e-08 >> Sum of squared relative error: 4.98420694339e-30 >> Max / min relative error: 2.20881302691e-16 -2.18321571939e-16 >> eps: 2.22044604925e-16 >> Proportion of relative err >= eps: 0.0 >> >> >> Windows 32 bit python with official MingW binaries >> >> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit >> (Intel)] on win32 >> >> Proportion of zeros: 0.99464 >> Sum of error: -3.91621083118e-07 >> Sum of squared error: 9.2239247812e-14 >> Max / min error: 5.96046447754e-08 -5.96046447754e-08 >> Sum of squared relative error: 1.3334972729e-29 >> Max / min relative error: 2.21593462148e-16 -2.2098803255e-16 >> eps: 2.22044604925e-16 >> Proportion of relative err >= eps: 0.0 >> >> >> >>> >>> Is mingw-w64 accurate enough? Do we have any policy on this? >>> >> >> I wouldn't worry about a missing or an extra eps in our applications, but >> the competition is more accurate. >> > > > Just for comparison, I increased `until` to 300 > the proportion of zeros and relative error stays about the same for both > MKL and your wheels > > absolute error are huge, the following is MKL > > Sum of error: -3.78802736366e+112 > Sum of squared error: 1.51136754049e+225 > Max / min error: 4.80981520952e+111 -3.84785216762e+112 > > (I looked a lot at the behavior of exp in the hundreds recently :( > > As illustration why I don't care about one **relative** eps > >>> np.finfo(np.double).eps + 10 == 10 > True > > https://github.com/scipy/scipy/pull/3547 and many others > another variation on the theme with wheels there is a positive bias, pretty large >>> errors.sum() 8.0841533112163688e-07 but where did it go ? >>> exact_exp.sum() - np.exp(vals).sum() 0.0 (Law of Large Numbers ? errors get swamped or cancel) with MKL smaller negative bias >>> errors.sum() -2.1054685550581098e-07 >>> exact_exp.sum() - np.exp(vals).sum() 0.0 Josef > > > Josef > > >> >> Josef >> >> >>> >>> Cheers, >>> >>> Matthew >>> >>> [1] http://matthew-brett.github.io/pydagogue/floating_error.html >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Apr 27 03:25:58 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 27 Apr 2014 00:25:58 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi Carl, On Sat, Apr 26, 2014 at 11:10 AM, Carl Kleffner wrote: > Hi, > > basically the toolchain was created with a local fork of the "mingw-builds" > build process along with some addons and patches. It is NOT a mingw-w64 > fork. BTW: there are numerous mingw-w64 based toolchains out there, most of > them build without any information about the build process and patches they > used. > > As long as the "mingw-builds" maintainers continue working on their project, > maintaining usuable toolchain for Python development on Windows should be > feasible. > > More details are given here: > http://article.gmane.org/gmane.comp.python.numeric.general/57446 I hope you don't mind, but I took the liberty of putting some of your email explanations and notes into the numpy wiki: https://github.com/numpy/numpy/wiki/Mingw-w64-faq https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Do you have anywhere a description of what you did to create your fork of the build process? Maybe we can automate this using a Fedora or other cross-compiler? I think we need to make sure that the build system need not die if you get hired by some great company and can't work on this anymore. Do you think that is possible? Thanks again for the all the hard work you've done here. I think we are getting very close to a good solution, and that has seemed a long way off until now... Cheers, Matthew From pav at iki.fi Sun Apr 27 07:57:40 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 27 Apr 2014 14:57:40 +0300 Subject: [Numpy-discussion] Slightly off-topic - accuracy of C exp function? In-Reply-To: References: Message-ID: Hi, 27.04.2014 01:37, Matthew Brett kirjoitti: [clip] > Take-home : exp implementation for mingw-w64 is exactly (floating > point) correct 82% of the time, and one unit-at-the-last-place off for > the rest [1]. OSX is off by 1 ULP only 0.2% of the time. > > Is mingw-w64 accurate enough? Do we have any policy on this? I think as mentioned, the C standards don't specify accuracy requirements. Errors of a couple of ULP should be still acceptable. Re: powell test --- if this turns out to be complicated to deal with, just go ahead and disable the trace test. Optimization routines contain statements of the form `if a > b: ...` with floating point numbers, so that the execution path can be sensitive to rounding error if you're unlucky, and the chances go up as the iteration count increases. -- Pauli Virtanen From pav at iki.fi Sun Apr 27 09:09:27 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 27 Apr 2014 16:09:27 +0300 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, 25.04.2014 00:56, Matthew Brett kirjoitti:> Thanks to Cark Kleffner's toolchain and some help from Clint Whaley > (main author of ATLAS), I've built 64-bit windows numpy and scipy > wheels for testing. Where can I get your numpy.patch scipy.patch and what's in them? Cheers, Pauli From matthew.brett at gmail.com Sun Apr 27 14:29:58 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 27 Apr 2014 11:29:58 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, On Sun, Apr 27, 2014 at 6:09 AM, Pauli Virtanen wrote: > Hi, > > 25.04.2014 00:56, Matthew Brett kirjoitti:> Thanks to Cark Kleffner's > toolchain and some help from Clint Whaley >> (main author of ATLAS), I've built 64-bit windows numpy and scipy >> wheels for testing. > > Where can I get your > > numpy.patch > scipy.patch They are Carl's patches - here: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads The scipy patch is tiny, the numpy patch more substantial. Carl - any interest into working up a pull-request for these? I'm happy to do it if you don't have time. Cheers, Matthew From cmkleffner at gmail.com Sun Apr 27 17:34:03 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sun, 27 Apr 2014 23:34:03 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, I will definitly don't have not time until thursday this week working out the github workflow for a numpy pull request. So feel free to do it for me. BTW: There is a missing feature in the mingw-w64 toolchain. By now it features linking to msvcrt90 runtime only. I have do extend the specs file to allow linking to msvcr100 with an addional flag. Or create a dedicated toolchain - what do you think? Cheers, Carl 2014-04-27 20:29 GMT+02:00 Matthew Brett : > Hi, > > On Sun, Apr 27, 2014 at 6:09 AM, Pauli Virtanen wrote: > > Hi, > > > > 25.04.2014 00:56, Matthew Brett kirjoitti:> Thanks to Cark Kleffner's > > toolchain and some help from Clint Whaley > >> (main author of ATLAS), I've built 64-bit windows numpy and scipy > >> wheels for testing. > > > > Where can I get your > > > > numpy.patch > > scipy.patch > > They are Carl's patches - here: > > https://bitbucket.org/carlkl/mingw-w64-for-python/downloads > > The scipy patch is tiny, the numpy patch more substantial. > > Carl - any interest into working up a pull-request for these? I'm > happy to do it if you don't have time. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Apr 27 17:46:52 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 27 Apr 2014 14:46:52 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, On Sun, Apr 27, 2014 at 2:34 PM, Carl Kleffner wrote: > Hi, > > I will definitly don't have not time until thursday this week working out > the github workflow for a numpy pull request. So feel free to do it for me. OK - I will have a go at this tomorrow. > BTW: There is a missing feature in the mingw-w64 toolchain. By now it > features linking to msvcrt90 runtime only. I have do extend the specs file > to allow linking to msvcr100 with an addional flag. Or create a dedicated > toolchain - what do you think? I don't know. Is this a discussion that should go to the mingw-w64 list do you think? It must be a very common feature. As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. The ideal would be a devkit that transparently picked up 32 vs 64 bit, and MSVC runtime according to the Python version. For example, OSX compilation automatically picks up the OSX SDK with which the relevant Python was built. Do you think something like this is possible? That would be a great improvement for people building extensions and wheels on Windows. Cheers, Matthew [1] http://rubyinstaller.org/add-ons/devkit/ From cmkleffner at gmail.com Sun Apr 27 18:06:39 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Mon, 28 Apr 2014 00:06:39 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Cheers, Carl 2014-04-27 23:46 GMT+02:00 Matthew Brett : > Hi, > > On Sun, Apr 27, 2014 at 2:34 PM, Carl Kleffner > wrote: > > Hi, > > > > I will definitly don't have not time until thursday this week working out > > the github workflow for a numpy pull request. So feel free to do it for > me. > > OK - I will have a go at this tomorrow. > > > BTW: There is a missing feature in the mingw-w64 toolchain. By now it > > features linking to msvcrt90 runtime only. I have do extend the specs > file > > to allow linking to msvcr100 with an addional flag. Or create a dedicated > > toolchain - what do you think? > > I don't know. > > Is this a discussion that should go to the mingw-w64 list do you > think? It must be a very common feature. > > As you know, I'm really hoping it will be possible make a devkit for > Python similar to the Ruby devkits [1]. > > The ideal would be a devkit that transparently picked up 32 vs 64 bit, > and MSVC runtime according to the Python version. For example, OSX > compilation automatically picks up the OSX SDK with which the relevant > Python was built. Do you think something like this is possible? That > would be a great improvement for people building extensions and wheels > on Windows. > > Cheers, > > Matthew > > [1] http://rubyinstaller.org/add-ons/devkit/ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Apr 27 18:19:03 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 27 Apr 2014 15:19:03 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner wrote: > A possible option is to install the toolchain inside site-packages and to > deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could > be extended during import of the package. But I have no idea, whats the best > strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I can also give access to the dedicated machine doing the builds. Cheers, Matthew From josef.pktd at gmail.com Sun Apr 27 18:26:38 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 27 Apr 2014 18:26:38 -0400 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Sun, Apr 27, 2014 at 6:06 PM, Carl Kleffner wrote: > A possible option is to install the toolchain inside site-packages and to > deploy it as PYPI wheel or wininst packages. The PATH to the toolchain > could be extended during import of the package. But I have no idea, whats > the best strategy to additionaly install ATLAS or other third party > libraries. > What I did in the past is just to download the ATLAS binaries from the scipy/numpy wiki and move them into the Python/Dlls directory. IIRC My impression was that finding ATLAS binaries was the difficult part, not moving them into the right directory. > > Cheers, > > Carl > > > 2014-04-27 23:46 GMT+02:00 Matthew Brett : > > Hi, >> >> On Sun, Apr 27, 2014 at 2:34 PM, Carl Kleffner >> wrote: >> > Hi, >> > >> > I will definitly don't have not time until thursday this week working >> out >> > the github workflow for a numpy pull request. So feel free to do it for >> me. >> >> OK - I will have a go at this tomorrow. >> >> > BTW: There is a missing feature in the mingw-w64 toolchain. By now it >> > features linking to msvcrt90 runtime only. I have do extend the specs >> file >> > to allow linking to msvcr100 with an addional flag. Or create a >> dedicated >> > toolchain - what do you think? >> >> I don't know. >> >> Is this a discussion that should go to the mingw-w64 list do you >> think? It must be a very common feature. >> >> As you know, I'm really hoping it will be possible make a devkit for >> Python similar to the Ruby devkits [1]. >> > I got my entire initial setup on the computer I'm using right now through python-xy, including MingW 32. The only thing I ever had to do was to create the `distutils.cfg` in new python install. python-xy relies on the availability of a open source development environment for numpy and scipy, and has been restricted so far to python 32 versions. winpython is only a python distribution and is also available for 64bit (with Gohlke binaries I think) I think it would be very helpful to get python-xy set up for development for 64 bit versions, now that the toolchain with MingW is available. I'm skeptical about having lot's of distributions that install all their own full toolchain (I always worry about which one is actually on the path. I deleted my first git for Windows version because it came with a built-in MSYS/MingW toolchain and now just use the nice and small portable version.) > >> The ideal would be a devkit that transparently picked up 32 vs 64 bit, >> and MSVC runtime according to the Python version. For example, OSX >> compilation automatically picks up the OSX SDK with which the relevant >> Python was built. Do you think something like this is possible? That >> would be a great improvement for people building extensions and wheels >> on Windows. >> > How does MingW64 decide whether to build 32 or to build 64 bit versions? Does the python version matter for MingW? or should this pick up one of the Visual SDK's that the user needs to install? Josef > >> Cheers, >> >> Matthew >> >> [1] http://rubyinstaller.org/add-ons/devkit/ >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sun Apr 27 18:39:51 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 27 Apr 2014 22:39:51 +0000 (UTC) Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> Message-ID: <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> Pauli Virtanen wrote: > Yes, Windows is the only platform on which Fortran was problematic. OSX > is somewhat saner in this respect. Oh yes, it seems there are official "unofficial gfortran binaries" available for OSX: http://gcc.gnu.org/wiki/GFortranBinaries#MacOS Cool :) Sturla From matthew.brett at gmail.com Sun Apr 27 18:50:44 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 27 Apr 2014 15:50:44 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett wrote: > Hi, > > On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner wrote: >> A possible option is to install the toolchain inside site-packages and to >> deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could >> be extended during import of the package. But I have no idea, whats the best >> strategy to additionaly install ATLAS or other third party libraries. > > Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the > devkit package. It sounds like OpenBLAS will be much easier to build, > so we could start with ATLAS binaries as a default, expecting OpenBLAS > to be built more often with the toolchain. I think that's how numpy > binary installers are built at the moment - using old binary builds of > ATLAS. > > I'm happy to provide the builds of ATLAS - e.g. here: > > https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Cheers, Matthew From michael.lehn at uni-ulm.de Mon Apr 28 06:25:09 2014 From: michael.lehn at uni-ulm.de (Michael Lehn) Date: Mon, 28 Apr 2014 12:25:09 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : > Sturla Molden wrote: > >> Making a totally new BLAS might seem like a crazy idea, but it might be the >> best solution in the long run. > > To see if this can be done, I'll try to re-implement cblas_dgemm and then > benchmark against MKL, Accelerate and OpenBLAS. If I can get the > performance better than 75% of their speed, without any assembly or dark So what percentage on performance did you achieve so far? Cheers, Michael From lists at hilboll.de Fri Apr 25 07:57:01 2014 From: lists at hilboll.de (Andreas Hilboll) Date: Fri, 25 Apr 2014 13:57:01 +0200 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <53521FAA.4030004@hilboll.de> References: <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> <53521FAA.4030004@hilboll.de> Message-ID: <535A4D8D.6020606@hilboll.de> On 19.04.2014 09:03, Andreas Hilboll wrote: > On 14.04.2014 20:59, Chris Barker wrote: >> On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer > > wrote: >> >> On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris >> > wrote: >> >> Are we in a position to start looking at implementation? If so, >> it would be useful to have a collection of test cases, i.e., >> typical uses with specified results. That should also cover >> conversion from/(to?) datetime.datetime. >> >> >> yup -- tests are always good! >> >> Indeed, my personal wish-list for np.datetime64 is centered much >> more on robust conversion to/from native date objects, including >> comparison. >> >> >> A good use case. >> >> >> Here are some of my particular points of frustration (apologies for >> the thread jacking!): >> - NaT should have similar behavior to NaN when used for comparisons >> (i.e., comparisons should always be False). >> >> >> make sense. >> >> >> - You can't compare a datetime object to a datetime64 object. >> >> >> that would be nice to have. >> >> >> - datetime64 objects with high precision (e.g., ns) can't compare to >> datetime objects. >> >> >> That's a problem, but how do you think it should be handled? My thought >> is that it should round to microseconds, and then compare -- kind of >> like comparing float32 and float64... >> >> >> Pandas has a very nice wrapper around datetime64 arrays that solves >> most of these issues, but it would be nice to get much of that >> functionality in core numpy, >> >> >> yes -- it would -- but learning from pandas is certainly a good idea. >> >> >> from numpy import datetime64 >> from datetime import datetime >> >> print np.datetime64('NaT') < np.datetime64('2011-01-01') # this >> should not to true >> print datetime(2010, 1, 1) < np.datetime64('2011-01-01') # raises >> exception >> print np.datetime64('2011-01-01T00:00', 'ns') > datetime(2010, 1, 1) >> # another exception >> print np.datetime64('2011-01-01T00:00') > datetime(2010, 1, 1) # >> finally something works! >> >> >> now to get them into proper unit tests.... > > As one further suggestion, I think it would be nice if doing arithmetic > using np.datetime64 and datetime.timedelta objects would work: > > np.datetime64(2011,1,1) + datetime.timedelta(1) == > np.datetime64(2011,1,2) > > And of course, but this is probably in the loop anyways, > np.asarray([list_of_datetime.datetime_objects]) should work as expected. One more wish / suggestion from my side (apologies if this isn't the place to make wishes): Array-wide access to the individual datetime components should work, i.e., datetime64array.year should yield an array of dtype int with the years. That would allow boolean indexing to filter data, like datetime64array[datetime64array.year == 2014] would yield all entries from 2014. Cheers, -- Andreas. From ralf.gommers at gmail.com Mon Apr 28 11:39:58 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 28 Apr 2014 17:39:58 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Apr 28, 2014 at 12:39 AM, Sturla Molden wrote: > Pauli Virtanen wrote: > > > Yes, Windows is the only platform on which Fortran was problematic. OSX > > is somewhat saner in this respect. > > Oh yes, it seems there are official "unofficial gfortran binaries" > available for OSX: > > http://gcc.gnu.org/wiki/GFortranBinaries#MacOS > I'd be interested to hear if those work well for you. For people that just want to get things working, I would recommend to use the gfortran installers recommended at http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython. Those work for sure, and alternatives have usually proven to be problematic in the past. Ralf > Cool :) > > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Apr 28 12:06:31 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 28 Apr 2014 16:06:31 +0000 (UTC) Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> Message-ID: <1039873720420393027.053304sturla.molden-gmail.com@news.gmane.org> Ralf Gommers wrote: > I'd be interested to hear if those work well for you. For people that just > want to get things working, I would recommend to use the gfortran > installers recommended at > href="http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython.">http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython. > Those work for sure, and alternatives have usually proven to be problematic > in the past. No problems thus far, but I only installed it yesterday. :-) I am not sure gcc-4.2 is needed anymore. Apple has retired it as platform C compiler on OS X. We need a Fortran compiler that can be used together with clang as C compiler. Sturla From ralf.gommers at gmail.com Mon Apr 28 12:21:59 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 28 Apr 2014 18:21:59 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: <1039873720420393027.053304sturla.molden-gmail.com@news.gmane.org> References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> <1039873720420393027.053304sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Apr 28, 2014 at 6:06 PM, Sturla Molden wrote: > Ralf Gommers wrote: > > > I'd be interested to hear if those work well for you. For people that > just > > want to get things working, I would recommend to use the gfortran > > installers recommended at > > > href=" > http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython > ."> > http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython > . > > Those work for sure, and alternatives have usually proven to be > problematic > > in the past. > > No problems thus far, but I only installed it yesterday. :-) > Sounds good. Let's give it a bit more time, once you've given it a good workout we can add that those gfortran 4.8.x compilers seem to work fine to the scipy build instructions. I am not sure gcc-4.2 is needed anymore. Apple has retired it as platform C > compiler on OS X. We need a Fortran compiler that can be used together with > clang as C compiler. > Clang together with gfortran 4.2 works fine on OS X 10.9. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Apr 28 13:22:19 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 28 Apr 2014 17:22:19 +0000 (UTC) Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> <1039873720420393027.053304sturla.molden-gmail.com@news.gmane.org> Message-ID: <805358462420397589.507528sturla.molden-gmail.com@news.gmane.org> Ralf Gommers wrote: > Sounds good. Let's give it a bit more time, once you've given it a good > workout we can add that those gfortran 4.8.x compilers seem to work fine to > the scipy build instructions. Yes, it needs to be tested properly. The build instructions for OS X Mavericks should also mention where to obtain Xcode (Appstore) and the secret command to retrieve the command-line utils after Xcode is installed: $ /usr/bin/xcode-select --install Probably it should also mention how to use alternative BLAS and LAPACK versions (MKL and OpenBLAS), although all three are equally performant on Mavericks (except Accelerate is not fork safe): https://twitter.com/nedlom/status/437427557919891457 Sturla From ndbecker2 at gmail.com Mon Apr 28 13:36:43 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 28 Apr 2014 13:36:43 -0400 Subject: [Numpy-discussion] should rint return int? Message-ID: I notice rint returns float. Shouldn't it return int? Would be useful when float is no longer acceptable as an index. I think conversion to an index using rint is a common idiom. From chris.barker at noaa.gov Mon Apr 28 13:40:49 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Apr 2014 10:40:49 -0700 Subject: [Numpy-discussion] Dates and times and Datetime64 (again) In-Reply-To: <535A4D8D.6020606@hilboll.de> References: <614DCCFD-BFFC-496D-B721-A08F68FFD6D2@uwaterloo.ca> <4E01B838-715F-4844-AA4A-C8905AFD3ADA@uwaterloo.ca> <53521FAA.4030004@hilboll.de> <535A4D8D.6020606@hilboll.de> Message-ID: On Fri, Apr 25, 2014 at 4:57 AM, Andreas Hilboll wrote: > Array-wide access to the individual datetime components should work, i.e., > > datetime64array.year > > should yield an array of dtype int with the years. That would allow > boolean indexing to filter data, like > > datetime64array[datetime64array.year == 2014] > > would yield all entries from 2014. > that would be nice, yes, but datetime64 doesn't support anything like that at all -- i.e. array-wide or not access to the components. In this case, you could kludge it with: In [19]: datetimearray Out[19]: array(['2014-02-03', '2013-03-08', '2012-03-07', '2014-04-06'], dtype='datetime64[D]') In [20]: datetimearray[datetimearray.astype('datetime64[Y]') == np.datetime64('2014')] Out[20]: array(['2014-02-03', '2014-04-06'], dtype='datetime64[D]') but that wouldn't work for months, for instance. I think the current NEP should stick with simply fixing the timezone thing -- no new functionality or consequence. But: Maybe it's time for a new NEP for what we want datetime64 to be in the future -- maybe borrow from the blaze proposal cited earlier? Or wait and see how that works out, then maybe port that code over to numpy? In the meantime, a set of utilities that do the kind of things you're looking for might make sense. You could do it as a ndarray subclass, and add those sorts of methods, though ndarray subclasses do get messy.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Apr 28 13:49:21 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Apr 2014 10:49:21 -0700 Subject: [Numpy-discussion] should rint return int? In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 10:36 AM, Neal Becker wrote: > I notice rint returns float. Shouldn't it return int? > AFAICT, rint() is the same as round(), except with slightly different rules for the "halfway" case. So returning a float makes sense, as round() and ceil() and floor() all do. ( though I've always thought those should return inegers, too... ) By the way, what IS the difference between rint and round? In [37]: for val in [-2.5, -3.5, 2.5, 3.5]: ....: assert np.rint(val) == np.round(val) ....: In [38]: -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.go -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Apr 28 13:54:11 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Apr 2014 10:54:11 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett wrote: > As you know, I'm really hoping it will be possible make a devkit for > Python similar to the Ruby devkits [1]. > That would be great! >From a really quick glance, it looks like we could almost use the Ruby Devkit, maybe adding a couple add-ons.. What do they do for 64 bit? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 28 15:21:59 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 28 Apr 2014 20:21:59 +0100 Subject: [Numpy-discussion] should rint return int? In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 6:36 PM, Neal Becker wrote: > I notice rint returns float. Shouldn't it return int? > > Would be useful when float is no longer acceptable as an index. I think > conversion to an index using rint is a common idiom. C's rint() does not: http://linux.die.net/man/3/rint This is because there are many integers that are representable as floats/doubles/long doubles that are well outside of the range of any C integer type, e.g. 1e20. Python 3's round() can return a Python int because Python ints are unbounded. Ours aren't. That said, typically the first thing anyone does with the result of rounding is to coerce it to a native int dtype without any checking. It would not be terrible to have a function that rounds, then coerces to int but checks for overflow and passes that through the numpy error mechanism to be controlled. But it shouldn't be called rint(), which is intended to be as thin a wrapper over the C function as possible. -- Robert Kern From ndbecker2 at gmail.com Mon Apr 28 15:29:59 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 28 Apr 2014 15:29:59 -0400 Subject: [Numpy-discussion] should rint return int? References: Message-ID: Robert Kern wrote: > On Mon, Apr 28, 2014 at 6:36 PM, Neal Becker wrote: >> I notice rint returns float. Shouldn't it return int? >> >> Would be useful when float is no longer acceptable as an index. I think >> conversion to an index using rint is a common idiom. > > C's rint() does not: > > http://linux.die.net/man/3/rint > > This is because there are many integers that are representable as > floats/doubles/long doubles that are well outside of the range of any > C integer type, e.g. 1e20. > > Python 3's round() can return a Python int because Python ints are > unbounded. Ours aren't. > > That said, typically the first thing anyone does with the result of > rounding is to coerce it to a native int dtype without any checking. > It would not be terrible to have a function that rounds, then coerces > to int but checks for overflow and passes that through the numpy error > mechanism to be controlled. But it shouldn't be called rint(), which > is intended to be as thin a wrapper over the C function as possible. > Well I'd spell it nint, and it works like: def nint (x): return int (x + 0.5) if x >= 0 else int (x - 0.5) From alan.isaac at gmail.com Mon Apr 28 15:36:53 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Mon, 28 Apr 2014 15:36:53 -0400 Subject: [Numpy-discussion] should rint return int? In-Reply-To: References: Message-ID: <535EADD5.1010200@gmail.com> On 4/28/2014 3:29 PM, Neal Becker wrote: > Well I'd spell it nint, and it works like: Wouldn't it be simpler to add a dtype argument to `rint`? Or does that violate the "simple wrapper" intent? Alan Isaac From robert.kern at gmail.com Mon Apr 28 15:38:32 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 28 Apr 2014 20:38:32 +0100 Subject: [Numpy-discussion] should rint return int? In-Reply-To: <535EADD5.1010200@gmail.com> References: <535EADD5.1010200@gmail.com> Message-ID: On Mon, Apr 28, 2014 at 8:36 PM, Alan G Isaac wrote: > On 4/28/2014 3:29 PM, Neal Becker wrote: >> Well I'd spell it nint, and it works like: > > Wouldn't it be simpler to add a dtype argument to `rint`? > Or does that violate the "simple wrapper" intent? `np.rint()` is a ufunc. -- Robert Kern From sturla.molden at gmail.com Mon Apr 28 16:44:53 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 28 Apr 2014 22:44:53 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: <1032885051420097733.605102sturla.molden-gmail.com@news.gmane.org> <1000506843420330985.125857sturla.molden-gmail.com@news.gmane.org> <1039873720420393027.053304sturla.molden-gmail.com@news.gmane.org> Message-ID: On 28/04/14 18:21, Ralf Gommers wrote: > No problems thus far, but I only installed it yesterday. :-) > > > Sounds good. Let's give it a bit more time, once you've given it a good > workout we can add that those gfortran 4.8.x compilers seem to work fine > to the scipy build instructions. I have not looked at building SciPy yet, but I was able to build MPICH 3.0.4 from source without a problem. I worked on the first attempt without any error or warnings. That is more than I hoped for... Using BLAS and LAPACK from Accelerate also worked correctly with flags -ff2c and -framework Accelerate. I can use it from Python (NumPy) with ctypes and Cython. I get correct results and it does not segfault. (It does segfault without -ff2c, but that is as expected, given that Accelerate has f2c/g77 ABI.) I was also able to build OpenBLAS with Clang as C compiler and gfortran as Fortran compiler. It works correctly as well (both the build process and the binaries I get). So far it looks damn good :-) The next step is to build NumPy and SciPy and run some tests :-) Sturla P.S. Here is what I did to build MPICH from source, for those interested: $./configure CC=clang CXX=clang++ F77=gfortran FC=gfortran --enable-fast=all,O3 --with-pm=gforker --prefix=/opt/mpich $ make $ sudo make install $ export PATH="/opt/mpich/bin:$PATH" # actually in ~/.bash_profile Now testing with some hello worlds: $ mpif77 -o hello hello.f $ mpiexec -np 4 ./hello Hello world Hello world Hello world Hello world $ rm hello $ mpicc -o hello hello.c $ mpiexec -np 4 ./hello Hello world from process 0 of 4 Hello world from process 1 of 4 Hello world from process 2 of 4 Hello world from process 3 of 4 The hello world programs looked like this: #include #include int main (int argc, char *argv[]) { int rank, size; MPI_Init (&argc, &argv); MPI_Comm_rank (MPI_COMM_WORLD, &rank); MPI_Comm_size (MPI_COMM_WORLD, &size); printf( "Hello world from process %d of %d\n", rank, size); MPI_Finalize(); return 0; } program hello_world include 'mpif.h' integer ierr call MPI_INIT(ierr) print *, "Hello world" call MPI_FINALIZE(ierr) stop end From njs at pobox.com Mon Apr 28 17:04:37 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 28 Apr 2014 22:04:37 +0100 Subject: [Numpy-discussion] should rint return int? In-Reply-To: References: Message-ID: On 28 Apr 2014 20:22, "Robert Kern" wrote: > C's rint() does not: > > http://linux.die.net/man/3/rint > > This is because there are many integers that are representable as > floats/doubles/long doubles that are well outside of the range of any > C integer type, e.g. 1e20. By the time you have a double integer that isn't representable as an int64, you're well into the range where all doubles are integers but not all integers are floats. "Round to the nearest integer" is already a pretty semantically weird operation for such values. I'm not sure what the consequences of this are for the discussion but it seems worth pointing out. > Python 3's round() can return a Python int because Python ints are > unbounded. Ours aren't. > > That said, typically the first thing anyone does with the result of > rounding is to coerce it to a native int dtype without any checking. > It would not be terrible to have a function that rounds, then coerces > to int but checks for overflow and passes that through the numpy error > mechanism to be controlled. It would help here if we had a consistent mechanism for handling integer representability errors :-). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 28 18:29:31 2014 From: cournape at gmail.com (David Cournapeau) Date: Mon, 28 Apr 2014 23:29:31 +0100 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett wrote: > Aha, > > On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett > wrote: > > Hi, > > > > On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner > wrote: > >> A possible option is to install the toolchain inside site-packages and > to > >> deploy it as PYPI wheel or wininst packages. The PATH to the toolchain > could > >> be extended during import of the package. But I have no idea, whats the > best > >> strategy to additionaly install ATLAS or other third party libraries. > > > > Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the > > devkit package. It sounds like OpenBLAS will be much easier to build, > > so we could start with ATLAS binaries as a default, expecting OpenBLAS > > to be built more often with the toolchain. I think that's how numpy > > binary installers are built at the moment - using old binary builds of > > ATLAS. > > > > I'm happy to provide the builds of ATLAS - e.g. here: > > > > https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds > > I just found the official numpy binary builds of ATLAS: > > https://github.com/numpy/vendor/tree/master/binaries > > But - they are from an old version of ATLAS / Lapack, and only for 32-bit. > > David - what say we update these to latest ATLAS stable? > Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. David > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Apr 28 19:30:25 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Apr 2014 00:30:25 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: > > Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : > >> Sturla Molden wrote: >> >>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>> best solution in the long run. >> >> To see if this can be done, I'll try to re-implement cblas_dgemm and then >> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >> performance better than 75% of their speed, without any assembly or dark > > So what percentage on performance did you achieve so far? I finally read this paper: http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf and I have to say that I'm no longer so convinced that OpenBLAS is the right starting point. They make a compelling argument that BLIS *is* the cleaned up, maintainable, and yet still competitive reimplementation of GotoBLAS/OpenBLAS that we all want, and that getting there required a qualitative reorganization of the code (i.e., very hard to do incrementally). But they've done it. And, I get the impression that the stuff they're missing -- threading, cross-platform build stuff, and runtime CPU adaptation -- is all pretty straightforward stuff that is only missing because no-one's gotten around to sitting down and implementing it. (In particular that paper does include impressive threading results; it sounds like given a decent thread pool library one could get competitive performance pretty trivially, it's just that they haven't been bothered yet to do thread pools properly or systematically test which of the pretty-good approaches to threading is "best". Which is important if your goal is to write papers about BLAS libraries but irrelevant to reaching minimal-viable-product stage.) It would be really interesting if someone were to try hacking simple runtime CPU detection into BLIS and see how far you could get -- right now they do kernel selection via the C preprocessor, but hacking in some function pointer thing instead would not be that hard I think. A maintainable library that builds on Linux/OSX/Windows, gets competitive performance on last-but-one generation x86-64 CPUs, and gets better-than-reference-BLAS performance everywhere else, would be a very very compelling product that I bet would quickly attract the necessary attention to make it competitive on all CPUs. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From sturla.molden at gmail.com Mon Apr 28 19:52:00 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 29 Apr 2014 01:52:00 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On 29/04/14 01:30, Nathaniel Smith wrote: > I finally read this paper: > > http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf > > and I have to say that I'm no longer so convinced that OpenBLAS is the > right starting point. I think OpenBLAS in the long run is doomed as an OSS project. Having huge portions of the source in assembly is not sustainable in 2014. OpenBLAS (like GotoBLAS2 before it) runs a high risk of becoming abandonware. Sturla From njs at pobox.com Mon Apr 28 20:01:14 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Apr 2014 01:01:14 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 29, 2014 at 12:52 AM, Sturla Molden wrote: > On 29/04/14 01:30, Nathaniel Smith wrote: > >> I finally read this paper: >> >> http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf >> >> and I have to say that I'm no longer so convinced that OpenBLAS is the >> right starting point. > > I think OpenBLAS in the long run is doomed as an OSS project. Having > huge portions of the source in assembly is not sustainable in 2014. > OpenBLAS (like GotoBLAS2 before it) runs a high risk of becoming > abandonware. Have you read the paper I linked? I really recommend it. BLIS is apparently 95% straight-up-C, plus a slot where you stick in a tiny CPU-specific super-optimized kernel [1]. So this localizes the nasty stuff to one tiny function, plus most of the kernels that have been written so far do in fact use intrinsics [2]. [1] https://code.google.com/p/blis/wiki/KernelsHowTo [2] https://code.google.com/p/blis/wiki/HardwareSupport -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Mon Apr 28 20:05:35 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Apr 2014 17:05:35 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: > On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: >> >> Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : >> >>> Sturla Molden wrote: >>> >>>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>>> best solution in the long run. >>> >>> To see if this can be done, I'll try to re-implement cblas_dgemm and then >>> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >>> performance better than 75% of their speed, without any assembly or dark >> >> So what percentage on performance did you achieve so far? > > I finally read this paper: > > http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf > > and I have to say that I'm no longer so convinced that OpenBLAS is the > right starting point. They make a compelling argument that BLIS *is* > the cleaned up, maintainable, and yet still competitive > reimplementation of GotoBLAS/OpenBLAS that we all want, and that > getting there required a qualitative reorganization of the code (i.e., > very hard to do incrementally). But they've done it. And, I get the > impression that the stuff they're missing -- threading, cross-platform > build stuff, and runtime CPU adaptation -- is all pretty > straightforward stuff that is only missing because no-one's gotten > around to sitting down and implementing it. (In particular that paper > does include impressive threading results; it sounds like given a > decent thread pool library one could get competitive performance > pretty trivially, it's just that they haven't been bothered yet to do > thread pools properly or systematically test which of the pretty-good > approaches to threading is "best". Which is important if your goal is > to write papers about BLAS libraries but irrelevant to reaching > minimal-viable-product stage.) > > It would be really interesting if someone were to try hacking simple > runtime CPU detection into BLIS and see how far you could get -- right > now they do kernel selection via the C preprocessor, but hacking in > some function pointer thing instead would not be that hard I think. A > maintainable library that builds on Linux/OSX/Windows, gets > competitive performance on last-but-one generation x86-64 CPUs, and > gets better-than-reference-BLAS performance everywhere else, would be > a very very compelling product that I bet would quickly attract the > necessary attention to make it competitive on all CPUs. I wonder - is there anyone who might be able to do this work, if we found funding for a couple of months to do it? Cheers, Matthew From jtaylor.debian at googlemail.com Mon Apr 28 20:10:25 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 29 Apr 2014 02:10:25 +0200 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: <535EEDF1.6000302@googlemail.com> On 29.04.2014 02:05, Matthew Brett wrote: > Hi, > > On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: >> On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: >>> >>> Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : >>> >>>> Sturla Molden wrote: >>>> >>>>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>>>> best solution in the long run. >>>> >>>> To see if this can be done, I'll try to re-implement cblas_dgemm and then >>>> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >>>> performance better than 75% of their speed, without any assembly or dark >>> >>> So what percentage on performance did you achieve so far? >> >> I finally read this paper: >> >> http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf >> >> and I have to say that I'm no longer so convinced that OpenBLAS is the >> right starting point. They make a compelling argument that BLIS *is* >> the cleaned up, maintainable, and yet still competitive >> reimplementation of GotoBLAS/OpenBLAS that we all want, and that >> getting there required a qualitative reorganization of the code (i.e., >> very hard to do incrementally). But they've done it. And, I get the >> impression that the stuff they're missing -- threading, cross-platform >> build stuff, and runtime CPU adaptation -- is all pretty >> straightforward stuff that is only missing because no-one's gotten >> around to sitting down and implementing it. (In particular that paper >> does include impressive threading results; it sounds like given a >> decent thread pool library one could get competitive performance >> pretty trivially, it's just that they haven't been bothered yet to do >> thread pools properly or systematically test which of the pretty-good >> approaches to threading is "best". Which is important if your goal is >> to write papers about BLAS libraries but irrelevant to reaching >> minimal-viable-product stage.) >> >> It would be really interesting if someone were to try hacking simple >> runtime CPU detection into BLIS and see how far you could get -- right >> now they do kernel selection via the C preprocessor, but hacking in >> some function pointer thing instead would not be that hard I think. A >> maintainable library that builds on Linux/OSX/Windows, gets >> competitive performance on last-but-one generation x86-64 CPUs, and >> gets better-than-reference-BLAS performance everywhere else, would be >> a very very compelling product that I bet would quickly attract the >> necessary attention to make it competitive on all CPUs. > > I wonder - is there anyone who might be able to do this work, if we > found funding for a couple of months to do it? > On scipy-dev a interesting BLIS related message was posted recently: http://mail.scipy.org/pipermail/scipy-dev/2014-April/019790.html http://www.cs.utexas.edu/~flame/web/ It seems some work of integrating BLIS into a proper BLAS/LAPACK library is already done. From njs at pobox.com Mon Apr 28 20:23:36 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Apr 2014 01:23:36 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <535EEDF1.6000302@googlemail.com> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> <535EEDF1.6000302@googlemail.com> Message-ID: On Tue, Apr 29, 2014 at 1:10 AM, Julian Taylor wrote: > On 29.04.2014 02:05, Matthew Brett wrote: >> Hi, >> >> On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: >>> It would be really interesting if someone were to try hacking simple >>> runtime CPU detection into BLIS and see how far you could get -- right >>> now they do kernel selection via the C preprocessor, but hacking in >>> some function pointer thing instead would not be that hard I think. A >>> maintainable library that builds on Linux/OSX/Windows, gets >>> competitive performance on last-but-one generation x86-64 CPUs, and >>> gets better-than-reference-BLAS performance everywhere else, would be >>> a very very compelling product that I bet would quickly attract the >>> necessary attention to make it competitive on all CPUs. >> >> I wonder - is there anyone who might be able to do this work, if we >> found funding for a couple of months to do it? > > On scipy-dev a interesting BLIS related message was posted recently: > http://mail.scipy.org/pipermail/scipy-dev/2014-April/019790.html > http://www.cs.utexas.edu/~flame/web/ > > It seems some work of integrating BLIS into a proper BLAS/LAPACK library > is already done. BLIS itself ships with a BLAS-compatible interface, that you can use with reference LAPACK (just like OpenBLAS). I wouldn't be surprised if there are various annoying Fortran/C ABI hacks remaining to be worked out, but at least in principle BLIS is a BLAS. The problem is that this BLAS has no threading, runtime configuration (you have to edit a config file and recompile to change CPU support), or windows build goop. Basically the authors seem to still be thinking of a BLAS library's target audience as being supercomputer sysadmins, not naive end-users. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Mon Apr 28 20:41:35 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Apr 2014 17:41:35 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: <535EEDF1.6000302@googlemail.com> References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> <535EEDF1.6000302@googlemail.com> Message-ID: Hi, On Mon, Apr 28, 2014 at 5:10 PM, Julian Taylor wrote: > On 29.04.2014 02:05, Matthew Brett wrote: >> Hi, >> >> On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: >>> On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: >>>> >>>> Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : >>>> >>>>> Sturla Molden wrote: >>>>> >>>>>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>>>>> best solution in the long run. >>>>> >>>>> To see if this can be done, I'll try to re-implement cblas_dgemm and then >>>>> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >>>>> performance better than 75% of their speed, without any assembly or dark >>>> >>>> So what percentage on performance did you achieve so far? >>> >>> I finally read this paper: >>> >>> http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf >>> >>> and I have to say that I'm no longer so convinced that OpenBLAS is the >>> right starting point. They make a compelling argument that BLIS *is* >>> the cleaned up, maintainable, and yet still competitive >>> reimplementation of GotoBLAS/OpenBLAS that we all want, and that >>> getting there required a qualitative reorganization of the code (i.e., >>> very hard to do incrementally). But they've done it. And, I get the >>> impression that the stuff they're missing -- threading, cross-platform >>> build stuff, and runtime CPU adaptation -- is all pretty >>> straightforward stuff that is only missing because no-one's gotten >>> around to sitting down and implementing it. (In particular that paper >>> does include impressive threading results; it sounds like given a >>> decent thread pool library one could get competitive performance >>> pretty trivially, it's just that they haven't been bothered yet to do >>> thread pools properly or systematically test which of the pretty-good >>> approaches to threading is "best". Which is important if your goal is >>> to write papers about BLAS libraries but irrelevant to reaching >>> minimal-viable-product stage.) >>> >>> It would be really interesting if someone were to try hacking simple >>> runtime CPU detection into BLIS and see how far you could get -- right >>> now they do kernel selection via the C preprocessor, but hacking in >>> some function pointer thing instead would not be that hard I think. A >>> maintainable library that builds on Linux/OSX/Windows, gets >>> competitive performance on last-but-one generation x86-64 CPUs, and >>> gets better-than-reference-BLAS performance everywhere else, would be >>> a very very compelling product that I bet would quickly attract the >>> necessary attention to make it competitive on all CPUs. >> >> I wonder - is there anyone who might be able to do this work, if we >> found funding for a couple of months to do it? >> > > On scipy-dev a interesting BLIS related message was posted recently: > http://mail.scipy.org/pipermail/scipy-dev/2014-April/019790.html > http://www.cs.utexas.edu/~flame/web/ > > It seems some work of integrating BLIS into a proper BLAS/LAPACK library > is already done. Has anyone tried building scipy with libflame yet? Cheers, Matthew From njs at pobox.com Mon Apr 28 20:50:02 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Apr 2014 01:50:02 +0100 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Apr 29, 2014 at 1:05 AM, Matthew Brett wrote: > Hi, > > On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: >> On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: >>> >>> Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : >>> >>>> Sturla Molden wrote: >>>> >>>>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>>>> best solution in the long run. >>>> >>>> To see if this can be done, I'll try to re-implement cblas_dgemm and then >>>> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >>>> performance better than 75% of their speed, without any assembly or dark >>> >>> So what percentage on performance did you achieve so far? >> >> I finally read this paper: >> >> http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf >> >> and I have to say that I'm no longer so convinced that OpenBLAS is the >> right starting point. They make a compelling argument that BLIS *is* >> the cleaned up, maintainable, and yet still competitive >> reimplementation of GotoBLAS/OpenBLAS that we all want, and that >> getting there required a qualitative reorganization of the code (i.e., >> very hard to do incrementally). But they've done it. And, I get the >> impression that the stuff they're missing -- threading, cross-platform >> build stuff, and runtime CPU adaptation -- is all pretty >> straightforward stuff that is only missing because no-one's gotten >> around to sitting down and implementing it. (In particular that paper >> does include impressive threading results; it sounds like given a >> decent thread pool library one could get competitive performance >> pretty trivially, it's just that they haven't been bothered yet to do >> thread pools properly or systematically test which of the pretty-good >> approaches to threading is "best". Which is important if your goal is >> to write papers about BLAS libraries but irrelevant to reaching >> minimal-viable-product stage.) >> >> It would be really interesting if someone were to try hacking simple >> runtime CPU detection into BLIS and see how far you could get -- right >> now they do kernel selection via the C preprocessor, but hacking in >> some function pointer thing instead would not be that hard I think. A >> maintainable library that builds on Linux/OSX/Windows, gets >> competitive performance on last-but-one generation x86-64 CPUs, and >> gets better-than-reference-BLAS performance everywhere else, would be >> a very very compelling product that I bet would quickly attract the >> necessary attention to make it competitive on all CPUs. > > I wonder - is there anyone who might be able to do this work, if we > found funding for a couple of months to do it? Not much point in worrying about this I think until someone tries a proof of concept. But potentially even the labs working on BLIS would be interested in a small grant from NumFOCUS or something. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From matthew.brett at gmail.com Tue Apr 29 00:09:47 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Apr 2014 21:09:47 -0700 Subject: [Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) In-Reply-To: References: <46818810418925962.495791sturla.molden-gmail.com@news.gmane.org> <517271708418928107.376969sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Mon, Apr 28, 2014 at 5:50 PM, Nathaniel Smith wrote: > On Tue, Apr 29, 2014 at 1:05 AM, Matthew Brett wrote: >> Hi, >> >> On Mon, Apr 28, 2014 at 4:30 PM, Nathaniel Smith wrote: >>> On Mon, Apr 28, 2014 at 11:25 AM, Michael Lehn wrote: >>>> >>>> Am 11 Apr 2014 um 19:05 schrieb Sturla Molden : >>>> >>>>> Sturla Molden wrote: >>>>> >>>>>> Making a totally new BLAS might seem like a crazy idea, but it might be the >>>>>> best solution in the long run. >>>>> >>>>> To see if this can be done, I'll try to re-implement cblas_dgemm and then >>>>> benchmark against MKL, Accelerate and OpenBLAS. If I can get the >>>>> performance better than 75% of their speed, without any assembly or dark >>>> >>>> So what percentage on performance did you achieve so far? >>> >>> I finally read this paper: >>> >>> http://www.cs.utexas.edu/users/flame/pubs/blis2_toms_rev2.pdf >>> >>> and I have to say that I'm no longer so convinced that OpenBLAS is the >>> right starting point. They make a compelling argument that BLIS *is* >>> the cleaned up, maintainable, and yet still competitive >>> reimplementation of GotoBLAS/OpenBLAS that we all want, and that >>> getting there required a qualitative reorganization of the code (i.e., >>> very hard to do incrementally). But they've done it. And, I get the >>> impression that the stuff they're missing -- threading, cross-platform >>> build stuff, and runtime CPU adaptation -- is all pretty >>> straightforward stuff that is only missing because no-one's gotten >>> around to sitting down and implementing it. (In particular that paper >>> does include impressive threading results; it sounds like given a >>> decent thread pool library one could get competitive performance >>> pretty trivially, it's just that they haven't been bothered yet to do >>> thread pools properly or systematically test which of the pretty-good >>> approaches to threading is "best". Which is important if your goal is >>> to write papers about BLAS libraries but irrelevant to reaching >>> minimal-viable-product stage.) >>> >>> It would be really interesting if someone were to try hacking simple >>> runtime CPU detection into BLIS and see how far you could get -- right >>> now they do kernel selection via the C preprocessor, but hacking in >>> some function pointer thing instead would not be that hard I think. A >>> maintainable library that builds on Linux/OSX/Windows, gets >>> competitive performance on last-but-one generation x86-64 CPUs, and >>> gets better-than-reference-BLAS performance everywhere else, would be >>> a very very compelling product that I bet would quickly attract the >>> necessary attention to make it competitive on all CPUs. >> >> I wonder - is there anyone who might be able to do this work, if we >> found funding for a couple of months to do it? > > Not much point in worrying about this I think until someone tries a > proof of concept. But potentially even the labs working on BLIS would > be interested in a small grant from NumFOCUS or something. The problem is the time and mental energy involved in the proof-of-concept may be enough to prevent it being done, and having some money to pay for time and to placate employers may be useful in overcoming that. To be clear - not me - I will certainly help if I can, but being paid isn't going to help me work on this. Cheers, Matthew From matthew.brett at gmail.com Tue Apr 29 05:19:08 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Apr 2014 02:19:08 -0700 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: Hi, On Mon, Apr 28, 2014 at 10:54 AM, Chris Barker wrote: > On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett > wrote: >> >> As you know, I'm really hoping it will be possible make a devkit for >> Python similar to the Ruby devkits [1]. > > > That would be great! > > From a really quick glance, it looks like we could almost use the Ruby > Devkit, maybe adding a couple add-ons.. Please, Carl correct me if I'm wrong, but I think the main issues are: 1) Linking to the right MSVC runtime for each Python version (seems to need different gcc specs) 2) Static linking - Carl's toolchain does full static linking including C runtimes so we don't need to deal with the DLL path. I'm not sure what the ruby devkit does to solve that. > What do they do for 64 bit? It looks like they have their own Mingw-w64 toolchain. Cheers, Matthew From cmkleffner at gmail.com Tue Apr 29 09:37:31 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Tue, 29 Apr 2014 15:37:31 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: References: Message-ID: (1) Yes, Support for MSVC100 (python-3.3 and up) is on the TODO list (2) both toolchains are configured for static linking. No need to deploy: libgcc_s_dw2-1.dll, libgomp-1.dll, libquadmath-0.dll, libstdc++-6.dll, libgfortran-3.dll or libwinpthread-1.dll (3) I decided to create two dedicated toolchains for 32bit and for 64bit Regards, Carl 2014-04-29 11:19 GMT+02:00 Matthew Brett : > Hi, > > On Mon, Apr 28, 2014 at 10:54 AM, Chris Barker > wrote: > > On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett > > wrote: > >> > >> As you know, I'm really hoping it will be possible make a devkit for > >> Python similar to the Ruby devkits [1]. > > > > > > That would be great! > > > > From a really quick glance, it looks like we could almost use the Ruby > > Devkit, maybe adding a couple add-ons.. > > Please, Carl correct me if I'm wrong, but I think the main issues are: > > 1) Linking to the right MSVC runtime for each Python version (seems to > need different gcc specs) > 2) Static linking - Carl's toolchain does full static linking > including C runtimes so we don't need to deal with the DLL path. I'm > not sure what the ruby devkit does to solve that. > > > What do they do for 64 bit? > > It looks like they have their own Mingw-w64 toolchain. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Apr 29 11:10:06 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 29 Apr 2014 15:10:06 +0000 (UTC) Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing References: Message-ID: <1693694134420476724.034354sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > 2) Static linking - Carl's toolchain does full static linking > including C runtimes The C runtime cannot be statically linked. It would mean that we get multiple copies of errno and multiple malloc heaps in the process ? one of each static CRT. We must use the same C runtime DLL as Python. But loading it is not a problem because Python has done that before NumPy is imported. Sturla From cmkleffner at gmail.com Tue Apr 29 14:21:52 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Tue, 29 Apr 2014 20:21:52 +0200 Subject: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing In-Reply-To: <1693694134420476724.034354sturla.molden-gmail.com@news.gmane.org> References: <1693694134420476724.034354sturla.molden-gmail.com@news.gmane.org> Message-ID: Correction: gcc (mingw) runtimes are statically linked. The C-runtime DLL msvcrXXX is linked dynamically. Carl 2014-04-29 17:10 GMT+02:00 Sturla Molden : > Matthew Brett wrote: > > > 2) Static linking - Carl's toolchain does full static linking > > including C runtimes > > The C runtime cannot be statically linked. It would mean that we get > multiple copies of errno and multiple malloc heaps in the process ? one of > each static CRT. We must use the same C runtime DLL as Python. But loading > it is not a problem because Python has done that before NumPy is imported. > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: