From warren.weckesser at gmail.com Sat Nov 1 10:31:28 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 1 Nov 2014 10:31:28 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: <5423196F.4000507@gmail.com> References: <5423196F.4000507@gmail.com> Message-ID: On 9/24/14, Alan G Isaac wrote: > On 9/24/2014 2:52 PM, Jaime Fern?ndez del R?o wrote: >> There is a PR in github that adds a new keyword to the genfromtxt >> function, to limit the number of rows that actually get read in: >> https://github.com/numpy/numpy/pull/5103 > > Sorry to come late to this party, but it seems to me that > more versatile than an `nrows` keyword for the number of rows > would be a "rows" keyword for a slice argument. > > fwiw, > Alan Isaac > I've continued the PR for the addition of the `nrows` (now `max_rows`) argument to `genfromtxt` here: https://github.com/numpy/numpy/pull/5253 Alan's suggestion to use a slice is interesting, but I'd like to see a more concrete proposal for the API. For example, how does it interact with `skip_header` and `skip_footer`? How would one use it to read a file in batches? The following are a couple use-cases for `max_rows` (originally added as comments at https://github.com/numpy/numpy/pull/5103): (1) Read a file in batches: Suppose the file "a.csv" contains: 0 10 1 11 2 12 3 13 4 14 5 15 6 16 7 17 8 18 9 19 With `max_rows`, the file can be read in batches of, say, 4: In [31]: f = open("a.csv", "r") In [32]: genfromtxt(f, dtype=None, max_rows=4) Out[32]: array([[ 0, 10], [ 1, 11], [ 2, 12], [ 3, 13]]) In [33]: genfromtxt(f, dtype=None, max_rows=4) Out[33]: array([[ 4, 14] [ 5, 15], [ 6, 16], [ 7, 17]]) In [33]: genfromtxt(f, dtype=None, max_rows=4) Out[33]: array([[ 8, 18], [ 9, 19]]) (2) Multiple arrays in a single file: I've seen a file format of the form 3 5 1.0 1.5 2.1 2.5 4.8 3.5 1.0 8.7 6.0 2.0 4.2 0.7 4.4 5.3 2.0 2 3 89.1 66.3 42.1 12.3 19.0 56.6 The file contains multiple arrays. Each array is preceded by a line containing the number of rows and columns in that array. The `max_rows` argument would make it easy to read this file with genfromtxt: In [7]: f = open("b.dat", "r") In [8]: nrows, ncols = genfromtxt(f, dtype=None, max_rows=1) In [9]: A = genfromtxt(f, max_rows=nrows) In [10]: nrows, ncols = genfromtxt(f, dtype=None, max_rows=1) In [11]: B = genfromtxt(f, max_rows=nrows) In [12]: A Out[12]: array([[ 1. , 1.5, 2.1, 2.5, 4.8], [ 3.5, 1. , 8.7, 6. , 2. ], [ 4.2, 0.7, 4.4, 5.3, 2. ]]) In [13]: B Out[13]: array([[ 89.1, 66.3, 42.1], [ 12.3, 19. , 56.6]]) Warren > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From alan.isaac at gmail.com Sat Nov 1 10:54:40 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 01 Nov 2014 10:54:40 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> Message-ID: <5454F430.9090206@gmail.com> On 11/1/2014 10:31 AM, Warren Weckesser wrote: > Alan's suggestion to use a slice is interesting, but I'd like to > see a more concrete proposal for the API. For example, how does > it interact with `skip_header` and `skip_footer`? How would one > use it to read a file in batches? I'm probably just not understanding the question, but the initial answer I will give is, "just like the proposal for `max_rows`". That is, skip_header and skip_footer are honored, and the remainder of the file is sliced. For the equivalent of say `max_rows=500`, one would say `slice_rows=slice(500)`. Perhaps you could provide an example illustrating the issues this reply overlooks. Cheers, Alan From bas.swinckels at gmail.com Sat Nov 1 14:28:40 2014 From: bas.swinckels at gmail.com (Bas Swinckels) Date: Sat, 1 Nov 2014 18:28:40 +0000 (UTC) Subject: [Numpy-discussion] np.float64 * Python list vs np.float64 + Python list Message-ID: Hi group, I just answered a question on Stackoverflow from some new user, who was bitten by some unexpected behavior when using a list instead of a np.array, see here: http://stackoverflow.com/q/26690480/2647279 The unexpected thing that triggered the error is that multiplying a list with a Numpy scalar behaves like standard integer multiplication of a list as in Python without raising an error, while addition in the same way works as expected: In [1]: import numpy as np In [2]: np.version.version Out[2]: '1.8.2' In [3]: 2 * [1, 2, 3] Out[3]: [1, 2, 3, 1, 2, 3] In [4]: 2.5 * [1, 2, 3] [...] TypeError: can't multiply sequence by non-int of type 'float' In [5]: np.float64(2.5) * [1, 2, 3] # like standard Python list tiling Out[5]: [1, 2, 3, 1, 2, 3] In [6]: np.float64(2.5) + [1, 2, 3] # element-wise addition Out[6]: array([ 3.5, 4.5, 5.5]) I would have expected that the multiplication would have either raised an exception, or would have given the element-wise result np.array([2.5, 5.0, 7.5]). Is there any good reason why multiplication and addition behave so different? I am using standard, up-to-date python2.7/numpy under 64-bit Ubuntu 14.04. Cheers, Bas From warren.weckesser at gmail.com Sat Nov 1 15:15:26 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 1 Nov 2014 15:15:26 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: <5454F430.9090206@gmail.com> References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sat, Nov 1, 2014 at 10:54 AM, Alan G Isaac wrote: > On 11/1/2014 10:31 AM, Warren Weckesser wrote: > > Alan's suggestion to use a slice is interesting, but I'd like to > > see a more concrete proposal for the API. For example, how does > > it interact with `skip_header` and `skip_footer`? How would one > > use it to read a file in batches? > > > I'm probably just not understanding the question, but the initial > answer I will give is, "just like the proposal for `max_rows`". > > That is, skip_header and skip_footer are honored, and the remainder > of the file is sliced. For the equivalent of say `max_rows=500`, > one would say `slice_rows=slice(500)`. > > Perhaps you could provide an example illustrating the issues this > reply overlooks. > > Cheers, > Alan > OK, so `slice_rows=slice(n)` should behave the same as `max_rows=n`. Here's my take on how `slice_rows` could be handled. I intended the result of `genfromtxt(..., max_rows=n)` to produce the same array as produced by `genfromtxt(...)[:n]`. So a reasonable way to define the behavior of `slice_rows` is that `gengromtxt(..., slice_rows=arg)` returns the same array as `genfromtxt(...)[arg]`. With that specification, it is natural for `slice_rows` to accept any object that is valid for indexing, e.g. `slice_rows=[0,2,3]` or `slice_rows=10`. (But that wouldn't necessarily have to be implemented.) The two differences between `genfromtxt(..., slice_rows=arg)` and `genfromtxt(...)[arg]` are (1) the former is more efficient--it can simply ignore the rows that won't be part of the final result; and (2) the former doesn't consume the input iterator beyond what is requested by `arg`. For example, `slice_rows=(2,10,2)` would consume 10 items from the input (or fewer, if there aren't 10 items in the input). Note that the actual indices for that slice are [2, 4, 6, 8]; even though index 9 is not included in the result, the corresponding item is consumed from the input iterator. (That's how I would interpret it, anyway.) Because the input argument to `genfromtxt` can be an arbitrary iterator, the use of `slice_rows=slice(n)` is not compatible with the use of `skip_footer=m`. Handling `skip_footer=m` requires looking ahead in the iterator to see if the end of the input is within `m` items, but in general, looking ahead is not possible without consuming the items. (The `max_rows` argument has the same problem. In the current PR, a ValueError is raised if both `skip_footer` and `max_rows` are given.) Related to this is how to handle `slice_rows=slice(-3)`. Either this is not allowed (for the same reason that `slice_rows=slice(n), skip_footer=m` is disallowed), or it results in the entire iterator being consumed (and it is explained in the docstring that this is the effect of a negative `stop` value in a slice). Is there wider interest in such an argument to `genfromtxt`? For my use-cases, `max_rows` is sufficient. I can't recall ever needing the full generality of a slice for pulling apart a text file. Does anyone have compelling use-cases that are not handled by `max_rows`? Warren > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Sat Nov 1 16:41:21 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sat, 1 Nov 2014 16:41:21 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sat, Nov 1, 2014 at 3:15 PM, Warren Weckesser wrote: > Is there wider interest in such an argument to `genfromtxt`? For my > use-cases, `max_rows` is sufficient. I can't recall ever needing the full > generality of a slice for pulling apart a text file. Does anyone have > compelling use-cases that are not handled by `max_rows`? > It is occasionally useful to be able to skip rows after the header. Maybe we should de-deprecate skip_rows and give it the meaning different from skip_header in case of names = None? For example, genfromtxt(fname, skip_header= 3, skip_rows = 1, max_rows = 100) would mean skip 3 lines, read column names from the 4-th, skip 5-th, process up to 100 more lines. This may be useful if the file contains some meta-data about the column below the header line. For example, it is common to put units of measurement below the column names. Another application could be processing a large text file in chunks, which again can be covered nicely by skip_rows/max_rows. I cannot think of a situation where I would need more generality such as reading every 3rd row or rows with the given numbers. Such processing is normally done after the text data is loaded into an array. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Sat Nov 1 16:56:19 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 01 Nov 2014 16:56:19 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: <545548F3.2060209@gmail.com> On 11/1/2014 4:41 PM, Alexander Belopolsky wrote: > I cannot think of a situation where I would need more generality such as reading every 3rd row or rows with the given numbers. Such processing is > normally done after the text data is loaded into an array. I have done this as cheaper than random selection for a quick and dirty look at large data sets. Setting maxrows can be very different if the data has been stored in some structured manner. I suppose my view is something like this. We are considering adding a keyword. If we can get greater functionality at about the same cost, why not? In that case, it is not really useful to speculate about use cases. If the costs are substantially greater, then that should be stated. Cost is a good reason not to do something. fwiw, Alan Isaac From alan.isaac at gmail.com Sat Nov 1 17:04:36 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 01 Nov 2014 17:04:36 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: <54554AE4.9010602@gmail.com> On 11/1/2014 3:15 PM, Warren Weckesser wrote: > I intended the result of `genfromtxt(..., max_rows=n)` to produce the same array as produced by `genfromtxt(...)[:n]`. I find that counterintuitive. I would first honor skip_header. Cheers, Alan From warren.weckesser at gmail.com Sat Nov 1 20:58:42 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 1 Nov 2014 20:58:42 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: <545548F3.2060209@gmail.com> References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> <545548F3.2060209@gmail.com> Message-ID: On 11/1/14, Alan G Isaac wrote: > On 11/1/2014 4:41 PM, Alexander Belopolsky wrote: >> I cannot think of a situation where I would need more generality such as >> reading every 3rd row or rows with the given numbers. Such processing is >> normally done after the text data is loaded into an array. > > > I have done this as cheaper than random selection for a quick and dirty > look at large data sets. Setting maxrows can be very different if the > data has been stored in some structured manner. > > I suppose my view is something like this. We are considering adding a > keyword. > If we can get greater functionality at about the same cost, why not? > In that case, it is not really useful to speculate about use cases. > If the costs are substantially greater, then that should be stated. > Cost is a good reason not to do something. > `slice_rows` is a generalization of `max_rows`. It will probably take a bit more code to implement, and it will require more tests and more documentation. So the cost isn't really the same. But if it solves real problems for users, the cost may be worth it. Warren > fwiw, > Alan Isaac > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From warren.weckesser at gmail.com Sat Nov 1 21:02:04 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 1 Nov 2014 21:02:04 -0400 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: <54554AE4.9010602@gmail.com> References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> <54554AE4.9010602@gmail.com> Message-ID: On 11/1/14, Alan G Isaac wrote: > On 11/1/2014 3:15 PM, Warren Weckesser wrote: >> I intended the result of `genfromtxt(..., max_rows=n)` to produce the same >> array as produced by `genfromtxt(...)[:n]`. > > I find that counterintuitive. > I would first honor skip_header. Sorry for the terse explanation. I meant for `...` to indicate any other arguments, including skip_header. Warren > Cheers, > Alan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jtaylor.debian at googlemail.com Sun Nov 2 08:33:50 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 02 Nov 2014 14:33:50 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.9.1 bugfix release Message-ID: <545632BE.5080100@googlemail.com> Hello, We am pleased to announce the release of NumPy 1.9.1, a bugfix only release for the 1.9.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ The upgrade is recommended for all users of the 1.9.x series. Following issues have been fixed: * gh-5184: restore linear edge behaviour of gradient to as it was in < 1.9. The second order behaviour is available via the `edge_order` keyword * gh-4007: workaround Accelerate sgemv crash on OSX 10.9 * gh-5100: restore object dtype inference from iterable objects without `len()` * gh-5163: avoid gcc-4.1.2 (red hat 5) miscompilation causing a crash * gh-5138: fix nanmedian on arrays containing inf * gh-5240: fix not returning out array from ufuncs with subok=False set * gh-5203: copy inherited masks in MaskedArray.__array_finalize__ * gh-2317: genfromtxt did not handle filling_values=0 correctly * gh-5067: restore api of npy_PyFile_DupClose in python2 * gh-5063: cannot convert invalid sequence index to tuple * gh-5082: Segmentation fault with argmin() on unicode arrays * gh-5095: don't propagate subtypes from np.where * gh-5104: np.inner segfaults with SciPy's sparse matrices * gh-5251: Issue with fromarrays not using correct format for unicode arrays * gh-5136: Import dummy_threading if importing threading fails * gh-5148: Make numpy import when run with Python flag '-OO' * gh-5147: Einsum double contraction in particular order causes ValueError * gh-479: Make f2py work with intent(in out) * gh-5170: Make python2 .npy files readable in python3 * gh-5027: Use 'll' as the default length specifier for long long * gh-4896: fix build error with MSVC 2013 caused by C99 complex support * gh-4465: Make PyArray_PutTo respect writeable flag * gh-5225: fix crash when using arange on datetime without dtype set * gh-5231: fix build in c99 mode The source distributions have been uploaded to PyPI. The Windows installers, documentation and release notes can be found at: https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ Cheers, Julian Taylor -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From maniteja.modesty067 at gmail.com Sun Nov 2 13:21:01 2014 From: maniteja.modesty067 at gmail.com (Maniteja Nandana) Date: Sun, 2 Nov 2014 23:51:01 +0530 Subject: [Numpy-discussion] Guidance to start contribution to Numpy Message-ID: Hi everyone, I'm a third year undergraduate student in Computer Science. I have worked in Python libraries for past two years and am interested in getting some open source experience in Numpy and it would be great if anyone could suggest some bugs or issues that are suitable for a beginner to try working on ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Nov 2 13:56:31 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 2 Nov 2014 13:56:31 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sat, Nov 1, 2014 at 4:41 PM, Alexander Belopolsky wrote: > > On Sat, Nov 1, 2014 at 3:15 PM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> Is there wider interest in such an argument to `genfromtxt`? For my >> use-cases, `max_rows` is sufficient. I can't recall ever needing the full >> generality of a slice for pulling apart a text file. Does anyone have >> compelling use-cases that are not handled by `max_rows`? >> > > It is occasionally useful to be able to skip rows after the header. Maybe > we should de-deprecate skip_rows and give it the meaning different from > skip_header in case of names = None? For example, > > genfromtxt(fname, skip_header= 3, skip_rows = 1, max_rows = 100) > > would mean skip 3 lines, read column names from the 4-th, skip 5-th, > process up to 100 more lines. This may be useful if the file contains some > meta-data about the column below the header line. For example, it is > common to put units of measurement below the column names. > Or you could just call genfromtxt() once with `max_rows=1` to skip a row. (I'm assuming that the first argument to genfromtxt is the open file object--or some other iterator--and not the filename.) > > Another application could be processing a large text file in chunks, which > again can be covered nicely by skip_rows/max_rows. > You don't really need `skip_rows` for this. In a previous email (and in https://github.com/numpy/numpy/pull/5103) I gave an example of using `max_rows` for handling a file that doesn't have a header. If the file has a header, you could process the file in batches using something like the following example, where the dtype determined in the first batch is used when reading the subsequent batches: In [12]: !cat foo.dat a b c 1.0 2.0 -9.0 3.0 4.0 -7.6 5.0 6.0 -1.0 7.0 8.0 -3.3 9.0 0.0 -3.4 In [13]: f = open("foo.dat", "r") In [14]: batch1 = genfromtxt(f, dtype=None, names=True, max_rows=2) In [15]: batch1 Out[15]: array([(1.0, 2.0, -9.0), (3.0, 4.0, -7.6)], dtype=[('a', ' I cannot think of a situation where I would need more generality such as > reading every 3rd row or rows with the given numbers. Such processing is > normally done after the text data is loaded into an array. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Sun Nov 2 14:18:38 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sun, 2 Nov 2014 14:18:38 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sun, Nov 2, 2014 at 1:56 PM, Warren Weckesser wrote: > Or you could just call genfromtxt() once with `max_rows=1` to skip a row. > (I'm assuming that the first argument to genfromtxt is the open file > object--or some other iterator--and not the filename.) That's hackish. If I have to resort to something like this, I would just call next() on the open file object or iterator. Still, the case of dtype=None, name=None is problematic. Suppose I want genfromtxt() to detect the column names from the 1-st row and data types from the 3-rd. How would you do that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Sun Nov 2 14:20:57 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sun, 2 Nov 2014 14:20:57 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: Sorry, I meant names=True, not name=None. On Sun, Nov 2, 2014 at 2:18 PM, Alexander Belopolsky wrote: > > On Sun, Nov 2, 2014 at 1:56 PM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> Or you could just call genfromtxt() once with `max_rows=1` to skip a >> row. (I'm assuming that the first argument to genfromtxt is the open file >> object--or some other iterator--and not the filename.) > > > That's hackish. If I have to resort to something like this, I would just > call next() on the open file object or iterator. > > Still, the case of dtype=None, name=None is problematic. Suppose I want > genfromtxt() to detect the column names from the 1-st row and data types > from the 3-rd. How would you do that? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Nov 2 14:32:55 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 2 Nov 2014 14:32:55 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sun, Nov 2, 2014 at 2:18 PM, Alexander Belopolsky wrote: > > On Sun, Nov 2, 2014 at 1:56 PM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> Or you could just call genfromtxt() once with `max_rows=1` to skip a >> row. (I'm assuming that the first argument to genfromtxt is the open file >> object--or some other iterator--and not the filename.) > > > That's hackish. If I have to resort to something like this, I would just > call next() on the open file object or iterator. > I agree, calling genfromtxt to skip a line is silly. Calling next() makes much more sense. > > Still, the case of dtype=None, name=None is problematic. Suppose I want > genfromtxt() to detect the column names from the 1-st row and data types > from the 3-rd. How would you do that? > > This may sound like a cop out, but at some point, I stop trying to make genfromtxt() handle every possible case, and instead I would write a custom header reader to handle this. Warren > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Sun Nov 2 16:18:51 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Sun, 2 Nov 2014 16:18:51 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On Sun, Nov 2, 2014 at 2:32 PM, Warren Weckesser wrote: > >> Still, the case of dtype=None, name=None is problematic. Suppose I want >> genfromtxt() to detect the column names from the 1-st row and data types >> from the 3-rd. How would you do that? >> >> > > This may sound like a cop out, but at some point, I stop trying to make > genfromtxt() handle every possible case, and instead I would write a custom > header reader to handle this. > In the abstract, I would agree with you. It is often the case that 2-3 lines of clear Python code is better than a terse function call with half a dozen non-obvious options. Specifically, I would be against the proposed slice_rows because it is either equivalent to genfromtxt(islice(..), ..) or hard to specify. On the other hand, skip_rows is different for two reasons: 1. It is not a new option. It is currently a deprecated alias to skip_header, so a change is expected - either removal or redefinition. 2. The intended use-case - inferring column names and type information from a file where data is separated from the column names is hard to code explicitly. (Try it!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Nov 2 17:24:38 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 2 Nov 2014 17:24:38 -0500 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> <5454F430.9090206@gmail.com> Message-ID: On 11/2/14, Alexander Belopolsky wrote: > On Sun, Nov 2, 2014 at 2:32 PM, Warren Weckesser > > wrote: > >> >>> Still, the case of dtype=None, name=None is problematic. Suppose I >>> want >>> genfromtxt() to detect the column names from the 1-st row and data >>> types >>> from the 3-rd. How would you do that? >>> >>> >> >> This may sound like a cop out, but at some point, I stop trying to make >> genfromtxt() handle every possible case, and instead I would write a >> custom >> header reader to handle this. >> > > In the abstract, I would agree with you. It is often the case that 2-3 > lines of clear Python code is better than a terse function call with half a > dozen non-obvious options. Specifically, I would be against the proposed > slice_rows because it is either equivalent to genfromtxt(islice(..), ..) > or hard to specify. I don't have much more to add to the API discussion at the moment, but I want to make sure one aspect is clear. (Sorry for the noise if the following is obvious.) In an earlier email, I gave my interpretation of the semantics of `slice_rows` (and `max_rows`), which is that `genfromtxt(f, ..., slice_rows=arg)` produces the same result as `genfromtxt(f, ...)[arg]`. (The difference is that it only consumes items from the input iterator f as required by `arg`). This isn't the same as `genfromtxt(islice(f, ), ...)`, because `genfromtxt` skips comments and blank lines. (It also skips invalid lines if the argument `invalid_raise=False` is used.) So if the input file was ----- 1 10 # A comment. 2 20 3 30 4 40 5 50 ----- Then `genfromtxt(f, dtype=int, slice_rows=slice(4))` would produce `array([[1, 10], [2, 20], [3, 30], [4, 40]])`, while `genfromtxt(islice(f, 4), dtype=int)` would produce `array([1, 10], [2, 20]])`. That's my interpretation of how `max_rows` or `slice_rows` should work. If that is not what other folks expect, than that should also be part of the discussion. Warren > > On the other hand, skip_rows is different for two reasons: > > 1. It is not a new option. It is currently a deprecated alias to > skip_header, so a change is expected - either removal or redefinition. > 2. The intended use-case - inferring column names and type information from > a file where data is separated from the column names is hard to code > explicitly. (Try it!) > From charlesr.harris at gmail.com Mon Nov 3 11:13:15 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 3 Nov 2014 09:13:15 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.9.1 bugfix release In-Reply-To: <545632BE.5080100@googlemail.com> References: <545632BE.5080100@googlemail.com> Message-ID: Hi Julian, On Sun, Nov 2, 2014 at 6:33 AM, Julian Taylor wrote: > Hello, > > We am pleased to announce the release of NumPy 1.9.1, a > bugfix only release for the 1.9.x series. > https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ > The upgrade is recommended for all users of the 1.9.x series. > > Following issues have been fixed: > * gh-5184: restore linear edge behaviour of gradient to as it was in < 1.9. > The second order behaviour is available via the `edge_order` keyword > * gh-4007: workaround Accelerate sgemv crash on OSX 10.9 > * gh-5100: restore object dtype inference from iterable objects without > `len()` > * gh-5163: avoid gcc-4.1.2 (red hat 5) miscompilation causing a crash > * gh-5138: fix nanmedian on arrays containing inf > * gh-5240: fix not returning out array from ufuncs with subok=False set > * gh-5203: copy inherited masks in MaskedArray.__array_finalize__ > * gh-2317: genfromtxt did not handle filling_values=0 correctly > * gh-5067: restore api of npy_PyFile_DupClose in python2 > * gh-5063: cannot convert invalid sequence index to tuple > * gh-5082: Segmentation fault with argmin() on unicode arrays > * gh-5095: don't propagate subtypes from np.where > * gh-5104: np.inner segfaults with SciPy's sparse matrices > * gh-5251: Issue with fromarrays not using correct format for unicode > arrays > * gh-5136: Import dummy_threading if importing threading fails > * gh-5148: Make numpy import when run with Python flag '-OO' > * gh-5147: Einsum double contraction in particular order causes ValueError > * gh-479: Make f2py work with intent(in out) > * gh-5170: Make python2 .npy files readable in python3 > * gh-5027: Use 'll' as the default length specifier for long long > * gh-4896: fix build error with MSVC 2013 caused by C99 complex support > * gh-4465: Make PyArray_PutTo respect writeable flag > * gh-5225: fix crash when using arange on datetime without dtype set > * gh-5231: fix build in c99 mode > > > The source distributions have been uploaded to PyPI. The Windows > installers, documentation and release notes can be found at: > https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ > > Cheers, > Julian Taylor > > > Thanks for getting this out. Hopefully this will keep the users happy for a while ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Nov 3 11:37:50 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 3 Nov 2014 09:37:50 -0700 Subject: [Numpy-discussion] Guidance to start contribution to Numpy In-Reply-To: References: Message-ID: Hi Maniteja, On Sun, Nov 2, 2014 at 11:21 AM, Maniteja Nandana < maniteja.modesty067 at gmail.com> wrote: > Hi everyone, I'm a third year undergraduate student in Computer Science. I > have worked in Python libraries for past two years and am interested in > getting some open source experience in Numpy and it would be great if > anyone could suggest some bugs or issues that are suitable for a beginner > to try working on ? > The best way to get started is to read the hacking document , set up a github account, and then look through the issues and pull requests for the numpy repository on github. Documentation fixes, code review, and bug fixes are all good ways to get involved in Numpy. If you are unfamiliar with git it will be helpful to read up on it and if you write code you should also read the howtos and style guides . Also note the bottom posting convention on the list ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 3 13:07:03 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 3 Nov 2014 10:07:03 -0800 Subject: [Numpy-discussion] Add `nrows` to `genfromtxt` In-Reply-To: References: <5423196F.4000507@gmail.com> Message-ID: On Sat, Nov 1, 2014 at 7:31 AM, Warren Weckesser wrote: > (2) Multiple arrays in a single file: > > ... > The file contains multiple arrays. Each array is > preceded by a line containing the number of rows > and columns in that array. The `max_rows` argument > would make it easy to read this file with genfromtxt: > +inf on this one -- this is a use case I've been looking for support for ages! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Nov 3 13:10:23 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 3 Nov 2014 10:10:23 -0800 Subject: [Numpy-discussion] ANN: NumPy 1.9.1 bugfix release In-Reply-To: References: <545632BE.5080100@googlemail.com> Message-ID: Hi, On Mon, Nov 3, 2014 at 8:13 AM, Charles R Harris wrote: > Hi Julian, > > On Sun, Nov 2, 2014 at 6:33 AM, Julian Taylor > wrote: >> >> Hello, >> >> We am pleased to announce the release of NumPy 1.9.1, a >> bugfix only release for the 1.9.x series. >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ >> The upgrade is recommended for all users of the 1.9.x series. >> >> Following issues have been fixed: >> * gh-5184: restore linear edge behaviour of gradient to as it was in < >> 1.9. >> The second order behaviour is available via the `edge_order` keyword >> * gh-4007: workaround Accelerate sgemv crash on OSX 10.9 >> * gh-5100: restore object dtype inference from iterable objects without >> `len()` >> * gh-5163: avoid gcc-4.1.2 (red hat 5) miscompilation causing a crash >> * gh-5138: fix nanmedian on arrays containing inf >> * gh-5240: fix not returning out array from ufuncs with subok=False set >> * gh-5203: copy inherited masks in MaskedArray.__array_finalize__ >> * gh-2317: genfromtxt did not handle filling_values=0 correctly >> * gh-5067: restore api of npy_PyFile_DupClose in python2 >> * gh-5063: cannot convert invalid sequence index to tuple >> * gh-5082: Segmentation fault with argmin() on unicode arrays >> * gh-5095: don't propagate subtypes from np.where >> * gh-5104: np.inner segfaults with SciPy's sparse matrices >> * gh-5251: Issue with fromarrays not using correct format for unicode >> arrays >> * gh-5136: Import dummy_threading if importing threading fails >> * gh-5148: Make numpy import when run with Python flag '-OO' >> * gh-5147: Einsum double contraction in particular order causes ValueError >> * gh-479: Make f2py work with intent(in out) >> * gh-5170: Make python2 .npy files readable in python3 >> * gh-5027: Use 'll' as the default length specifier for long long >> * gh-4896: fix build error with MSVC 2013 caused by C99 complex support >> * gh-4465: Make PyArray_PutTo respect writeable flag >> * gh-5225: fix crash when using arange on datetime without dtype set >> * gh-5231: fix build in c99 mode >> >> >> The source distributions have been uploaded to PyPI. The Windows >> installers, documentation and release notes can be found at: >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/ >> >> Cheers, >> Julian Taylor >> >> > > Thanks for getting this out. Hopefully this will keep the users happy for a > while ;) Thanks from me too. I added OSX wheels to pypi... Cheers, Matthew From maniteja.modesty067 at gmail.com Mon Nov 3 16:54:32 2014 From: maniteja.modesty067 at gmail.com (Maniteja Nandana) Date: Tue, 4 Nov 2014 03:24:32 +0530 Subject: [Numpy-discussion] Guidance to start contribution to Numpy In-Reply-To: References: Message-ID: Hello Charles, Thanks for the help. I am a computer science major with little maths background, with interest in natural language processing and machine learning, both of which involve a lot of calculus, statistics and linear algebra. I have used numpy quite a lot for clustering, fft and polynomials like legendre. I have gone through the hacking and howtocode documentation. My github account is https://github.com/maniteja123. I have been using git for my local repositories. It would be great if you could suggest where to have a look at for the ideas, maybe like the roadmap, since there are many issues and pull requests on repository and I have not yet explored the breadths and depths of Numpy and scipy, but am totally eager to do so. Regards, Maniteja _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http:// mail.scipy.org /mailman/ listinfo / numpy -discussion Hi Maniteja, On Sun, Nov 2, 2014 at 11:21 AM, Maniteja Nandana < maniteja.modesty067 at gmail.com> wrote: > Hi everyone, I'm a third year undergraduate student in Computer Science. I > have worked in Python libraries for past two years and am interested in > getting some open source experience in Numpy and it would be great if > anyone could suggest some bugs or issues that are suitable for a beginner > to try working on ? > The best way to get started is to read the hacking document , set up a github account, and then look through the issues and pull requests for the numpy repository on github. Documentation fixes, code review, and bug fixes are all good ways to get involved in Numpy. If you are unfamiliar with git it will be helpful to read up on it and if you write code you should also read the howtos and style guides . Also note the bottom posting convention on the list ;) Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Tue Nov 4 08:50:48 2014 From: sebix at sebix.at (Sebastian Wagner) Date: Tue, 04 Nov 2014 14:50:48 +0100 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers Message-ID: <5415270265975505d5db65eab7550627@fizeau.net> Hello, I want to bring up Issue #2522 'numpy.diff fails on unsigned integers (Trac #1929)' [1], as it was resonsible for an error in one of our programs. Short explanation of the bug: np.diff performs a subtraction on the input array. If this is of type uint and the data contains falling data, it results in an artihmetic underflow. >>> np.diff(np.array([0,1,0], dtype=np.uint8)) array([ 1, 255], dtype=uint8) @charris proposed either - a note to the doc string and maybe an example to clarify things - or raise a warning but with a discussion on the list. I would like to start it now, as it is an error which is not easily detectable (no errors or warnings are thrown). In our case the type of a data sequence, with only zeros and ones, had type f8 as also every other one, has been changed to u4. As the programs looked for values ==1 and ==-1, it broke silently. In my opinion, a note in the docs is not enough and does not help if the type changed or set after the program has been written. I'd go for automatic upcasting of uints by default and an option to turn it off, if this behavior is explicitly wanted. This wouldn't be correct from the point of view of a programmer, but as most of the users have a scientific background who excpect it 'to work', instead of sth is theoretically correct but not convenient. (I count myself to the first group) When a consens has been achieved I could implement it for a PR. [1]: https://github.com/numpy/numpy/issues/2522 -- gpg --keyserver keys.gnupg.net --recv-key DC9B463B From toddrjen at gmail.com Tue Nov 4 09:06:14 2014 From: toddrjen at gmail.com (Todd) Date: Tue, 4 Nov 2014 15:06:14 +0100 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: <5415270265975505d5db65eab7550627@fizeau.net> References: <5415270265975505d5db65eab7550627@fizeau.net> Message-ID: On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner wrote: > Hello, > > I want to bring up Issue #2522 'numpy.diff fails on unsigned integers > (Trac #1929)' [1], as it was resonsible for an error in one of our > programs. Short explanation of the bug: np.diff performs a subtraction > on the input array. If this is of type uint and the data contains > falling data, it results in an artihmetic underflow. > > >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > array([ 1, 255], dtype=uint8) > > @charris proposed either > - a note to the doc string and maybe an example to clarify things > - or raise a warning > but with a discussion on the list. > > I would like to start it now, as it is an error which is not easily > detectable (no errors or warnings are thrown). In our case the type of a > data sequence, with only zeros and ones, had type f8 as also every other > one, has been changed to u4. As the programs looked for values ==1 and > ==-1, it broke silently. > In my opinion, a note in the docs is not enough and does not help if the > type changed or set after the program has been written. > I'd go for automatic upcasting of uints by default and an option to turn > it off, if this behavior is explicitly wanted. This wouldn't be correct > from the point of view of a programmer, but as most of the users have a > scientific background who excpect it 'to work', instead of sth is > theoretically correct but not convenient. (I count myself to the first > group) > When you say "automatic upcasting", that would be, for example uint8 to int16? What about for uint64? There is no int128. Also, when you say "by default", is this only when an overflow is detected, or always? How would the option to turn it off be implemented? An argument to np.diff or some sort of global option? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Tue Nov 4 13:19:24 2014 From: sebix at sebix.at (Sebastian) Date: Tue, 04 Nov 2014 19:19:24 +0100 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: References: <5415270265975505d5db65eab7550627@fizeau.net> Message-ID: <545918AC.2020707@sebix.at> On 2014-11-04 15:06, Todd wrote: > On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner > wrote: > > Hello, > > I want to bring up Issue #2522 'numpy.diff fails on unsigned integers > (Trac #1929)' [1], as it was resonsible for an error in one of our > programs. Short explanation of the bug: np.diff performs a subtraction > on the input array. If this is of type uint and the data contains > falling data, it results in an artihmetic underflow. > > >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > array([ 1, 255], dtype=uint8) > > @charris proposed either > - a note to the doc string and maybe an example to clarify things > - or raise a warning > but with a discussion on the list. > > I would like to start it now, as it is an error which is not easily > detectable (no errors or warnings are thrown). In our case the > type of a > data sequence, with only zeros and ones, had type f8 as also every > other > one, has been changed to u4. As the programs looked for values ==1 and > ==-1, it broke silently. > In my opinion, a note in the docs is not enough and does not help > if the > type changed or set after the program has been written. > I'd go for automatic upcasting of uints by default and an option > to turn > it off, if this behavior is explicitly wanted. This wouldn't be > correct > from the point of view of a programmer, but as most of the users > have a > scientific background who excpect it 'to work', instead of sth is > theoretically correct but not convenient. (I count myself to the first > group) > > > > When you say "automatic upcasting", that would be, for example uint8 > to int16? What about for uint64? There is no int128. The upcast should go to the next bigger, otherwise it would again result in wrong values. uint64 we can't do that, so it has to stay. > Also, when you say "by default", is this only when an overflow is > detected, or always? I don't know how I could detect an overflow in the diff-function. In subtraction it should be possible, but that's very deep in the numpy-internals. > How would the option to turn it off be implemented? An argument to > np.diff or some sort of global option? I thought of a parameter upcast_int=True for the function. > -- > gpg --keyserver keys.gnupg.net --recv-key DC9B463B From charlesr.harris at gmail.com Tue Nov 4 13:44:36 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 4 Nov 2014 11:44:36 -0700 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: <545918AC.2020707@sebix.at> References: <5415270265975505d5db65eab7550627@fizeau.net> <545918AC.2020707@sebix.at> Message-ID: On Tue, Nov 4, 2014 at 11:19 AM, Sebastian wrote: > On 2014-11-04 15:06, Todd wrote: > > On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner > > wrote: > > > > Hello, > > > > I want to bring up Issue #2522 'numpy.diff fails on unsigned integers > > (Trac #1929)' [1], as it was resonsible for an error in one of our > > programs. Short explanation of the bug: np.diff performs a > subtraction > > on the input array. If this is of type uint and the data contains > > falling data, it results in an artihmetic underflow. > > > > >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > > array([ 1, 255], dtype=uint8) > > > > @charris proposed either > > - a note to the doc string and maybe an example to clarify things > > - or raise a warning > > but with a discussion on the list. > > > > I would like to start it now, as it is an error which is not easily > > detectable (no errors or warnings are thrown). In our case the > > type of a > > data sequence, with only zeros and ones, had type f8 as also every > > other > > one, has been changed to u4. As the programs looked for values ==1 > and > > ==-1, it broke silently. > > In my opinion, a note in the docs is not enough and does not help > > if the > > type changed or set after the program has been written. > > I'd go for automatic upcasting of uints by default and an option > > to turn > > it off, if this behavior is explicitly wanted. This wouldn't be > > correct > > from the point of view of a programmer, but as most of the users > > have a > > scientific background who excpect it 'to work', instead of sth is > > theoretically correct but not convenient. (I count myself to the > first > > group) > > > > > > > > When you say "automatic upcasting", that would be, for example uint8 > > to int16? What about for uint64? There is no int128. > The upcast should go to the next bigger, otherwise it would again result > in wrong values. uint64 we can't do that, so it has to stay. > > Also, when you say "by default", is this only when an overflow is > > detected, or always? > I don't know how I could detect an overflow in the diff-function. In > subtraction it should be possible, but that's very deep in the > numpy-internals. > > How would the option to turn it off be implemented? An argument to > > np.diff or some sort of global option? > I thought of a parameter upcast_int=True for the function. > Could check for non-decreasing sequence in the unsigned case. Note that differences of signed integers can also overflow. One way to check in general is to determine the expected sign using comparisons. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Catherine.M.Moroney at jpl.nasa.gov Wed Nov 5 13:11:15 2014 From: Catherine.M.Moroney at jpl.nasa.gov (Moroney, Catherine M (398E)) Date: Wed, 5 Nov 2014 18:11:15 +0000 Subject: [Numpy-discussion] median filtering a masked array Message-ID: <71F9591C-2AEC-4324-89AC-C42C3712CC95@jpl.nasa.gov> Hello, I have to perform a median filter on only selected elements in an array and am wondering if using a masked array will speed things up by preventing the computation of the median on those elements where it's not needed. I'm using a Fortran code to fill up a (NX,NY,NZ) array where (NZ) is the number of points in the median filter window and NX and NY are the array dimensions themselves. If a filter is not required at a certain (I,J) element, I set all the elements along the NZ dimension to a fill value, and then create a masked array where the "mask" argument is set to True for all fill values. So, I am hoping that the numpy.median(data, axis=2) call will not even try to compute the median for those (I,J) elements where all the data in the NZ direction are masked out, thus saving me valuable computational time. Is this a valid assumption? Or does numpy go ahead and calculate the median for all (I,J) elements irregardless of whether they're masked out or not? What is the recommended way of doing a fast median filter on an array where only certain elements of the array need to be calculated? I'm trying to avoid a nested loop over all (I,J) elements. Catherine From davidmenhur at gmail.com Thu Nov 6 04:10:29 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 6 Nov 2014 10:10:29 +0100 Subject: [Numpy-discussion] median filtering a masked array In-Reply-To: <71F9591C-2AEC-4324-89AC-C42C3712CC95@jpl.nasa.gov> References: <71F9591C-2AEC-4324-89AC-C42C3712CC95@jpl.nasa.gov> Message-ID: On 5 November 2014 19:11, Moroney, Catherine M (398E) wrote: > What is the recommended way of doing a fast median filter on an array where only > certain elements of the array need to be calculated? I'm trying to avoid a > nested loop over all (I,J) elements. Since you are using FORTRAN, I believe the simplest way is to do this double loop in Cython. While you are at it, if you want to squeeze cycles, you can implement the median in Cython too. Here is some code: http://numpy-discussion.10968.n7.nabble.com/A-faster-median-Wirth-s-method-td14267.html /David. From stefan at sun.ac.za Thu Nov 6 04:48:51 2014 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 06 Nov 2014 11:48:51 +0200 Subject: [Numpy-discussion] median filtering a masked array In-Reply-To: References: <71F9591C-2AEC-4324-89AC-C42C3712CC95@jpl.nasa.gov> Message-ID: <87vbms64n0.fsf@sun.ac.za> On 2014-11-06 11:10:29, Da?id wrote: > On 5 November 2014 19:11, Moroney, Catherine M (398E) > wrote: >> What is the recommended way of doing a fast median filter on an array where only >> certain elements of the array need to be calculated? I'm trying to avoid a >> nested loop over all (I,J) elements. > > Since you are using FORTRAN, I believe the simplest way is to do this > double loop in Cython. If you'd prefer to stay on the Python side, have a look at scipy.ndimage.generic_filter St?fan From bburan at alum.mit.edu Thu Nov 6 17:51:02 2014 From: bburan at alum.mit.edu (Brad Buran) Date: Thu, 6 Nov 2014 14:51:02 -0800 Subject: [Numpy-discussion] FFT of 2D array along last axis Message-ID: Given the following code: import numpy as np x = np.random.random(size=2**14) y = x.copy() z = np.concatenate([x[np.newaxis], y[np.newaxis]], axis=0) print(np.all(np.fft.fft(z, axis=-1)[0] == np.fft.fft(z[0]))) On Windows 7 using Anaconda with numpy 1.9.1 I get False (indicating that the FFT is not treating each row separately). When I test on a Ubuntu box using numpy 1.9.1 I get True. Is this expected behavior? If I understand the documentation correctly, the FFT on each row should be independent (i.e. the result should not be influenced by the other rows). Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Nov 6 18:28:16 2014 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 07 Nov 2014 01:28:16 +0200 Subject: [Numpy-discussion] FFT of 2D array along last axis In-Reply-To: References: Message-ID: <871tpf6h9r.fsf@sun.ac.za> Hi Brad On 2014-11-07 00:51:02, Brad Buran wrote: > On Windows 7 using Anaconda with numpy 1.9.1 I get False (indicating that > the FFT is not treating each row separately). When I test on a Ubuntu box > using numpy 1.9.1 I get True. Is this expected behavior? If I understand > the documentation correctly, the FFT on each row should be independent > (i.e. the result should not be influenced by the other rows). The results should be the same. As an additional test, can you check: np.testing.assert_array_almost_equal(np.fft.fft(z, axis=-1)[0], np.fft.fft(z[0])) Thanks St?fan From stefan.otte at gmail.com Wed Nov 12 05:30:33 2014 From: stefan.otte at gmail.com (Stefan Otte) Date: Wed, 12 Nov 2014 11:30:33 +0100 Subject: [Numpy-discussion] Subscribing to mailinglist not possible / sites down Message-ID: Hey *, The websites to subscribe to the numpy/scipy mailinglists seem to be down: - http://mail.scipy.org/mailman/listinfo/scipy-user - http://mail.scipy.org/mailman/listinfo/scipy-user - http://projects.scipy.org/pipermail/scipy-dev/ And it's not just me: http://www.downforeveryoneorjustme.com/http://projects.scipy.org/pipermail/scipy-dev/ Best, Stefan From bburan at alum.mit.edu Wed Nov 12 14:24:39 2014 From: bburan at alum.mit.edu (Brad Buran) Date: Wed, 12 Nov 2014 11:24:39 -0800 Subject: [Numpy-discussion] FFT of 2D array along last axis In-Reply-To: <871tpf6h9r.fsf@sun.ac.za> References: <871tpf6h9r.fsf@sun.ac.za> Message-ID: So it seems that the error only occurs in np.fft.rfft (not np.fft.fft). The following code: import numpy as np r = np.random.RandomState(seed=0) z = r.randn(2**14).reshape((2, -1)) print(np.abs(np.fft.rfft(z)[0])[:5]) print(np.abs(np.fft.rfft(z[0]))[:5]) Prints out on a Windows 7 with Anaconda 64-bit: [ 94.22136166 149.3181169 145.77500588 177.39556729 172.05616019] [ 128.55889109 77.35670747 69.63121452 109.06772033 66.22919485] Prints out on a Ubuntu box: [ 128.55889109 77.35670747 69.63121452 109.06772033 66.22919485] [ 128.55889109 77.35670747 69.63121452 109.06772033 66.22919485] On a Windows 7 with Python(x,y): [ 128.55889109 77.35670747 69.63121452 109.06772033 66.22919485] [ 128.55889109 77.35670747 69.63121452 109.06772033 66.22919485] Brad On Thu, Nov 6, 2014 at 3:28 PM, Stefan van der Walt wrote: > Hi Brad > > On 2014-11-07 00:51:02, Brad Buran wrote: > > On Windows 7 using Anaconda with numpy 1.9.1 I get False (indicating that > > the FFT is not treating each row separately). When I test on a Ubuntu > box > > using numpy 1.9.1 I get True. Is this expected behavior? If I > understand > > the documentation correctly, the FFT on each row should be independent > > (i.e. the result should not be influenced by the other rows). > > The results should be the same. As an additional test, can you check: > > np.testing.assert_array_almost_equal(np.fft.fft(z, axis=-1)[0], > np.fft.fft(z[0])) > > Thanks > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Tue Nov 11 15:46:44 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Tue, 11 Nov 2014 12:46:44 -0800 Subject: [Numpy-discussion] truthiness of object arrays Message-ID: I am puzzled by the following (numpy 1.9.0, python 3.4.2): In [1]: t = array(None); t[()] = array([None, None]) # Construct a 0d array of dtype object, containing a single numpy array with 2 elements In [2]: bool(t) Out[2]: True In [3]: if t: pass --------------------------------------------------------------------------- ValueError Traceback (most recent call last) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() I thought that "if x" simply calls "bool", but apparently this is not even the case... Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From msarahan at gmail.com Wed Nov 12 23:15:18 2014 From: msarahan at gmail.com (Michael Sarahan) Date: Wed, 12 Nov 2014 20:15:18 -0800 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: Message-ID: Hi Antony, In general, you can't use numpy arrays in if statements without using any() or all() or some other means of obtaining a single boolean value from the whole array. I think your confusion is that bool() uses truth testing rules outlined here: https://docs.python.org/release/2.5.2/lib/truth.html If statements in theory have equivalent behavior, but take slightly different paths (they don't call bool() directly). This SO post was enlightening to me: http://stackoverflow.com/questions/11885382/is-there-any-difference-between-if-boolx-and-if-x-in-python Without looking at Numpy's code, I'd bet Numpy arrays probably define __bool__ or __nonzero__ in such a way that the ValueError is raised when it makes sense to do so. HTH, Mike On Tue, Nov 11, 2014 at 12:46 PM, Antony Lee wrote: > I am puzzled by the following (numpy 1.9.0, python 3.4.2): > > In [1]: t = array(None); t[()] = array([None, None]) # Construct a 0d > array of dtype object, containing a single numpy array with 2 elements > > In [2]: bool(t) > Out[2]: True > > In [3]: if t: pass > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > > I thought that "if x" simply calls "bool", but apparently this is not even > the case... > > Antony > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Thu Nov 13 01:19:35 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Wed, 12 Nov 2014 22:19:35 -0800 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: Message-ID: I know you can't in general, but this was in a context where I knew the array contained a single element, which "works" (it checks the truthiness of the contained element). Of course I didn't consider the case where the element contained was itself a (non-trivial) array, thus the finding. The link you posted doesn't seem to address what magic numpy can do to make "bool(x)" and "if x:" have different behaviors (FWIW, "t.__bool__()" also returns True). Antony 2014-11-12 20:15 GMT-08:00 Michael Sarahan : > Hi Antony, > > In general, you can't use numpy arrays in if statements without using > any() or all() or some other means of obtaining a single boolean value from > the whole array. > > I think your confusion is that bool() uses truth testing rules outlined > here: https://docs.python.org/release/2.5.2/lib/truth.html > > If statements in theory have equivalent behavior, but take slightly > different paths (they don't call bool() directly). This SO post was > enlightening to me: > > http://stackoverflow.com/questions/11885382/is-there-any-difference-between-if-boolx-and-if-x-in-python > > Without looking at Numpy's code, I'd bet Numpy arrays probably define > __bool__ or __nonzero__ in such a way that the ValueError is raised when it > makes sense to do so. > > HTH, > Mike > > On Tue, Nov 11, 2014 at 12:46 PM, Antony Lee > wrote: > >> I am puzzled by the following (numpy 1.9.0, python 3.4.2): >> >> In [1]: t = array(None); t[()] = array([None, None]) # Construct a 0d >> array of dtype object, containing a single numpy array with 2 elements >> >> In [2]: bool(t) >> Out[2]: True >> >> In [3]: if t: pass >> >> --------------------------------------------------------------------------- >> ValueError Traceback (most recent call >> last) >> ValueError: The truth value of an array with more than one element is >> ambiguous. Use a.any() or a.all() >> >> I thought that "if x" simply calls "bool", but apparently this is not >> even the case... >> >> Antony >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Thu Nov 13 02:10:52 2014 From: sebix at sebix.at (Sebastian) Date: Thu, 13 Nov 2014 08:10:52 +0100 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: <545C848B.1090708@sebix.at> References: <545C848B.1090708@sebix.at> Message-ID: <5464597C.1060803@sebix.at> On 2014-11-04 19:44, Charles R Harris wrote: > On Tue, Nov 4, 2014 at 11:19 AM, Sebastian wrote: > >> On 2014-11-04 15:06, Todd wrote: >>> On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner > >>> > wrote: >>> >>> Hello, >>> >>> I want to bring up Issue #2522 'numpy.diff fails on unsigned >> integers >>> (Trac #1929)' [1], as it was resonsible for an error in one >> of our >>> programs. Short explanation of the bug: np.diff performs a >> subtraction >>> on the input array. If this is of type uint and the data >> contains >>> falling data, it results in an artihmetic underflow. >>> >>> >>> np.diff(np.array([0,1,0], dtype=np.uint8)) >>> array([ 1, 255], dtype=uint8) >>> >>> @charris proposed either >>> - a note to the doc string and maybe an example to clarify >> things >>> - or raise a warning >>> but with a discussion on the list. >>> >>> I would like to start it now, as it is an error which is not >> easily >>> detectable (no errors or warnings are thrown). In our case >> the >>> type of a >>> data sequence, with only zeros and ones, had type f8 as also >> every >>> other >>> one, has been changed to u4. As the programs looked for >> values ==1 and >>> ==-1, it broke silently. >>> In my opinion, a note in the docs is not enough and does not >> help >>> if the >>> type changed or set after the program has been written. >>> I'd go for automatic upcasting of uints by default and an >> option >>> to turn >>> it off, if this behavior is explicitly wanted. This wouldn't >> be >>> correct >>> from the point of view of a programmer, but as most of the >> users >>> have a >>> scientific background who excpect it 'to work', instead of >> sth is >>> theoretically correct but not convenient. (I count myself to >> the first >>> group) >>> >>> >>> >>> When you say "automatic upcasting", that would be, for example >> uint8 >>> to int16? What about for uint64? There is no int128. >> The upcast should go to the next bigger, otherwise it would again >> result >> in wrong values. uint64 we can't do that, so it has to stay. >>> Also, when you say "by default", is this only when an overflow is >>> detected, or always? >> I don't know how I could detect an overflow in the diff-function. >> In >> subtraction it should be possible, but that's very deep in the >> numpy-internals. >>> How would the option to turn it off be implemented? An argument >> to >>> np.diff or some sort of global option? >> I thought of a parameter upcast_int=True for the function. > > Could check for non-decreasing sequence in the unsigned case. Note > that differences of signed integers can also overflow. One way to > check in general is to determine the expected sign using comparisons. I think you mean a decreasing/non-increasing instead of non-decreasing sequence? It's also the same check as checking for a sorted sequence. But I currently don't know how I could do that efficiently without np.diff in Python, in Cython it should be easily possible. np.gradient has the same problem: >>> np.random.seed(89) >>> d = np.random.randint(0,2,size=10).astype(np.uint8); d array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0], dtype=uint8) >>> np.diff(d) array([255, 0, 1, 255, 1, 0, 255, 0, 0], dtype=uint8) >>> np.gradient(d) array([ 255. , 127.5, 0.5, 0. , 0. , 0.5, 127.5, 127.5, 0. , 0. ]) --- gpg --keyserver keys.gnupg.net --recv-key DC9B463B From robert.kern at gmail.com Thu Nov 13 03:20:50 2014 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Nov 2014 08:20:50 +0000 Subject: [Numpy-discussion] Subscribing to mailinglist not possible / sites down In-Reply-To: References: Message-ID: Thanks, they have been restored. You can email me directly next time, since if those sites are down, so is the mailing list, most likely. On Wed, Nov 12, 2014 at 10:30 AM, Stefan Otte wrote: > Hey *, > > The websites to subscribe to the numpy/scipy mailinglists seem to be down: > > - http://mail.scipy.org/mailman/listinfo/scipy-user > - http://mail.scipy.org/mailman/listinfo/scipy-user > - http://projects.scipy.org/pipermail/scipy-dev/ > > > And it's not just me: > http://www.downforeveryoneorjustme.com/http://projects.scipy.org/pipermail/scipy-dev/ > > Best, > Stefan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Robert Kern From sebastian at sipsolutions.net Thu Nov 13 03:42:45 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 13 Nov 2014 09:42:45 +0100 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: Message-ID: <1415868165.3721.0.camel@sebastian-t440> On Mi, 2014-11-12 at 22:19 -0800, Antony Lee wrote: > I know you can't in general, but this was in a context where I knew > the array contained a single element, which "works" (it checks the > truthiness of the contained element). Of course I didn't consider the > case where the element contained was itself a (non-trivial) array, > thus the finding. > The link you posted doesn't seem to address what magic numpy can do to > make "bool(x)" and "if x:" have different behaviors (FWIW, > "t.__bool__()" also returns True). > Antony > First sight, sounds like a bug. It should be passing out the error raised by the array inside. Can you open an issue? - Sebastian > 2014-11-12 20:15 GMT-08:00 Michael Sarahan : > Hi Antony, > > In general, you can't use numpy arrays in if statements > without using any() or all() or some other means of obtaining > a single boolean value from the whole array. > > I think your confusion is that bool() uses truth testing rules > outlined here: > https://docs.python.org/release/2.5.2/lib/truth.html > > If statements in theory have equivalent behavior, but take > slightly different paths (they don't call bool() directly). > This SO post was enlightening to me: > http://stackoverflow.com/questions/11885382/is-there-any-difference-between-if-boolx-and-if-x-in-python > > Without looking at Numpy's code, I'd bet Numpy arrays probably > define __bool__ or __nonzero__ in such a way that the > ValueError is raised when it makes sense to do so. > > HTH, > Mike > > > On Tue, Nov 11, 2014 at 12:46 PM, Antony Lee > wrote: > > I am puzzled by the following (numpy 1.9.0, python > 3.4.2): > > > In [1]: t = array(None); t[()] = array([None, None]) > # Construct a 0d array of dtype object, containing a > single numpy array with 2 elements > > > In [2]: bool(t) > Out[2]: True > > > In [3]: if t: pass > --------------------------------------------------------------------------- > ValueError Traceback > (most recent call last) > ValueError: The truth value of an array with more than > one element is ambiguous. Use a.any() or a.all() > > > I thought that "if x" simply calls "bool", but > apparently this is not even the case... > > > Antony > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From alan.isaac at gmail.com Thu Nov 13 08:24:37 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 13 Nov 2014 08:24:37 -0500 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: Message-ID: <5464B115.1040104@gmail.com> On 11/13/2014 1:19 AM, Antony Lee wrote: > "t.__bool__()" also returns True But t.__nonzero__() is being called in the `if` test. The question is: is the difference between `__nonzero__` and `__bool__` intentional. By the way, there has been a change in behavior. For example, in 1.7.1 if you call `t.__bool__()` it raised an attribute error -- unless one first called `t.__nonzero__()` and then called `t.__bool__()`, which was of course very weird and needed to be fixed. Maybe (?) not like this. In fact the oddity probably remains but moved. in 1.9.0 I see this: >>> np.__version__ '1.9.0' >>> t = np.array(None); t[()] = np.array([None, None]) >>> t.__nonzero__() Traceback (most recent call last): File "", line 1, in AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' >>> t.__bool__() True >>> t.__nonzero__() ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Alan Isaac From sebastian at sipsolutions.net Thu Nov 13 08:53:09 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 13 Nov 2014 14:53:09 +0100 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: <5464B115.1040104@gmail.com> References: <5464B115.1040104@gmail.com> Message-ID: <1415886789.10875.7.camel@sebastian-t440> On Do, 2014-11-13 at 08:24 -0500, Alan G Isaac wrote: > On 11/13/2014 1:19 AM, Antony Lee wrote: > > "t.__bool__()" also returns True > > > But t.__nonzero__() is being called in the `if` test. > The question is: is the difference between `__nonzero__` > and `__bool__` intentional. > > By the way, there has been a change in behavior. > For example, in 1.7.1 if you call `t.__bool__()` > it raised an attribute error -- unless one first > called `t.__nonzero__()` and then called `t.__bool__()`, > which was of course very weird and needed to be fixed. > Maybe (?) not like this. > > In fact the oddity probably remains but moved. in 1.9.0 I see this: > > >>> np.__version__ > '1.9.0' > >>> t = np.array(None); t[()] = np.array([None, None]) > >>> t.__nonzero__() > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' > >>> t.__bool__() > True > >>> t.__nonzero__() > ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() > Note that the error is set, but appears to be not indicated within the C-api. This can cause errors to be missed or appear later then they should... Might be just that you are seeing, though I could not find a reason for it in the numpy code on first glance. - Sebastian > Alan Isaac > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From toddrjen at gmail.com Thu Nov 13 08:57:03 2014 From: toddrjen at gmail.com (Todd) Date: Thu, 13 Nov 2014 14:57:03 +0100 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: <5464597C.1060803@sebix.at> References: <545C848B.1090708@sebix.at> <5464597C.1060803@sebix.at> Message-ID: On Thu, Nov 13, 2014 at 8:10 AM, Sebastian wrote: > On 2014-11-04 19:44, Charles R Harris wrote: > > On Tue, Nov 4, 2014 at 11:19 AM, Sebastian wrote: > > > >> On 2014-11-04 15:06, Todd wrote: > >>> On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner >> > >>> > wrote: > >>> > >>> Hello, > >>> > >>> I want to bring up Issue #2522 'numpy.diff fails on unsigned > >> integers > >>> (Trac #1929)' [1], as it was resonsible for an error in one > >> of our > >>> programs. Short explanation of the bug: np.diff performs a > >> subtraction > >>> on the input array. If this is of type uint and the data > >> contains > >>> falling data, it results in an artihmetic underflow. > >>> > >>> >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > >>> array([ 1, 255], dtype=uint8) > >>> > >>> @charris proposed either > >>> - a note to the doc string and maybe an example to clarify > >> things > >>> - or raise a warning > >>> but with a discussion on the list. > >>> > >>> I would like to start it now, as it is an error which is not > >> easily > >>> detectable (no errors or warnings are thrown). In our case > >> the > >>> type of a > >>> data sequence, with only zeros and ones, had type f8 as also > >> every > >>> other > >>> one, has been changed to u4. As the programs looked for > >> values ==1 and > >>> ==-1, it broke silently. > >>> In my opinion, a note in the docs is not enough and does not > >> help > >>> if the > >>> type changed or set after the program has been written. > >>> I'd go for automatic upcasting of uints by default and an > >> option > >>> to turn > >>> it off, if this behavior is explicitly wanted. This wouldn't > >> be > >>> correct > >>> from the point of view of a programmer, but as most of the > >> users > >>> have a > >>> scientific background who excpect it 'to work', instead of > >> sth is > >>> theoretically correct but not convenient. (I count myself to > >> the first > >>> group) > >>> > >>> > >>> > >>> When you say "automatic upcasting", that would be, for example > >> uint8 > >>> to int16? What about for uint64? There is no int128. > >> The upcast should go to the next bigger, otherwise it would again > >> result > >> in wrong values. uint64 we can't do that, so it has to stay. > >>> Also, when you say "by default", is this only when an overflow is > >>> detected, or always? > >> I don't know how I could detect an overflow in the diff-function. > >> In > >> subtraction it should be possible, but that's very deep in the > >> numpy-internals. > >>> How would the option to turn it off be implemented? An argument > >> to > >>> np.diff or some sort of global option? > >> I thought of a parameter upcast_int=True for the function. > > > > Could check for non-decreasing sequence in the unsigned case. Note > > that differences of signed integers can also overflow. One way to > > check in general is to determine the expected sign using comparisons. > > I think you mean a decreasing/non-increasing instead of non-decreasing > sequence? > It's also the same check as checking for a sorted sequence. But I > currently don't know how I could do that efficiently without np.diff in > Python, in Cython it should be easily possible. > > > np.gradient has the same problem: > >>> np.random.seed(89) > >>> d = np.random.randint(0,2,size=10).astype(np.uint8); d > array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0], dtype=uint8) > >>> np.diff(d) > array([255, 0, 1, 255, 1, 0, 255, 0, 0], dtype=uint8) > >>> np.gradient(d) > array([ 255. , 127.5, 0.5, 0. , 0. , 0.5, 127.5, 127.5, > 0. , 0. ]) > > Consider it is generally an error, might it be good to have a general warning built into the int dtypes regarding overflow errors? That warning can then be caught by the diff function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Nov 13 10:10:07 2014 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 13 Nov 2014 07:10:07 -0800 Subject: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers In-Reply-To: <5464597C.1060803@sebix.at> References: <545C848B.1090708@sebix.at> <5464597C.1060803@sebix.at> Message-ID: On Wed, Nov 12, 2014 at 11:10 PM, Sebastian wrote: > On 2014-11-04 19:44, Charles R Harris wrote: > > On Tue, Nov 4, 2014 at 11:19 AM, Sebastian wrote: > > > >> On 2014-11-04 15:06, Todd wrote: > >>> On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner >> > >>> > wrote: > >>> > >>> Hello, > >>> > >>> I want to bring up Issue #2522 'numpy.diff fails on unsigned > >> integers > >>> (Trac #1929)' [1], as it was resonsible for an error in one > >> of our > >>> programs. Short explanation of the bug: np.diff performs a > >> subtraction > >>> on the input array. If this is of type uint and the data > >> contains > >>> falling data, it results in an artihmetic underflow. > >>> > >>> >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > >>> array([ 1, 255], dtype=uint8) > >>> > >>> @charris proposed either > >>> - a note to the doc string and maybe an example to clarify > >> things > >>> - or raise a warning > >>> but with a discussion on the list. > >>> > >>> I would like to start it now, as it is an error which is not > >> easily > >>> detectable (no errors or warnings are thrown). In our case > >> the > >>> type of a > >>> data sequence, with only zeros and ones, had type f8 as also > >> every > >>> other > >>> one, has been changed to u4. As the programs looked for > >> values ==1 and > >>> ==-1, it broke silently. > >>> In my opinion, a note in the docs is not enough and does not > >> help > >>> if the > >>> type changed or set after the program has been written. > >>> I'd go for automatic upcasting of uints by default and an > >> option > >>> to turn > >>> it off, if this behavior is explicitly wanted. This wouldn't > >> be > >>> correct > >>> from the point of view of a programmer, but as most of the > >> users > >>> have a > >>> scientific background who excpect it 'to work', instead of > >> sth is > >>> theoretically correct but not convenient. (I count myself to > >> the first > >>> group) > >>> > >>> > >>> > >>> When you say "automatic upcasting", that would be, for example > >> uint8 > >>> to int16? What about for uint64? There is no int128. > >> The upcast should go to the next bigger, otherwise it would again > >> result > >> in wrong values. uint64 we can't do that, so it has to stay. > >>> Also, when you say "by default", is this only when an overflow is > >>> detected, or always? > >> I don't know how I could detect an overflow in the diff-function. > >> In > >> subtraction it should be possible, but that's very deep in the > >> numpy-internals. > >>> How would the option to turn it off be implemented? An argument > >> to > >>> np.diff or some sort of global option? > >> I thought of a parameter upcast_int=True for the function. > > > > Could check for non-decreasing sequence in the unsigned case. Note > > that differences of signed integers can also overflow. One way to > > check in general is to determine the expected sign using comparisons. > > I think you mean a decreasing/non-increasing instead of non-decreasing > sequence? > It's also the same check as checking for a sorted sequence. But I > currently don't know how I could do that efficiently without np.diff in > Python, in Cython it should be easily possible. > For a 1D array, you could simply do: if a.dtype.kind == 'u' and np.any(a[1:] < a[:-1]): # warn, upcast, or whatever return a[1:] - a[:-1] The general case is a little more complicated because of the axis handling, but if you look at the source https://github.com/numpy/numpy/blob/v1.9.1/numpy/lib/function_base.py#L1055, you just need to replace 1: with slice1 and :-1 with slice2. > > np.gradient has the same problem: > >>> np.random.seed(89) > >>> d = np.random.randint(0,2,size=10).astype(np.uint8); d > array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0], dtype=uint8) > >>> np.diff(d) > array([255, 0, 1, 255, 1, 0, 255, 0, 0], dtype=uint8) > >>> np.gradient(d) > array([ 255. , 127.5, 0.5, 0. , 0. , 0.5, 127.5, 127.5, > 0. , 0. ]) > The case of gradient is kind of sad, because it actually goes through the trouble of figuring out the right dtype for the output if yt isn't floating point already, but doesn't upcast the operations... Take a look at the source here, https://github.com/numpy/numpy/blob/v1.9.1/numpy/lib/function_base.py#L886, and wherever you find something like out[slice1] = (y[slice2] - y[slice3]) simply replace it with: np.subtract(y[slice2], y[slice3], dtype=dtype, out=out[slice1]) and it should work fine. > --- > gpg --keyserver keys.gnupg.net --recv-key DC9B463B > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Thu Nov 13 12:37:02 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 13 Nov 2014 09:37:02 -0800 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: <5464B115.1040104@gmail.com> References: <5464B115.1040104@gmail.com> Message-ID: On Python3, __nonzero__ is never defined (always raises an AttributeError), even after calling __bool__. 2014-11-13 5:24 GMT-08:00 Alan G Isaac : > On 11/13/2014 1:19 AM, Antony Lee wrote: > > "t.__bool__()" also returns True > > > But t.__nonzero__() is being called in the `if` test. > The question is: is the difference between `__nonzero__` > and `__bool__` intentional. > > By the way, there has been a change in behavior. > For example, in 1.7.1 if you call `t.__bool__()` > it raised an attribute error -- unless one first > called `t.__nonzero__()` and then called `t.__bool__()`, > which was of course very weird and needed to be fixed. > Maybe (?) not like this. > > In fact the oddity probably remains but moved. in 1.9.0 I see this: > > >>> np.__version__ > '1.9.0' > >>> t = np.array(None); t[()] = np.array([None, None]) > >>> t.__nonzero__() > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' > >>> t.__bool__() > True > >>> t.__nonzero__() > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > > Alan Isaac > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Thu Nov 13 13:05:12 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 13 Nov 2014 13:05:12 -0500 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: <5464B115.1040104@gmail.com> Message-ID: <5464F2D8.2040602@gmail.com> On 11/13/2014 12:37 PM, Antony Lee wrote: > On Python3, __nonzero__ is never defined (always raises an AttributeError), even after calling __bool__. The example I posted was Python 3.4.1 with numpy 1.9.0. fwiw, Alan Isaac Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '1.9.0' >>> t = np.array(None); t[()] = np.array([None, None]) >>> t.__nonzero__() Traceback (most recent call last): File "", line 1, in AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' >>> t.__bool__() True >>> t.__nonzero__() ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>> From antony.lee at berkeley.edu Thu Nov 13 13:08:45 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 13 Nov 2014 10:08:45 -0800 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: <5464F2D8.2040602@gmail.com> References: <5464B115.1040104@gmail.com> <5464F2D8.2040602@gmail.com> Message-ID: Dunno, seems unlikely that something changed with Python 3.4.2... $ python --version Python 3.4.2 $ python -c 'import numpy as np; print(np.__version__); t = np.array(None); t[()] = np.array([None, None]); t.__bool__(); t.__nonzero__()' 1.9.0 Traceback (most recent call last): File "", line 1, in AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' 2014-11-13 10:05 GMT-08:00 Alan G Isaac : > On 11/13/2014 12:37 PM, Antony Lee wrote: > > On Python3, __nonzero__ is never defined (always raises an > AttributeError), even after calling __bool__. > > > The example I posted was Python 3.4.1 with numpy 1.9.0. > > fwiw, > Alan Isaac > > Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 > bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > >>> np.__version__ > '1.9.0' > >>> t = np.array(None); t[()] = np.array([None, None]) > >>> t.__nonzero__() > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' > >>> t.__bool__() > True > >>> t.__nonzero__() > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > >>> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Nov 13 13:32:06 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 13 Nov 2014 18:32:06 +0000 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: <5464B115.1040104@gmail.com> References: <5464B115.1040104@gmail.com> Message-ID: On Thu, Nov 13, 2014 at 1:24 PM, Alan G Isaac wrote: > On 11/13/2014 1:19 AM, Antony Lee wrote: >> "t.__bool__()" also returns True > > But t.__nonzero__() is being called in the `if` test. > The question is: is the difference between `__nonzero__` > and `__bool__` intentional. __nonzero__ and __bool__ should be identical; they refer to the same magic method, it's just that Py2 calls it __nonzero__ and Py3 calls it __bool__. bool(t) and 'if t: ...' should always be identical at the interpreter level; that they aren't is probably because we're someone violating some invariant in the Python C API (like @seberg suggested). In Py2 they both call __nonzero__, and in Py3 they both call __bool__. > By the way, there has been a change in behavior. > For example, in 1.7.1 if you call `t.__bool__()` > it raised an attribute error -- unless one first > called `t.__nonzero__()` and then called `t.__bool__()`, > which was of course very weird and needed to be fixed. > Maybe (?) not like this. > > In fact the oddity probably remains but moved. in 1.9.0 I see this: > > >>> np.__version__ > '1.9.0' > >>> t = np.array(None); t[()] = np.array([None, None]) > >>> t.__nonzero__() > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' > >>> t.__bool__() > True > >>> t.__nonzero__() > ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() I think you're being misled by buggy exception handling weirdness, where the ValueError raised by calling __bool__ is getting delayed, and then pre-empting the AttributeError that should be generated by the call to __nonzero__. Consider: >>> t.__bool__() True >>> 1 # any expression here will do ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>> t.__nonzero__() Traceback (most recent call last): File "", line 1, in AttributeError: 'numpy.ndarray' object has no attribute '__nonzero__' -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From alan.isaac at gmail.com Thu Nov 13 14:29:03 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 13 Nov 2014 14:29:03 -0500 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: References: <5464B115.1040104@gmail.com> Message-ID: <5465067F.7090400@gmail.com> On 11/13/2014 1:32 PM, Nathaniel Smith wrote: > I think you're being misled by buggy exception handling weirdness, > where the ValueError raised by calling __bool__ is getting delayed, > and then pre-empting the AttributeError that should be generated by > the call to __nonzero__. Aha! Thanks. From charlesr.harris at gmail.com Fri Nov 14 11:15:11 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 14 Nov 2014 11:15:11 -0500 Subject: [Numpy-discussion] truthiness of object arrays In-Reply-To: <5464B115.1040104@gmail.com> References: <5464B115.1040104@gmail.com> Message-ID: On Thu, Nov 13, 2014 at 8:24 AM, Alan G Isaac wrote: > On 11/13/2014 1:19 AM, Antony Lee wrote: > > "t.__bool__()" also returns True > > > But t.__nonzero__() is being called in the `if` test. > The question is: is the difference between `__nonzero__` > and `__bool__` intentional. > > By the way, there has been a change in behavior. > For example, in 1.7.1 if you call `t.__bool__()` > it raised an attribute error -- unless one first > called `t.__nonzero__()` and then called `t.__bool__()`, > which was of course very weird and needed to be fixed. > Maybe (?) not like this. > The change probably came in with the Python2/3 common code base. Python3 replaces `__nonzero__` with `__bool__`. IIRC, the two are now the same method. I don't have the code available to check... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmabey at swri.org Fri Nov 14 14:06:31 2014 From: gmabey at swri.org (Glen Mabey) Date: Fri, 14 Nov 2014 19:06:31 +0000 Subject: [Numpy-discussion] Fwd: numpy.i and std::complex References: <46BC1BC4-4E02-4700-843E-8388E7923C53@electro.swri.edu> Message-ID: <91E1D2AA-0265-413C-967F-5FAD4D7C66DE@electro.swri.edu> Hello, Ok, here's my attempt -- https://github.com/gmabey/numpy/compare/swig-std-complex Glen On Oct 27, 2014, at 11:13 AM, Bill Spotz wrote: > Supporting std::complex<> was just low enough priority for me that I decided to wait until someone expressed interest ... and now, many years later, someone finally has. > > I would be happy to include this into numpy.i, but I would like to see some tests in the numpy repository demonstrating that it works. These could be relatively short and simple, and since float and double are the only scalar data types that I could foresee supporting, there would not be a need for testing the large numbers of data types that the other tests cover. > > I would also want to protect the references to C++ objects with '#ifdef __cplusplus', but that is easy enough. > > -Bill > > On Oct 27, 2014, at 9:06 AM, Glen Mabey wrote: > >> Hello, >> >> I was very excited to learn about numpy.i for easy numpy+swigification of C code -- it's really handy. >> >> Knowing that swig wraps C code, I wasn't too surprised that there was the issue with complex data types (as described at http://docs.scipy.org/doc/numpy/reference/swig.interface-file.html#other-common-types-complex), but still it was pretty disappointing because most of my data is complex, and I'm invoking methods written to use C++'s std::complex class. >> >> After quite a bit of puzzling and not much help from previous mailing list posts, I created this very brief but very useful file, which I call numpy_std_complex.i -- >> >> /* -*- C -*- (not really, but good for syntax highlighting) */ >> #ifdef SWIGPYTHON >> >> %include "numpy.i" >> >> %include >> >> %numpy_typemaps(std::complex, NPY_CFLOAT , int) >> %numpy_typemaps(std::complex, NPY_CDOUBLE, int) >> >> #endif /* SWIGPYTHON */ >> >> >> I'd really like for this to be included alongside numpy.i -- but maybe I overestimate the number of numpy users who use complex data (let your voice be heard!) and who also end up using std::complex in C++ land. >> >> Or if anyone wants to improve upon this usage I would be very happy to hear about what I'm missing. >> >> I'm sure there's a documented way to submit this file to the git repo, but let me simultaneously ask whether list subscribers think this is worthwhile and ask someone to add+push it for me ? >> >> Thanks, >> Glen Mabey >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From andyspiros at gmail.com Sun Nov 16 12:42:09 2014 From: andyspiros at gmail.com (Andrea Arteaga) Date: Sun, 16 Nov 2014 18:42:09 +0100 Subject: [Numpy-discussion] Initializing array from buffer Message-ID: Hello. Using the numpy.frombuffer function [1] one can initialize a numpy array using an existing python object that implements the buffer protocol [2]. This is great, but currently this function supports only 1D buffers, even if the provided buffer is multidimensional and it exposes all information about its structure (shape, strides, data type). Apparently, one can extract every kind of buffer information out of a buffer of a numpy array (pointer, number of dimensions, shape, strides, suboffsets,...), but the other way around is only partially implemented: providing a multidimensional buffer does not mean being able of creating a numpy array the uses that buffer with the desired structure. My use case is the following: we have a some 3D arrays in our C++ framework. The ordering of the elements in these arrays is neither C nor Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. IKJ. Moreover we put some padding to optimize aligned access. This kind of memory structure cannot be just expressed as 'C' or 'Fortran', but it can be perfectly expressed using the Python buffer protocol by providing the shape and the strides. We would like to export this structure to a numpy array that should be able of accessing the same memory locations in a consistent way and make some operations like initializing the content or plotting it. Is this currently possible? If not, is it planned to implement such a feature? ========== Maybe just to clarify I could show an example entirely in python. Assume a in a 2D numpy array: a = np.ones((10,20)) It contains information about its structure which can be portably accessed using its data member: a.data.format == 'd' a.data.ndim == 2 a.data.shape == (10,20) a.data.strides == (160,8) Unfortunately, when initializing an array b from this buffer, the structure of the buffer is "downgraded" to unidimensional shape: b = np.frombuffer(a.data) b.ndim == 1 b.shape == (200,) b.strides == (8,) I wished b had the same multi-dimensional structure of a. (This is of course a very simple example. In my use case I would initialize b with my own buffer instead of that of another numpy array). Best regards [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.frombuffer.html [2] https://docs.python.org/3/c-api/buffer.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Nov 17 06:03:09 2014 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 17 Nov 2014 13:03:09 +0200 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: Message-ID: <87k32uozrm.fsf@sun.ac.za> Hi Andrea On 2014-11-16 19:42:09, Andrea Arteaga wrote: > My use case is the following: we have a some 3D arrays in our C++ > framework. The ordering of the elements in these arrays is neither C nor > Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in > memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. IKJ. > Moreover we put some padding to optimize aligned access. This kind of > memory structure cannot be just expressed as 'C' or 'Fortran', but it can > be perfectly expressed using the Python buffer protocol by providing the > shape and the strides. We would like to export this structure to a numpy > array that should be able of accessing the same memory locations in a > consistent way and make some operations like initializing the content or > plotting it. > > Is this currently possible? > If not, is it planned to implement such a feature? This looks like something that should be accomplished fairly easily using the ``__array_interface__`` dictionary, as described here: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html Any object that exposes a suitable dictionary named ``__array_interface__`` may be converted to a NumPy array. It has the following important keys: shape typestr data: (20495857, True); 2-tuple?pointer to data and boolean to indicate whether memory is read-only strides version: 3 Regards St?fan From edisongustavo at gmail.com Mon Nov 17 12:08:47 2014 From: edisongustavo at gmail.com (Edison Gustavo Muenz) Date: Mon, 17 Nov 2014 15:08:47 -0200 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: Message-ID: Have you tried using the C-API to create the array? This link might be of help: http://docs.scipy.org/doc/numpy/reference/c-api.array.html#creating-arrays I know that Boost.Python can handle this. On Sun, Nov 16, 2014 at 3:42 PM, Andrea Arteaga wrote: > Hello. > Using the numpy.frombuffer function [1] one can initialize a numpy array > using an existing python object that implements the buffer protocol [2]. > This is great, but currently this function supports only 1D buffers, even > if the provided buffer is multidimensional and it exposes all information > about its structure (shape, strides, data type). > > Apparently, one can extract every kind of buffer information out of a > buffer of a numpy array (pointer, number of dimensions, shape, strides, > suboffsets,...), but the other way around is only partially implemented: > providing a multidimensional buffer does not mean being able of creating a > numpy array the uses that buffer with the desired structure. > > My use case is the following: we have a some 3D arrays in our C++ > framework. The ordering of the elements in these arrays is neither C nor > Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in > memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. IKJ. > Moreover we put some padding to optimize aligned access. This kind of > memory structure cannot be just expressed as 'C' or 'Fortran', but it can > be perfectly expressed using the Python buffer protocol by providing the > shape and the strides. We would like to export this structure to a numpy > array that should be able of accessing the same memory locations in a > consistent way and make some operations like initializing the content or > plotting it. > > Is this currently possible? > If not, is it planned to implement such a feature? > > ========== > > Maybe just to clarify I could show an example entirely in python. Assume a > in a 2D numpy array: > > a = np.ones((10,20)) > > It contains information about its structure which can be portably accessed > using its data member: > > a.data.format == 'd' > a.data.ndim == 2 > a.data.shape == (10,20) > a.data.strides == (160,8) > > Unfortunately, when initializing an array b from this buffer, the > structure of the buffer is "downgraded" to unidimensional shape: > > b = np.frombuffer(a.data) > > b.ndim == 1 > b.shape == (200,) > b.strides == (8,) > > I wished b had the same multi-dimensional structure of a. > > (This is of course a very simple example. In my use case I would > initialize b with my own buffer instead of that of another numpy array). > > Best regards > > [1] > http://docs.scipy.org/doc/numpy/reference/generated/numpy.frombuffer.html > [2] https://docs.python.org/3/c-api/buffer.html > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyspiros at gmail.com Mon Nov 17 15:33:32 2014 From: andyspiros at gmail.com (Andrea Arteaga) Date: Mon, 17 Nov 2014 21:33:32 +0100 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: Message-ID: Hi. Yesterday I tried to make use of the C API, but I did not manage to have anything useful. The reference is very well done, but I feel the lack for some tutorial that would guide with some examples. Do you know of any? The array interface looks sounds like a very good solution. In a sense it is a Numpy specific version of the buffer protocol, a bit simpler but less generic. It looks very easy to implement and clean. I will try this way. Thank you so much for the useful links. Andrea Arteaga 2014-11-17 18:08 GMT+01:00 Edison Gustavo Muenz : > Have you tried using the C-API to create the array? This link might be of > help: > http://docs.scipy.org/doc/numpy/reference/c-api.array.html#creating-arrays > > I know that Boost.Python can handle this. > > On Sun, Nov 16, 2014 at 3:42 PM, Andrea Arteaga > wrote: > >> Hello. >> Using the numpy.frombuffer function [1] one can initialize a numpy array >> using an existing python object that implements the buffer protocol [2]. >> This is great, but currently this function supports only 1D buffers, even >> if the provided buffer is multidimensional and it exposes all information >> about its structure (shape, strides, data type). >> >> Apparently, one can extract every kind of buffer information out of a >> buffer of a numpy array (pointer, number of dimensions, shape, strides, >> suboffsets,...), but the other way around is only partially implemented: >> providing a multidimensional buffer does not mean being able of creating a >> numpy array the uses that buffer with the desired structure. >> >> My use case is the following: we have a some 3D arrays in our C++ >> framework. The ordering of the elements in these arrays is neither C nor >> Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in >> memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. IKJ. >> Moreover we put some padding to optimize aligned access. This kind of >> memory structure cannot be just expressed as 'C' or 'Fortran', but it can >> be perfectly expressed using the Python buffer protocol by providing the >> shape and the strides. We would like to export this structure to a numpy >> array that should be able of accessing the same memory locations in a >> consistent way and make some operations like initializing the content or >> plotting it. >> >> Is this currently possible? >> If not, is it planned to implement such a feature? >> >> ========== >> >> Maybe just to clarify I could show an example entirely in python. Assume >> a in a 2D numpy array: >> >> a = np.ones((10,20)) >> >> It contains information about its structure which can be portably >> accessed using its data member: >> >> a.data.format == 'd' >> a.data.ndim == 2 >> a.data.shape == (10,20) >> a.data.strides == (160,8) >> >> Unfortunately, when initializing an array b from this buffer, the >> structure of the buffer is "downgraded" to unidimensional shape: >> >> b = np.frombuffer(a.data) >> >> b.ndim == 1 >> b.shape == (200,) >> b.strides == (8,) >> >> I wished b had the same multi-dimensional structure of a. >> >> (This is of course a very simple example. In my use case I would >> initialize b with my own buffer instead of that of another numpy array). >> >> Best regards >> >> [1] >> http://docs.scipy.org/doc/numpy/reference/generated/numpy.frombuffer.html >> [2] https://docs.python.org/3/c-api/buffer.html >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Nov 17 15:52:16 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 17 Nov 2014 20:52:16 +0000 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 5:42 PM, Andrea Arteaga wrote: > Hello. > Using the numpy.frombuffer function [1] one can initialize a numpy array > using an existing python object that implements the buffer protocol [2]. > This is great, but currently this function supports only 1D buffers, even if > the provided buffer is multidimensional and it exposes all information about > its structure (shape, strides, data type). np.frombuffer is not often used, and I'm not sure if it's been updated for the new py3 buffer protocol. (The old buffer protocol only supported 1d buffers.) Have you tried just using np.asarray? It seems to work fine with multidimensional memoryview objects at least, which use the new buffer protocol: In [12]: a = np.ones((2, 3)) In [13]: a_buf = memoryview(a) In [14]: a_buf Out[14]: In [15]: np.asarray(a_buf) Out[15]: array([[ 1., 1., 1.], [ 1., 1., 1.]]) -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From sturla.molden at gmail.com Sun Nov 16 19:27:51 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 17 Nov 2014 00:27:51 +0000 (UTC) Subject: [Numpy-discussion] Initializing array from buffer References: Message-ID: <357693592437876378.268056sturla.molden-gmail.com@news.gmane.org> Andrea Arteaga wrote: > My use case is the following: we have a some 3D arrays in our C++ > framework. The ordering of the elements in these arrays is neither C nor > Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in > memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. IKJ. > Moreover we put some padding to optimize aligned access. This kind of > memory structure cannot be just expressed as 'C' or 'Fortran', but it can > be perfectly expressed using the Python buffer protocol by providing the > shape and the strides. We would like to export this structure to a numpy > array that should be able of accessing the same memory locations in a > consistent way and make some operations like initializing the content or > plotting it. > > Is this currently possible? > If not, is it planned to implement such a feature? If you are already coding in C++, just use PyArray_New or PyArray_NewFromDescr: http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_New http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_NewFromDescr Apart from that, numpy.array and numpy.asarray can also accept a PEP 3118 buffer. Sturla From rmcgibbo at gmail.com Mon Nov 17 22:21:32 2014 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Mon, 17 Nov 2014 19:21:32 -0800 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: <357693592437876378.268056sturla.molden-gmail.com@news.gmane.org> References: <357693592437876378.268056sturla.molden-gmail.com@news.gmane.org> Message-ID: The np.ndarray constructor takes a strides argument argument, and a buffer. Is it not sufficiently flexible? -Robert On Sun, Nov 16, 2014 at 4:27 PM, Sturla Molden wrote: > Andrea Arteaga wrote: > > > My use case is the following: we have a some 3D arrays in our C++ > > framework. The ordering of the elements in these arrays is neither C nor > > Fortran style: it might be IJK (i.e. C style, 3rd dimension contiguous in > > memory), KJI (i.e. Fortran style, first dimension contiguous) or, e.g. > IKJ. > > Moreover we put some padding to optimize aligned access. This kind of > > memory structure cannot be just expressed as 'C' or 'Fortran', but it can > > be perfectly expressed using the Python buffer protocol by providing the > > shape and the strides. We would like to export this structure to a numpy > > array that should be able of accessing the same memory locations in a > > consistent way and make some operations like initializing the content or > > plotting it. > > > > Is this currently possible? > > If not, is it planned to implement such a feature? > > If you are already coding in C++, just use PyArray_New or > PyArray_NewFromDescr: > > http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_New > > http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_NewFromDescr > > Apart from that, numpy.array and numpy.asarray can also accept a PEP 3118 > buffer. > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 18 13:20:30 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 18 Nov 2014 18:20:30 +0000 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement Message-ID: Hi, I have not followed closely the changes that happen in 1.9.1, but was surprised by the following: x = np.zeros(12, "d") assert x.flags.aligned # fails This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Nov 18 13:26:37 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 18 Nov 2014 11:26:37 -0700 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 11:20 AM, David Cournapeau wrote: > Hi, > > I have not followed closely the changes that happen in 1.9.1, but was > surprised by the following: > > x = np.zeros(12, "d") > assert x.flags.aligned # fails > > This is running numpy 1.9.1 built on windows with VS 2008. Is it expected > that zeros may return a non-aligned array ? > The alignment requirements may have become stricter, Julian would know. Is this on 32 or 64 Windows? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Nov 18 13:26:39 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 18 Nov 2014 19:26:39 +0100 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: Message-ID: <546B8F5F.4000507@googlemail.com> On 18.11.2014 19:20, David Cournapeau wrote: > Hi, > > I have not followed closely the changes that happen in 1.9.1, but was > surprised by the following: > > x = np.zeros(12, "d") > assert x.flags.aligned # fails > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > expected that zeros may return a non-aligned array ? > what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"] there are problems with complex types but doubles should be aligned even on 32 bit. From cournape at gmail.com Tue Nov 18 13:35:00 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 18 Nov 2014 18:35:00 +0000 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: <546B8F5F.4000507@googlemail.com> References: <546B8F5F.4000507@googlemail.com> Message-ID: It is on windows 32 bits, but I would need to make this work for complex (pair of double) as well. Is this a bug (I assumed array creation methods would always create aligned arrays for their type) ? Seems like quite a bit of code out there would assume this (scipy itself does for example). (the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., because f2py intent(inout) fails on work arrays created by zeros, this is a windows-32 only failure). David On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 18.11.2014 19:20, David Cournapeau wrote: > > Hi, > > > > I have not followed closely the changes that happen in 1.9.1, but was > > surprised by the following: > > > > x = np.zeros(12, "d") > > assert x.flags.aligned # fails > > > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > > expected that zeros may return a non-aligned array ? > > > > what is the real alignment of the array? Are you on 32 bit or 64 bit? > What is the alignment of doubles in windows (linux its 4 byte on 32 bit > 8 byte on 64 bit (% special compiler flags)? > print x.__array_interface__["data"] > > there are problems with complex types but doubles should be aligned even > on 32 bit. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 18 13:37:14 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 18 Nov 2014 18:37:14 +0000 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: <546B8F5F.4000507@googlemail.com> Message-ID: Additional point: it seems to always return aligned data on 1.8.1 (same platform/compiler/everything). On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau wrote: > It is on windows 32 bits, but I would need to make this work for complex > (pair of double) as well. > > Is this a bug (I assumed array creation methods would always create > aligned arrays for their type) ? Seems like quite a bit of code out there > would assume this (scipy itself does for example). > > (the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., > because f2py intent(inout) fails on work arrays created by zeros, this is a > windows-32 only failure). > > David > > On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor < > jtaylor.debian at googlemail.com> wrote: > >> On 18.11.2014 19:20, David Cournapeau wrote: >> > Hi, >> > >> > I have not followed closely the changes that happen in 1.9.1, but was >> > surprised by the following: >> > >> > x = np.zeros(12, "d") >> > assert x.flags.aligned # fails >> > >> > This is running numpy 1.9.1 built on windows with VS 2008. Is it >> > expected that zeros may return a non-aligned array ? >> > >> >> what is the real alignment of the array? Are you on 32 bit or 64 bit? >> What is the alignment of doubles in windows (linux its 4 byte on 32 bit >> 8 byte on 64 bit (% special compiler flags)? >> print x.__array_interface__["data"] >> >> there are problems with complex types but doubles should be aligned even >> on 32 bit. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Nov 18 13:40:39 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 18 Nov 2014 19:40:39 +0100 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: <546B8F5F.4000507@googlemail.com> Message-ID: <546B92A7.90707@googlemail.com> < 1.9 lies about alignment it doesn't actually check for new arrays. is the array aligned? On 18.11.2014 19:37, David Cournapeau wrote: > Additional point: it seems to always return aligned data on 1.8.1 (same > platform/compiler/everything). > > On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau > wrote: > > It is on windows 32 bits, but I would need to make this work for > complex (pair of double) as well. > > Is this a bug (I assumed array creation methods would always create > aligned arrays for their type) ? Seems like quite a bit of code out > there would assume this (scipy itself does for example). > > (the context is > 100 test failures on scipy 0.14.x on top of numpy > 1.9., because f2py intent(inout) fails on work arrays created by > zeros, this is a windows-32 only failure). > > David > > On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor > > wrote: > > On 18.11.2014 19:20, David Cournapeau wrote: > > Hi, > > > > I have not followed closely the changes that happen in 1.9.1, > but was > > surprised by the following: > > > > x = np.zeros(12, "d") > > assert x.flags.aligned # fails > > > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > > expected that zeros may return a non-aligned array ? > > > > what is the real alignment of the array? Are you on 32 bit or 64 > bit? > What is the alignment of doubles in windows (linux its 4 byte on > 32 bit > 8 byte on 64 bit (% special compiler flags)? > print x.__array_interface__["data"] > > there are problems with complex types but doubles should be > aligned even > on 32 bit. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Tue Nov 18 13:56:06 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 18 Nov 2014 18:56:06 +0000 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: <546B92A7.90707@googlemail.com> References: <546B8F5F.4000507@googlemail.com> <546B92A7.90707@googlemail.com> Message-ID: On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > < 1.9 lies about alignment it doesn't actually check for new arrays. > When I do the following on 1.8.1 with win 32 bits: x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # always true but on 1.9.1: x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # not always true I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this case (unlikely to be a coincidence, as I try quite a few times with different sizes). > > is the array aligned? > > On 18.11.2014 19:37, David Cournapeau wrote: > > Additional point: it seems to always return aligned data on 1.8.1 (same > > platform/compiler/everything). > > > > On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau > > wrote: > > > > It is on windows 32 bits, but I would need to make this work for > > complex (pair of double) as well. > > > > Is this a bug (I assumed array creation methods would always create > > aligned arrays for their type) ? Seems like quite a bit of code out > > there would assume this (scipy itself does for example). > > > > (the context is > 100 test failures on scipy 0.14.x on top of numpy > > 1.9., because f2py intent(inout) fails on work arrays created by > > zeros, this is a windows-32 only failure). > > > > David > > > > On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor > > > > wrote: > > > > On 18.11.2014 19:20, David Cournapeau wrote: > > > Hi, > > > > > > I have not followed closely the changes that happen in 1.9.1, > > but was > > > surprised by the following: > > > > > > x = np.zeros(12, "d") > > > assert x.flags.aligned # fails > > > > > > This is running numpy 1.9.1 built on windows with VS 2008. Is > it > > > expected that zeros may return a non-aligned array ? > > > > > > > what is the real alignment of the array? Are you on 32 bit or 64 > > bit? > > What is the alignment of doubles in windows (linux its 4 byte on > > 32 bit > > 8 byte on 64 bit (% special compiler flags)? > > print x.__array_interface__["data"] > > > > there are problems with complex types but doubles should be > > aligned even > > on 32 bit. > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Nov 18 14:05:59 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 18 Nov 2014 20:05:59 +0100 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: <546B8F5F.4000507@googlemail.com> <546B92A7.90707@googlemail.com> Message-ID: <546B9897.2060306@googlemail.com> 32 bit windows should not provide 16 byte alignment, at least it doesn't for me. That is typically a property of 64 bit OS. But that does not explain why normal double is not aligned for you, that only needs 4 bytes on i386 which even 32 bit OS should provide. I though I tested scipy on 32 bit linux which has very similar properties to win32, time to rerun the test. On 18.11.2014 19:56, David Cournapeau wrote: > > > On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor > > > wrote: > > < 1.9 lies about alignment it doesn't actually check for new arrays. > > > When I do the following on 1.8.1 with win 32 bits: > > x = np.zeros(12, "D") > print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # > always true > print x.aligned.flags # always true > > but on 1.9.1: > > x = np.zeros(12, "D") > print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # > always true > print x.aligned.flags # not always true > > I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this > case (unlikely to be a coincidence, as I try quite a few times with > different sizes). > > > > is the array aligned? > > On 18.11.2014 19:37, David Cournapeau wrote: > > Additional point: it seems to always return aligned data on 1.8.1 (same > > platform/compiler/everything). > > > > On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau > > >> wrote: > > > > It is on windows 32 bits, but I would need to make this work for > > complex (pair of double) as well. > > > > Is this a bug (I assumed array creation methods would always create > > aligned arrays for their type) ? Seems like quite a bit of code out > > there would assume this (scipy itself does for example). > > > > (the context is > 100 test failures on scipy 0.14.x on top of numpy > > 1.9., because f2py intent(inout) fails on work arrays created by > > zeros, this is a windows-32 only failure). > > > > David > > > > On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor > > > > >> wrote: > > > > On 18.11.2014 19:20, David Cournapeau wrote: > > > Hi, > > > > > > I have not followed closely the changes that happen in 1.9.1, > > but was > > > surprised by the following: > > > > > > x = np.zeros(12, "d") > > > assert x.flags.aligned # fails > > > > > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > > > expected that zeros may return a non-aligned array ? > > > > > > > what is the real alignment of the array? Are you on 32 bit or 64 > > bit? > > What is the alignment of doubles in windows (linux its 4 byte on > > 32 bit > > 8 byte on 64 bit (% special compiler flags)? > > print x.__array_interface__["data"] > > > > there are problems with complex types but doubles should be > > aligned even > > on 32 bit. > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Tue Nov 18 14:44:36 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 18 Nov 2014 19:44:36 +0000 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: <546B9897.2060306@googlemail.com> References: <546B8F5F.4000507@googlemail.com> <546B92A7.90707@googlemail.com> <546B9897.2060306@googlemail.com> Message-ID: On Tue, Nov 18, 2014 at 7:05 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > 32 bit windows should not provide 16 byte alignment, at least it doesn't > for me. That is typically a property of 64 bit OS. > > But that does not explain why normal double is not aligned for you, that > only needs 4 bytes on i386 which even 32 bit OS should provide. > Sorry for the confusion, doubles are aligned, only complex128 are not. But I see that on linux 32 bits, this is the same as on windows (zeros output not always aligned on "D" dtype), and yet I don't see the issues with f2py not being able to I am starting to think I got on the wrong track regarding the original issue I see with scipy 0.14.x on win 32 bits... I will create a separate ticket for that. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Nov 18 17:10:39 2014 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 19 Nov 2014 00:10:39 +0200 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: <546B8F5F.4000507@googlemail.com> <546B92A7.90707@googlemail.com> <546B9897.2060306@googlemail.com> Message-ID: 18.11.2014, 21:44, David Cournapeau kirjoitti: > On Tue, Nov 18, 2014 at 7:05 PM, Julian Taylor < > jtaylor.debian at googlemail.com> wrote: > >> 32 bit windows should not provide 16 byte alignment, at least it doesn't >> for me. That is typically a property of 64 bit OS. > >> But that does not explain why normal double is not aligned for you, that >> only needs 4 bytes on i386 which even 32 bit OS should provide. > > Sorry for the confusion, doubles are aligned, only complex128 are not. But > I see that on linux 32 bits, this is the same as on windows (zeros output > not always aligned on "D" dtype), and yet I don't see the issues with f2py > not being able to The scipy ticket is here, btw: https://github.com/scipy/scipy/issues/4168 The second question is whether F2py actually *needs* to check the dtype-size alignment, or is just something like sizeof(double) enough for Fortran compilers. Fortran compilers however apparently do generate code that crashes and burns if there's no alignment also on x86: https://github.com/scipy/scipy/pull/2698 All this is probably unspecified, as it's just F77 up out there. Apparently, everything has worked OK with the old Numpy behavior, or at least, nobody managed to pinpoint a crash on Win32 because of this? Can the F2py alignment checks be relaxed? Maybe it is enough to assume the Fortran compiler is happy with whatever alignment the system malloc() assures? If not, the second option is a bit nasty, since I'd believe many people have f2py code out there with complex inout arrays, and I think no-one uses special aligned allocators... -- Pauli Virtanen From geoffrey.megardon at gmail.com Wed Nov 19 11:06:58 2014 From: geoffrey.megardon at gmail.com (=?UTF-8?Q?Geoffrey_M=C3=A9gardon?=) Date: Wed, 19 Nov 2014 11:06:58 -0500 Subject: [Numpy-discussion] Problem with _dotblas.pyd when using Matplotlib for 3d plot Message-ID: ? Hi, I am on Windows 8.1, 64-bit, using the usual version of Anaconda-64-bit for windows. I just come from the mailing list of matplotlib: my PC is not able to do 3D plot with matplotlib. Python crashes completely if I try to plot something in 3D and to show it: "Python.exe has stopped working..." The main problem in resolving my issue is that the python traceback and the faulthandler package do not succeed in catching the errors. Only the Windows crash report allows us to see what happened before the crash. The windows report tells that the file _dotblas.pyd from numpy is likely responsible for the crash. So I tried a simple: import numpy; numpy.test() numpy.test() does not work! I dont get any traceback from python. It just crashes as before: "Python.exe has stopped working..." And again, the Windows report blames _dotblas.pyd :) Here the new Windows report for the numpy.test(), I highlighted the lines which blame _dotblas.pyd: Version=1 EventType=APPCRASH EventTime=130608138227608275 ReportType=2 Consent=1 UploadTime=130608138229728384 ReportIdentifier=216b16d2-6f5c-11e4-bec3-48d22435da2b IntegratorReportIdentifier=216b16d1-6f5c-11e4-bec3-48d22435da2b NsAppName=python.exe Response.BucketId=398b7eee350a0fd8a7a96f705d3488f6 Response.BucketTable=4 Response.LegacyBucketId=85979911964 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=python.exe Sig[1].Name=Application Version Sig[1].Value=0.0.0.0 Sig[2].Name=Application Timestamp Sig[2].Value=53b4679e *Sig[3].Name=Fault Module NameSig[3].Value=_dotblas.pyd* Sig[4].Name=Fault Module Version Sig[4].Value=0.0.0.0 Sig[5].Name=Fault Module Timestamp Sig[5].Value=545678cb Sig[6].Name=Exception Code Sig[6].Value=c000001d Sig[7].Name=Exception Offset Sig[7].Value=0000000000324022 DynamicSig[1].Name=OS Version DynamicSig[1].Value=6.3.9600.2.0.0.768.101 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=2057 DynamicSig[22].Name=Additional Information 1 DynamicSig[22].Value=f32f DynamicSig[23].Name=Additional Information 2 DynamicSig[23].Value=f32feb95f950f918532aa47b4372840e DynamicSig[24].Name=Additional Information 3 DynamicSig[24].Value=9882 DynamicSig[25].Name=Additional Information 4 DynamicSig[25].Value=98823b6c7f579b24e92112ab827fe4a1 UI[2]=C:\Anaconda\python.exe UI[3]=python.exe has stopped working UI[4]=Windows can check online for a solution to the problem. UI[5]=Check online for a solution and close the program UI[6]=Check online for a solution later and close the program UI[7]=Close the program LoadedModule[0]=C:\Anaconda\python.exe LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll LoadedModule[2]=C:\WINDOWS\system32\KERNEL32.DLL LoadedModule[3]=C:\WINDOWS\system32\KERNELBASE.dll LoadedModule[4]=C:\Anaconda\python27.dll LoadedModule[5]=C:\WINDOWS\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\MSVCR90.dll LoadedModule[6]=C:\WINDOWS\system32\USER32.dll LoadedModule[7]=C:\WINDOWS\system32\ADVAPI32.dll LoadedModule[8]=C:\WINDOWS\system32\SHELL32.dll LoadedModule[9]=C:\WINDOWS\system32\GDI32.dll LoadedModule[10]=C:\WINDOWS\system32\msvcrt.dll LoadedModule[11]=C:\WINDOWS\SYSTEM32\sechost.dll LoadedModule[12]=C:\WINDOWS\system32\RPCRT4.dll LoadedModule[13]=C:\WINDOWS\SYSTEM32\combase.dll LoadedModule[14]=C:\WINDOWS\system32\SHLWAPI.dll LoadedModule[15]=C:\WINDOWS\system32\IMM32.DLL LoadedModule[16]=C:\WINDOWS\system32\MSCTF.dll LoadedModule[17]=C:\Anaconda\DLLs\_socket.pyd LoadedModule[18]=C:\WINDOWS\system32\WS2_32.dll LoadedModule[19]=C:\WINDOWS\system32\NSI.dll LoadedModule[20]=C:\Anaconda\DLLs\_ssl.pyd LoadedModule[21]=C:\Anaconda\DLLs\_ctypes.pyd LoadedModule[22]=C:\WINDOWS\system32\ole32.dll LoadedModule[23]=C:\WINDOWS\system32\OLEAUT32.dll LoadedModule[24]=C:\Anaconda\DLLs\_hashlib.pyd LoadedModule[25]=C:\WINDOWS\SYSTEM32\CRYPTSP.dll LoadedModule[26]=C:\WINDOWS\system32\rsaenh.dll LoadedModule[27]=C:\WINDOWS\SYSTEM32\bcrypt.dll LoadedModule[28]=C:\WINDOWS\SYSTEM32\CRYPTBASE.dll LoadedModule[29]=C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll LoadedModule[30]=C:\Users\User\AppData\Roaming\Python-Eggs\faulthandler-2.4-py2.7-win-amd64.egg-tmp\faulthandler.pyd LoadedModule[31]=C:\Anaconda\lib\site-packages\numpy\core\multiarray.pyd LoadedModule[32]=C:\Anaconda\lib\site-packages\numpy\core\umath.pyd LoadedModule[33]=C:\Anaconda\lib\site-packages\numpy\core\_dotblas.pyd LoadedModule[34]=C:\Anaconda\lib\site-packages\numpy\core\libiomp5md.dll LoadedModule[35]=C:\Anaconda\lib\site-packages\numpy\core\scalarmath.pyd LoadedModule[36]=C:\Anaconda\lib\site-packages\numpy\lib\_compiled_base.pyd LoadedModule[37]=C:\Anaconda\lib\site-packages\numpy\linalg\lapack_lite.pyd LoadedModule[38]=C:\Anaconda\lib\site-packages\numpy\linalg\_umath_linalg.pyd LoadedModule[39]=C:\Anaconda\lib\site-packages\numpy\fft\fftpack_lite.pyd LoadedModule[40]=C:\Anaconda\lib\site-packages\numpy\random\mtrand.pyd LoadedModule[41]=C:\Anaconda\DLLs\_multiprocessing.pyd State[0].Key=Transport.DoneStage1 State[0].Value=1 FriendlyEventName=Stopped working ConsentKey=APPCRASH AppName=python.exe AppPath=C:\Anaconda\python.exe NsPartner=windows NsGroup=windows8 ApplicationIdentity=5B036AF1EC2E20F320DBF28D119DE93D Any idea, of what I can do? I try to reinstall anaconda, I also try conda remove numpy and conda install numpy. ?That does not seem to change anything. ? -- -- MEGARDON Geoffrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 19 16:56:18 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 19 Nov 2014 21:56:18 +0000 Subject: [Numpy-discussion] Problem with _dotblas.pyd when using Matplotlib for 3d plot In-Reply-To: References: Message-ID: Some things that might be worth trying: - run the test suite in verbose mode (numpy.test(verbose=2)) - this should let you narrow down which test is causing the crash. - instead of using anaconda, try installing python from python.org and numpy from the http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 . This won't give nearly as usable an environment as anaconda does, but if anaconda doesn't work and the sourceforge installer does, then that would suggest the problem is something broken in the anaconda build scripts rather than numpy itself. -n On 19 Nov 2014 16:07, "Geoffrey M?gardon" wrote: > ? > Hi, > > I am on Windows 8.1, 64-bit, using the usual version of Anaconda-64-bit > for windows. > > I just come from the mailing list of matplotlib: my PC is not able to do > 3D plot with matplotlib. > Python crashes completely if I try to plot something in 3D and to show it: > "Python.exe has stopped working..." > > The main problem in resolving my issue is that the python traceback and > the faulthandler package do not succeed in catching the errors. > Only the Windows crash report allows us to see what happened before the > crash. > The windows report tells that the file _dotblas.pyd from numpy is likely > responsible for the crash. > > So I tried a simple: > import numpy; numpy.test() > > numpy.test() does not work! I dont get any traceback from python. It just > crashes as before: "Python.exe has stopped working..." > > And again, the Windows report blames _dotblas.pyd :) > > Here the new Windows report for the numpy.test(), I highlighted the lines > which blame _dotblas.pyd: > Version=1 > EventType=APPCRASH > EventTime=130608138227608275 > ReportType=2 > Consent=1 > UploadTime=130608138229728384 > ReportIdentifier=216b16d2-6f5c-11e4-bec3-48d22435da2b > IntegratorReportIdentifier=216b16d1-6f5c-11e4-bec3-48d22435da2b > NsAppName=python.exe > Response.BucketId=398b7eee350a0fd8a7a96f705d3488f6 > Response.BucketTable=4 > Response.LegacyBucketId=85979911964 > Response.type=4 > Sig[0].Name=Application Name > Sig[0].Value=python.exe > Sig[1].Name=Application Version > Sig[1].Value=0.0.0.0 > Sig[2].Name=Application Timestamp > Sig[2].Value=53b4679e > > *Sig[3].Name=Fault Module NameSig[3].Value=_dotblas.pyd* > Sig[4].Name=Fault Module Version > Sig[4].Value=0.0.0.0 > Sig[5].Name=Fault Module Timestamp > Sig[5].Value=545678cb > Sig[6].Name=Exception Code > Sig[6].Value=c000001d > Sig[7].Name=Exception Offset > Sig[7].Value=0000000000324022 > DynamicSig[1].Name=OS Version > DynamicSig[1].Value=6.3.9600.2.0.0.768.101 > DynamicSig[2].Name=Locale ID > DynamicSig[2].Value=2057 > DynamicSig[22].Name=Additional Information 1 > DynamicSig[22].Value=f32f > DynamicSig[23].Name=Additional Information 2 > DynamicSig[23].Value=f32feb95f950f918532aa47b4372840e > DynamicSig[24].Name=Additional Information 3 > DynamicSig[24].Value=9882 > DynamicSig[25].Name=Additional Information 4 > DynamicSig[25].Value=98823b6c7f579b24e92112ab827fe4a1 > UI[2]=C:\Anaconda\python.exe > UI[3]=python.exe has stopped working > UI[4]=Windows can check online for a solution to the problem. > UI[5]=Check online for a solution and close the program > UI[6]=Check online for a solution later and close the program > UI[7]=Close the program > LoadedModule[0]=C:\Anaconda\python.exe > LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll > LoadedModule[2]=C:\WINDOWS\system32\KERNEL32.DLL > LoadedModule[3]=C:\WINDOWS\system32\KERNELBASE.dll > LoadedModule[4]=C:\Anaconda\python27.dll > > LoadedModule[5]=C:\WINDOWS\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\MSVCR90.dll > LoadedModule[6]=C:\WINDOWS\system32\USER32.dll > LoadedModule[7]=C:\WINDOWS\system32\ADVAPI32.dll > LoadedModule[8]=C:\WINDOWS\system32\SHELL32.dll > LoadedModule[9]=C:\WINDOWS\system32\GDI32.dll > LoadedModule[10]=C:\WINDOWS\system32\msvcrt.dll > LoadedModule[11]=C:\WINDOWS\SYSTEM32\sechost.dll > LoadedModule[12]=C:\WINDOWS\system32\RPCRT4.dll > LoadedModule[13]=C:\WINDOWS\SYSTEM32\combase.dll > LoadedModule[14]=C:\WINDOWS\system32\SHLWAPI.dll > LoadedModule[15]=C:\WINDOWS\system32\IMM32.DLL > LoadedModule[16]=C:\WINDOWS\system32\MSCTF.dll > LoadedModule[17]=C:\Anaconda\DLLs\_socket.pyd > LoadedModule[18]=C:\WINDOWS\system32\WS2_32.dll > LoadedModule[19]=C:\WINDOWS\system32\NSI.dll > LoadedModule[20]=C:\Anaconda\DLLs\_ssl.pyd > LoadedModule[21]=C:\Anaconda\DLLs\_ctypes.pyd > LoadedModule[22]=C:\WINDOWS\system32\ole32.dll > LoadedModule[23]=C:\WINDOWS\system32\OLEAUT32.dll > LoadedModule[24]=C:\Anaconda\DLLs\_hashlib.pyd > LoadedModule[25]=C:\WINDOWS\SYSTEM32\CRYPTSP.dll > LoadedModule[26]=C:\WINDOWS\system32\rsaenh.dll > LoadedModule[27]=C:\WINDOWS\SYSTEM32\bcrypt.dll > LoadedModule[28]=C:\WINDOWS\SYSTEM32\CRYPTBASE.dll > LoadedModule[29]=C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll > > LoadedModule[30]=C:\Users\User\AppData\Roaming\Python-Eggs\faulthandler-2.4-py2.7-win-amd64.egg-tmp\faulthandler.pyd > LoadedModule[31]=C:\Anaconda\lib\site-packages\numpy\core\multiarray.pyd > LoadedModule[32]=C:\Anaconda\lib\site-packages\numpy\core\umath.pyd > LoadedModule[33]=C:\Anaconda\lib\site-packages\numpy\core\_dotblas.pyd > LoadedModule[34]=C:\Anaconda\lib\site-packages\numpy\core\libiomp5md.dll > LoadedModule[35]=C:\Anaconda\lib\site-packages\numpy\core\scalarmath.pyd > LoadedModule[36]=C:\Anaconda\lib\site-packages\numpy\lib\_compiled_base.pyd > LoadedModule[37]=C:\Anaconda\lib\site-packages\numpy\linalg\lapack_lite.pyd > > LoadedModule[38]=C:\Anaconda\lib\site-packages\numpy\linalg\_umath_linalg.pyd > LoadedModule[39]=C:\Anaconda\lib\site-packages\numpy\fft\fftpack_lite.pyd > LoadedModule[40]=C:\Anaconda\lib\site-packages\numpy\random\mtrand.pyd > LoadedModule[41]=C:\Anaconda\DLLs\_multiprocessing.pyd > State[0].Key=Transport.DoneStage1 > State[0].Value=1 > FriendlyEventName=Stopped working > ConsentKey=APPCRASH > AppName=python.exe > AppPath=C:\Anaconda\python.exe > NsPartner=windows > NsGroup=windows8 > ApplicationIdentity=5B036AF1EC2E20F320DBF28D119DE93D > > Any idea, of what I can do? > I try to reinstall anaconda, > I also try conda remove numpy and conda install numpy. > ?That does not seem to change anything. > ? > > -- > -- > MEGARDON Geoffrey > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoffrey.megardon at gmail.com Wed Nov 19 17:03:10 2014 From: geoffrey.megardon at gmail.com (=?iso-8859-15?Q?M=E9gardon_Geoffrey?=) Date: Wed, 19 Nov 2014 17:03:10 -0500 Subject: [Numpy-discussion] Problem with _dotblas.pyd when using Matplotlib for 3d plot In-Reply-To: References: Message-ID: Hi, In verbose mode, it stops at this line: test_blasdot.test_dot_3args ... Ok, I will try with numpy and python :/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 19 18:21:14 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 19 Nov 2014 16:21:14 -0700 Subject: [Numpy-discussion] Problem with _dotblas.pyd when using Matplotlib for 3d plot In-Reply-To: References: Message-ID: On Wed, Nov 19, 2014 at 3:03 PM, M?gardon Geoffrey < geoffrey.megardon at gmail.com> wrote: > Hi, > > In verbose mode, it stops at this line: > test_blasdot.test_dot_3args ... > > Ok, I will try with numpy and python :/ > Might want to file a bug report with Continuum Analytics. Plain old numpy will not be using _dotblas, that requires ATLAS or MKL for blas support. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Wed Nov 19 21:11:49 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 20 Nov 2014 03:11:49 +0100 Subject: [Numpy-discussion] Numpy 1.9.1, zeros and alignement In-Reply-To: References: <546B8F5F.4000507@googlemail.com> <546B92A7.90707@googlemail.com> <546B9897.2060306@googlemail.com> Message-ID: On 18/11/14 23:10, Pauli Virtanen wrote: > The second question is whether F2py actually *needs* to check the > dtype-size alignment, or is just something like sizeof(double) enough > for Fortran compilers. The Fortran standard does not specify an ABI so this is likely compiler dependent. The only safe way of calling Fortran from C is through a proxy Fortran routine exported to C using Fortran 2003 ISO C bindings. That is what f2py should do behind the scenes now that the Fortran 77 limitation is lifted from SciPy. With this method we can just pass an arbitrary C pointer to Fortran and construct a Fortran array pointer with c_f_function. The Fortran compiler will make an aligned copy of the array referenced by the pointer if required. No special coding will be required to ensure correct alignment. Sturla From mwojc at p.lodz.pl Thu Nov 20 12:47:41 2014 From: mwojc at p.lodz.pl (Marek Wojciechowski) Date: Thu, 20 Nov 2014 18:47:41 +0100 Subject: [Numpy-discussion] return type from ufuncs Message-ID: <1743805.dMy5v138kg@think> Hi! I wrote a simple subclass of np.ndarray and now i do call np.sum() on it. I expected that the result will be a python float (or int) just like when summing up regular arrays. Instead i obtain the (scalar) view of my subclass. How can i change this behavior? I tried writing __array_wrap__ method like this: def __array_wrap__(self, out_arr, context=None): selfv = self.view(np.ndarray) return np.ndarray.__array_wrap__(selfv, out_arr, context) but this just returns np.ndarray type and not float. Regards, -- Marek Wojciechowski From njs at pobox.com Thu Nov 20 19:09:51 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 21 Nov 2014 00:09:51 +0000 Subject: [Numpy-discussion] return type from ufuncs In-Reply-To: <1743805.dMy5v138kg@think> References: <1743805.dMy5v138kg@think> Message-ID: On Thu, Nov 20, 2014 at 5:47 PM, Marek Wojciechowski wrote: > Hi! > > I wrote a simple subclass of np.ndarray and now i do call np.sum() on it. I > expected that the result will be a python float (or int) just like when summing > up regular arrays. Instead i obtain the (scalar) view of my subclass. How can > i change this behavior? I tried writing __array_wrap__ method like this: > > def __array_wrap__(self, out_arr, context=None): > selfv = self.view(np.ndarray) > return np.ndarray.__array_wrap__(selfv, out_arr, context) > > but this just returns np.ndarray type and not float. There isn't really any reason to call __array_wrap__ here. Just compute whatever value you want and return it. If you want a float then return that :-). (Something like if selfv.ndim == 0: return selfv[()] ) As a bit of advice, if you can possibly avoid subclassing ndarray, then you probably should. There are a lot of painful little quirks that you'll run into. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From mwojc at p.lodz.pl Fri Nov 21 04:49:39 2014 From: mwojc at p.lodz.pl (Marek Wojciechowski) Date: Fri, 21 Nov 2014 10:49:39 +0100 Subject: [Numpy-discussion] return type from ufuncs In-Reply-To: References: <1743805.dMy5v138kg@think> Message-ID: <2111060.vPxnFLLBTU@think> Dnia pi?tek, 21 listopada 2014 00:09:51 Nathaniel Smith pisze: > On Thu, Nov 20, 2014 at 5:47 PM, Marek Wojciechowski wrote: > > Hi! > > > > I wrote a simple subclass of np.ndarray and now i do call np.sum() on it. > > I > > expected that the result will be a python float (or int) just like when > > summing up regular arrays. Instead i obtain the (scalar) view of my > > subclass. How can i change this behavior? I tried writing __array_wrap__ > > method like this:> > > def __array_wrap__(self, out_arr, context=None): > > selfv = self.view(np.ndarray) > > return np.ndarray.__array_wrap__(selfv, out_arr, context) > > > > but this just returns np.ndarray type and not float. > > There isn't really any reason to call __array_wrap__ here. Just > compute whatever value you want and return it. If you want a float > then return that :-). (Something like > if selfv.ndim == 0: > return selfv[()] > ) > > As a bit of advice, if you can possibly avoid subclassing ndarray, > then you probably should. There are a lot of painful little quirks > that you'll run into. Thanks for the answer. I thought similar, but the 'context' argument confused me a bit. I do not know what for it is here. Thanks also for the (reasonable) advice. Subclassing was just the fastest way to implement what i wanted to do, because of all these methods being on place. However i do see now, that that was not necessarily the best choice... -- Marek Wojciechowski From sturla.molden at gmail.com Fri Nov 21 22:04:40 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 22 Nov 2014 04:04:40 +0100 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: <357693592437876378.268056sturla.molden-gmail.com@news.gmane.org> Message-ID: On 18/11/14 04:21, Robert McGibbon wrote: > The np.ndarray constructor > takes > a strides argument argument, and a buffer. Is it not sufficiently flexible? > > -Robert AFAIK the buffer argument is not a memory address but an object exporting the old buffer protocol. We can abuse the __array_interface__ to do this though, but I prefer the C API functions. Wrapping a C pointer with __array_interface__ then becomes something like this (not tested, but should work): import numpy as np cdef class wrapper_array(object): cdef: object readonly __array_interface__ def __init__(wrapper_array self, addr, shape, dtype, order, strides, offset): if strides is None: if order == 'C': strides = None else: strides = _get_fortran_strides(shape, dtype) self.__array_interface__ = dict( data = (addr + offset, False), descr = dtype.descr, shape = shape, strides = strides, typestr = dtype.str, version = 3, ) cdef object _get_fortran_strides(shape, dtype): strides = tuple(dtype.itemsize * np.cumprod((1,) + shape[:-1])) return strides def wrap_pointer(void *addr, shape, dtype, order, strides, offset): """Wraps a C pointer with an ndarray""" return np.asarray(wrapper_array( addr, shape, dtype, order, strides, offset)) https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/array.py Sturla From ram at rachum.com Sat Nov 22 14:39:40 2014 From: ram at rachum.com (Ram Rachum) Date: Sat, 22 Nov 2014 21:39:40 +0200 Subject: [Numpy-discussion] ANN: First release of combi, a combinatorics package for Python Message-ID: Hi everyone, Just a quick announcement: Last week I made the first release of an open-source package called Combi. It's a package for doing combinatorics in Python. The pitch: Combi lets you explore spaces of permutations and combinations as if they were Python sequences, but without generating all the permutations/combinations in advance. It lets you specify a lot of special conditions on these spaces. It also provides a few more classes that might be useful in combinatorics programming. Information about it: https://pypi.python.org/pypi/combi I hope this package will be useful to people here. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyspiros at gmail.com Sat Nov 22 19:57:05 2014 From: andyspiros at gmail.com (Andrea Arteaga) Date: Sun, 23 Nov 2014 01:57:05 +0100 Subject: [Numpy-discussion] Initializing array from buffer In-Reply-To: References: <357693592437876378.268056sturla.molden-gmail.com@news.gmane.org> Message-ID: Thanks everybody for suggesting many different ways to achieve this result. While all of them seem valid methods, I decided to use the constructor, as proposed by Robert: > The np.ndarray constructor takes a strides argument argument, and a buffer. I could easily do this from within C++ in a clean way and without making use of Numpy-specific C code. It sounds just a bit redundant to me the fact that you have to provide the strides and the shape, even if the buffer object already contains this information, but I suppose this is done so to support an older buffer protocol, where only 1D arrays could be defined. I have it working. Thanks once more. All the best Andrea 2014-11-22 4:04 GMT+01:00 Sturla Molden : > On 18/11/14 04:21, Robert McGibbon wrote: > > The np.ndarray constructor > > > takes > > a strides argument argument, and a buffer. Is it not sufficiently > flexible? > > > > -Robert > > AFAIK the buffer argument is not a memory address but an object > exporting the old buffer protocol. We can abuse the __array_interface__ > to do this though, but I prefer the C API functions. Wrapping a C > pointer with __array_interface__ then becomes something like this (not > tested, but should work): > > > import numpy as np > > cdef class wrapper_array(object): > > cdef: > object readonly __array_interface__ > > def __init__(wrapper_array self, addr, shape, dtype, > order, strides, offset): > if strides is None: > if order == 'C': > strides = None > else: > strides = _get_fortran_strides(shape, dtype) > self.__array_interface__ = dict( > data = (addr + offset, False), > descr = dtype.descr, > shape = shape, > strides = strides, > typestr = dtype.str, > version = 3, > ) > > cdef object _get_fortran_strides(shape, dtype): > strides = tuple(dtype.itemsize * np.cumprod((1,) + shape[:-1])) > return strides > > def wrap_pointer(void *addr, shape, dtype, order, strides, offset): > """Wraps a C pointer with an ndarray""" > return np.asarray(wrapper_array( addr, shape, > dtype, order, strides, offset)) > > > > https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/array.py > > > > Sturla > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Nov 23 18:13:03 2014 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 24 Nov 2014 01:13:03 +0200 Subject: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release Message-ID: <547269FF.7010905@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, We have finally finished preparing the Scipy 0.15.0 beta 1 release. Please try it and report any issues on the scipy-dev mailing list, and/or on Github. If no surprises turn up, the final release is planned on Dec 20 in three weeks. Source tarballs and full release notes are available at https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ Binary installers should also be up soon. Best regards, Pauli Virtanen - -------------------------------------------- SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or greater. New features ============ Linear Programming Interface - - ---------------------------- The new function ``scipy.optimize.linprog`` provides a generic linear programming similar to the way ``scipy.optimize.minimize`` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported,allowing the user to monitor the progress of the algorithm. Differential_evolution, a global optimizer - - ------------------------------------------ A new ``differential_evolution`` function is available in the ``scipy.optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques. ``scipy.signal`` improvements - - ----------------------------- The function ``max_len_seq`` was added, which computes a Maximum Length Sequence (MLS) signal. ``scipy.integrate`` improvements - - -------------------------------- It is now possible to use ``scipy.integrate`` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance. ``scipy.linalg`` improvements - - ----------------------------- Add function ``orthogonal_procrustes`` for solving the procrustes linear algebra problem. ``scipy.sparse`` improvements - - ----------------------------- ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its main input. ``scipy.special`` improvements - - ------------------------------ Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``. New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added. ``scipy.sparse.csgraph`` improvements - - ------------------------------------- Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added. ``scipy.stats`` improvements - - ---------------------------- Added a Dirichlet distribution as multivariate distribution. The new function ``scipy.stats.median_test`` computes Mood's median test. The new function ``scipy.stats.combine_pvalues`` implements Fisher's and Stouffer's methods for combining p-values. ``scipy.stats.describe`` returns a namedtuple rather than a tuple, allowing users to access results by index or by name. Deprecated features =================== The ``scipy.weave`` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: `https://github.com/scipy/weave`_. It is a pure Python package, and can easily be installed with ``pip install weave``. ``scipy.special.bessel_diff_formula`` is deprecated. It is a private function, and therefore will be removed from the public API in a following release. Backwards incompatible changes ============================== scipy.ndimage - - ------------- The functions ``scipy.ndimage.minimum_positions``, ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return positions as ints instead of floats. scipy.integrate - - --------------- The format of banded Jacobians in ``scipy.integrate.ode`` solvers is changed. Note that the previous documentation of this feature was erroneous. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRyaf8ACgkQ6BQxb7O0pWC7XQCeNtdJD4ZNDXvFeNFs7N3KjQn6 8QkAoK3pFmhMrTwCrgkusl+fRNMboN2r =WSpM -----END PGP SIGNATURE----- From matthew.brett at gmail.com Mon Nov 24 21:26:16 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 24 Nov 2014 18:26:16 -0800 Subject: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release In-Reply-To: <547269FF.7010905@iki.fi> References: <547269FF.7010905@iki.fi> Message-ID: Hi, On Sun, Nov 23, 2014 at 3:13 PM, Pauli Virtanen wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dear all, > > We have finally finished preparing the Scipy 0.15.0 beta 1 release. > Please try it and report any issues on the scipy-dev mailing list, > and/or on Github. > > If no surprises turn up, the final release is planned on Dec 20 in > three weeks. > > Source tarballs and full release notes are available at > https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ > Binary installers should also be up soon. > > Best regards, > Pauli Virtanen > > > - -------------------------------------------- > > SciPy 0.15.0 is the culmination of 6 months of hard work. It contains > several new features, numerous bug-fixes, improved test coverage and > better documentation. There have been a number of deprecations and > API changes in this release, which are documented below. All users > are encouraged to upgrade to this release, as there are a large number > of bug-fixes and optimizations. Moreover, our development attention > will now shift to bug-fix releases on the 0.16.x branch, and on adding > new features on the master branch. > > This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or > greater. > > > New features > ============ > > Linear Programming Interface > - - ---------------------------- > > The new function ``scipy.optimize.linprog`` provides a generic > linear programming similar to the way ``scipy.optimize.minimize`` > provides a generic interface to nonlinear programming optimizers. > Currently the only method supported is *simplex* which provides > a two-phase, dense-matrix-based simplex algorithm. Callbacks > functions are supported,allowing the user to monitor the progress > of the algorithm. > > Differential_evolution, a global optimizer > - - ------------------------------------------ > > A new ``differential_evolution`` function is available in the > ``scipy.optimize`` > module. Differential Evolution is an algorithm used for finding the > global > minimum of multivariate functions. It is stochastic in nature (does > not use > gradient methods), and can search large areas of candidate space, but > often > requires larger numbers of function evaluations than conventional gradient > based techniques. > > ``scipy.signal`` improvements > - - ----------------------------- > > The function ``max_len_seq`` was added, which computes a Maximum > Length Sequence (MLS) signal. > > ``scipy.integrate`` improvements > - - -------------------------------- > > It is now possible to use ``scipy.integrate`` routines to integrate > multivariate ctypes functions, thus avoiding callbacks to Python and > providing better performance. > > ``scipy.linalg`` improvements > - - ----------------------------- > > Add function ``orthogonal_procrustes`` for solving the procrustes > linear algebra problem. > > ``scipy.sparse`` improvements > - - ----------------------------- > > ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its > main input. > > ``scipy.special`` improvements > - - ------------------------------ > > Values of ellipsoidal harmonic (i.e. Lame) functions and associated > normalization constants can be now computed using ``ellip_harm``, > ``ellip_harm_2``, and ``ellip_normal``. > > New convenience functions ``entr``, ``rel_entr`` ``kl_div``, > ``huber``, and ``pseudo_huber`` were added. > > ``scipy.sparse.csgraph`` improvements > - - ------------------------------------- > > Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` > for computing reorderings of sparse graphs were added. > > ``scipy.stats`` improvements > - - ---------------------------- > > Added a Dirichlet distribution as multivariate distribution. > > The new function ``scipy.stats.median_test`` computes Mood's median test. > > The new function ``scipy.stats.combine_pvalues`` implements Fisher's > and Stouffer's methods for combining p-values. > > ``scipy.stats.describe`` returns a namedtuple rather than a tuple, > allowing > users to access results by index or by name. > > Deprecated features > =================== > > The ``scipy.weave`` module is deprecated. It was the only module > never ported > to Python 3.x, and is not recommended to be used for new code - use Cython > instead. In order to support existing code, ``scipy.weave`` has been > packaged > separately: `https://github.com/scipy/weave`_. It is a pure Python > package, and > can easily be installed with ``pip install weave``. > > ``scipy.special.bessel_diff_formula`` is deprecated. It is a private > function, > and therefore will be removed from the public API in a following release. > > > Backwards incompatible changes > ============================== > > scipy.ndimage > - - ------------- > > The functions ``scipy.ndimage.minimum_positions``, > ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return > positions as ints instead of floats. > > scipy.integrate > - - --------------- > > The format of banded Jacobians in ``scipy.integrate.ode`` solvers is > changed. Note that the previous documentation of this feature was > erroneous. I ran against current scipy stack packages, and get these errors: https://travis-ci.org/MacPython/scipy-stack-osx-testing/jobs/42022929#L324 ====================================================================== ERROR: test_netcdf.test_byte_gatts ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py", line 268, in test_byte_gatts f = netcdf_file(filename, 'w') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py", line 226, in __init__ self.fp = open(self.filename, '%sb' % omode) IOError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/g_byte_atts.nc' ====================================================================== ERROR: test_netcdf.test_open_append ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py", line 284, in test_open_append f = netcdf_file(filename, 'w') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py", line 226, in __init__ self.fp = open(self.filename, '%sb' % omode) IOError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/append_dat.nc' This turns out to be because the netcdf tests are trying to write into the scipy/io/tests/data directory: https://github.com/scipy/scipy/pull/4198 Otherwise, it looks clean. I uploaded the OSX wheels to sourceforge. Cheers, Matthew From cournape at gmail.com Tue Nov 25 07:53:03 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Nov 2014 12:53:03 +0000 Subject: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release In-Reply-To: <547269FF.7010905@iki.fi> References: <547269FF.7010905@iki.fi> Message-ID: Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker (the issue arises on scipy master as well as 0.14.1) ? On Sun, Nov 23, 2014 at 11:13 PM, Pauli Virtanen wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dear all, > > We have finally finished preparing the Scipy 0.15.0 beta 1 release. > Please try it and report any issues on the scipy-dev mailing list, > and/or on Github. > > If no surprises turn up, the final release is planned on Dec 20 in > three weeks. > > Source tarballs and full release notes are available at > https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ > Binary installers should also be up soon. > > Best regards, > Pauli Virtanen > > > - -------------------------------------------- > > SciPy 0.15.0 is the culmination of 6 months of hard work. It contains > several new features, numerous bug-fixes, improved test coverage and > better documentation. There have been a number of deprecations and > API changes in this release, which are documented below. All users > are encouraged to upgrade to this release, as there are a large number > of bug-fixes and optimizations. Moreover, our development attention > will now shift to bug-fix releases on the 0.16.x branch, and on adding > new features on the master branch. > > This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or > greater. > > > New features > ============ > > Linear Programming Interface > - - ---------------------------- > > The new function ``scipy.optimize.linprog`` provides a generic > linear programming similar to the way ``scipy.optimize.minimize`` > provides a generic interface to nonlinear programming optimizers. > Currently the only method supported is *simplex* which provides > a two-phase, dense-matrix-based simplex algorithm. Callbacks > functions are supported,allowing the user to monitor the progress > of the algorithm. > > Differential_evolution, a global optimizer > - - ------------------------------------------ > > A new ``differential_evolution`` function is available in the > ``scipy.optimize`` > module. Differential Evolution is an algorithm used for finding the > global > minimum of multivariate functions. It is stochastic in nature (does > not use > gradient methods), and can search large areas of candidate space, but > often > requires larger numbers of function evaluations than conventional gradient > based techniques. > > ``scipy.signal`` improvements > - - ----------------------------- > > The function ``max_len_seq`` was added, which computes a Maximum > Length Sequence (MLS) signal. > > ``scipy.integrate`` improvements > - - -------------------------------- > > It is now possible to use ``scipy.integrate`` routines to integrate > multivariate ctypes functions, thus avoiding callbacks to Python and > providing better performance. > > ``scipy.linalg`` improvements > - - ----------------------------- > > Add function ``orthogonal_procrustes`` for solving the procrustes > linear algebra problem. > > ``scipy.sparse`` improvements > - - ----------------------------- > > ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its > main input. > > ``scipy.special`` improvements > - - ------------------------------ > > Values of ellipsoidal harmonic (i.e. Lame) functions and associated > normalization constants can be now computed using ``ellip_harm``, > ``ellip_harm_2``, and ``ellip_normal``. > > New convenience functions ``entr``, ``rel_entr`` ``kl_div``, > ``huber``, and ``pseudo_huber`` were added. > > ``scipy.sparse.csgraph`` improvements > - - ------------------------------------- > > Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` > for computing reorderings of sparse graphs were added. > > ``scipy.stats`` improvements > - - ---------------------------- > > Added a Dirichlet distribution as multivariate distribution. > > The new function ``scipy.stats.median_test`` computes Mood's median test. > > The new function ``scipy.stats.combine_pvalues`` implements Fisher's > and Stouffer's methods for combining p-values. > > ``scipy.stats.describe`` returns a namedtuple rather than a tuple, > allowing > users to access results by index or by name. > > Deprecated features > =================== > > The ``scipy.weave`` module is deprecated. It was the only module > never ported > to Python 3.x, and is not recommended to be used for new code - use Cython > instead. In order to support existing code, ``scipy.weave`` has been > packaged > separately: `https://github.com/scipy/weave`_. It is a pure Python > package, and > can easily be installed with ``pip install weave``. > > ``scipy.special.bessel_diff_formula`` is deprecated. It is a private > function, > and therefore will be removed from the public API in a following release. > > > Backwards incompatible changes > ============================== > > scipy.ndimage > - - ------------- > > The functions ``scipy.ndimage.minimum_positions``, > ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return > positions as ints instead of floats. > > scipy.integrate > - - --------------- > > The format of banded Jacobians in ``scipy.integrate.ode`` solvers is > changed. Note that the previous documentation of this feature was > erroneous. > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iEYEARECAAYFAlRyaf8ACgkQ6BQxb7O0pWC7XQCeNtdJD4ZNDXvFeNFs7N3KjQn6 > 8QkAoK3pFmhMrTwCrgkusl+fRNMboN2r > =WSpM > -----END PGP SIGNATURE----- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Nov 25 13:10:55 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 25 Nov 2014 18:10:55 +0000 (UTC) Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release References: <547269FF.7010905@iki.fi> Message-ID: <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> David Cournapeau wrote: > Shall we consider href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168 > to be a > blocker (the issue arises on scipy master as well as 0.14.1) ? > It is really bad, but does anyone know what is really going on? Which changes to NumPy set this off? Sturla From cournape at gmail.com Tue Nov 25 13:33:25 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Nov 2014 18:33:25 +0000 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden wrote: > David Cournapeau wrote: > > Shall we consider > href="https://github.com/scipy/scipy/issues/4168"> > https://github.com/scipy/scipy/issues/4168 > > to be a > > blocker (the issue arises on scipy master as well as 0.14.1) ? > > > > It is really bad, but does anyone know what is really going on? > Yes, it is in the bug report. David > > Which changes to NumPy set this off? > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 25 13:39:28 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 25 Nov 2014 18:39:28 +0000 Subject: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> Message-ID: On Tue, Nov 25, 2014 at 12:53 PM, David Cournapeau wrote: > Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker > (the issue arises on scipy master as well as 0.14.1) ? Do you think there's anything scipy can do about it? It looks like a pure numpy bug to me. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Tue Nov 25 14:14:46 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 25 Nov 2014 19:14:46 +0000 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Scipy 0.15.0 beta 1 release In-Reply-To: <1792165665438634567.990743sturla.molden-gmail.com@news.gmane.org> References: <547269FF.7010905@iki.fi> <1792165665438634567.990743sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Nov 25, 2014 at 7:07 PM, Sturla Molden wrote: > Nathaniel Smith wrote: >> On Tue, Nov 25, 2014 at 12:53 PM, David Cournapeau wrote: >>> Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker >>> (the issue arises on scipy master as well as 0.14.1) ? >> >> Do you think there's anything scipy can do about it? It looks like a >> pure numpy bug to me. > > I think we should assume it is not a bug in NumPy. There is nothing that > dictates that an ndarray's buffer must be aligned. It can just as well be > an unaligned view or wrap an external buffer. An ndarray is valud with any > alignment, but it does not mean it can be used with intent(inout) in f2py. > > Then by deduction if it is not a NumPy bug it must be a SciPy bug, which > means the bug must be the unprotected use of intent(inout) in the f2py > wrapper. I would think it is the responsibility of SciPy to ensure that > whatever is passed with intent(inout) in SciPy is properly aligned. In general you have a point, but in this case the reason the array is "unaligned" is that numpy is being overly strict about its definition of "aligned". In fact, it's so strict that numpy itself doesn't provide any standard and reliable way to *create* an aligned array by this definition. (I guess scipy could create an overallocated copy and then take a slice at the right offset, but asking scipy to use such hacks to work around our bugs is clearly wrong.) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From pav at iki.fi Tue Nov 25 14:50:43 2014 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 25 Nov 2014 21:50:43 +0200 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> <1792165665438634567.990743sturla.molden-gmail.com@news.gmane.org> Message-ID: 25.11.2014, 21:14, Nathaniel Smith kirjoitti: [clip] > (I guess scipy could create an overallocated copy and then take a > slice at the right offset, but asking scipy to use such hacks to work > around our bugs is clearly wrong.) Note that the issue is not just with Scipy, but with *all* f2py code that is out there. Everyone who uses "double complex, intent(inout)" in their f2py wrapped code will start getting random exceptions on Windows. Users of "double complex, intent(in)" pay a performance penalty. -- Pauli Virtanen From andrea.gavana at gmail.com Tue Nov 25 18:50:55 2014 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Wed, 26 Nov 2014 00:50:55 +0100 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> Message-ID: On 25 November 2014 at 19:33, David Cournapeau wrote: > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > wrote: > >> David Cournapeau wrote: >> > Shall we consider > > href="https://github.com/scipy/scipy/issues/4168"> >> https://github.com/scipy/scipy/issues/4168 >> > to be a >> > blocker (the issue arises on scipy master as well as 0.14.1) ? >> > >> >> It is really bad, but does anyone know what is really going on? >> > > Yes, it is in the bug report. > Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) Happy hacking. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- # -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 25 19:00:12 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 26 Nov 2014 00:00:12 +0000 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Nov 25, 2014 at 11:50 PM, Andrea Gavana wrote: > > On 25 November 2014 at 19:33, David Cournapeau wrote: >> >> >> >> On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden >> wrote: >>> >>> David Cournapeau wrote: >>> > Shall we consider >> > >>> > href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168 >>> > to be a >>> > blocker (the issue arises on scipy master as well as 0.14.1) ? >>> > >>> >>> It is really bad, but does anyone know what is really going on? >> >> >> Yes, it is in the bug report. > > > > Nice move. > > I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun > thing on NumPy/SciPy across the whole user base of Python in the company. We > will probably move to 64bit-in-any-sense soon enough, I guess before this > issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is > NumPy stricter in its checking because of SPARC? SPARC?!? It's a regression in 1.9, yes; 1.8 is fine. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From yw5aj at virginia.edu Tue Nov 25 23:06:57 2014 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Tue, 25 Nov 2014 23:06:57 -0500 Subject: [Numpy-discussion] Implementation of tensor product and tensor contraction? Message-ID: Dear all, I have been doing tensor algebra recently (for continuum mechanics) and was looking into two common operations: tensor product & tensor contraction. 1. Tensor product One common usage is: a[i1, i2, i3, ..., iN, j1, j2, j3, ..., jM] = b[i1, i2, i3, ..., iN] * c[j1, j2, j3, ..., jM] I looked into the current np.outer(), and the only difference is that it always flattens the input array. So actually, the function for tensor product is simply np.outer(a, b, out=out).reshape(a.shape+b.shape) <-- I think I got this right but please do correct me if I am wrong Would anyone think it helpful or harmful to add such a function, np.tensorprod()? it will simply be like def tensorprod(a, b, out=None): return outer(a, b, out=out).reshape(a.shape+b.shape) 2. Tensor contraction It is currently the np.tensordot(a, b) and it will do np.tensordot(a, b, axes=2) by default. I think this is all great, but it would be even better if we specify in the doc, that: i) say explicitly that by default it will be the double-dot or contraction operator, and ii) explain that in cases where axes is an integer-like scalar, which axes were selected from the two array and in what order. Like: if axes is an integer-like scalar, it is the number axes to sum over, equivalent to axes=(list(range(-axes, 0)), list(range(0, axes))) (or something like this) It'd be great to hear what you would think about it, Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From charlesr.harris at gmail.com Tue Nov 25 23:12:26 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 25 Nov 2014 21:12:26 -0700 Subject: [Numpy-discussion] Implementation of tensor product and tensor contraction? In-Reply-To: References: Message-ID: Take a look as einsum, it is quite good for such things. Chuck On Tue, Nov 25, 2014 at 9:06 PM, Yuxiang Wang wrote: > Dear all, > > I have been doing tensor algebra recently (for continuum mechanics) > and was looking into two common operations: tensor product & tensor > contraction. > > 1. Tensor product > > One common usage is: > a[i1, i2, i3, ..., iN, j1, j2, j3, ..., jM] = b[i1, i2, i3, ..., iN] * > c[j1, j2, j3, ..., jM] > > I looked into the current np.outer(), and the only difference is that > it always flattens the input array. So actually, the function for > tensor product is simply > > np.outer(a, b, out=out).reshape(a.shape+b.shape) <-- I think I got > this right but please do correct me if I am wrong > > Would anyone think it helpful or harmful to add such a function, > np.tensorprod()? it will simply be like > def tensorprod(a, b, out=None): > return outer(a, b, out=out).reshape(a.shape+b.shape) > > > 2. Tensor contraction > > It is currently the np.tensordot(a, b) and it will do np.tensordot(a, > b, axes=2) by default. I think this is all great, but it would be even > better if we specify in the doc, that: > i) say explicitly that by default it will be the double-dot or > contraction operator, and > ii) explain that in cases where axes is an integer-like scalar, > which axes were selected from the two array and in what order. Like: > if axes is an integer-like scalar, it is the number axes to sum over, > equivalent to axes=(list(range(-axes, 0)), list(range(0, axes))) (or > something like this) > > > It'd be great to hear what you would think about it, > > Shawn > > > -- > Yuxiang "Shawn" Wang > Gerling Research Lab > University of Virginia > yw5aj at virginia.edu > +1 (434) 284-0836 > https://sites.google.com/a/virginia.edu/yw5aj/ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yw5aj at virginia.edu Tue Nov 25 23:16:59 2014 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Tue, 25 Nov 2014 23:16:59 -0500 Subject: [Numpy-discussion] Implementation of tensor product and tensor contraction? In-Reply-To: References: Message-ID: Hi Charles, Thank you for your response! I do think np.einsum() is really great. I am not clear about how that ties to my question though, because I was thinking more in lines of wrapping and reshaping the output (#1) and improve the documentation (#2), where the proper outcome was already calculated. -Shawn On Tue, Nov 25, 2014 at 11:12 PM, Charles R Harris wrote: > Take a look as einsum, it is quite good for such things. > > Chuck > > On Tue, Nov 25, 2014 at 9:06 PM, Yuxiang Wang wrote: >> >> Dear all, >> >> I have been doing tensor algebra recently (for continuum mechanics) >> and was looking into two common operations: tensor product & tensor >> contraction. >> >> 1. Tensor product >> >> One common usage is: >> a[i1, i2, i3, ..., iN, j1, j2, j3, ..., jM] = b[i1, i2, i3, ..., iN] * >> c[j1, j2, j3, ..., jM] >> >> I looked into the current np.outer(), and the only difference is that >> it always flattens the input array. So actually, the function for >> tensor product is simply >> >> np.outer(a, b, out=out).reshape(a.shape+b.shape) <-- I think I got >> this right but please do correct me if I am wrong >> >> Would anyone think it helpful or harmful to add such a function, >> np.tensorprod()? it will simply be like >> def tensorprod(a, b, out=None): >> return outer(a, b, out=out).reshape(a.shape+b.shape) >> >> >> 2. Tensor contraction >> >> It is currently the np.tensordot(a, b) and it will do np.tensordot(a, >> b, axes=2) by default. I think this is all great, but it would be even >> better if we specify in the doc, that: >> i) say explicitly that by default it will be the double-dot or >> contraction operator, and >> ii) explain that in cases where axes is an integer-like scalar, >> which axes were selected from the two array and in what order. Like: >> if axes is an integer-like scalar, it is the number axes to sum over, >> equivalent to axes=(list(range(-axes, 0)), list(range(0, axes))) (or >> something like this) >> >> >> It'd be great to hear what you would think about it, >> >> Shawn >> >> >> -- >> Yuxiang "Shawn" Wang >> Gerling Research Lab >> University of Virginia >> yw5aj at virginia.edu >> +1 (434) 284-0836 >> https://sites.google.com/a/virginia.edu/yw5aj/ >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From cournape at gmail.com Wed Nov 26 03:44:52 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 26 Nov 2014 08:44:52 +0000 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? Message-ID: Hi, Would anybody mind if I create a label "newcomers" on GH, and start labelling simple issues ? This is in anticipation to the bloomberg lab event in London this WE. I will try to give a hand to people interested in numpy/scipy, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Wed Nov 26 04:01:57 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 26 Nov 2014 10:01:57 +0100 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: References: Message-ID: <54759705.2090203@googlemail.com> On 11/26/2014 09:44 AM, David Cournapeau wrote: > Hi, > > Would anybody mind if I create a label "newcomers" on GH, and start > labelling simple issues ? > > This is in anticipation to the bloomberg lab event in London this WE. I > will try to give a hand to people interested in numpy/scipy, > > David > we have the easy-fix tag which we use for this purpose, though we have not been applying it very systematically, the bug list could probably do with a new sweep for these issues. From jtaylor.debian at googlemail.com Wed Nov 26 04:06:50 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 26 Nov 2014 10:06:50 +0100 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> Message-ID: <5475982A.8060201@googlemail.com> On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > On 25 November 2014 at 19:33, David Cournapeau > wrote: > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > > wrote: > > David Cournapeau > wrote: > > Shall we consider > href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168 > > to be a > > blocker (the issue arises on scipy master as well as 0.14.1) ? > > > > It is really bad, but does anyone know what is really going on? > > > Yes, it is in the bug report. > > > > Nice move. > > I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun > thing on NumPy/SciPy across the whole user base of Python in the > company. We will probably move to 64bit-in-any-sense soon enough, I > guess before this issue is solved. Tell me, NumPy, was the array aligned > enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc. > > Dozens of f2py compiled extensions are going to fail soon here - which > I'll reluctantly check tomorrow. I don't want to think about other > people on Win32 facing the same issue elsewhere... :-) > There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it. From cournape at gmail.com Wed Nov 26 04:08:42 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 26 Nov 2014 09:08:42 +0000 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: <54759705.2090203@googlemail.com> References: <54759705.2090203@googlemail.com> Message-ID: oups, I missed it. Will use that one then. On Wed, Nov 26, 2014 at 9:01 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 11/26/2014 09:44 AM, David Cournapeau wrote: > > Hi, > > > > Would anybody mind if I create a label "newcomers" on GH, and start > > labelling simple issues ? > > > > This is in anticipation to the bloomberg lab event in London this WE. I > > will try to give a hand to people interested in numpy/scipy, > > > > David > > > > we have the easy-fix tag which we use for this purpose, though we have > not been applying it very systematically, the bug list could probably do > with a new sweep for these issues. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Nov 26 04:36:29 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 26 Nov 2014 10:36:29 +0100 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: References: Message-ID: <1416994589.5657.6.camel@sebastian-t440> On Mi, 2014-11-26 at 08:44 +0000, David Cournapeau wrote: > Hi, > > > Would anybody mind if I create a label "newcomers" on GH, and start > labelling simple issues ? We actually have an "easy fix" label, which I think had this in mind. However, I admit that I think some of these issues may not be easy at all (I guess it depends on what you consider easy ;)). In any case, I think just go ahead with creating a new label or reusing the current one. "easy fix" might be a starting point to find some candidate issues. - Sebsatian > > > This is in anticipation to the bloomberg lab event in London this WE. > I will try to give a hand to people interested in numpy/scipy, > > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From pav at iki.fi Wed Nov 26 07:48:24 2014 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 26 Nov 2014 12:48:24 +0000 (UTC) Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> <5475982A.8060201@googlemail.com> Message-ID: Julian Taylor googlemail.com> writes: [clip] > There haven't been any real complaints from applications yet, only > testsuite failure of scipy. > Either the one thing that is broken in scipy isn't much used or windows > 32 users aren't using 1.9 yet. What is broken is calculating eigenvalues of complex-valued sparse matrices and iterative solution of complex-valued linear equations. I.e., nothing obscure. A likely explanation is that win32 + Numpy 1.9 is a less common platform, and users whose code started failing just infrequently do not report bugs as easily... > The majority of f2py should still be working, numpys own f2py testsuite > passes on win32. Perhaps the arrays are aligned by chance? I don't think the test suite repeats the complex valued intent(inout) parameter test many times. [clip] > I still don't know what exactly arpack is doing > different but I also did not have time yet to look at the testcase david > created. David's test case is this: n = 4 x = np.zeros(n * 3, dtype="D") _dummy.zfoo(x, n) where the argument is declared as "double complex, dimension(3*n), intent(inout)" in f2py. The ARPACK stuff in Scipy also does pretty much just this. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Nov 26 08:19:31 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 26 Nov 2014 06:19:31 -0700 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: <5475982A.8060201@googlemail.com> References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> <5475982A.8060201@googlemail.com> Message-ID: On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > > > On 25 November 2014 at 19:33, David Cournapeau > > wrote: > > > > > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > > > wrote: > > > > David Cournapeau > > wrote: > > > Shall we consider > > href="https://github.com/scipy/scipy/issues/4168"> > https://github.com/scipy/scipy/issues/4168 > > > to be a > > > blocker (the issue arises on scipy master as well as 0.14.1) ? > > > > > > > It is really bad, but does anyone know what is really going on? > > > > > > Yes, it is in the bug report. > > > > > > > > Nice move. > > > > I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun > > thing on NumPy/SciPy across the whole user base of Python in the > > company. We will probably move to 64bit-in-any-sense soon enough, I > > guess before this issue is solved. Tell me, NumPy, was the array aligned > > enough in 1.8? Is NumPy stricter in its checking because of SPARC? > SPARC?!? > > yes, before the change numpy accepted a factor 10 performance regression > in complex indexing to satisfy sparc. > > > > > Dozens of f2py compiled extensions are going to fail soon here - which > > I'll reluctantly check tomorrow. I don't want to think about other > > people on Win32 facing the same issue elsewhere... :-) > > > > There haven't been any real complaints from applications yet, only > testsuite failure of scipy. > Either the one thing that is broken in scipy isn't much used or windows > 32 users aren't using 1.9 yet. > The majority of f2py should still be working, numpys own f2py testsuite > passes on win32. I still don't know what exactly arpack is doing > different but I also did not have time yet to look at the testcase david > created. > > It should of course be fixed, I have an old branch with aligned > allocators lying around somewhere which should fix the issue or at least > give users a way to work around it. > The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned. If anyone is to blame, it is the reviewer, namely, myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 26 08:24:07 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 26 Nov 2014 06:24:07 -0700 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: <1416994589.5657.6.camel@sebastian-t440> References: <1416994589.5657.6.camel@sebastian-t440> Message-ID: On Wed, Nov 26, 2014 at 2:36 AM, Sebastian Berg wrote: > On Mi, 2014-11-26 at 08:44 +0000, David Cournapeau wrote: > > Hi, > > > > > > Would anybody mind if I create a label "newcomers" on GH, and start > > labelling simple issues ? > > We actually have an "easy fix" label, which I think had this in mind. > However, I admit that I think some of these issues may not be easy at > all (I guess it depends on what you consider easy ;)). In any case, I > think just go ahead with creating a new label or reusing the current > one. "easy fix" might be a starting point to find some candidate issues. > > - Sebsatian > > > > > > > This is in anticipation to the bloomberg lab event in London this WE. > > I will try to give a hand to people interested in numpy/scipy, > There is also a documentation label, and about 30 tickets with that label. That should be good for just practicing the mechanics. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Wed Nov 26 08:59:34 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 26 Nov 2014 14:59:34 +0100 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> <5475982A.8060201@googlemail.com> Message-ID: <5475DCC6.3010209@googlemail.com> On 11/26/2014 02:19 PM, Charles R Harris wrote: > > > On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor > > > wrote: > > On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > > > On 25 November 2014 at 19:33, David Cournapeau > > >> wrote: > > > > > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > > > >> > wrote: > > > > David Cournapeau > > >> wrote: > > > Shall we consider > > href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168 > > > to be a > > > blocker (the issue arises on scipy master as well as 0.14.1) ? > > > > > > > It is really bad, but does anyone know what is really going on? > > > > > > Yes, it is in the bug report. > > > > > > > > Nice move. > > > > I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun > > thing on NumPy/SciPy across the whole user base of Python in the > > company. We will probably move to 64bit-in-any-sense soon enough, I > > guess before this issue is solved. Tell me, NumPy, was the array aligned > > enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? > > yes, before the change numpy accepted a factor 10 performance regression > in complex indexing to satisfy sparc. > > > > > Dozens of f2py compiled extensions are going to fail soon here - which > > I'll reluctantly check tomorrow. I don't want to think about other > > people on Win32 facing the same issue elsewhere... :-) > > > > There haven't been any real complaints from applications yet, only > testsuite failure of scipy. > Eithe the one thing that is broken in scipy isn't much used or windows > 32 users aren't using 1.9 yet. > The majority of f2py should still be working, numpys own f2py testsuite > passes on win32. I still don't know what exactly arpack is doing > different but I also did not have time yet to look at the testcase david > created. > > It should of course be fixed, I have an old branch with aligned > allocators lying around somewhere which should fix the issue or at least > give users a way to work around it. > > > The lesson I take from this is that we need a test ;) Also, that it > would be nice if we got more testing by users before releases. But > things are as they are, this will get fixed, and we will move on with > one more lesson learned. > > If anyone is to blame, it is the reviewer, namely, myself. > concerning actually fixing it I guess we have 3 options: 1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows. fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...) From charlesr.harris at gmail.com Wed Nov 26 09:14:04 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 26 Nov 2014 07:14:04 -0700 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: <5475DCC6.3010209@googlemail.com> References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> <5475982A.8060201@googlemail.com> <5475DCC6.3010209@googlemail.com> Message-ID: On Wed, Nov 26, 2014 at 6:59 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 11/26/2014 02:19 PM, Charles R Harris wrote: > > > > > > On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor > > > > > wrote: > > > > On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > > > > > On 25 November 2014 at 19:33, David Cournapeau > > > >> wrote: > > > > > > > > > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > > > > > >> > > wrote: > > > > > > David Cournapeau cournape at gmail.com> > > > >> > wrote: > > > > Shall we consider > > > href="https://github.com/scipy/scipy/issues/4168"> > https://github.com/scipy/scipy/issues/4168 > > > > to be a > > > > blocker (the issue arises on scipy master as well as > 0.14.1) ? > > > > > > > > > > It is really bad, but does anyone know what is really > going on? > > > > > > > > > Yes, it is in the bug report. > > > > > > > > > > > > Nice move. > > > > > > I've now recommended to hold back any > upgrade/update/pip-crap/enpkg-fun > > > thing on NumPy/SciPy across the whole user base of Python in the > > > company. We will probably move to 64bit-in-any-sense soon enough, I > > > guess before this issue is solved. Tell me, NumPy, was the array > aligned > > > enough in 1.8? Is NumPy stricter in its checking because of SPARC? > SPARC?!? > > > > yes, before the change numpy accepted a factor 10 performance > regression > > in complex indexing to satisfy sparc. > > > > > > > > Dozens of f2py compiled extensions are going to fail soon here - > which > > > I'll reluctantly check tomorrow. I don't want to think about other > > > people on Win32 facing the same issue elsewhere... :-) > > > > > > > There haven't been any real complaints from applications yet, only > > testsuite failure of scipy. > > Eithe the one thing that is broken in scipy isn't much used or > windows > > 32 users aren't using 1.9 yet. > > The majority of f2py should still be working, numpys own f2py > testsuite > > passes on win32. I still don't know what exactly arpack is doing > > different but I also did not have time yet to look at the testcase > david > > created. > > > > It should of course be fixed, I have an old branch with aligned > > allocators lying around somewhere which should fix the issue or at > least > > give users a way to work around it. > > > > > > The lesson I take from this is that we need a test ;) Also, that it > > would be nice if we got more testing by users before releases. But > > things are as they are, this will get fixed, and we will move on with > > one more lesson learned. > > > > If anyone is to blame, it is the reviewer, namely, myself. > > > > > concerning actually fixing it I guess we have 3 options: > > 1. reduce maximum copy alignment from currently 16 to 8 on platforms > that need it. > That will automatically reduce the needed alignment of complex on win32 > to 8 bytes. Do other compilers provide something similar to gccs > __BIGGEST_ALIGNMENT__? > 2. remove bloating of complex alignment and let sparc crash. > 3. add an aligned allocator > > I somewhat favor 1, it has the risk that a vectorizing compiler might > wreak havoc but to my knowledge numpy does not actually have real 16 > byte copies. There are some occurrences of npy_int128, but those likely > are not used on windows. > > fwiw I could reproduce the issue on linux i386 with -malign-double, (I > could have sworn I tested that configuration too...) > I would also go for 1) on the general principal that alignment needs to be platform dependent. I suppose the problem here is that is might also be compiler dependent. Where do the aligned copies take place? What is the downside to that solution? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 26 11:46:32 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 26 Nov 2014 16:46:32 +0000 Subject: [Numpy-discussion] Scipy 0.15.0 beta 1 release In-Reply-To: <5475DCC6.3010209@googlemail.com> References: <547269FF.7010905@iki.fi> <135351674438631559.674600sturla.molden-gmail.com@news.gmane.org> <5475982A.8060201@googlemail.com> <5475DCC6.3010209@googlemail.com> Message-ID: On 26 Nov 2014 13:59, "Julian Taylor" wrote: > > concerning actually fixing it I guess we have 3 options: > > 1. reduce maximum copy alignment from currently 16 to 8 on platforms > that need it. > That will automatically reduce the needed alignment of complex on win32 > to 8 bytes. Do other compilers provide something similar to gccs > __BIGGEST_ALIGNMENT__? > 2. remove bloating of complex alignment and let sparc crash. > 3. add an aligned allocator > > I somewhat favor 1, it has the risk that a vectorizing compiler might > wreak havoc but to my knowledge numpy does not actually have real 16 > byte copies. There are some occurrences of npy_int128, but those likely > are not used on windows. > > fwiw I could reproduce the issue on linux i386 with -malign-double, (I > could have sworn I tested that configuration too...) I'm not sure what "maximum copy alignment" means in this context, but (1) does sound the most like a proper fix to me too. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Wed Nov 26 13:19:31 2014 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Wed, 26 Nov 2014 13:19:31 -0500 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: References: <1416994589.5657.6.camel@sebastian-t440> Message-ID: On Wed, Nov 26, 2014 at 8:24 AM, Charles R Harris wrote: > > > On Wed, Nov 26, 2014 at 2:36 AM, Sebastian Berg < > sebastian at sipsolutions.net> wrote: > >> On Mi, 2014-11-26 at 08:44 +0000, David Cournapeau wrote: >> > Hi, >> > >> > >> > Would anybody mind if I create a label "newcomers" on GH, and start >> > labelling simple issues ? >> >> We actually have an "easy fix" label, which I think had this in mind. >> However, I admit that I think some of these issues may not be easy at >> all (I guess it depends on what you consider easy ;)). In any case, I >> think just go ahead with creating a new label or reusing the current >> one. "easy fix" might be a starting point to find some candidate issues. >> >> - Sebsatian >> >> > >> > >> > This is in anticipation to the bloomberg lab event in London this WE. >> > I will try to give a hand to people interested in numpy/scipy, >> > > There is also a documentation label, and about 30 tickets with that label. > That should be good for just practicing the mechanics. > FWIW in astropy we settled on two properties, level of effort and level of sub-package expertise, with corresponding labels: - effort-low, effort-medium, and effort-high - package-novice, package-intermediate, package-expert This has been used with reasonable success. - Tom > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Nov 26 20:29:31 2014 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 26 Nov 2014 20:29:31 -0500 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: References: <1416994589.5657.6.camel@sebastian-t440> Message-ID: FWIW, matplotlib calls it "low hanging fruit". I think it is a better name than "newcomers". On Wed, Nov 26, 2014 at 1:19 PM, Aldcroft, Thomas < aldcroft at head.cfa.harvard.edu> wrote: > > > On Wed, Nov 26, 2014 at 8:24 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Wed, Nov 26, 2014 at 2:36 AM, Sebastian Berg < >> sebastian at sipsolutions.net> wrote: >> >>> On Mi, 2014-11-26 at 08:44 +0000, David Cournapeau wrote: >>> > Hi, >>> > >>> > >>> > Would anybody mind if I create a label "newcomers" on GH, and start >>> > labelling simple issues ? >>> >>> We actually have an "easy fix" label, which I think had this in mind. >>> However, I admit that I think some of these issues may not be easy at >>> all (I guess it depends on what you consider easy ;)). In any case, I >>> think just go ahead with creating a new label or reusing the current >>> one. "easy fix" might be a starting point to find some candidate issues. >>> >>> - Sebsatian >>> >>> > >>> > >>> > This is in anticipation to the bloomberg lab event in London this WE. >>> > I will try to give a hand to people interested in numpy/scipy, >>> >> >> There is also a documentation label, and about 30 tickets with that >> label. That should be good for just practicing the mechanics. >> > > FWIW in astropy we settled on two properties, level of effort and level of > sub-package expertise, with corresponding labels: > > - effort-low, effort-medium, and effort-high > - package-novice, package-intermediate, package-expert > > This has been used with reasonable success. > > - Tom > > >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwojc at p.lodz.pl Thu Nov 27 04:00:01 2014 From: mwojc at p.lodz.pl (Marek Wojciechowski) Date: Thu, 27 Nov 2014 10:00:01 +0100 Subject: [Numpy-discussion] return type from ufuncs In-Reply-To: <1743805.dMy5v138kg@think> References: <1743805.dMy5v138kg@think> Message-ID: <11082192.sWhnnZHngt@think> Dnia czwartek, 20 listopada 2014 18:47:41 Marek Wojciechowski pisze: > Hi! > > I wrote a simple subclass of np.ndarray and now i do call np.sum() on it. I > expected that the result will be a python float (or int) just like when > summing up regular arrays. Instead i obtain the (scalar) view of my > subclass. How can i change this behavior? I tried writing __array_wrap__ > method like this: > > def __array_wrap__(self, out_arr, context=None): > selfv = self.view(np.ndarray) > return np.ndarray.__array_wrap__(selfv, out_arr, context) > > but this just returns np.ndarray type and not float. Hi! I'm back with the problem of returning types form ndarray subclasses, now with ravelling the array. I wrote the __array_wrap__ function like this: def __array_wrap__(self, out_arr, context=None): arr = out_arr.view(np.ndarray) if arr.ndim == 0: arr = arr[()] return arr I was convinced that now ufuncs wil return always numpy arrays. But now i discovered that for example: a.ravel() returns still the subclass type, not ndarray type. Ravel method apparently does not call __array_wrap__ (whereas np.ravel(a) does!). Is there a systematic way to resolve this or i need to write new ravel method? Regards, -- Marek Wojciechowski From toddrjen at gmail.com Thu Nov 27 04:57:35 2014 From: toddrjen at gmail.com (Todd) Date: Thu, 27 Nov 2014 10:57:35 +0100 Subject: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ? In-Reply-To: References: <1416994589.5657.6.camel@sebastian-t440> Message-ID: KDE calls them"junior jobs". On Nov 27, 2014 2:29 AM, "Benjamin Root" wrote: > FWIW, matplotlib calls it "low hanging fruit". I think it is a better name > than "newcomers". > > On Wed, Nov 26, 2014 at 1:19 PM, Aldcroft, Thomas < > aldcroft at head.cfa.harvard.edu> wrote: > >> >> >> On Wed, Nov 26, 2014 at 8:24 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Wed, Nov 26, 2014 at 2:36 AM, Sebastian Berg < >>> sebastian at sipsolutions.net> wrote: >>> >>>> On Mi, 2014-11-26 at 08:44 +0000, David Cournapeau wrote: >>>> > Hi, >>>> > >>>> > >>>> > Would anybody mind if I create a label "newcomers" on GH, and start >>>> > labelling simple issues ? >>>> >>>> We actually have an "easy fix" label, which I think had this in mind. >>>> However, I admit that I think some of these issues may not be easy at >>>> all (I guess it depends on what you consider easy ;)). In any case, I >>>> think just go ahead with creating a new label or reusing the current >>>> one. "easy fix" might be a starting point to find some candidate issues. >>>> >>>> - Sebsatian >>>> >>>> > >>>> > >>>> > This is in anticipation to the bloomberg lab event in London this WE. >>>> > I will try to give a hand to people interested in numpy/scipy, >>>> >>> >>> There is also a documentation label, and about 30 tickets with that >>> label. That should be good for just practicing the mechanics. >>> >> >> FWIW in astropy we settled on two properties, level of effort and level >> of sub-package expertise, with corresponding labels: >> >> - effort-low, effort-medium, and effort-high >> - package-novice, package-intermediate, package-expert >> >> This has been used with reasonable success. >> >> - Tom >> >> >>> >>> Chuck >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Thu Nov 27 22:15:41 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Thu, 27 Nov 2014 22:15:41 -0500 Subject: [Numpy-discussion] Finding values in an array Message-ID: I probably miss something very basic, but how given two arrays a and b, can I find positions in a where elements of b are located? If a were sorted, I could use searchsorted, but I don't want to get valid positions for elements that are not in a. In my case, a has unique elements, but in the general case I would accept the first match. In other words, I am looking for an array analog of list.index() method. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Thu Nov 27 23:33:33 2014 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 27 Nov 2014 23:33:33 -0500 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: Message-ID: On Thu, Nov 27, 2014 at 10:15 PM, Alexander Belopolsky wrote: > I probably miss something very basic, but how given two arrays a and b, > can I find positions in a where elements of b are located? If a were > sorted, I could use searchsorted, but I don't want to get valid positions > for elements that are not in a. In my case, a has unique elements, but in > the general case I would accept the first match. In other words, I am > looking for an array analog of list.index() method. > I don't know an easy solution to this problem in pure numpy, but if you could do this pretty easily (and quite efficiently) if you are willing to use pandas. Something like: locs = pd.Index(a).get_indexer(b) Note that -1 is used to denote a non-match, and get_indexer will raise if the match is non-unique instead of returning the first element. If your array is not 1d, you can still make this work but you'll need to use np.ravel and np.unravel_index. Actually, you may find that putting your data into pandas data structures is a good solution, since pandas is designed to make exactly these sort of alignment operations easy (and automatic). I suppose the simplest solution to this problem would be to convert your data into a list and use list.index() repeatedly (or you could even write it yourself in a few lines), but I'd guess that was never implemented for ndarrays because it's rather slow -- better to use a hash-table like a dict or pandas.Index for repeated lookups. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Nov 28 03:22:47 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 28 Nov 2014 09:22:47 +0100 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: Message-ID: <547830D7.2020008@googlemail.com> On 28.11.2014 04:15, Alexander Belopolsky wrote: > I probably miss something very basic, but how given two arrays a and b, > can I find positions in a where elements of b are located? If a were > sorted, I could use searchsorted, but I don't want to get valid > positions for elements that are not in a. In my case, a has unique > elements, but in the general case I would accept the first match. In > other words, I am looking for an array analog of list.index() method. np.where(np.in1d(a, b)) From robert.kern at gmail.com Fri Nov 28 03:37:32 2014 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Nov 2014 08:37:32 +0000 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: <547830D7.2020008@googlemail.com> References: <547830D7.2020008@googlemail.com> Message-ID: On Fri, Nov 28, 2014 at 8:22 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > > On 28.11.2014 04:15, Alexander Belopolsky wrote: > > I probably miss something very basic, but how given two arrays a and b, > > can I find positions in a where elements of b are located? If a were > > sorted, I could use searchsorted, but I don't want to get valid > > positions for elements that are not in a. In my case, a has unique > > elements, but in the general case I would accept the first match. In > > other words, I am looking for an array analog of list.index() method. > > np.where(np.in1d(a, b)) Only if the matching elements in `b` have the same order as they do in `a`. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Nov 28 03:40:29 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 28 Nov 2014 09:40:29 +0100 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: <547830D7.2020008@googlemail.com> Message-ID: <547834FD.9070600@googlemail.com> On 28.11.2014 09:37, Robert Kern wrote: > On Fri, Nov 28, 2014 at 8:22 AM, Julian Taylor > > > wrote: >> >> On 28.11.2014 04:15, Alexander Belopolsky wrote: >> > I probably miss something very basic, but how given two arrays a and b, >> > can I find positions in a where elements of b are located? If a were >> > sorted, I could use searchsorted, but I don't want to get valid >> > positions for elements that are not in a. In my case, a has unique >> > elements, but in the general case I would accept the first match. In >> > other words, I am looking for an array analog of list.index() method. >> >> np.where(np.in1d(a, b)) > > Only if the matching elements in `b` have the same order as they do in `a`. > seems to work also if unordered: In [32]: a = np.arange(1000) In [33]: b = np.arange(500,550, 3) In [34]: np.random.shuffle(a) In [35]: np.random.shuffle(b) In [36]: np.where(np.in1d(a, b)) Out[36]: (array([ 0, 106, 133, 149, 238, 398, 418, 498, 533, 541, 545, 589, 634, 798, 846, 891, 965]),) From robert.kern at gmail.com Fri Nov 28 04:01:16 2014 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Nov 2014 09:01:16 +0000 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: <547834FD.9070600@googlemail.com> References: <547830D7.2020008@googlemail.com> <547834FD.9070600@googlemail.com> Message-ID: On Fri, Nov 28, 2014 at 8:40 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 28.11.2014 09:37, Robert Kern wrote: > > On Fri, Nov 28, 2014 at 8:22 AM, Julian Taylor > > > > > wrote: > >> > >> On 28.11.2014 04:15, Alexander Belopolsky wrote: > >> > I probably miss something very basic, but how given two arrays a and > b, > >> > can I find positions in a where elements of b are located? If a were > >> > sorted, I could use searchsorted, but I don't want to get valid > >> > positions for elements that are not in a. In my case, a has unique > >> > elements, but in the general case I would accept the first match. In > >> > other words, I am looking for an array analog of list.index() method. > >> > >> np.where(np.in1d(a, b)) > > > > Only if the matching elements in `b` have the same order as they do in > `a`. > > > > seems to work also if unordered: > In [32]: a = np.arange(1000) > > In [33]: b = np.arange(500,550, 3) > > In [34]: np.random.shuffle(a) > > In [35]: np.random.shuffle(b) > > In [36]: np.where(np.in1d(a, b)) > Out[36]: > (array([ 0, 106, 133, 149, 238, 398, 418, 498, 533, 541, 545, 589, 634, > 798, 846, 891, 965]),) I meant that the OP is asking for something stricter than that. He wants this array of indices to be in the order in which those matching elements appear in `b` so that he can use this information to merge two datasets. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Fri Nov 28 09:54:08 2014 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 28 Nov 2014 15:54:08 +0100 Subject: [Numpy-discussion] ANN: SfePy 2014.4 Message-ID: <54788C90.1080405@ntc.zcu.cz> I am pleased to announce release 2014.4 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - preliminary support for 1D problems - data probes using pyVTK library For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Lubos Kejzlar, Vladimir Lukes From nickeubank at gmail.com Fri Nov 28 16:12:07 2014 From: nickeubank at gmail.com (Nick Eubank) Date: Fri, 28 Nov 2014 21:12:07 +0000 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? Message-ID: Hello Numpy! Curious if there were any timelines for when NA / NaN support might arrive for Ints. (I'm sure this is asked and answered many times, but what I can find seems very out of date.) I'm strongly considering migrating from R/Stata/Matlab to Pandas/Numpy, but the inability to handle Na in Int types seems like a big limitation for someone always working with dirty data like me (a social scientist), and the Pandas docs suggest they're waiting for Numpy to add support. Just curious when that might happen! Thanks! nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 28 20:08:55 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 01:08:55 +0000 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? In-Reply-To: References: Message-ID: On Fri, Nov 28, 2014 at 9:12 PM, Nick Eubank wrote: > Hello Numpy! > > Curious if there were any timelines for when NA / NaN support might arrive > for Ints. > > (I'm sure this is asked and answered many times, but what I can find seems > very out of date.) > > I'm strongly considering migrating from R/Stata/Matlab to Pandas/Numpy, but > the inability to handle Na in Int types seems like a big limitation for > someone always working with dirty data like me (a social scientist), and the > Pandas docs suggest they're waiting for Numpy to add support. Just curious > when that might happen! There's no particular timeline, no. We don't have any budget or employees so any timeline would be a fantasy anyway :-). It's definitely possible to do and we'd like to do it, but it'll happen when someone with the right skills and interests finds the time to do the work. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Fri Nov 28 20:15:27 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 01:15:27 +0000 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: Message-ID: On Fri, Nov 28, 2014 at 3:15 AM, Alexander Belopolsky wrote: > I probably miss something very basic, but how given two arrays a and b, can > I find positions in a where elements of b are located? If a were sorted, I > could use searchsorted, but I don't want to get valid positions for elements > that are not in a. In my case, a has unique elements, but in the general > case I would accept the first match. In other words, I am looking for an > array analog of list.index() method. How about this? def index(haystack, needle): haystack = np.asarray(haystack) haystack_sort = np.argsort(haystack) haystack_sorted = haystack[haystack_sort] return haystack_sort[np.searchsorted(haystack_sorted, needle)] (Note that this will return incorrect results if any entries in needle are missing from haystack entirely. If this is a concern then you need to do some extra error-checking on the searchsorted return value.) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ben.root at ou.edu Fri Nov 28 20:51:52 2014 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 28 Nov 2014 20:51:52 -0500 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? In-Reply-To: References: Message-ID: That being said, doesn't Pandas support something like what you are asking for? While Pandas would like to push the NA support down into the NumPy level for seamless interaction with other SciPy libraries, it does a very decent job with NA on its own, and depending on the sort of code you need to write, you might rarely ever have to directly interface with NumPy. Pandas is very powerful for data analysis, particularly for those coming from R and S+. Those of us who use NumPy directly tend to be involved in more generalized numerical tasks than those developing Pandas. Perhaps you should raise your concerns on the Pandas mailing list? They might be better suited to help you with any hurdles that you foresee. The people on that list are very familiar with dealing with "dirty data". Cheers! Ben Root On Fri, Nov 28, 2014 at 8:08 PM, Nathaniel Smith wrote: > On Fri, Nov 28, 2014 at 9:12 PM, Nick Eubank wrote: > > Hello Numpy! > > > > Curious if there were any timelines for when NA / NaN support might > arrive > > for Ints. > > > > (I'm sure this is asked and answered many times, but what I can find > seems > > very out of date.) > > > > I'm strongly considering migrating from R/Stata/Matlab to Pandas/Numpy, > but > > the inability to handle Na in Int types seems like a big limitation for > > someone always working with dirty data like me (a social scientist), and > the > > Pandas docs suggest they're waiting for Numpy to add support. Just > curious > > when that might happen! > > There's no particular timeline, no. We don't have any budget or > employees so any timeline would be a fantasy anyway :-). It's > definitely possible to do and we'd like to do it, but it'll happen > when someone with the right skills and interests finds the time to do > the work. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 28 20:57:14 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 01:57:14 +0000 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? In-Reply-To: References: Message-ID: On Sat, Nov 29, 2014 at 1:51 AM, Benjamin Root wrote: > That being said, doesn't Pandas support something like what you are asking > for? While Pandas would like to push the NA support down into the NumPy > level for seamless interaction with other SciPy libraries, it does a very > decent job with NA on its own, and depending on the sort of code you need to > write, you might rarely ever have to directly interface with NumPy. Pandas > is very powerful for data analysis, particularly for those coming from R and > S+. Those of us who use NumPy directly tend to be involved in more > generalized numerical tasks than those developing Pandas. > > Perhaps you should raise your concerns on the Pandas mailing list? They > might be better suited to help you with any hurdles that you foresee. The > people on that list are very familiar with dealing with "dirty data". Nick is way ahead of you :-) https://groups.google.com/forum/#!topic/pydata/g9UPbgFHPxk -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From nickeubank at gmail.com Fri Nov 28 21:02:15 2014 From: nickeubank at gmail.com (Nick Eubank) Date: Sat, 29 Nov 2014 02:02:15 +0000 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? References: Message-ID: Thanks Nathaniel and Ben. Nathaniel, I completely understand! Believe me, I appreciate all you folks do already, and am hoping to some day improve my skills enough to contribute myself. Ben: they only handle it for Floats. If you introduce an NA to an int64 series, it just coerces the type into Float64. With that said, someone on their list suggested they were going to try and integrate it with dynd. (ha! good catch Nathaniel -- sorry for double dipping, Internet initially suggested they were waiting to see what the numpy team would do) On Fri Nov 28 2014 at 5:52:19 PM Benjamin Root wrote: > That being said, doesn't Pandas support something like what you are asking > for? While Pandas would like to push the NA support down into the NumPy > level for seamless interaction with other SciPy libraries, it does a very > decent job with NA on its own, and depending on the sort of code you need > to write, you might rarely ever have to directly interface with NumPy. > Pandas is very powerful for data analysis, particularly for those coming > from R and S+. Those of us who use NumPy directly tend to be involved in > more generalized numerical tasks than those developing Pandas. > > Perhaps you should raise your concerns on the Pandas mailing list? They > might be better suited to help you with any hurdles that you foresee. The > people on that list are very familiar with dealing with "dirty data". > > Cheers! > Ben Root > > > > On Fri, Nov 28, 2014 at 8:08 PM, Nathaniel Smith wrote: > >> On Fri, Nov 28, 2014 at 9:12 PM, Nick Eubank >> wrote: >> > Hello Numpy! >> > >> > Curious if there were any timelines for when NA / NaN support might >> arrive >> > for Ints. >> > >> > (I'm sure this is asked and answered many times, but what I can find >> seems >> > very out of date.) >> > >> > I'm strongly considering migrating from R/Stata/Matlab to Pandas/Numpy, >> but >> > the inability to handle Na in Int types seems like a big limitation for >> > someone always working with dirty data like me (a social scientist), >> and the >> > Pandas docs suggest they're waiting for Numpy to add support. Just >> curious >> > when that might happen! >> >> There's no particular timeline, no. We don't have any budget or >> employees so any timeline would be a fantasy anyway :-). It's >> definitely possible to do and we'd like to do it, but it'll happen >> when someone with the right skills and interests finds the time to do >> the work. >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Fri Nov 28 22:21:32 2014 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Fri, 28 Nov 2014 19:21:32 -0800 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: Message-ID: On Fri, Nov 28, 2014 at 5:15 PM, Nathaniel Smith wrote: > On Fri, Nov 28, 2014 at 3:15 AM, Alexander Belopolsky > wrote: > > I probably miss something very basic, but how given two arrays a and b, > can > > I find positions in a where elements of b are located? If a were > sorted, I > > could use searchsorted, but I don't want to get valid positions for > elements > > that are not in a. In my case, a has unique elements, but in the general > > case I would accept the first match. In other words, I am looking for an > > array analog of list.index() method. > > How about this? > > def index(haystack, needle): > haystack = np.asarray(haystack) > haystack_sort = np.argsort(haystack) > haystack_sorted = haystack[haystack_sort] > return haystack_sort[np.searchsorted(haystack_sorted, needle)] > > (Note that this will return incorrect results if any entries in needle > are missing from haystack entirely. If this is a concern then you need > to do some extra error-checking on the searchsorted return value.) > I like this approach a lot. You can actually skip the creation of the haystack_sorted array using the sorter kwarg: idx = haystack_sort[np.searchsorted(haystack, needle, sorter=haystack_sort)] But either using haystack_sorted or not, if any item in the needle is larger than the largest entry in the haystack, the indexing will error out with an index out of bounds. So the whole thing with proper error checking gets kind of messy, something along the lines of: sorted_idx = np.searchsorted(haystack, needle, sorter=haystack_sort) mask_idx = sorted_idx < len(haystack) idx = haystack_sort[sorted_idx[mask_idx]] mask_in_haystack = haystack[idx] == needle[mask_idx] mask_idx[mask_idx] &= mask_in_haystack So using -1 to indicate items in needle not found in haystack, you could do: ret = np.empty_like(needle, dtype=np.intp) ret[~mask_idx] = -1 ret[mask_idx] = idx[mask_in_haystack] In the end, it does get kind of messy, but I am not sure how could it be improved. Perhaps giving searchsorted an option to figure out the exact matches? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Nov 28 22:23:09 2014 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 28 Nov 2014 22:23:09 -0500 Subject: [Numpy-discussion] Updates on NA / NaN values for Ints? In-Reply-To: References: Message-ID: Ah, I didn't notice the coercion before. Dynnd has been an interesting project for a couple years now, and it is in the right position of being able to develop and experiment with new concepts. I hope they can work out the NA/NaN/mask semantics in a way that is intuitive and can coexist. That was the major sticking point in previous discussions that prevented its incorporation. Cheers! Ben Root On Nov 28, 2014 9:02 PM, "Nick Eubank" wrote: > Thanks Nathaniel and Ben. > > Nathaniel, I completely understand! Believe me, I appreciate all you folks > do already, and am hoping to some day improve my skills enough to > contribute myself. > > Ben: they only handle it for Floats. If you introduce an NA to an int64 > series, it just coerces the type into Float64. > > With that said, someone on their list suggested they were going to try and > integrate it with dynd. > > (ha! good catch Nathaniel -- sorry for double dipping, Internet initially > suggested they were waiting to see what the numpy team would do) > > On Fri Nov 28 2014 at 5:52:19 PM Benjamin Root wrote: > >> That being said, doesn't Pandas support something like what you are >> asking for? While Pandas would like to push the NA support down into the >> NumPy level for seamless interaction with other SciPy libraries, it does a >> very decent job with NA on its own, and depending on the sort of code you >> need to write, you might rarely ever have to directly interface with NumPy. >> Pandas is very powerful for data analysis, particularly for those coming >> from R and S+. Those of us who use NumPy directly tend to be involved in >> more generalized numerical tasks than those developing Pandas. >> >> Perhaps you should raise your concerns on the Pandas mailing list? They >> might be better suited to help you with any hurdles that you foresee. The >> people on that list are very familiar with dealing with "dirty data". >> >> Cheers! >> Ben Root >> >> >> >> On Fri, Nov 28, 2014 at 8:08 PM, Nathaniel Smith wrote: >> >>> On Fri, Nov 28, 2014 at 9:12 PM, Nick Eubank >>> wrote: >>> > Hello Numpy! >>> > >>> > Curious if there were any timelines for when NA / NaN support might >>> arrive >>> > for Ints. >>> > >>> > (I'm sure this is asked and answered many times, but what I can find >>> seems >>> > very out of date.) >>> > >>> > I'm strongly considering migrating from R/Stata/Matlab to >>> Pandas/Numpy, but >>> > the inability to handle Na in Int types seems like a big limitation for >>> > someone always working with dirty data like me (a social scientist), >>> and the >>> > Pandas docs suggest they're waiting for Numpy to add support. Just >>> curious >>> > when that might happen! >>> >>> There's no particular timeline, no. We don't have any budget or >>> employees so any timeline would be a fantasy anyway :-). It's >>> definitely possible to do and we'd like to do it, but it'll happen >>> when someone with the right skills and interests finds the time to do >>> the work. >>> >>> -n >>> >>> -- >>> Nathaniel J. Smith >>> Postdoctoral researcher - Informatics - University of Edinburgh >>> http://vorpus.org >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Nov 28 22:26:37 2014 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 28 Nov 2014 22:26:37 -0500 Subject: [Numpy-discussion] Finding values in an array In-Reply-To: References: Message-ID: If we don't have an operation for this in numpy's setops module, it probably should be added. Ben Root On Nov 28, 2014 10:21 PM, "Jaime Fern?ndez del R?o" wrote: > On Fri, Nov 28, 2014 at 5:15 PM, Nathaniel Smith wrote: > >> On Fri, Nov 28, 2014 at 3:15 AM, Alexander Belopolsky >> wrote: >> > I probably miss something very basic, but how given two arrays a and b, >> can >> > I find positions in a where elements of b are located? If a were >> sorted, I >> > could use searchsorted, but I don't want to get valid positions for >> elements >> > that are not in a. In my case, a has unique elements, but in the >> general >> > case I would accept the first match. In other words, I am looking for >> an >> > array analog of list.index() method. >> >> How about this? >> >> def index(haystack, needle): >> haystack = np.asarray(haystack) >> haystack_sort = np.argsort(haystack) >> haystack_sorted = haystack[haystack_sort] >> return haystack_sort[np.searchsorted(haystack_sorted, needle)] >> >> (Note that this will return incorrect results if any entries in needle >> are missing from haystack entirely. If this is a concern then you need >> to do some extra error-checking on the searchsorted return value.) >> > > I like this approach a lot. You can actually skip the creation of the > haystack_sorted array using the sorter kwarg: > > idx = haystack_sort[np.searchsorted(haystack, needle, > sorter=haystack_sort)] > > But either using haystack_sorted or not, if any item in the needle is > larger than the largest entry in the haystack, the indexing will error out > with an index out of bounds. So the whole thing with proper error checking > gets kind of messy, something along the lines of: > > sorted_idx = np.searchsorted(haystack, needle, sorter=haystack_sort) > mask_idx = sorted_idx < len(haystack) > idx = haystack_sort[sorted_idx[mask_idx]] > mask_in_haystack = haystack[idx] == needle[mask_idx] > mask_idx[mask_idx] &= mask_in_haystack > > So using -1 to indicate items in needle not found in haystack, you could > do: > > ret = np.empty_like(needle, dtype=np.intp) > ret[~mask_idx] = -1 > ret[mask_idx] = idx[mask_in_haystack] > > In the end, it does get kind of messy, but I am not sure how could it be > improved. Perhaps giving searchsorted an option to figure out the exact > matches? > > Jaime > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Sat Nov 29 20:49:47 2014 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sat, 29 Nov 2014 20:49:47 -0500 Subject: [Numpy-discussion] packbits / unpackbits bugs or misfeatures? Message-ID: <547A77BB.5010900@gmail.com> Six years ago Zachary Pincus wrote: > it would kind of make more sense, at least to me, for > unpackbits to return a boolean array, and packbits to take > boolean arrays. I could try to look into fixing the code > instead of the docs, if others agree. I agree with this. (Better late than never?) More relevant at this point: it seems very odd that `packbits` does not *accept* a boolean array, even if it does not insist upon one. Might this be considered a bug? Cheers, Alan Isaac From charlesr.harris at gmail.com Sat Nov 29 21:05:40 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 29 Nov 2014 19:05:40 -0700 Subject: [Numpy-discussion] packbits / unpackbits bugs or misfeatures? In-Reply-To: <547A77BB.5010900@gmail.com> References: <547A77BB.5010900@gmail.com> Message-ID: On Sat, Nov 29, 2014 at 6:49 PM, Alan G Isaac wrote: > Six years ago Zachary Pincus wrote: > > it would kind of make more sense, at least to me, for > > unpackbits to return a boolean array, and packbits to take > > boolean arrays. I could try to look into fixing the code > > instead of the docs, if others agree. > > > I agree with this. (Better late than never?) > More relevant at this point: it seems very odd > that `packbits` does not *accept* a boolean array, > even if it does not insist upon one. Might this > be considered a bug? > > Cheers, > Alan Isaac > Both packbits and unpack bits are being refactored at this very moment, you might want to comment there . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From radhakrishnan.mohan at gmail.com Sun Nov 30 02:36:08 2014 From: radhakrishnan.mohan at gmail.com (Mohan Radhakrishnan) Date: Sun, 30 Nov 2014 13:06:08 +0530 Subject: [Numpy-discussion] Maximum-margin linear classifier Message-ID: Hi, I am a beginner. I have 4 points and wish to plot the classifier separation line. I believe this code without the plot is correct. Can I look at sample plotting code(with basic explanation) for this simple SVM ? X = [[3,3], [7,2], [5,4], [8,3]] Y = [-1, -1, 1, 1] clf = svm.SVC(kernel='linear',C=2) clf.fit(X,Y) for i in range(4): x = np.array(X[i])[0]*clf.coef_[0][0] + np.array(X[i])[1]*clf.coef_[0][1] + clf.intercept_; print x print clf.intercept_ print clf.coef_ Thanks, Mohan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Nov 30 06:54:27 2014 From: cournape at gmail.com (David Cournapeau) Date: Sun, 30 Nov 2014 11:54:27 +0000 Subject: [Numpy-discussion] GDB macros for numpy Message-ID: Hi there, I remember having seen some numpy-aware gdb macros at some point, but cannot find any reference. Does anyone know of any ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Nov 30 09:22:43 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 30 Nov 2014 14:22:43 +0000 Subject: [Numpy-discussion] Maximum-margin linear classifier In-Reply-To: References: Message-ID: Hi Mohan, While you might get responses here, you'll probably have better luck asking on the scikit-learn list. (I assume you are using scikit-learn for the svm?) That's where you'll find experts on the details of these models hanging out - this is the list for numpy itself, and numpy doesn't include svms or plotting support directly. (StackOverflow is also sometimes a good place to try for general questions that aren't really specific to any one project.) Good luck! -n On 30 Nov 2014 07:36, "Mohan Radhakrishnan" wrote: > Hi, > I am a beginner. > > I have 4 points and wish to plot the classifier separation line. I believe > this code without the plot is correct. > > Can I look at sample plotting code(with basic explanation) for this simple > SVM ? > > X = [[3,3], [7,2], [5,4], [8,3]] > > Y = [-1, -1, 1, 1] > > clf = svm.SVC(kernel='linear',C=2) > > clf.fit(X,Y) > > > for i in range(4): > > x = np.array(X[i])[0]*clf.coef_[0][0] + np.array(X[i])[1]*clf.coef_[0 > ][1] + clf.intercept_; > > print x > > > print clf.intercept_ > > print clf.coef_ > > Thanks, > Mohan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Nov 30 12:45:00 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 30 Nov 2014 10:45:00 -0700 Subject: [Numpy-discussion] GDB macros for numpy In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau wrote: > Hi there, > > I remember having seen some numpy-aware gdb macros at some point, but > cannot find any reference. Does anyone know of any ? > What would numpy aware gdb macros be? Never heard of them. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.eberspaecher at gmail.com Sun Nov 30 15:06:10 2014 From: alex.eberspaecher at gmail.com (=?ISO-8859-15?Q?Alexander_Ebersp=E4cher?=) Date: Sun, 30 Nov 2014 21:06:10 +0100 Subject: [Numpy-discussion] GDB macros for numpy In-Reply-To: References: Message-ID: <547B78B2.3070004@gmail.com> On 30.11.2014 12:54, David Cournapeau wrote: > I remember having seen some numpy-aware gdb macros at some point, but > cannot find any reference. Does anyone know of any ? Though I have no idea what a NumPy-aware GDB macro looked like, I'd be interested in other people's input on debugging cross-language projects (Cython/Python/C/Fortran) involving NumPy. I use cygdb but find it hard (or rather impossible) to maintain debug builds of all parts of the toolchain I use. Python itself is fine, but how do you deal with debug builds for the scientific parts of the toolchain if you build from source? Thanks for your answers! Alex From cournape at gmail.com Sun Nov 30 16:41:07 2014 From: cournape at gmail.com (David Cournapeau) Date: Sun, 30 Nov 2014 21:41:07 +0000 Subject: [Numpy-discussion] GDB macros for numpy In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris wrote: > > > On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau > wrote: > >> Hi there, >> >> I remember having seen some numpy-aware gdb macros at some point, but >> cannot find any reference. Does anyone know of any ? >> > > What would numpy aware gdb macros be? Never heard of them. > Python itself has some gdb macros ( https://wiki.python.org/moin/DebuggingWithGdb). You can do things like: # p is a PyObject * pointer to the list [1, 2] > pyo p [1, 2] The idea would be to have a nice representation for arrays, dtypes, etc... David > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sun Nov 30 19:44:31 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 1 Dec 2014 01:44:31 +0100 Subject: [Numpy-discussion] GDB macros for numpy In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 10:41 PM, David Cournapeau wrote: > > > On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau >> wrote: >> >>> Hi there, >>> >>> I remember having seen some numpy-aware gdb macros at some point, but >>> cannot find any reference. Does anyone know of any ? >>> >> >> What would numpy aware gdb macros be? Never heard of them. >> > > Python itself has some gdb macros ( > https://wiki.python.org/moin/DebuggingWithGdb). > > You can do things like: > > # p is a PyObject * pointer to the list [1, 2] > > pyo p > [1, 2] > > The idea would be to have a nice representation for arrays, dtypes, etc... > > David > > this wiki entry is very very old or not aware of the official python gdb macros (which are built into gdb in typical linux distributions, no extra configuration required). and the macros are awesome, they are extremely helpful in debugging python extension codes. The current commands are: py-list py-bt py-print py-locals py-up/py-down having similar macros for numpy types might be useful, but at least I more often than not one does want the actual C structure than a pretty representation which is just *(type*)PyArray_DATA(arr)@10 away. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sun Nov 30 19:50:59 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 1 Dec 2014 01:50:59 +0100 Subject: [Numpy-discussion] GDB macros for numpy In-Reply-To: References: Message-ID: On Mon, Dec 1, 2014 at 1:44 AM, Julian Taylor wrote: > > On Sun, Nov 30, 2014 at 10:41 PM, David Cournapeau > wrote: > >> >> >> On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau >>> wrote: >>> >>>> Hi there, >>>> >>>> I remember having seen some numpy-aware gdb macros at some point, but >>>> cannot find any reference. Does anyone know of any ? >>>> >>> >>> What would numpy aware gdb macros be? Never heard of them. >>> >> >> Python itself has some gdb macros ( >> https://wiki.python.org/moin/DebuggingWithGdb). >> >> You can do things like: >> >> # p is a PyObject * pointer to the list [1, 2] >> > pyo p >> [1, 2] >> >> The idea would be to have a nice representation for arrays, dtypes, etc... >> >> David >> >> > this wiki entry is very very old or not aware of the official python gdb > macros (which are built into gdb in typical linux distributions, no extra > configuration required). > > ups I scrolled to far to the legacy section, the earlier parts do mention the new macros :) -------------- next part -------------- An HTML attachment was scrubbed... URL: