From sturla.molden at gmail.com Fri Jul 1 05:16:10 2016 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 01 Jul 2016 11:16:10 +0200 Subject: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels? In-Reply-To: References: Message-ID: On 29/06/16 21:55, Nathaniel Smith wrote: > Accelerate is closed, so when we hit bugs then there's often nothing we > can do except file a bug with apple and hope that it gets fixed within a > year or two. This isn't hypothetical -- we've hit cases where accelerate > gave wrong answers. Numpy actually carries some scary code right now to > work around one of these bugs by monkeypatching (!) accelerate using > dynamic linker trickiness. And, of course, there's the thing where > accelerate totally breaks multiprocessing. Yes, those are the cons. > Apple has said that they don't consider this a bug. Theoretically they are right, but from a practical perspective... Sturla From sturla.molden at gmail.com Fri Jul 1 05:18:48 2016 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 01 Jul 2016 11:18:48 +0200 Subject: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels? In-Reply-To: References: Message-ID: On 29/06/16 21:55, Nathaniel Smith wrote: > Speed is important, but it's far from the only consideration, especially > since differences between the top tier libraries are usually rather > small. It is not even the more important consideration. I would say that correctness matters most. Everything else comes second. Sturla From ralf.gommers at gmail.com Fri Jul 1 17:54:47 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 1 Jul 2016 23:54:47 +0200 Subject: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels? In-Reply-To: <374618154488923939.947496sturla.molden-gmail.com@news.gmane.org> References: <374618154488923939.947496sturla.molden-gmail.com@news.gmane.org> Message-ID: On Wed, Jun 29, 2016 at 11:06 PM, Sturla Molden wrote: > Ralf Gommers wrote: > > > For most routines performance seems to be comparable, and both are much > > better than ATLAS. When there's a significant difference, I have the > > impression that OpenBLAS is more often the slower one (example: > > > href="https://github.com/xianyi/OpenBLAS/issues/533"> > https://github.com/xianyi/OpenBLAS/issues/533). > > Accelerate is in general better optimized for level-1 and level-2 BLAS than > OpenBLAS. There are two reasons for this: > > First, OpenBLAS does not use AVX for these kernels, but Accelerate does. > This is the more important difference. It seems the OpenBLAS devs are now > working on this. > > Second, the thread pool in OpenBLAS is not as scalable on small tasks as > the "Grand Central Dispatch" (GCD) used by Accelerate. The GCD thread-pool > used by Accelerate is actually quite unique in having a very tiny overhead: > It takes only 16 extra opcodes (IIRC) for running a task on the global > parallel queue instead of the current thread. (Even if my memory is not > perfect and it is not exactly 16 opcodes, it is within that order of > magnitude.) GCD can do this because the global queues and threadpool is > actually built into the kernel of the OS. On the other hand, OpenBLAS and > MKL depends on thread pools managed in userspace, for which the scheduler > in the OS have no special knowledge. When you need fine-grained parallelism > and synchronization, there is nothing like GCD. Even a user-space spinlock > will have bigger overhead than a sequential queue in GCD. With a userspace > threadpool all threads are scheduled on a round robin basis, but with GCD > the scheduler has special knowledge about the tasks put on the queues, and > executes them as fast as possible. Accelerate therefore has an unique > advantage when running level-1 and 2 BLAS routines, with which OpenBLAS or > MKL probably never can properly compete. Programming with GCD can actually > often be counter-intuitive to someone used to deal with OpenMP, MPI or > pthreads. For example it is often better to enqueue a lot of small tasks > instead of splitting up the computation into large chunks of work. When > parallelising a tight loop, a chunk size of 1 can be great on GCD but is > likely to be horrible on OpenMP and anything else that has userspace > threads. > Thanks Sturla, interesting details as always. You didn't state your preference by the way, do you have one? We're building binaries for the average user, so I'd say the AVX thing is of relevance for the decision to be made, the GCD one less so (people who care about that will not have any trouble building their own numpy). So far the score is: one +1, one +0.5, one +0, one -1 and one "still a bit nervous". Any other takers? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Jul 1 19:55:24 2016 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 1 Jul 2016 23:55:24 +0000 (UTC) Subject: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels? References: <374618154488923939.947496sturla.molden-gmail.com@news.gmane.org> Message-ID: <1784305886489108487.737048sturla.molden-gmail.com@news.gmane.org> Ralf Gommers wrote: > Thanks Sturla, interesting details as always. You didn't state your > preference by the way, do you have one? I use Accelerate because it is the easier for me to use when building SciPy. But that is from a developer's perspective. As you know, Accelerate breaks a common (ab)use of multiprocessing on POSIX systems. While the bug is strictly speaking in multiprocessing (but partially fixed in Python 3.4 and later), it is still a nasty surprise to many users. E.g. a call to np.dot never returns, and there is no error message indicating why. That speaks against using it in the wheels. Accelerate, like MKL and FFTW, has nifty FFTs. If we start to use MKL and Accelerate for numpy.fft (which I sometimes have fantacies about), that would shift the balance the other way, in favour of Accelerate. Speed wise Accelerate wins for things like dot product of two vectors or multiplication of a vector and a matrix. For general matrix multiplication the performance is about the same, except when matrices are very small and Accelerate can benefit from the tiny GCD overhead. But then the Python overhead probably dominates, so they are going to be about equal anyway. I am going to vote ? 0. I am really not sure which will be the better for the binary wheels. They seem about equal to me right now. There are pros and cons with either. Sturla From leon.woo at db.com Sat Jul 2 08:02:15 2016 From: leon.woo at db.com (Leon Woo) Date: Sat, 2 Jul 2016 14:02:15 +0200 Subject: [Numpy-discussion] AUTO: Leon Woo is out of the office (returning 11/07/2016) Message-ID: I am out of the office until 11/07/2016. For standard requests within the scope of EMG PWM Berlin, please write to EMG PWM Berlin at DBEMEA. For non-standard requests, please cc Hien Pham-Thu. Note: This is an automated response to your message "NumPy-Discussion Digest, Vol 118, Issue 2" sent on 02.07.2016 14:00:01. This is the only notification you will receive while this person is away.-- Informationen (einschlie?lich Pflichtangaben) zu einzelnen, innerhalb der EU t?tigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enth?lt vertrauliche und/ oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Sat Jul 2 22:08:30 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 3 Jul 2016 02:08:30 +0000 (UTC) Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key Message-ID: (I'm probably going to botch the description...) Suppose I have a 2D array of Python objects, the first n elements of each row form a key, the rest of the elements form the value. Each key can (and generally does) occur multiple times. I'd like to generate a new array consisting of just the first (or last) row for each key occurrence. Rows retain their relative order on output. For example, suppose I have this array with key length 2: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] Selecting the first occurrence of each key would return this array: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 17, 100.3 ] while selecting the last occurrence would return this array: [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] In real life, my array is a bit larger than this example, with the input being on the order of a million rows, and the output being around 5000 rows. Avoiding processing all those extra rows at the Python level would speed things up. I don't know what this filter might be called (though I'm sure I haven't thought of something new), so searching Google or Bing for it would seem to be fruitless. It strikes me as something which numpy or Pandas might already have in their bag(s) of tricks. Pointers appreciated, Skip From jni.soma at gmail.com Sat Jul 2 22:18:00 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Sat, 2 Jul 2016 22:18:00 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: Hey Skip, Any way that you can make your keys numeric? Then you can run np.diff on that first column, and use the indices of nonzero entries (np.flatnonzero) to know where values change. With a +1/-1 offset (that I am too lazy to figure out right now ;) you can then index into the original rows to get either the first or last occurrence of each run. Juan. On 2 July 2016 at 10:10:16 PM, Skip Montanaro (skip.montanaro at gmail.com) wrote: (I'm probably going to botch the description...) Suppose I have a 2D array of Python objects, the first n elements of each row form a key, the rest of the elements form the value. Each key can (and generally does) occur multiple times. I'd like to generate a new array consisting of just the first (or last) row for each key occurrence. Rows retain their relative order on output. For example, suppose I have this array with key length 2: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] Selecting the first occurrence of each key would return this array: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 17, 100.3 ] while selecting the last occurrence would return this array: [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] In real life, my array is a bit larger than this example, with the input being on the order of a million rows, and the output being around 5000 rows. Avoiding processing all those extra rows at the Python level would speed things up. I don't know what this filter might be called (though I'm sure I haven't thought of something new), so searching Google or Bing for it would seem to be fruitless. It strikes me as something which numpy or Pandas might already have in their bag(s) of tricks. Pointers appreciated, Skip _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Sun Jul 3 10:13:18 2016 From: alan.isaac at gmail.com (Alan Isaac) Date: Sun, 3 Jul 2016 10:13:18 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: <76560c10-70fa-b863-86a8-062a57684fc9@gmail.com> 1. This is not a NumPy question; StackExchange would be more appropriate. 2. Do some bookkeeping: def initialKeyFilter(data, length, invert=False): result = list() seen = set() if invert: data = reversed(data) for datum in data: k = tuple(datum[:length]) if (k not in seen): result.append(datum) seen.add(k) if invert: result.reverse() return result Cheers, Alan Isaac From tom.kooij at gmail.com Mon Jul 4 16:25:41 2016 From: tom.kooij at gmail.com (Tom Kooij) Date: Mon, 4 Jul 2016 22:25:41 +0200 Subject: [Numpy-discussion] ANN: PyTables 3.2.3 released. Message-ID: =========================== Announcing PyTables 3.2.3 =========================== We are happy to announce PyTables 3.2.3. What's new ========== This is a bug fix release. It solves many issues reported in the months since the release of 3.2.2. In case you want to know more in detail what has changed in this version, please refer to: http://www.pytables.org/release_notes.html For an online version of the manual, visit: http://www.pytables.org/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org Github: http://www.github.com/PyTables/PyTables About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Jul 4 19:22:13 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 4 Jul 2016 23:22:13 +0000 (UTC) Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key References: <76560c10-70fa-b863-86a8-062a57684fc9@gmail.com> Message-ID: > 1. This is not a NumPy question; StackExchange would be more appropriate. Thanks, that is the fairly straightforward -- but slow -- solution, which I have already implemented. I was asking if numpy had some filtering functions which might speed things up (it's a huge library, with which I'm not terribly familiar). It's fine if the answer is "no". From skip.montanaro at gmail.com Mon Jul 4 19:27:14 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 4 Jul 2016 23:27:14 +0000 (UTC) Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key References: Message-ID: > Any way that you can make your keys numeric? Then you can run np.diff on > that first column, and use the indices of nonzero entries (np.flatnonzero) > to know where values change. With a +1/-1 offset (that I am too lazy to > figure out right now ;) you can then index into the original rows to get > either the first or last occurrence of each run. I'll give it some thought, but one of the elements of the key is definitely a (short, < six characters) string. Hashing it probably wouldn't work, too great a chance for collisions. S From jeffreback at gmail.com Mon Jul 4 19:31:11 2016 From: jeffreback at gmail.com (Jeff Reback) Date: Mon, 4 Jul 2016 19:31:11 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: This is trivial in pandas. a simple groupby. In [6]: data = [[ 'a', 27, 14.5 ],['b', 12, 99.0],['a', 17, 100.3], ['b', 12, -329.0]] In [7]: df = DataFrame(data, columns=list('ABC')) In [8]: df Out[8]: A B C 0 a 27 14.5 1 b 12 99.0 2 a 17 100.3 3 b 12 -329.0 In [9]: df.groupby('A').first() Out[9]: B C A a 27 14.5 b 12 99.0 In [10]: df.groupby('A').last() Out[10]: B C A a 17 100.3 b 12 -329.0 On Mon, Jul 4, 2016 at 7:27 PM, Skip Montanaro wrote: > > Any way that you can make your keys numeric? Then you can run np.diff on > > that first column, and use the indices of nonzero entries > (np.flatnonzero) > > to know where values change. With a +1/-1 offset (that I am too lazy to > > figure out right now ;) you can then index into the original rows to get > > either the first or last occurrence of each run. > > I'll give it some thought, but one of the elements of the key is definitely > a (short, < six characters) string. Hashing it probably wouldn't work, too > great a chance for collisions. > > S > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Jul 4 19:38:31 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 4 Jul 2016 23:38:31 +0000 (UTC) Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key References: Message-ID: > This is trivial in pandas. a simple groupby. Oh, cool. Precisely the sort of solution I was hoping would turn up. Straightforward, easy for a novice data slinger like me to understand, and likely a bazillion times faster than the straightforward version. Skip From jni.soma at gmail.com Tue Jul 5 00:49:41 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Tue, 5 Jul 2016 00:49:41 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: On 4 July 2016 at 7:38:48 PM, Skip Montanaro (skip.montanaro at gmail.com) wrote: Oh, cool. Precisely the sort of solution I was hoping would turn up. Except it doesn?t seem to meet your original spec, which retrieved the first item of each *run* of an index value? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Tue Jul 5 01:03:59 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Tue, 5 Jul 2016 01:03:59 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: On 4 July 2016 at 7:27:47 PM, Skip Montanaro (skip.montanaro at gmail.com) wrote: Hashing it probably wouldn't work, too great a chance for collisions. If the string is ASCII, you can always interpret the bytes as part of an 8 byte integer. Or, you can map unique values to consecutive integers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Jul 5 02:50:13 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 5 Jul 2016 02:50:13 -0400 Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key In-Reply-To: References: Message-ID: On Tue, Jul 5, 2016 at 1:03 AM, Juan Nunez-Iglesias wrote: > On 4 July 2016 at 7:27:47 PM, Skip Montanaro (skip.montanaro at gmail.com) > wrote: > > Hashing it probably wouldn't work, too > great a chance for collisions. > > > If the string is ASCII, you can always interpret the bytes as part of an 8 > byte integer. Or, you can map unique values to consecutive integers. > IIUC np.nonzero(a[1] == a[:-1]) gives all changes independent of dtype. add or remove a 1 to adjust which element is indexed. (IIRC from a long time ago, arraysetops used/uses something like this.) Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Tue Jul 5 04:22:48 2016 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Tue, 5 Jul 2016 10:22:48 +0200 Subject: [Numpy-discussion] numpy.dtype has the wrong size, try recompiling Message-ID: This question on Stack Overflow: http://stackoverflow.com/questions/38197086/sklearn-numpy-dtype-has-the-wrong-size-try-recompiling-in-both-pycharm-and-te If I remember correctly this has something to do with Cython, right? Can't remeber the details though, can someone help that poor soul out? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Jul 5 04:29:17 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 5 Jul 2016 04:29:17 -0400 Subject: [Numpy-discussion] numpy.dtype has the wrong size, try recompiling In-Reply-To: References: Message-ID: On Tue, Jul 5, 2016 at 4:22 AM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > This question on Stack Overflow: > > > http://stackoverflow.com/questions/38197086/sklearn-numpy-dtype-has-the-wrong-size-try-recompiling-in-both-pycharm-and-te > > If I remember correctly this has something to do with Cython, right? Can't > remeber the details though, can someone help that poor soul out? > from what I understood many years ago (based mainly on what David Cournapeau was saying) http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling Josef > > Jaime > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Tue Jul 5 08:31:09 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 5 Jul 2016 12:31:09 +0000 (UTC) Subject: [Numpy-discussion] Picking rows with the first (or last) occurrence of each key References: Message-ID: > Except it doesn?t seem to meet your original spec, which retrieved > the first item of each *run* of an index value? No, just what I was looking for. As I indicated in my original post, "I'm probably going to botch the description..." It's quite likely that my problem specification wasn't as crisp as it should have been. S From klonuo at gmail.com Tue Jul 5 15:18:56 2016 From: klonuo at gmail.com (klo uo) Date: Tue, 5 Jul 2016 21:18:56 +0200 Subject: [Numpy-discussion] f2py output module name Message-ID: Hi, I'm following this guide: http://docs.scipy.org/doc/numpy-dev/f2py/getting-started.html#the-quick-and-smart-way I'm on Windows with gfortran and VS2015. When I run: f2py -c -m fib3 fib3.f as output I dont get "fib3.pyd", but "fib3.cp35-win_amd64.pyd". Does anyone know how to get correctly named module in this case? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 5 15:59:07 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 5 Jul 2016 21:59:07 +0200 Subject: [Numpy-discussion] f2py output module name In-Reply-To: References: Message-ID: On Tue, Jul 5, 2016 at 9:18 PM, klo uo wrote: > Hi, I'm following this guide: > http://docs.scipy.org/doc/numpy-dev/f2py/getting-started.html#the-quick-and-smart-way > > I'm on Windows with gfortran and VS2015. When I run: > > f2py -c -m fib3 fib3.f > > as output I dont get "fib3.pyd", but "fib3.cp35-win_amd64.pyd". > > Does anyone know how to get correctly named module in this case? > That is the right output, see https://www.python.org/dev/peps/pep-3149. You can check the expected extension tag with `sysconfig.get_config_var('SOABI')`. There may be a way to turn that off, but why would you? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From klonuo at gmail.com Tue Jul 5 16:14:21 2016 From: klonuo at gmail.com (klo uo) Date: Tue, 5 Jul 2016 22:14:21 +0200 Subject: [Numpy-discussion] f2py output module name In-Reply-To: References: Message-ID: So on Linux too, there is "ABI" suffix too, for generated module... I misunderstood. I was renaming generated module to "fib3.pyd" to be able to do "import fib3", but now I see it's not necessary - it's importable the same regardless the name of generated module :) Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 00:09:29 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 00:09:29 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d Message-ID: Hi, I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a function np.atleast_nd in PR#7804 (https://github.com/numpy/numpy/pull/7804). As a result of this PR, I have a couple of questions about `np.atleast_3d`. `np.atleast_3d` appears to do something weird with the dimensions: If the input is 1D, it prepends and appends a size-1 dimension. If the input is 2D, it appends a size-1 dimension. This is inconsistent with `np.atleast_2d`, which always prepends (as does `np.atleast_nd`). - Is there any reason for this behavior? - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in terms of `np.atleast_nd`, which is actually much simpler)? This would be a slight API change since the output would not be exactly the same. Thanks, -Joe From njs at pobox.com Wed Jul 6 01:06:44 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 5 Jul 2016 22:06:44 -0700 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" wrote: > > Hi, > > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a > function np.atleast_nd in PR#7804 > (https://github.com/numpy/numpy/pull/7804). > > As a result of this PR, I have a couple of questions about > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > the dimensions: If the input is 1D, it prepends and appends a size-1 > dimension. If the input is 2D, it appends a size-1 dimension. This is > inconsistent with `np.atleast_2d`, which always prepends (as does > `np.atleast_nd`). > > - Is there any reason for this behavior? > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > terms of `np.atleast_nd`, which is actually much simpler)? This would > be a slight API change since the output would not be exactly the same. Changing atleast_3d seems likely to break a bunch of stuff... Beyond that, I find it hard to have an opinion about the best design for these functions, because I don't think I've ever encountered a situation where they were actually what I wanted. I'm not a big fan of coercing dimensions in the first place, for the usual "refuse to guess" reasons. And then generally if I do want to coerce an array to another dimension, then I have some opinion about where the new dimensions should go, and/or I have some opinion about the minimum acceptable starting dimension, and/or I have a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that requirements list.) I don't know how typical I am in this. But it does make me wonder if the atleast_* functions act as an attractive nuisance, where new users take their presence as an implicit recommendation that they are actually a useful thing to reach for, even though they... aren't that. And maybe we should be recommending folk move away from them rather than trying to extend them further? Or maybe they're totally useful and I'm just missing it. What's your use case that motivates atleast_nd? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 6 02:21:10 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 6 Jul 2016 08:21:10 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > wrote: > > > > Hi, > > > > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a > > function np.atleast_nd in PR#7804 > > (https://github.com/numpy/numpy/pull/7804). > > > > As a result of this PR, I have a couple of questions about > > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > > the dimensions: If the input is 1D, it prepends and appends a size-1 > > dimension. If the input is 2D, it appends a size-1 dimension. This is > > inconsistent with `np.atleast_2d`, which always prepends (as does > > `np.atleast_nd`). > > > > - Is there any reason for this behavior? > > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > > terms of `np.atleast_nd`, which is actually much simpler)? This would > > be a slight API change since the output would not be exactly the same. > > Changing atleast_3d seems likely to break a bunch of stuff... > > Beyond that, I find it hard to have an opinion about the best design for > these functions, because I don't think I've ever encountered a situation > where they were actually what I wanted. I'm not a big fan of coercing > dimensions in the first place, for the usual "refuse to guess" reasons. And > then generally if I do want to coerce an array to another dimension, then I > have some opinion about where the new dimensions should go, and/or I have > some opinion about the minimum acceptable starting dimension, and/or I have > a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; > 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that > requirements list.) > > I don't know how typical I am in this. But it does make me wonder if the > atleast_* functions act as an attractive nuisance, where new users take > their presence as an implicit recommendation that they are actually a > useful thing to reach for, even though they... aren't that. And maybe we > should be recommending folk move away from them rather than trying to > extend them further? > > Or maybe they're totally useful and I'm just missing it. What's your use > case that motivates atleast_nd? > I think you're just missing it:) atleast_1d/2d are used quite a bit in Scipy and Statsmodels (those are the only ones I checked), and in the large majority of cases it's the best thing to use there. There's a bunch of atleast_2d calls with a transpose appended because the input needs to be treated as columns instead of rows, but that's still efficient and readable enough. For 3D/nD I can see that you'd need more control over where the dimensions go, but 1D/2D are fine. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Jul 6 03:29:43 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 6 Jul 2016 03:29:43 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers wrote: > > > On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: > > On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" >> wrote: >> > >> > Hi, >> > >> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >> > function np.atleast_nd in PR#7804 >> > (https://github.com/numpy/numpy/pull/7804). >> > >> > As a result of this PR, I have a couple of questions about >> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >> > the dimensions: If the input is 1D, it prepends and appends a size-1 >> > dimension. If the input is 2D, it appends a size-1 dimension. This is >> > inconsistent with `np.atleast_2d`, which always prepends (as does >> > `np.atleast_nd`). >> > >> > - Is there any reason for this behavior? >> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >> > terms of `np.atleast_nd`, which is actually much simpler)? This would >> > be a slight API change since the output would not be exactly the same. >> >> Changing atleast_3d seems likely to break a bunch of stuff... >> >> Beyond that, I find it hard to have an opinion about the best design for >> these functions, because I don't think I've ever encountered a situation >> where they were actually what I wanted. I'm not a big fan of coercing >> dimensions in the first place, for the usual "refuse to guess" reasons. And >> then generally if I do want to coerce an array to another dimension, then I >> have some opinion about where the new dimensions should go, and/or I have >> some opinion about the minimum acceptable starting dimension, and/or I have >> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; >> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that >> requirements list.) >> >> I don't know how typical I am in this. But it does make me wonder if the >> atleast_* functions act as an attractive nuisance, where new users take >> their presence as an implicit recommendation that they are actually a >> useful thing to reach for, even though they... aren't that. And maybe we >> should be recommending folk move away from them rather than trying to >> extend them further? >> >> Or maybe they're totally useful and I'm just missing it. What's your use >> case that motivates atleast_nd? >> > I think you're just missing it:) atleast_1d/2d are used quite a bit in > Scipy and Statsmodels (those are the only ones I checked), and in the large > majority of cases it's the best thing to use there. There's a bunch of > atleast_2d calls with a transpose appended because the input needs to be > treated as columns instead of rows, but that's still efficient and readable > enough. > As Ralph pointed out its usage in statsmodels. I do find them useful as replacement for several lines of ifs and reshapes We stilll need in many cases the atleast_2d_cols, that appends the newaxis if necessary. roughly the equivalent of if x.ndim == 1: x = x[:, None] else: x = np.atleast_2d(x) Josef > > For 3D/nD I can see that you'd need more control over where the dimensions > go, but 1D/2D are fine. > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Jul 6 03:41:54 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 6 Jul 2016 03:41:54 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 3:29 AM, wrote: > > > On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: >> >> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" >>> wrote: >>> > >>> > Hi, >>> > >>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >>> > function np.atleast_nd in PR#7804 >>> > (https://github.com/numpy/numpy/pull/7804). >>> > >>> > As a result of this PR, I have a couple of questions about >>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >>> > the dimensions: If the input is 1D, it prepends and appends a size-1 >>> > dimension. If the input is 2D, it appends a size-1 dimension. This is >>> > inconsistent with `np.atleast_2d`, which always prepends (as does >>> > `np.atleast_nd`). >>> > >>> > - Is there any reason for this behavior? >>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >>> > terms of `np.atleast_nd`, which is actually much simpler)? This would >>> > be a slight API change since the output would not be exactly the same. >>> >>> Changing atleast_3d seems likely to break a bunch of stuff... >>> >>> Beyond that, I find it hard to have an opinion about the best design for >>> these functions, because I don't think I've ever encountered a situation >>> where they were actually what I wanted. I'm not a big fan of coercing >>> dimensions in the first place, for the usual "refuse to guess" reasons. And >>> then generally if I do want to coerce an array to another dimension, then I >>> have some opinion about where the new dimensions should go, and/or I have >>> some opinion about the minimum acceptable starting dimension, and/or I have >>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; >>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that >>> requirements list.) >>> >>> I don't know how typical I am in this. But it does make me wonder if the >>> atleast_* functions act as an attractive nuisance, where new users take >>> their presence as an implicit recommendation that they are actually a >>> useful thing to reach for, even though they... aren't that. And maybe we >>> should be recommending folk move away from them rather than trying to >>> extend them further? >>> >>> Or maybe they're totally useful and I'm just missing it. What's your use >>> case that motivates atleast_nd? >>> >> I think you're just missing it:) atleast_1d/2d are used quite a bit in >> Scipy and Statsmodels (those are the only ones I checked), and in the large >> majority of cases it's the best thing to use there. There's a bunch of >> atleast_2d calls with a transpose appended because the input needs to be >> treated as columns instead of rows, but that's still efficient and readable >> enough. >> > > > As Ralph pointed out its usage in statsmodels. I do find them useful as > replacement for several lines of ifs and reshapes > > We stilll need in many cases the atleast_2d_cols, that appends the newaxis > if necessary. > > roughly the equivalent of > > if x.ndim == 1: > x = x[:, None] > else: > x = np.atleast_2d(x) > > Josef > > >> >> For 3D/nD I can see that you'd need more control over where the >> dimensions go, but 1D/2D are fine. >> > statsmodels has currently very little code with ndim >2, so I have no overview of possible use cases, but it would be necessary to have full control over the added axis since axis have a strict meaning and stats still prefer Fortran order to default numpy/C ordering. Josef > >> >> Ralf >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 09:12:32 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 09:12:32 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: I can add a keyword-only argument that lets you put the new dims before or after the existing ones. I am not sure how to specify arbitrary patterns for the new dimensions, but that should take care of most use cases. The use case that motivated this function in the first place is that I am doing some processing on 4D arrays and I need to reduce them but return a result with the original dimensionality (but not shape). atleast_nd seemed like a better solution than atleast_4d. -Joe On Wed, Jul 6, 2016 at 3:41 AM, wrote: > > > On Wed, Jul 6, 2016 at 3:29 AM, wrote: >> >> >> >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers >> wrote: >>> >>> >>> >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: >>> >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" >>>> wrote: >>>> > >>>> > Hi, >>>> > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >>>> > function np.atleast_nd in PR#7804 >>>> > (https://github.com/numpy/numpy/pull/7804). >>>> > >>>> > As a result of this PR, I have a couple of questions about >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >>>> > the dimensions: If the input is 1D, it prepends and appends a size-1 >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This is >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does >>>> > `np.atleast_nd`). >>>> > >>>> > - Is there any reason for this behavior? >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This would >>>> > be a slight API change since the output would not be exactly the same. >>>> >>>> Changing atleast_3d seems likely to break a bunch of stuff... >>>> >>>> Beyond that, I find it hard to have an opinion about the best design for >>>> these functions, because I don't think I've ever encountered a situation >>>> where they were actually what I wanted. I'm not a big fan of coercing >>>> dimensions in the first place, for the usual "refuse to guess" reasons. And >>>> then generally if I do want to coerce an array to another dimension, then I >>>> have some opinion about where the new dimensions should go, and/or I have >>>> some opinion about the minimum acceptable starting dimension, and/or I have >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that >>>> requirements list.) >>>> >>>> I don't know how typical I am in this. But it does make me wonder if the >>>> atleast_* functions act as an attractive nuisance, where new users take >>>> their presence as an implicit recommendation that they are actually a useful >>>> thing to reach for, even though they... aren't that. And maybe we should be >>>> recommending folk move away from them rather than trying to extend them >>>> further? >>>> >>>> Or maybe they're totally useful and I'm just missing it. What's your use >>>> case that motivates atleast_nd? >>> >>> I think you're just missing it:) atleast_1d/2d are used quite a bit in >>> Scipy and Statsmodels (those are the only ones I checked), and in the large >>> majority of cases it's the best thing to use there. There's a bunch of >>> atleast_2d calls with a transpose appended because the input needs to be >>> treated as columns instead of rows, but that's still efficient and readable >>> enough. >> >> >> >> As Ralph pointed out its usage in statsmodels. I do find them useful as >> replacement for several lines of ifs and reshapes >> >> We stilll need in many cases the atleast_2d_cols, that appends the newaxis >> if necessary. >> >> roughly the equivalent of >> >> if x.ndim == 1: >> x = x[:, None] >> else: >> x = np.atleast_2d(x) >> >> Josef >> >>> >>> >>> For 3D/nD I can see that you'd need more control over where the >>> dimensions go, but 1D/2D are fine. > > > > statsmodels has currently very little code with ndim >2, so I have no > overview of possible use cases, but it would be necessary to have full > control over the added axis since axis have a strict meaning and stats still > prefer Fortran order to default numpy/C ordering. > > Josef > > >>> >>> >>> >>> Ralf >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From ben.v.root at gmail.com Wed Jul 6 09:51:16 2016 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 6 Jul 2016 09:51:16 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: While atleast_1d/2d/3d predates my involvement in numpy, I am probably partly to blame for popularizing them as I helped to fix them up a fair amount. I wouldn't call its use "guessing". Rather, I would treat them as useful input sanitizers. If your function is going to be doing 2d indexing on an input, then it is very convenient to have atleast_2d() at the top of your function, not only to sanitize the input, but to make it clear that your code expects at least two dimensions. One place where it is used is in np.loadtxt(..., ndmin=N) to protect against the situation of a single row of data becoming a 1-D array rather than a 2-D array (or an empty text file returning something completely useless). I have previously pointed out the oddity with atleast_3d(). I can't remember the explanation I got though. Maybe someone can find the old thread that has the explanation, if any? I think the keyword argument approach for controlling the behavior might be a good approach, provided that a suitable design could be devised. 1 & 2 dimensions is fairly trivial to control, but 3+ dimensions has too many degrees of freedom for me to consider. Cheers! Ben Root On Wed, Jul 6, 2016 at 9:12 AM, Joseph Fox-Rabinovitz < jfoxrabinovitz at gmail.com> wrote: > I can add a keyword-only argument that lets you put the new dims > before or after the existing ones. I am not sure how to specify > arbitrary patterns for the new dimensions, but that should take care > of most use cases. > > The use case that motivated this function in the first place is that I > am doing some processing on 4D arrays and I need to reduce them but > return a result with the original dimensionality (but not shape). > atleast_nd seemed like a better solution than atleast_4d. > > -Joe > > > On Wed, Jul 6, 2016 at 3:41 AM, wrote: > > > > > > On Wed, Jul 6, 2016 at 3:29 AM, wrote: > >> > >> > >> > >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers > >> wrote: > >>> > >>> > >>> > >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: > >>> > >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > >>>> wrote: > >>>> > > >>>> > Hi, > >>>> > > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with > a > >>>> > function np.atleast_nd in PR#7804 > >>>> > (https://github.com/numpy/numpy/pull/7804). > >>>> > > >>>> > As a result of this PR, I have a couple of questions about > >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > >>>> > the dimensions: If the input is 1D, it prepends and appends a size-1 > >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This > is > >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does > >>>> > `np.atleast_nd`). > >>>> > > >>>> > - Is there any reason for this behavior? > >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This > would > >>>> > be a slight API change since the output would not be exactly the > same. > >>>> > >>>> Changing atleast_3d seems likely to break a bunch of stuff... > >>>> > >>>> Beyond that, I find it hard to have an opinion about the best design > for > >>>> these functions, because I don't think I've ever encountered a > situation > >>>> where they were actually what I wanted. I'm not a big fan of coercing > >>>> dimensions in the first place, for the usual "refuse to guess" > reasons. And > >>>> then generally if I do want to coerce an array to another dimension, > then I > >>>> have some opinion about where the new dimensions should go, and/or I > have > >>>> some opinion about the minimum acceptable starting dimension, and/or > I have > >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column > matrix; > >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that > >>>> requirements list.) > >>>> > >>>> I don't know how typical I am in this. But it does make me wonder if > the > >>>> atleast_* functions act as an attractive nuisance, where new users > take > >>>> their presence as an implicit recommendation that they are actually a > useful > >>>> thing to reach for, even though they... aren't that. And maybe we > should be > >>>> recommending folk move away from them rather than trying to extend > them > >>>> further? > >>>> > >>>> Or maybe they're totally useful and I'm just missing it. What's your > use > >>>> case that motivates atleast_nd? > >>> > >>> I think you're just missing it:) atleast_1d/2d are used quite a bit in > >>> Scipy and Statsmodels (those are the only ones I checked), and in the > large > >>> majority of cases it's the best thing to use there. There's a bunch of > >>> atleast_2d calls with a transpose appended because the input needs to > be > >>> treated as columns instead of rows, but that's still efficient and > readable > >>> enough. > >> > >> > >> > >> As Ralph pointed out its usage in statsmodels. I do find them useful as > >> replacement for several lines of ifs and reshapes > >> > >> We stilll need in many cases the atleast_2d_cols, that appends the > newaxis > >> if necessary. > >> > >> roughly the equivalent of > >> > >> if x.ndim == 1: > >> x = x[:, None] > >> else: > >> x = np.atleast_2d(x) > >> > >> Josef > >> > >>> > >>> > >>> For 3D/nD I can see that you'd need more control over where the > >>> dimensions go, but 1D/2D are fine. > > > > > > > > statsmodels has currently very little code with ndim >2, so I have no > > overview of possible use cases, but it would be necessary to have full > > control over the added axis since axis have a strict meaning and stats > still > > prefer Fortran order to default numpy/C ordering. > > > > Josef > > > > > >>> > >>> > >>> > >>> Ralf > >>> > >>> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion at scipy.org > >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >> > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Jul 6 10:22:05 2016 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 6 Jul 2016 10:22:05 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: Hi All, I'm with Nathaniel here, in that I don't really see the point of these routines in the first place: broadcasting takes care of many of the initial use cases one might think of, and others are generally not all that well served by them: the examples from scipy to me do not really support `at_least?d`, but rather suggest that little thought has been put into higher-dimensional objects which should be treated as stacks of row or column vectors. My sense is that we're better off developing the direction started with `matmul`, perhaps adding `matvecmul` etc. More to the point of the initial inquiry: what is the advantage of having a general `np.at_leastnd` routine over doing ``` np.array(a, copy=False, ndim=n) ``` or, for a list of inputs, ``` [np.array(a, copy=False, ndim=n) for a in input_list] ``` All the best, Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From plucek at ssaris.com Wed Jul 6 10:25:06 2016 From: plucek at ssaris.com (Paul Lucek) Date: Wed, 6 Jul 2016 10:25:06 -0400 Subject: [Numpy-discussion] unsubscribe Message-ID: On Wednesday, July 6, 2016, wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request at scipy.org > > You can reach the person managing the list at > numpy-discussion-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Joseph Fox-Rabinovitz) > 2. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Benjamin Root) > 3. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Marten van Kerkwijk) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 6 Jul 2016 09:12:32 -0400 > From: Joseph Fox-Rabinovitz > > To: Discussion of Numerical Python > > Subject: Re: [Numpy-discussion] Added atleast_nd, request for > clarification/cleanup of atleast_3d > Message-ID: > sp4dQJRbXoGzkEgBdGPw at mail.gmail.com > > Content-Type: text/plain; charset=UTF-8 > > I can add a keyword-only argument that lets you put the new dims > before or after the existing ones. I am not sure how to specify > arbitrary patterns for the new dimensions, but that should take care > of most use cases. > > The use case that motivated this function in the first place is that I > am doing some processing on 4D arrays and I need to reduce them but > return a result with the original dimensionality (but not shape). > atleast_nd seemed like a better solution than atleast_4d. > > -Joe > > > On Wed, Jul 6, 2016 at 3:41 AM, > > wrote: > > > > > > On Wed, Jul 6, 2016 at 3:29 AM, > > wrote: > >> > >> > >> > >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers > > >> wrote: > >>> > >>> > >>> > >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith > wrote: > >>> > >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > >>>> > wrote: > >>>> > > >>>> > Hi, > >>>> > > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with > a > >>>> > function np.atleast_nd in PR#7804 > >>>> > (https://github.com/numpy/numpy/pull/7804). > >>>> > > >>>> > As a result of this PR, I have a couple of questions about > >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > >>>> > the dimensions: If the input is 1D, it prepends and appends a size-1 > >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This > is > >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does > >>>> > `np.atleast_nd`). > >>>> > > >>>> > - Is there any reason for this behavior? > >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This > would > >>>> > be a slight API change since the output would not be exactly the > same. > >>>> > >>>> Changing atleast_3d seems likely to break a bunch of stuff... > >>>> > >>>> Beyond that, I find it hard to have an opinion about the best design > for > >>>> these functions, because I don't think I've ever encountered a > situation > >>>> where they were actually what I wanted. I'm not a big fan of coercing > >>>> dimensions in the first place, for the usual "refuse to guess" > reasons. And > >>>> then generally if I do want to coerce an array to another dimension, > then I > >>>> have some opinion about where the new dimensions should go, and/or I > have > >>>> some opinion about the minimum acceptable starting dimension, and/or > I have > >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column > matrix; > >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that > >>>> requirements list.) > >>>> > >>>> I don't know how typical I am in this. But it does make me wonder if > the > >>>> atleast_* functions act as an attractive nuisance, where new users > take > >>>> their presence as an implicit recommendation that they are actually a > useful > >>>> thing to reach for, even though they... aren't that. And maybe we > should be > >>>> recommending folk move away from them rather than trying to extend > them > >>>> further? > >>>> > >>>> Or maybe they're totally useful and I'm just missing it. What's your > use > >>>> case that motivates atleast_nd? > >>> > >>> I think you're just missing it:) atleast_1d/2d are used quite a bit in > >>> Scipy and Statsmodels (those are the only ones I checked), and in the > large > >>> majority of cases it's the best thing to use there. There's a bunch of > >>> atleast_2d calls with a transpose appended because the input needs to > be > >>> treated as columns instead of rows, but that's still efficient and > readable > >>> enough. > >> > >> > >> > >> As Ralph pointed out its usage in statsmodels. I do find them useful as > >> replacement for several lines of ifs and reshapes > >> > >> We stilll need in many cases the atleast_2d_cols, that appends the > newaxis > >> if necessary. > >> > >> roughly the equivalent of > >> > >> if x.ndim == 1: > >> x = x[:, None] > >> else: > >> x = np.atleast_2d(x) > >> > >> Josef > >> > >>> > >>> > >>> For 3D/nD I can see that you'd need more control over where the > >>> dimensions go, but 1D/2D are fine. > > > > > > > > statsmodels has currently very little code with ndim >2, so I have no > > overview of possible use cases, but it would be necessary to have full > > control over the added axis since axis have a strict meaning and stats > still > > prefer Fortran order to default numpy/C ordering. > > > > Josef > > > > > >>> > >>> > >>> > >>> Ralf > >>> > >>> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion at scipy.org > >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >> > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > ------------------------------ > > Message: 2 > Date: Wed, 6 Jul 2016 09:51:16 -0400 > From: Benjamin Root > > To: Discussion of Numerical Python > > Subject: Re: [Numpy-discussion] Added atleast_nd, request for > clarification/cleanup of atleast_3d > Message-ID: > < > CANNq6FkrEmDM3DYW2ztHxxELW+wBeq2uj+viF3VNq4KsFEu_dQ at mail.gmail.com > > > Content-Type: text/plain; charset="utf-8" > > While atleast_1d/2d/3d predates my involvement in numpy, I am probably > partly to blame for popularizing them as I helped to fix them up a fair > amount. I wouldn't call its use "guessing". Rather, I would treat them as > useful input sanitizers. If your function is going to be doing 2d indexing > on an input, then it is very convenient to have atleast_2d() at the top of > your function, not only to sanitize the input, but to make it clear that > your code expects at least two dimensions. > > One place where it is used is in np.loadtxt(..., ndmin=N) to protect > against the situation of a single row of data becoming a 1-D array rather > than a 2-D array (or an empty text file returning something completely > useless). > > I have previously pointed out the oddity with atleast_3d(). I can't > remember the explanation I got though. Maybe someone can find the old > thread that has the explanation, if any? > > I think the keyword argument approach for controlling the behavior might be > a good approach, provided that a suitable design could be devised. 1 & 2 > dimensions is fairly trivial to control, but 3+ dimensions has too many > degrees of freedom for me to consider. > > Cheers! > Ben Root > > > On Wed, Jul 6, 2016 at 9:12 AM, Joseph Fox-Rabinovitz < > jfoxrabinovitz at gmail.com > wrote: > > > I can add a keyword-only argument that lets you put the new dims > > before or after the existing ones. I am not sure how to specify > > arbitrary patterns for the new dimensions, but that should take care > > of most use cases. > > > > The use case that motivated this function in the first place is that I > > am doing some processing on 4D arrays and I need to reduce them but > > return a result with the original dimensionality (but not shape). > > atleast_nd seemed like a better solution than atleast_4d. > > > > -Joe > > > > > > On Wed, Jul 6, 2016 at 3:41 AM, > > wrote: > > > > > > > > > On Wed, Jul 6, 2016 at 3:29 AM, > > wrote: > > >> > > >> > > >> > > >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers > > > >> wrote: > > >>> > > >>> > > >>> > > >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith > wrote: > > >>> > > >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > > >>>> > wrote: > > >>>> > > > >>>> > Hi, > > >>>> > > > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d > with > > a > > >>>> > function np.atleast_nd in PR#7804 > > >>>> > (https://github.com/numpy/numpy/pull/7804). > > >>>> > > > >>>> > As a result of this PR, I have a couple of questions about > > >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird > with > > >>>> > the dimensions: If the input is 1D, it prepends and appends a > size-1 > > >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This > > is > > >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does > > >>>> > `np.atleast_nd`). > > >>>> > > > >>>> > - Is there any reason for this behavior? > > >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` > in > > >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This > > would > > >>>> > be a slight API change since the output would not be exactly the > > same. > > >>>> > > >>>> Changing atleast_3d seems likely to break a bunch of stuff... > > >>>> > > >>>> Beyond that, I find it hard to have an opinion about the best design > > for > > >>>> these functions, because I don't think I've ever encountered a > > situation > > >>>> where they were actually what I wanted. I'm not a big fan of > coercing > > >>>> dimensions in the first place, for the usual "refuse to guess" > > reasons. And > > >>>> then generally if I do want to coerce an array to another dimension, > > then I > > >>>> have some opinion about where the new dimensions should go, and/or I > > have > > >>>> some opinion about the minimum acceptable starting dimension, and/or > > I have > > >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column > > matrix; > > >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on > that > > >>>> requirements list.) > > >>>> > > >>>> I don't know how typical I am in this. But it does make me wonder if > > the > > >>>> atleast_* functions act as an attractive nuisance, where new users > > take > > >>>> their presence as an implicit recommendation that they are actually > a > > useful > > >>>> thing to reach for, even though they... aren't that. And maybe we > > should be > > >>>> recommending folk move away from them rather than trying to extend > > them > > >>>> further? > > >>>> > > >>>> Or maybe they're totally useful and I'm just missing it. What's your > > use > > >>>> case that motivates atleast_nd? > > >>> > > >>> I think you're just missing it:) atleast_1d/2d are used quite a bit > in > > >>> Scipy and Statsmodels (those are the only ones I checked), and in the > > large > > >>> majority of cases it's the best thing to use there. There's a bunch > of > > >>> atleast_2d calls with a transpose appended because the input needs to > > be > > >>> treated as columns instead of rows, but that's still efficient and > > readable > > >>> enough. > > >> > > >> > > >> > > >> As Ralph pointed out its usage in statsmodels. I do find them useful > as > > >> replacement for several lines of ifs and reshapes > > >> > > >> We stilll need in many cases the atleast_2d_cols, that appends the > > newaxis > > >> if necessary. > > >> > > >> roughly the equivalent of > > >> > > >> if x.ndim == 1: > > >> x = x[:, None] > > >> else: > > >> x = np.atleast_2d(x) > > >> > > >> Josef > > >> > > >>> > > >>> > > >>> For 3D/nD I can see that you'd need more control over where the > > >>> dimensions go, but 1D/2D are fine. > > > > > > > > > > > > statsmodels has currently very little code with ndim >2, so I have no > > > overview of possible use cases, but it would be necessary to have full > > > control over the added axis since axis have a strict meaning and stats > > still > > > prefer Fortran order to default numpy/C ordering. > > > > > > Josef > > > > > > > > >>> > > >>> > > >>> > > >>> Ralf > > >>> > > >>> > > >>> _______________________________________________ > > >>> NumPy-Discussion mailing list > > >>> NumPy-Discussion at scipy.org > > >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion > > >>> > > >> > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.scipy.org/pipermail/numpy-discussion/attachments/20160706/4241522b/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Wed, 6 Jul 2016 10:22:05 -0400 > From: Marten van Kerkwijk > > To: Discussion of Numerical Python > > Subject: Re: [Numpy-discussion] Added atleast_nd, request for > clarification/cleanup of atleast_3d > Message-ID: > < > CAJNV+9vZX4xN6FvjUyn_h1zw+0a6qtt4nG3_NfOy7-cTL+RFMA at mail.gmail.com > > > Content-Type: text/plain; charset="utf-8" > > Hi All, > > I'm with Nathaniel here, in that I don't really see the point of these > routines in the first place: broadcasting takes care of many of the initial > use cases one might think of, and others are generally not all that well > served by them: the examples from scipy to me do not really support > `at_least?d`, but rather suggest that little thought has been put into > higher-dimensional objects which should be treated as stacks of row or > column vectors. My sense is that we're better off developing the direction > started with `matmul`, perhaps adding `matvecmul` etc. > > More to the point of the initial inquiry: what is the advantage of having a > general `np.at_leastnd` routine over doing > ``` > np.array(a, copy=False, ndim=n) > ``` > or, for a list of inputs, > ``` > [np.array(a, copy=False, ndim=n) for a in input_list] > ``` > > All the best, > > Marten > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.scipy.org/pipermail/numpy-discussion/attachments/20160706/75f87f2d/attachment.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------ > > End of NumPy-Discussion Digest, Vol 118, Issue 9 > ************************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Jul 6 10:56:23 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Wed, 6 Jul 2016 16:56:23 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: We use np.at_least2d extensively in scikit-image, and I also use it in a *lot* of my own code now that scikit-learn stopped accepting 1D arrays as feature vectors. > what is the advantage of np.at_leastnd` over `np.array(a, copy=False, ndim=n)` Readability, clearly. My only concern is the described behavior of np.at_least3d, which came as a surprise. I certainly would expect the ?at_least? family to all work in the same way as broadcasting, ie prepending singleton dimensions. Prepend/append behavior can be controlled either by keyword or simply by using .T, I don?t mind either way. Juan. On 6 July 2016 at 10:22:15 AM, Marten van Kerkwijk ( m.h.vankerkwijk at gmail.com) wrote: Hi All, I'm with Nathaniel here, in that I don't really see the point of these routines in the first place: broadcasting takes care of many of the initial use cases one might think of, and others are generally not all that well served by them: the examples from scipy to me do not really support `at_least?d`, but rather suggest that little thought has been put into higher-dimensional objects which should be treated as stacks of row or column vectors. My sense is that we're better off developing the direction started with `matmul`, perhaps adding `matvecmul` etc. More to the point of the initial inquiry: what is the advantage of having a general `np.at_leastnd` routine over doing ``` np.array(a, copy=False, ndim=n) ``` or, for a list of inputs, ``` [np.array(a, copy=False, ndim=n) for a in input_list] ``` All the best, Marten _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Jul 6 11:14:00 2016 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 06 Jul 2016 17:14:00 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: <1467818040.9455.27.camel@sipsolutions.net> On Mi, 2016-07-06 at 10:22 -0400, Marten van Kerkwijk wrote: > Hi All, > > I'm with Nathaniel here, in that I don't really see the point of > these routines in the first place: broadcasting takes care of many of > the initial use cases one might think of, and others are generally > not all that well served by them: the examples from scipy to me do > not really support `at_least?d`, but rather suggest that little > thought has been put into higher-dimensional objects which should be > treated as stacks of row or column vectors. My sense is that we're > better off developing the direction started with `matmul`, perhaps > adding `matvecmul` etc. > > More to the point of the initial inquiry: what is the advantage of > having a general `np.at_leastnd` routine over doing There is another wonky reason for using the atleast_?d functions, in that they use reshape to be fully duck typed ;) (in newer versions at least, probably mostly for sparse arrays, not sure). Tend to agree though, especially considering the confusing order of 3d, which I suppose is likely due to some linalg considerations. Of course you could supply something like an insertion order of (1, 0, 2) to denote the current 3D case in the nd one, but frankly it seems to me likely harder to understand how it works then to write your own functions to just do it. Scipy uses the 3D case exactly never (once in a test). I have my doubts many would notice if we deprecate the 3D case, but then it is likely more trouble then gain. - Sebastian > ``` > np.array(a, copy=False, ndim=n) > ``` > or, for a list of inputs, > ``` > [np.array(a, copy=False, ndim=n) for a in input_list] > ``` > > All the best, > > Marten > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Wed Jul 6 12:26:49 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 09:26:49 -0700 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Jul 5, 2016 11:21 PM, "Ralf Gommers" wrote: > > > > On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: > >> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" wrote: >> > >> > Hi, >> > >> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >> > function np.atleast_nd in PR#7804 >> > (https://github.com/numpy/numpy/pull/7804). >> > >> > As a result of this PR, I have a couple of questions about >> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >> > the dimensions: If the input is 1D, it prepends and appends a size-1 >> > dimension. If the input is 2D, it appends a size-1 dimension. This is >> > inconsistent with `np.atleast_2d`, which always prepends (as does >> > `np.atleast_nd`). >> > >> > - Is there any reason for this behavior? >> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >> > terms of `np.atleast_nd`, which is actually much simpler)? This would >> > be a slight API change since the output would not be exactly the same. >> >> Changing atleast_3d seems likely to break a bunch of stuff... >> >> Beyond that, I find it hard to have an opinion about the best design for these functions, because I don't think I've ever encountered a situation where they were actually what I wanted. I'm not a big fan of coercing dimensions in the first place, for the usual "refuse to guess" reasons. And then generally if I do want to coerce an array to another dimension, then I have some opinion about where the new dimensions should go, and/or I have some opinion about the minimum acceptable starting dimension, and/or I have a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that requirements list.) >> >> I don't know how typical I am in this. But it does make me wonder if the atleast_* functions act as an attractive nuisance, where new users take their presence as an implicit recommendation that they are actually a useful thing to reach for, even though they... aren't that. And maybe we should be recommending folk move away from them rather than trying to extend them further? >> >> Or maybe they're totally useful and I'm just missing it. What's your use case that motivates atleast_nd? > > I think you're just missing it:) atleast_1d/2d are used quite a bit in Scipy and Statsmodels (those are the only ones I checked), and in the large majority of cases it's the best thing to use there. There's a bunch of atleast_2d calls with a transpose appended because the input needs to be treated as columns instead of rows, but that's still efficient and readable enough. I know people *use* it :-). What I'm confused about is in what situations you would invent it if it didn't exist. Can you point me to an example or two where it's "the best thing"? I actually had statsmodels in mind with my example of wanting the semantics "coerce 1d inputs into a column matrix; 0d or 3d inputs are an error". I'm surprised if there are places where you really want 0d arrays converted into 1x1, or want to allow high dimensional arrays to pass through - and if you do want to allow high dimensional arrays to pass through, then transposing might help with 2d cases but will silently mangle high-d cases, right? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 6 12:35:24 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 09:35:24 -0700 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Jul 6, 2016 6:12 AM, "Joseph Fox-Rabinovitz" wrote: > > I can add a keyword-only argument that lets you put the new dims > before or after the existing ones. I am not sure how to specify > arbitrary patterns for the new dimensions, but that should take care > of most use cases. > > The use case that motivated this function in the first place is that I > am doing some processing on 4D arrays and I need to reduce them but > return a result with the original dimensionality (but not shape). > atleast_nd seemed like a better solution than atleast_4d. This is a tangent that might not apply given the details of your code, but isn't this what keepdims is for? (And keepdims has the huge advantage that it knows which axes are being reduced and thus where to put the new axes.) I guess even if I couldn't use keepdims for some reason, my inclination would be to try to emulate it by fixing up the axes as I went, because I'd find it easier to verify that I hadn't accidentally misaligned things if the reductions and fix-ups were local to each other, and explicit axis insertions are much easier than trying to remember whether atleast_nd prepends or appends. This of course is all based on some vague guess at what your code actually looks like though... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 12:48:30 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 12:48:30 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: I was using "reduce" in an abstract sense. I put in a 4D array in, get a 1-3D array out, depending on some other parameters (not strictly just by reduction, although that is the net effect). The placement of the dimensions is irrelevant, I just need to make the output 4D again for further calculations. Since I can have cases where the output is of different number of dims, I wrote this function as a handy tool to avoid conditionals. I realize that this is not a common use-case, but it seemed like a thing someone else might find useful one day. -Joe On Wed, Jul 6, 2016 at 12:35 PM, Nathaniel Smith wrote: > On Jul 6, 2016 6:12 AM, "Joseph Fox-Rabinovitz" > wrote: >> >> I can add a keyword-only argument that lets you put the new dims >> before or after the existing ones. I am not sure how to specify >> arbitrary patterns for the new dimensions, but that should take care >> of most use cases. >> >> The use case that motivated this function in the first place is that I >> am doing some processing on 4D arrays and I need to reduce them but >> return a result with the original dimensionality (but not shape). >> atleast_nd seemed like a better solution than atleast_4d. > > This is a tangent that might not apply given the details of your code, but > isn't this what keepdims is for? (And keepdims has the huge advantage that > it knows which axes are being reduced and thus where to put the new axes.) > > I guess even if I couldn't use keepdims for some reason, my inclination > would be to try to emulate it by fixing up the axes as I went, because I'd > find it easier to verify that I hadn't accidentally misaligned things if the > reductions and fix-ups were local to each other, and explicit axis > insertions are much easier than trying to remember whether atleast_nd > prepends or appends. This of course is all based on some vague guess at what > your code actually looks like though... > > -n > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From shoyer at gmail.com Wed Jul 6 13:43:57 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 6 Jul 2016 10:43:57 -0700 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Tue, Jul 5, 2016 at 10:06 PM, Nathaniel Smith wrote: > I don't know how typical I am in this. But it does make me wonder if the > atleast_* functions act as an attractive nuisance, where new users take > their presence as an implicit recommendation that they are actually a > useful thing to reach for, even though they... aren't that. And maybe we > should be recommending folk move away from them rather than trying to > extend them further? > Agreed. I would avoid adding atleast_nd. We could discourage using atleast_3d (certainly the behavior is indeed surprising), but I'm not sure it's worth the trouble. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 14:21:44 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 14:21:44 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: I still think this function is useful. I have made a change so that it only accepts one array, as Marten suggested, making the API much cleaner than that of its siblings. The side on which the new dimensions will be added is configurable via the `where` parameter, which currently accepts 'before' and 'after', but can be changed to accept sequences or even dicts. The change also resulted in finding a bug in the masked array versions of the atleast functions, which the PR now fixes and adds regression tests for. If the devs do decide to discard this PR, I will of course submit the bug fix separately. -Joe On Wed, Jul 6, 2016 at 1:43 PM, Stephan Hoyer wrote: > On Tue, Jul 5, 2016 at 10:06 PM, Nathaniel Smith wrote: >> >> I don't know how typical I am in this. But it does make me wonder if the >> atleast_* functions act as an attractive nuisance, where new users take >> their presence as an implicit recommendation that they are actually a useful >> thing to reach for, even though they... aren't that. And maybe we should be >> recommending folk move away from them rather than trying to extend them >> further? > > Agreed. I would avoid adding atleast_nd. We could discourage using > atleast_3d (certainly the behavior is indeed surprising), but I'm not sure > it's worth the trouble. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From ben.v.root at gmail.com Wed Jul 6 14:25:41 2016 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 6 Jul 2016 14:25:41 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: I wouldn't have the keyword be "where", as that collides with the notion of "where" elsewhere in numpy. On Wed, Jul 6, 2016 at 2:21 PM, Joseph Fox-Rabinovitz < jfoxrabinovitz at gmail.com> wrote: > I still think this function is useful. I have made a change so that it > only accepts one array, as Marten suggested, making the API much > cleaner than that of its siblings. The side on which the new > dimensions will be added is configurable via the `where` parameter, > which currently accepts 'before' and 'after', but can be changed to > accept sequences or even dicts. The change also resulted in finding a > bug in the masked array versions of the atleast functions, which the > PR now fixes and adds regression tests for. If the devs do decide to > discard this PR, I will of course submit the bug fix separately. > > -Joe > > On Wed, Jul 6, 2016 at 1:43 PM, Stephan Hoyer wrote: > > On Tue, Jul 5, 2016 at 10:06 PM, Nathaniel Smith wrote: > >> > >> I don't know how typical I am in this. But it does make me wonder if the > >> atleast_* functions act as an attractive nuisance, where new users take > >> their presence as an implicit recommendation that they are actually a > useful > >> thing to reach for, even though they... aren't that. And maybe we > should be > >> recommending folk move away from them rather than trying to extend them > >> further? > > > > Agreed. I would avoid adding atleast_nd. We could discourage using > > atleast_3d (certainly the behavior is indeed surprising), but I'm not > sure > > it's worth the trouble. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 14:36:58 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 14:36:58 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: Agreed. I was originally going with "side", but I want something that can be changed to accepting arbitrary specs without changing the word. Perhaps "pos"? I am open to suggestion. -Joe On Wed, Jul 6, 2016 at 2:25 PM, Benjamin Root wrote: > I wouldn't have the keyword be "where", as that collides with the notion of > "where" elsewhere in numpy. > > On Wed, Jul 6, 2016 at 2:21 PM, Joseph Fox-Rabinovitz > wrote: >> >> I still think this function is useful. I have made a change so that it >> only accepts one array, as Marten suggested, making the API much >> cleaner than that of its siblings. The side on which the new >> dimensions will be added is configurable via the `where` parameter, >> which currently accepts 'before' and 'after', but can be changed to >> accept sequences or even dicts. The change also resulted in finding a >> bug in the masked array versions of the atleast functions, which the >> PR now fixes and adds regression tests for. If the devs do decide to >> discard this PR, I will of course submit the bug fix separately. >> >> -Joe >> >> On Wed, Jul 6, 2016 at 1:43 PM, Stephan Hoyer wrote: >> > On Tue, Jul 5, 2016 at 10:06 PM, Nathaniel Smith wrote: >> >> >> >> I don't know how typical I am in this. But it does make me wonder if >> >> the >> >> atleast_* functions act as an attractive nuisance, where new users take >> >> their presence as an implicit recommendation that they are actually a >> >> useful >> >> thing to reach for, even though they... aren't that. And maybe we >> >> should be >> >> recommending folk move away from them rather than trying to extend them >> >> further? >> > >> > Agreed. I would avoid adding atleast_nd. We could discourage using >> > atleast_3d (certainly the behavior is indeed surprising), but I'm not >> > sure >> > it's worth the trouble. >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From efiring at hawaii.edu Wed Jul 6 14:57:48 2016 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 6 Jul 2016 08:57:48 -1000 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On 2016/07/06 8:25 AM, Benjamin Root wrote: > I wouldn't have the keyword be "where", as that collides with the notion > of "where" elsewhere in numpy. Agreed. Maybe "side"? (I find atleast_1d and atleast_2d to be very helpful for handling inputs, as Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.) Eric From jni.soma at gmail.com Wed Jul 6 15:01:44 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Wed, 6 Jul 2016 21:01:44 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: at_leastnd would be useful for nd image processing in a very analogous way to how at_least2d is used by scikit-image, assuming it prepends. The at_least3d choice is baffling, seems analogous to the 0.5-based indexing presented at PyCon, and should be "fun" to deprecate. =P On 6 July 2016 at 2:57:57 PM, Eric Firing (efiring at hawaii.edu) wrote: On 2016/07/06 8:25 AM, Benjamin Root wrote: > I wouldn't have the keyword be "where", as that collides with the notion > of "where" elsewhere in numpy. Agreed. Maybe "side"? (I find atleast_1d and atleast_2d to be very helpful for handling inputs, as Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.) Eric _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 15:15:42 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 15:15:42 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 3:01 PM, Juan Nunez-Iglesias wrote: > at_leastnd would be useful for nd image processing in a very analogous way > to how at_least2d is used by scikit-image, assuming it prepends. The > at_least3d choice is baffling, seems analogous to the 0.5-based indexing > presented at PyCon, and should be "fun" to deprecate. =P at_leastnd prepends by default, has an option to append instead and certainly does not 0.5-pend under any circumstances. `np.swapaxes` and `np.rollaxis` are there for a reason. If atleast_3d is deprecated because of its funky behavior, atleast_nd may be useful replacement. -Joe From jfoxrabinovitz at gmail.com Wed Jul 6 15:20:12 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 15:20:12 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing wrote: > On 2016/07/06 8:25 AM, Benjamin Root wrote: >> >> I wouldn't have the keyword be "where", as that collides with the notion >> of "where" elsewhere in numpy. > > > Agreed. Maybe "side"? I have tentatively changed it to "pos". The reason that I don't like "side" is that it implies only a subset of the possible ways that that the position of the new dimensions can be specified. The current implementation only puts things on one side or the other, but I have considered also allowing an array of indices at which to place new dimensions, and/or a dictionary keyed by the starting ndims. I do not think "side" would be appropriate for these extended cases, even if they are very unlikely to ever materialize. -Joe > (I find atleast_1d and atleast_2d to be very helpful for handling inputs, as > Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.) > > Eric > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From ben.v.root at gmail.com Wed Jul 6 15:30:10 2016 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 6 Jul 2016 15:30:10 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: I don't see how one could define a spec that would take an arbitrary array of indices at which to place new dimensions. By definition, you don't know how many dimensions are going to be added. If you knew, then you wouldn't be calling this function. I can only imagine simple rules such as 'left' or 'right' or maybe something akin to what at_least3d() implements. On Wed, Jul 6, 2016 at 3:20 PM, Joseph Fox-Rabinovitz < jfoxrabinovitz at gmail.com> wrote: > On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing wrote: > > On 2016/07/06 8:25 AM, Benjamin Root wrote: > >> > >> I wouldn't have the keyword be "where", as that collides with the notion > >> of "where" elsewhere in numpy. > > > > > > Agreed. Maybe "side"? > > I have tentatively changed it to "pos". The reason that I don't like > "side" is that it implies only a subset of the possible ways that that > the position of the new dimensions can be specified. The current > implementation only puts things on one side or the other, but I have > considered also allowing an array of indices at which to place new > dimensions, and/or a dictionary keyed by the starting ndims. I do not > think "side" would be appropriate for these extended cases, even if > they are very unlikely to ever materialize. > > -Joe > > > (I find atleast_1d and atleast_2d to be very helpful for handling > inputs, as > > Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.) > > > > Eric > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 6 16:56:11 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 6 Jul 2016 22:56:11 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 6:26 PM, Nathaniel Smith wrote: On Jul 5, 2016 11:21 PM, "Ralf Gommers" wrote: > > > > > > > > On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: > > > >> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" < > jfoxrabinovitz at gmail.com> wrote: > >> > > >> > Hi, > >> > > >> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a > >> > function np.atleast_nd in PR#7804 > >> > (https://github.com/numpy/numpy/pull/7804). > >> > > >> > As a result of this PR, I have a couple of questions about > >> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > >> > the dimensions: If the input is 1D, it prepends and appends a size-1 > >> > dimension. If the input is 2D, it appends a size-1 dimension. This is > >> > inconsistent with `np.atleast_2d`, which always prepends (as does > >> > `np.atleast_nd`). > >> > > >> > - Is there any reason for this behavior? > >> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > >> > terms of `np.atleast_nd`, which is actually much simpler)? This would > >> > be a slight API change since the output would not be exactly the same. > >> > >> Changing atleast_3d seems likely to break a bunch of stuff... > >> > >> Beyond that, I find it hard to have an opinion about the best design > for these functions, because I don't think I've ever encountered a > situation where they were actually what I wanted. I'm not a big fan of > coercing dimensions in the first place, for the usual "refuse to guess" > reasons. And then generally if I do want to coerce an array to another > dimension, then I have some opinion about where the new dimensions should > go, and/or I have some opinion about the minimum acceptable starting > dimension, and/or I have a maximum dimension in mind. (E.g. "coerce 1d > inputs into a column matrix; 0d or 3d inputs are an error" -- atleast_2d is > zero-for-three on that requirements list.) > >> > >> I don't know how typical I am in this. But it does make me wonder if > the atleast_* functions act as an attractive nuisance, where new users take > their presence as an implicit recommendation that they are actually a > useful thing to reach for, even though they... aren't that. And maybe we > should be recommending folk move away from them rather than trying to > extend them further? > >> > >> Or maybe they're totally useful and I'm just missing it. What's your > use case that motivates atleast_nd? > > > > I think you're just missing it:) atleast_1d/2d are used quite a bit in > Scipy and Statsmodels (those are the only ones I checked), and in the large > majority of cases it's the best thing to use there. There's a bunch of > atleast_2d calls with a transpose appended because the input needs to be > treated as columns instead of rows, but that's still efficient and readable > enough. > > I know people *use* it :-). What I'm confused about is in what situations > you would invent it if it didn't exist. Can you point me to an example or > two where it's "the best thing"? I actually had statsmodels in mind with my > example of wanting the semantics "coerce 1d inputs into a column matrix; 0d > or 3d inputs are an error". I'm surprised if there are places where you > really want 0d arrays converted into 1x1, > Scalar to shape (1,1) is less common, but 1-D to 2-D or scalar to shape (1,) is very common. Example is at the top of scipy/stats/stats.py: the _chk_asarray functions (used in many other functions) take care to never return scalar arrays because those are plain annoying to deal with. If that sounds weird to you, you're probably one of those people who was never surprised by this: In [3]: x0 = np.array(1) In [4]: x1 = np.array([1]) In [5]: x0[0] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in () ----> 1 x0[0] IndexError: too many indices for array In [6]: x1[0] Out[6]: 1 or want to allow high dimensional arrays to pass through - and if you do > want to allow high dimensional arrays to pass through, then transposing > might help with 2d cases but will silently mangle high-d cases, right? > >2d input handling is usually irrelevant. The vast majority of cases is "function that accepts scalar and 1-D array" or "function that accepts 1-D and 2-D arrays". Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 6 17:17:50 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 6 Jul 2016 17:17:50 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 4:56 PM, Ralf Gommers wrote: > > > On Wed, Jul 6, 2016 at 6:26 PM, Nathaniel Smith wrote: > >> On Jul 5, 2016 11:21 PM, "Ralf Gommers" wrote: >> > >> > >> > >> > On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: >> > >> >> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" >> >> wrote: >> >> > >> >> > Hi, >> >> > >> >> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >> >> > function np.atleast_nd in PR#7804 >> >> > (https://github.com/numpy/numpy/pull/7804). >> >> > >> >> > As a result of this PR, I have a couple of questions about >> >> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >> >> > the dimensions: If the input is 1D, it prepends and appends a size-1 >> >> > dimension. If the input is 2D, it appends a size-1 dimension. This is >> >> > inconsistent with `np.atleast_2d`, which always prepends (as does >> >> > `np.atleast_nd`). >> >> > >> >> > - Is there any reason for this behavior? >> >> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >> >> > terms of `np.atleast_nd`, which is actually much simpler)? This would >> >> > be a slight API change since the output would not be exactly the >> >> > same. >> >> >> >> Changing atleast_3d seems likely to break a bunch of stuff... >> >> >> >> Beyond that, I find it hard to have an opinion about the best design >> >> for these functions, because I don't think I've ever encountered a situation >> >> where they were actually what I wanted. I'm not a big fan of coercing >> >> dimensions in the first place, for the usual "refuse to guess" reasons. And >> >> then generally if I do want to coerce an array to another dimension, then I >> >> have some opinion about where the new dimensions should go, and/or I have >> >> some opinion about the minimum acceptable starting dimension, and/or I have >> >> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; >> >> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that >> >> requirements list.) >> >> >> >> I don't know how typical I am in this. But it does make me wonder if >> >> the atleast_* functions act as an attractive nuisance, where new users take >> >> their presence as an implicit recommendation that they are actually a useful >> >> thing to reach for, even though they... aren't that. And maybe we should be >> >> recommending folk move away from them rather than trying to extend them >> >> further? >> >> >> >> Or maybe they're totally useful and I'm just missing it. What's your >> >> use case that motivates atleast_nd? >> > >> > I think you're just missing it:) atleast_1d/2d are used quite a bit in >> > Scipy and Statsmodels (those are the only ones I checked), and in the large >> > majority of cases it's the best thing to use there. There's a bunch of >> > atleast_2d calls with a transpose appended because the input needs to be >> > treated as columns instead of rows, but that's still efficient and readable >> > enough. >> >> I know people *use* it :-). What I'm confused about is in what situations >> you would invent it if it didn't exist. Can you point me to an example or >> two where it's "the best thing"? I actually had statsmodels in mind with my >> example of wanting the semantics "coerce 1d inputs into a column matrix; 0d >> or 3d inputs are an error". I'm surprised if there are places where you >> really want 0d arrays converted into 1x1, > > Scalar to shape (1,1) is less common, but 1-D to 2-D or scalar to shape (1,) > is very common. Example is at the top of scipy/stats/stats.py: the > _chk_asarray functions (used in many other functions) take care to never > return scalar arrays because those are plain annoying to deal with. If that > sounds weird to you, you're probably one of those people who was never > surprised by this: > > In [3]: x0 = np.array(1) > > In [4]: x1 = np.array([1]) > > In [5]: x0[0] > --------------------------------------------------------------------------- > IndexError Traceback (most recent call last) > in () > ----> 1 x0[0] > > IndexError: too many indices for array > > In [6]: x1[0] > Out[6]: 1 > >> or want to allow high dimensional arrays to pass through - and if you do >> want to allow high dimensional arrays to pass through, then transposing >> might help with 2d cases but will silently mangle high-d cases, right? > >>2d input handling is usually irrelevant. The vast majority of cases is >> "function that accepts scalar and 1-D array" or "function that accepts 1-D >> and 2-D arrays". Often such a function would want to convert inputs internally. > > Ralf > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From novin01 at gmail.com Wed Jul 6 18:20:07 2016 From: novin01 at gmail.com (David) Date: Wed, 6 Jul 2016 22:20:07 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Added_atleast=5Fnd=2C_request_for_cl?= =?utf-8?q?arification/cleanup_of_atleast=5F3d?= References: Message-ID: Joseph Fox-Rabinovitz gmail.com> writes: > > On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing hawaii.edu> wrote: > > On 2016/07/06 8:25 AM, Benjamin Root wrote: > >> > >> I wouldn't have the keyword be "where", as that collides with the notion > >> of "where" elsewhere in numpy. > > > > > > Agreed. Maybe "side"? > > I have tentatively changed it to "pos". The reason that I don't like > "side" is that it implies only a subset of the possible ways that that > the position of the new dimensions can be specified. The current > implementation only puts things on one side or the other, but I have > considered also allowing an array of indices at which to place new > dimensions, and/or a dictionary keyed by the starting ndims. I do not > think "side" would be appropriate for these extended cases, even if > they are very unlikely to ever materialize. > > -Joe > > > (I find atleast_1d and atleast_2d to be very helpful for handling inputs, as > > Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.) > > > > Eric > > > > __________________________________________ _____ > > NumPy-Discussion mailing list > > NumPy-Discussion scipy.org > > https://mail.scipy.org/mailman/listinfo/nu mpy-discussion > About `order='C'` or `order='F'` for the argument name? From njs at pobox.com Wed Jul 6 18:22:01 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 15:22:01 -0700 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 1:56 PM, Ralf Gommers wrote: > > > On Wed, Jul 6, 2016 at 6:26 PM, Nathaniel Smith wrote: > >> On Jul 5, 2016 11:21 PM, "Ralf Gommers" wrote: >> > >> > >> > >> > On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith wrote: >> > >> >> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" >> >> wrote: >> >> > >> >> > Hi, >> >> > >> >> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a >> >> > function np.atleast_nd in PR#7804 >> >> > (https://github.com/numpy/numpy/pull/7804). >> >> > >> >> > As a result of this PR, I have a couple of questions about >> >> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with >> >> > the dimensions: If the input is 1D, it prepends and appends a size-1 >> >> > dimension. If the input is 2D, it appends a size-1 dimension. This is >> >> > inconsistent with `np.atleast_2d`, which always prepends (as does >> >> > `np.atleast_nd`). >> >> > >> >> > - Is there any reason for this behavior? >> >> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in >> >> > terms of `np.atleast_nd`, which is actually much simpler)? This would >> >> > be a slight API change since the output would not be exactly the >> >> > same. >> >> >> >> Changing atleast_3d seems likely to break a bunch of stuff... >> >> >> >> Beyond that, I find it hard to have an opinion about the best design >> >> for these functions, because I don't think I've ever encountered a situation >> >> where they were actually what I wanted. I'm not a big fan of coercing >> >> dimensions in the first place, for the usual "refuse to guess" reasons. And >> >> then generally if I do want to coerce an array to another dimension, then I >> >> have some opinion about where the new dimensions should go, and/or I have >> >> some opinion about the minimum acceptable starting dimension, and/or I have >> >> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column matrix; >> >> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that >> >> requirements list.) >> >> >> >> I don't know how typical I am in this. But it does make me wonder if >> >> the atleast_* functions act as an attractive nuisance, where new users take >> >> their presence as an implicit recommendation that they are actually a useful >> >> thing to reach for, even though they... aren't that. And maybe we should be >> >> recommending folk move away from them rather than trying to extend them >> >> further? >> >> >> >> Or maybe they're totally useful and I'm just missing it. What's your >> >> use case that motivates atleast_nd? >> > >> > I think you're just missing it:) atleast_1d/2d are used quite a bit in >> > Scipy and Statsmodels (those are the only ones I checked), and in the large >> > majority of cases it's the best thing to use there. There's a bunch of >> > atleast_2d calls with a transpose appended because the input needs to be >> > treated as columns instead of rows, but that's still efficient and readable >> > enough. >> >> I know people *use* it :-). What I'm confused about is in what situations >> you would invent it if it didn't exist. Can you point me to an example or >> two where it's "the best thing"? I actually had statsmodels in mind with my >> example of wanting the semantics "coerce 1d inputs into a column matrix; 0d >> or 3d inputs are an error". I'm surprised if there are places where you >> really want 0d arrays converted into 1x1, > > Scalar to shape (1,1) is less common, but 1-D to 2-D or scalar to shape (1,) > is very common. That's ravel, though, not atleast_*, right? > Example is at the top of scipy/stats/stats.py: the > _chk_asarray functions (used in many other functions) I feel like this actually argues for my point :-). scipy.stats needs some uniform prepping of input, so there's a helper function to do that, and the helper function's semantics are not at all the semantics of atleast_*. And they don't even use atleast_* in any necessary way -- the only thing they do is if arr.ndim ==0: arr = np.atleast_1d(arr) but this could be written just as well as if arr.ndim == 0: arr = arr[np.newaxis] (In any case, atleast_1d definitely makes more sense to me than any of the others, since it so obviously corresponds to exactly that 2-line incantation as the only reasonable implementation.) > take care to never > return scalar arrays because those are plain annoying to deal with. If that > sounds weird to you, you're probably one of those people who was never > surprised by this: > > In [3]: x0 = np.array(1) > > In [4]: x1 = np.array([1]) > > In [5]: x0[0] > --------------------------------------------------------------------------- > IndexError Traceback (most recent call last) > in () > ----> 1 x0[0] > > IndexError: too many indices for array > > In [6]: x1[0] > Out[6]: 1 I was surprised by it the first time I hit it, but then thought it over and decided that it was better than the alternatives :-). (It does strike me as really odd, I would even say a bug, that e.g. scipy.stats.mode returns a 1d array for 1d input, and a 2d array (!) for 2d input. Mode is semantically a reduction operation, and we have pretty strong conventions for how those work -- drop a dimension unless keepdims=True. Obviously this is old existing code, and that's fine, but it's not how we'd recommend people write new code, I think? I guess this is orthogonal to the whole atleast_* discussion anyway :-).) >> or want to allow high dimensional arrays to pass through - and if you do >> want to allow high dimensional arrays to pass through, then transposing >> might help with 2d cases but will silently mangle high-d cases, right? > >>2d input handling is usually irrelevant. The vast majority of cases is >> "function that accepts scalar and 1-D array" or "function that accepts 1-D >> and 2-D arrays". So maybe we should have functions that actually handle those cases, instead of recommending atleast_*? -n -- Nathaniel J. Smith -- https://vorpus.org From numpy-cpbxr at q33.us Wed Jul 6 18:43:02 2016 From: numpy-cpbxr at q33.us (Rob Nagler) Date: Wed, 6 Jul 2016 16:43:02 -0600 Subject: [Numpy-discussion] Remove FCompiler flags? Message-ID: I would like to remove "-fno-second-underscore" and "-Wall" from the fcompiler. I had hacked a build_clib subclass to remove the flags dynamically, but that is fragile (and is now broken). Is there a right way to remove flags? Thanks, Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Jul 7 04:34:19 2016 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 07 Jul 2016 10:34:19 +0200 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: (sfid-20160706_213210_200485_F2370C62) References: (sfid-20160706_213210_200485_F2370C62) Message-ID: <1467880459.17128.10.camel@sipsolutions.net> On Mi, 2016-07-06 at 15:30 -0400, Benjamin Root wrote: > I don't see how one could define a spec that would take an arbitrary > array of indices at which to place new dimensions. By definition, you > You just give a reordered range, so that (1, 0, 2) would be the current 3D version. If 1D, fill in `1` and `2`, if 2D, fill in only `2` (0D, add everything of course). However, I have my doubts that it is actually easier to understand then to write yourself ;). - Sebastian ? > don't know how many dimensions are going to be added. If you knew, > then you wouldn't be calling this function. I can only imagine simple > rules such as 'left' or 'right' or maybe something akin to what > at_least3d() implements. > > On Wed, Jul 6, 2016 at 3:20 PM, Joseph Fox-Rabinovitz @gmail.com> wrote: > > On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing > > wrote: > > > On 2016/07/06 8:25 AM, Benjamin Root wrote: > > >> > > >> I wouldn't have the keyword be "where", as that collides with > > the notion > > >> of "where" elsewhere in numpy. > > > > > > > > > Agreed.? Maybe "side"? > > > > I have tentatively changed it to "pos". The reason that I don't > > like > > "side" is that it implies only a subset of the possible ways that > > that > > the position of the new dimensions can be specified. The current > > implementation only puts things on one side or the other, but I > > have > > considered also allowing an array of indices at which to place new > > dimensions, and/or a dictionary keyed by the starting ndims. I do > > not > > think "side" would be appropriate for these extended cases, even if > > they are very unlikely to ever materialize. > > > > ? ? -Joe > > > > > (I find atleast_1d and atleast_2d to be very helpful for handling > > inputs, as > > > Ben noted; I'm skeptical as to the value of atleast_3d and > > atleast_nd.) > > > > > > Eric > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From jfoxrabinovitz at gmail.com Thu Jul 7 09:11:59 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Thu, 7 Jul 2016 09:11:59 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: <1467880459.17128.10.camel@sipsolutions.net> References: <1467880459.17128.10.camel@sipsolutions.net> Message-ID: On Thu, Jul 7, 2016 at 4:34 AM, Sebastian Berg wrote: > On Mi, 2016-07-06 at 15:30 -0400, Benjamin Root wrote: >> I don't see how one could define a spec that would take an arbitrary >> array of indices at which to place new dimensions. By definition, you >> > > You just give a reordered range, so that (1, 0, 2) would be the current > 3D version. If 1D, fill in `1` and `2`, if 2D, fill in only `2` (0D, > add everything of course). I was originally thinking (-1, 0) for the 2D case. Just go along the list and fill as many dims as necessary. Your way is much better since it does not require a different operation for positive and negative indices. > However, I have my doubts that it is actually easier to understand then > to write yourself ;). A dictionary or ragged list would be better for that: either {1: (1, 0), 2: (2,)} or [(1, 0), (2,)]. The first is more clear since the index in the list is the starting ndim - 1. > > - Sebastian > > >> don't know how many dimensions are going to be added. If you knew, >> then you wouldn't be calling this function. I can only imagine simple >> rules such as 'left' or 'right' or maybe something akin to what >> at_least3d() implements. >> >> On Wed, Jul 6, 2016 at 3:20 PM, Joseph Fox-Rabinovitz > @gmail.com> wrote: >> > On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing >> > wrote: >> > > On 2016/07/06 8:25 AM, Benjamin Root wrote: >> > >> >> > >> I wouldn't have the keyword be "where", as that collides with >> > the notion >> > >> of "where" elsewhere in numpy. >> > > >> > > >> > > Agreed. Maybe "side"? >> > >> > I have tentatively changed it to "pos". The reason that I don't >> > like >> > "side" is that it implies only a subset of the possible ways that >> > that >> > the position of the new dimensions can be specified. The current >> > implementation only puts things on one side or the other, but I >> > have >> > considered also allowing an array of indices at which to place new >> > dimensions, and/or a dictionary keyed by the starting ndims. I do >> > not >> > think "side" would be appropriate for these extended cases, even if >> > they are very unlikely to ever materialize. >> > >> > -Joe >> > >> > > (I find atleast_1d and atleast_2d to be very helpful for handling >> > inputs, as >> > > Ben noted; I'm skeptical as to the value of atleast_3d and >> > atleast_nd.) >> > > >> > > Eric >> > > >> > > _______________________________________________ >> > > NumPy-Discussion mailing list >> > > NumPy-Discussion at scipy.org >> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at gmail.com Thu Jul 7 16:19:39 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 7 Jul 2016 22:19:39 +0200 Subject: [Numpy-discussion] Remove FCompiler flags? In-Reply-To: References: Message-ID: On Thu, Jul 7, 2016 at 12:43 AM, Rob Nagler wrote: > I would like to remove "-fno-second-underscore" and "-Wall" from the > fcompiler. I had hacked a build_clib subclass to remove the flags > dynamically, but that is fragile (and is now broken). Is there a right way > to remove flags? > > No good way to do that I'm afraid, every method is pretty fragile. If it's on my own machine, I would just remove the flags from fcompiler/gnu.py by hand. If it needs to be dynamically, then I'd probably go for monkeypatching the GnuFCompiler/Gnu95FCompiler classes from that package instead of a command subclass. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-cpbxr at q33.us Thu Jul 7 17:29:27 2016 From: numpy-cpbxr at q33.us (Rob Nagler) Date: Thu, 7 Jul 2016 15:29:27 -0600 Subject: [Numpy-discussion] Remove FCompiler flags? In-Reply-To: References: Message-ID: Yes, that's what I ended up doing. For the curious, here's the code: https://github.com/robnagler/shadow3/blob/5a2676b8d20156e33603b898d7a4d690c4511619/setup.py#L21 Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Fri Jul 8 19:36:41 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 9 Jul 2016 00:36:41 +0100 Subject: [Numpy-discussion] scipy 0.18.0 release candidate 2 Message-ID: On behalf of the scipy development team, I am pleased to announce the second release candidate for scipy 0.18.0. Please test it --- both the release itself on your machines and your code against this release --- and report results on the issue tracker or scipy-dev mailing list. Source tarballs and release notes are available from Github releases, https://github.com/scipy/scipy/releases/tag/v0.18.0rc2 OS X and Linux wheels are being built now, and I will upload them to PyPI when they are ready. I would like to flag two areas where additional scrutiny and testing would be most appreciated: * Precision loss in some corner cases was reported to occur on Fedora 23 and 24 with OpenBLAS, see https://github.com/scipy/scipy/issues/6286. * On Windows, a CRT incompatibility was reported when building with MSVC and MKL, see https://github.com/scipy/scipy/issues/6336. There are additional issues on s390x (https://github.com/scipy/scipy/issues/6338), so if you are using this platform, we appreciate your help resolving them. The (revised) release schedule is: 25 July: final release Thanks to everyone who contributed! Cheers, Evgeni From asbuzza at gmail.com Sat Jul 9 14:57:23 2016 From: asbuzza at gmail.com (asbuzza) Date: Sat, 9 Jul 2016 11:57:23 -0700 (MST) Subject: [Numpy-discussion] F2PY subroutine referenced in another file Message-ID: <1468090643073-43261.post@n7.nabble.com> Hello everyone, I am relatively new to f2py and have been trying to create a module of a main fortran file that references a subroutine in another fortran file. I have been doing: >f2py -c -m mainmodule mainmodule.f90 othermodule.f90 Any help would be greatly appreciated. Thanks, Andy -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/F2PY-subroutine-referenced-in-another-file-tp43261.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From rmay31 at gmail.com Sat Jul 9 21:44:27 2016 From: rmay31 at gmail.com (Ryan May) Date: Sat, 9 Jul 2016 19:44:27 -0600 Subject: [Numpy-discussion] Custom Dtype/Units discussion Message-ID: Greetings! I've been beating my head against a wall trying to work seamlessly with pint's unit support and arrays from numpy and xarray; these same issues seem to apply to other unit frameworks as well. Last time I dug into these problems, things like custom dtypes were raised as a more extensible solution that works within numpy (and other libraries) without needing a bunch of custom support. Anyone around SciPy this week want to get together and talk about how we can move ahead? (or acquaint me with another/better path forward?) I feel like I need to get this figured out one way or another before I can move forward in my corner of the world, and I have time I can dedicate to implementing a solution. Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jul 10 02:20:02 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 9 Jul 2016 23:20:02 -0700 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Hi Ryan, I'll be and SciPy and I'd love to talk about this :-). Things are a bit hectic for me on Mon/Tue/Wed between the Python Compilers Workshop and my talk, but do you want to meet up Thursday maybe? -n On Sat, Jul 9, 2016 at 6:44 PM, Ryan May wrote: > Greetings! > > I've been beating my head against a wall trying to work seamlessly with > pint's unit support and arrays from numpy and xarray; these same issues seem > to apply to other unit frameworks as well. Last time I dug into these > problems, things like custom dtypes were raised as a more extensible > solution that works within numpy (and other libraries) without needing a > bunch of custom support. > > Anyone around SciPy this week want to get together and talk about how we can > move ahead? (or acquaint me with another/better path forward?) I feel like I > need to get this figured out one way or another before I can move forward in > my corner of the world, and I have time I can dedicate to implementing a > solution. > > Ryan > > -- > Ryan May > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith -- https://vorpus.org From jjstickel at gmail.com Sun Jul 10 10:19:20 2016 From: jjstickel at gmail.com (Jonathan Stickel) Date: Sun, 10 Jul 2016 08:19:20 -0600 Subject: [Numpy-discussion] F2PY subroutine referenced in another file In-Reply-To: References: Message-ID: <75f3fdc9-9386-ebab-080e-fde944da256b@gmail.com> Andy First compile "othermodule" with fortran (not f2py): $ gfortran -shared -o libothermodule.so othermodule.f90 Then link to it when compiling "mainmodule" with f2py: $ f2py -c -L[path to libothermodule.so] -lothermodule -m mainmodule mainmodule.f90 You can also make othermodule a static library and incorporate it into mainmodule. The syntax is a little different, but I do not have an example in front of me right now. HTH, Jonathan On 7/10/16 06:00 , numpy-discussion-request at scipy.org wrote: > Date: Sat, 9 Jul 2016 11:57:23 -0700 (MST) > From: asbuzza > To: numpy-discussion at scipy.org > Subject: [Numpy-discussion] F2PY subroutine referenced in another file > Message-ID: <1468090643073-43261.post at n7.nabble.com> > Content-Type: text/plain; charset=us-ascii > > Hello everyone, > > I am relatively new to f2py and have been trying to create a module of a > main fortran file that references a subroutine in another fortran file. I > have been doing: > >> >f2py -c -m mainmodule mainmodule.f90 othermodule.f90 > Any help would be greatly appreciated. > > Thanks, > Andy From rmay31 at gmail.com Sun Jul 10 22:45:57 2016 From: rmay31 at gmail.com (Ryan May) Date: Sun, 10 Jul 2016 20:45:57 -0600 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Hi Nathaniel, Thursday works for me; anyone else interested is welcome to join. Ryan On Sun, Jul 10, 2016 at 12:20 AM, Nathaniel Smith wrote: > Hi Ryan, > > I'll be and SciPy and I'd love to talk about this :-). Things are a > bit hectic for me on Mon/Tue/Wed between the Python Compilers Workshop > and my talk, but do you want to meet up Thursday maybe? > > -n > > On Sat, Jul 9, 2016 at 6:44 PM, Ryan May wrote: > > Greetings! > > > > I've been beating my head against a wall trying to work seamlessly with > > pint's unit support and arrays from numpy and xarray; these same issues > seem > > to apply to other unit frameworks as well. Last time I dug into these > > problems, things like custom dtypes were raised as a more extensible > > solution that works within numpy (and other libraries) without needing a > > bunch of custom support. > > > > Anyone around SciPy this week want to get together and talk about how we > can > > move ahead? (or acquaint me with another/better path forward?) I feel > like I > > need to get this figured out one way or another before I can move > forward in > > my corner of the world, and I have time I can dedicate to implementing a > > solution. > > > > Ryan > > > > -- > > Ryan May > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Sun Jul 10 23:12:33 2016 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Sun, 10 Jul 2016 22:12:33 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Hi Ryan, As a maintainer of a unit-aware ndarray subclass, I'm also interested in sitting in. Maybe this can be an informal BOF session? Nathan On Sunday, July 10, 2016, Ryan May wrote: > Hi Nathaniel, > > Thursday works for me; anyone else interested is welcome to join. > > Ryan > > On Sun, Jul 10, 2016 at 12:20 AM, Nathaniel Smith > wrote: > >> Hi Ryan, >> >> I'll be and SciPy and I'd love to talk about this :-). Things are a >> bit hectic for me on Mon/Tue/Wed between the Python Compilers Workshop >> and my talk, but do you want to meet up Thursday maybe? >> >> -n >> >> On Sat, Jul 9, 2016 at 6:44 PM, Ryan May > > wrote: >> > Greetings! >> > >> > I've been beating my head against a wall trying to work seamlessly with >> > pint's unit support and arrays from numpy and xarray; these same issues >> seem >> > to apply to other unit frameworks as well. Last time I dug into these >> > problems, things like custom dtypes were raised as a more extensible >> > solution that works within numpy (and other libraries) without needing a >> > bunch of custom support. >> > >> > Anyone around SciPy this week want to get together and talk about how >> we can >> > move ahead? (or acquaint me with another/better path forward?) I feel >> like I >> > need to get this figured out one way or another before I can move >> forward in >> > my corner of the world, and I have time I can dedicate to implementing a >> > solution. >> > >> > Ryan >> > >> > -- >> > Ryan May >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Ryan May > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Sun Jul 10 23:25:30 2016 From: rmay31 at gmail.com (Ryan May) Date: Sun, 10 Jul 2016 22:25:30 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Sounds like an apt description of what this is intended to be. Ryan On Sun, Jul 10, 2016 at 10:12 PM, Nathan Goldbaum wrote: > Hi Ryan, > > As a maintainer of a unit-aware ndarray subclass, I'm also interested in > sitting in. > > Maybe this can be an informal BOF session? > > Nathan > > > On Sunday, July 10, 2016, Ryan May wrote: > >> Hi Nathaniel, >> >> Thursday works for me; anyone else interested is welcome to join. >> >> Ryan >> >> On Sun, Jul 10, 2016 at 12:20 AM, Nathaniel Smith wrote: >> >>> Hi Ryan, >>> >>> I'll be and SciPy and I'd love to talk about this :-). Things are a >>> bit hectic for me on Mon/Tue/Wed between the Python Compilers Workshop >>> and my talk, but do you want to meet up Thursday maybe? >>> >>> -n >>> >>> On Sat, Jul 9, 2016 at 6:44 PM, Ryan May wrote: >>> > Greetings! >>> > >>> > I've been beating my head against a wall trying to work seamlessly with >>> > pint's unit support and arrays from numpy and xarray; these same >>> issues seem >>> > to apply to other unit frameworks as well. Last time I dug into these >>> > problems, things like custom dtypes were raised as a more extensible >>> > solution that works within numpy (and other libraries) without needing >>> a >>> > bunch of custom support. >>> > >>> > Anyone around SciPy this week want to get together and talk about how >>> we can >>> > move ahead? (or acquaint me with another/better path forward?) I feel >>> like I >>> > need to get this figured out one way or another before I can move >>> forward in >>> > my corner of the world, and I have time I can dedicate to implementing >>> a >>> > solution. >>> > >>> > Ryan >>> > >>> > -- >>> > Ryan May >>> > >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> >>> >>> >>> -- >>> Nathaniel J. Smith -- https://vorpus.org >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> >> -- >> Ryan May >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 11 00:53:28 2016 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 10 Jul 2016 22:53:28 -0600 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Sun, Jul 10, 2016 at 12:20 AM, Nathaniel Smith wrote: > Hi Ryan, > > I'll be and SciPy and I'd love to talk about this :-). Things are a > bit hectic for me on Mon/Tue/Wed between the Python Compilers Workshop > and my talk, but do you want to meet up Thursday maybe? > > I'll be at scipy also and Thursday sounds fine. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Mon Jul 11 10:41:24 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Mon, 11 Jul 2016 10:41:24 -0400 Subject: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d In-Reply-To: References: <1467880459.17128.10.camel@sipsolutions.net> Message-ID: I would like to follow up on my original PR (7804). While there appears to be some debate as to whether the PR is numpy material to begin with, there do not appear to be any technical issues with it. To make the decision more straightforward, I factored out the non-controversial bug fixes to masked arrays into PR #7823, along with their regression tests. This way, the original enhancement can be closed or left hanging indefinitely, (even though I hope neither happens). PR 7804 still has the bug fixes duplicated in it. Regards, -Joe On Thu, Jul 7, 2016 at 9:11 AM, Joseph Fox-Rabinovitz wrote: > On Thu, Jul 7, 2016 at 4:34 AM, Sebastian Berg > wrote: >> On Mi, 2016-07-06 at 15:30 -0400, Benjamin Root wrote: >>> I don't see how one could define a spec that would take an arbitrary >>> array of indices at which to place new dimensions. By definition, you >>> >> >> You just give a reordered range, so that (1, 0, 2) would be the current >> 3D version. If 1D, fill in `1` and `2`, if 2D, fill in only `2` (0D, >> add everything of course). > > I was originally thinking (-1, 0) for the 2D case. Just go along the > list and fill as many dims as necessary. Your way is much better since > it does not require a different operation for positive and negative > indices. > >> However, I have my doubts that it is actually easier to understand then >> to write yourself ;). > > A dictionary or ragged list would be better for that: either {1: (1, > 0), 2: (2,)} or [(1, 0), (2,)]. The first is more clear since the > index in the list is the starting ndim - 1. > >> >> - Sebastian >> >> >>> don't know how many dimensions are going to be added. If you knew, >>> then you wouldn't be calling this function. I can only imagine simple >>> rules such as 'left' or 'right' or maybe something akin to what >>> at_least3d() implements. >>> >>> On Wed, Jul 6, 2016 at 3:20 PM, Joseph Fox-Rabinovitz >> @gmail.com> wrote: >>> > On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing >>> > wrote: >>> > > On 2016/07/06 8:25 AM, Benjamin Root wrote: >>> > >> >>> > >> I wouldn't have the keyword be "where", as that collides with >>> > the notion >>> > >> of "where" elsewhere in numpy. >>> > > >>> > > >>> > > Agreed. Maybe "side"? >>> > >>> > I have tentatively changed it to "pos". The reason that I don't >>> > like >>> > "side" is that it implies only a subset of the possible ways that >>> > that >>> > the position of the new dimensions can be specified. The current >>> > implementation only puts things on one side or the other, but I >>> > have >>> > considered also allowing an array of indices at which to place new >>> > dimensions, and/or a dictionary keyed by the starting ndims. I do >>> > not >>> > think "side" would be appropriate for these extended cases, even if >>> > they are very unlikely to ever materialize. >>> > >>> > -Joe >>> > >>> > > (I find atleast_1d and atleast_2d to be very helpful for handling >>> > inputs, as >>> > > Ben noted; I'm skeptical as to the value of atleast_3d and >>> > atleast_nd.) >>> > > >>> > > Eric >>> > > >>> > > _______________________________________________ >>> > > NumPy-Discussion mailing list >>> > > NumPy-Discussion at scipy.org >>> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> From nevion at gmail.com Mon Jul 11 13:01:49 2016 From: nevion at gmail.com (Jason Newton) Date: Mon, 11 Jul 2016 13:01:49 -0400 Subject: [Numpy-discussion] deterministic, reproducible matmul / __matmult_ Message-ID: Hello I'm a long time user of numpy - but an issue I've had with it is making sure I can reproduce the results of a floating point matrix multiplication in other languages/modules (like c or GPU) in another, or across installations. I take great pains in doing this type of work because it allows me to both prototype with python/numpy and to also in a fairly strong/useful capacity, use it as a strict reference implementation. For me - small differences accumulate such that allclose type thinking starts failing in a few iterations of algorithms, things diverge. I've had success with einsum before for some cases (by chance) where no difference was ever observed between it and Eigen (C++), but I'm not sure if I should use this any longer. The new @ operator is very tempting to use too, in prototyping. Does the ML have any ideas on how one could get a matmul that will not allow any funny business on the evaluation of the products? Funny business here is something like changing the evaluation order additions of terms. I want strict IEEE 754 compliance - no 80 bit registers, and perhaps control of the rounding mode, no unsafe math optimizations. I'm definitely willing to sacrifice performance (esp multi threaded based enhancements which already cause problems in reduction ordering) in order to get these guarantees. I was looking around and found a few BLAS's that might be worth a mention, comments on these would also be welcome: http://bebop.cs.berkeley.edu/reproblas/ https://exblas.lip6.fr/ -Jason From chris.barker at noaa.gov Mon Jul 11 13:39:05 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 11 Jul 2016 10:39:05 -0700 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum wrote: > > Maybe this can be an informal BOF session? > or maybe a formal BoF? after all, how formal do they get? Anyway, it was my understanding that we really needed to do some significant refactoring of how numpy deals with dtypes in order to do this kind of thing cleanly -- so where has that gone since last year? Maybe this conversation should be about how to build a more flexible dtype system generally, rather than specifically about unit support. (though unit support is a great use-case to focus on) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Mon Jul 11 13:49:22 2016 From: rmay31 at gmail.com (Ryan May) Date: Mon, 11 Jul 2016 12:49:22 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker wrote: > > Maybe this conversation should be about how to build a more flexible dtype > system generally, rather than specifically about unit support. (though unit > support is a great use-case to focus on) > I agree that a more general solution is a good goal--just that units is my "sine qua non". Also, I would have love to have heard that someone solved the unit + ndarray-like thing problem. :) Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 11 13:58:53 2016 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 11 Jul 2016 11:58:53 -0600 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 11:39 AM, Chris Barker wrote: > > > On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum > wrote: > >> >> Maybe this can be an informal BOF session? >> > > or maybe a formal BoF? after all, how formal do they get? > > Anyway, it was my understanding that we really needed to do some > significant refactoring of how numpy deals with dtypes in order to do this > kind of thing cleanly -- so where has that gone since last year? > > Maybe this conversation should be about how to build a more flexible dtype > system generally, rather than specifically about unit support. (though unit > support is a great use-case to focus on) > Note that Mark Wiebe will also be giving a talk Friday, so he may be around. As the last person to add a type to Numpy and the designer of DyND he might have some useful input. DyND development is pretty active and I'm always curious how we can somehow move in that direction. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Jul 11 15:27:37 2016 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 11 Jul 2016 19:27:37 +0000 (UTC) Subject: [Numpy-discussion] deterministic, reproducible matmul / __matmult_ References: Message-ID: Mon, 11 Jul 2016 13:01:49 -0400, Jason Newton kirjoitti: > Does the ML have any ideas on how one could get a matmul that will not > allow any funny business on the evaluation of the products? Funny > business here is something like changing the evaluation order additions > of terms. I want strict IEEE 754 compliance - no 80 bit registers, and > perhaps control of the rounding mode, no unsafe math optimizations. If you link Numpy with a BLAS and LAPACK libraries that have been compiled for this purpose, and turn on the compiler flags that enforce strict IEEE (and disable SSE) when compiling Numpy, you probably will get reproducible builds. Numpy itself just offloads the dot computations to BLAS, so if your BLAS is reproducible, things should mainly be OK. You may also need to turn off the SSE optimizations in Numpy, because these can make results depend on memory alignment --- not in dot products, but in other computations. Out of curiosity, what is the application where this is necessary? Maybe there is a numerically stable formulation? -- Pauli Virtanen From travis at continuum.io Tue Jul 12 01:56:01 2016 From: travis at continuum.io (Travis Oliphant) Date: Tue, 12 Jul 2016 00:56:01 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 12:58 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Mon, Jul 11, 2016 at 11:39 AM, Chris Barker > wrote: > >> >> >> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum >> wrote: >> >>> >>> Maybe this can be an informal BOF session? >>> >> >> or maybe a formal BoF? after all, how formal do they get? >> >> Anyway, it was my understanding that we really needed to do some >> significant refactoring of how numpy deals with dtypes in order to do this >> kind of thing cleanly -- so where has that gone since last year? >> >> Maybe this conversation should be about how to build a more flexible >> dtype system generally, rather than specifically about unit support. >> (though unit support is a great use-case to focus on) >> > > Note that Mark Wiebe will also be giving a talk Friday, so he may be > around. As the last person to add a type to Numpy and the designer of DyND > he might have some useful input. DyND development is pretty active and I'm > always curious how we can somehow move in that direction. > > There has been a lot of work over the past 6 months on making DyND implement the "pluribus" concept that I have talked about briefly in the past. DyND now has a separate C++ ndt data-type library. The Python interface to that type library is still unified in the dynd module but it is separable and work is in progress to make a separate Python-wrapper to this type library. The dynd type library is datashape described at http://datashape.pydata.org This type system is extensible and could be the foundation of a re-factored NumPy. My view (and what I am encouraging work in the direction of) is that array computing in Python should be refactored into a "type-subsystem" (I think ndt is the right model there), a generic ufunc-system (I think dynd has a very promising approach there as well), and then a container (the memoryview already in Python might be enough already). These modules could be separately installed, maintained and eventually moved into Python itself. Then, a potential future NumPy project could be ported to be a layer of calculations and connections to other C-libraries on-top of this system. Many parts of the current code could be re-used in that effort --- or the new system could be part of a re-factoring of NumPy to make the innards of NumPy more accessible to a JIT compiler. We are already far enough along that this could be pursued with a motivated person. It would take 18 months to complete the system but first-light would be less than 6 months for a dedicated, motivated, and talented resource. DyND is far enough along as well as Cython and/or Numba to make this pretty straight-forward. For this re-factored array-computing project to take the NumPy name, this community would have to decide that that is the right thing to do. But, other projects like Pandas and/or xarray and/or numpy-py and/or NumPy on Jython could use this sub-system also. It has taken me a long time to actually get to the point where I would recommend a specific way forward. I have thought about this for many years and don't make these recommendations lightly. The pluribus concept is my recommendation about what would be best now and in the future --- and I will be pursuing this concept and working to get to a point where this community will accept it if possible because it would be ideal if this new array library were still called NumPy. My working view is that someone will have to build the new prototype NumPy for the community to evaluate whether it's the right approach and get consensus that it is the right way forward. There is enough there now with DyND, data-shape, and Numba/Cython to do this fairly quickly. It is not strictly necessary to use DyND or Numba or even data-shape to accomplish this general plan --- but these are already available and a great place to start as they have been built explicitly with the intention of improving array-computing in Python. This potential NumPy could be backwards compatible from an API perspective (including a C-API) --- though recompliation would be necessary and there would be some semantic differences in corner-cases that could either be fixed where necessary but potentially just made part of the new version. I will be at the Continuum Happy hour on Thursday at our offices and welcome anyone to come discuss things with me there --- I am also willing to meet with anyone on Thursday and Friday if I can --- but I don't have a ticket to ScPy itself. Please CC me directly if you have questions. I try to follow the numpy-discussion mailing list but I am not always successful at keeping up. To be clear as some have mis-interpreted me in the past, while I originally wrote NumPy (borrowing heavily from Numeric and drawing inspiration from Numarray and receiving a lot of help for specific modules from many of you), the community has continued to develop NumPy and now has a proper governance model. I am now simply an interested NumPy user and previous NumPy developer who finally has some concrete ideas to share based on work that I have been funding, leading, and encouraging for the past several years. I am still very interested in helping NumPy progress, but we are also going to be taking these ideas to create a general concept of the "buffer protocol in Python" to enable cross-language code-sharing to enable more code re-use for data analytics among language communities. This is the concept of "data-fabric" which is pre-alpha vapor-ware at this point but with some ideas expressed at http://datashape.pydata.org and here: https://github.com/blaze/datafabric and is something DyND is enabling. NumPy itself has a clear governance model and whether NumPy (the project) adopts any of the new array-computing concepts I am proposing will depend on this community's decisions as well as work done by motivated developers willing to work on prototypes. I will be wiling to help get funding for someone motivated to work on this. Best, -Travis > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant, PhD* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 12 02:21:22 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 12 Jul 2016 08:21:22 +0200 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Tue, Jul 12, 2016 at 7:56 AM, Travis Oliphant wrote: > > > > http://www.continuum.io > On Mon, Jul 11, 2016 at 12:58 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Mon, Jul 11, 2016 at 11:39 AM, Chris Barker >> wrote: >> >>> >>> >>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum >>> wrote: >>> >>>> >>>> Maybe this can be an informal BOF session? >>>> >>> >>> or maybe a formal BoF? after all, how formal do they get? >>> >>> Anyway, it was my understanding that we really needed to do some >>> significant refactoring of how numpy deals with dtypes in order to do this >>> kind of thing cleanly -- so where has that gone since last year? >>> >>> Maybe this conversation should be about how to build a more flexible >>> dtype system generally, rather than specifically about unit support. >>> (though unit support is a great use-case to focus on) >>> >> >> Note that Mark Wiebe will also be giving a talk Friday, so he may be >> around. As the last person to add a type to Numpy and the designer of DyND >> he might have some useful input. DyND development is pretty active and I'm >> always curious how we can somehow move in that direction. >> >> > There has been a lot of work over the past 6 months on making DyND > implement the "pluribus" concept that I have talked about briefly in the > past. DyND now has a separate C++ ndt data-type library. The Python > interface to that type library is still unified in the dynd module but it > is separable and work is in progress to make a separate Python-wrapper to > this type library. The dynd type library is datashape described at > http://datashape.pydata.org > > This type system is extensible and could be the foundation of a > re-factored NumPy. My view (and what I am encouraging work in the > direction of) is that array computing in Python should be refactored into a > "type-subsystem" (I think ndt is the right model there), a generic > ufunc-system (I think dynd has a very promising approach there as well), > and then a container (the memoryview already in Python might be enough > already). These modules could be separately installed, maintained and > eventually moved into Python itself. > > Then, a potential future NumPy project could be ported to be a layer of > calculations and connections to other C-libraries on-top of this system. > Many parts of the current code could be re-used in that effort --- or the > new system could be part of a re-factoring of NumPy to make the innards of > NumPy more accessible to a JIT compiler. > > We are already far enough along that this could be pursued with a > motivated person. It would take 18 months to complete the system but > first-light would be less than 6 months for a dedicated, motivated, and > talented resource. DyND is far enough along as well as Cython and/or > Numba to make this pretty straight-forward. For this re-factored > array-computing project to take the NumPy name, this community would have > to decide that that is the right thing to do. But, other projects like > Pandas and/or xarray and/or numpy-py and/or NumPy on Jython could use this > sub-system also. > > It has taken me a long time to actually get to the point where I would > recommend a specific way forward. I have thought about this for many > years and don't make these recommendations lightly. The pluribus concept > is my recommendation about what would be best now and in the future --- and > I will be pursuing this concept and working to get to a point where this > community will accept it if possible because it would be ideal if this new > array library were still called NumPy. > > My working view is that someone will have to build the new prototype NumPy > for the community to evaluate whether it's the right approach and get > consensus that it is the right way forward. There is enough there now > with DyND, data-shape, and Numba/Cython to do this fairly quickly. It > is not strictly necessary to use DyND or Numba or even data-shape to > accomplish this general plan --- but these are already available and a > great place to start as they have been built explicitly with the intention > of improving array-computing in Python. > > This potential NumPy could be backwards compatible from an API perspective > (including a C-API) --- though recompliation would be necessary and there > would be some semantic differences in corner-cases that could either be > fixed where necessary but potentially just made part of the new version. > > I will be at the Continuum Happy hour on Thursday at our offices and > welcome anyone to come discuss things with me there --- I am also willing > to meet with anyone on Thursday and Friday if I can --- but I don't have a > ticket to ScPy itself. Please CC me directly if you have questions. I > try to follow the numpy-discussion mailing list but I am not always > successful at keeping up. > > To be clear as some have mis-interpreted me in the past, while I > originally wrote NumPy (borrowing heavily from Numeric and drawing > inspiration from Numarray and receiving a lot of help for specific modules > from many of you), the community has continued to develop NumPy and now has > a proper governance model. I am now simply an interested NumPy user and > previous NumPy developer who finally has some concrete ideas to share based > on work that I have been funding, leading, and encouraging for the past > several years. > > I am still very interested in helping NumPy progress, but we are also > going to be taking these ideas to create a general concept of the "buffer > protocol in Python" to enable cross-language code-sharing to enable more > code re-use for data analytics among language communities. This is the > concept of "data-fabric" which is pre-alpha vapor-ware at this point but > with some ideas expressed at http://datashape.pydata.org and here: > https://github.com/blaze/datafabric and is something DyND is enabling. > > NumPy itself has a clear governance model and whether NumPy (the project) > adopts any of the new array-computing concepts I am proposing will depend > on this community's decisions as well as work done by motivated developers > willing to work on prototypes. I will be wiling to help get funding for > someone motivated to work on this. > Thanks Travis! I'm going to let the technical parts sink in for a bit first, but wanted to say already that your continued interest and sharing of new ideas are much appreciated. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Wed Jul 13 15:43:10 2016 From: rmay31 at gmail.com (Ryan May) Date: Wed, 13 Jul 2016 14:43:10 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker wrote: > > > On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum > wrote: > >> >> Maybe this can be an informal BOF session? >> > > or maybe a formal BoF? after all, how formal do they get? > > Anyway, it was my understanding that we really needed to do some > significant refactoring of how numpy deals with dtypes in order to do this > kind of thing cleanly -- so where has that gone since last year? > > Maybe this conversation should be about how to build a more flexible dtype > system generally, rather than specifically about unit support. (though unit > support is a great use-case to focus on) > > So Thursday's options seem to be in the standard BOF slot (up against the Numfocus BOF), or doing something that evening, which would overlap at least part of multiple happy hour events. I lean towards evening. Thoughts? Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jul 13 20:46:41 2016 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Jul 2016 18:46:41 -0600 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Evening would work for me. Dinner? On Jul 13, 2016 2:43 PM, "Ryan May" wrote: > On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker > wrote: > >> >> >> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum >> wrote: >> >>> >>> Maybe this can be an informal BOF session? >>> >> >> or maybe a formal BoF? after all, how formal do they get? >> >> Anyway, it was my understanding that we really needed to do some >> significant refactoring of how numpy deals with dtypes in order to do this >> kind of thing cleanly -- so where has that gone since last year? >> >> Maybe this conversation should be about how to build a more flexible >> dtype system generally, rather than specifically about unit support. >> (though unit support is a great use-case to focus on) >> >> > So Thursday's options seem to be in the standard BOF slot (up against the > Numfocus BOF), or doing something that evening, which would overlap at > least part of multiple happy hour events. I lean towards evening. Thoughts? > > Ryan > > -- > Ryan May > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at kinnear.ca Thu Jul 14 00:16:15 2016 From: ryan at kinnear.ca (Ryan J. Kinnear) Date: Thu, 14 Jul 2016 00:16:15 -0400 Subject: [Numpy-discussion] Numpy Jenkin's Traub? Message-ID: <5787120F.102@kinnear.ca> Dear list, I'm working on implementing an ARMA(p, q) modeling method in Python. The particular method is described in section 4.7.1 of Statistical Digital Signal Processing and Modeling by Hayes. The upshot is that I need to calculate the roots of a polynomial. I've learned that this is a numerically ill conditioned problem, and have found many cases where np.roots produces roots that are clearly spurious. This is causing me a lot of issues. I have done some further searching and learned of the "Jenkins-Traub" algorithm for root finding, which seems to be considered the most robust method. I found a Python implementation of this method here (https://github.com/vrdabomb5717/jenkins_traub), and it is certainly much more robust than np.roots. Are there reasons for Jenkins-Traub not being implemented as part of Numpy? It is built into Scilab (https://help.scilab.org/docs/6.0.0/en_US/roots.html) Is anyone working on getting it into Numpy? -RJK From njs at pobox.com Thu Jul 14 01:48:01 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 14 Jul 2016 00:48:01 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: I have something at lunch, so dinner would be good for me too. On Jul 13, 2016 7:46 PM, "Charles R Harris" wrote: > Evening would work for me. Dinner? > On Jul 13, 2016 2:43 PM, "Ryan May" wrote: > >> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >> wrote: >> >>> >>> >>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum >>> wrote: >>> >>>> >>>> Maybe this can be an informal BOF session? >>>> >>> >>> or maybe a formal BoF? after all, how formal do they get? >>> >>> Anyway, it was my understanding that we really needed to do some >>> significant refactoring of how numpy deals with dtypes in order to do this >>> kind of thing cleanly -- so where has that gone since last year? >>> >>> Maybe this conversation should be about how to build a more flexible >>> dtype system generally, rather than specifically about unit support. >>> (though unit support is a great use-case to focus on) >>> >>> >> So Thursday's options seem to be in the standard BOF slot (up against the >> Numfocus BOF), or doing something that evening, which would overlap at >> least part of multiple happy hour events. I lean towards evening. Thoughts? >> >> Ryan >> >> -- >> Ryan May >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Thu Jul 14 11:49:40 2016 From: rmay31 at gmail.com (Ryan May) Date: Thu, 14 Jul 2016 10:49:40 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Fine with me. Ryan On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith wrote: > I have something at lunch, so dinner would be good for me too. > On Jul 13, 2016 7:46 PM, "Charles R Harris" > wrote: > >> Evening would work for me. Dinner? >> On Jul 13, 2016 2:43 PM, "Ryan May" wrote: >> >>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >>> wrote: >>> >>>> >>>> >>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum >>> > wrote: >>>> >>>>> >>>>> Maybe this can be an informal BOF session? >>>>> >>>> >>>> or maybe a formal BoF? after all, how formal do they get? >>>> >>>> Anyway, it was my understanding that we really needed to do some >>>> significant refactoring of how numpy deals with dtypes in order to do this >>>> kind of thing cleanly -- so where has that gone since last year? >>>> >>>> Maybe this conversation should be about how to build a more flexible >>>> dtype system generally, rather than specifically about unit support. >>>> (though unit support is a great use-case to focus on) >>>> >>>> >>> So Thursday's options seem to be in the standard BOF slot (up against >>> the Numfocus BOF), or doing something that evening, which would overlap at >>> least part of multiple happy hour events. I lean towards evening. Thoughts? >>> >>> Ryan >>> >>> -- >>> Ryan May >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Thu Jul 14 11:51:53 2016 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Thu, 14 Jul 2016 10:51:53 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Fine with me as well. Meet in the downstairs lobby after the lightning talks? On Thu, Jul 14, 2016 at 10:49 AM, Ryan May wrote: > Fine with me. > > Ryan > > On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith wrote: > >> I have something at lunch, so dinner would be good for me too. >> On Jul 13, 2016 7:46 PM, "Charles R Harris" >> wrote: >> >>> Evening would work for me. Dinner? >>> On Jul 13, 2016 2:43 PM, "Ryan May" wrote: >>> >>>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >>>> wrote: >>>> >>>>> >>>>> >>>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum < >>>>> nathan12343 at gmail.com> wrote: >>>>> >>>>>> >>>>>> Maybe this can be an informal BOF session? >>>>>> >>>>> >>>>> or maybe a formal BoF? after all, how formal do they get? >>>>> >>>>> Anyway, it was my understanding that we really needed to do some >>>>> significant refactoring of how numpy deals with dtypes in order to do this >>>>> kind of thing cleanly -- so where has that gone since last year? >>>>> >>>>> Maybe this conversation should be about how to build a more flexible >>>>> dtype system generally, rather than specifically about unit support. >>>>> (though unit support is a great use-case to focus on) >>>>> >>>>> >>>> So Thursday's options seem to be in the standard BOF slot (up against >>>> the Numfocus BOF), or doing something that evening, which would overlap at >>>> least part of multiple happy hour events. I lean towards evening. Thoughts? >>>> >>>> Ryan >>>> >>>> -- >>>> Ryan May >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Ryan May > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Thu Jul 14 11:56:30 2016 From: rmay31 at gmail.com (Ryan May) Date: Thu, 14 Jul 2016 10:56:30 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Sounds good. On Thu, Jul 14, 2016 at 10:51 AM, Nathan Goldbaum wrote: > Fine with me as well. Meet in the downstairs lobby after the lightning > talks? > > On Thu, Jul 14, 2016 at 10:49 AM, Ryan May wrote: > >> Fine with me. >> >> Ryan >> >> On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith wrote: >> >>> I have something at lunch, so dinner would be good for me too. >>> On Jul 13, 2016 7:46 PM, "Charles R Harris" >>> wrote: >>> >>>> Evening would work for me. Dinner? >>>> On Jul 13, 2016 2:43 PM, "Ryan May" wrote: >>>> >>>>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum < >>>>>> nathan12343 at gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> Maybe this can be an informal BOF session? >>>>>>> >>>>>> >>>>>> or maybe a formal BoF? after all, how formal do they get? >>>>>> >>>>>> Anyway, it was my understanding that we really needed to do some >>>>>> significant refactoring of how numpy deals with dtypes in order to do this >>>>>> kind of thing cleanly -- so where has that gone since last year? >>>>>> >>>>>> Maybe this conversation should be about how to build a more flexible >>>>>> dtype system generally, rather than specifically about unit support. >>>>>> (though unit support is a great use-case to focus on) >>>>>> >>>>>> >>>>> So Thursday's options seem to be in the standard BOF slot (up against >>>>> the Numfocus BOF), or doing something that evening, which would overlap at >>>>> least part of multiple happy hour events. I lean towards evening. Thoughts? >>>>> >>>>> Ryan >>>>> >>>>> -- >>>>> Ryan May >>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> Ryan May >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jul 14 12:05:30 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 14 Jul 2016 11:05:30 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: Where is "the downstairs lobby"? I can think of 4 places that I might describe that way depending on context :-) (Alternative: meet by the conference registration desk?) On Jul 14, 2016 10:52, "Nathan Goldbaum" wrote: > Fine with me as well. Meet in the downstairs lobby after the lightning > talks? > > On Thu, Jul 14, 2016 at 10:49 AM, Ryan May wrote: > >> Fine with me. >> >> Ryan >> >> On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith wrote: >> >>> I have something at lunch, so dinner would be good for me too. >>> On Jul 13, 2016 7:46 PM, "Charles R Harris" >>> wrote: >>> >>>> Evening would work for me. Dinner? >>>> On Jul 13, 2016 2:43 PM, "Ryan May" wrote: >>>> >>>>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum < >>>>>> nathan12343 at gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> Maybe this can be an informal BOF session? >>>>>>> >>>>>> >>>>>> or maybe a formal BoF? after all, how formal do they get? >>>>>> >>>>>> Anyway, it was my understanding that we really needed to do some >>>>>> significant refactoring of how numpy deals with dtypes in order to do this >>>>>> kind of thing cleanly -- so where has that gone since last year? >>>>>> >>>>>> Maybe this conversation should be about how to build a more flexible >>>>>> dtype system generally, rather than specifically about unit support. >>>>>> (though unit support is a great use-case to focus on) >>>>>> >>>>>> >>>>> So Thursday's options seem to be in the standard BOF slot (up against >>>>> the Numfocus BOF), or doing something that evening, which would overlap at >>>>> least part of multiple happy hour events. I lean towards evening. Thoughts? >>>>> >>>>> Ryan >>>>> >>>>> -- >>>>> Ryan May >>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> Ryan May >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Thu Jul 14 12:06:30 2016 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Thu, 14 Jul 2016 11:06:30 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: On Thu, Jul 14, 2016 at 11:05 AM, Nathaniel Smith wrote: > Where is "the downstairs lobby"? I can think of 4 places that I might > describe that way depending on context :-) > > (Alternative: meet by the conference registration desk?) > Sorry, that's what I meant. > > On Jul 14, 2016 10:52, "Nathan Goldbaum" wrote: > >> Fine with me as well. Meet in the downstairs lobby after the lightning >> talks? >> >> On Thu, Jul 14, 2016 at 10:49 AM, Ryan May wrote: >> >>> Fine with me. >>> >>> Ryan >>> >>> On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith wrote: >>> >>>> I have something at lunch, so dinner would be good for me too. >>>> On Jul 13, 2016 7:46 PM, "Charles R Harris" >>>> wrote: >>>> >>>>> Evening would work for me. Dinner? >>>>> On Jul 13, 2016 2:43 PM, "Ryan May" wrote: >>>>> >>>>>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker >>>>> > wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum < >>>>>>> nathan12343 at gmail.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> Maybe this can be an informal BOF session? >>>>>>>> >>>>>>> >>>>>>> or maybe a formal BoF? after all, how formal do they get? >>>>>>> >>>>>>> Anyway, it was my understanding that we really needed to do some >>>>>>> significant refactoring of how numpy deals with dtypes in order to do this >>>>>>> kind of thing cleanly -- so where has that gone since last year? >>>>>>> >>>>>>> Maybe this conversation should be about how to build a more flexible >>>>>>> dtype system generally, rather than specifically about unit support. >>>>>>> (though unit support is a great use-case to focus on) >>>>>>> >>>>>>> >>>>>> So Thursday's options seem to be in the standard BOF slot (up against >>>>>> the Numfocus BOF), or doing something that evening, which would overlap at >>>>>> least part of multiple happy hour events. I lean towards evening. Thoughts? >>>>>> >>>>>> Ryan >>>>>> >>>>>> -- >>>>>> Ryan May >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> >>> -- >>> Ryan May >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmbelgod at us.ibm.com Thu Jul 14 15:06:13 2016 From: bmbelgod at us.ibm.com (Brian M Belgodere) Date: Thu, 14 Jul 2016 19:06:13 +0000 Subject: [Numpy-discussion] 1.11.1 install fails on RHEL 7.2 (IBM Power8) with Python 2.7.12 Message-ID: An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jul 14 16:00:44 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 14 Jul 2016 15:00:44 -0500 Subject: [Numpy-discussion] 1.11.1 install fails on RHEL 7.2 (IBM Power8) with Python 2.7.12 In-Reply-To: References: Message-ID: Can you post the complete output from pip? On Jul 14, 2016 2:06 PM, "Brian M Belgodere" wrote: > > > While attempting to install numpy on a Power8 (ppc) machine Running RHEL > 7.2 I'm encountering the following error below. Numpy 1.11.0 installs and > works fine. > pip install numpy==1.11.1 > ERROR > error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g > -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include > -Ibuild/src.linux-ppc64le-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/src/npysort > -I/opt/share/Python-2.7.12/ppc64le/include/python2.7 > -Ibuild/src.linux-ppc64le-2.7/numpy/core/src/private > -Ibuild/src.linux-ppc64le-2.7/numpy/core/src/private > -Ibuild/src.linux-ppc64le-2.7/numpy/core/src/private -c > build/src.linux-ppc64le-2.7/numpy/core/src/npymath/npy_math_complex.c -o > build/temp.linux-ppc64le-2.7/build/src.linux-ppc64le-2.7/numpy/core/src/npymath/npy_math_complex.o" > failed with exit status 1 > > ---------------------------------------- > Rolling back uninstall of numpy > > Command "/opt/share/Python-2.7.12/ppc64le/bin/python -u -c "import > setuptools, > tokenize;__file__='/tmp/pip-build-LvkRpf/numpy/setup.py';exec(compile(getattr(tokenize, > 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" > install --record /tmp/pip-Bg_8Ko-record/install-record.txt > --single-version-externally-managed --compile" failed with error code 1 in > /tmp/pip-build-LvkRpf/numpy/ > ENVIRONMENT DETAILS > OS: Red Hat Enterprise Linux 7.2 Maipo > Kernel: ppc64le Linux 3.10.0-327.el7.ppc64le > CPU: IBM PowerPC G3 POWER8 (raw) @ 160x 3.491GHz > $pip list > Cython (0.24) > numpy (1.11.0) > pip (8.1.2) > scipy (0.17.1) > setuptools (24.0.3) > virtualenv (15.0.2) > $gcc -v > Using built-in specs. > COLLECT_GCC=gcc > COLLECT_LTO_WRAPPER=/usr/libexec/gcc/ppc64le-redhat-linux/4.8.5/lto-wrapper > Target: ppc64le-redhat-linux > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info --with-bugurl= > http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared > --enable-threads=posix --enable-checking=release --with-system-zlib > --enable-__cxa_atexit --disable-libunwind-exceptions > --enable-gnu-unique-object --enable-linker-build-id > --with-linker-hash-style=gnu > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto > --enable-plugin --enable-initfini-array --disable-libgcj > --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/isl-install > --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/cloog-install > --enable-gnu-indirect-function --enable-secureplt --with-long-double-128 > --enable-targets=powerpcle-linux --disable-multilib --with-cpu-64=power7 > --with-tune-64=power8 --build=ppc64le-redhat-linux > Thread model: posix > gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) > > > > > > *Brian Michael Belgodere* > > > Systems Management Engineering and Optimization > Software Engineer > IBM Research > Phone: +1-724-510-5308 > e-mail: bmbelgod at us.ibm.com > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Thu Jul 14 19:46:49 2016 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Thu, 14 Jul 2016 18:46:49 -0500 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: References: Message-ID: We are in room 203 On Thursday, July 14, 2016, Nathan Goldbaum wrote: > > > On Thu, Jul 14, 2016 at 11:05 AM, Nathaniel Smith > wrote: > >> Where is "the downstairs lobby"? I can think of 4 places that I might >> describe that way depending on context :-) >> >> (Alternative: meet by the conference registration desk?) >> > Sorry, that's what I meant. > > >> >> On Jul 14, 2016 10:52, "Nathan Goldbaum" > > wrote: >> >>> Fine with me as well. Meet in the downstairs lobby after the lightning >>> talks? >>> >>> On Thu, Jul 14, 2016 at 10:49 AM, Ryan May >> > wrote: >>> >>>> Fine with me. >>>> >>>> Ryan >>>> >>>> On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith >>> > wrote: >>>> >>>>> I have something at lunch, so dinner would be good for me too. >>>>> On Jul 13, 2016 7:46 PM, "Charles R Harris" >>>> > wrote: >>>>> >>>>>> Evening would work for me. Dinner? >>>>>> On Jul 13, 2016 2:43 PM, "Ryan May" >>>>> > wrote: >>>>>> >>>>>>> On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker < >>>>>>> chris.barker at noaa.gov >>>>>>> > wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum < >>>>>>>> nathan12343 at gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Maybe this can be an informal BOF session? >>>>>>>>> >>>>>>>> >>>>>>>> or maybe a formal BoF? after all, how formal do they get? >>>>>>>> >>>>>>>> Anyway, it was my understanding that we really needed to do some >>>>>>>> significant refactoring of how numpy deals with dtypes in order to do this >>>>>>>> kind of thing cleanly -- so where has that gone since last year? >>>>>>>> >>>>>>>> Maybe this conversation should be about how to build a more >>>>>>>> flexible dtype system generally, rather than specifically about unit >>>>>>>> support. (though unit support is a great use-case to focus on) >>>>>>>> >>>>>>>> >>>>>>> So Thursday's options seem to be in the standard BOF slot (up >>>>>>> against the Numfocus BOF), or doing something that evening, which would >>>>>>> overlap at least part of multiple happy hour events. I lean towards >>>>>>> evening. Thoughts? >>>>>>> >>>>>>> Ryan >>>>>>> >>>>>>> -- >>>>>>> Ryan May >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> >>>>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> >>>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> >>>> -- >>>> Ryan May >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmbelgod at us.ibm.com Thu Jul 14 21:36:46 2016 From: bmbelgod at us.ibm.com (Brian M Belgodere) Date: Fri, 15 Jul 2016 01:36:46 +0000 Subject: [Numpy-discussion] 1.11.1 install fails on RHEL 7.2 (IBM Power8) with Python 2.7.12 (Nathaniel Smith response) FULL INSTALL LOG In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From oleksandr.pavlyk at intel.com Thu Jul 14 21:53:19 2016 From: oleksandr.pavlyk at intel.com (Pavlyk, Oleksandr) Date: Fri, 15 Jul 2016 01:53:19 +0000 Subject: [Numpy-discussion] Design feedback solicitation In-Reply-To: References: <4C9EDA7282E297428F3986994EB0FBD38342AA@ORSMSX110.amr.corp.intel.com> Message-ID: <4C9EDA7282E297428F3986994EB0FBD38381AE@ORSMSX110.amr.corp.intel.com> Hi Robert, Thank you for the pointers. I think numpy.random should have a mechanism to choose between methods for generating the underlying randomness dynamically, at a run-time, as well as an extensible framework, where developers could add more methods. The default would be MT19937 for backwards compatibility. It is important to be able to do this at a run-time, as it would allow one to use different algorithms in different threads (like different members of the parallel Mersenne twister family of generators, see MT2203). The framework should allow to define randomness as a bit stream, a stream of fixed size integers, or a stream of uniform reals (32 or 64 bits). This is a lot of like MKL?s abstract method for basic pseudo-random number generation. https://software.intel.com/en-us/node/590373 Each method should provide routines to sample from uniform distributions over reals (in floats and doubles), as well as over integers. All remaining non-uniform distributions build on top of these uniform streams. I think it is pretty important to refactor numpy.random to allow the underlying generators to produce a given number of independent variates at a time. There could be convenience wrapper functions to allow to get one variate for backwards compatibility, but this change in design would allow for better efficiency, as sampling a vector of random variates at once is often faster than repeated sampling of one at a time due to set-up cost, vectorization, etc. Finally, methods to sample particular distribution should uniformly support method keyword argument. Because method names vary from distribution to distribution, it should ideally be programmatically discoverable which methods are supported for a given distribution. For instance, the standard normal distribution could support method=?Inversion?, method=?Box-Muller?, method=?Ziggurat?, method=?Box-Muller-Marsaglia? (the one used in numpy.random right now), as well as bunch of non-named methods based on transformed rejection method (see http://statistik.wu-wien.ac.at/anuran/ ) It would also be good if one could dynamically register a new method to sample from a non-uniform distribution. This would allow, for instance, to automatically add methods to sample certain non-uniform distribution by directly calling into MKL (or other library), when available, instead of building them from uniforms (which may remain a fall-through method). The linked project is a good start, but the choice of the underlying algorithm needs to be made at a run-time, as far as I understood, and the only provided interface to query random variates is one at a time, just like it is currently the case in numpy.random. Oleksandr From: NumPy-Discussion [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern Sent: Friday, June 17, 2016 10:23 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Design feedback solicitation On Fri, Jun 17, 2016 at 4:08 PM, Pavlyk, Oleksandr > wrote: > > Hi, > > I am new to this list, so I will start with an introduction. My name is Oleksandr Pavlyk. I now work at Intel Corp. on the Intel Distribution for Python, and previously worked at Wolfram Research for 12 years. My latest project was to write a mirror to numpy.random, named numpy.random_intel. The module uses MKL to sample from different distributions for efficiency. It provides support for different underlying algorithms for basic pseudo-random number generation, i.e. in addition to MT19937, it also provides SFMT19937, MT2203, etc. > > I recently published a blog about it: > > https://software.intel.com/en-us/blogs/2016/06/15/faster-random-number-generation-in-intel-distribution-for-python > > I originally attempted to simply replace numpy.random in the Intel Distribution for Python with the new module, but due to fixed seed backwards incompatibility this results in numerous test failures in numpy, scipy, pandas and other modules. > > Unlike numpy.random, the new module generates a vector of random numbers at a time, which can be done faster than repeatedly generating the same number of variates one at a time. > > The source code for the new module is not upstreamed yet, and this email is meant to solicit early community feedback to allow for faster acceptance of the proposed changes. Cool! You can find pertinent discussion here: https://github.com/numpy/numpy/issues/6967 And the current effort for adding new core PRNGs here: https://github.com/bashtage/ng-numpy-randomstate -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 14 22:14:53 2016 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Jul 2016 03:14:53 +0100 Subject: [Numpy-discussion] Design feedback solicitation In-Reply-To: <4C9EDA7282E297428F3986994EB0FBD38381AE@ORSMSX110.amr.corp.intel.com> References: <4C9EDA7282E297428F3986994EB0FBD38342AA@ORSMSX110.amr.corp.intel.com> <4C9EDA7282E297428F3986994EB0FBD38381AE@ORSMSX110.amr.corp.intel.com> Message-ID: On Fri, Jul 15, 2016 at 2:53 AM, Pavlyk, Oleksandr < oleksandr.pavlyk at intel.com> wrote: > > Hi Robert, > > Thank you for the pointers. > > I think numpy.random should have a mechanism to choose between methods for generating the underlying randomness dynamically, at a run-time, as well as an extensible framework, where developers could add more methods. The default would be MT19937 for backwards compatibility. It is important to be able to do this at a run-time, as it would allow one to use different algorithms in different threads (like different members of the parallel Mersenne twister family of generators, see MT2203). > > The framework should allow to define randomness as a bit stream, a stream of fixed size integers, or a stream of uniform reals (32 or 64 bits). This is a lot of like MKL?s abstract method for basic pseudo-random number generation. > > Each method should provide routines to sample from uniform distributions over reals (in floats and doubles), as well as over integers. > > All remaining non-uniform distributions build on top of these uniform streams. ng-numpy-randomstate does all of these. > I think it is pretty important to refactor numpy.random to allow the underlying generators to produce a given number of independent variates at a time. There could be convenience wrapper functions to allow to get one variate for backwards compatibility, but this change in design would allow for better efficiency, as sampling a vector of random variates at once is often faster than repeated sampling of one at a time due to set-up cost, vectorization, etc. The underlying C implementation is an implementation detail, so the refactoring that you suggest has no backwards compatibility constraints. > Finally, methods to sample particular distribution should uniformly support method keyword argument. Because method names vary from distribution to distribution, it should ideally be programmatically discoverable which methods are supported for a given distribution. For instance, the standard normal distribution could support method=?Inversion?, method=?Box-Muller?, method=?Ziggurat?, method=?Box-Muller-Marsaglia? (the one used in numpy.random right now), as well as bunch of non-named methods based on transformed rejection method (see http://statistik.wu-wien.ac.at/anuran/ ) That is one of the items under discussion. I personally prefer that one simply exposes named methods for each different scheme (e.g. ziggurat_normal(), etc.). > It would also be good if one could dynamically register a new method to sample from a non-uniform distribution. This would allow, for instance, to automatically add methods to sample certain non-uniform distribution by directly calling into MKL (or other library), when available, instead of building them from uniforms (which may remain a fall-through method). > > The linked project is a good start, but the choice of the underlying algorithm needs to be made at a run-time, That's what happens. You instantiate the RandomState class that you want. > as far as I understood, and the only provided interface to query random variates is one at a time, just like it is currently the case > in numpy.random. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jul 15 04:22:35 2016 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 15 Jul 2016 10:22:35 +0200 Subject: [Numpy-discussion] Custom Dtype/Units discussion In-Reply-To: (sfid-20160715_014815_905698_4FD28EAA) References: (sfid-20160715_014815_905698_4FD28EAA) Message-ID: <1468570955.31164.1.camel@sipsolutions.net> On Do, 2016-07-14 at 18:46 -0500, Nathan Goldbaum wrote: > We are in room 203 > You guys were probably doing that anyway, and I know you are too busy right now. But if there were some nice ideas/plans from this discussion, related to Numpy or not, I would appreciate a lot if one of you can send a few notes. Have some nice last days at SciPy! - Sebastian > On Thursday, July 14, 2016, Nathan Goldbaum > wrote: > > > > > > On Thu, Jul 14, 2016 at 11:05 AM, Nathaniel Smith > > wrote: > > > Where is "the downstairs lobby"? I can think of 4 places that I > > > might describe that way depending on context :-) > > > (Alternative: meet by the conference registration desk?) > > > > > Sorry, that's what I meant. > > ? > > > > > > On Jul 14, 2016 10:52, "Nathan Goldbaum" > > > wrote: > > > > Fine with me as well. Meet in the downstairs lobby after the > > > > lightning talks? > > > > > > > > On Thu, Jul 14, 2016 at 10:49 AM, Ryan May > > > > wrote: > > > > > Fine with me. > > > > > > > > > > Ryan > > > > > > > > > > On Thu, Jul 14, 2016 at 12:48 AM, Nathaniel Smith > > > > com> wrote: > > > > > > I have something at lunch, so dinner would be good for me > > > > > > too. > > > > > > On Jul 13, 2016 7:46 PM, "Charles R Harris" > > > > > s at gmail.com> wrote: > > > > > > > Evening would work for me. Dinner? > > > > > > > On Jul 13, 2016 2:43 PM, "Ryan May" > > > > > > > wrote: > > > > > > > > On Mon, Jul 11, 2016 at 12:39 PM, Chris Barker > > > > > > > arker at noaa.gov> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum > > > > > > > > han12343 at gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > Maybe this can be an informal BOF session? > > > > > > > > > > > > > > > > > > > or ?maybe a formal BoF? after all, how formal do they > > > > > > > > > get? > > > > > > > > > > > > > > > > > > Anyway, it was my understanding that we really needed > > > > > > > > > to do some significant refactoring of how numpy deals > > > > > > > > > with dtypes in order to do this kind of thing cleanly > > > > > > > > > -- so where has that gone since last year?? > > > > > > > > > > > > > > > > > > Maybe this conversation should be about how to build > > > > > > > > > a more flexible dtype system generally, rather than > > > > > > > > > specifically about unit support. (though unit support > > > > > > > > > is a great use-case to focus on) > > > > > > > > > > > > > > > > > > > > > > > > > > So Thursday's options seem to be in the standard BOF > > > > > > > > slot (up against the Numfocus BOF), or doing something > > > > > > > > that evening, which would overlap at least part of > > > > > > > > multiple happy hour events. I lean towards evening. > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > > > --? > > > > > > > > Ryan May > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > NumPy-Discussion mailing list > > > > > > > > NumPy-Discussion at scipy.org > > > > > > > > https://mail.scipy.org/mailman/listinfo/numpy-discussio > > > > > > > > n > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > NumPy-Discussion mailing list > > > > > > > NumPy-Discussion at scipy.org > > > > > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > _______________________________________________ > > > > > > NumPy-Discussion mailing list > > > > > > NumPy-Discussion at scipy.org > > > > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > > > > > > > > > --? > > > > > Ryan May > > > > > > > > > > > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at scipy.org > > > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at scipy.org > > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From Nicolas.Rougier at inria.fr Fri Jul 15 09:48:34 2016 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Fri, 15 Jul 2016 08:48:34 -0500 Subject: [Numpy-discussion] 100 Numpy exercises complete ! Message-ID: <4E3FBC3F-C790-4B75-BF49-15DB12EDA24E@inria.fr> Hi all, It's my great pleasure to announce that "100 Numpy exercises" is now complete. I've also made a notebook out of them such that you can now test them on binder. https://github.com/rougier/numpy-100 If you spot errors or have better solutions to propose, PR welcome. (I'm still fighting to fix exercise #54 that does not work as expected). Nicolas From matti.picus at gmail.com Fri Jul 15 09:50:37 2016 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 15 Jul 2016 08:50:37 -0500 Subject: [Numpy-discussion] runtests.py fails f2py test Message-ID: <5788EA2D.5000205@gmail.com> Am I missing something simple? I - install git, subversion, gcc, gfortran (Ubuntu 16.04) - create a clean python2 virtual env (no numpy) - activate it - git clone numpy - cd into it - python runtests.py - wait And it fails tests because it cannot find f2py. Then I - python setup.py install # to install numpy - cd numpy/f2py - python setup.py build And setup fails: F2PY Version 2 Traceback (most recent call last): File "setup.py", line 117, in **config) TypeError: setup() got multiple values for keyword argument 'version' Can someone give me a one-line hint what I am doing wrong? Matti From faltet at gmail.com Sun Jul 17 05:57:54 2016 From: faltet at gmail.com (Francesc Alted) Date: Sun, 17 Jul 2016 11:57:54 +0200 Subject: [Numpy-discussion] ANN: numexpr 2.6.1 released Message-ID: ========================= Announcing Numexpr 2.6.1 ========================= What's new ========== This is a maintenance release that fixes a performance regression in some situations. More specifically, the BLOCK_SIZE1 constant has been set to 1024 (down from 8192). This allows for better cache utilization when there are many operands and with VML. Fixes #221. Also, support for NetBSD has been added. Thanks to Thomas Klausner. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst What's Numexpr ============== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From nr4qewd6v4 at snkmail.com Sun Jul 17 14:53:42 2016 From: nr4qewd6v4 at snkmail.com (.) Date: Sun, 17 Jul 2016 14:53:42 -0400 Subject: [Numpy-discussion] Numpy Jenkin's Traub? In-Reply-To: <5787120F.102@kinnear.ca> References: <5787120F.102@kinnear.ca> Message-ID: <589-1468781649-221769@sneakemail.com> There was a little discussion of putting Aberth's method into SciPy. I'm not sure how those methods compare. https://github.com/scipy/scipy/issues/5999 On Thu, Jul 14, 2016 at 12:16 AM, Ryan J. Kinnear ryan-at-kinnear.ca |numpy mailing list/Example Allow| wrote: > Dear list, > > I'm working on implementing an ARMA(p, q) modeling method in Python. The > particular method is described in section 4.7.1 of Statistical Digital > Signal Processing and Modeling by Hayes. > > The upshot is that I need to calculate the roots of a polynomial. > > I've learned that this is a numerically ill conditioned problem, and have > found many cases where np.roots produces roots that are clearly spurious. > This is causing me a lot of issues. > > I have done some further searching and learned of the "Jenkins-Traub" > algorithm for root finding, which seems to be considered the most robust > method. I found a Python implementation of this method here ( > https://github.com/vrdabomb5717/jenkins_traub), and it is certainly much > more robust than np.roots. > > Are there reasons for Jenkins-Traub not being implemented as part of > Numpy? It is built into Scilab ( > https://help.scilab.org/docs/6.0.0/en_US/roots.html) Is anyone working > on getting it into Numpy? > > -RJK > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitikanth.kashyap at icloud.com Mon Jul 18 12:14:00 2016 From: shitikanth.kashyap at icloud.com (Shitikanth Kashyap) Date: Mon, 18 Jul 2016 12:14:00 -0400 Subject: [Numpy-discussion] Building numpy with ATLAS Message-ID: <495ACA7D-FD81-435A-9118-1AB06CC28B12@icloud.com> Hey Everyone, I am trying to build numpy-1.11.1 with ATLAS libraries. The build completes successfully, and I am able to use numpy, but there seems to be some problem with my config because numpy only ever uses one core, even when multiplying 5000x5000 matrices with numpy.dot. My site.cfg is: [default] library_dirs = /u3/s4kashya/.local/atlas3.11/lib:/u3/s4kashya/.local/lib:/usr/local/lib:/usr/lib include_dirs = /u3/s4kashya/.local/atlas3.11/include:/u3/s4kashya/.local/include:/usr/local/include:/usr/include [atlas] libraries = ptlapack, ptf77blas, ptcblas, atlas library_dirs = /u3/s4kashya/.local/atlas3.11/lib include_dirs = /u3/s4kashya/.local/atlas3.11/include Output from numpy.__config__.show(): atlas_3_10_blas_threads_info: libraries = ['tatlas', 'ptlapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/u3/s4kashya/.local/atlas3.11/lib'] define_macros = [('HAVE_CBLAS', None), ('NO_ATLAS_INFO', -1)] language = c include_dirs = ['/u3/s4kashya/.local/atlas3.11/include'] lapack_opt_info: libraries = ['tatlas', 'tatlas', 'tatlas', 'ptlapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/u3/s4kashya/.local/atlas3.11/lib'] define_macros = [('NO_ATLAS_INFO', -1)] language = f77 include_dirs = ['/u3/s4kashya/.local/atlas3.11/include'] blas_opt_info: libraries = ['tatlas', 'ptlapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/u3/s4kashya/.local/atlas3.11/lib'] define_macros = [('HAVE_CBLAS', None), ('NO_ATLAS_INFO', -1)] language = c include_dirs = ['/u3/s4kashya/.local/atlas3.11/include'] openblas_info: NOT AVAILABLE openblas_lapack_info: NOT AVAILABLE atlas_3_10_threads_info: libraries = ['tatlas', 'tatlas', 'tatlas', 'ptlapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/u3/s4kashya/.local/atlas3.11/lib'] define_macros = [('NO_ATLAS_INFO', -1)] language = f77 include_dirs = ['/u3/s4kashya/.local/atlas3.11/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE Any ideas what could be going wrong? Thanks, Shitikanth From nevion at gmail.com Mon Jul 18 17:15:54 2016 From: nevion at gmail.com (Jason Newton) Date: Mon, 18 Jul 2016 17:15:54 -0400 Subject: [Numpy-discussion] deterministic, reproducible matmul / __matmult_ In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 3:27 PM, Pauli Virtanen wrote: > Mon, 11 Jul 2016 13:01:49 -0400, Jason Newton kirjoitti: >> Does the ML have any ideas on how one could get a matmul that will not >> allow any funny business on the evaluation of the products? Funny >> business here is something like changing the evaluation order additions >> of terms. I want strict IEEE 754 compliance - no 80 bit registers, and >> perhaps control of the rounding mode, no unsafe math optimizations. > > If you link Numpy with a BLAS and LAPACK libraries that have been > compiled for this purpose, and turn on the compiler flags that enforce > strict IEEE (and disable SSE) when compiling Numpy, you probably will get > reproducible builds. Numpy itself just offloads the dot computations to > BLAS, so if your BLAS is reproducible, things should mainly be OK. The matrix multiplication of the reference blas hard to follow, so that's good. Generalized Inverse is a little more difficult. I've actually had problems building numpy w/ ref blas under windows..., anyone know where and how to take a ready made cblas dll and get an existing numpy installation (e.g. anaconda) to use it? > > You may also need to turn off the SSE optimizations in Numpy, because > these can make results depend on memory alignment --- not in dot > products, but in other computations. > > Out of curiosity, what is the application where this is necessary? > Maybe there is a numerically stable formulation? This can come up in recursive processes or anywhere where parallelism (vectorization or other kinds) and the need for reproducable code exists (there are many reasons you'd want this, see slides below). As I mentioned, I desire to have reproducable calculations when involving alternative implementations with things like c modules, MPI, FPGAs, GPUs coming into the picture. You can usually figure out a strategy to do this when you write everything yourself, but I'd love to write things in numpy and have it just choose simple / straight forward implementations that are usually what everything boils down to onto these other devices, at least in meaningful peaces. There may be other times where it gets more complicated than what i mention here, but it is still very useful as a building block for those cases (which are often multiple accumulators/partitioning, tree like reduction ordering). I looked into reproblas further and also I asked the authors of repoblas to add a cblas wrapper in the hopes I might sometime have numpy working on top of it. I read alot of their research recently and it's pretty cool - they get better accuracy than most implementations and the performance is pretty good. Check out slides here http://people.eecs.berkeley.edu/~hdnguyen/public/talks/ARITH21_Fast_Sum.pdf (skip to slide 24 for accuracy) - the takeaway here is you actually do quite a bit better in precision/accuracy with their sum method, than the naive and alternative implementations. The cpu performance is worth the trade and really not bad at all for most operations and purposes - especially since they hand scalability very well. They also just posted a monster of a technical report here https://www.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-121.html - this details recent performance using avx and sse, if anyone is interested. I'd love to have a library like this that I could just tell numpy to switch out to at runtime - at the start of a script. -Jason From ssanderson at quantopian.com Mon Jul 18 17:20:33 2016 From: ssanderson at quantopian.com (Scott Sanderson) Date: Mon, 18 Jul 2016 17:20:33 -0400 Subject: [Numpy-discussion] isnan() equivalent for np.NaT? Message-ID: Hi All, I'm working on upgrading Zipline (github.com/quantopian/zipline) to the latest numpy, and I'm getting a FutureWarnings about the upcoming change in the behavior of comparisons on np.NaT. I'd like to be able to do checks for NaT in a way that's forwards-compatible, but I couldn't find a function analogous to `np.isnan` for NaT. Am I missing something that already exists? If not, is there interest in such a function? I'd like to be able to pre-emptively fix the warnings in Zipline so that we're ready when the change actually happens, but it doesn't seem like the necessary tools are available right now. -Scott -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerrit.holl at gmail.com Mon Jul 18 18:39:15 2016 From: gerrit.holl at gmail.com (Gerrit Holl) Date: Mon, 18 Jul 2016 23:39:15 +0100 Subject: [Numpy-discussion] isnan() equivalent for np.NaT? In-Reply-To: References: Message-ID: On 18 July 2016 at 22:20, Scott Sanderson wrote: > I'm working on upgrading Zipline (github.com/quantopian/zipline) to the > latest numpy, and I'm getting a FutureWarnings about the upcoming change in > the behavior of comparisons on np.NaT. I'd like to be able to do checks for > NaT in a way that's forwards-compatible, but I couldn't find a function > analogous to `np.isnan` for NaT. Am I missing something that already > exists? If not, is there interest in such a function? I'd like to be able > to pre-emptively fix the warnings in Zipline so that we're ready when the > change actually happens, but it doesn't seem like the necessary tools are > available right now. Hi Scott, see https://github.com/numpy/numpy/issues/5610 Gerrit. From shoyer at gmail.com Mon Jul 18 19:35:59 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 18 Jul 2016 16:35:59 -0700 Subject: [Numpy-discussion] isnan() equivalent for np.NaT? In-Reply-To: References: Message-ID: Agreed -- this would be really nice to have. For now, the best you can do is something like the following: def is_na(x): x = np.asarray(x) if np.issubdtype(x.dtype, (np.datetime64, np.timedelta64)): # ugh int_min = np.iinfo(np.int64).min return x.view('int64') == int_min else: return np.isnan(x) On Mon, Jul 18, 2016 at 3:39 PM, Gerrit Holl wrote: > On 18 July 2016 at 22:20, Scott Sanderson > wrote: > > I'm working on upgrading Zipline (github.com/quantopian/zipline) to the > > latest numpy, and I'm getting a FutureWarnings about the upcoming change > in > > the behavior of comparisons on np.NaT. I'd like to be able to do checks > for > > NaT in a way that's forwards-compatible, but I couldn't find a function > > analogous to `np.isnan` for NaT. Am I missing something that already > > exists? If not, is there interest in such a function? I'd like to be > able > > to pre-emptively fix the warnings in Zipline so that we're ready when the > > change actually happens, but it doesn't seem like the necessary tools are > > available right now. > > Hi Scott, > > see https://github.com/numpy/numpy/issues/5610 > > Gerrit. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitikanth.kashyap at icloud.com Tue Jul 19 00:35:36 2016 From: shitikanth.kashyap at icloud.com (Shitikanth Kashyap) Date: Tue, 19 Jul 2016 00:35:36 -0400 Subject: [Numpy-discussion] Building numpy with ATLAS In-Reply-To: <495ACA7D-FD81-435A-9118-1AB06CC28B12@icloud.com> References: <495ACA7D-FD81-435A-9118-1AB06CC28B12@icloud.com> Message-ID: <83231E2D-32BE-4B15-9968-088A8A527CE3@icloud.com> As it turns out, I just had to run "python setup.py build --force" after changing my site.cfg file in order to recompile numpy/core/multiarray.so. From ecemsogancioglu at gmail.com Tue Jul 19 09:53:52 2016 From: ecemsogancioglu at gmail.com (=?UTF-8?Q?Ecem_soganc=C4=B1oglu?=) Date: Tue, 19 Jul 2016 13:53:52 +0000 Subject: [Numpy-discussion] Performance issue in covariance function in Numpy 1.9 and later Message-ID: Hello All, there seems to be a performance issue with the covariance function in numpy 1.9 and later. Code example: *np.cov(np.random.randn(700,37000))* In numpy 1.8, this line of code requires 4.5755 seconds. In numpy 1.9 and later, the same line of code requires more than 30.3709 s execution time. Has anyone else observed this problem and is there a known bugfix? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 19 15:44:23 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 19 Jul 2016 21:44:23 +0200 Subject: [Numpy-discussion] Performance issue in covariance function in Numpy 1.9 and later In-Reply-To: References: Message-ID: On Tue, Jul 19, 2016 at 3:53 PM, Ecem soganc?oglu wrote: > Hello All, > > there seems to be a performance issue with the covariance function in > numpy 1.9 and later. > > Code example: > *np.cov(np.random.randn(700,37000))* > > In numpy 1.8, this line of code requires 4.5755 seconds. > In numpy 1.9 and later, the same line of code requires more than 30.3709 s > execution time. > Hi Ecem, can you make sure to use the exact same random array as input to np.cov when testing this? Also timing just the function call you're interested in would be good; the creating of your 2-D array takes longer than the np.cov call: In [5]: np.random.seed(1234) In [6]: x = np.random.randn(700,37000) In [7]: %timeit np.cov(x) 1 loops, best of 3: 572 ms per loop In [8]: %timeit np.random.randn(700, 37000) 1 loops, best of 3: 1.26 s per loop Cheers, Ralf > Has anyone else observed this problem and is there a known bugfix? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 19 17:11:20 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 19 Jul 2016 23:11:20 +0200 Subject: [Numpy-discussion] runtests.py fails f2py test In-Reply-To: <5788EA2D.5000205@gmail.com> References: <5788EA2D.5000205@gmail.com> Message-ID: On Fri, Jul 15, 2016 at 3:50 PM, Matti Picus wrote: > Am I missing something simple? I > - install git, subversion, gcc, gfortran (Ubuntu 16.04) > - create a clean python2 virtual env (no numpy) > - activate it > - git clone numpy > - cd into it > - python runtests.py > - wait > And it fails tests because it cannot find f2py. > I can confirm that runtests.py doesn't install f2py into the virtualenv bin directory. This issue is hard to run into though, because if you've ever installed numpy on your system before (outside a virtualenv), then you probably already have f2py on your PATH (it's at .local/bin/f2py). > > Then I > - python setup.py install # to install numpy > - cd numpy/f2py > - python setup.py build > And setup fails: > > F2PY Version 2 > Traceback (most recent call last): > File "setup.py", line 117, in > **config) > TypeError: setup() got multiple values for keyword argument 'version' > > f2py isn't supposed to be installed standalone, I'm not surprised that that doesn't work. Ralf > Can someone give me a one-line hint what I am doing wrong? > Matti > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jladasky at itu.edu Tue Jul 19 17:41:04 2016 From: jladasky at itu.edu (John Ladasky) Date: Tue, 19 Jul 2016 14:41:04 -0700 Subject: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? Message-ID: Hi there, I've been using Numpy for several years and appreciate it very much. The following minimal code has been tried on Python 3.4 and 3.5, with Numpy 1.8 and Numpy 1.11, respectively. I want to temporarily change the way that a Numpy array is printed, then change it back. import numpy as np a = np.random.random((4,3)) print(a, "\n") opt = np.get_printoptions() np.set_printoptions(precision = 3, suppress = True) print(a, "\n") np.set_printoptions(opt) print(a, "\n\nDone.\n") Here is the traceback: Traceback (most recent call last): File "set_printoptions test.py", line 11, in print(a, "\n\nDone.\n") File "/usr/lib/python3/dist-packages/numpy/core/numeric.py", line 1615, in array_str return array2string(a, max_line_width, precision, suppress_small, ' ', "", str) File "/usr/lib/python3/dist-packages/numpy/core/arrayprint.py", line 454, in array2string separator, prefix, formatter=formatter) File "/usr/lib/python3/dist-packages/numpy/core/arrayprint.py", line 328, in _array2string _summaryEdgeItems, summary_insert)[:-1] File "/usr/lib/python3/dist-packages/numpy/core/arrayprint.py", line 523, in _formatArray summary_insert) File "/usr/lib/python3/dist-packages/numpy/core/arrayprint.py", line 497, in _formatArray word = format_function(a[-i]) + separator File "/usr/lib/python3/dist-packages/numpy/core/arrayprint.py", line 616, in __call__ s = self.format % x AttributeError: 'FloatFormat' object has no attribute 'format' It took me a while to discover the bug, on the second to last line. That line should read: np.set_printoptions(**opt) This unpacks the dictionary, opt, which was retrieved earlier in the program with the call to get_printoptions. I am wondering why set_printoptions would accept the dictionary as a single argument. Shouldn't that raise a TypeError or something? The AttributeError that is raised the next time that one attempts to print a numpy array is remote from the problem, and thus rather uninformative, which is why I did not immediately diagnose my error. Should this be considered a Numpy bug, or is there some reason that set_printoptions would legitimately need to accept a dictionary as a single argument? -- *John J. Ladasky Jr., Ph.D.* *Research Scientist* *International Technological University* *2711 N. First St, San Jose, CA 95134 USA* -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jul 19 17:43:56 2016 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Jul 2016 22:43:56 +0100 Subject: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? In-Reply-To: References: Message-ID: On Tue, Jul 19, 2016 at 10:41 PM, John Ladasky wrote: > Should this be considered a Numpy bug, or is there some reason that set_printoptions would legitimately need to accept a dictionary as a single argument? There is no such reason. One could certainly add more validation to the arguments to np.set_printoptions(). -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jladasky at itu.edu Tue Jul 19 17:49:10 2016 From: jladasky at itu.edu (John Ladasky) Date: Tue, 19 Jul 2016 14:49:10 -0700 Subject: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? In-Reply-To: References: Message-ID: Hi Robert, Thanks for your reply. If no one disagrees with you or with me that this is a Numpy bug, I would appreciate being directed to the appropriate page to submit a bug-fix request. On Tue, Jul 19, 2016 at 2:43 PM, Robert Kern wrote: > On Tue, Jul 19, 2016 at 10:41 PM, John Ladasky wrote: > > > Should this be considered a Numpy bug, or is there some reason that > set_printoptions would legitimately need to accept a dictionary as a single > argument? > > There is no such reason. One could certainly add more validation to the > arguments to np.set_printoptions(). > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *John J. Ladasky Jr., Ph.D.* *Research Scientist* *International Technological University* *2711 N. First St, San Jose, CA 95134 USA* -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Tue Jul 19 17:55:11 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Tue, 19 Jul 2016 14:55:11 -0700 Subject: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? In-Reply-To: References: Message-ID: https://github.com/numpy/numpy/issues From: John Ladasky Reply: Discussion of Numerical Python Date: 20 July 2016 at 7:49:10 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? Hi Robert, > > Thanks for your reply. If no one disagrees with you or with me that this > is a Numpy bug, I would appreciate being directed to the appropriate page > to submit a bug-fix request. > > > On Tue, Jul 19, 2016 at 2:43 PM, Robert Kern > wrote: > >> On Tue, Jul 19, 2016 at 10:41 PM, John Ladasky wrote: >> >> > Should this be considered a Numpy bug, or is there some reason that >> set_printoptions would legitimately need to accept a dictionary as a single >> argument? >> >> There is no such reason. One could certainly add more validation to the >> arguments to np.set_printoptions(). >> >> -- >> Robert Kern >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > *John J. Ladasky Jr., Ph.D.* > *Research Scientist* > *International Technological University* > *2711 N. First St, San Jose, CA 95134 USA* > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jladasky at itu.edu Tue Jul 19 18:37:34 2016 From: jladasky at itu.edu (John Ladasky) Date: Tue, 19 Jul 2016 15:37:34 -0700 Subject: [Numpy-discussion] Numpy set_printoptions, silent failure, bug? In-Reply-To: References: Message-ID: Thank you Juan, I've just joined GitHub and I've submitted the description of the bug. On Tue, Jul 19, 2016 at 2:55 PM, Juan Nunez-Iglesias wrote: > https://github.com/numpy/numpy/issues > > > From: John Ladasky > Reply: Discussion of Numerical Python > > Date: 20 July 2016 at 7:49:10 AM > To: Discussion of Numerical Python > > Subject: Re: [Numpy-discussion] Numpy set_printoptions, silent failure, > bug? > > Hi Robert, >> >> Thanks for your reply. If no one disagrees with you or with me that this >> is a Numpy bug, I would appreciate being directed to the appropriate page >> to submit a bug-fix request. >> >> >> On Tue, Jul 19, 2016 at 2:43 PM, Robert Kern >> wrote: >> >>> On Tue, Jul 19, 2016 at 10:41 PM, John Ladasky wrote: >>> >>> > Should this be considered a Numpy bug, or is there some reason that >>> set_printoptions would legitimately need to accept a dictionary as a single >>> argument? >>> >>> There is no such reason. One could certainly add more validation to the >>> arguments to np.set_printoptions(). >>> >>> -- >>> Robert Kern >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> *John J. Ladasky Jr., Ph.D.* >> *Research Scientist* >> *International Technological University* >> *2711 N. First St, San Jose, CA 95134 USA* >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *John J. Ladasky Jr., Ph.D.* *Research Scientist* *International Technological University* *2711 N. First St, San Jose, CA 95134 USA* -------------- next part -------------- An HTML attachment was scrubbed... URL: From story645 at gmail.com Tue Jul 19 22:28:04 2016 From: story645 at gmail.com (Hannah) Date: Tue, 19 Jul 2016 22:28:04 -0400 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) Message-ID: Hi, I started venturing down the rabbit hole of trying to write a patch to add support for numpy to convert python 3 dictionary keys (collections.abc.ViewMapping objects), which is open issue #5718 and am having trouble orienting myself. I'm unclear as to where the python entry point into array is (basically, what function np.array drops into and if this is in Python or C) and where/what language (fine with writing either) a patch that supports MappingViews would go. Any help getting oriented would be much appreciated. The reasoning for the patch is s that dicts are one of the most common Python datatypes and this specifically is because of an upstream issue of wanting dict support in matplotlib. Thanks, Hannah -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Wed Jul 20 08:52:55 2016 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Wed, 20 Jul 2016 14:52:55 +0200 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: On Wed, Jul 20, 2016 at 4:28 AM, Hannah wrote: > Hi, > I started venturing down the rabbit hole of trying to write a patch to add > support for numpy to convert python 3 dictionary keys > (collections.abc.ViewMapping objects), which is open issue #5718 and am > having trouble orienting myself. I'm unclear as to where the python entry > point into array is (basically, what function np.array drops into and if > this is in Python or C) and where/what language (fine with writing either) > a patch that supports MappingViews would go. Any help getting oriented > would be much appreciated. > Hi Hannah, ?p.array is written in C, and is part of the multiarray module that is defined here: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c The "array" name is mapped here: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L4093 to the function _array_fromobject defined here: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L1557 That functions does some checking and has a couple of fast paths for the case where the input is already an array or a subclass, but for the general case it relies on PyArray_CheckFromAny: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1848 which in turn calls Pyarray_FromAny: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1674 You will also haveto take a look at what goes on in PyArray_GetArrayParamsFromObject, which gets called by PyArray_FromAny; https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1428 as well as several other places, but I think they are all (or most of them) in ctors.c. You may also want to take a llok at PyArray_FromIter, which is the function that ultimately takes care of calls to np.fromiter: https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L3657 It's messy, but not that bad once you get used to it: good luck! Jaime > > The reasoning for the patch is s that dicts are one of the most common > Python datatypes and this specifically is because of an upstream issue of > wanting dict support in matplotlib. > > Thanks, > Hannah > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Wed Jul 20 11:07:15 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Wed, 20 Jul 2016 11:07:15 -0400 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: Jaime, This is a great intro for people looking to jump into the C side of things. I have been trying to figure out which bits are the important ones from looking at the code and the docs. Your post cut out most of the confusion. Is there some way you would consider adding something like this this to the docs? -Joe On Wed, Jul 20, 2016 at 8:52 AM, Jaime Fern?ndez del R?o wrote: > On Wed, Jul 20, 2016 at 4:28 AM, Hannah wrote: >> >> Hi, >> I started venturing down the rabbit hole of trying to write a patch to add >> support for numpy to convert python 3 dictionary keys >> (collections.abc.ViewMapping objects), which is open issue #5718 and am >> having trouble orienting myself. I'm unclear as to where the python entry >> point into array is (basically, what function np.array drops into and if >> this is in Python or C) and where/what language (fine with writing either) a >> patch that supports MappingViews would go. Any help getting oriented would >> be much appreciated. > > > Hi Hannah, > > ?p.array is written in C, and is part of the multiarray module that is > defined here: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c > > The "array" name is mapped here: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L4093 > > to the function _array_fromobject defined here: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L1557 > > That functions does some checking and has a couple of fast paths for the > case where the input is already an array or a subclass, but for the general > case it relies on PyArray_CheckFromAny: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1848 > > which in turn calls Pyarray_FromAny: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1674 > > You will also haveto take a look at what goes on in > PyArray_GetArrayParamsFromObject, which gets called by PyArray_FromAny; > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1428 > > as well as several other places, but I think they are all (or most of them) > in ctors.c. > > You may also want to take a llok at PyArray_FromIter, which is the function > that ultimately takes care of calls to np.fromiter: > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L3657 > > It's messy, but not that bad once you get used to it: good luck! > > Jaime > >> >> >> The reasoning for the patch is s that dicts are one of the most common >> Python datatypes and this specifically is because of an upstream issue of >> wanting dict support in matplotlib. >> >> Thanks, >> Hannah >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de > dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From story645 at gmail.com Wed Jul 20 15:39:11 2016 From: story645 at gmail.com (Hannah) Date: Wed, 20 Jul 2016 15:39:11 -0400 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: I second (and third & fourth &...) this Thanks so much for this, Jaime, it's exactly what I was looking for and couldn't find. Maybe this can be linked to in the contribute docs as an "orienting yourself in the codebase" page? On Wed, Jul 20, 2016 at 11:07 AM, Joseph Fox-Rabinovitz < jfoxrabinovitz at gmail.com> wrote: > Jaime, > > This is a great intro for people looking to jump into the C side of > things. I have been trying to figure out which bits are the important > ones from looking at the code and the docs. Your post cut out most of > the confusion. Is there some way you would consider adding something > like this this to the docs? > > -Joe > > > On Wed, Jul 20, 2016 at 8:52 AM, Jaime Fern?ndez del R?o > wrote: > > On Wed, Jul 20, 2016 at 4:28 AM, Hannah wrote: > >> > >> Hi, > >> I started venturing down the rabbit hole of trying to write a patch to > add > >> support for numpy to convert python 3 dictionary keys > >> (collections.abc.ViewMapping objects), which is open issue #5718 and am > >> having trouble orienting myself. I'm unclear as to where the python > entry > >> point into array is (basically, what function np.array drops into and if > >> this is in Python or C) and where/what language (fine with writing > either) a > >> patch that supports MappingViews would go. Any help getting oriented > would > >> be much appreciated. > > > > > > Hi Hannah, > > > > ?p.array is written in C, and is part of the multiarray module that is > > defined here: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c > > > > The "array" name is mapped here: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L4093 > > > > to the function _array_fromobject defined here: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L1557 > > > > That functions does some checking and has a couple of fast paths for the > > case where the input is already an array or a subclass, but for the > general > > case it relies on PyArray_CheckFromAny: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1848 > > > > which in turn calls Pyarray_FromAny: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1674 > > > > You will also haveto take a look at what goes on in > > PyArray_GetArrayParamsFromObject, which gets called by PyArray_FromAny; > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1428 > > > > as well as several other places, but I think they are all (or most of > them) > > in ctors.c. > > > > You may also want to take a llok at PyArray_FromIter, which is the > function > > that ultimately takes care of calls to np.fromiter: > > > > > https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L3657 > > > > It's messy, but not that bad once you get used to it: good luck! > > > > Jaime > > > >> > >> > >> The reasoning for the patch is s that dicts are one of the most common > >> Python datatypes and this specifically is because of an upstream issue > of > >> wanting dict support in matplotlib. > >> > >> Thanks, > >> Hannah > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > > > > > -- > > (\__/) > > ( O.o) > > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > planes de > > dominaci?n mundial. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Jul 21 06:07:59 2016 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 21 Jul 2016 12:07:59 +0200 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: On Wed, Jul 20, 2016 at 9:39 PM, Hannah wrote: > I second (and third & fourth &...) this > > Thanks so much for this, Jaime, it's exactly what I was looking for and > couldn't find. Maybe this can be linked to in the contribute docs as an > "orienting yourself in the codebase" page? > Glad to know it helped, realizing how this works was one of the big leaps in understanding the code base for myself too. We (my wife and I) have sent the kids on vacation with their grandparents, so when we are done enjoying our shortly recovered freedom hunting for Pokemons around Zurich, I'll try to find time to write this up in a form suitable for the docs, hopefully over the next couple of weeks. I have been known to not keep commitments like this in the past, so don't hold your breath just in case... Jaime > > On Wed, Jul 20, 2016 at 11:07 AM, Joseph Fox-Rabinovitz < > jfoxrabinovitz at gmail.com> wrote: > >> Jaime, >> >> This is a great intro for people looking to jump into the C side of >> things. I have been trying to figure out which bits are the important >> ones from looking at the code and the docs. Your post cut out most of >> the confusion. Is there some way you would consider adding something >> like this this to the docs? >> >> -Joe >> >> >> On Wed, Jul 20, 2016 at 8:52 AM, Jaime Fern?ndez del R?o >> wrote: >> > On Wed, Jul 20, 2016 at 4:28 AM, Hannah wrote: >> >> >> >> Hi, >> >> I started venturing down the rabbit hole of trying to write a patch to >> add >> >> support for numpy to convert python 3 dictionary keys >> >> (collections.abc.ViewMapping objects), which is open issue #5718 and am >> >> having trouble orienting myself. I'm unclear as to where the python >> entry >> >> point into array is (basically, what function np.array drops into and >> if >> >> this is in Python or C) and where/what language (fine with writing >> either) a >> >> patch that supports MappingViews would go. Any help getting oriented >> would >> >> be much appreciated. >> > >> > >> > Hi Hannah, >> > >> > ?p.array is written in C, and is part of the multiarray module that is >> > defined here: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c >> > >> > The "array" name is mapped here: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L4093 >> > >> > to the function _array_fromobject defined here: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/multiarraymodule.c#L1557 >> > >> > That functions does some checking and has a couple of fast paths for the >> > case where the input is already an array or a subclass, but for the >> general >> > case it relies on PyArray_CheckFromAny: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1848 >> > >> > which in turn calls Pyarray_FromAny: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1674 >> > >> > You will also haveto take a look at what goes on in >> > PyArray_GetArrayParamsFromObject, which gets called by PyArray_FromAny; >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L1428 >> > >> > as well as several other places, but I think they are all (or most of >> them) >> > in ctors.c. >> > >> > You may also want to take a llok at PyArray_FromIter, which is the >> function >> > that ultimately takes care of calls to np.fromiter: >> > >> > >> https://github.com/numpy/numpy/blob/maintenance/1.11.x/numpy/core/src/multiarray/ctors.c#L3657 >> > >> > It's messy, but not that bad once you get used to it: good luck! >> > >> > Jaime >> > >> >> >> >> >> >> The reasoning for the patch is s that dicts are one of the most common >> >> Python datatypes and this specifically is because of an upstream issue >> of >> >> wanting dict support in matplotlib. >> >> >> >> Thanks, >> >> Hannah >> >> >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > >> > >> > >> > -- >> > (\__/) >> > ( O.o) >> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >> planes de >> > dominaci?n mundial. >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Jul 21 07:47:53 2016 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 21 Jul 2016 13:47:53 +0200 Subject: [Numpy-discussion] StackOverflow documentation Message-ID: StackOverflow now also has documentation, and there already is a NumPy tag: http://stackoverflow.com/documentation/numpy Not sure what, if anything, do we want to do with this, nor how to handle not having to different sources with the same information. Any thoughts? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Thu Jul 21 09:12:14 2016 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Thu, 21 Jul 2016 15:12:14 +0200 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: I know it is slightly obnoxious to hold the "making a suggestion is to volunteer for it" -- but usually a PR to the docs is best made by someone who is trying to understand it rather than someone who already knows everything........ -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jul 21 09:37:37 2016 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Jul 2016 07:37:37 -0600 Subject: [Numpy-discussion] StackOverflow documentation In-Reply-To: References: Message-ID: On Thu, Jul 21, 2016 at 5:47 AM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > StackOverflow now also has documentation, and there already is a NumPy tag: > > http://stackoverflow.com/documentation/numpy > > Not sure what, if anything, do we want to do with this, nor how to handle > not having to different sources with the same information. Any thoughts? > > That's interesting. Not sure what to do there, maybe upload some of our documentation? I'm a bit worried as numpy documentation changes with every release. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at telenczuk.pl Thu Jul 21 10:24:15 2016 From: mail at telenczuk.pl (mail at telenczuk.pl) Date: Thu, 21 Jul 2016 16:24:15 +0200 Subject: [Numpy-discussion] StackOverflow documentation In-Reply-To: References: Message-ID: <5790db0ff744_146814d76f4b2@Pct-EqAlain-Z30.notmuch> >> StackOverflow now also has documentation, and there already is a NumPy tag: >> >> http://stackoverflow.com/documentation/numpy >> >> Not sure what, if anything, do we want to do with this, nor how to handle >> not having to different sources with the same information. Any thoughts? >From what I understand, it's not meant to replace real documentation. It's rather a collection of crowd-sourced instructive examples with explanations: https://stackoverflow.com/tour/documentation Since it is run by the community, perhaps it's not a bad idea to encourage people to share their examples. Cheers, Bartosz From jfoxrabinovitz at gmail.com Thu Jul 21 10:46:40 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Thu, 21 Jul 2016 10:46:40 -0400 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: I agree with you, and in that spirit I started working on something. I do welcome suggestions on where to place the new page though. Regards, -Joe On Thu, Jul 21, 2016 at 9:12 AM, Marten van Kerkwijk wrote: > I know it is slightly obnoxious to hold the "making a suggestion is to > volunteer for it" -- but usually a PR to the docs is best made by someone > who is trying to understand it rather than someone who already knows > everything........ > -- Marten > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From m.h.vankerkwijk at gmail.com Thu Jul 21 12:09:15 2016 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Thu, 21 Jul 2016 18:09:15 +0200 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: Yes, indeed, where should this be!? The logical place would be in the developer documentation, under "understanding the code & getting started" [1], but that is right now just a single paragraph. Others should definitely correct me if wrong, but I *think* this simply does not exist, and would suggest to move that subsection to its own section, and make your contribution a first entry in it ("the path to creating an array" or so). Hopefully, someone else will at some point add, e.g., a section on how ufuncs work (I always get quite confused following the long chain there...). Or even something as simple as copying an array (where transposing axes can have major impacts on speed). All the best, Marten [1] http://docs.scipy.org/doc/numpy-dev/dev/development_environment.html#understanding-the-code-getting-started ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Thu Jul 21 13:10:22 2016 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Thu, 21 Jul 2016 13:10:22 -0400 Subject: [Numpy-discussion] Python 3 dict support (issue #5718) In-Reply-To: References: Message-ID: I will do my best. I am not that familiar with rst or numpy docs, but that's what PRs are for after all. -Joe On Thu, Jul 21, 2016 at 12:09 PM, Marten van Kerkwijk wrote: > Yes, indeed, where should this be!? > > The logical place would be in the developer documentation, under > "understanding the code & getting started" [1], but that is right now just a > single paragraph. > > Others should definitely correct me if wrong, but I *think* this simply does > not exist, and would suggest to move that subsection to its own section, and > make your contribution a first entry in it ("the path to creating an array" > or so). Hopefully, someone else will at some point add, e.g., a section on > how ufuncs work (I always get quite confused following the long chain > there...). Or even something as simple as copying an array (where > transposing axes can have major impacts on speed). > > All the best, > > Marten > > [1] > http://docs.scipy.org/doc/numpy-dev/dev/development_environment.html#understanding-the-code-getting-started > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Thu Jul 21 15:10:27 2016 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 21 Jul 2016 19:10:27 +0000 (UTC) Subject: [Numpy-discussion] StackOverflow documentation References: <5790db0ff744_146814d76f4b2@Pct-EqAlain-Z30.notmuch> Message-ID: Thu, 21 Jul 2016 16:24:15 +0200, mail kirjoitti: [clip] > Since it is run by the community, perhaps it's not a bad idea to > encourage people to share their examples. I would perhaps rather encourage people to improve the "Numpy User Guide" or the main documentation. Of course, working on that requires a somewhat different level of committment than editing what's essentially Stackexchange-provided wiki (where content gets relicensed with attribution clauses where you have to reference stackexchange and not the original author directly). -- Pauli Virtanen From tombird1992 at gmail.com Thu Jul 21 15:43:30 2016 From: tombird1992 at gmail.com (Tom Bird) Date: Thu, 21 Jul 2016 20:43:30 +0100 Subject: [Numpy-discussion] Adding covariance matrix of fitted parameters to fit methods of numpy.polynomial In-Reply-To: <579124bb.88711c0a.38c7f.14d8@mx.google.com> References: <579124bb.88711c0a.38c7f.14d8@mx.google.com> Message-ID: <579125e9.8bc71c0a.18326.0f31@mx.google.com> See issue #7780: https://github.com/numpy/numpy/issues/7780 Would be good if the numpy.polynomial fit methods (e.g. numpy.polynomial.legendre.Legendre.fit) could return the covariance matrix of the fitted parameters, in the same way as numpy.polyfit. Let me know if there are any suggestions/concerns. Otherwise I?m happy to add the code and update the docs. Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Jul 21 17:59:32 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 21 Jul 2016 17:59:32 -0400 Subject: [Numpy-discussion] StackOverflow documentation In-Reply-To: References: <5790db0ff744_146814d76f4b2@Pct-EqAlain-Z30.notmuch> Message-ID: On Thu, Jul 21, 2016 at 3:10 PM, Pauli Virtanen wrote: > Thu, 21 Jul 2016 16:24:15 +0200, mail kirjoitti: > [clip] >> Since it is run by the community, perhaps it's not a bad idea to >> encourage people to share their examples. > > I would perhaps rather encourage people to improve the "Numpy User Guide" > or the main documentation. Of course, working on that requires a somewhat > different level of committment than editing what's essentially > Stackexchange-provided wiki (where content gets relicensed with > attribution clauses where you have to reference stackexchange and not the > original author directly). The way it looks right now, it is more of an example collection by topic, similar to the old scipy wiki examples for numpy. http://stackoverflow.com/documentation/python/196/comprehensions#t=201607212153155953853 It doesn't look like it will be a substitute for proper API documentation, docstrings with numpy standard. Longer examples or recipes on stackoverflow also have the problem that code doesn't have a different license and falls under the documentation license. So, I don't think it will become a substitute for good notebook or blog post examples either. (license is the usual one way street, they copy, we cannot copy back under our license) Josef > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From ecemsogancioglu at gmail.com Fri Jul 22 11:09:28 2016 From: ecemsogancioglu at gmail.com (=?UTF-8?Q?Ecem_soganc=C4=B1oglu?=) Date: Fri, 22 Jul 2016 15:09:28 +0000 Subject: [Numpy-discussion] Performance issue in covariance function in Numpy 1.9 and later In-Reply-To: References: Message-ID: Hi Ralf, Thank you so much for your answer. I finally figured out that the problem was because Numpy 1.9 was not linked to BLAS. I do not know why because I simply installed numpy 1.9 via the commands: apt-get install python3-numpy If anybody has the same problem, you may want to take a look into this: http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html Best Regards, Ecem On Fri, Jul 22, 2016 at 5:19 PM Ecem soganc?oglu wrote: > Dear Ralf, > > Thank you so much for your answer. > > I finally figured out that the problem was because Numpy 1.9 was not > linked to BLAS. I do not know why because I simply installed numpy 1.9 via > the commands: > > apt-get install python3-numpy > > If anybody has the same problem, you may want to take a look into this: > http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html > > Best Regards, > Ecem > > > > > On Tue, Jul 19, 2016 at 9:44 PM Ralf Gommers > wrote: > >> On Tue, Jul 19, 2016 at 3:53 PM, Ecem soganc?oglu < >> ecemsogancioglu at gmail.com> wrote: >> >>> Hello All, >>> >>> there seems to be a performance issue with the covariance function in >>> numpy 1.9 and later. >>> >>> Code example: >>> *np.cov(np.random.randn(700,37000))* >>> >>> In numpy 1.8, this line of code requires 4.5755 seconds. >>> In numpy 1.9 and later, the same line of code requires more than 30.3709 >>> s execution time. >>> >> >> Hi Ecem, can you make sure to use the exact same random array as input to >> np.cov when testing this? Also timing just the function call you're >> interested in would be good; the creating of your 2-D array takes longer >> than the np.cov call: >> >> In [5]: np.random.seed(1234) >> >> In [6]: x = np.random.randn(700,37000) >> >> In [7]: %timeit np.cov(x) >> 1 loops, best of 3: 572 ms per loop >> >> In [8]: %timeit np.random.randn(700, 37000) >> 1 loops, best of 3: 1.26 s per loop >> >> >> Cheers, >> Ralf >> >> >> >>> Has anyone else observed this problem and is there a known bugfix? >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asbuzza at gmail.com Sun Jul 24 15:54:49 2016 From: asbuzza at gmail.com (andy buzza) Date: Sun, 24 Jul 2016 14:54:49 -0500 Subject: [Numpy-discussion] f2py NotImplementedError Message-ID: Hello all, I have been trying to compile a relatively simple pair of Fortran files, one referencing a subroutine from another file (mainmodule.f90 references othermodule.f90). I have been able to compile them using a Fortran compiler, but receive a NotImplementedError when using f2py. Steps I use for f2py: $gfortran -shared -o othermodule.so othermodule.f90 -fPIC $f2py -c -l/path/othermodule -m mainmodule mainmodule.f90 I am running this on linux and wasn't sure how to correct the error. othermodule.f90 module moderator implicit none integer*1 :: i integer*8 :: fact,len real*8,dimension(:),allocatable :: ex contains subroutine submarine(ii,ff,exex) implicit none integer*1 :: ii integer*8 :: ff real*8 :: exex exex=exp(real(ii))/ff end subroutine submarine end module moderator mainmodule.f90 program mains use moderator implicit none len=10 allocate(ex(len)) fact=1 do i=1,len fact=fact*i call submarine(i,fact,ex(i)) if (i==1) then print*,"here's your ",i,"st number: ",ex(i) elseif (i==2) then print*,"here's your ",i,"nd number: ",ex(i) elseif (i==3) then print*,"here's your ",i,"rd number: ",ex(i) else print*,"here's your ",i,"th number: ",ex(i) endif enddo deallocate(ex) end program Thanks for the help, Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Mon Jul 25 13:54:00 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Mon, 25 Jul 2016 18:54:00 +0100 Subject: [Numpy-discussion] ANN: scipy 0.18.0 release Message-ID: On behalf of the Scipy development team I am pleased to announce the availability of Scipy 0.18.0. This release contains several great new features and a large number of bug fixes and various improvements, as detailed in the release notes below. 99 people contributed to this release over the course of six months. Thanks to everyone who contributed! This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.2 or greater. Source tarballs and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.18.0. OS X and Linux wheels are available from PyPI. For security-conscious, the wheels themselves are signed with my GPG key. Additionally, you can checksum the wheels and verify the checksums with those listed in the README file at https://github.com/scipy/scipy/releases/tag/v0.18.0. Cheers, Evgeni -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ========================== SciPy 0.18.0 Release Notes ========================== .. contents:: SciPy 0.18.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.19.x branch, and on adding new features on the master branch. This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.1 or greater. Highlights of this release include: - - A new ODE solver for two-point boundary value problems, `scipy.optimize.solve_bvp`. - - A new class, `CubicSpline`, for cubic spline interpolation of data. - - N-dimensional tensor product polynomials, `scipy.interpolate.NdPPoly`. - - Spherical Voronoi diagrams, `scipy.spatial.SphericalVoronoi`. - - Support for discrete-time linear systems, `scipy.signal.dlti`. New features ============ `scipy.integrate` improvements - ------------------------------ A solver of two-point boundary value problems for ODE systems has been implemented in `scipy.integrate.solve_bvp`. The solver allows for non-separated boundary conditions, unknown parameters and certain singular terms. It finds a C1 continious solution using a fourth-order collocation algorithm. `scipy.interpolate` improvements - -------------------------------- Cubic spline interpolation is now available via `scipy.interpolate.CubicSpline`. This class represents a piecewise cubic polynomial passing through given points and C2 continuous. It is represented in the standard polynomial basis on each segment. A representation of n-dimensional tensor product piecewise polynomials is available as the `scipy.interpolate.NdPPoly` class. Univariate piecewise polynomial classes, `PPoly` and `Bpoly`, can now be evaluated on periodic domains. Use ``extrapolate="periodic"`` keyword argument for this. `scipy.fftpack` improvements - ---------------------------- `scipy.fftpack.next_fast_len` function computes the next "regular" number for FFTPACK. Padding the input to this length can give significant performance increase for `scipy.fftpack.fft`. `scipy.signal` improvements - --------------------------- Resampling using polyphase filtering has been implemented in the function `scipy.signal.resample_poly`. This method upsamples a signal, applies a zero-phase low-pass FIR filter, and downsamples using `scipy.signal.upfirdn` (which is also new in 0.18.0). This method can be faster than FFT-based filtering provided by `scipy.signal.resample` for some signals. `scipy.signal.firls`, which constructs FIR filters using least-squares error minimization, was added. `scipy.signal.sosfiltfilt`, which does forward-backward filtering like `scipy.signal.filtfilt` but for second-order sections, was added. Discrete-time linear systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `scipy.signal.dlti` provides an implementation of discrete-time linear systems. Accordingly, the `StateSpace`, `TransferFunction` and `ZerosPolesGain` classes have learned a the new keyword, `dt`, which can be used to create discrete-time instances of the corresponding system representation. `scipy.sparse` improvements - --------------------------- The functions `sum`, `max`, `mean`, `min`, `transpose`, and `reshape` in `scipy.sparse` have had their signatures augmented with additional arguments and functionality so as to improve compatibility with analogously defined functions in `numpy`. Sparse matrices now have a `count_nonzero` method, which counts the number of nonzero elements in the matrix. Unlike `getnnz()` and ``nnz`` propety, which return the number of stored entries (the length of the data attribute), this method counts the actual number of non-zero entries in data. `scipy.optimize` improvements - ----------------------------- The implementation of Nelder-Mead minimization, `scipy.minimize(..., method="Nelder-Mead")`, obtained a new keyword, `initial_simplex`, which can be used to specify the initial simplex for the optimization process. Initial step size selection in CG and BFGS minimizers has been improved. We expect that this change will improve numeric stability of optimization in some cases. See pull request gh-5536 for details. Handling of infinite bounds in SLSQP optimization has been improved. We expect that this change will improve numeric stability of optimization in the some cases. See pull request gh-6024 for details. A large suite of global optimization benchmarks has been added to ``scipy/benchmarks/go_benchmark_functions``. See pull request gh-4191 for details. Nelder-Mead and Powell minimization will now only set defaults for maximum iterations or function evaluations if neither limit is set by the caller. In some cases with a slow converging function and only 1 limit set, the minimization may continue for longer than with previous versions and so is more likely to reach convergence. See issue gh-5966. `scipy.stats` improvements - -------------------------- Trapezoidal distribution has been implemented as `scipy.stats.trapz`. Skew normal distribution has been implemented as `scipy.stats.skewnorm`. Burr type XII distribution has been implemented as `scipy.stats.burr12`. Three- and four-parameter kappa distributions have been implemented as `scipy.stats.kappa3` and `scipy.stats.kappa4`, respectively. New `scipy.stats.iqr` function computes the interquartile region of a distribution. Random matrices ~~~~~~~~~~~~~~~ `scipy.stats.special_ortho_group` and `scipy.stats.ortho_group` provide generators of random matrices in the SO(N) and O(N) groups, respectively. They generate matrices in the Haar distribution, the only uniform distribution on these group manifolds. `scipy.stats.random_correlation` provides a generator for random correlation matrices, given specified eigenvalues. `scipy.linalg` improvements - --------------------------- `scipy.linalg.svd` gained a new keyword argument, ``lapack_driver``. Available drivers are ``gesdd`` (default) and ``gesvd``. `scipy.linalg.lapack.ilaver` returns the version of the LAPACK library SciPy links to. `scipy.spatial` improvements - ---------------------------- Boolean distances, `scipy.spatial.pdist`, have been sped up. Improvements vary by the function and the input size. In many cases, one can expect a speed-up of x2--x10. New class `scipy.spatial.SphericalVoronoi` constructs Voronoi diagrams on the surface of a sphere. See pull request gh-5232 for details. `scipy.cluster` improvements - ---------------------------- A new clustering algorithm, the nearest neighbor chain algorithm, has been implemented for `scipy.cluster.hierarchy.linkage`. As a result, one can expect a significant algorithmic improvement (:math:`O(N^2)` instead of :math:`O(N^3)`) for several linkage methods. `scipy.special` improvements - ---------------------------- The new function `scipy.special.loggamma` computes the principal branch of the logarithm of the Gamma function. For real input, ``loggamma`` is compatible with `scipy.special.gammaln`. For complex input, it has more consistent behavior in the complex plane and should be preferred over ``gammaln``. Vectorized forms of spherical Bessel functions have been implemented as `scipy.special.spherical_jn`, `scipy.special.spherical_kn`, `scipy.special.spherical_in` and `scipy.special.spherical_yn`. They are recommended for use over ``sph_*`` functions, which are now deprecated. Several special functions have been extended to the complex domain and/or have seen domain/stability improvements. This includes `spence`, `digamma`, `log1p` and several others. Deprecated features =================== The cross-class properties of `lti` systems have been deprecated. The following properties/setters will raise a `DeprecationWarning`: Name - (accessing/setting raises warning) - (setting raises warning) * StateSpace - (`num`, `den`, `gain`) - (`zeros`, `poles`) * TransferFunction (`A`, `B`, `C`, `D`, `gain`) - (`zeros`, `poles`) * ZerosPolesGain (`A`, `B`, `C`, `D`, `num`, `den`) - () Spherical Bessel functions, ``sph_in``, ``sph_jn``, ``sph_kn``, ``sph_yn``, ``sph_jnyn`` and ``sph_inkn`` have been deprecated in favor of `scipy.special.spherical_jn` and ``spherical_kn``, ``spherical_yn``, ``spherical_in``. The following functions in `scipy.constants` are deprecated: ``C2K``, ``K2C``, ``C2F``, ``F2C``, ``F2K`` and ``K2F``. They are superceded by a new function `scipy.constants.convert_temperature` that can perform all those conversions plus to/from the Rankine temperature scale. Backwards incompatible changes ============================== `scipy.optimize` - ---------------- The convergence criterion for ``optimize.bisect``, ``optimize.brentq``, ``optimize.brenth``, and ``optimize.ridder`` now works the same as ``numpy.allclose``. `scipy.ndimage` - --------------- The offset in ``ndimage.iterpolation.affine_transform`` is now consistently added after the matrix is applied, independent of if the matrix is specified using a one-dimensional or a two-dimensional array. `scipy.stats` - ------------- ``stats.ks_2samp`` used to return nonsensical values if the input was not real or contained nans. It now raises an exception for such inputs. Several deprecated methods of `scipy.stats` distributions have been removed: ``est_loc_scale``, ``vecfunc``, ``veccdf`` and ``vec_generic_moment``. Deprecated functions ``nanmean``, ``nanstd`` and ``nanmedian`` have been removed from `scipy.stats`. These functions were deprecated in scipy 0.15.0 in favor of their `numpy` equivalents. A bug in the ``rvs()`` method of the distributions in `scipy.stats` has been fixed. When arguments to ``rvs()`` were given that were shaped for broadcasting, in many cases the returned random samples were not random. A simple example of the problem is ``stats.norm.rvs(loc=np.zeros(10))``. Because of the bug, that call would return 10 identical values. The bug only affected code that relied on the broadcasting of the shape, location and scale parameters. The ``rvs()`` method also accepted some arguments that it should not have. There is a potential for backwards incompatibility in cases where ``rvs()`` accepted arguments that are not, in fact, compatible with broadcasting. An example is stats.gamma.rvs([2, 5, 10, 15], size=(2,2)) The shape of the first argument is not compatible with the requested size, but the function still returned an array with shape (2, 2). In scipy 0.18, that call generates a ``ValueError``. `scipy.io` - ---------- `scipy.io.netcdf` masking now gives precedence to the ``_FillValue`` attribute over the ``missing_value`` attribute, if both are given. Also, data are only treated as missing if they match one of these attributes exactly: values that differ by roundoff from ``_FillValue`` or ``missing_value`` are no longer treated as missing values. `scipy.interpolate` - ------------------- `scipy.interpolate.PiecewisePolynomial` class has been removed. It has been deprecated in scipy 0.14.0, and `scipy.interpolate.BPoly.from_derivatives` serves as a drop-in replacement. Other changes ============= Scipy now uses ``setuptools`` for its builds instead of plain distutils. This fixes usage of ``install_requires='scipy'`` in the ``setup.py`` files of projects that depend on Scipy (see Numpy issue gh-6551 for details). It potentially affects the way that build/install methods for Scipy itself behave though. Please report any unexpected behavior on the Scipy issue tracker. PR `#6240 `__ changes the interpretation of the `maxfun` option in `L-BFGS-B` based routines in the `scipy.optimize` module. An `L-BFGS-B` search consists of multiple iterations, with each iteration consisting of one or more function evaluations. Whereas the old search strategy terminated immediately upon reaching `maxfun` function evaluations, the new strategy allows the current iteration to finish despite reaching `maxfun`. The bundled copy of Qhull in the `scipy.spatial` subpackage has been upgraded to version 2015.2. The bundled copy of ARPACK in the `scipy.sparse.linalg` subpackage has been upgraded to arpack-ng 3.3.0. The bundled copy of SuperLU in the `scipy.sparse` subpackage has been upgraded to version 5.1.1. Authors ======= * @endolith * @yanxun827 + * @kleskjr + * @MYheavyGo + * @solarjoe + * Gregory Allen + * Gilles Aouizerate + * Tom Augspurger + * Henrik Bengtsson + * Felix Berkenkamp * Per Brodtkorb * Lars Buitinck * Daniel Bunting + * Evgeni Burovski * CJ Carey * Tim Cera * Grey Christoforo + * Robert Cimrman * Philip DeBoer + * Yves Delley + * D?vid Bodn?r + * Ion Elberdin + * Gabriele Farina + * Yu Feng * Andrew Fowlie + * Joseph Fox-Rabinovitz * Simon Gibbons + * Neil Girdhar + * Kolja Glogowski + * Christoph Gohlke * Ralf Gommers * Todd Goodall + * Johnnie Gray + * Alex Griffing * Olivier Grisel * Thomas Haslwanter + * Michael Hirsch + * Derek Homeier * Golnaz Irannejad + * Marek Jacob + * InSuk Joung + * Tetsuo Koyama + * Eugene Krokhalev + * Eric Larson * Denis Laxalde * Antony Lee * Jerry Li + * Henry Lin + * Nelson Liu + * Lo?c Est?ve * Lei Ma + * Osvaldo Martin + * Stefano Martina + * Nikolay Mayorov * Matthieu Melot + * Sturla Molden * Eric Moore * Alistair Muldal + * Maniteja Nandana * Tavi Nathanson + * Andrew Nelson * Joel Nothman * Behzad Nouri * Nikolai Nowaczyk + * Juan Nunez-Iglesias + * Ted Pudlik * Eric Quintero * Yoav Ram * Jonas Rauber + * Tyler Reddy + * Juha Remes * Garrett Reynolds + * Ariel Rokem + * Fabian Rost + * Bill Sacks + * Jona Sassenhagen + * Kari Schoonbee + * Marcello Seri + * Sourav Singh + * Martin Spacek + * S?ren Fuglede J?rgensen + * Bhavika Tekwani + * Martin Thoma + * Sam Tygier + * Meet Udeshi + * Utkarsh Upadhyay * Bram Vandekerckhove + * Sebasti?n Vanrell + * Ze Vinicius + * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Jakub Wilk + * Josh Wilson * Phillip J. Wolfram + * Nathan Woods * Haochen Wu * G Young + A total of 99 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.18.0 - ------------------------ - - `#1484 `__: SVD using *GESVD lapack drivers (Trac #957) - - `#1547 `__: Inconsistent use of offset in ndimage.interpolation.affine_transform()... - - `#1609 `__: special.hyp0f1 returns nan (Trac #1082) - - `#1656 `__: fmin_slsqp enhancement (Trac #1129) - - `#2069 `__: stats broadcasting in rvs (Trac #1544) - - `#2165 `__: sph_jn returns false results for some orders/values (Trac #1640) - - `#2255 `__: Incorrect order of translation and rotation in affine_transform... - - `#2332 `__: hyp0f1 args and return values are unnumpyic (Trac #1813) - - `#2534 `__: The sparse .sum() method with uint8 dtype does not act like the... - - `#3113 `__: Implement ufuncs for CSPHJY, SPHJ, SPHY, CSPHIK, SPHI, SPHIK... - - `#3568 `__: SciPy 0.13.3 - CentOS5 - Errors in test_arpack - - `#3581 `__: optimize: stepsize in fmin_bfgs is "bad" - - `#4476 `__: scipy.sparse non-native endian bug - - `#4484 `__: ftol in optimize.fmin fails to work - - `#4510 `__: sparsetools.cxx call_thunk can segfault due to out of bounds... - - `#5051 `__: ftol and xtol for _minimize_neldermead are absolute instead of... - - `#5097 `__: proposal: spherical Voronoi diagrams - - `#5123 `__: Call to `scipy.sparse.coo_matrix` fails when passed Cython typed... - - `#5220 `__: scipy.cluster.hierarchy.{ward,median,centroid} does not work... - - `#5379 `__: Add a build step at the end of .travis.yml that uploads working... - - `#5440 `__: scipy.optimize.basinhopping: accept_test returning numpy.bool_... - - `#5452 `__: Error in scipy.integrate.nquad when using variable integration... - - `#5520 `__: Cannot inherit csr_matrix properly - - `#5533 `__: Kendall tau implementation uses Python mergesort - - `#5553 `__: stats.tiecorrect overflows - - `#5589 `__: Add the Type XII Burr distribution to stats. - - `#5612 `__: sparse.linalg factorizations slow for small k due to default... - - `#5626 `__: io.netcdf masking should use masked_equal rather than masked_value - - `#5637 `__: Simple cubic spline interpolation? - - `#5683 `__: BUG: Akima1DInterpolator may return nans given multidimensional... - - `#5686 `__: scipy.stats.ttest_ind_from_stats does not accept arrays - - `#5702 `__: scipy.ndimage.interpolation.affine_transform lacks documentation... - - `#5718 `__: Wrong computation of weighted minkowski distance in cdist - - `#5745 `__: move to setuptools for next release - - `#5752 `__: DOC: solve_discrete_lyapunov equation puts transpose in wrong... - - `#5760 `__: signal.ss2tf doesn't handle zero-order state-space models - - `#5764 `__: Hypergeometric function hyp0f1 behaves incorrectly for complex... - - `#5814 `__: stats NaN Policy Error message inconsistent with code - - `#5833 `__: docstring of stats.binom_test() needs an update - - `#5853 `__: Error in scipy.linalg.expm for complex matrix with shape (1,1) - - `#5856 `__: Specify Nelder-Mead initial simplex - - `#5865 `__: scipy.linalg.expm fails for certain numpy matrices - - `#5915 `__: optimize.basinhopping - variable referenced before assignment. - - `#5916 `__: LSQUnivariateSpline fitting failed with knots generated from... - - `#5927 `__: unicode vs. string comparison in scipy.stats.binned_statistic_dd - - `#5936 `__: faster implementation of ks_2samp - - `#5948 `__: csc matrix .mean returns single element matrix rather than scalar - - `#5959 `__: BUG: optimize test error for root when using lgmres - - `#5972 `__: Test failures for sparse sum tests on 32-bit Python - - `#5976 `__: Unexpected exception in scipy.sparse.bmat while using 0 x 0 matrix - - `#6008 `__: scipy.special.kl_div not available in 0.14.1 - - `#6011 `__: The von-Mises entropy is broken - - `#6016 `__: python crashes for linalg.interpolative.svd with certain large... - - `#6017 `__: Wilcoxon signed-rank test with zero_method="pratt" or "zsplit"... - - `#6028 `__: stats.distributions does not have trapezoidal distribution - - `#6035 `__: Wrong link in f_oneway - - `#6056 `__: BUG: signal.decimate should only accept discrete LTI objects - - `#6093 `__: Precision error on Linux 32 bit with openblas - - `#6101 `__: Barycentric transforms test error on Python3, 32-bit Linux - - `#6105 `__: scipy.misc.face docstring is incorrect - - `#6113 `__: scipy.linalg.logm fails for a trivial matrix - - `#6128 `__: Error in dot method of sparse COO array, when used with numpy... - - `#6132 `__: Failures with latest MKL - - `#6136 `__: Failures on `master` with MKL - - `#6162 `__: fmin_l_bfgs_b returns inconsistent results (fmin ? f(xmin)) and... - - `#6165 `__: optimize.minimize infinite loop with Newton-CG - - `#6167 `__: incorrect distribution fitting for data containing boundary values. - - `#6194 `__: lstsq() and others detect numpy.complex256 as real - - `#6216 `__: ENH: improve accuracy of ppf cdf roundtrip for bradford - - `#6217 `__: BUG: weibull_min.logpdf return nan for c=1 and x=0 - - `#6218 `__: Is there a method to cap shortest path search distances? - - `#6222 `__: PchipInterpolator no longer handles a 2-element array - - `#6226 `__: ENH: improve accuracy for logistic.ppf and logistic.isf - - `#6227 `__: ENH: improve accuracy for rayleigh.logpdf and rayleigh.logsf... - - `#6228 `__: ENH: improve accuracy of ppf cdf roundtrip for gumbel_l - - `#6235 `__: BUG: alpha.pdf and alpha.logpdf returns nan for x=0 - - `#6245 `__: ENH: improve accuracy for ppf-cdf and sf-isf roundtrips for invgamma - - `#6263 `__: BUG: stats: Inconsistency in the multivariate_normal docstring - - `#6292 `__: Python 3 unorderable type errors in test_sparsetools.TestInt32Overflow - - `#6316 `__: TestCloughTocher2DInterpolator.test_dense crashes python3.5.2rc1_64bit... - - `#6318 `__: Scipy interp1d 'nearest' not working for high values on x-axis Pull requests for 0.18.0 - ------------------------ - - `#3226 `__: DOC: Change `nb` and `na` to conventional m and n - - `#3867 `__: allow cKDTree.query taking a list input in k. - - `#4191 `__: ENH: Benchmarking global optimizers - - `#4356 `__: ENH: add PPoly.solve(y) for solving ``p(x) == y`` - - `#4370 `__: DOC separate boolean distance functions for clarity - - `#4678 `__: BUG: sparse: ensure index dtype is large enough to pass all parameters... - - `#4881 `__: scipy.signal: Add the class dlti for linear discrete-time systems.... - - `#4901 `__: MAINT: add benchmark and improve docstring for signal.lfilter - - `#5043 `__: ENH: sparse: add count_nonzero method - - `#5136 `__: Attribute kurtosistest() to Anscombe & Glynn (1983) - - `#5186 `__: ENH: Port upfirdn - - `#5232 `__: ENH: adding spherical Voronoi diagram algorithm to scipy.spatial - - `#5279 `__: ENH: Bessel filters with different normalizations, high order - - `#5384 `__: BUG: Closes #5027 distance function always casts bool to double - - `#5392 `__: ENH: Add zero_phase kwarg to signal.decimate - - `#5394 `__: MAINT: sparse: non-canonical test cleanup and fixes - - `#5424 `__: DOC: add Scipy developers guide - - `#5442 `__: STY: PEP8 amendments - - `#5472 `__: Online QR in LGMRES - - `#5526 `__: BUG: stats: Fix broadcasting in the rvs() method of the distributions. - - `#5530 `__: MAINT: sparse: set `format` attr explicitly - - `#5536 `__: optimize: fix up cg/bfgs initial step sizes - - `#5548 `__: PERF: improves performance in stats.kendalltau - - `#5549 `__: ENH: Nearest-neighbor chain algorithm for hierarchical clustering - - `#5554 `__: MAINT/BUG: closes overflow bug in stats.tiecorrect - - `#5557 `__: BUG: modify optimize.bisect to achieve desired tolerance - - `#5581 `__: DOC: Tutorial for least_squares - - `#5606 `__: ENH: differential_evolution - moving core loop of solve method... - - `#5609 `__: [MRG] test against numpy dev - - `#5611 `__: use setuptools for bdist_egg distributions - - `#5615 `__: MAINT: linalg: tighten _decomp_update + special: remove unused... - - `#5622 `__: Add SO(N) rotation matrix generator - - `#5623 `__: ENH: special: Add vectorized spherical Bessel functions. - - `#5627 `__: Response to issue #5160, implements the skew normal distribution... - - `#5628 `__: DOC: Align the description and operation - - `#5632 `__: DOC: special: Expanded docs for Airy, elliptic, Bessel functions. - - `#5633 `__: MAINT: linalg: unchecked malloc in _decomp_update - - `#5634 `__: MAINT: optimize: tighten _group_columns - - `#5640 `__: Fixes for io.netcdf masking - - `#5645 `__: MAINT: size 0 vector handling in cKDTree range queries - - `#5649 `__: MAINT: update license text - - `#5650 `__: DOC: Clarify Exponent Order in ltisys.py - - `#5651 `__: DOC: Clarify Documentation for scipy.special.gammaln - - `#5652 `__: DOC: Fixed scipy.special.betaln Doc - - `#5653 `__: [MRG] ENH: CubicSpline interpolator - - `#5654 `__: ENH: Burr12 distribution to stats module - - `#5659 `__: DOC: Define BEFORE/AFTER in runtests.py -h for bench-compare - - `#5660 `__: MAINT: remove functions deprecated before 0.16.0 - - `#5662 `__: ENH: Circular statistic optimization - - `#5663 `__: MAINT: remove uses of np.testing.rand - - `#5665 `__: MAINT: spatial: remove matching distance implementation - - `#5667 `__: Change some HTTP links to HTTPS - - `#5669 `__: DOC: zpk2sos can't do analog, array_like, etc. - - `#5670 `__: Update conf.py - - `#5672 `__: MAINT: move a sample distribution to a subclass of rv_discrete - - `#5678 `__: MAINT: stats: remove est_loc_scale method - - `#5679 `__: MAINT: DRY up generic computations for discrete distributions - - `#5680 `__: MAINT: stop shadowing builtins in stats.distributions - - `#5681 `__: forward port ENH: Re-enable broadcasting of fill_value - - `#5684 `__: BUG: Fix Akima1DInterpolator returning nans - - `#5690 `__: BUG: fix stats.ttest_ind_from_stats to handle arrays. - - `#5691 `__: BUG: fix generator in io._loadarff to comply with PEP 0479 - - `#5693 `__: ENH: use math.factorial for exact factorials - - `#5695 `__: DOC: dx might be a float, not only an integer - - `#5699 `__: MAINT: io: micro-optimize Matlab reading code for size - - `#5701 `__: Implement OptimizeResult.__dir__ - - `#5703 `__: ENH: stats: make R? printing optional in probplot - - `#5704 `__: MAINT: typo ouf->out - - `#5705 `__: BUG: fix typo in query_pairs - - `#5707 `__: DOC:Add some explanation for ftol xtol in scipy.optimize.fmin - - `#5708 `__: DOC: optimize: PEP8 minimize docstring - - `#5709 `__: MAINT: optimize Cython code for speed and size - - `#5713 `__: [DOC] Fix broken link to reference - - `#5717 `__: DOC: curve_fit raises RuntimeError on failure. - - `#5724 `__: forward port gh-5720 - - `#5728 `__: STY: remove a blank line - - `#5729 `__: ENH: spatial: speed up boolean distances - - `#5732 `__: MAINT: differential_evolution changes to default keywords break... - - `#5733 `__: TST: differential_evolution - population initiation tests - - `#5736 `__: Complex number support in log1p, expm1, and xlog1py - - `#5741 `__: MAINT: sparse: clean up extraction functions - - `#5742 `__: DOC: signal: Explain fftbins in get_window - - `#5748 `__: ENH: Add O(N) random matrix generator - - `#5749 `__: ENH: Add polyphase resampling - - `#5756 `__: RFC: Bump the minimum numpy version, drop older python versions - - `#5761 `__: DOC: Some improvements to least squares docstrings - - `#5762 `__: MAINT: spatial: distance refactoring - - `#5768 `__: DOC: Fix io.loadmat docstring for mdict param - - `#5770 `__: BUG: Accept anything np.dtype can handle for a dtype in sparse.random - - `#5772 `__: Update sparse.csgraph.laplacian docstring - - `#5777 `__: BUG: fix special.hyp0f1 to work correctly for complex inputs. - - `#5780 `__: DOC: Update PIL error install URL - - `#5781 `__: DOC: Fix documentation on solve_discrete_lyapunov - - `#5782 `__: DOC: cKDTree and KDTree now reference each other - - `#5783 `__: DOC: Clarify finish behaviour in scipy.optimize.brute - - `#5784 `__: MAINT: Change default tolerances of least_squares to 1e-8 - - `#5787 `__: BUG: Allow Processing of Zero Order State Space Models in signal.ss2tf - - `#5788 `__: DOC, BUG: Clarify and Enforce Input Types to 'Data' Objects - - `#5789 `__: ENH: sparse: speedup LIL matrix slicing (was #3338) - - `#5791 `__: DOC: README: remove coveralls.io - - `#5792 `__: MAINT: remove uses of deprecated np.random.random_integers - - `#5794 `__: fix affine_transform (fixes #1547 and #5702) - - `#5795 `__: DOC: Removed uniform method from kmeans2 doc - - `#5797 `__: DOC: Clarify the computation of weighted minkowski - - `#5798 `__: BUG: Ensure scipy's _asfarray returns ndarray - - `#5799 `__: TST: Mpmath testing patch - - `#5801 `__: allow reading of certain IDL 8.0 .sav files - - `#5803 `__: DOC: fix module name in error message - - `#5804 `__: DOC: special: Expanded docs for special functions. - - `#5805 `__: DOC: Fix order of returns in _spectral_helper - - `#5806 `__: ENH: sparse: vectorized coo_matrix.diagonal - - `#5808 `__: ENH: Added iqr function to compute IQR metric in scipy/stats/stats.py - - `#5810 `__: MAINT/BENCH: sparse: Benchmark cleanup and additions - - `#5811 `__: DOC: sparse.linalg: shape, not size - - `#5813 `__: Update sparse ARPACK functions min `ncv` value - - `#5815 `__: BUG: Error message contained wrong values - - `#5816 `__: remove dead code from stats tests - - `#5820 `__: "in"->"a" in order_filter docstring - - `#5821 `__: DOC: README: INSTALL.txt was renamed in 2014 - - `#5825 `__: DOC: typo in the docstring of least_squares - - `#5826 `__: MAINT: sparse: increase test coverage - - `#5827 `__: NdPPoly rebase - - `#5828 `__: Improve numerical stability of hyp0f1 for large orders - - `#5829 `__: ENH: sparse: Add copy parameter to all .toXXX() methods in sparse... - - `#5830 `__: DOC: rework INSTALL.rst.txt - - `#5831 `__: Adds plotting options to voronoi_plot_2d - - `#5834 `__: Update stats.binom_test() docstring - - `#5836 `__: ENH, TST: Allow SIMO tf's for tf2ss - - `#5837 `__: DOC: Image examples - - `#5838 `__: ENH: sparse: add eliminate_zeros() to coo_matrix - - `#5839 `__: BUG: Fixed name of NumpyVersion.__repr__ - - `#5845 `__: MAINT: Fixed typos in documentation - - `#5847 `__: Fix bugs in sparsetools - - `#5848 `__: BUG: sparse.linalg: add locks to ensure ARPACK threadsafety - - `#5849 `__: ENH: sparse.linalg: upgrade to superlu 5.1.1 - - `#5851 `__: ENH: expose lapack's ilaver to python to allow lapack verion... - - `#5852 `__: MAINT: runtests.py: ensure Ctrl-C interrupts the build - - `#5854 `__: DOC: Minor update to documentation - - `#5855 `__: Pr 5640 - - `#5859 `__: ENH: Add random correlation matrix generator - - `#5862 `__: BUG: Allow expm for complex matrix with shape (1, 1) - - `#5863 `__: FIX: Fix test - - `#5864 `__: DOC: add a little note about the Normal survival function (Q-function) - - `#5867 `__: Fix for #5865 - - `#5869 `__: extend normal distribution cdf to complex domain - - `#5872 `__: DOC: Note that morlet and cwt don't work together - - `#5875 `__: DOC: interp2d class description - - `#5876 `__: MAINT: spatial: remove a stray print statement - - `#5878 `__: MAINT: Fixed noisy UserWarnings in ndimage tests. Fixes #5877 - - `#5879 `__: MAINT: sparse.linalg/superlu: add explicit casts to resolve compiler... - - `#5880 `__: MAINT: signal: import gcd from math and not fractions when on... - - `#5887 `__: Neldermead initial simplex - - `#5894 `__: BUG: _CustomLinearOperator unpickalable in python3.5 - - `#5895 `__: DOC: special: slightly improve the multigammaln docstring - - `#5900 `__: Remove duplicate assignment. - - `#5901 `__: Update bundled ARPACK - - `#5904 `__: ENH: Make convolve and correlate order-agnostic - - `#5905 `__: ENH: sparse.linalg: further LGMRES cleanups - - `#5906 `__: Enhancements and cleanup in scipy.integrate (attempt #2) - - `#5907 `__: ENH: Change sparse `.sum` and `.mean` dtype casting to match... - - `#5909 `__: changes for convolution symmetry - - `#5913 `__: MAINT: basinhopping remove instance test closes #5440 - - `#5919 `__: MAINT: uninitialised var if basinhopping niter=0. closes #5915 - - `#5920 `__: BLD: Fix missing lsame.c error for MKL - - `#5921 `__: DOC: interpolate: add example showing how to work around issue... - - `#5926 `__: MAINT: spatial: upgrade to Qhull 2015.2 - - `#5928 `__: MAINT: sparse: optimize DIA sum/diagonal, csgraph.laplacian - - `#5929 `__: Update info/URL for octave-maintainers discussion - - `#5930 `__: TST: special: silence DeprecationWarnings from sph_yn - - `#5931 `__: ENH: implement the principle branch of the logarithm of Gamma. - - `#5934 `__: Typo: "mush" => "must" - - `#5935 `__: BUG:string comparison stats._binned_statistic closes #5927 - - `#5938 `__: Cythonize stats.ks_2samp for a ~33% gain in speed. - - `#5939 `__: DOC: fix optimize.fmin convergence docstring - - `#5941 `__: Fix minor typo in squareform docstring - - `#5942 `__: Update linregress stderr description. - - `#5943 `__: ENH: Improve numerical accuracy of lognorm - - `#5944 `__: Merge vonmises into stats pyx - - `#5945 `__: MAINT: interpolate: Tweak declaration to avoid cython warning... - - `#5946 `__: MAINT: sparse: clean up format conversion methods - - `#5949 `__: BUG: fix sparse .mean to return a scalar instead of a matrix - - `#5955 `__: MAINT: Replace calls to `hanning` with `hann` - - `#5956 `__: DOC: Missing periods interfering with parsing - - `#5958 `__: MAINT: add a test for lognorm.sf underflow - - `#5961 `__: MAINT _centered(): rename size to shape - - `#5962 `__: ENH: constants: Add multi-scale temperature conversion function - - `#5965 `__: ENH: special: faster way for calculating comb() for exact=True - - `#5975 `__: ENH: Improve FIR path of signal.decimate - - `#5977 `__: MAINT/BUG: sparse: remove overzealous bmat checks - - `#5978 `__: minimize_neldermead() stop at user requested maxiter or maxfev - - `#5983 `__: ENH: make sparse `sum` cast dtypes like NumPy `sum` for 32-bit... - - `#5985 `__: BUG, API: Add `jac` parameter to curve_fit - - `#5989 `__: ENH: Add firls least-squares fitting - - `#5990 `__: BUG: read tries to handle 20-bit WAV files but shouldn't - - `#5991 `__: DOC: Cleanup wav read/write docs and add tables for common types - - `#5994 `__: ENH: Add gesvd method for svd - - `#5996 `__: MAINT: Wave cleanup - - `#5997 `__: TST: Break up upfirdn tests & compare to lfilter - - `#6001 `__: Filter design docs - - `#6002 `__: COMPAT: Expand compatibility fromnumeric.py - - `#6007 `__: ENH: Skip conversion of TF to TF in freqresp - - `#6009 `__: DOC: fix incorrect versionadded for entr, rel_entr, kl_div - - `#6013 `__: Fixed the entropy calculation of the von Mises distribution. - - `#6014 `__: MAINT: make gamma, rgamma use loggamma for complex arguments - - `#6020 `__: WIP: ENH: add exact=True factorial for vectors - - `#6022 `__: Added 'lanczos' to the image interpolation function list. - - `#6024 `__: BUG: optimize: do not use dummy constraints in SLSQP when no... - - `#6025 `__: ENH: Boundary value problem solver for ODE systems - - `#6029 `__: MAINT: Future imports for optimize._lsq - - `#6030 `__: ENH: stats.trap - adding trapezoidal distribution closes #6028 - - `#6031 `__: MAINT: Some improvements to optimize._numdiff - - `#6032 `__: MAINT: Add special/_comb.c to .gitignore - - `#6033 `__: BUG: check the requested approximation rank in interpolative.svd - - `#6034 `__: DOC: Doc for mannwhitneyu in stats.py corrected - - `#6040 `__: FIX: Edit the wrong link in f_oneway - - `#6044 `__: BUG: (ordqz) always increase parameter lwork by 1. - - `#6047 `__: ENH: extend special.spence to complex arguments. - - `#6049 `__: DOC: Add documentation of PR #5640 to the 0.18.0 release notes - - `#6050 `__: MAINT: small cleanups related to loggamma - - `#6070 `__: Add asarray to explicitly cast list to numpy array in wilcoxon... - - `#6071 `__: DOC: antialiasing filter and link decimate resample, etc. - - `#6075 `__: MAINT: reimplement special.digamma for complex arguments - - `#6080 `__: avoid multiple computation in kstest - - `#6081 `__: Clarified pearson correlation return value - - `#6085 `__: ENH: allow long indices of sparse matrix with umfpack in spsolve() - - `#6086 `__: fix description for associated Laguerre polynomials - - `#6087 `__: Corrected docstring of splrep. - - `#6094 `__: ENH: special: change zeta signature to zeta(x, q=1) - - `#6095 `__: BUG: fix integer overflow in special.spence - - `#6106 `__: Fixed Issue #6105 - - `#6116 `__: BUG: matrix logarithm edge case - - `#6119 `__: TST: DeprecationWarnings in stats on python 3.5 closes #5885 - - `#6120 `__: MAINT: sparse: clean up sputils.isintlike - - `#6122 `__: DOC: optimize: linprog docs should say minimize instead of maximize - - `#6123 `__: DOC: optimize: document the `fun` field in `scipy.optimize.OptimizeResult` - - `#6124 `__: Move FFT zero-padding calculation from signaltools to fftpack - - `#6125 `__: MAINT: improve special.gammainc in the ``a ~ x`` regime. - - `#6130 `__: BUG: sparse: Fix COO dot with zero columns - - `#6138 `__: ENH: stats: Improve behavior of genextreme.sf and genextreme.isf - - `#6146 `__: MAINT: simplify the expit implementation - - `#6151 `__: MAINT: special: make generate_ufuncs.py output deterministic - - `#6152 `__: TST: special: better test for gammainc at large arguments - - `#6153 `__: ENH: Make next_fast_len public and faster - - `#6154 `__: fix typo "mush"-->"must" - - `#6155 `__: DOC: Fix some incorrect RST definition lists - - `#6160 `__: make logsumexp error out on a masked array - - `#6161 `__: added missing bracket to rosen documentation - - `#6163 `__: ENH: Added "kappa4" and "kappa3" distributions. - - `#6164 `__: DOC: Minor clean-up in integrate._bvp - - `#6169 `__: Fix mpf_assert_allclose to handle iterable results, such as maps - - `#6170 `__: Fix pchip_interpolate convenience function - - `#6172 `__: Corrected misplaced bracket in doc string - - `#6175 `__: ENH: sparse.csgraph: Pass indices to shortest_path - - `#6178 `__: TST: increase test coverage of sf and isf of a generalized extreme... - - `#6179 `__: TST: avoid a deprecation warning from numpy - - `#6181 `__: ENH: Boundary conditions for CubicSpline - - `#6182 `__: DOC: Add examples/graphs to max_len_seq - - `#6183 `__: BLD: update Bento build config files for recent changes. - - `#6184 `__: BUG: fix issue in io/wavfile for float96 input. - - `#6186 `__: ENH: Periodic extrapolation for PPoly and BPoly - - `#6192 `__: MRG: Add circle-CI - - `#6193 `__: ENH: sparse: avoid setitem densification - - `#6196 `__: Fixed missing sqrt in docstring of Mahalanobis distance in cdist,... - - `#6206 `__: MAINT: Minor changes in solve_bvp - - `#6207 `__: BUG: linalg: for BLAS, downcast complex256 to complex128, not... - - `#6209 `__: BUG: io.matlab: avoid buffer overflows in read_element_into - - `#6210 `__: BLD: use setuptools when building. - - `#6214 `__: BUG: sparse.linalg: fix bug in LGMRES breakdown handling - - `#6215 `__: MAINT: special: make loggamma use zdiv - - `#6220 `__: DOC: Add parameter - - `#6221 `__: ENH: Improve Newton solver for solve_bvp - - `#6223 `__: pchip should work for length-2 arrays - - `#6224 `__: signal.lti: deprecate cross-class properties/setters - - `#6229 `__: BUG: optimize: avoid an infinite loop in Newton-CG - - `#6230 `__: Add example for application of gaussian filter - - `#6236 `__: MAINT: gumbel_l accuracy - - `#6237 `__: MAINT: rayleigh accuracy - - `#6238 `__: MAINT: logistic accuracy - - `#6239 `__: MAINT: bradford distribution accuracy - - `#6240 `__: MAINT: avoid bad fmin in l-bfgs-b due to maxfun interruption - - `#6241 `__: MAINT: weibull_min accuracy - - `#6246 `__: ENH: Add _support_mask to distributions - - `#6247 `__: fixed a print error for an example of ode - - `#6249 `__: MAINT: change x-axis label for stats.probplot to "theoretical... - - `#6250 `__: DOC: fix typos - - `#6251 `__: MAINT: constants: filter out test noise from deprecated conversions - - `#6252 `__: MAINT: io/arff: remove unused variable - - `#6253 `__: Add examples to scipy.ndimage.filters - - `#6254 `__: MAINT: special: fix some build warnings - - `#6258 `__: MAINT: inverse gamma distribution accuracy - - `#6260 `__: MAINT: signal.decimate - Use discrete-time objects - - `#6262 `__: BUG: odr: fix string formatting - - `#6267 `__: TST: fix some test issues in interpolate and stats. - - `#6269 `__: TST: fix some warnings in the test suite - - `#6274 `__: ENH: Add sosfiltfilt - - `#6276 `__: DOC: update release notes for 0.18.0 - - `#6277 `__: MAINT: update the author name mapping - - `#6282 `__: DOC: Correcting references for scipy.stats.normaltest - - `#6283 `__: DOC: some more additions to 0.18.0 release notes. - - `#6284 `__: Add `.. versionadded::` directive to `loggamma`. - - `#6285 `__: BUG: stats: Inconsistency in the multivariate_normal docstring... - - `#6290 `__: Add author list, gh-lists to 0.18.0 release notes - - `#6293 `__: TST: special: relax a test's precision - - `#6295 `__: BUG: sparse: stop comparing None and int in bsr_matrix constructor - - `#6313 `__: MAINT: Fix for python 3.5 travis-ci build problem. - - `#6327 `__: TST: signal: use assert_allclose for testing near-equality in... - - `#6330 `__: BUG: spatial/qhull: allocate qhT via malloc to ensure CRT likes... - - `#6332 `__: TST: fix stats.iqr test to not emit warnings, and fix line lengths. - - `#6334 `__: MAINT: special: fix a test for hyp0f1 - - `#6347 `__: TST: spatial.qhull: skip a test on 32-bit platforms - - `#6350 `__: BUG: optimize/slsqp: don't overwrite an array out of bounds - - `#6351 `__: BUG: #6318 Interp1d 'nearest' integer x-axis overflow issue fixed - - `#6355 `__: Backports for 0.18.0 Checksums ========= MD5 ~~~ e9ddb97336421e110a3f20e1b1f6c057 scipy-0.18.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl f203cdea575c4e5d9c11f88480e609c4 scipy-0.18.0-cp27-cp27m-manylinux1_i686.whl ccca6aeb87717ca383a15e1d3cd3b3e9 scipy-0.18.0-cp27-cp27m-manylinux1_x86_64.whl 3426dd5731fcdb9f56ee30623def9ac0 scipy-0.18.0-cp27-cp27mu-manylinux1_i686.whl 96f1e6ac2ed3a845802050840929681a scipy-0.18.0-cp27-cp27mu-manylinux1_x86_64.whl aec151bd64fee0651dc804f8b57a1948 scipy-0.18.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 6e7bcd119bfcd3504f5d432424779fd2 scipy-0.18.0-cp34-cp34m-manylinux1_i686.whl d8b88298b048884225708494bc51f249 scipy-0.18.0-cp34-cp34m-manylinux1_x86_64.whl 3886c82aa705b44662964d5d1de179b8 scipy-0.18.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 1bdc3664de8456bbc45620afbfc3d2cf scipy-0.18.0-cp35-cp35m-manylinux1_i686.whl 8f28a30aba8dd7fa4c1704088347684e scipy-0.18.0-cp35-cp35m-manylinux1_x86_64.whl d70e7f533622ab705bc016dac328d93e scipy-0.18.0.tar.gz 59bceff108f58b0e72dfac6fb719476e scipy-0.18.0.tar.xz 9ec1363dde2f2c16e833d3cd09f0dd13 scipy-0.18.0.zip SHA256 ~~~~~~ c1357a8d7da900f3eb72c817a554fa82f71a123d232c1d28c4ef5624694ec937 scipy-0.18.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl f1ce34715420670253aef746ca9ac0de6af0db975bc2f7145697b227ed43a411 scipy-0.18.0-cp27-cp27m-manylinux1_i686.whl b10ccbb451d781f5a31bce940150c626260369c31dc7c6cc609c012aae5b8b77 scipy-0.18.0-cp27-cp27m-manylinux1_x86_64.whl 919c035bfd63a8ecd9076e46488108db0847a13ff7bb3e2eb52561af68ffb798 scipy-0.18.0-cp27-cp27mu-manylinux1_i686.whl 7b77d7721a2017fe6fe2369edf6631c1cb4d2665f7a0e0562ed3796e8d8007d4 scipy-0.18.0-cp27-cp27mu-manylinux1_x86_64.whl 61441deb6c9ff093c60b48362a2104fa22212bb8baa09b62643f71df3f1fd361 scipy-0.18.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 41252b1083e657be1beb391c8508fc70298cc3c47214d7b874e5b50e2676229e scipy-0.18.0-cp34-cp34m-manylinux1_i686.whl 1709f0696508047742a80c5f3c7699c7bd28f006cbbe4d2b001ed6ee9fc862a8 scipy-0.18.0-cp34-cp34m-manylinux1_x86_64.whl 5ed943f23f7553c34aec6444a3f929b9eccb0f330b7b4416a3132dd0c4c1068f scipy-0.18.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d71c4ac8019b5820acd881d1178461e4d900e8dcfe920f3b63faf9381db24a1c scipy-0.18.0-cp35-cp35m-manylinux1_i686.whl ab9db1de7140eaf602d4ba190bbbe07ca608d5d298203afbfeceb170a3f03ef4 scipy-0.18.0-cp35-cp35m-manylinux1_x86_64.whl f01784fb1c2bc246d4211f2482ecf4369db5abaecb9d5afb9d94f6c59663286a scipy-0.18.0.tar.gz fc2d01887276839b1c709e661ba318d8db69a5c398217c79b97d6460a93c2d41 scipy-0.18.0.tar.xz a4f9fb8cddbe681e2d5465a8dcf715ef454b16fe8c9eddf92bffb10bb50ac75e scipy-0.18.0.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJXljjOAAoJEIp0pQ0zQcu+748IAL8pSPGDmi0PQOASCM00DUgK 2tq/3gnI5RIqpMDshQmcuI0/+6iqn0dGYVMJVIWYYBCVSm5D13xtz/3jTC9XGB6E hsnVdSBI/RiZHoEpB7VguzyvOjZIFngkFk5XUpGbr8uaqYcfdQhzKfLnzq7ZHcuB hUoYcku7gdQY5LZDlFTKZYjEBJM9odHa4uYIwezic09Rvr5X1PVjwIhahSXZekpo zH37Zwi/bMNJxjUWtLonIdOB1xFi3rSFgEkycGNYWaiwtxy8dJGrkstiLf9T6HCC ZV0lIVJcgUDG8VuRg8D7dQAoNSpDkYuR3HA17OvPfaKXETb6BDPcFFtqSxwvOF8= =GM4X -----END PGP SIGNATURE----- From aronne.merrelli at gmail.com Tue Jul 26 09:19:02 2016 From: aronne.merrelli at gmail.com (Aronne Merrelli) Date: Tue, 26 Jul 2016 08:19:02 -0500 Subject: [Numpy-discussion] f2py NotImplementedError In-Reply-To: References: Message-ID: Hello Andy, I'm not an expert on f2py, but I've used it in a few cases. I can recreate a NotImplementedError by copying your commands. The problem is that you are using -l (lower case L), which I think is supposed to specify the library name, not a path. The NotImplementedError is not really what the code should produce in this case, but I don't know what is going on there. I think you normally want to use the typical flags, like "-lfoo -L/path/to/foo" or similar. In any case for this simple example, you don't really need to use those flags. I can get your code to compile with this command (I have to specify fcompiler here, otherwise it uses ifort on my system): $ gfortran -fPIC -c othermodule.f90 -o othermodule.o $ f2py --fcompiler=gfortran -m mainmodule -c mainmodule.f90 othermodule.o This compiles, but then the shared object has no functions in it when you import it in python. I think that's because your main file is a program, and I do not think f2py wraps fortran programs, rather only subroutines/functions. If you change the mainmodule file slightly, replacing the 'program mains' with: module mains use moderator contains subroutine foo ... end subroutine foo end module mains Compile it similarly, then it looks to work: $ python -c "import mainmodule ; mainmodule.mains.foo()" here's your 1 st number: 2.7182817459106445 here's your 2 nd number: 3.6945281028747559 here's your 3 rd number: 3.3475894927978516 here's your 4 th number: 2.2749228477478027 here's your 5 th number: 1.2367763519287109 here's your 6 th number: 0.56031775474548340 here's your 7 th number: 0.21758595108985901 here's your 8 th number: 7.3932491242885590E-002 here's your 9 th number: 2.2329926490783691E-002 here's your 10 th number: 6.0699032619595528E-003 On Sun, Jul 24, 2016 at 2:54 PM, andy buzza wrote: > Hello all, > > I have been trying to compile a relatively simple pair of Fortran files, one > referencing a subroutine from another file (mainmodule.f90 references > othermodule.f90). I have been able to compile them using a Fortran > compiler, but receive a NotImplementedError when using f2py. > > Steps I use for f2py: > > $gfortran -shared -o othermodule.so othermodule.f90 -fPIC > > $f2py -c -l/path/othermodule -m mainmodule mainmodule.f90 > > I am running this on linux and wasn't sure how to correct the error. > > othermodule.f90 > > module moderator > > implicit none > integer*1 :: i > integer*8 :: fact,len > real*8,dimension(:),allocatable :: ex > > contains > > subroutine submarine(ii,ff,exex) > > implicit none > integer*1 :: ii > integer*8 :: ff > real*8 :: exex > > exex=exp(real(ii))/ff > > end subroutine submarine > > end module moderator > > > mainmodule.f90 > > program mains > > use moderator > > implicit none > > len=10 > allocate(ex(len)) > fact=1 > do i=1,len > fact=fact*i > call submarine(i,fact,ex(i)) > if (i==1) then > print*,"here's your ",i,"st number: ",ex(i) > elseif (i==2) then > print*,"here's your ",i,"nd number: ",ex(i) > elseif (i==3) then > print*,"here's your ",i,"rd number: ",ex(i) > else > print*,"here's your ",i,"th number: ",ex(i) > endif > enddo > deallocate(ex) > > end program > > > Thanks for the help, > Andy > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From dmoisset at machinalis.com Wed Jul 27 15:07:05 2016 From: dmoisset at machinalis.com (Daniel Moisset) Date: Wed, 27 Jul 2016 20:07:05 +0100 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? Message-ID: Hi, I work at Machinalis were we use a lot of numpy (and the pydata stack in general). Recently we've also been getting involved with mypy, which is a tool to type check (not on runtime, think of it as a linter) annotated python code (the way of annotating python types has been recently standarized in PEP 484). As part of that involvement we've started creating type annotations for the Python libraries we use most, which include numpy. Mypy provides a way to specify types with annotations in separate files in case you don't have control over a library, so we have created an initial proof of concept at [1], and we are actively improving it. You can find some additional information about it and some problems we've found on the way at this blogpost [2]. What I wanted to ask is if the people involved on the numpy project are aware of PEP484 and if you have some interest in starting using them. The main benefit is that annotations serve as clear (and automatically testable) documentation for users, and secondary benefits is that users discovers bugs more quickly and that some IDEs (like pycharm) are starting to use this information for smart editor features (autocompletion, online checking, refactoring tools); eventually tools like jupyter could take advantage of these annotations in the future. And the cost of writing and including these are relatively low. We're doing the work anyway, but contributing our typespecs back could make it easier for users to benefit from this, and for us to maintain it and keep it in sync with future releases. If you've never heard about PEP484 or mypy (it happens a lot) I'll be happy to clarify anything about it that might helpunderstand this situation Thanks! D. [1] https://github.com/machinalis/mypy-data [2] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Thu Jul 28 15:35:52 2016 From: bryanv at continuum.io (Bryan Van de Ven) Date: Thu, 28 Jul 2016 14:35:52 -0500 Subject: [Numpy-discussion] ANN: Bokeh 0.12.1 released Message-ID: <89C5E2A9-F81A-4DCA-869E-9505E26B9700@continuum.io> Hi all, On behalf of the Bokeh team, I am pleased to announce the release of version 0.12.1 of Bokeh! This is a minor, incremental update that adds a few new small features and fixes several bugs. Please see the announcement post at: https://bokeh.github.io/blog/2016/6/28/release-0-12-1/ which has much more information as well as live demonstrations. And as always, see the CHANGELOG and Release Notes for full details. If you are using Anaconda/miniconda, you can install it with conda: conda install -c bokeh bokeh Alternatively, you can also install it with pip: pip install bokeh Full information including details about how to use and obtain BokehJS are at: http://bokeh.pydata.org/en/0.12.1/docs/installation.html Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/bokeh/bokeh Documentation is available at http://bokeh.pydata.org/en/0.12.1 Questions can be directed to the Bokeh mailing list: bokeh at continuum.io or the Gitter Chat room: https://gitter.im/bokeh/bokeh Thanks, Bryan Van de Ven Continuum Analytics ----- Bokeh is a Python interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of versatile graphics with high-performance interactivity over very large or streaming datasets. Bokeh can help anyone who would like to quickly and easily create interactive plots, dashboards, and data applications. From sebastian at sipsolutions.net Fri Jul 29 03:30:59 2016 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 29 Jul 2016 09:30:59 +0200 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? In-Reply-To: (sfid-20160727_210902_786748_22B418E5) References: (sfid-20160727_210902_786748_22B418E5) Message-ID: <1469777459.16772.13.camel@sipsolutions.net> On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote: > > Hi,? > > I work at Machinalis were we use a lot of numpy (and the pydata stack > in general). Recently we've also been getting involved with mypy, > which is a tool to type check (not on runtime, think of it as a > linter) annotated python code (the way of annotating python types has > been recently standarized in PEP 484). > > As part of that involvement we've started creating type annotations > for the Python libraries we use most, which include numpy. Mypy > provides a way to specify types with annotations in separate files in > case you don't have control over a library, so we have created an > initial proof of concept at [1], and we are actively improving it. > You can find some additional information about it and some problems > we've found on the way at this blogpost [2]. > > What I wanted to ask is if the people involved on the numpy project > are aware of PEP484 and if you have some interest in starting using > them. The main benefit is that annotations serve as clear (and > automatically testable) documentation for users, and secondary > benefits is that users discovers bugs more quickly and that some IDEs > (like pycharm) are starting to use this information for smart editor > features (autocompletion, online checking, refactoring tools); > eventually tools like jupyter could take advantage of these > annotations in the future. And the cost of writing and including > these are relatively low. > There is currently no plan to do it as far as I know, but with these things it is often more of a problem that someone volunteers to maintain it then to convince everyone that it is a good idea. If there is enough interest we could talk about hosting it on the numpy github group as a separate project to make it a bit more visible/obvious that such a project exists. For inclusion in numpy, it seems to me that currently this would probably be better of improved separately? In the long run, would it be possible to do something like run all numpy tests and then check whether the definitions cover (almost) everything, or test against the documentation or so? Otherwise it might get tricky to keep things quite up to date, at least until these type checks are very widely used. Also I wonder if all or most of numpy can be easily put into it. Anyway, it seems like a great project to have as much support for type annotations as possible. I have never used them, but with editors picking up on these things it sounds like it could be very useful in the future. - Sebastian > We're doing the work anyway, but contributing our typespecs back > could make it easier for users to benefit from this, and for us to > maintain it and keep it in sync with future releases. > > If you've never heard about PEP484 or mypy (it happens a lot) I'll be > happy to clarify anything about it that might helpunderstand this > situation > > Thanks! > > D. > > > [1]?https://github.com/machinalis/mypy-data? > [2]?http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ > > --? > Daniel F. Moisset -?UK Country Manager > www.machinalis.com > Skype: @dmoisset > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From dmoisset at machinalis.com Fri Jul 29 05:33:06 2016 From: dmoisset at machinalis.com (Daniel Moisset) Date: Fri, 29 Jul 2016 10:33:06 +0100 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? In-Reply-To: <1469777459.16772.13.camel@sipsolutions.net> References: <1469777459.16772.13.camel@sipsolutions.net> Message-ID: Hi Sebastian, thanks for your reply I'm glad to hear that you see value in having type annotations. Just to clarify, my question was aimed at surveying if there was interest in accepting the work we're already doing if we contribute it and if it has value for the numpy project. I'm aware there's effort involved; some colleagues and me are already involved doing that at https://github.com/machinalis/mypy-data because it's valuable for ourselves, so the volunteers are already here. You of course are invited to comment on the existing code and try it :) (or joining the effort, goes without saying) Running the checker on the test suite is probably the best way to validate the annotations (the normal way would be checking the annotations against the code, but that doesn't work with C extensions like numpy). That's something we haven't been doing yet but it's an obvious next step now that some simple examples are working. WRT "I wonder if all or most of numpy can be easily put into it.", we've covered ndarray (and matrix soon) which are the core types, things built upon that shouldn't be too hard. We found some snags along the way [1] [2], but no of it is a showstopper and I'm quite sure we'll fix those in time. But of course, if someone wants to try it out it will be a better validation than my optimism to see if this makes sense :) Thanks again and I'd be happy to hear more opinions from other numpy devs! Best, D. [1] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ [2] https://github.com/machinalis/mypy-data/issues On 29 Jul 2016 08:31, "Sebastian Berg" wrote: > On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote: > > > > Hi, > > > > I work at Machinalis were we use a lot of numpy (and the pydata stack > > in general). Recently we've also been getting involved with mypy, > > which is a tool to type check (not on runtime, think of it as a > > linter) annotated python code (the way of annotating python types has > > been recently standarized in PEP 484). > > > > As part of that involvement we've started creating type annotations > > for the Python libraries we use most, which include numpy. Mypy > > provides a way to specify types with annotations in separate files in > > case you don't have control over a library, so we have created an > > initial proof of concept at [1], and we are actively improving it. > > You can find some additional information about it and some problems > > we've found on the way at this blogpost [2]. > > > > What I wanted to ask is if the people involved on the numpy project > > are aware of PEP484 and if you have some interest in starting using > > them. The main benefit is that annotations serve as clear (and > > automatically testable) documentation for users, and secondary > > benefits is that users discovers bugs more quickly and that some IDEs > > (like pycharm) are starting to use this information for smart editor > > features (autocompletion, online checking, refactoring tools); > > eventually tools like jupyter could take advantage of these > > annotations in the future. And the cost of writing and including > > these are relatively low. > > > > There is currently no plan to do it as far as I know, but with these > things it is often more of a problem that someone volunteers to > maintain it then to convince everyone that it is a good idea. > If there is enough interest we could talk about hosting it on the numpy > github group as a separate project to make it a bit more > visible/obvious that such a project exists. > > For inclusion in numpy, it seems to me that currently this would > probably be better of improved separately? In the long run, would it be > possible to do something like run all numpy tests and then check > whether the definitions cover (almost) everything, or test against the > documentation or so? Otherwise it might get tricky to keep things quite > up to date, at least until these type checks are very widely used. Also > I wonder if all or most of numpy can be easily put into it. > > Anyway, it seems like a great project to have as much support for type > annotations as possible. I have never used them, but with editors > picking up on these things it sounds like it could be very useful in > the future. > > - Sebastian > > > > We're doing the work anyway, but contributing our typespecs back > > could make it easier for users to benefit from this, and for us to > > maintain it and keep it in sync with future releases. > > > > If you've never heard about PEP484 or mypy (it happens a lot) I'll be > > happy to clarify anything about it that might helpunderstand this > > situation > > > > Thanks! > > > > D. > > > > > > [1] https://github.com/machinalis/mypy-data > > [2] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ > > > > -- > > Daniel F. Moisset - UK Country Manager > > www.machinalis.com > > Skype: @dmoisset > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Fri Jul 29 09:31:16 2016 From: ben.v.root at gmail.com (Benjamin Root) Date: Fri, 29 Jul 2016 09:31:16 -0400 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? In-Reply-To: References: <1469777459.16772.13.camel@sipsolutions.net> Message-ID: One thing that I have always wished for from a project like mypy is the ability to annotate what the expected shape should be. Too often, I get a piece of code from a coworker and it comes with no docstring explaining the expected dimensions of the input arrays and what the output array is going to be. What would be really awesome is the ability to do something like annotate that "a" is MxN, and "b" is NxP, and that "c" is Px3. Even if the linter can't really check to make sure that the shapes would be respected, it would still be nice to have a common way of expressing the expected shapes in this annotation format. As for matplotlib, we would need to express much more complicated annotations, because our API is so flexible. It would be useful to keep an eye out to those needs as well. Cheers! Ben Root On Fri, Jul 29, 2016 at 5:33 AM, Daniel Moisset wrote: > Hi Sebastian, thanks for your reply > > I'm glad to hear that you see value in having type annotations. Just to > clarify, my question was aimed at surveying if there was interest in > accepting the work we're already doing if we contribute it and if it has > value for the numpy project. I'm aware there's effort involved; some > colleagues and me are already involved doing that at > https://github.com/machinalis/mypy-data because it's valuable for > ourselves, so the volunteers are already here. You of course are invited to > comment on the existing code and try it :) (or joining the effort, goes > without saying) > > Running the checker on the test suite is probably the best way to validate > the annotations (the normal way would be checking the annotations against > the code, but that doesn't work with C extensions like numpy). That's > something we haven't been doing yet but it's an obvious next step now that > some simple examples are working. > WRT "I wonder if all or most of numpy can be easily put into it.", we've > covered ndarray (and matrix soon) which are the core types, things built > upon that shouldn't be too hard. We found some snags along the way [1] [2], > but no of it is a showstopper and I'm quite sure we'll fix those in time. > But of course, if someone wants to try it out it will be a better > validation than my optimism to see if this makes sense :) > > Thanks again and I'd be happy to hear more opinions from other numpy devs! > > Best, > D. > > [1] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ > [2] https://github.com/machinalis/mypy-data/issues > > > On 29 Jul 2016 08:31, "Sebastian Berg" wrote: > >> On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote: >> > >> > Hi, >> > >> > I work at Machinalis were we use a lot of numpy (and the pydata stack >> > in general). Recently we've also been getting involved with mypy, >> > which is a tool to type check (not on runtime, think of it as a >> > linter) annotated python code (the way of annotating python types has >> > been recently standarized in PEP 484). >> > >> > As part of that involvement we've started creating type annotations >> > for the Python libraries we use most, which include numpy. Mypy >> > provides a way to specify types with annotations in separate files in >> > case you don't have control over a library, so we have created an >> > initial proof of concept at [1], and we are actively improving it. >> > You can find some additional information about it and some problems >> > we've found on the way at this blogpost [2]. >> > >> > What I wanted to ask is if the people involved on the numpy project >> > are aware of PEP484 and if you have some interest in starting using >> > them. The main benefit is that annotations serve as clear (and >> > automatically testable) documentation for users, and secondary >> > benefits is that users discovers bugs more quickly and that some IDEs >> > (like pycharm) are starting to use this information for smart editor >> > features (autocompletion, online checking, refactoring tools); >> > eventually tools like jupyter could take advantage of these >> > annotations in the future. And the cost of writing and including >> > these are relatively low. >> > >> >> There is currently no plan to do it as far as I know, but with these >> things it is often more of a problem that someone volunteers to >> maintain it then to convince everyone that it is a good idea. >> If there is enough interest we could talk about hosting it on the numpy >> github group as a separate project to make it a bit more >> visible/obvious that such a project exists. >> >> For inclusion in numpy, it seems to me that currently this would >> probably be better of improved separately? In the long run, would it be >> possible to do something like run all numpy tests and then check >> whether the definitions cover (almost) everything, or test against the >> documentation or so? Otherwise it might get tricky to keep things quite >> up to date, at least until these type checks are very widely used. Also >> I wonder if all or most of numpy can be easily put into it. >> >> Anyway, it seems like a great project to have as much support for type >> annotations as possible. I have never used them, but with editors >> picking up on these things it sounds like it could be very useful in >> the future. >> >> - Sebastian >> >> >> > We're doing the work anyway, but contributing our typespecs back >> > could make it easier for users to benefit from this, and for us to >> > maintain it and keep it in sync with future releases. >> > >> > If you've never heard about PEP484 or mypy (it happens a lot) I'll be >> > happy to clarify anything about it that might helpunderstand this >> > situation >> > >> > Thanks! >> > >> > D. >> > >> > >> > [1] https://github.com/machinalis/mypy-data >> > [2] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ >> > >> > -- >> > Daniel F. Moisset - UK Country Manager >> > www.machinalis.com >> > Skype: @dmoisset >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmoisset at machinalis.com Fri Jul 29 12:31:41 2016 From: dmoisset at machinalis.com (Daniel Moisset) Date: Fri, 29 Jul 2016 17:31:41 +0100 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? In-Reply-To: References: <1469777459.16772.13.camel@sipsolutions.net> Message-ID: I don't think a tool like mypy nor PEP 484 can talk about specific sizes (like the MxN and NxP for a matrix multiplication), but probably there are things that can be done at least about dimensionality (saying "a and b are 2d matrices, v is a 1-d vector"). But that's much farther about the road. For now you'll be able to detect simpler errors like treating an ndarray as a python list, method names mispells, or wrong counts/order of method arguments. Best, D. On Fri, Jul 29, 2016 at 2:31 PM, Benjamin Root wrote: > One thing that I have always wished for from a project like mypy is the > ability to annotate what the expected shape should be. Too often, I get a > piece of code from a coworker and it comes with no docstring explaining the > expected dimensions of the input arrays and what the output array is going > to be. What would be really awesome is the ability to do something like > annotate that "a" is MxN, and "b" is NxP, and that "c" is Px3. Even if the > linter can't really check to make sure that the shapes would be respected, > it would still be nice to have a common way of expressing the expected > shapes in this annotation format. > > As for matplotlib, we would need to express much more complicated > annotations, because our API is so flexible. It would be useful to keep an > eye out to those needs as well. > > Cheers! > Ben Root > > > On Fri, Jul 29, 2016 at 5:33 AM, Daniel Moisset > wrote: > >> Hi Sebastian, thanks for your reply >> >> I'm glad to hear that you see value in having type annotations. Just to >> clarify, my question was aimed at surveying if there was interest in >> accepting the work we're already doing if we contribute it and if it has >> value for the numpy project. I'm aware there's effort involved; some >> colleagues and me are already involved doing that at >> https://github.com/machinalis/mypy-data because it's valuable for >> ourselves, so the volunteers are already here. You of course are invited to >> comment on the existing code and try it :) (or joining the effort, goes >> without saying) >> >> Running the checker on the test suite is probably the best way to >> validate the annotations (the normal way would be checking the annotations >> against the code, but that doesn't work with C extensions like numpy). >> That's something we haven't been doing yet but it's an obvious next step >> now that some simple examples are working. >> WRT "I wonder if all or most of numpy can be easily put into it.", we've >> covered ndarray (and matrix soon) which are the core types, things built >> upon that shouldn't be too hard. We found some snags along the way [1] [2], >> but no of it is a showstopper and I'm quite sure we'll fix those in time. >> But of course, if someone wants to try it out it will be a better >> validation than my optimism to see if this makes sense :) >> >> Thanks again and I'd be happy to hear more opinions from other numpy devs! >> >> Best, >> D. >> >> [1] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ >> [2] https://github.com/machinalis/mypy-data/issues >> >> >> On 29 Jul 2016 08:31, "Sebastian Berg" >> wrote: >> >>> On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote: >>> > >>> > Hi, >>> > >>> > I work at Machinalis were we use a lot of numpy (and the pydata stack >>> > in general). Recently we've also been getting involved with mypy, >>> > which is a tool to type check (not on runtime, think of it as a >>> > linter) annotated python code (the way of annotating python types has >>> > been recently standarized in PEP 484). >>> > >>> > As part of that involvement we've started creating type annotations >>> > for the Python libraries we use most, which include numpy. Mypy >>> > provides a way to specify types with annotations in separate files in >>> > case you don't have control over a library, so we have created an >>> > initial proof of concept at [1], and we are actively improving it. >>> > You can find some additional information about it and some problems >>> > we've found on the way at this blogpost [2]. >>> > >>> > What I wanted to ask is if the people involved on the numpy project >>> > are aware of PEP484 and if you have some interest in starting using >>> > them. The main benefit is that annotations serve as clear (and >>> > automatically testable) documentation for users, and secondary >>> > benefits is that users discovers bugs more quickly and that some IDEs >>> > (like pycharm) are starting to use this information for smart editor >>> > features (autocompletion, online checking, refactoring tools); >>> > eventually tools like jupyter could take advantage of these >>> > annotations in the future. And the cost of writing and including >>> > these are relatively low. >>> > >>> >>> There is currently no plan to do it as far as I know, but with these >>> things it is often more of a problem that someone volunteers to >>> maintain it then to convince everyone that it is a good idea. >>> If there is enough interest we could talk about hosting it on the numpy >>> github group as a separate project to make it a bit more >>> visible/obvious that such a project exists. >>> >>> For inclusion in numpy, it seems to me that currently this would >>> probably be better of improved separately? In the long run, would it be >>> possible to do something like run all numpy tests and then check >>> whether the definitions cover (almost) everything, or test against the >>> documentation or so? Otherwise it might get tricky to keep things quite >>> up to date, at least until these type checks are very widely used. Also >>> I wonder if all or most of numpy can be easily put into it. >>> >>> Anyway, it seems like a great project to have as much support for type >>> annotations as possible. I have never used them, but with editors >>> picking up on these things it sounds like it could be very useful in >>> the future. >>> >>> - Sebastian >>> >>> >>> > We're doing the work anyway, but contributing our typespecs back >>> > could make it easier for users to benefit from this, and for us to >>> > maintain it and keep it in sync with future releases. >>> > >>> > If you've never heard about PEP484 or mypy (it happens a lot) I'll be >>> > happy to clarify anything about it that might helpunderstand this >>> > situation >>> > >>> > Thanks! >>> > >>> > D. >>> > >>> > >>> > [1] https://github.com/machinalis/mypy-data >>> > [2] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ >>> > >>> > -- >>> > Daniel F. Moisset - UK Country Manager >>> > www.machinalis.com >>> > Skype: @dmoisset >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Jul 29 13:30:35 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 29 Jul 2016 10:30:35 -0700 Subject: [Numpy-discussion] Is there any official position on PEP484/mypy? In-Reply-To: References: <1469777459.16772.13.camel@sipsolutions.net> Message-ID: I'm a big fan of type annotations and would support moving your repo over to the official typeshed repo or the NumPy GitHub organization to indicate it's official status. This is excellent work -- thank you for putting in the effort! Like Ben, I have also wished for type annotation support for dimension shapes/sizes. Someone recently suggested using datashape as potential syntax for this on the Blaze mailing list [1]. I have no idea how hard it would be to actually implement type inference for shape. Possibly an interesting research project? I know it's out of scope for mypy / PEP 484 for now. [1] https://groups.google.com/a/continuum.io/forum/#!topic/blaze-dev/0vNo4f-tNSk On Fri, Jul 29, 2016 at 9:31 AM, Daniel Moisset wrote: > I don't think a tool like mypy nor PEP 484 can talk about specific sizes > (like the MxN and NxP for a matrix multiplication), but probably there are > things that can be done at least about dimensionality (saying "a and b are > 2d matrices, v is a 1-d vector"). But that's much farther about the road. > For now you'll be able to detect simpler errors like treating an ndarray as > a python list, method names mispells, or wrong counts/order of method > arguments. > > Best, > D. > > On Fri, Jul 29, 2016 at 2:31 PM, Benjamin Root > wrote: > >> One thing that I have always wished for from a project like mypy is the >> ability to annotate what the expected shape should be. Too often, I get a >> piece of code from a coworker and it comes with no docstring explaining the >> expected dimensions of the input arrays and what the output array is going >> to be. What would be really awesome is the ability to do something like >> annotate that "a" is MxN, and "b" is NxP, and that "c" is Px3. Even if the >> linter can't really check to make sure that the shapes would be respected, >> it would still be nice to have a common way of expressing the expected >> shapes in this annotation format. >> >> As for matplotlib, we would need to express much more complicated >> annotations, because our API is so flexible. It would be useful to keep an >> eye out to those needs as well. >> >> Cheers! >> Ben Root >> >> >> On Fri, Jul 29, 2016 at 5:33 AM, Daniel Moisset >> wrote: >> >>> Hi Sebastian, thanks for your reply >>> >>> I'm glad to hear that you see value in having type annotations. Just to >>> clarify, my question was aimed at surveying if there was interest in >>> accepting the work we're already doing if we contribute it and if it has >>> value for the numpy project. I'm aware there's effort involved; some >>> colleagues and me are already involved doing that at >>> https://github.com/machinalis/mypy-data because it's valuable for >>> ourselves, so the volunteers are already here. You of course are invited to >>> comment on the existing code and try it :) (or joining the effort, goes >>> without saying) >>> >>> Running the checker on the test suite is probably the best way to >>> validate the annotations (the normal way would be checking the annotations >>> against the code, but that doesn't work with C extensions like numpy). >>> That's something we haven't been doing yet but it's an obvious next step >>> now that some simple examples are working. >>> WRT "I wonder if all or most of numpy can be easily put into it.", >>> we've covered ndarray (and matrix soon) which are the core types, things >>> built upon that shouldn't be too hard. We found some snags along the way >>> [1] [2], but no of it is a showstopper and I'm quite sure we'll fix those >>> in time. But of course, if someone wants to try it out it will be a better >>> validation than my optimism to see if this makes sense :) >>> >>> Thanks again and I'd be happy to hear more opinions from other numpy >>> devs! >>> >>> Best, >>> D. >>> >>> [1] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ >>> [2] https://github.com/machinalis/mypy-data/issues >>> >>> >>> On 29 Jul 2016 08:31, "Sebastian Berg" >>> wrote: >>> >>>> On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote: >>>> > >>>> > Hi, >>>> > >>>> > I work at Machinalis were we use a lot of numpy (and the pydata stack >>>> > in general). Recently we've also been getting involved with mypy, >>>> > which is a tool to type check (not on runtime, think of it as a >>>> > linter) annotated python code (the way of annotating python types has >>>> > been recently standarized in PEP 484). >>>> > >>>> > As part of that involvement we've started creating type annotations >>>> > for the Python libraries we use most, which include numpy. Mypy >>>> > provides a way to specify types with annotations in separate files in >>>> > case you don't have control over a library, so we have created an >>>> > initial proof of concept at [1], and we are actively improving it. >>>> > You can find some additional information about it and some problems >>>> > we've found on the way at this blogpost [2]. >>>> > >>>> > What I wanted to ask is if the people involved on the numpy project >>>> > are aware of PEP484 and if you have some interest in starting using >>>> > them. The main benefit is that annotations serve as clear (and >>>> > automatically testable) documentation for users, and secondary >>>> > benefits is that users discovers bugs more quickly and that some IDEs >>>> > (like pycharm) are starting to use this information for smart editor >>>> > features (autocompletion, online checking, refactoring tools); >>>> > eventually tools like jupyter could take advantage of these >>>> > annotations in the future. And the cost of writing and including >>>> > these are relatively low. >>>> > >>>> >>>> There is currently no plan to do it as far as I know, but with these >>>> things it is often more of a problem that someone volunteers to >>>> maintain it then to convince everyone that it is a good idea. >>>> If there is enough interest we could talk about hosting it on the numpy >>>> github group as a separate project to make it a bit more >>>> visible/obvious that such a project exists. >>>> >>>> For inclusion in numpy, it seems to me that currently this would >>>> probably be better of improved separately? In the long run, would it be >>>> possible to do something like run all numpy tests and then check >>>> whether the definitions cover (almost) everything, or test against the >>>> documentation or so? Otherwise it might get tricky to keep things quite >>>> up to date, at least until these type checks are very widely used. Also >>>> I wonder if all or most of numpy can be easily put into it. >>>> >>>> Anyway, it seems like a great project to have as much support for type >>>> annotations as possible. I have never used them, but with editors >>>> picking up on these things it sounds like it could be very useful in >>>> the future. >>>> >>>> - Sebastian >>>> >>>> >>>> > We're doing the work anyway, but contributing our typespecs back >>>> > could make it easier for users to benefit from this, and for us to >>>> > maintain it and keep it in sync with future releases. >>>> > >>>> > If you've never heard about PEP484 or mypy (it happens a lot) I'll be >>>> > happy to clarify anything about it that might helpunderstand this >>>> > situation >>>> > >>>> > Thanks! >>>> > >>>> > D. >>>> > >>>> > >>>> > [1] https://github.com/machinalis/mypy-data >>>> > [2] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/ >>>> > >>>> > -- >>>> > Daniel F. Moisset - UK Country Manager >>>> > www.machinalis.com >>>> > Skype: @dmoisset >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion at scipy.org >>>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Daniel F. Moisset - UK Country Manager > www.machinalis.com > Skype: @dmoisset > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.brinkman at umich.edu Fri Jul 29 14:42:23 2016 From: erik.brinkman at umich.edu (Erik Brinkman) Date: Fri, 29 Jul 2016 18:42:23 +0000 Subject: [Numpy-discussion] ufunc reduceat behavior on empty slices Message-ID: Hi, The behavior of a ufuncs reduceat on empty slices seems a little strange, and I wonder if there's a reason behind it / if there's a route to potentially changing it. First, I'll go into a little background. I've been making a lot of use of ufuncs reduceat functionality on staggered arrays. In general, I'll have "n" arrays, each with size "s[n]" and I'll store them in one array "x", such that "s.sum() == x.size". reduceat is great because I use ufunc.reduceat(x, np.insert(s[:-1].cumsum(), 0, 0)) to get some summary information about each array. However, reduceat seems to behave strangely for empty slices. To make things concrete, let's assume: import numpy as np s = np.array([3, 0, 2]) x = np.arange(s.sum()) inds = np.insert(s[:-1].cumsum(), 0, 0) # [0, 3, 3] np.add.reduceat(x, inds) # [3, 3, 7] not [3, 0, 7] # This is distinct from np.fromiter(map(np.add.reduce, np.array_split(x, inds[1:])), x.dtype, s.size - 1) # [3, 0, 7] what I wanted The current documentation on reduceat first states: For i in range(len(indices)), reduceat computes ufunc.reduce(a[indices[i]:indices[i+1]]) That would suggest the outcome, that I expected. However, in the examples section it goes into a bunch of examples which contradict that statement and instead suggest that the actual algorithm is more akin to: ufunc.reduce(a[indices[i]:indices[i+1]]) if indices[i+1] > indices[i] else indices[i] Looking at the source , it seems like it's copying indices[i], and then while there are more elements to process it keeps reducing, resulting in this unexpected behavior. It seems like the proper thing to do would be start with ufunc.identity, and then reduce. This is slightly less performant than what's implemented, but more "correct." There could, of course, just be a switch to copy the identity only when the slice is empty. Is there a reason it's implemented like this? Is it just for performance, or is this strange behavior *useful* somewhere? It seems like "fixing" this would be bad because you'll be changing somewhat documented functionality in a backwards incompatible way. What would the best approach to "fixing" this be? add another function "reduceat_"? add a flag to reduceat to do the proper thing for empty slices? Finally, is there a good way to work around this? I think for now I'm just going to mask out the empty slices and use insert to add them back in, but if I'm missing an obvious solution, I'll look at that too. I need to mask them out because, np.add.reduceat(x, 5) would ideally return 0, but instead it throws an error since 5 is out of range... Thanks for indulging my curiosity, Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Jul 29 16:14:03 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 29 Jul 2016 13:14:03 -0700 Subject: [Numpy-discussion] ufunc reduceat behavior on empty slices In-Reply-To: References: Message-ID: Jaime brought up the same issue recently, along with some other issues for ufunc.reduceat: https://mail.scipy.org/pipermail/numpy-discussion/2016-March/075199.html I completely agree with both of you that the current behavior for empty slices is very strange and should be changed to remove the special case. Nathaniel Smith voiced the same opinion on the GitHub issue [1]. I think the path forward here (as Nathaniel writes) is pretty clear: 1. Start issuing a FutureWarning about a future change. 2. Fix this in a release or two. [1] https://github.com/numpy/numpy/issues/834 On Fri, Jul 29, 2016 at 11:42 AM, Erik Brinkman wrote: > Hi, > > The behavior of a ufuncs reduceat on empty slices seems a little strange, > and I wonder if there's a reason behind it / if there's a route to > potentially changing it. First, I'll go into a little background. > > I've been making a lot of use of ufuncs reduceat functionality on > staggered arrays. In general, I'll have "n" arrays, each with size "s[n]" > and I'll store them in one array "x", such that "s.sum() == x.size". > reduceat is great because I use > > ufunc.reduceat(x, np.insert(s[:-1].cumsum(), 0, 0)) > > to get some summary information about each array. However, reduceat seems > to behave strangely for empty slices. To make things concrete, let's assume: > > import numpy as np > s = np.array([3, 0, 2]) > x = np.arange(s.sum()) > inds = np.insert(s[:-1].cumsum(), 0, 0) > # [0, 3, 3] > np.add.reduceat(x, inds) > # [3, 3, 7] not [3, 0, 7] > # This is distinct from > np.fromiter(map(np.add.reduce, np.array_split(x, inds[1:])), x.dtype, > s.size - 1) > # [3, 0, 7] what I wanted > > The current documentation > > on reduceat first states: > > For i in range(len(indices)), reduceat computes > ufunc.reduce(a[indices[i]:indices[i+1]]) > > That would suggest the outcome, that I expected. However, in the examples > section it goes into a bunch of examples which contradict that statement > and instead suggest that the actual algorithm is more akin to: > > ufunc.reduce(a[indices[i]:indices[i+1]]) if indices[i+1] > indices[i] else > indices[i] > > Looking at the source > , > it seems like it's copying indices[i], and then while there are more > elements to process it keeps reducing, resulting in this unexpected > behavior. It seems like the proper thing to do would be start with > ufunc.identity, and then reduce. This is slightly less performant than > what's implemented, but more "correct." There could, of course, just be a > switch to copy the identity only when the slice is empty. > > Is there a reason it's implemented like this? Is it just for performance, > or is this strange behavior *useful* somewhere? It seems like "fixing" > this would be bad because you'll be changing somewhat documented > functionality in a backwards incompatible way. What would the best approach > to "fixing" this be? add another function "reduceat_"? add a flag to > reduceat to do the proper thing for empty slices? > > Finally, is there a good way to work around this? I think for now I'm just > going to mask out the empty slices and use insert to add them back in, but > if I'm missing an obvious solution, I'll look at that too. I need to mask > them out because, np.add.reduceat(x, 5) would ideally return 0, but > instead it throws an error since 5 is out of range... > > Thanks for indulging my curiosity, > Erik > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: