NumPy-Discussion August 2015

numpy-discussion@python.org

63 participants
48 discussions

Re: [Numpy-discussion] [pytables-dev] ANN: PyTables 3.2.1 released
by Antonio Valentino 04 Aug '15

04 Aug '15

Good job. Thanks Andrea -- Antonio Valentino > Il giorno 04/ago/2015, alle ore 02:38, Andrea Bedini <andrea(a)andreabedini.com> ha scritto: > > =========================== > Announcing PyTables 3.2.1 > =========================== > > We are happy to announce PyTables 3.2.1. > > > What's new > ========== > > This is a bug fix release. It contains a fix for a segv fault in > indexesextension.keysort(). > > In case you want to know more in detail what has changed in this > version, please refer to: http://www.pytables.org/release_notes.html > > For an online version of the manual, visit: > http://www.pytables.org/usersguide/index.html > > > What it is? > =========== > > PyTables is a library for managing hierarchical datasets and > designed to efficiently cope with extremely large amounts of data with > support for full 64-bit file addressing. PyTables runs on top of > the HDF5 library and NumPy package for achieving maximum throughput and > convenient use. PyTables includes OPSI, a new indexing technology, > allowing to perform data lookups in tables exceeding 10 gigarows > (10**10 rows) in less than a tenth of a second. > > > Resources > ========= > > About PyTables: http://www.pytables.org > > About the HDF5 library: http://hdfgroup.org/HDF5/ > > About NumPy: http://numpy.scipy.org/ > > > Acknowledgments > =============== > > Thanks to many users who provided feature improvements, patches, bug > reports, support and suggestions. See the ``THANKS`` file in the > distribution package for a (incomplete) list of contributors. Most > specially, a lot of kudos go to the HDF5 and NumPy makers. > Without them, PyTables simply would not exist. > > > Share your experience > ===================== > > Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. > > > ---- > > **Enjoy data!** > > -- The PyTables Developers > > > -- > You received this message because you are subscribed to the Google Groups "pytables-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to pytables-dev+unsubscribe(a)googlegroups.com. > Visit this group at http://groups.google.com/group/pytables-dev. > For more options, visit https://groups.google.com/d/optout.

1 0

Re: [Numpy-discussion] Change default order to Fortran order
by Kang Wang 03 Aug '15

03 Aug '15

Thank you all for replying and providing useful insights and suggestions. The reasons I really want to use column-major are: I am image-oriented user (not matrix-oriented, as explained in http://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-a…) I am so used to read/write "I(x, y, z)" in textbook and code, and it is very likely that if the environment (row-major environment) forces me to write I(z, y, x), I will write a bug if I am not 100% focused. When this happens, it is difficult to debug, because everything compile and build fine. You will see run time error. Depending on environment, you may get useful error message (i.e. index out of range), but sometimes you just get bad image results. It actually has not too much to do with the actual data layout in memory. In imaging processing, especially medical imaging where I am working in, if you have a 3D image, everyone will agree that in memory, the X index is the fasted changing index, and the Z dimension (we often call it the "slice" dimension) has the largest stride in memory. So, if data layout is like this in memory, and image-oriented users are so used to read/write "I(x,y,z)", the only storage order that makes sense is column-major I also write code in MATLAB and C/C++. In MATLAB, matrix is column-major array. In C/C++, we often use ITK, which is also column-major (http://www.itk.org/Doxygen/html/classitk_1_1Image.html). I really prefer always read/write column-major code to minimize coding bugs related to storage order. I also prefer index to be 0-based; however, there is nothing I can do about it for MATLAB (which is 1-based). I can see that my original thought about "modifying NumPy source and re-compile" is probably a bad idea. The suggestions about using "fortran_zeros = partial(np.zeros(order='F'))" is probably the best way so far, in my opinion, and I am going to give it a try. Again, thank you all for replying. Kang On 08/02/15, Nathaniel Smith <njs(a)pobox.com> wrote: > > On Aug 2, 2015 7:30 AM, "Sturla Molden" <sturla.molden(a)gmail.com> wrote: > > > > > > On 02/08/15 15:55, Kang Wang wrote: > > > > > > > Can anyone provide any insight/help? > > > > > > There is no "default order". There was before, but now all operators > > > control the order of their return arrays from the order of their input > > > array. > > This is... overoptimistic. I would not rely on this in code that I wrote. > > It's true that many numpy operations do preserve the input order. But there are also many operations that don't. And which are which often changes between releases. (Not on purpose usually, but it's an easy bug to introduce. And sometimes it is done intentionally, e.g. to make functions faster. It sucks to have to make a function slower for everyone because someone somewhere is depending on memory layout default details.) And there are operations where it's not even clear what preserving order means (indexing a C array with a Fortran array, add(C, fortran), ...), and even lots of operations that intrinsically break contiguity/ordering (transpose, diagonal, slicing, swapaxes, ...), so you will end up with mixed orderings one way or another in any non-trivial program. > > Instead, it's better to explicitly specify order= just at the places where you care. That way your code is *locally* correct (you can check it will work by just reading the one function). The alternative is to try and enforce a *global* property on your program ("everyone everywhere is very careful to only use contiguity-preserving operations", where "everyone" includes third party libraries like numpy and others). In software design, local invariants invariants are always better than global invariants -- the most well known example is local variables versus global variables, but the principle is much broader. > > -n > > -- Kang Wang, Ph.D. 1111 Highland Ave., Room 1113 Madison, WI 53705-2275 ----------------------------------------

7 8

Re: [Numpy-discussion] Change default order to Fortran order
by Kang Wang 03 Aug '15

03 Aug '15

Thank you all for replying! I did a quick test, using python 2.6.6, and the original numpy package on my Linux computer without any change.== x = np.zeros((2,3),dtype=np.int32,order='F') print "x.strides =" print x.strides y = x + 1 print "y.strides =" print y.strides == Output: -------- x.strides = (4, 8) y.strides = (12, 4) -------- So, basically, "x" is Fortran-style column-major (because I explicitly write order='F'), but "y" is C-style row-major. This is going to be very annoying. What I really want is: - I do not have to write order='F' explicitly when declaring "x" - both "x" and "y" are Fortran-style column-major Which file should I modify to achieve this goal? Right now, I am just trying to get some basic stuff working with all arrays default to Fortran-style, and I can worry about interfacing with other code/libraries later. Thanks, Kang On 08/02/15, Sebastian Berg <sebastian(a)sipsolutions.net> wrote: > Well, numpy has a tendency to prefer C order. There is nothing you can do about that really. But you just cannot be sure what you get in some cases. > Often you need something specific for interfaceing other code. But in that case quite often you also do not need to fear the copy. > > - Sebastian > > > On Sun Aug 2 16:27:08 2015 GMT+0200, Sturla Molden wrote: > > On 02/08/15 15:55, Kang Wang wrote: > > > > > Can anyone provide any insight/help? > > > > There is no "default order". There was before, but now all operators > > control the order of their return arrays from the order of their input > > array. The only thing that makes C order "default" is the keyword > > argument to np.empty, np.ones and np.zeros. Just monkey patch those > > functions and it should be fine. > > > > Sturla > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Kang Wang, Ph.D. 1111 Highland Ave., Room 1113 Madison, WI 53705-2275 TEL 608-263-0066 http://www.medphysics.wisc.edu/~kang/ ----------------------------------------

5 5

Change default order to Fortran order
by Kang Wang 03 Aug '15

03 Aug '15

Hi, I am an imaging researcher, and a new Python user. My first Python project is to somehow modify NumPy source code such that everything is Fortran column-major by default. I read about the information in the link below, but for us, the fact is that we absolutely want to use Fortran column major, and we want to make it default. Explicitly writing " order = 'F' " all over the place is not acceptable to us. http://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-a… I tried searching in this email list, as well as google search in general. However, I have not found anything useful. This must be a common request/need, I believe. Can anyone provide any insight/help? Thank you very much, Kang -- Kang Wang, Ph.D. 1111 Highland Ave., Room 1113 Madison, WI 53705-2275 ----------------------------------------

6 7

Re: [Numpy-discussion] Change default order to Fortran order
by Kang Wang 03 Aug '15

03 Aug '15

This is very good discussion. Thank you all for replying. I can see the fundamental difference is that I always think/talk/read/write a 3D image as I(x, y, z), not (plane, row, column) . I am coming from MRI (magnetic resonance imaging) research, and I can assure you that the entire MRI community is using (x, y, z), including books, journal papers, conference abstracts, presentations, everything. We even talk about what we called "logical x/y/z" and "physical x/y/z", and the rotation matrix that converts the two coordinate systems. The radiologists are also used to (x, y, z). For example, we always say "my image is 256 by 256 by 20 slices", and we never say "20 by 256 by 256". So, basically, at least in MRI, we always talk about an image as I(x, y, z), and we always assume that "x" is the fastest changing index. That's why I prefer column-major (because it is more natural). Of course, I can totally get my work done by using row-major, I just have to always remind myself "write last dimension index first" when coding. I actually have done this before, and I found it would be so much easier if just using column-major. Kang On 08/02/15, Juan Nunez-Iglesias <jni.soma(a)gmail.com> wrote: > Hi Kang, > > Feel free to come chat about your application on the scikit-image list [1]! I'll note that we've been through the array order discussion many times there and even have a doc page about it [2]. > > The short version is that you'll save yourself a lot of pain by starting to think of your images as (plane, row, column) instead of (x, y, z). The syntax actually becomes friendlier too. For example, to do something to each slice of data, you do: > > for plane in image: > plane += foo > > > instead of > > > for z in image.shape[2]: > image[:, :, z] += foo > > > for example. > > > Juan. > > > [1] scikit-image(a)googlegroups.com > [2] http://scikit-image.org/docs/dev/user_guide/numpy_images.html#coordinate-co… > > > PS: As to the renamed Fortran-ordered numpy, may I suggest "funpy". The F is for Fortran and the fun is for all the fun you'll have maintaining it. =P > > On Mon, 3 Aug 2015 at 6:28 am Daniel Sank <sank.daniel(a)gmail.com> wrote: > > > > Kang, > > > > Thank you for explaining your motivation. It's clear from your last note, as you said, that your desire for column-first indexing has nothing to do with in-memory data layout. That being the case, I strongly urge you to just use bare numpy and do not use the "fortran_zeros" function I recommended before. Changing the in-memory layout via the "order" keyword in numpy.zeros will not change the way indexing works at all. You gain absolutely nothing by changing the in-memory order unless you are writing some C or Fortran code which will interact with the data in memory. > > > > > > To see what I mean, consider the following examples: > > > > > > x = np.array([1, 2, 3], [4, 5, 6]]) > > x.shape > > >>> (2, 3) > > > > > > and > > > > > > x = np.array([1, 2, 3], [4, 5, 6]], order='F') > > x.shape > > >>> (2, 3) > > > > > > > > You see that changing the in-memory order has nothing whatsoever to do with the array's shape or how you access it. > > > > > > > > > You will see run time error. Depending on environment, you may get useful error message > > > (i.e. index out of range), but sometimes you just get bad image results. > > > > > > > > Could you give a very simple example of what you mean? I can't think of how this could ever happen and your fear here makes me think there's a fundamental misunderstanding about how array operations in numpy and other programming languages work. As an example, iteration in numpy goes through the first index: > > > > > > x = np.array([[1, 2, 3], [4, 5, 6]]) > > for foo in x: > > ... > > > > > > Inside the for loop, foo takes on the values [1, 2, 3] on the first iteration and [4, 5, 6] on the second. If you want to iterate through the columns just do this instead > > > > > > x = np.array([[1, 2, 3], [4, 5, 6]]) > > for foo in x.T: > > ... > > > > > > > > If your complaint is that you want np.array([[1, 2, 3], [4, 5, 6]]) to produce an array with shape (3, 2) then you should own up to the fact that the array constructor expects it the other way around and do this > > > > > > > > x = np.array([[1, 2, 3], [4, 5, 6]]).T > > > > > > > > instead. This is infinity times better than trying to write a shim function or patch numpy because with .T you're using (fast) built-in functionality which other people your code will understand. > > > > The real message here is that whether the first index runs over rows or columns is actually meaningless. The only places the row versus column issue has any meaning is when doing input/output (in which case you should use the transpose if you actually need it), or when doing iteration. One thing that would make sense if you're reading from a binary file format which uses column-major format would be to write your own reader function: > > > > > > > > def read_fortran_style_binary_file(file): > > return np.fromfile(file).T > > > > > > Note that if you do this then you already have a column major array in numpy and you don't have to worry about any other transposes (except, again, when doing more I/O or passing to something like a plotting function). > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 2, 2015 at 7:16 PM, Kang Wang <kwang24(a)wisc.edu> wrote: > > > > > > > > > Thank you all for replying and providing useful insights and suggestions. > > > > > > The reasons I really want to use column-major are: > > > > > > I am image-oriented user (not matrix-oriented, as explained in http://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-a…) > > > I am so used to read/write "I(x, y, z)" in textbook and code, and it is very likely that if the environment (row-major environment) forces me to write I(z, y, x), I will write a bug if I am not 100% focused. When this happens, it is difficult to debug, because everything compile and build fine. You will see run time error. Depending on environment, you may get useful error message (i.e. index out of range), but sometimes you just get bad image results. > > > It actually has not too much to do with the actual data layout in memory. In imaging processing, especially medical imaging where I am working in, if you have a 3D image, everyone will agree that in memory, the X index is the fasted changing index, and the Z dimension (we often call it the "slice" dimension) has the largest stride in memory. So, if data layout is like this in memory, and image-oriented users are so used to read/write "I(x,y,z)", the only storage order that makes sense is column-major > > > I also write code in MATLAB and C/C++. In MATLAB, matrix is column-major array. In C/C++, we often use ITK, which is also column-major (http://www.itk.org/Doxygen/html/classitk_1_1Image.html). I really prefer always read/write column-major code to minimize coding bugs related to storage order. > > > I also prefer index to be 0-based; however, there is nothing I can do about it for MATLAB (which is 1-based). > > > > > > I can see that my original thought about "modifying NumPy source and re-compile" is probably a bad idea. The suggestions about using "fortran_zeros = partial(np.zeros(order='F'))" is probably the best way so far, in my opinion, and I am going to give it a try. > > > > > > > > > Again, thank you all for replying. > > > > > > > > > Kang > > > > > > On 08/02/15, Nathaniel Smith <njs(a)pobox.com> wrote: > > > > > > > > On Aug 2, 2015 7:30 AM, "Sturla Molden" <sturla.molden(a)gmail.com> wrote: > > > > > > > > > > > > > > > > > > On 02/08/15 15:55, Kang Wang wrote: > > > > > > > > > > > > > > > > > > > Can anyone provide any insight/help? > > > > > > > > > > > > > > > > > > There is no "default order". There was before, but now all operators > > > > > > > > > control the order of their return arrays from the order of their input > > > > > > > > > array. > > > > > > > > This is... overoptimistic. I would not rely on this in code that I wrote. > > > > > > > > It's true that many numpy operations do preserve the input order. But there are also many operations that don't. And which are which often changes between releases. (Not on purpose usually, but it's an easy bug to introduce. And sometimes it is done intentionally, e.g. to make functions faster. It sucks to have to make a function slower for everyone because someone somewhere is depending on memory layout default details.) And there are operations where it's not even clear what preserving order means (indexing a C array with a Fortran array, add(C, fortran), ...), and even lots of operations that intrinsically break contiguity/ordering (transpose, diagonal, slicing, swapaxes, ...), so you will end up with mixed orderings one way or another in any non-trivial program. > > > > > > > > Instead, it's better to explicitly specify order= just at the places where you care. That way your code is *locally* correct (you can check it will work by just reading the one function). The alternative is to try and enforce a *global* property on your program ("everyone everywhere is very careful to only use contiguity-preserving operations", where "everyone" includes third party libraries like numpy and others). In software design, local invariants invariants are always better than global invariants -- the most well known example is local variables versus global variables, but the principle is much broader. > > > > > > > > -n > > > > > > > > > > > > > > > > > -- > > > Kang Wang, Ph.D. > > > 1111 Highland Ave., Room 1113 > > > Madison, WI 53705-2275 > > > ---------------------------------------- > > > > > > > > > _______________________________________________ > > > > > > NumPy-Discussion mailing list > > > > > > NumPy-Discussion(a)scipy.org > > > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > > > > > > > > > > > -- > > Daniel Sank > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion(a)scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > -- Kang Wang, Ph.D. 1111 Highland Ave., Room 1113 Madison, WI 53705-2275 ----------------------------------------

5 6

1.10.x is branched
by Charles R Harris 03 Aug '15

03 Aug '15

Hi All, Numpy 1.10.x is branched. There is still some cleanup to do before the alpha release, but that should be coming in a couple of days. Chuck

2 1

mailmap update
by Charles R Harris 02 Aug '15

02 Aug '15

Hi All, I'm trying to update the .mailmap file on github and could use some help. The current version seems common to both numpy and scipy, hence the crosspost. Here is what I've got so far. Alex Griffing <argriffi <at> ncsu.edu> alex <argriffi <at> ncsu.edu> Alex Griffing <argriffi <at> ncsu.edu> argriffing <argriffi <at> ncsu.edu> Alex Griffing <argriffi <at> ncsu.edu> argriffing <argriffing <at> users.noreply.github.com> Behzad Nouri <behzadnouri <at> gmail.com> behzad nouri <behzadnouri <at> gmail.com> Carl Kleffner <cmkleffner <at> gmail.com> carlkl <cmkleffner <at> gmail.com> Christoph Gohlke <cgohlke <at> uci.edu> Christolph Gohlke <cgohlke <at> uci.edu> Christoph Gohlke <cgohlke <at> uci.edu> cgholke <? <at> ?> Christoph Gohlke <cgohlke <at> uci.edu> cgohlke <cgohlke <at> uci.edu> Han Genuit <hangenuit <at> gmail.com> Han <hangenuit <at> gmail.com> Jaime Fernandez <jaime.frio <at> gmail.com> Jaime <jaime.frio <at> gmail.com > Jaime Fernandez <jaime.frio <at> gmail.com> jaimefrio <jaime.frio <at> gmail.com> Mark Wiebe <mwwiebe <at> gmail.com> Mark <mwwiebe <at> gmail.com> Mark Wiebe <mwwiebe <at> gmail.com> Mark Wiebe <mwiebe <at> enthought.com> Mark Wiebe <mwwiebe <at> gmail.com> Mark Wiebe <mwiebe <at> georg.(none)> Nathaniel J. Smith <njs <at> pobox.com> njsmith <njs <at> pobox.com> Ondřej Čertík <ondrej.certik <at> gmail.com> Ondrej Certik <ondrej.certik <at> gmail.com> Ralf Gommers <ralf.gommers <at> googlemail.com> rgommers <ralf.gommers <at> googlemail.com> Saullo Giovani <saullogiovani <at> gmail.com> saullogiovani <saullogiovani <at> gmail.com> Sebastian Berg <sebastian <at> sipsolutions.net> seberg <sebastian <at> sipsolutions.net> Anon <anon(a)gmail.com> abdulmuneer <abdulmuneer <at> gmail.com> Anon <anon(a)gmail.com> amir <ladsgroup <at> gmail.com> Anon <anon(a)gmail.com> cel <cel.gentoo <at> gmail.com> Anon <anon(a)gmail.com> chebee7i <chebee7i <at> gmail.com> Anon <anon(a)gmail.com> empeeu <empeeu <at> yahoo.com> Anon <anon(a)gmail.com> endolith <endolith <at> gmail.com> Anon <anon(a)gmail.com> hannaro <hroehling <at> gmx.net> Anon <anon(a)gmail.com> hpaulj <hpj3 <at> myuw.net> Anon <anon(a)gmail.com> immerrr <immerrr <at> gmail.com> Anon <anon(a)gmail.com> jmrosen155 <Jordan <at> Jordans-MacBook-Pro.local> Anon <anon(a)gmail.com> jnothman <jnothman <at> student.usyd.edu.au> Anon <anon(a)gmail.com> kanhua <kanhwa <at> gmail.com> Anon <anon(a)gmail.com> mamikony <ernest.mamikonyan <at> sig.com> Anon <anon(a)gmail.com> mbyt <random.seed <at> web.de> Anon <anon(a)gmail.com> mlai <mlai <at> begws92.beg.utexas.edu> Anon <anon(a)gmail.com> ryanblak <rbtnet <at> gmail.com> Anon <anon(a)gmail.com> styr <styr.py <at> gmail.com> Anon <anon(a)gmail.com> tdihp <tdihp <at> hotmail.com> Anon <anon(a)gmail.com> tpoole <t.b.poole <at> gmail.com> Anon <anon(a)gmail.com> wim glenn <wim.glenn <at> melbourneit.com.au> The Anon author is just a standing in for unknown author. I can make a guess at some of those, but would prefer it if the people in question could supply their proper name and address. TIA, Chuck

2 1

Branching 1.10 Sunday, Aug 2.
by Charles R Harris 02 Aug '15

02 Aug '15

Hi All, Just a heads up. If anything absolutely needed has been left out, please make a noise. Chuck

1 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion August 2015