Mailman 3 March 2012 - NumPy-Discussion

Viewing a float64 array with a float32 array
by Hugo Gagnon March 21, 2012

March 21, 2012

Hi, Is it possible to have a view of a float64 array that is itself float32? So that: >>> A = np.arange(5, dtype='d') >>> A.view(dtype='f') would return a size 5 float32 array looking at A's data? Thanks, -- Hugo Gagnon

2 3

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
by Francesc Alted March 21, 2012

March 21, 2012

On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: > Francesc Alted <francesc(a)continuum.io> wrote: > >> On Mar 20, 2012, at 12:49 PM, mark florisson wrote: >>>> Cython and Numba certainly overlap. However, Cython requires: >>>> >>>> 1) learning another language >>>> 2) creating an extension module --- loading bit-code files >> and dynamically executing (even on a different machine from the one >> that initially created them) can be a powerful alternative for run-time >> compilation and distribution of code. >>>> >>>> These aren't show-stoppers obviously. But, I think some users >> would prefer an even simpler approach to getting fast-code than Cython >> (which currently doesn't do enought type-inference and requires >> building a dlopen extension module). >>> >>> Dag and I have been discussing this at PyCon, and here is my take on >>> it (at this moment :). >>> >>> Definitely, if you can avoid Cython then that is easier and more >>> desirable in many ways. So perhaps we can create a third project >>> called X (I'm not very creative, maybe ArrayExprOpt), that takes an >>> abstract syntax tree in a rather simple form, performs code >>> optimizations such as rewriting loops with array accesses to vector >>> expressions, fusing vector expressions and loops, etc, and spits out >> a >>> transformed AST containing these optimizations. If runtime >> information >>> is given such as actual shape and stride information the >>> transformations could figure out there and then whether to do things >>> like collapsing, axes swapping, blocking (as in, introducing more >> axes >>> or loops to retain discontiguous blocks in the cache), blocked memory >>> copies to contiguous chunks, etc. The AST could then also say whether >>> the final expressions are vectorizable. Part of this functionality is >>> already in numpy's nditer, except that this would be implicit and do >>> more (and hopefully with minimal overhead). >>> >>> So numba, Cython and maybe numexpr could use the functionality, >> simply >>> by building the AST from Python and converting back (if necessary) to >>> its own AST. As such, the AST optimizer would be only part of any >>> (runtime) compiler's pipeline, and it should be very flexible to >>> retain any information (metadata regarding actual types, control flow >>> information, etc) provided by the original AST. It would not do >>> control flow analysis, type inference or promotion, etc, but only >> deal >>> with abstract types like integers, reals and arrays (C, Fortran or >>> partly contiguous or strided). It would not deal with objects, but >>> would allow to insert nodes like UnreorderableNode and SideEffectNode >>> wrapping parts of the original AST. In short, it should be as easy as >>> possible to convert from an original AST to this project's AST and >>> back again afterwards. >> >> I think this is a very interesting project, and certainly projects like >> numba can benefit of it. So, in order to us have an idea on what you >> are after, can we assume that your project (call it X) would be kind of >> an compiler optimizer, and then the produced, optimized code could be >> feed into numba for optimized LLVM code generation (that on its turn, >> can be run on top of CPUs or GPUs or a combination)? Is that correct? > > I think so. Another way of thinking about it is that it is a reimplementation of the logic in the (good and closed source) Fortran 90 compilers, in a reusable component for inclusion in various compilers. > > Various c++ metaprogramming libraries (like Blitz++) are similar too. Aha, thanks. >> Giving that my interpretation above is correct, it is bit more >> difficult to me to see how your X project could be of benefit for >> numexpr. In fact, I actually see this the other way round: once the >> optimizer has discovered the vectorization parts, then go one step >> further and generate code that uses numexpr automatically (call this, >> vectorization through numexpr). This is what you mean, or I'm missing >> something? > > No. I think in some ways this is a competitor to numexpr -- you would gut out the middle of numexpr and keep the frontend and backend, but use this to optimize iteration order and blocking strategies. I see. Yes, I can easily see Mark's project X + numba more as a competitor (killer?) to numexpr too. > > I think the goal is for higher performance than what I understand numexpr can provide (in some cases, not all!). For instance, can numexpr deal well with > > a + a.T > > where a is a c-contiguous array? Any numpy-like iteration order will not work well, one needs to use higher-dimensional (eg 2D) blocking, not 1D blocking. No. numexpr cannot deal with the above problem efficiently. numexpr is about 1d blocking, so its approach is pretty naive (but effective for these 1d blocking tasks). > (if numexpr can do this then great, the task might then reduce to refactoring numexpr so that cython and numba can use the same logic) Well, now that I think about it, the virtual machine in latest numexpr (2.0) is based on the new nditer iterator included in NumPy 1.6, so I'm wondering if with a little bit of more effort, the 2d blocking could be implemented. While I'm pretty sure this is the case, I don't know if it would be really worth the effort. Perhaps concentrating on things like numba + projectX, or just Theano would be better targets. Perhaps Mark Wiebe (who contributed the new nditer-aware VM) could say a bit more about this. Hmm, some good food for brains :) -- Francesc Alted

3 3

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
by Nathaniel Smith March 21, 2012

March 21, 2012

[Re-adding the list to the To: field, after it got dropped accidentally] On Tue, Feb 28, 2012 at 12:28 AM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: > Excerpts from Nathaniel Smith's message of Mon Feb 27 17:33:52 -0500 2012: >> On Mon, Feb 27, 2012 at 6:02 PM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: >> > Excerpts from Nathaniel Smith's message of Mon Feb 27 12:07:11 -0500 2012: >> >> On Mon, Feb 27, 2012 at 2:44 PM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: >> >> > What I've got is a solution for writing and reading structured arrays to >> >> > and from files, both in text files and binary files. It is written in C >> >> > and python. It allows reading arbitrary subsets of the data efficiently >> >> > without reading in the whole file. It defines a class Recfile that >> >> > exposes an array like interface for reading, e.g. x=rf[columns][rows]. >> >> >> >> What format do you use for binary data? Something tiled? I don't >> >> understand how you can read in a single column of a standard text or >> >> mmap-style binary file any more efficiently than by reading the whole >> >> file. >> > >> > For binary, I just seek to the appropriate bytes on disk and read them, >> > no mmap. The user must have input an accurate dtype describing rows in >> > the file of course. This saves a lot of memory and time on big files if >> > you just need small subsets. >> >> Have you quantified the time savings? I'd expect this to either be the >> same speed or slower than reading the entire file. > > Nathaniel - > > Yes I've verified it, but as you point out below there are pathological > cases. See below. > >> The reason is that usually the OS cannot read just a few bytes from a >> middle of a file -- if it is reading at all, it will read at least a >> page (4K on linux). If your rows are less than 4K in size, then >> reading a little bit out of each row means that you will be loading >> the entire file from disk regardless. You avoid any parsing overhead >> for the skipped columns, but for binary files that should be zero. >> (Even if you're doing endian conversion or something it should still >> be trivial compared to disk speed.) > > I'll say up front, the speed gains for binary data are often huge over > even numpy.memmap because memmap is not column aware. My code doesn't > have that limitation. Hi Erin, I don't doubt your observations, but... there must be something more going on! In a modern VM implementation, what happens when you request to read from an arbitrary offset in the file is: 1) The OS works out which disk page (or set of pages, for a longer read) contains the given offset 2) It reads those pages from the disk, and loads them into some OS owned buffers (the "page cache") 3) It copies the relevant bytes out of the page cache into the buffer passed to read() And when you mmap() and then attempt to access some memory at an arbitrary offset within the mmap region, what happens is: 1) The processor notices that it doesn't actually know how the memory address given maps to real memory (a tlb miss), so it asks the OS 2) The OS notices that this is a memory-mapped region, and works out which disk page maps to the given memory address 3) It reads that page from disk, and loads it into some OS owned buffers (the "page cache") 4) It tells the processor That is, reading at a bunch of fixed offsets inside a large memory mapped array (which is what numpy does when you request a single column of a recarray) should end up issuing *exactly the same read commands* as writing code that explicitly seeks to those addresses and reads them. But, I realized I've never actually tested this myself, so I wrote a little test (attached). It reads a bunch of uint32's at equally-spaced offsets from a large file, using either mmap, explicit seeks, or the naive read-everything approach. I'm finding it very hard to get precise results, because I don't have a spare drive and anything that touches the disk really disrupts the timing here (and apparently Ubuntu no longer has a real single-user mode :-(), but here are some examples on a 200,000,000 byte file with different simulated row sizes: 1024 byte rows: Mode: MMAP. Checksum: bdd205e9. Time: 3.44 s Mode: SEEK. Checksum: bdd205e9. Time: 3.34 s Mode: READALL. Checksum: bdd205e9. Time: 3.53 s Mode: MMAP. Checksum: bdd205e9. Time: 3.39 s Mode: SEEK. Checksum: bdd205e9. Time: 3.30 s Mode: READALL. Checksum: bdd205e9. Time: 3.17 s Mode: MMAP. Checksum: bdd205e9. Time: 3.16 s Mode: SEEK. Checksum: bdd205e9. Time: 3.41 s Mode: READALL. Checksum: bdd205e9. Time: 3.43 s 65536 byte rows (64 KiB): Mode: MMAP. Checksum: da4f9d8d. Time: 3.25 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.27 s Mode: READALL. Checksum: da4f9d8d. Time: 3.16 s Mode: MMAP. Checksum: da4f9d8d. Time: 3.34 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.36 s Mode: READALL. Checksum: da4f9d8d. Time: 3.44 s Mode: MMAP. Checksum: da4f9d8d. Time: 3.18 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.19 s Mode: READALL. Checksum: da4f9d8d. Time: 3.16 s 1048576 byte rows (1 MiB): Mode: MMAP. Checksum: 22963df9. Time: 1.57 s Mode: SEEK. Checksum: 22963df9. Time: 1.44 s Mode: READALL. Checksum: 22963df9. Time: 3.13 s Mode: MMAP. Checksum: 22963df9. Time: 1.59 s Mode: SEEK. Checksum: 22963df9. Time: 1.43 s Mode: READALL. Checksum: 22963df9. Time: 3.16 s Mode: MMAP. Checksum: 22963df9. Time: 1.55 s Mode: SEEK. Checksum: 22963df9. Time: 1.66 s Mode: READALL. Checksum: 22963df9. Time: 3.15 s And for comparison: In [32]: a = np.memmap("src/bigfile", np.uint32, "r") In [33]: time hex(np.sum(a[::1048576//4][:-1], dtype=a.dtype)) CPU times: user 0.00 s, sys: 0.01 s, total: 0.01 s Wall time: 1.54 s Out[34]: '0x22963df9L' (Ubuntu Linux 2.6.38-13, traditional spinning-disk media) So, in this test: For small rows: seeking is irrelevant, reading everything is just as fast. (And the cutoff for "small" is not very small... I tried 512KiB too and it looked like 32KiB). For large rows: seeking is faster than reading everything. But mmap, explicit seeks, and np.memmap all act the same. I guess it's possible the difference you're seeing could just mean that, like, Windows has a terrible VM subsystem, but that would be weird. > In the ascii case the gains for speed are less > for the reasons you point out; you have to read through the data even to > skip rows and fields. Then it is really about memory. > > > Even for binary, there are pathological cases, e.g. 1) reading a random > subset of nearly all rows. 2) reading a single column when rows are > small. In case 2 you will only go this route in the first place if you > need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. > I wrote this code to deal with a typical use case of mine where small > subsets of binary or ascii data are begin read from a huge file. It > solves that problem. Cool. >> If your rows are greater than 4K in size, then seeking will allow you >> to avoid loading some parts of the file from disk... but it also might >> defeat the OS's readahead heuristics, which means that instead of >> doing a streaming read, you're might be doing a disk seek for every >> row. On an SSD, this is fine, and probably a nice win. On a >> traditional spinning-disk, losing readahead will cause a huge >> slowdown; you should only win if each row is like... a megabyte >> apiece. Seeks are much much slower than continuous reads. So I'm >> surprised if this actually helps! But that's just theory, so I am >> curious to see the actual numbers. >> >> Re: memory savings -- it's definitely a win to avoid allocating the >> whole array if you just want to read one column, but you can >> accomplish that without any particular cleverness in the low-level >> file reading code. You just read the first N rows into a fixed-size >> temp buffer, copy out the relevant column into the output array, >> repeat. > > Certainly this could also be done that way, and would be equally good > for some cases. > > I've already got all the C code to do this stuff, so it not much work > for me to hack it into my numpy fork. If it turns out there are cases > that are faster using another method, we should root them out during > testing and add the appropriate logic. Cool. I'm just a little concerned that, since we seem to have like... 5 different implementations of this stuff all being worked on at the same time, we need to get some consensus on which features actually matter, so they can be melded together into the Single Best File Reader Evar. An interface where indexing and file-reading are combined is significantly more complicated than one where the core file-reading inner-loop can ignore indexing. So far I'm not sure why this complexity would be worthwhile, so that's what I'm trying to understand. Cheers, -- Nathaniel > Also, for some crazy ascii files we may want to revert to pure python > anyway, but I think these should be special cases that can be flagged > at runtime through keyword arguments to the python functions. > > BTW, did you mean to go off-list? > > cheers, > > -e > -- > Erin Scott Sheldon > Brookhaven National Laboratory

7 11

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
by Dag Sverre Seljebotn March 20, 2012

March 20, 2012

Sorry, forgot to CC list on this. Lines staring with single greater-than are mine. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Dag Sverre Seljebotn <d.s.seljebotn(a)astro.uio.no> wrote: Francesc Alted <francesc(a)continuum.io> wrote: >On Mar 20, 2012, at 12:49 PM, mark florisson wrote: >>> Cython and Numba certainly overlap. However, Cython requires: >>> >>> 1) learning another language >>> 2) creating an extension module --- loading bit-code files >and dynamically executing (even on a different machine from the one >that initially created them) can be a powerful alternative for run-time >compilation and distribution of code. >>> >>> These aren't show-stoppers obviously. But, I think some users >would prefer an even simpler approach to getting fast-code than Cython >(which currently doesn't do enought type-inference and requires >building a dlopen extension module). >> >> Dag and I have been discussing this at PyCon, and here is my take on >> it (at this moment :). >> >> Definitely, if you can avoid Cython then that is easier and more >> desirable in many ways. So perhaps we can create a third project >> called X (I'm not very creative, maybe ArrayExprOpt), that takes an >> abstract syntax tree in a rather simple form, performs code >> optimizations such as rewriting loops with array accesses to vector >> expressions, fusing vector expressions and loops, etc, and spits out >a >> transformed AST containing these optimizations. If runtime >information >> is given such as actual shape and stride information the >> transformations could figure out there and then whether to do things >> like collapsing, axes swapping, blocking (as in, introducing more >axes >> or loops to retain discontiguous blocks in the cache), blocked memory >> copies to contiguous chunks, etc. The AST could then also say whether >> the final expressions are vectorizable. Part of this functionality is >> already in numpy's nditer, except that this would be implicit and do >> more (and hopefully with minimal overhead). >> >> So numba, Cython and maybe numexpr could use the functionality, >simply >> by building the AST from Python and converting back (if necessary) to >> its own AST. As such, the AST optimizer would be only part of any >> (runtime) compiler's pipeline, and it should be very flexible to >> retain any information (metadata regarding actual types, control flow >> information, etc) provided by the original AST. It would not do >> control flow analysis, type inference or promotion, etc, but only >deal >> with abstract types like integers, reals and arrays (C, Fortran or >> partly contiguous or strided). It would not deal with objects, but >> would allow to insert nodes like UnreorderableNode and SideEffectNode >> wrapping parts of the original AST. In short, it should be as easy as >> possible to convert from an original AST to this project's AST and >> back again afterwards. > >I think this is a very interesting project, and certainly projects like >numba can benefit of it. So, in order to us have an idea on what you >are after, can we assume that your project (call it X) would be kind of >an compiler optimizer, and then the produced, optimized code could be >feed into numba for optimized LLVM code generation (that on its turn, >can be run on top of CPUs or GPUs or a combination)? Is that correct? I think so. Another way of thinking about it is that it is a reimplementation of the logic in the (good and closed source) Fortran 90 compilers, in a reusable component for inclusion in various compilers. Various c++ metaprogramming libraries (like Blitz++) are similar too. > >Giving that my interpretation above is correct, it is bit more >difficult to me to see how your X project could be of benefit for >numexpr. In fact, I actually see this the other way round: once the >optimizer has discovered the vectorization parts, then go one step >further and generate code that uses numexpr automatically (call this, >vectorization through numexpr). This is what you mean, or I'm missing >something? No. I think in some ways this is a competitor to numexpr -- you would gut out the middle of numexpr and keep the frontend and backend, but use this to optimize iteration order and blocking strategies. I think the goal is for higher performance than what I understand numexpr can provide (in some cases, not all!). For instance, can numexpr deal well with a + a.T where a is a c-contiguous array? Any numpy-like iteration order will not work well, one needs to use higher-dimensional (eg 2D) blocking, not 1D blocking. (if numexpr can do this then great, the task might then reduce to refactoring numexpr so that cython and numba can use the same logic) Dag > >> As the project matures many optimizations may be added that deal with >> all sorts of loop restructuring and ways to efficiently utilize the >> cache as well as enable vectorization and possibly parallelism. >> Perhaps it could even generate a different AST depending on whether >> execution target the CPU or the GPU (with optionally available >> information such as cache sizes, GPU shared/local memory sizes, etc). >> >> Seeing that this would be a part of my master dissertation, my >> supervisor would require me to write the code, so at least until >> August I think I would have to write (at least the bulk of) this. >> Otherwise I can also make other parts of my dissertation's project >> more prominent to make up for it. Anyway, my question is, is there >> interest from at least the numba and numexpr projects (if code can be >> transformed into vector operations, it makes sense to use numexpr for >> that, I'm not sure what numba's interest is in that). > >I'm definitely interested for the numexpr part. It is just that I'm >still struggling to see the big picture on this. But the general idea >is really appealing. > >Thanks, > >-- Francesc Alted -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

1 0

Question on numpy.ma.masked_values
by Gökhan Sever March 20, 2012

March 20, 2012

Hello, >From the masked_values() documentation -> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ma.masked_values.… I10 np.ma.masked_values(x, 1.5) O10 masked_array(data = [ 1. 1.1 2. 1.1 3. ], mask = False, fill_value = 1.5) I12 np.ma.masked_values(x, 1.5, shrink=False) O12 masked_array(data = [ 1. 1.1 2. 1.1 3. ], mask = False, fill_value = 1.5) Shouldn't setting the 'shrink' to False return an array of False values for the mask field? If not so, how can I return a set of False values if my masking condition is not met? Using: I16 np.__version__ O16 '2.0.0.dev-7e202a2' Thanks. -- Gökhan

2 6

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 66, Issue 61
by Matthieu Rigal March 20, 2012

March 20, 2012

Hi Richard, Thanks for your answer and the related help ! In fact, I was hoping to have a less memory and more speed solution. Something equivalent to a "raster calculator" for numpy. Wouldn't it make sense to have some optimized function to work on more than 2 arrays for numpy anyway ? At the end, I am rather interested by more speed. I tried first a code-sparing version : array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() But this one is at the end more than 2 times slower than : array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) numpy.less(aBlueChannel, 1.0, out=array1[0]) numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) array = array1.all() (and this solution is about 30% faster than the original one) I could find another way which was fine for me too: array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * (aNirChannel < (aBlueChannel * 1.8)) But this one is only 5-10% faster than the original solution, even if probably using less memory than the 2 previous ones. (same was possible with operator +, but slower than operator *) Regards, Matthieu Rigal On Monday 19 March 2012 18:00:02 numpy-discussion-request(a)scipy.org wrote: > Message: 2 > Date: Mon, 19 Mar 2012 13:20:23 +0000 > From: Richard Hattersley <rhattersley(a)gmail.com> > Subject: Re: [Numpy-discussion] Using logical function on more than 2 > arrays, availability of a "between" function ? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAP=RS9=UBOc6Kmtmnne7W093t19w=T=oSrXUAW0WF8B49hqcXQ(a)mail.gmail.com > > Content-Type: text/plain; charset=ISO-8859-1 > > What do you mean by "efficient"? Are you trying to get it execute > faster? Or using less memory? Or have more concise source code? > > Less memory: > - numpy.vectorize would let you get to the end result without any > intermediate arrays but will be slow. > - Using the "out" parameter of numpy.logical_and will let you avoid > one of the intermediate arrays. > > More speed?: > Perhaps putting all three boolean temporary results into a single > boolean array (using the "out" parameter of numpy.greater, etc) and > using numpy.all might benefit from logical short-circuiting. > > And watch out for divide-by-zero from "aNirChannel/aBlueChannel". > > Regards, > Richard Hattersley > RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Ryan Johnson Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Robert Johnson Commercial Register/Handelsregister Potsdam HRB 24742 P Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified

2 1

Re: [Numpy-discussion] Using logical function on more than 2 arrays, availability of a "between" function ?
by Matthieu Rigal March 20, 2012

March 20, 2012

Auto-answer, sorry, Well, actually I made a mistake lower... that you may have noticed... On the faster (your) solution, even with a cleaner use of the out parameter, the fact that the all has then to be used with parameter axis=0 takes more time and makes it actually slower than the initial solution... So I may go for the "multiplier" solution. Regards, Matthieu On Tuesday 20 March 2012 13:13:15 you wrote: > Hi Richard, > > Thanks for your answer and the related help ! > > In fact, I was hoping to have a less memory and more speed solution. > Something equivalent to a "raster calculator" for numpy. Wouldn't it make > sense to have some optimized function to work on more than 2 arrays for > numpy anyway ? > > At the end, I am rather interested by more speed. > > I tried first a code-sparing version : > array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * > 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() > > But this one is at the end more than 2 times slower than : > array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) > numpy.less(aBlueChannel, 1.0, out=array1[0]) > numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) > numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) > array = array1.all() > > (and this solution is about 30% faster than the original one) > > I could find another way which was fine for me too: > array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * > (aNirChannel < (aBlueChannel * 1.8)) > > But this one is only 5-10% faster than the original solution, even if > probably using less memory than the 2 previous ones. (same was possible > with operator +, but slower than operator *) > > Regards, > Matthieu Rigal > > On Monday 19 March 2012 18:00:02 numpy-discussion-request(a)scipy.org wrote: > > Message: 2 > > Date: Mon, 19 Mar 2012 13:20:23 +0000 > > From: Richard Hattersley <rhattersley(a)gmail.com> > > Subject: Re: [Numpy-discussion] Using logical function on more than 2 > > arrays, availability of a "between" function ? > > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > Message-ID: > > > > <CAP=RS9=UBOc6Kmtmnne7W093t19w=T=oSrXUAW0WF8B49hqcXQ(a)mail.gmail.com > > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > What do you mean by "efficient"? Are you trying to get it execute > > faster? Or using less memory? Or have more concise source code? > > > > Less memory: > > - numpy.vectorize would let you get to the end result without any > > intermediate arrays but will be slow. > > - Using the "out" parameter of numpy.logical_and will let you avoid > > one of the intermediate arrays. > > > > More speed?: > > Perhaps putting all three boolean temporary results into a single > > boolean array (using the "out" parameter of numpy.greater, etc) and > > using numpy.all might benefit from logical short-circuiting. > > > > And watch out for divide-by-zero from "aNirChannel/aBlueChannel". > > > > Regards, > > Richard Hattersley > RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Ryan Johnson Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Robert Johnson Commercial Register/Handelsregister Potsdam HRB 24742 P Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified

2 1

Trying to read 500M txt file using numpy.genfromtxt within ipython shell
by Chao YUE March 20, 2012

March 20, 2012

Dear all, I received a file from others which contains ~30 million lines and in size of ~500M. I try read it with numpy.genfromtxt in ipython interactive mode. Then ipython crashed. The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. Finally I want to write the data to netcdf for separate years and feed them into the model. I guess I need a better way to do this? anyone would be any idea is highly appreciated. lon,lat,year,area_burned -180.0,65.0,1001,0 -180.0,65.0,1002,0 -180.0,65.0,1003,0 -180.0,65.0,1004,0 -180.0,65.0,1005,0 -180.0,65.0,1006,0 -180.0,65.0,1007,0 thanks and cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 3

Recovering from a STOP ?
by Pierre GM March 20, 2012

March 20, 2012

THYC Dear all, I'm working with some large inherited F90 code that needs to be wrapped in Python. if the code base itself cannot be modified (it's a static archive), some additional F90 files were written to help the interaction with the code. Writing a python extension combining the archive and these additional files is rather straightforward using F2PY, the extension loads, all is well but... ...when a problem occurs in a subroutine deep in the base code, it calls another routine (say, ENDRUN) which itself uses a STOP statement. As you know, this statement not only exits the fortran code, but makes the extension and then the interpreter crash... I patched ENDRUN use STOP, I tried to call a C external function instead of using STOP: """ *- ENDRUN.f90 -* subroutine ENDRUN(C_NOM) use endrun_wrap Implicit none character(LEN=*), intent(in) :: C_NOM ! write(*, *) C_NOM ! stop call pyraise_runtime(C_NOM) end subroutine ENDRUN *- ENDRUNWRAP.f90 -* module endrun_wrap interface subroutine pyraise_runtime(message) character(LEN=*), intent(in) :: message end subroutine pyraise_runtime end interface end module endrun_wrap *- pyraise_runtime.c -* #include <Python.h> void pyraise_runtime_(char *message); void pyraise_runtime_(char *message) { printf("calling PyErr_SetString(PyExc_RuntimeError, message)\n"); PyErr_SetString(PyExc_RuntimeError, message); } """ Alas, the RuntimeError doesn't look like it's passed back to the interpreter, which still crashes. (Adding a Py_Exit(-1) at the end of pyraise_runtime at least let the interpreter do some extra cleaning after the fortran code stopped, but still...) Note that ENDRUN is never supposed to be called directly by the user (so no point to define a callback function via f2py, right ?). Any idea would be quite welcome. Thanks an awful lot in advance. P. PS: Oh, BTW, I'm limited to numpy 1.4.1 and the f2py that comes with it...

3 3

numpy installation problem?
by avalle＠famaf.unc.edu.ar March 18, 2012

March 18, 2012

Dear list, I am having problems installing matplotlib (from source) and fipy. I had installed numpy from source and it is running well: :~$ python -c "import numpy; print numpy.__version__" 1.6.1 After being trying to solve this problem on matplotlib list, I was recommended to ask numpy list. I copy-paste the errors below and forward the mails from matplotlib users. ... .... creating /usr/local/lib/python2.7/dist-packages/FiPy-2.1.2-py2.7.egg Extracting FiPy-2.1.2-py2.7.egg to /usr/local/lib/python2.7/dist-packages FiPy 2.1.2 is already the active version in easy-install.pth Installed /usr/local/lib/python2.7/dist-packages/FiPy-2.1.2-py2.7.egg Processing dependencies for FiPy==2.1.2 Finished processing dependencies for FiPy==2.1.2 Traceback (most recent call last): File "setup.py", line 594, in <module> __import__(pkg) File "/usr/local/lib/python2.7/dist-packages/pysparse-1.2_dev224-py2.7-linux-x86_64.egg/pysparse/__init__.py", line 6, in <module> from numpy._import_tools import PackageLoader File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/usr/local/lib/python2.7/dist-packages/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' :~/FiPy-2.1.2$ :~/matplotlib$ sudo python setup.py install basedirlist is: ['/usr/local', '/usr'] ============================================================================ BUILDING MATPLOTLIB matplotlib: 1.2.x python: 2.7.2+ (default, Oct 4 2011, 20:06:09) [GCC 4.6.1] platform: linux2 REQUIRED DEPENDENCIES numpy: no * You must install numpy 1.4 or later to build * matplotlib. I do not understand why matplotlib can not see numpy. Please, see the forwarded message. My notebook description: Linux 3.0.0-13-generic #22-Ubuntu SMP Wed Nov 2 13:27:26 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Thank in advance for your help Regards, Lucia

2 1