From ndbecker2 at gmail.com Tue Jan 1 14:07:33 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 01 Jan 2008 14:07:33 -0500 Subject: [Numpy-discussion] fast iteration (I think I've got it) Message-ID: This is a c-api question. I'm trying to get iterators that are both fast and reasonably general. I did confirm that iterating using just the general PyArrayIterObject protocol is not as fast as using c-style pointers for contiguous arrays. Please confirm if my understanding is correct. There are 2 cases to consider. There are plain old dense arrays, which will be contiguous, and there are array 'views' or 'slices'. The views will not be contiguous. However, in all cases, the strides between elements in one dimension are constant. This last point is key. As long as the stride in the last dimension is a constant, a fast iterator is possible. I put together a small test, which seems to work for the cases I tried. The idea is to use the general iterator (via PyArray_IterAllButAxis), but iteration over the last dimension is done with a c-style pointer. I have tested it with these cases, and the results appear correct: a = array ((xrange(4),xrange(4,8),xrange(8,12)), int) b = a[:,::2] c = a[:,1::2] d = a[:,-1::-1] sum3(a) 0 1 2 3 4 5 6 7 8 9 10 11 sum3(b) 0 2 4 6 8 10 sum3(c) 1 3 5 7 9 11 sum3 (d) 3 2 1 0 7 6 5 4 11 10 9 8 template inline T sum3 (object const& o) { PyArrayObject* arr = (PyArrayObject*)(o.ptr()); int last_dim = arr->nd - 1; PyArrayIterObject* it = (PyArrayIterObject*)(PyArray_IterAllButAxis (o.ptr(), &last_dim)); for (int i = 0 ; i < arr->nd; ++i) std::cout << arr->dimensions[i] << ' '; std::cout << '\n'; while (it->index < it->size) { T * dptr = (T*)(it->dataptr); const int stride = arr->strides[last_dim]; for (int i = 0; i < arr->dimensions[last_dim]; ++i) { std::cout << *dptr << ' '; dptr += stride/sizeof(T); } std::cout << '\n'; PyArray_ITER_NEXT(it); } return 0; } From ondrej at certik.cz Tue Jan 1 15:07:52 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 1 Jan 2008 21:07:52 +0100 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> On Dec 31, 2007 11:43 PM, Jarrod Millman wrote: > Hey, > > I just wanted to announce that we now have a NumPy/SciPy blog > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ Cool! I was going to write an email that it could be a good idea. Thanks Gael. > Feel free to contact me if you have a blog that you would like included. Is it just for anything, or just scipy/numpy related? I use this blog: http://ondrejcertik.blogspot.com/ however, since I am new to blogging, I am not sure what the best practise is. I blog about a lot of things (SymPy, Sage, Debian), but I think just something is suitable to be on planet.scipy.org. So I think the best way is to just use tags. So could you please put this into the planet.conf? [http://ondrejcertik.blogspot.com/feeds/posts/default/-/scipy] name = Ondřej Čertík and I'll be tagging scipy related blogposts with the scipy tag (currently 0 posts:). Ondrej From peridot.faceted at gmail.com Tue Jan 1 15:34:06 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 1 Jan 2008 15:34:06 -0500 Subject: [Numpy-discussion] fast iteration (I think I've got it) In-Reply-To: References: Message-ID: On 01/01/2008, Neal Becker wrote: > This is a c-api question. > > I'm trying to get iterators that are both fast and reasonably general. I > did confirm that iterating using just the general PyArrayIterObject > protocol is not as fast as using c-style pointers for contiguous arrays. I'd like to point out that "contiguous" can be misleading as it is used in numpy. An array is flagged contiguous if all the elements are contiguous *and* they are arranged as a C-ordered multidimensional array - i.e., the last index changes the fastest as you walk through the array elements, and the next-to-last chenges the next fastest, and so on. > Please confirm if my understanding is correct. There are 2 cases to > consider. There are plain old dense arrays, which will be contiguous, and > there are array 'views' or 'slices'. The views will not be contiguous. > However, in all cases, the strides between elements in one dimension are > constant. This last point is key. As long as the stride in the last > dimension is a constant, a fast iterator is possible. > > I put together a small test, which seems to work for the cases I tried. > The idea is to use the general iterator (via PyArray_IterAllButAxis), but > iteration over the last dimension is done with a c-style pointer. This is not an unreasonable approach, but it can suffer badly if the last dimension is "small", for example if you have an array of shape (1000,1000,2). Such arrays can arise for example from working with complex numbers as pairs of real numbers. Where is the acceleration coming from in this code? You can walk through a multidimensional array in pure C, just incrementing pointers as needed, without too much difficulty. If you want to save bookkeeping overhead, you can partially flatten the array: if your last dimension has stride 7 and length 11, and your second-to-last dimension has stride 77, you can flatten (using reshape, in python, to get a view) the last two dimensions of the array and use a nice quick iterator on the combined dimension. For a "C contiguous" array, this means you just walk through the whole array with a single layer of looping. I suspect that the place you will see big slowdowns is where data is accessed in an order that doesn't make sense from a cache point of view. If four-byte chunks of data are spaced every eight bytes, then the processor spends twice as much time waiting for cache loads as if they are spaced every four bytes. And most numpy tasks are almost certainly memory-bound - all the ufuncs I can think of almost certainly are (arctan of a double almost certainly takes less time than loading and storing a double). If you walk through an array in the wrong order - changing the big strides fastest and the small strides slowest - you'll almost certainly have to shuffle the whole array through cache eight times or more. This is even ignoring cache misses. (To cut down on cache misses you can use the GCC prefetch instructions, or god-knows-what equivalent in other compilers.) I seem to recall that numpy already reorders its walks through arrays within ufuncs - does it do so with cache in mind? For that matter, is information on the CPU's caches available to numpy? Anne P.S. Here is code to flatten an array as much as possible without changing the order of flatiter: def flatteniter(A): i=0 while i Message-ID: Thank you for the response. I'm afraid I haven't explained what I'm doing. I have a lot of c++ code written in a generic c++ interface style. (The code is for signal processing, but that is irrelevant). The code is generic for data types as well as container types. To accomplish this, the interface uses the boost::range interface concept. An example of using this interface: template void F (in_t const& in) { typename boost::range_const_iterator::type i = boost::begin (in); for (; i != boost::end (in); ++i) do_something_with (*i); } In the above, in_t is a container type. It could be std::vector, for example. The concept has: range_iterator::type and range_const_iterator::type are the iterator types for the range begin(in) gives and iterator pointing to the beginning ++i increments the iterator *i derefs the iterator This allows writing functions that work with different container types. For example, std::vector, boost::ublas::vector. I'm trying to make this work with numpy. To do this, I'm using boost::iterator_facade to create appropriate iterators. In the simplest case, this is just a wrapper around PyArrayIterObject*. Using this directly results in code equivalent to: * uses PyArray_IterNew to create the iterator, * PyArray_ITER_NEXT to increment the iterator, * it->dataptr for deref This will be slower than necessary. So, I hope this explains the motivation behind the plan. I'm hoping that I can use PyArray_IterAllButAxis to iterate over arbitrary numpy arrays, but with the inner loop using dptr += stride/sizeof(T) to speed the access. I believe that all numpy arrays have a constant stride over any one dimension, which would allow this to work. From millman at berkeley.edu Tue Jan 1 16:31:11 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 1 Jan 2008 13:31:11 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: On Jan 1, 2008 12:07 PM, Ondrej Certik wrote: > So could you please put this into the planet.conf? > > [http://ondrejcertik.blogspot.com/feeds/posts/default/-/scipy] > name = Ondřej Čertík Done. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From oliphant at enthought.com Tue Jan 1 16:37:54 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 01 Jan 2008 15:37:54 -0600 Subject: [Numpy-discussion] fast iteration (I think I've got it) In-Reply-To: References: Message-ID: <477AB2B2.3090402@enthought.com> Neal Becker wrote: > This is a c-api question. > > I'm trying to get iterators that are both fast and reasonably general. I > did confirm that iterating using just the general PyArrayIterObject > protocol is not as fast as using c-style pointers for contiguous arrays. > > Please confirm if my understanding is correct. There are 2 cases to > consider. There are plain old dense arrays, which will be contiguous, and > there are array 'views' or 'slices'. The views will not be contiguous. > However, in all cases, the strides between elements in one dimension are > constant. This last point is key. As long as the stride in the last > dimension is a constant, a fast iterator is possible. > Your understanding is correct. You can use the iterator over all but one dimension. This is a common paradigm in NumPy (use an iterator over all but one dimension (usually chosen to be the one with the smallest striding) and then use strided access over the final dimension (see all the ufuncs for examples). IterAllButAxis takes a pointer to an integer. If that integer is negative on input, then IterAllButAxis will choose the axis for you to be the one with the smallest stride, then it will set the variable to the axis that was chosen. This is just what ufuncs do and is a pretty good idea generally to minimize the over-head of the final inner loop. -Travis O. From millman at berkeley.edu Tue Jan 1 20:14:51 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 1 Jan 2008 17:14:51 -0800 Subject: [Numpy-discussion] [OT] Which version of svn does svn.scipy.org run ? In-Reply-To: <4767756F.20507@ar.media.kyoto-u.ac.jp> References: <4767756F.20507@ar.media.kyoto-u.ac.jp> Message-ID: On Dec 17, 2007 11:23 PM, David Cournapeau wrote: > Is it possible to know which version of subversion is used by the > svn.scipy.org server ? I am trying to use svnsync on it, without any > success, and wanted to know if this was coming from svn.scipy.org or > from something else ? It is running subversion 1.2.3-2.1. In the near future, the server will be replaced with newer hardware and updated software. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From barrywark at gmail.com Tue Jan 1 21:18:34 2008 From: barrywark at gmail.com (Barry Wark) Date: Tue, 1 Jan 2008 18:18:34 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: Jarrod, I'd like to submit my blog for inclusion too... http://softwareforscientists.blogspot.com/search/label/scipy Again, not too many posts yet, but things will be ramping up. On Jan 1, 2008 1:31 PM, Jarrod Millman wrote: > On Jan 1, 2008 12:07 PM, Ondrej Certik wrote: > > So could you please put this into the planet.conf? > > > > [http://ondrejcertik.blogspot.com/feeds/posts/default/-/scipy] > > name = Ondřej Čertík > > Done. > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Tue Jan 1 21:35:34 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 1 Jan 2008 18:35:34 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: On Jan 1, 2008 6:18 PM, Barry Wark wrote: > I'd like to submit my blog for inclusion too... > > http://softwareforscientists.blogspot.com/search/label/scipy Hmm. I added you, but while I was adding you it looks like your blog disappeared. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From ndbecker2 at gmail.com Tue Jan 1 21:49:08 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 01 Jan 2008 21:49:08 -0500 Subject: [Numpy-discussion] need inverse of PyArray_ITER_GOTO1D Message-ID: I need a function that is the inverse of PyArray_ITER_GOTO1D. A bit of guesswork brought me to this, can anyone please tell me if it looks correct? inline npy_intp as_linear (PyArrayIterObject const* it) { if (it->nd_m1 == 0) return (it->dataptr - it->ao->data)/it->strides[0]; else if (it->contiguous) return (it->dataptr - it->ao->data)/it->ao->descr->elsize; else { npy_intp loc = 0; npy_intp offset = (it->dataptr - it->ao->data)/it->ao->descr->elsize; for (int i = 0; i <= it->nd_m1; ++i) { loc += offset / it->factors[i]; offset %= it->factors[i]; } return loc; } } From wbaxter at gmail.com Wed Jan 2 01:22:03 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 2 Jan 2008 15:22:03 +0900 Subject: [Numpy-discussion] Travis and the NumPy Documentation Message-ID: I just read the blog post about Travis switching jobs Enthought. Was that posted here? If so I'm surprised I missed it. Anyway that sounds like great news for the NumPy/SciPy communities. I hope it also is great news for you and your family, Travis. I have to say that I took some amount of inspiration from seeing how much you were able to do with the open source community while being a tenure track prof, since I've thought about going to academia myself, but would hate to give up involvement with open source projects. Maybe I need to rethink the academia thing now, though. My real question though -- I was wondering whether this move would have any impact on the NumPy documentation? I purchased my copy, and am glad to have been able to contribute to your efforts by doing so, but I know not everyone feels the same. If I recall, part of the incentive given for making the documentation for-fee was that your day-job didn't compensate you for writing such documentation or other NumPy efforts, so charging for documentation was one way to get back a little something. But if I understand correctly, your current employer is very enthusiastic about your work on NumPy. So how about opening up the vault on the documentation a couple of years early? Or is there some sort of contractual agreement with Tregol that prevents that? Just wondering. --bb From oliphant at enthought.com Wed Jan 2 02:19:16 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 02 Jan 2008 01:19:16 -0600 Subject: [Numpy-discussion] Travis and the NumPy Documentation In-Reply-To: References: Message-ID: <477B3AF4.6090206@enthought.com> Bill Baxter wrote: > I just read the blog post about Travis switching jobs Enthought. Was > that posted here? If so I'm surprised I missed it. > I don't think it was posted here. It was announced at SciPy and Enthought put out a press release (but the planet.scipy.org aggregator was not running at that point :-) ) > I have to say that I took some amount of inspiration from seeing how > much you were able to do with the open source community while being a > tenure track prof, since I've thought about going to academia myself, > but would hate to give up involvement with open source projects. > Maybe I need to rethink the academia thing now, though. > Perhaps times will change (and I would love it if they did). William Stein is doing some great things with SAGE that may help. But, you do need to be careful and clear about what your goals are. > My real question though -- I was wondering whether this move would > > have any impact on the NumPy documentation? Yes, it will have an impact but the exact details are still being worked out. I am comfortable, however, in stating that Enthought has already made a significant contribution towards the release of the book in hiring me. > I purchased my copy, and > am glad to have been able to contribute to your efforts by doing so, > but I know not everyone feels the same. If I recall, part of the > incentive given for making the documentation for-fee was that your > day-job didn't compensate you for writing such documentation or other > That was part of it. There were basically two reasons for doing what I did: 1) I needed the money (to pay graduate students and to cover other bills). 2) I believe that the more open source can continue to connect with the money economy, the bigger it will be. I want to see the kind of resources that are now spent on proprietary software development lead to open source software. I see the market-determined mechanism that Trelgol proposed as one possible way for that to happen (among others that are being explored). As a result, the "Guide to NumPy" experiment contains very useful information independent of anything else. As soon as my house in Utah sells (I moved at a rather bad time....), point #1 will not be as critical for me anymore. But, point #2 still remains until the book's market price is reached. As it stands, it looks like the revenues are on target for release in 2009. I'll know more details when the books are closed for 2007. I hope this helps. Best regards, -Travis O. From ramercer at gmail.com Wed Jan 2 06:45:25 2008 From: ramercer at gmail.com (Adam Mercer) Date: Wed, 2 Jan 2008 11:45:25 +0000 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: <799406d60801020345v716aad0oe96ab48664f33625@mail.gmail.com> On Dec 31, 2007 10:43 PM, Jarrod Millman wrote: > Hey, > > I just wanted to announce that we now have a NumPy/SciPy blog > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ When I try to load http://planet.scipy.org I get an error saying that the page doesn't exist, however the scipy home page, i.e. http://www.scipy.org, now appears to be the planet aggregator is this just a temporary DNS issue? Cheers Adam From ondrej at certik.cz Wed Jan 2 07:53:25 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 2 Jan 2008 13:53:25 +0100 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <799406d60801020345v716aad0oe96ab48664f33625@mail.gmail.com> References: <799406d60801020345v716aad0oe96ab48664f33625@mail.gmail.com> Message-ID: <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> On Jan 2, 2008 12:45 PM, Adam Mercer wrote: > On Dec 31, 2007 10:43 PM, Jarrod Millman wrote: > > Hey, > > > > I just wanted to announce that we now have a NumPy/SciPy blog > > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ > > When I try to load http://planet.scipy.org I get an error saying that > the page doesn't exist, however the scipy home page, i.e. > http://www.scipy.org, now appears to be the planet aggregator is this > just a temporary DNS issue? It works for me. Ondrej From ndbecker2 at gmail.com Wed Jan 2 08:45:48 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 02 Jan 2008 08:45:48 -0500 Subject: [Numpy-discussion] need inverse of PyArray_ITER_GOTO1D References: Message-ID: Neal Becker wrote: > I need a function that is the inverse of PyArray_ITER_GOTO1D. A bit of > guesswork brought me to this, can anyone please tell me if it looks > correct? > > inline npy_intp as_linear (PyArrayIterObject const* it) { > if (it->nd_m1 == 0) > return (it->dataptr - it->ao->data)/it->strides[0]; > else if (it->contiguous) > return (it->dataptr - it->ao->data)/it->ao->descr->elsize; > else { > npy_intp loc = 0; > npy_intp offset = (it->dataptr - it->ao->data)/it->ao->descr->elsize; > for (int i = 0; i <= it->nd_m1; ++i) { > loc += offset / it->factors[i]; > offset %= it->factors[i]; > } > return loc; > } > } That didn't work for negative strides. Actually, I think all I need is this: inline npy_intp as_linear (PyArrayIterObject const* it) { return it->index; } Is this correct? From fullung at gmail.com Wed Jan 2 09:45:36 2008 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 2 Jan 2008 16:45:36 +0200 Subject: [Numpy-discussion] planet.scipy.org References: <799406d60801020345v716aad0oe96ab48664f33625@mail.gmail.com> <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> Message-ID: <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> Hello I also seem to be experiencing this problem. I have been able to successfully browse Planet SciPy for the past two days, but right now the DNS isn't even resolving. www.scipy.org still works though. Cheers, Albert ----- Original Message ----- From: "Ondrej Certik" To: "Discussion of Numerical Python" Sent: Wednesday, January 02, 2008 2:53 PM Subject: Re: [Numpy-discussion] planet.scipy.org > On Jan 2, 2008 12:45 PM, Adam Mercer wrote: >> On Dec 31, 2007 10:43 PM, Jarrod Millman wrote: >> > Hey, >> > >> > I just wanted to announce that we now have a NumPy/SciPy blog >> > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ >> >> When I try to load http://planet.scipy.org I get an error saying that >> the page doesn't exist, however the scipy home page, i.e. >> http://www.scipy.org, now appears to be the planet aggregator is this >> just a temporary DNS issue? > > It works for me. > > Ondrej From sransom at nrao.edu Wed Jan 2 10:20:11 2008 From: sransom at nrao.edu (Scott Ransom) Date: Wed, 2 Jan 2008 10:20:11 -0500 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> References: <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> Message-ID: <20080102152011.GA22106@ssh.cv.nrao.edu> Hmmm. When I try to load: http://planet.scipy.org/ I get: "Unknown host planet.scipy.org" However, http://planet.scipy.org (without the final slash) resolves fine. Scott On Wed, Jan 02, 2008 at 04:45:36PM +0200, Albert Strasheim wrote: > Hello > > I also seem to be experiencing this problem. I have been able to > successfully browse Planet SciPy for the past two days, but right now the > DNS isn't even resolving. www.scipy.org still works though. > > Cheers, > > Albert > > ----- Original Message ----- > From: "Ondrej Certik" > To: "Discussion of Numerical Python" > Sent: Wednesday, January 02, 2008 2:53 PM > Subject: Re: [Numpy-discussion] planet.scipy.org > > > > On Jan 2, 2008 12:45 PM, Adam Mercer wrote: > >> On Dec 31, 2007 10:43 PM, Jarrod Millman wrote: > >> > Hey, > >> > > >> > I just wanted to announce that we now have a NumPy/SciPy blog > >> > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ > >> > >> When I try to load http://planet.scipy.org I get an error saying that > >> the page doesn't exist, however the scipy home page, i.e. > >> http://www.scipy.org, now appears to be the planet aggregator is this > >> just a temporary DNS issue? > > > > It works for me. > > > > Ondrej > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From matthieu.brucher at gmail.com Wed Jan 2 11:07:55 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 2 Jan 2008 17:07:55 +0100 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <20080102152011.GA22106@ssh.cv.nrao.edu> References: <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> <20080102152011.GA22106@ssh.cv.nrao.edu> Message-ID: As the entry was added some days ago, some of us may have some DNS cache issues, it should be fine in a couple of days ;) Matthieu 2008/1/2, Scott Ransom : > > Hmmm. When I try to load: > > http://planet.scipy.org/ > > I get: "Unknown host planet.scipy.org" > > However, http://planet.scipy.org (without the final slash) > resolves fine. > > Scott > > > On Wed, Jan 02, 2008 at 04:45:36PM +0200, Albert Strasheim wrote: > > Hello > > > > I also seem to be experiencing this problem. I have been able to > > successfully browse Planet SciPy for the past two days, but right now > the > > DNS isn't even resolving. www.scipy.org still works though. > > > > Cheers, > > > > Albert > > > > ----- Original Message ----- > > From: "Ondrej Certik" > > To: "Discussion of Numerical Python" > > Sent: Wednesday, January 02, 2008 2:53 PM > > Subject: Re: [Numpy-discussion] planet.scipy.org > > > > > > > On Jan 2, 2008 12:45 PM, Adam Mercer wrote: > > >> On Dec 31, 2007 10:43 PM, Jarrod Millman > wrote: > > >> > Hey, > > >> > > > >> > I just wanted to announce that we now have a NumPy/SciPy blog > > >> > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ > > >> > > >> When I try to load http://planet.scipy.org I get an error saying that > > >> the page doesn't exist, however the scipy home page, i.e. > > >> http://www.scipy.org, now appears to be the planet aggregator is this > > >> just a temporary DNS issue? > > > > > > It works for me. > > > > > > Ondrej > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- > -- > Scott M. Ransom Address: NRAO > Phone: (434) 296-0320 520 Edgemont Rd. > email: sransom at nrao.edu Charlottesville, VA 22903 USA > GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Jan 2 12:32:51 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 02 Jan 2008 09:32:51 -0800 Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector In-Reply-To: <4776D744.5010601@gmail.com> References: <4771F0AE.40406@ncsu.edu> <7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com> <47732B84.70400@ncsu.edu> <4773325D.3040907@gmail.com> <4773BD97.1000703@ncsu.edu> <4773C17B.4050802@ncsu.edu> <4775DF5D.3030400@ncsu.edu> <7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com> <4776B111.9020607@ncsu.edu> <4776C3FC.7030101@ncsu.edu> <4776D1A5.60106@ncsu.edu> <4776D744.5010601@gmail.com> Message-ID: <477BCAC3.3020200@noaa.gov> Robert Kern wrote: > Bruce Sherwood wrote: >> There is also the question of >> whether it would pay for numpy to make what is probably an exceedingly >> fast check and do much faster calculations of sqrt(scalar) and other >> such mathematical functions. > > There is no question that it would pay. It takes time and effort to implement, > though. There is a question of whether it would pay enough to be worth the time and effort. For most use (at least most of my use), I'm either working with a small number of scalars, and performance is a non-issue, or I"m working with arrays of data, and then I need the numpy versions. So I really don't see the point of optimizing for scalar arguments. IIUC, the issue at hand involved not a desire for a faster sqrt(scalar), but rather an interaction with C++ code that didn't efficiently deal with numpy scalars -- that's a totally different issue, one that may have been solved by using math.sqrt() to avoid the numpy scalar, but not one that would that provides an argument for cluttering up numpy with this sort of special casing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ondrej at certik.cz Wed Jan 2 14:02:07 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 2 Jan 2008 20:02:07 +0100 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <20080102152011.GA22106@ssh.cv.nrao.edu> References: <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> <20080102152011.GA22106@ssh.cv.nrao.edu> Message-ID: <85b5c3130801021102t4633f82ep9622cbb39ee60fb9@mail.gmail.com> On Jan 2, 2008 4:20 PM, Scott Ransom wrote: > Hmmm. When I try to load: > > http://planet.scipy.org/ > > I get: "Unknown host planet.scipy.org" > > However, http://planet.scipy.org (without the final slash) > resolves fine. Both work for me. Ondrej From stefan at sun.ac.za Wed Jan 2 16:40:59 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 2 Jan 2008 23:40:59 +0200 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: <20080102214059.GG5753@mentat.za.net> Hi Jarrod On Mon, Dec 31, 2007 at 02:43:37PM -0800, Jarrod Millman wrote: > I just wanted to announce that we now have a NumPy/SciPy blog > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ > > Feel free to contact me if you have a blog that you would like > included. Great, thanks! Would you mind putting links to templates like http://planet.scipy.org/rss20.xml somewhere on the page, so that it can be easily found using external RSS readers? Thanks St?fan From stefan at sun.ac.za Wed Jan 2 17:46:51 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 3 Jan 2008 00:46:51 +0200 Subject: [Numpy-discussion] Numpy code coverage Message-ID: <20080102224651.GI5753@mentat.za.net> Hi all, I read about Titus Brown's Figleaf code coverage tool [1] on the Planet SciPy aggregator [2]. The results of running figleaf on the numpy test-suite [3] covers Python code only. What the best way of discovering the C and C++ code coverage as well? Regards St?fan [1] http://darcs.idyll.org/~t/projects/figleaf/README.html [2] http://planet.scipy.org [3] http://mentat.za.net/refer/numpy_figleaf/ From barrywark at gmail.com Wed Jan 2 19:26:24 2008 From: barrywark at gmail.com (Barry Wark) Date: Wed, 2 Jan 2008 16:26:24 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: Jarrod, Sorry. Correct URL: http://physionconsultants.blogspot.com/search/label/scipy Sorry for the mix up. Barry On Jan 1, 2008 6:35 PM, Jarrod Millman wrote: > On Jan 1, 2008 6:18 PM, Barry Wark wrote: > > I'd like to submit my blog for inclusion too... > > > > http://softwareforscientists.blogspot.com/search/label/scipy > > Hmm. I added you, but while I was adding you it looks like your blog > disappeared. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Wed Jan 2 20:23:42 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 2 Jan 2008 17:23:42 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <20080102214059.GG5753@mentat.za.net> References: <20080102214059.GG5753@mentat.za.net> Message-ID: On Jan 2, 2008 1:40 PM, Stefan van der Walt wrote: > Great, thanks! Would you mind putting links to templates like > > http://planet.scipy.org/rss20.xml > > somewhere on the page, so that it can be easily found using external > RSS readers? Done. Let me know if you would like the wording changed or the link to be moved elsewhere. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From fperez.net at gmail.com Wed Jan 2 20:26:24 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jan 2008 18:26:24 -0700 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <20080102214059.GG5753@mentat.za.net> Message-ID: On Jan 2, 2008 6:23 PM, Jarrod Millman wrote: > On Jan 2, 2008 1:40 PM, Stefan van der Walt wrote: > > Great, thanks! Would you mind putting links to templates like > > > > http://planet.scipy.org/rss20.xml > > > > somewhere on the page, so that it can be easily found using external > > RSS readers? > > Done. Let me know if you would like the wording changed or the link > to be moved elsewhere. BTW, now that the planet aggregator seems to be up and running well, it should probably be linked from the main scipy.org page, no? I don't think many of us can edit the main page, so that might be up to one of you guys... Cheers, f From barrywark at gmail.com Wed Jan 2 20:27:25 2008 From: barrywark at gmail.com (Barry Wark) Date: Wed, 2 Jan 2008 17:27:25 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <20080102214059.GG5753@mentat.za.net> Message-ID: Looks great. Thanks for setting this up! barry On Jan 2, 2008 5:23 PM, Jarrod Millman wrote: > On Jan 2, 2008 1:40 PM, Stefan van der Walt wrote: > > Great, thanks! Would you mind putting links to templates like > > > > http://planet.scipy.org/rss20.xml > > > > somewhere on the page, so that it can be easily found using external > > RSS readers? > > Done. Let me know if you would like the wording changed or the link > to be moved elsewhere. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Wed Jan 2 20:54:11 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 2 Jan 2008 17:54:11 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: On Dec 31, 2007 2:43 PM, Jarrod Millman wrote: > I just wanted to announce that we now have a NumPy/SciPy blog > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ http://scipy.com sends me to the planet scipy page. Shouldn't it go to http://scipy.org instead? From millman at berkeley.edu Wed Jan 2 20:58:54 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 2 Jan 2008 17:58:54 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <20080102214059.GG5753@mentat.za.net> Message-ID: On Jan 2, 2008 5:26 PM, Fernando Perez wrote: > BTW, now that the planet aggregator seems to be up and running well, > it should probably be linked from the main scipy.org page, no? I > don't think many of us can edit the main page, so that might be up to > one of you guys... I just added a link to the Blog to the navigation panel. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From gkc1000 at gmail.com Thu Jan 3 10:08:34 2008 From: gkc1000 at gmail.com (Garnet Chan) Date: Thu, 3 Jan 2008 10:08:34 -0500 Subject: [Numpy-discussion] non-intuitive behaviour in numpy.array([list], numpy.object_) Message-ID: <8596294f0801030708u63857a20m5f41b7f924a301f7@mail.gmail.com> When constructing an numpy object array from a list of numpy arrays, one observes the following behaviour >>> import numpy as N >>> a=[N.zeros([2,2], N.object_), N.zeros([2,2], N.object_)] >>> b=N.array(a, N.object_) >>> print b.shape (2, 2, 2) >>> a=[N.zeros([2,2], N.object_), N.zeros([2,1], N.object_)] >>> b=N.array(a, N.object_) >>> print b.shape (2,) I understand that this is because in the 1st instance, numpy is trying to be clever and recognises that the list of arrays "a" can be reshaped into a 3-d array. But surely, if we are constructing an array with the object_ dtype, the first should also be converted into a shape (2,) array? From Chris.Barker at noaa.gov Thu Jan 3 12:50:05 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 03 Jan 2008 09:50:05 -0800 Subject: [Numpy-discussion] non-intuitive behaviour in numpy.array([list], numpy.object_) In-Reply-To: <8596294f0801030708u63857a20m5f41b7f924a301f7@mail.gmail.com> References: <8596294f0801030708u63857a20m5f41b7f924a301f7@mail.gmail.com> Message-ID: <477D204D.1050706@noaa.gov> Garnet Chan wrote: > When constructing an numpy object array from a list of numpy arrays, > one observes the following behaviour > >>>> import numpy as N >>>> a=[N.zeros([2,2], N.object_), N.zeros([2,2], N.object_)] >>>> b=N.array(a, N.object_) >>>> print b.shape > (2, 2, 2) >>>> a=[N.zeros([2,2], N.object_), N.zeros([2,1], N.object_)] >>>> b=N.array(a, N.object_) >>>> print b.shape > (2,) > > I understand that this is because in the 1st instance, numpy is trying > to be clever and recognises that the list of arrays "a" can be > reshaped into a 3-d array. But surely, if we are constructing an array > with the object_ dtype, the first should also be converted into a > shape (2,) array? arrays of the _object dtype are plain weird -- as arbitrary objects can be sequences (and mutable ones at that!), it's impossible to automatically create the shape of arrays of objects that the user wants. Perhaps it could be a bit smarter to be more consistent, but what you really need to do is define the shape for numpy, and not expect it to figure out what you want -- even if it does the right thing with one example. If you want a (2,) array, then do this: >>> b = N.empty(2, dtype=N.object) >>> b[:]=a >>> b.shape (2,) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strawman at astraw.com Thu Jan 3 14:08:02 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 03 Jan 2008 11:08:02 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) Message-ID: <477D3292.80702@astraw.com> Apologies if I've missed the discussion of this, but I was recently surprised by the following behavior (in svn trunk 4673). The following code runs without triggering the assertion. import numpy as np print np.__version__ a=np.int32(42) b=np.array([],dtype=np.int32) assert np.allclose(a,b) Is this expected behavior of numpy or is this a bug I should report? Thanks, Andrew From robert.kern at gmail.com Thu Jan 3 15:06:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 03 Jan 2008 14:06:37 -0600 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <477D3292.80702@astraw.com> References: <477D3292.80702@astraw.com> Message-ID: <477D404D.5060407@gmail.com> Andrew Straw wrote: > Apologies if I've missed the discussion of this, but I was recently > surprised by the following behavior (in svn trunk 4673). The following > code runs without triggering the assertion. > > import numpy as np > print np.__version__ > a=np.int32(42) > b=np.array([],dtype=np.int32) > assert np.allclose(a,b) > > Is this expected behavior of numpy or is this a bug I should report? Bug, I think. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lxander.m at gmail.com Thu Jan 3 15:41:16 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 3 Jan 2008 15:41:16 -0500 Subject: [Numpy-discussion] MaskedArray and Record Arrays Message-ID: <525f23e80801031241o2f82110nbbb774ac6da1cf3d@mail.gmail.com> I am experimenting with the new MaskedArray (from ) as a replacement for my own home-brewed masked data handling mechanisms. In what I have built myself, I often work with record arrays that have a single mask for the whole record (no fieldmask). It seems like it is almost possible to do this with MaskedArray: >>> a = masked_array( ... [(1,10,100),(0,0,0),(2,20,200)], ... mask=[False,True,False], ... dtype=[('x',float), ('y',float), ('c',int)] ... ) masked_array(data = [(1.0, 10.0, 100) -- (2.0, 20.0, 200)], mask = [False True False], fill_value=???) except MaskedArray.__getitem__ doesn't check to see if I'm asking for a field instead of an index: >>> a['x'] Traceback (most recent call last): File "", line 1, in File "qlab\ma\core.py", line 1269, in __getitem__ dout._mask = ndarray.__getitem__(m, indx).reshape(dout.shape) ValueError: field named x not found. modifying ma.core.py so that def __getitem__(self, indx): ... if m is not nomask: if not isinstance(indx, basestring): dout._mask = ndarray.__getitem__(m, indx).reshape(dout.shape) ... gets me a little father. Any plans to make this style of record array usage work with the new MaskedArray? I saw and experimented with mrecords, but I am concerned about the performance overhead as I do not require the granularity of fieldmask. Nevertheless, I will continue with mrecords since it does what I want (and it is just performance that I am worried about) and see how it works for me, but I thought I would toss this other usage style out there because it might be easy. Thanks, Alex From lxander.m at gmail.com Thu Jan 3 15:49:45 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 3 Jan 2008 15:49:45 -0500 Subject: [Numpy-discussion] MaskedArray and the min,max,sum,prod Methods Message-ID: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> Working with the new MaskedArray, I noticed the following differences with numpy.array behavior: >>> masked_array([1, 2, 3], mask=True).min() 2147483647 >>> array([]).min() Traceback (most recent call last): File "", line 1, in ValueError: zero-size array to ufunc.reduce without identity >>> masked_array([1, 2, 3], mask=True).max() -2147483648 >>> array([]).max() Traceback (most recent call last): File "", line 1, in ValueError: zero-size array to ufunc.reduce without identity >>> masked_array([1, 2, 3], mask=True).prod() masked_array(data = --, mask = True, fill_value=1e+020) >>> array([]).prod() 1.0 >>> masked_array([1, 2, 3], mask=True).sum() masked_array(data = --, mask = True, fill_value=1e+020) >>> numpy.array([]).sum() 0.0 Are these corner cases intentionally different? Thanks, Alex From charlesr.harris at gmail.com Thu Jan 3 15:57:31 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 3 Jan 2008 13:57:31 -0700 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <477D404D.5060407@gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> Message-ID: On Jan 3, 2008 1:06 PM, Robert Kern wrote: > Andrew Straw wrote: > > Apologies if I've missed the discussion of this, but I was recently > > surprised by the following behavior (in svn trunk 4673). The following > > code runs without triggering the assertion. > > > > import numpy as np > > print np.__version__ > > a=np.int32(42) > > b=np.array([],dtype=np.int32) > > assert np.allclose(a,b) > > > > Is this expected behavior of numpy or is this a bug I should report? > > Bug, I think. Isn't it trivially true that all elements of an empty array are close to any number? For instance, it is a well known fact that all blue, cheese eating martians speak esperanto. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gkc1000 at gmail.com Thu Jan 3 15:58:25 2008 From: gkc1000 at gmail.com (Garnet Chan) Date: Thu, 3 Jan 2008 15:58:25 -0500 Subject: [Numpy-discussion] non-intuitive behaviour in numpy.array([list], numpy.object_) In-Reply-To: <477D204D.1050706@noaa.gov> References: <8596294f0801030708u63857a20m5f41b7f924a301f7@mail.gmail.com> <477D204D.1050706@noaa.gov> Message-ID: <8596294f0801031258y5b17fb70ydb3e6e9a9f3cb656@mail.gmail.com> Thanks - that's clear I guess, although I still think that it might be less confusing if numpy did not try to be clever! On 1/3/08, Christopher Barker wrote: > Garnet Chan wrote: > > When constructing an numpy object array from a list of numpy arrays, > > one observes the following behaviour > > > >>>> import numpy as N > >>>> a=[N.zeros([2,2], N.object_), N.zeros([2,2], N.object_)] > >>>> b=N.array(a, N.object_) > >>>> print b.shape > > (2, 2, 2) > >>>> a=[N.zeros([2,2], N.object_), N.zeros([2,1], N.object_)] > >>>> b=N.array(a, N.object_) > >>>> print b.shape > > (2,) > > > > I understand that this is because in the 1st instance, numpy is trying > > to be clever and recognises that the list of arrays "a" can be > > reshaped into a 3-d array. But surely, if we are constructing an array > > with the object_ dtype, the first should also be converted into a > > shape (2,) array? > > arrays of the _object dtype are plain weird -- as arbitrary objects can > be sequences (and mutable ones at that!), it's impossible to > automatically create the shape of arrays of objects that the user wants. > Perhaps it could be a bit smarter to be more consistent, but what you > really need to do is define the shape for numpy, and not expect it to > figure out what you want -- even if it does the right thing with one > example. > > If you want a (2,) array, then do this: > > >>> b = N.empty(2, dtype=N.object) > >>> b[:]=a > >>> b.shape > (2,) > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From aisaac at american.edu Thu Jan 3 16:20:38 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 3 Jan 2008 16:20:38 -0500 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> Message-ID: On Thu, 3 Jan 2008, Charles R Harris apparently wrote: > Isn't it trivially true that all elements of an empty > array are close to any number? Sure, but might not one expect a ValueError due to shape mismatch? (Doesn't allclose usually use normal broadcasting rules?) Cheers, Alan Isaac From matthew.brett at gmail.com Thu Jan 3 16:22:12 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 3 Jan 2008 13:22:12 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> Message-ID: <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> Hi, > > > import numpy as np > > > print np.__version__ > > > a=np.int32(42) > > > b=np.array([],dtype=np.int32) > > > assert np.allclose(a,b) > > > > > > Is this expected behavior of numpy or is this a bug I should report? > > > > Bug, I think. I think this bug - which may be mine - follows from this line in allclose: return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) and: In [39]: all([]) Out[39]: True Matthew From aisaac at american.edu Thu Jan 3 16:31:14 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 3 Jan 2008 16:31:14 -0500 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> Message-ID: > On Thu, 3 Jan 2008, Charles R Harris apparently wrote: >> Isn't it trivially true that all elements of an empty >> array are close to any number? On Thu, 3 Jan 2008, Alan G Isaac apparently wrote: > Sure, but might not one expect a ValueError due to > shape mismatch? (Doesn't allclose usually use > normal broadcasting rules?) Ooops, forgot that was a "scalar", so it was "normal": >>> a*b array([], dtype=int32) Cheers, Alan Isaac From matthew.brett at gmail.com Thu Jan 3 16:37:34 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 3 Jan 2008 13:37:34 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> Message-ID: <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> Just to ask - is there a reason why this: > In [39]: all([]) > Out[39]: True is the case? From charlesr.harris at gmail.com Thu Jan 3 17:25:57 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 3 Jan 2008 15:25:57 -0700 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> Message-ID: On Jan 3, 2008 2:37 PM, Matthew Brett wrote: > Just to ask - is there a reason why this: > > > In [39]: all([]) > > Out[39]: True > > is the case? Because it's True. Anything is true about the elements of an empty set, because there aren't any. In this case, all asks if all elements in [] are true, i.e., does x member [] -> x is true. Since x member [] is always false, the implication is always true. Recall that the statement x -> y has the same truth value as the statement x' or xy. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jan 3 17:37:56 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 3 Jan 2008 14:37:56 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> Message-ID: <1e2af89e0801031437me945b0ejd25d06a4b7ec3d8f@mail.gmail.com> So, currently we have all and allclose giving the same answer: In [19]: a = array([]) In [20]: b = array([1]) In [21]: all(a == b) Out[21]: True In [22]: allclose(a, b) Out[22]: True Would we want the answers to be different? From ondrej at certik.cz Thu Jan 3 17:51:19 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 3 Jan 2008 23:51:19 +0100 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <85b5c3130801021102t4633f82ep9622cbb39ee60fb9@mail.gmail.com> References: <85b5c3130801020453r771a0ac9mbecd2cf260b81ad6@mail.gmail.com> <000c01c84d4e$22f5a190$0a83a8c0@sun.ac.za> <20080102152011.GA22106@ssh.cv.nrao.edu> <85b5c3130801021102t4633f82ep9622cbb39ee60fb9@mail.gmail.com> Message-ID: <85b5c3130801031451n16849b42v8dbdd92f7821ef1e@mail.gmail.com> On Jan 2, 2008 8:02 PM, Ondrej Certik wrote: > On Jan 2, 2008 4:20 PM, Scott Ransom wrote: > > Hmmm. When I try to load: > > > > http://planet.scipy.org/ > > > > I get: "Unknown host planet.scipy.org" > > > > However, http://planet.scipy.org (without the final slash) > > resolves fine. > > Both work for me. Something is wrong with the dns, because it stopped working for me from the Czech Republic. But it works from USA. So hopefully tomorrow it'd be ok. Ondrej > > Ondrej > From robert.kern at gmail.com Thu Jan 3 17:56:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 03 Jan 2008 16:56:15 -0600 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <1e2af89e0801031437me945b0ejd25d06a4b7ec3d8f@mail.gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> <1e2af89e0801031437me945b0ejd25d06a4b7ec3d8f@mail.gmail.com> Message-ID: <477D680F.8000607@gmail.com> Matthew Brett wrote: > So, currently we have all and allclose giving the same answer: > > In [19]: a = array([]) > > In [20]: b = array([1]) > > In [21]: all(a == b) > Out[21]: True > > In [22]: allclose(a, b) > Out[22]: True > > Would we want the answers to be different? No. I wasn't thinking correctly, previously. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Jan 3 18:02:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 03 Jan 2008 17:02:44 -0600 Subject: [Numpy-discussion] Numpy code coverage In-Reply-To: <20080102224651.GI5753@mentat.za.net> References: <20080102224651.GI5753@mentat.za.net> Message-ID: <477D6994.1080800@gmail.com> Stefan van der Walt wrote: > Hi all, > > I read about Titus Brown's Figleaf code coverage tool [1] on the > Planet SciPy aggregator [2]. The results of running figleaf on the > numpy test-suite [3] covers Python code only. > > What the best way of discovering the C and C++ code coverage as well? I've never encountered any documents about checking C code coverage of Python extension modules. It is possible that the standard tools for C code coverage (e.g. gcov) would work fine with some finagling. If you try this, I would love to hear what you did. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthew.brett at gmail.com Thu Jan 3 18:04:52 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 3 Jan 2008 15:04:52 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <477D680F.8000607@gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> <1e2af89e0801031437me945b0ejd25d06a4b7ec3d8f@mail.gmail.com> <477D680F.8000607@gmail.com> Message-ID: <1e2af89e0801031504v692751a7q15d7788400885ed0@mail.gmail.com> > > So, currently we have all and allclose giving the same answer: > > > > In [19]: a = array([]) > > > > In [20]: b = array([1]) > > > > In [21]: all(a == b) > > Out[21]: True > > > > In [22]: allclose(a, b) > > Out[22]: True > > > > Would we want the answers to be different? > > No. I wasn't thinking correctly, previously. It's unfortunate that all of us immediately think the given answer is wrong. From lists.steve at arachnedesign.net Thu Jan 3 18:29:03 2008 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 3 Jan 2008 18:29:03 -0500 Subject: [Numpy-discussion] Error importing from numpy.matlib Message-ID: Hi all, I can't figure out why this is happening ... I just recently recompiled numpy/scipy from svn just for the heck of it. Anyway, somewhere in my codebase (for a long time now) I'm doing: from numpy.matlib import * Now, when I try to use this code, or just type that in the interpreter, I get this message: AttributeError: 'module' object has no attribute 'pkgload' This doesn't happen when I do: import numpy.matlib as M Anyway, can anyone shed some light on this? Thanks, -steve From strawman at astraw.com Thu Jan 3 18:35:17 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 03 Jan 2008 15:35:17 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <1e2af89e0801031504v692751a7q15d7788400885ed0@mail.gmail.com> References: <477D3292.80702@astraw.com> <477D404D.5060407@gmail.com> <1e2af89e0801031322l6a251d50pa6c0c1ab1970c774@mail.gmail.com> <1e2af89e0801031337x49e6e8eak13beeb6ab55ac984@mail.gmail.com> <1e2af89e0801031437me945b0ejd25d06a4b7ec3d8f@mail.gmail.com> <477D680F.8000607@gmail.com> <1e2af89e0801031504v692751a7q15d7788400885ed0@mail.gmail.com> Message-ID: <477D7135.9040502@astraw.com> Matthew Brett wrote: >>> So, currently we have all and allclose giving the same answer: >>> >>> In [19]: a = array([]) >>> >>> In [20]: b = array([1]) >>> >>> In [21]: all(a == b) >>> Out[21]: True >>> >>> In [22]: allclose(a, b) >>> Out[22]: True >>> >>> Would we want the answers to be different? >>> >> No. I wasn't thinking correctly, previously. >> > > It's unfortunate that all of us immediately think the given answer is wrong. Maybe we need "allclose_sameshape()" (my ability to name stuff is terrible, but you get the idea). Regardless of the name issue, I have no idea how this is viewed against the no-namespace-bloat principle. From myeates at jpl.nasa.gov Thu Jan 3 19:16:39 2008 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 03 Jan 2008 16:16:39 -0800 Subject: [Numpy-discussion] weird indexing Message-ID: <477D7AE7.6000105@jpl.nasa.gov> Hi Okay, here's a weird one. In Fortran you can specify the upper/lower bounds of an array e.g. REAL A(3:7) What would be the best way to translate this to a Numpy array? I would like to do something like A=numpy.zeros(shape=(5,)) and have the expression A[3] actually return A[0]. Or something. Any thoughts? Mathew From david at ar.media.kyoto-u.ac.jp Fri Jan 4 02:59:59 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 16:59:59 +0900 Subject: [Numpy-discussion] [OT] Which version of svn does svn.scipy.org run ? In-Reply-To: References: <4767756F.20507@ar.media.kyoto-u.ac.jp> Message-ID: <477DE77F.8020002@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > On Dec 17, 2007 11:23 PM, David Cournapeau wrote: > >> Is it possible to know which version of subversion is used by the >> svn.scipy.org server ? I am trying to use svnsync on it, without any >> success, and wanted to know if this was coming from svn.scipy.org or >> from something else ? >> > > It is running subversion 1.2.3-2.1. In the near future, the server > will be replaced with newer hardware and updated software. > > Thanks for the information. Would it be possible to have a public announcement at this time ? David From cournape at gmail.com Fri Jan 4 05:01:41 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 4 Jan 2008 19:01:41 +0900 Subject: [Numpy-discussion] Towards using scons in numpy: 1st step In-Reply-To: <5b8d13220712201704v1e4d7acboee36c05c7d46f70a@mail.gmail.com> References: <5b8d13220712180221r2da5e9d9t645eeec21914d51e@mail.gmail.com> <4769C56D.3000809@enthought.com> <5b8d13220712201704v1e4d7acboee36c05c7d46f70a@mail.gmail.com> Message-ID: <5b8d13220801040201q512f99f8w933bc5201fde36ce@mail.gmail.com> On Dec 21, 2007 10:04 AM, David Cournapeau wrote: > On Dec 20, 2007 10:29 AM, Travis E. Oliphant wrote: > > How did you get around using #ifdef etc? This concerns me to just > > change it, unless I'm convinced there is no problem. > I don't get around them, I just do not use them inside generated > files. For example, NPY_ALLOW_THREADS is defined in ndarrayobject.h, > now, depending on some values defined in the config headers. Having > #ifdef inside generated files is a bit against the spirit of config > headers, and it makes also more complicated to generate them through > standard tools (neither scons nor autoheader supports this, for > example). Travis, did you get any chance to look at it ? If the goal is to merge the changes needed for scons support for 1.0.5, I would prefer having a few days between the merge and the release to take care about problems cheers, David From haase at msg.ucsf.edu Fri Jan 4 05:31:31 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 4 Jan 2008 11:31:31 +0100 Subject: [Numpy-discussion] weird indexing In-Reply-To: <477D7AE7.6000105@jpl.nasa.gov> References: <477D7AE7.6000105@jpl.nasa.gov> Message-ID: On Jan 4, 2008 1:16 AM, Mathew Yeates wrote: > Hi > Okay, here's a weird one. In Fortran you can specify the upper/lower > bounds of an array > e.g. REAL A(3:7) > > What would be the best way to translate this to a Numpy array? I would > like to do something like > A=numpy.zeros(shape=(5,)) > and have the expression A[3] actually return A[0]. > > Or something. Any thoughts? > As far as I know, numpy only supports the lower boundary to be 0 -- just like C. (I didn't even know that Fortran supported this ;-) ) -Sebastian Haase From gael.varoquaux at normalesup.org Fri Jan 4 05:33:22 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Jan 2008 11:33:22 +0100 Subject: [Numpy-discussion] weird indexing In-Reply-To: <477D7AE7.6000105@jpl.nasa.gov> References: <477D7AE7.6000105@jpl.nasa.gov> Message-ID: <20080104103322.GB10801@clipper.ens.fr> On Thu, Jan 03, 2008 at 04:16:39PM -0800, Mathew Yeates wrote: > Hi > Okay, here's a weird one. In Fortran you can specify the upper/lower > bounds of an array > e.g. REAL A(3:7) > What would be the best way to translate this to a Numpy array? I would > like to do something like > A=numpy.zeros(shape=(5,)) > and have the expression A[3] actually return A[0]. > Or something. Any thoughts? Create a proxy object with an overriden __getitem__ method? That seems quite ugly to me. Ga?l From robert.kern at gmail.com Fri Jan 4 05:36:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 04:36:17 -0600 Subject: [Numpy-discussion] weird indexing In-Reply-To: <477D7AE7.6000105@jpl.nasa.gov> References: <477D7AE7.6000105@jpl.nasa.gov> Message-ID: <477E0C21.7070300@gmail.com> Mathew Yeates wrote: > Hi > Okay, here's a weird one. In Fortran you can specify the upper/lower > bounds of an array > e.g. REAL A(3:7) > > What would be the best way to translate this to a Numpy array? I would > like to do something like > A=numpy.zeros(shape=(5,)) > and have the expression A[3] actually return A[0]. > > Or something. Any thoughts? The best way would depend on what you want to do with it. Why do you need this? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pearu at cens.ioc.ee Fri Jan 4 05:37:39 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 12:37:39 +0200 (EET) Subject: [Numpy-discussion] weird indexing In-Reply-To: <477D7AE7.6000105@jpl.nasa.gov> References: <477D7AE7.6000105@jpl.nasa.gov> Message-ID: <50770.85.166.31.187.1199443059.squirrel@cens.ioc.ee> On Fri, January 4, 2008 2:16 am, Mathew Yeates wrote: > Hi > Okay, here's a weird one. In Fortran you can specify the upper/lower > bounds of an array > e.g. REAL A(3:7) > > What would be the best way to translate this to a Numpy array? I would > like to do something like > A=numpy.zeros(shape=(5,)) > and have the expression A[3] actually return A[0]. > > Or something. Any thoughts? f2py wraps A(3:7) to numpy array A[0:5] and keeps it that way as it is most natural for working in Python side. Pearu From david at ar.media.kyoto-u.ac.jp Fri Jan 4 06:24:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 20:24:04 +0900 Subject: [Numpy-discussion] Moving away from svn ? Message-ID: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Hi, First things first, happy new year to all ! Having recently felt the pain to use subversion merge, I was wondering about people's feeling on moving away from subversion and using a better system, ala mercurial or bzr (I will talk about bzr because that's the one I know the most, but this discussion is really about using something better than subversion, not that much about bzr). I think this could be an important step forward, and is somewhat related to the discusions on scikits and co. As some of you are certainly aware, there has been a recent trend towards so called Distributed Version Control Systems (DVCS). I won't go into the details, because it varies from system to system, and I am in no position to explain technical details. But for people who are wondering, here is a small description of DVCS, and why I think this can be a significant step forward for numpy/scipy. You can skip it if you know about them What is a DVCS ============== DVCS, contrary to centralized systems like CVS or SVN, has no technical concept of a central repository from which everybodoy pulls and push changes. Instead, the DVCS are centered around the branch concept, which contains a local copy of the history. As a consequence: 1 you can do most traditionnal svn/cvs operations locally, and disconnected from the network (getting the log, getting annotations, commiting, branching, merging from other branches). 2 because the branch is local, no rights is needed: anybody can jump-in, commit to a local branch. Of course, integration in an official numpy branch would need some special approval. Also, this has the following consequence: since branching/merging is such a key point of DVCS, merging actually works with DVCS. In perticular, merging several times the same changes works, and you certainly do not have to do the whole svn madness of tracking versions. For more informations, here are some links which go much deeper: - some discussion from K. Richard, the maintained of X.org: http://keithp.com/blogs/Tyrannical_SCM_selection/, http://keithp.com/blogs/Repository_Formats_Matter/ - Linus Torvald on the advantages of git, the DVCS he wrote for linux developement, versus svn for kde (long but it really makes all the points really clearly): http://lists.kde.org/?l=kde-core-devel&m=118764715705846&w=2 Why using a DVCS ? ================== Some people argue that DVCS are intrinsically more complicated, which is something I really don't understand. I've been programming 'seriously' for only about 2-3 years, and I find bzr much easier to use and setting up than subversion; the key point I think is that I started using DVCS before centralized ones. Some things which are utterly complicated with subversion and are trivial with bzr: merging, going back into the history (that is at rev 150, you realize that everything from rev 140 is rubbish, and you want to go back: this is extremely tedious to do with subversion). Basically, most of the things which are the reasons why we use VCS in the first place are easier with DVCS than VCS (at least as far as svn is concerned). Also: - For a casual user who wants to use the last development instead of a release, getting it from a bzr repository, a git repository, a mercurial repository or a svn repository is extremely similar. It is one step in all cases. - For casual developers: being able to use branches means that they can implements their new features in a change-set oriented way instead of one big patch. Also, bzr enables things like uncommit if you made a mistake and wants to go back. More generally, going back in history is much easier. - For core developers: I personally find the ability to use branches for each new feature to be extremely useful. It makes me feel much safer when I do something. I am not afraid of doing something totally stupid which may end up screwing other people. And finally, I find the ability to do things locally to be really pleasant and it enables workflows not really possible with systems such as SVN. In particular, I work at three distant places every week, and the ability to work in the transportation, and the trivial synchronization between computers is definitely helpful. Instantaneous log and annotations is also really useful IMHO. Which DVCS ? ============ The 3 ones which keeps coming up are: - git (the one used for linux kernel development). That's the one I know the least (only from a user point of view, never used it for developement). It is supposed to be more powerful, more complicated than the others. It is also known to be really fast (the kernel is not a small codebase for sure). - mercurial: started at the same time than git. Is written in python except for a few things written in C. It is reasonably fast, and has been recently selected for some bigs projects, in perticular by Sun (openJDK, openSolaris, open Netbeans). - bzr: also written in python. Sponsored by Cannonical, the company between Ubuntu. It has just reached the 1.0 version. The focus is on the UI; handles renaming really well. It has a vibrant community, with dedicated developers working on it; it has the reputation of being slow, which was somewhat true previously, but in my experience, it is on par with mercurial, at least for local operations. Anyway, it is not a problem for numpy or scipy, which are small codebases (a few thousand of files, a few thousand revisions). Problems: ========= Assuming people think it worths being tried out, I mainly see two problems: - importing the current history - integration with trac For bzr, I can say that the bzr-svn plugin works really well; in perticular, it can import numpy and scipy repositories with the whole history, I am using it regurlarly as a proxy between local bzr and the scipy and scikits trunk. Incidentally, this makes it possible for me to give numbers if numbers are needed wrt bzr's speed, repository size, etc... For mercurial, I tried one method once which did not go really far, but I did not try really hard; anyway, I think people at enthought use mercurial a lot, so they would know better. Integration with trac is the real problem, I think. According to one bzr developer, trac model (0.10, the last released one) is really based around subversion notion of repository, which does not fit well with mercurial and bzr. I don't know if this is true for the not yet released 0.11. If bzr is considered a possible candidate, I can get more informations from bzr developers. What is the experience wrt trac from enthought developers ? This email is already getting pretty long, so to conclude, I think DVCS would be helpful for future development of numpy/scipy. I believe it would both enable easier participation from different people, enabling safer developement schedules, etc... What do other people think ? Would it be worthwhile to discuss further around the issues and how to resolve things ? cheers, David P.S: I would be willing to take care about the bzr side of things: trying conversion, setting up experimental repositories for trial, and asking advices to the bzr community. From dmitrey.kroshko at scipy.org Fri Jan 4 06:48:49 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 04 Jan 2008 13:48:49 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <477E1D21.1010203@scipy.org> As for me, I would wait until DVCS became more popular than svn. Jump often from one VSC to another isn't a good idea, moreover, it's not clear for now which DVCS will suppress others and became standard (being installed in many OS by default). Also, I would prefer (for example my openopt) changes being available 24 hours/day immediately after I have commit them; also, keeping them too long only in my HDD makes data more vulnerable - computer viruses, often electricity drops, other possible causes to lose data. // my 2 cents Regards, D. David Cournapeau wrote: > Hi, > From ndbecker2 at gmail.com Fri Jan 4 06:56:51 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 04 Jan 2008 06:56:51 -0500 Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> Message-ID: dmitrey wrote: > As for me, I would wait until DVCS became more popular than svn. Jump > often from one VSC to another isn't a good idea, moreover, it's not > clear for now which DVCS will suppress others and became standard (being > installed in many OS by default). > > Also, I would prefer (for example my openopt) changes being available 24 > hours/day immediately after I have commit them; also, keeping them too > long only in my HDD makes data more vulnerable - computer viruses, often > electricity drops, other possible causes to lose data. > // my 2 cents > You misunderstand dvcs. There is no problem to maintain a centralized copy, I believe all the well known projects that have adopted dvcs all maintain a canonical centralized copy. From ndbecker2 at gmail.com Fri Jan 4 06:57:29 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 04 Jan 2008 06:57:29 -0500 Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: There is a mercurial plugin for trac. From david at ar.media.kyoto-u.ac.jp Fri Jan 4 06:54:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 20:54:13 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1D21.1010203@scipy.org> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> Message-ID: <477E1E65.3040604@ar.media.kyoto-u.ac.jp> dmitrey wrote: > As for me, I would wait until DVCS became more popular than svn. Jump > often from one VSC to another isn't a good idea, moreover, it's not > clear for now which DVCS will suppress others and became standard (being > installed in many OS by default). I don't think one will become standard. Git will stay for sure, since it is used and developers by kernel hackers; it is used by at least two big open source projects: linux and xorg (as well as many freedesktop projects). bzr is pushed really hard by Canonical, and I don't think Canonical will be going away soon. Mercurial is used by Sun for all its open sourced projects. I don't see why this would be a problem. On Linux, getting open source softwares is trivial, and windows has never been distributed with any VCS system :) Only mac os X has svn by default (if you install the dev tools at least). But as long as binary installers are available, I don't see that as a big problem either. Now, the points you raised (concerning the popularity) have direct consequences on the availability of third party tools, which certainly is a problem to consider (GUI, etc...). As far as bzr is concerned, I would say that's the core problem (Gui on windows, integration with trac). > > Also, I would prefer (for example my openopt) changes being available 24 > hours/day immediately after I have commit them; also, keeping them too > long only in my HDD makes data more vulnerable - computer viruses, often > electricity drops, other possible causes to lose data. Nothing prevents you from putting the changes on a backup server: for example, bzr supports the concept of sending any commited changed to a 'bound branch' automatically, to have a more svn-like workflow. I certainly agree that changing the VCS is a big change, and requires a lot of thinking, though. I am not suggesting to change for the next week. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Jan 4 06:56:31 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 20:56:31 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <477E1EEF.8000302@ar.media.kyoto-u.ac.jp> Neal Becker wrote: > There is a mercurial plugin for trac. > as well as a bzr one. The problem is more related to performance issues (cheap things in svn are not cheap in DVCS, and vice-versa). For example, the trac-bzr plugin is really slow for timelines (it takes almost one second on a local server !); I have not tried the trac-mercurial one much. cheers, David From gael.varoquaux at normalesup.org Fri Jan 4 07:11:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Jan 2008 13:11:59 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1E65.3040604@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> Message-ID: <20080104121159.GA13665@clipper.ens.fr> On Fri, Jan 04, 2008 at 08:54:13PM +0900, David Cournapeau wrote: > I certainly agree that changing the VCS is a big change, and requires a > lot of thinking, though. I am not suggesting to change for the next week. In the mean time, do you want to tell us more about how you use bzr with svn. This seems like a good transitory option. Ga?l From ndbecker2 at gmail.com Fri Jan 4 07:17:53 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 04 Jan 2008 07:17:53 -0500 Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> Message-ID: Gael Varoquaux wrote: > On Fri, Jan 04, 2008 at 08:54:13PM +0900, David Cournapeau wrote: >> I certainly agree that changing the VCS is a big change, and requires a >> lot of thinking, though. I am not suggesting to change for the next week. > > In the mean time, do you want to tell us more about how you use bzr with > svn. This seems like a good transitory option. > > Ga?l Mercurial has a very powerful extension called 'convert'. http://www.selenic.com/mercurial/wiki/index.cgi/ConvertExtension From ondrej at certik.cz Fri Jan 4 07:25:55 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 4 Jan 2008 13:25:55 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1EEF.8000302@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1EEF.8000302@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130801040425n3a8dea4ft60d868b4c7a87b16@mail.gmail.com> On Jan 4, 2008 12:56 PM, David Cournapeau wrote: > Neal Becker wrote: > > There is a mercurial plugin for trac. > > > as well as a bzr one. The problem is more related to performance issues > (cheap things in svn are not cheap in DVCS, and vice-versa). For > example, the trac-bzr plugin is really slow for timelines (it takes > almost one second on a local server !); I have not tried the > trac-mercurial one much. We switched from svn to Mercurial in SymPy, I wrote some info here: http://code.google.com/p/sympy/wiki/Mercurial But basically, once you try "svn merge" and you go through all the pain and then try DVCS (I only use mercurial and git), you never want to come back. Our central repo is here: http://hg.sympy.org/sympy/ and I can just fully recommend. We converted all our svn history to it, so now, I frequently browse the history of sympy (because every clone of the repo has it) if I need to look at something. I never used that with svn, because it's painfully slow. We were basically only deciding between git and Mercurial, but we chose mercurial, because * we are python guys and Mercurial is in python+C, very nicely written and they accept patches (Kirill, one sympy developer, has posted several already, to implement features he was missing - he used to use darcs before) * Sage uses it Ondrej From david at ar.media.kyoto-u.ac.jp Fri Jan 4 07:21:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 21:21:21 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <20080104121159.GA13665@clipper.ens.fr> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> Message-ID: <477E24C1.80505@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > On Fri, Jan 04, 2008 at 08:54:13PM +0900, David Cournapeau wrote: >> I certainly agree that changing the VCS is a big change, and requires a >> lot of thinking, though. I am not suggesting to change for the next week. > > In the mean time, do you want to tell us more about how you use bzr with > svn. This seems like a good transitory option. Once you installed bzr-svn, you can import the whole scikits trunk using the svn-import command. This will create a shared repository (that is no working tree is actually put in scikits.bzr: this means that having the whole repository is quite cheap storage-wise; as an example, the whole scipy history takes around 68 Mb, that is less than a svn checkout with the working tree). Once you have the shared repository, you do not care anymore that it is imported from svn. It is exactly the same worklow as a native bzr shared repository. The problems: - importing is really slow, because you need to get the history per revision; this is a network-bound operation (for local svn mirrors, on a macbook, I can import around 15 revisions/second for matplotlib, at which point it becomes CPU bound). That's why I asked about svn server informations, to be able to use svnsync (which makes a local mirror of a svn repository) for thorough experiments on my own. - because of the above, it would be really bad if many people start to import directly, because of the burden on the svn server. - bzr-svn uses a different format than usual bzr. UI wise, it does not change anything, but this means it is less performant than the format used since bzr 0.92 (it is an over-simplification, because you can have a better format). - It does not work right now with numpy because I made a mistake + a bzr-svn bug, which should be easily solved, though. It works with scipy and scikits. David From david at ar.media.kyoto-u.ac.jp Fri Jan 4 07:47:33 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Jan 2008 21:47:33 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <85b5c3130801040425n3a8dea4ft60d868b4c7a87b16@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1EEF.8000302@ar.media.kyoto-u.ac.jp> <85b5c3130801040425n3a8dea4ft60d868b4c7a87b16@mail.gmail.com> Message-ID: <477E2AE5.4020401@ar.media.kyoto-u.ac.jp> Ondrej Certik wrote: > On Jan 4, 2008 12:56 PM, David Cournapeau wrote: >> Neal Becker wrote: >>> There is a mercurial plugin for trac. >>> >> as well as a bzr one. The problem is more related to performance issues >> (cheap things in svn are not cheap in DVCS, and vice-versa). For >> example, the trac-bzr plugin is really slow for timelines (it takes >> almost one second on a local server !); I have not tried the >> trac-mercurial one much. > > > We switched from svn to Mercurial in SymPy, I wrote some info here: > > http://code.google.com/p/sympy/wiki/Mercurial > > But basically, once you try "svn merge" and you go through all the pain > and then try DVCS (I only use mercurial and git), you never want to come back. Imagine the pain in the other direction, which was my experience :) I actually did not believe at first that it was so bad, and thought I was doing something wrong. At least, it certainly convinced me that SVN was not easier than DVCS. > > Our central repo is here: > > http://hg.sympy.org/sympy/ > > and I can just fully recommend. We converted all our svn history to > it, so now, I frequently > browse the history of sympy (because every clone of the repo has it) > if I need to look at something. I never used that with svn, because > it's painfully slow. I am not familiar with sympy: you are not using trac at all ? Also, how did you convert the svn history ? I like the mercurial's way of showing branches and co; bzr does not have anything like that out of the box (there are separate projects to show sources; there is also launchpad of course, but since it is not open source, I do not even consider it for numpy/scipy). On the other hand, the bzr community is more user-friendly: the tool is easier to use I think, the graphical tools are more advanced, at least from my experience. > > We were basically only deciding between git and Mercurial, but we > chose mercurial, because > > * we are python guys and Mercurial is in python+C, very nicely written > and they accept patches (Kirill, one sympy developer, has posted > several already, to implement features he was missing - he used to use > darcs before) > * Sage uses it For some time, the big problem of bzr was speed. But bzr accomplished quite a lot the last year: the first time I used mercurial, the speed difference was obvious; it is not so true anymore (they 'feel' the same, basically, but I have not used mercurial extensively, at least compared to bzr). So I think it really boils down to the difficulty of the transition, the availability of third party tools (and also the tool used by other projects similar to numpy, as you mentionned). cheers, David From ndbecker2 at gmail.com Fri Jan 4 09:05:45 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 04 Jan 2008 09:05:45 -0500 Subject: [Numpy-discussion] PyArray_FromAny does not accept a generator? Message-ID: It seems that PyArray_FromAny does not accept a generator? Seems like this would be useful. From ondrej at certik.cz Fri Jan 4 09:16:10 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 4 Jan 2008 15:16:10 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E2AE5.4020401@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1EEF.8000302@ar.media.kyoto-u.ac.jp> <85b5c3130801040425n3a8dea4ft60d868b4c7a87b16@mail.gmail.com> <477E2AE5.4020401@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130801040616j5a54d640ta4fe5abac917fd3d@mail.gmail.com> > Imagine the pain in the other direction, which was my experience :) I > actually did not believe at first that it was so bad, and thought I was > doing something wrong. At least, it certainly convinced me that SVN was > not easier than DVCS. It would made me sick. :) > I am not familiar with sympy: you are not using trac at all ? Also, how We use googlecode: http://code.google.com/p/sympy/ it works nice for us. > did you convert the svn history ? using the mercurial extension. Kirill submitted some patches, so that also branches are converted and tags too. > I like the mercurial's way of showing branches and co; bzr does not have > anything like that out of the box (there are separate projects to show > sources; there is also launchpad of course, but since it is not open > source, I do not even consider it for numpy/scipy). > > On the other hand, the bzr community is more user-friendly: the tool is > easier to use I think, the graphical tools are more advanced, at least > from my experience. I never used bzr, so I cannot judge. > > We were basically only deciding between git and Mercurial, but we > > chose mercurial, because > > > > * we are python guys and Mercurial is in python+C, very nicely written > > and they accept patches (Kirill, one sympy developer, has posted > > several already, to implement features he was missing - he used to use > > darcs before) > > * Sage uses it > For some time, the big problem of bzr was speed. But bzr accomplished > quite a lot the last year: the first time I used mercurial, the speed > difference was obvious; it is not so true anymore (they 'feel' the same, > basically, but I have not used mercurial extensively, at least compared > to bzr). > > So I think it really boils down to the difficulty of the transition, the > availability of third party tools (and also the tool used by other > projects similar to numpy, as you mentionned). I know that bzr is used by Canonical, but I think socially, you should choose either mercurial or git. Those are imho the most widespread DVCS. As I said, Mercurial being Python+C was very important for us, so that we can easily fix bugs and implement new functionality in mercurial. Also the commands of mercurial are very similar to svn. Ondrej From matthieu.brucher at gmail.com Fri Jan 4 09:26:52 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 4 Jan 2008 15:26:52 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E24C1.80505@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> <477E24C1.80505@ar.media.kyoto-u.ac.jp> Message-ID: > > > In the mean time, do you want to tell us more about how you use bzr with > > svn. This seems like a good transitory option. > Once you installed bzr-svn, you can import the whole scikits trunk using > the svn-import command. This works OK for Linux, but for Windows, the packages needed by bzr-svn (the python wrappers that are in the usual python-subversion package) are in the Subversion trunk (1.5). So we have to compile them first. Beside this, I'm starting to use bazaar (in fact it's the successor of arch) for a small project of mine hosted on launchpad.net, and it works great. As David stated, the only problem is the UI : on Linux, I'm using mainly the command line because olive-gtk is buggy and not really user friendly (there is room for improvement, it has even deprecation warnings because it uses a pre-0.18 bzr API). For Windows, I tried to use it also, but I saw that there might be a TortoiseBZR program, but I didn't try it. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Jan 4 10:01:02 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 10:01:02 -0500 Subject: [Numpy-discussion] MaskedArray and the min, max, sum, prod Methods In-Reply-To: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> References: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> Message-ID: <200801041001.02660.pgmdevlist@gmail.com> On Thursday 03 January 2008 15:49:45 Alexander Michael wrote: > Working with the new MaskedArray, I noticed the following differences > > with numpy.array behavior: > >>> masked_array([1, 2, 3], mask=True).min() > > 2147483647 That's a bug, the result should be maskedarray.masked. Samething for max, of course. > >>> masked_array([1, 2, 3], mask=True).prod() > > masked_array(data = --, > mask = True, > fill_value=1e+020) > > >>> array([]).prod() > > 1.0 That's OK here, the result is maskedarray.masked. Ditto for sum From pgmdevlist at gmail.com Fri Jan 4 10:01:20 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 10:01:20 -0500 Subject: [Numpy-discussion] MaskedArray and Record Arrays In-Reply-To: <525f23e80801031241o2f82110nbbb774ac6da1cf3d@mail.gmail.com> References: <525f23e80801031241o2f82110nbbb774ac6da1cf3d@mail.gmail.com> Message-ID: <200801041001.20773.pgmdevlist@gmail.com> On Thursday 03 January 2008 15:41:16 Alexander Michael wrote: > I am experimenting with the new MaskedArray (from > ) as a > replacement for my own home-brewed masked data handling mechanisms. ... > Any plans to make this style of record array > usage work with the new MaskedArray? Alexander, Let me check what I can do. Thanks a lot for the suggestion! From stefan at sun.ac.za Fri Jan 4 10:09:50 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 4 Jan 2008 17:09:50 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> <477E24C1.80505@ar.media.kyoto-u.ac.jp> Message-ID: <20080104150950.GA7649@mentat.za.net> Hi Matthieu On Fri, Jan 04, 2008 at 03:26:52PM +0100, Matthieu Brucher wrote: > Beside this, I'm starting to use bazaar (in fact it's the successor of arch) > for a small project of mine hosted on launchpad.net, and it works > great. As Note that bzr refers to bazaar-ng (new generation), which is not the same as the bazaar which succeeded arch/tla. Regards St?fan From stefan at sun.ac.za Fri Jan 4 10:22:02 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 4 Jan 2008 17:22:02 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <20080104152202.GB7649@mentat.za.net> Hi David On Fri, Jan 04, 2008 at 08:24:04PM +0900, David Cournapeau wrote: > First things first, happy new year to all ! Happy new year! It's been great so far :) > Having recently felt the pain to use subversion merge, I was > wondering about people's feeling on moving away from subversion and > using a better system, ala mercurial or bzr (I will talk about bzr > because that's the one I know the most, but this discussion is really > about using something better than subversion, not that much about bzr). > I think this could be an important step forward, and is somewhat related > to the discusions on scikits and co. I have a couple of questions, that you may be able to answer more quickly than what I could, googling for a few hours: 1) Is it easy to setup bzr so that many people have submit-access to the main-branch, i.e. so that we don't need a central "patch-manager"? 2) With a DRCS, we have to check out the whole repository, instead of the current sources only. Currently, the history amounts to roughly 70Mb, but that includes files that have been deleted etc. Is there any way to compact the repository, or to say "let's only go 100 revisions back, and for the rest query the main branch"? I'm just worried that, some day in the future, a user will need to do an extremely large checkout to hack on a fairly small codebase. 3) Which of bzr, mercurial and git has the best merging capabilities? I heard a while back that git does not try to be too clever about it, while the others do -- I wonder how that worked out. I am very fond of the distributed model, and use it for my own development, too. Regardless, I would still like to hear more from people who have used it on a larger scale. Regards St?fan From cournape at gmail.com Fri Jan 4 10:24:33 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 00:24:33 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> <477E24C1.80505@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220801040724r2de5e738m7128295883835215@mail.gmail.com> On Jan 4, 2008 11:26 PM, Matthieu Brucher wrote: > > > > In the mean time, do you want to tell us more about how you use bzr with > > > svn. This seems like a good transitory option. > > Once you installed bzr-svn, you can import the whole scikits trunk using > > the svn-import command. > > This works OK for Linux, but for Windows, the packages needed by bzr-svn > (the python wrappers that are in the usual python-subversion package) are in > the Subversion trunk ( 1.5). So we have to compile them first. Yes, there is indeed a problem with bzr-svn in that respect on windows. People are trying to improve the situation, though (at least one developer of bzr is a windows user, by which I mean he mainly uses windows, and bzr-svn is one of the top priority for obvious reasons). But the transition Gael talked about is not about everyone using bzr-svn, this would be overkill and a waste of ressources. The solution would be to have a bzr mirror of svn, hosted somewhere on scipy.org, so that we use bzr, on a bzr repository, e.g only the mirror system would need bzr-svn. > Beside this, I'm starting to use bazaar (in fact it's the successor of arch) > for a small project of mine hosted on launchpad.net, and it works great. As > David stated, the only problem is the UI : on Linux, I'm using mainly the > command line because olive-gtk is buggy and not really user friendly (there > is room for improvement, it has even deprecation warnings because it uses a > pre-0.18 bzr API). For Windows, I tried to use it also, but I saw that there > might be a TortoiseBZR program, but I didn't try it. As a linux user, I don't see the point of a GUI for most tasks related to bzr :) I sometimes use bzr-gtk, though. On windows, it seems that qbzr is the most windows-friendly. I don't know the state of TortoiseBzr, but I will find out soon, since I have to set up a system to coordinate change at my lab, and I intend to test more thoroughly trac+bzr+TortoiseBzr. David From lxander.m at gmail.com Fri Jan 4 10:27:32 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 4 Jan 2008 10:27:32 -0500 Subject: [Numpy-discussion] MaskedArray and the min, max, sum, prod Methods In-Reply-To: <200801041001.02660.pgmdevlist@gmail.com> References: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> <200801041001.02660.pgmdevlist@gmail.com> Message-ID: <525f23e80801040727m6ea3eef6m2b8f83d9cd5b50bd@mail.gmail.com> On Jan 4, 2008 10:01 AM, Pierre GM wrote: > On Thursday 03 January 2008 15:49:45 Alexander Michael wrote: > > Working with the new MaskedArray, I noticed the following differences > > > > with numpy.array behavior: > > >>> masked_array([1, 2, 3], mask=True).min() > > > > 2147483647 > > That's a bug, the result should be maskedarray.masked. Samething for max, of > course. > > > >>> masked_array([1, 2, 3], mask=True).prod() > > > > masked_array(data = --, > > mask = True, > > fill_value=1e+020) > > > > >>> array([]).prod() > > > > 1.0 > > That's OK here, the result is maskedarray.masked. Ditto for sum Hmm. I liked the base ndarray behavior as it makes a lot of sense to me and provides an easy default that avoids needing to check the result between steps. Does MaskedArray do this to be compatible with the original ma, or is there a theoretically good reason for it that I am missing? Thanks! Alex From pgmdevlist at gmail.com Fri Jan 4 10:42:54 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 10:42:54 -0500 Subject: [Numpy-discussion] MaskedArray and the min, max, sum, prod Methods In-Reply-To: <525f23e80801040727m6ea3eef6m2b8f83d9cd5b50bd@mail.gmail.com> References: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> <200801041001.02660.pgmdevlist@gmail.com> <525f23e80801040727m6ea3eef6m2b8f83d9cd5b50bd@mail.gmail.com> Message-ID: <200801041042.54533.pgmdevlist@gmail.com> On Friday 04 January 2008 10:27:32 Alexander Michael wrote: > Hmm. I liked the base ndarray behavior as it makes a lot of sense to > me and provides an easy default that avoids needing to check the > result between steps. I must admit I have troubles conceptualizing the product of an empty array, so the ndarray behavior is a bit puzzling to me. > Does MaskedArray do this to be compatible > with the original ma, or is there a theoretically good reason for it > that I am missing? Yes for the part. For the second, well, I think there's a difference between a maskedarray of a given size, where all values are masked, and an empty array. Any operation on a fully-masked array should result in maskedarray.masked. From cournape at gmail.com Fri Jan 4 10:45:49 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 00:45:49 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <20080104152202.GB7649@mentat.za.net> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> Message-ID: <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> On Jan 5, 2008 12:22 AM, Stefan van der Walt wrote: > Hi David > > On Fri, Jan 04, 2008 at 08:24:04PM +0900, David Cournapeau wrote: > > First things first, happy new year to all ! > > Happy new year! It's been great so far :) > > > Having recently felt the pain to use subversion merge, I was > > wondering about people's feeling on moving away from subversion and > > using a better system, ala mercurial or bzr (I will talk about bzr > > because that's the one I know the most, but this discussion is really > > about using something better than subversion, not that much about bzr). > > I think this could be an important step forward, and is somewhat related > > to the discusions on scikits and co. > > I have a couple of questions, that you may be able to answer more > quickly than what I could, googling for a few hours: > > 1) Is it easy to setup bzr so that many people have submit-access to > the main-branch, i.e. so that we don't need a central "patch-manager"? If you read the linus' email I linked, he says a word about that, as well as several related problems (having several "main branches", and problems related to access/regression testing/build bot). I don't know much about it, but bzr's team uses a software which tracks merge requests. The trunk is thus read only (in the sense that nobody pushes change to it), and pull changes from the software tracking merge requests. https://launchpad.net/pqm My understanding, but I have never used that feature, is that mercurial has something similar. > > 2) With a DRCS, we have to check out the whole repository, instead of > the current sources only. Currently, the history amounts to roughly > 70Mb, but that includes files that have been deleted etc. Is there > any way to compact the repository, or to say "let's only go 100 > revisions back, and for the rest query the main branch"? I'm just > worried that, some day in the future, a user will need to do an > extremely large checkout to hack on a fairly small codebase. Git has this feature, bzr not yet. I don't know about mercurial. In itself, it is not so much a problem because the amount of data is not so big compared to a svn checkout. As I mentionned previously, for scikits and scipy import, the whole history imported un bzr is smaller than a svn checkout ! It depends a lot on the codebase details, but again, my understanding is that the repository formats used by DVCS makes it quite efficient to get the whole history. This is the reason why I linked the Keith Richard blog: in his discussion about why repository format matters, there is the story of the 2.7 Gb mozilla CVS repository going to 8.2 Gb when imported in svn, going back to 450 Mb when imported with git (compared to a 350 Mb checkout !). > > 3) Which of bzr, mercurial and git has the best merging capabilities? > I heard a while back that git does not try to be too clever about it, > while the others do -- I wonder how that worked out. I don't know. Mercurial historically used an external merge tool, I think. I can just say that most of the time, bzr merge works without any problem, whereas svn never works. I basically gave up on using merge with svn: for my scons related work, I just checked out from svn, created a bzr repository from it, and made my changes inside bzr, because I was fed up with svnmerge failing every single time I wanted to pull merges from other branches. Maybe I am too stupide, but I don't see how anyone would want to use the merge command from svn. cheers, David From matthieu.brucher at gmail.com Fri Jan 4 10:50:52 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 4 Jan 2008 16:50:52 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801040724r2de5e738m7128295883835215@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <477E1D21.1010203@scipy.org> <477E1E65.3040604@ar.media.kyoto-u.ac.jp> <20080104121159.GA13665@clipper.ens.fr> <477E24C1.80505@ar.media.kyoto-u.ac.jp> <5b8d13220801040724r2de5e738m7128295883835215@mail.gmail.com> Message-ID: > > As a linux user, I don't see the point of a GUI for most tasks related > to bzr :) I sometimes use bzr-gtk, though. On windows, it seems that > qbzr is the most windows-friendly. I don't know the state of > TortoiseBzr, but I will find out soon, since I have to set up a system > to coordinate change at my lab, and I intend to test more thoroughly > trac+bzr+TortoiseBzr. I tried to test qbzr, but I could find where and how to launch it, and it seemed less advanced than olive, so I stopped trying :| But as you said it, a lot can be done in command line (and should be done this way, even for SVN). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Jan 4 10:53:55 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 4 Jan 2008 17:53:55 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <20080104155355.GC7649@mentat.za.net> Hi David On Fri, Jan 04, 2008 at 08:24:04PM +0900, David Cournapeau wrote: > Having recently felt the pain to use subversion merge, I was [...] Also take a look at https://launchpad.net/pqm I don't know why we haven't set up a hook like this a long time ago in SVN, it just makes so much sense -- no checkins without the unit tests passing! Regards St?fan From charlesr.harris at gmail.com Fri Jan 4 11:30:00 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 09:30:00 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 4, 2008 4:24 AM, David Cournapeau wrote: > Hi, > > First things first, happy new year to all ! > > Having recently felt the pain to use subversion merge, I was > wondering about people's feeling on moving away from subversion and > using a better system, ala mercurial or bzr (I will talk about bzr > because that's the one I know the most, but this discussion is really > about using something better than subversion, not that much about bzr). > I think this could be an important step forward, and is somewhat related > to the discusions on scikits and co. > As some of you are certainly aware, there has been a recent trend > towards so called Distributed Version Control Systems (DVCS). I won't go > into the details, because it varies from system to system, and I am in > no position to explain technical details. But for people who are > wondering, here is a small description of DVCS, and why I think this can > be a significant step forward for numpy/scipy. You can skip it if you > know about them > I like Mercurial and use it a lot, but I'm not convinced we have enough developers and code to justify the pain of changing the VCS at this time. SVN generally works well and has good support on Windows through tortoise. Mercurial also has tortoise support these days, but I haven't ever used it and can't comment on it. In fact, I've never even used Mercurial on windows, perhaps someone can relate their experiences with it. I suppose a shift might be justified if there is a lot of branch merging and such in our future. Anyone know what folks are working in branches? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Jan 4 11:56:49 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 01:56:49 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> On Jan 5, 2008 1:30 AM, Charles R Harris wrote: > > I like Mercurial and use it a lot, but I'm not convinced we have enough > developers and code to justify the pain of changing the VCS at this time. I don't understand the number of developers argument: on most of the projects I am working on, I am the only developer, and I much prefer bzr to svn, although for reasons which are not really relevant to a project like numpy/scipy. > SVN g!enerally works well and has good support on Windows through tortoise. That's where I don't agree: I don't think svn works really well. As long as you use it as an history backup, it works ok, but that's it. The non functional merge makes branching almost useless, reverting back in time is extremely cumbersome, > Mercurial also has tortoise support these days, but I haven't ever used it > and can't comment on it. In fact, I've never even used Mercurial on windows, > perhaps someone can relate their experiences with it. I suppose a shift > might be justified if there is a lot of branch merging and such in our > future. Anyone know what folks are working in branches? Well, I started this discussion because of the scikits discussion. A typical use of branches is for sandboxes: it makes a lot of sense to use branches instead of sandboxes. Also, when branching actually works, you really start using many branches: I do it all the time on all my projects, and I am the only developer on most of them. It means that you commit smaller changes (because comitting does not mean makeing your changes available to the trunk), and instead of submitting one big changeset, you actually submit a serie of small changes. This really makes a big difference IMHO. Also, things like log, blame are actually usable, since they are much faster on DVCS. For something like scipy (less for numpy), where many people develop different things, I think it really makes a lot of sense to use a DVCS. I actually think scipy to be more distributed in nature than many open source projects (again, this is much less true for numpy, IMHO). David From lxander.m at gmail.com Fri Jan 4 12:01:14 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 4 Jan 2008 12:01:14 -0500 Subject: [Numpy-discussion] MaskedArray and the min, max, sum, prod Methods In-Reply-To: <200801041042.54533.pgmdevlist@gmail.com> References: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> <200801041001.02660.pgmdevlist@gmail.com> <525f23e80801040727m6ea3eef6m2b8f83d9cd5b50bd@mail.gmail.com> <200801041042.54533.pgmdevlist@gmail.com> Message-ID: <525f23e80801040901p257d1eeftda36e1f9cc13e3ce@mail.gmail.com> On Jan 4, 2008 10:42 AM, Pierre GM wrote: > On Friday 04 January 2008 10:27:32 Alexander Michael wrote: > > > Hmm. I liked the base ndarray behavior as it makes a lot of sense to > > me and provides an easy default that avoids needing to check the > > result between steps. > > I must admit I have troubles conceptualizing the product of an empty array, so > the ndarray behavior is a bit puzzling to me. I conceptualize the prod method as the foldl (i.e. reduce) combinator for the multiplication operator. Combinators of this kind are "initialized" with the identity value of the operator being applied, which is 1 for multiplication and 0 for summation (for the integers and reals). These are very rigourous definitions from category theory and adhered to within the computer science community (but I'm just a voyeur, so someone correct me if I am wrong). As for the masking, at least I am prone to think "skip masked elements during iteration", which may not apply beyond 1D (which could be the issue you are solving). Concretely, this is how I would write my own prod function: def ma_prod(a): p = 1.0 for x in a.data[~a.mask]: p *= x return p > > Does MaskedArray do this to be compatible > > with the original ma, or is there a theoretically good reason for it > > that I am missing? > > Yes for the part. For the second, well, I think there's a difference between a > maskedarray of a given size, where all values are masked, and an empty array. > Any operation on a fully-masked array should result in maskedarray.masked. All the CS mumbo-jumbo aside, the real question appears to be are fully masked arrays empty or undefined? I would consider them conceptually empty, if only because it makes for a convenient continuity as the number of masked elements increases from one to the whole of the array. Thanks for the hard work on the new MaskedArray. I'm attempting to be constructive by raising these issues, so please let me know if my comments are otherwise. Regards, Alex From pearu at cens.ioc.ee Fri Jan 4 12:23:50 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 19:23:50 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? Message-ID: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> Hi, Say, one defines class A(tuple): def __repr__(self): return 'A(%s)' % (tuple.__repr__(self)) and I'd like to create an array of A instances. Currently I get >>> array([A((1,2))], dtype=object) array([[1, 2]], dtype=object) but what I'd like to get is array([A((1, 2))], dtype=object) I tried defining A.__array__ method but that does not have effect when A instance is inside a sequence to array call. Thanks for any information for how to resolve this problem, PEaru From oliphant at enthought.com Fri Jan 4 12:33:52 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Jan 2008 11:33:52 -0600 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> Message-ID: <477E6E00.4070402@enthought.com> Pearu Peterson wrote: > Hi, > > Say, one defines > > class A(tuple): > def __repr__(self): > return 'A(%s)' % (tuple.__repr__(self)) > > and I'd like to create an array of A instances. > > Currently I get > > The array function was designed a long time ago to inspect sequences and flatten them. Arguably, there should be more intelligence when an "object" array is requested, but there is ambiguity about what the "right" thing to do is. Thus, the current situation is that if you are creating object arrays, the advice is to populate it after the fact. So, create an empty object array and insert the entries the way you want them: a = np.empty(1,dtype=object) a[0] = A((1,2)) -Travis From pearu at cens.ioc.ee Fri Jan 4 13:00:47 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 20:00:47 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477E6E00.4070402@enthought.com> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> Message-ID: <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: > Pearu Peterson wrote: >> Hi, >> >> Say, one defines >> >> class A(tuple): >> def __repr__(self): >> return 'A(%s)' % (tuple.__repr__(self)) >> >> and I'd like to create an array of A instances. > > The array function was designed a long time ago to inspect sequences and > flatten them. > > Arguably, there should be more intelligence when an "object" array is > requested, but there is ambiguity about what the "right" thing to do is. > > Thus, the current situation is that if you are creating object arrays, > the advice is to populate it after the fact. > > So, create an empty object array and insert the entries the way you want > them: > > a = np.empty(1,dtype=object) > a[0] = A((1,2)) Thanks Travis for the hint! The solution is too verbose for an end-user, though. Meantime I was reading arrayobject.c and it seems that before objects are checked for being sequences, their __array_interface__ is accessed (eg in Array_FromSequence, discover_depth). Would this provide a solution if the class A would define a property __array_interface__? I just don't know what the data field should be for an object. Thanks, Pearu From pearu at cens.ioc.ee Fri Jan 4 13:13:27 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 20:13:27 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477E6E00.4070402@enthought.com> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> Message-ID: <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: > So, create an empty object array and insert the entries the way you want > them: > > a = np.empty(1,dtype=object) > a[0] = A((1,2)) Actually this is still an option if to put the above to a A.as_array method something like class A(tuple): def as_array(self): import numpy obj = numpy.empty(1,dtype=object) obj[0] = self return obj but it would be nice if the __array__ method also worked. Could this be something that should be put to issues or would it just get a WontFix flag? Thanks again, Pearu From charlesr.harris at gmail.com Fri Jan 4 13:33:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 11:33:36 -0700 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> Message-ID: On Jan 4, 2008 11:13 AM, Pearu Peterson wrote: > On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: > > > So, create an empty object array and insert the entries the way you want > > them: > > > > a = np.empty(1,dtype=object) > > a[0] = A((1,2)) > > Actually this is still an option if to put the above > to a A.as_array method something like > > class A(tuple): > def as_array(self): > import numpy > obj = numpy.empty(1,dtype=object) > obj[0] = self > return obj > > but it would be nice if the __array__ method also worked. > Could this be something that should be put to issues or > would it just get a WontFix flag? IIRC the previous discussions, there simply isn't enough information in the array interface to decide what needs to be done with regards to object arrays. I think a general solution would require a specialized creation function with more arguments, say, a depth argument which would create objects at a certain depth of nesting. In your case that would be at level 0(1?). I think that would solve most problems, and I suppose it could be added to the current array interface with a default value implying current behavior, i.e., as deep as possible. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Fri Jan 4 13:45:06 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 4 Jan 2008 19:45:06 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> Message-ID: <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> On Jan 4, 2008 5:56 PM, David Cournapeau wrote: > On Jan 5, 2008 1:30 AM, Charles R Harris wrote: > > > > > I like Mercurial and use it a lot, but I'm not convinced we have enough > > developers and code to justify the pain of changing the VCS at this time. > > I don't understand the number of developers argument: on most of the > projects I am working on, I am the only developer, and I much prefer > bzr to svn, although for reasons which are not really relevant to a > project like numpy/scipy. > > > SVN g!enerally works well and has good support on Windows through tortoise. > That's where I don't agree: I don't think svn works really well. As > long as you use it as an history backup, it works ok, but that's it. > The non functional merge makes branching almost useless, reverting > back in time is extremely cumbersome, > > Mercurial also has tortoise support these days, but I haven't ever used it > > and can't comment on it. In fact, I've never even used Mercurial on windows, > > perhaps someone can relate their experiences with it. I suppose a shift > > might be justified if there is a lot of branch merging and such in our > > future. Anyone know what folks are working in branches? > Well, I started this discussion because of the scikits discussion. A > typical use of branches is for sandboxes: it makes a lot of sense to > use branches instead of sandboxes. Also, when branching actually > works, you really start using many branches: I do it all the time on > all my projects, and I am the only developer on most of them. It means > that you commit smaller changes (because comitting does not mean > makeing your changes available to the trunk), and instead of > submitting one big changeset, you actually submit a serie of small > changes. This really makes a big difference IMHO. Also, things like > log, blame are actually usable, since they are much faster on DVCS. > > For something like scipy (less for numpy), where many people develop > different things, I think it really makes a lot of sense to use a > DVCS. I actually think scipy to be more distributed in nature than > many open source projects (again, this is much less true for numpy, > IMHO). David is 100% right, I fully support this. I would be just repeating what he says. Charles actually said another point in favor of Mercurial - it works on Windows (at least people say so), while git not that much (at least people say so). I never use Windows myself, so I don't know. Subversion sucks not only in the merge thing, but especially when providing patches. Because most of the people don't have access to the repo, and not being able to commit locally (=work incrementally), is just bad. So, I use mercurial even when providing patches for svn. Ondrej From oliphant at enthought.com Fri Jan 4 13:55:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Jan 2008 12:55:56 -0600 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> Message-ID: <477E813C.8060803@enthought.com> Charles R Harris wrote: > > > On Jan 4, 2008 11:13 AM, Pearu Peterson > wrote: > > On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: > > > So, create an empty object array and insert the entries the way > you want > > them: > > > > a = np.empty(1,dtype=object) > > a[0] = A((1,2)) > > Actually this is still an option if to put the above > to a A.as_array method something like > > class A(tuple): > def as_array(self): > import numpy > obj = numpy.empty(1,dtype=object) > obj[0] = self > return obj > > but it would be nice if the __array__ method also worked. > Could this be something that should be put to issues or > would it just get a WontFix flag? > > > IIRC the previous discussions, there simply isn't enough information > in the array interface to decide what needs to be done with regards to > object arrays. I think a general solution would require a specialized > creation function with more arguments, say, a depth argument which > would create objects at a certain depth of nesting. In your case that > would be at level 0(1?). I think that would solve most problems, and I > suppose it could be added to the current array interface with a > default value implying current behavior, i.e., as deep as possible. > I like the depth option, and that could be used with the current interface without too much difficulty, I think. This question does come up quite often and should probably be addressed. -Travis From millman at berkeley.edu Fri Jan 4 13:56:12 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 4 Jan 2008 10:56:12 -0800 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: In general I think that this is a good direction to go in. My general preference would be to use git or mercurial. I haven't had time to read the entire thread, but since I won't get a chance to catch up on this thread until much later today -- here are my concerns: 1. We use as vanilla a version of Trac as possible. In particular, we should avoid using experimental plugins. 2. Before making any major changes on the scipy.org server we first get the server upgraded and cleaned up. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From fperez.net at gmail.com Fri Jan 4 13:58:07 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 4 Jan 2008 11:58:07 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: On Jan 4, 2008 11:45 AM, Ondrej Certik wrote: > David is 100% right, I fully support this. I would be just repeating > what he says. > > Charles actually said another point in favor of Mercurial - it works > on Windows (at least people say so), while git not that much (at least > people say so). I never use Windows myself, so I don't know. FWIW, we (ipython) have also gone around a few times on this, and would like (at some point to switch to a DVCS as well). I think the benefits are many, so I won't rehash it here, others have done it well. One point that hasn't been mentioned is how useful a DVCS is when doing dev sprints: people can work and sync off their own private repos without touching SVN, with lots and lots of cross-developer information flow that doesn't affect the main server or even other devs. In fact, when doing sprints I always end up making a local hg repo just for that purpose, and then committing back to svn upstream at the end of the sprint. As much as git looks really good, the Windows issue is, I think, a deal killer: last I checked support was poor, and I think our core dev tools should be truly, 100% cross-platform without any discrimination (kinda-sorta-works on platform X isn't enough). My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... f From pearu at cens.ioc.ee Fri Jan 4 13:58:40 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 20:58:40 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> Message-ID: <56663.85.166.31.187.1199473120.squirrel@cens.ioc.ee> On Fri, January 4, 2008 8:00 pm, Pearu Peterson wrote: > On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: >> Pearu Peterson wrote: >>> Hi, >>> >>> Say, one defines >>> >>> class A(tuple): >>> def __repr__(self): >>> return 'A(%s)' % (tuple.__repr__(self)) >>> >>> and I'd like to create an array of A instances. >> So, create an empty object array and insert the entries the way you want >> them: >> >> a = np.empty(1,dtype=object) >> a[0] = A((1,2)) > > Meantime I was reading arrayobject.c and it seems that > before objects are checked for being sequences, their > __array_interface__ is accessed (eg in Array_FromSequence, > discover_depth). > > Would this provide a solution if the class A would define > a property __array_interface__? I just don't know what > the data field should be for an object. Ok, I found a partial solution: class A(tuple): def __repr__(self): return 'A(%s)' % (tuple.__repr__(self)) @property def __array_interface__(self): import numpy obj = numpy.empty(1,dtype=object) obj[0] = self return obj.__array_interface__.copy() From pearu at cens.ioc.ee Fri Jan 4 14:03:29 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 21:03:29 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> Message-ID: <57391.85.166.31.187.1199473409.squirrel@cens.ioc.ee> On Fri, January 4, 2008 8:00 pm, Pearu Peterson wrote: > On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: >> Pearu Peterson wrote: >>> Hi, >>> >>> Say, one defines >>> >>> class A(tuple): >>> def __repr__(self): >>> return 'A(%s)' % (tuple.__repr__(self)) >>> >>> and I'd like to create an array of A instances. >> >> The array function was designed a long time ago to inspect sequences and >> flatten them. >> >> Arguably, there should be more intelligence when an "object" array is >> requested, but there is ambiguity about what the "right" thing to do is. >> >> Thus, the current situation is that if you are creating object arrays, >> the advice is to populate it after the fact. >> >> So, create an empty object array and insert the entries the way you want >> them: >> >> a = np.empty(1,dtype=object) >> a[0] = A((1,2)) > Meantime I was reading arrayobject.c and it seems that > before objects are checked for being sequences, their > __array_interface__ is accessed (eg in Array_FromSequence, > discover_depth). > > Would this provide a solution if the class A would define > a property __array_interface__? I just don't know what > the data field should be for an object. Ok, I found a partial solution: class A(tuple): def __repr__(self): return 'A('+tuple.__repr__(self)+')' @property def __array_interface__(self): import numpy obj = numpy.empty(1,dtype=object) obj[0] = self return obj.__array_interface__.copy() >>> from numpy import * >>> array([A((1,2))]) array([[1, 2]], dtype=object) but >>> array(A((1,2))) array([None], dtype=object) Pearu PS: sorry about previous mail, Send was pressed accidentaly. From stefan at sun.ac.za Fri Jan 4 14:03:15 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 4 Jan 2008 21:03:15 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: <20080104190315.GE7649@mentat.za.net> On Fri, Jan 04, 2008 at 07:45:06PM +0100, Ondrej Certik wrote: > Charles actually said another point in favor of Mercurial - it works > on Windows (at least people say so), while git not that much (at least > people say so). I never use Windows myself, so I don't know. Note that bzr also runs under Windows, and is also written in Python+C. Here is the URL I referred to this afternoon on IRC, regarding the diff-algorithm: http://bramcohen.livejournal.com/37690.html Regards St?fan From cournape at gmail.com Fri Jan 4 14:04:54 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 04:04:54 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220801041104p1450223due8100095ce9bdd10@mail.gmail.com> On Jan 5, 2008 3:56 AM, Jarrod Millman wrote: > In general I think that this is a good direction to go in. My general > preference would be to use git or mercurial. > > I haven't had time to read the entire thread, but since I won't get a > chance to catch up on this thread until much later today -- here are > my concerns: > 1. We use as vanilla a version of Trac as possible. In particular, > we should avoid using experimental plugins. > 2. Before making any major changes on the scipy.org server we first > get the server upgraded and cleaned up. I personally think that this is a change which should be carefully planned. Typically, I would not think about doing it before say the 1.1 release of numpy (that is, having several months at least to test things and let people get used to it). My email should really be understood as a proposal for a roadmap, and how to proceed (having mirrors in bzr, hg, etc... testing with an experimental trac which would run in parrallel with the main one, etc...). cheers, David From cournape at gmail.com Fri Jan 4 14:21:46 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 04:21:46 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> On Jan 5, 2008 3:58 AM, Fernando Perez wrote: > On Jan 4, 2008 11:45 AM, Ondrej Certik wrote: > > > David is 100% right, I fully support this. I would be just repeating > > what he says. > > > > Charles actually said another point in favor of Mercurial - it works > > on Windows (at least people say so), while git not that much (at least > > people say so). I never use Windows myself, so I don't know. > > FWIW, we (ipython) have also gone around a few times on this, and > would like (at some point to switch to a DVCS as well). I think the > benefits are many, so I won't rehash it here, others have done it > well. > > One point that hasn't been mentioned is how useful a DVCS is when > doing dev sprints: people can work and sync off their own private > repos without touching SVN, with lots and lots of cross-developer > information flow that doesn't affect the main server or even other > devs. In fact, when doing sprints I always end up making a local hg > repo just for that purpose, and then committing back to svn upstream > at the end of the sprint. > > As much as git looks really good, the Windows issue is, I think, a > deal killer: last I checked support was poor, and I think our core dev > tools should be truly, 100% cross-platform without any discrimination > (kinda-sorta-works on platform X isn't enough). I agree. This is not enough, but for me, the following are non negotiable: - the tool must work on unix, mac os X and windows - the tool must be open source. I guess everyone agrees on those points anyway. > > My vote so far is for hg, for performance reasons but also partly > because sage and sympy already use it, two projects I'm likely to > interact a lot with and that are squarely in line with the > ipython/numpy/scipy/matplotlib world. Since they went first and made > the choice, I'm happy to let that be a factor in my decision. I'd > rather use a tool that others in the same community are also using, > especially when the choice is a sound one on technical merit alone. > I understand the "sumpy uses it" reason, it is definitely a factor. But I would rather have a more thorough study on the merits of each system. For example, being a user of bzr for a year and a half now, I think I have a pretty good idea on how it works, and its advantages. We could then decide on a set of attributes to compare, and people who knows about one tool could then tell about it. Performances-wise, hg and bzr really are comparable nowadays for common, local operations. I don't think it is a relevant parameter for the hg vs bzr choice anymor, specially for scipy/numpy which are small projects (I have bzr imports of scipy and scikits, so I can give some numbers if you need them). Third party tools, special abilities (svn import, storage efficiency, special commands, etc...) are more important I think David From pearu at cens.ioc.ee Fri Jan 4 14:24:31 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 4 Jan 2008 21:24:31 +0200 (EET) Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <56663.85.166.31.187.1199473120.squirrel@cens.ioc.ee> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <49424.85.166.31.187.1199469647.squirrel@cens.ioc.ee> <56663.85.166.31.187.1199473120.squirrel@cens.ioc.ee> Message-ID: <58532.85.166.31.187.1199474671.squirrel@cens.ioc.ee> Just ignore this solution. It was not quite working and I was able to get a segfault from it. Pearu On Fri, January 4, 2008 8:58 pm, Pearu Peterson wrote: > On Fri, January 4, 2008 8:00 pm, Pearu Peterson wrote: >> On Fri, January 4, 2008 7:33 pm, Travis E. Oliphant wrote: >>> Pearu Peterson wrote: >>>> Hi, >>>> >>>> Say, one defines >>>> >>>> class A(tuple): >>>> def __repr__(self): >>>> return 'A(%s)' % (tuple.__repr__(self)) >>>> >>>> and I'd like to create an array of A instances. > >>> So, create an empty object array and insert the entries the way you >>> want >>> them: >>> >>> a = np.empty(1,dtype=object) >>> a[0] = A((1,2)) >> >> Meantime I was reading arrayobject.c and it seems that >> before objects are checked for being sequences, their >> __array_interface__ is accessed (eg in Array_FromSequence, >> discover_depth). >> >> Would this provide a solution if the class A would define >> a property __array_interface__? I just don't know what >> the data field should be for an object. > > Ok, I found a partial solution: > > > class A(tuple): > def __repr__(self): > return 'A(%s)' % (tuple.__repr__(self)) > @property > def __array_interface__(self): > import numpy > obj = numpy.empty(1,dtype=object) > obj[0] = self > return obj.__array_interface__.copy() > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From strawman at astraw.com Fri Jan 4 14:27:34 2008 From: strawman at astraw.com (Andrew Straw) Date: Fri, 04 Jan 2008 11:27:34 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <477D3292.80702@astraw.com> References: <477D3292.80702@astraw.com> Message-ID: <477E88A6.2070600@astraw.com> I have added a page to the wiki describing this issue: http://scipy.org/numpy_warts_and_gotchas I'll link it into the main documentation pages over the next few days, but I ask for a review the following text for correctness and clarity: (You can simply edit the wiki page or post your reply here and I'll do it.) Like most of numpy, allclose() uses the broadcasting rules when performing its operation. This leads to the following behavior: >>> a=32 >>> b=numpy.array([]) >>> numpy.allclose(a,b) True Upon closer inspection, we can see that the broadcasting rules cause a to become a zero-dimensional array like b. The default truth value of a zero-dimensional array is True, so the following holds and illustrates how the above result is consistent with numpy's rules. Andrew Straw wrote: > Apologies if I've missed the discussion of this, but I was recently > surprised by the following behavior (in svn trunk 4673). The following > code runs without triggering the assertion. > > import numpy as np > print np.__version__ > a=np.int32(42) > b=np.array([],dtype=np.int32) > assert np.allclose(a,b) > > Is this expected behavior of numpy or is this a bug I should report? > > Thanks, > Andrew > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Jan 4 14:50:31 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 12:50:31 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: On Jan 4, 2008 12:21 PM, David Cournapeau wrote: > On Jan 5, 2008 3:58 AM, Fernando Perez wrote: > > On Jan 4, 2008 11:45 AM, Ondrej Certik wrote: > > > > > David is 100% right, I fully support this. I would be just repeating > > > what he says. > > > > > > Charles actually said another point in favor of Mercurial - it works > > > on Windows (at least people say so), while git not that much (at least > > > people say so). I never use Windows myself, so I don't know. > > > > FWIW, we (ipython) have also gone around a few times on this, and > > would like (at some point to switch to a DVCS as well). I think the > > benefits are many, so I won't rehash it here, others have done it > > well. > > > > One point that hasn't been mentioned is how useful a DVCS is when > > doing dev sprints: people can work and sync off their own private > > repos without touching SVN, with lots and lots of cross-developer > > information flow that doesn't affect the main server or even other > > devs. In fact, when doing sprints I always end up making a local hg > > repo just for that purpose, and then committing back to svn upstream > > at the end of the sprint. > > > > As much as git looks really good, the Windows issue is, I think, a > > deal killer: last I checked support was poor, and I think our core dev > > tools should be truly, 100% cross-platform without any discrimination > > (kinda-sorta-works on platform X isn't enough). > > I agree. This is not enough, but for me, the following are non negotiable: > - the tool must work on unix, mac os X and windows > - the tool must be open source. > > I guess everyone agrees on those points anyway. > > > > > My vote so far is for hg, for performance reasons but also partly > > because sage and sympy already use it, two projects I'm likely to > > interact a lot with and that are squarely in line with the > > ipython/numpy/scipy/matplotlib world. Since they went first and made > > the choice, I'm happy to let that be a factor in my decision. I'd > > rather use a tool that others in the same community are also using, > > especially when the choice is a sound one on technical merit alone. > > > > I understand the "sumpy uses it" reason, it is definitely a factor. > But I would rather have a more thorough study on the merits of each > system. For example, being a user of bzr for a year and a half now, I > think I have a pretty good idea on how it works, and its advantages. > We could then decide on a set of attributes to compare, and people who > knows about one tool could then tell about it. At this point, it might be more efficient to ask if anyone has objections or knows of any problems. I suspect that both hg and bzr probably do most of what is needed. My own preference is hg because I have several years experience with it, it has a long history with trac, and it is in pretty widespread use. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rowen at cesmail.net Fri Jan 4 14:51:11 2008 From: rowen at cesmail.net (Russell E. Owen) Date: Fri, 04 Jan 2008 11:51:11 -0800 Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> Message-ID: In article <5b8d13220801040856n85b7f5fv29c608cb95d022a5 at mail.gmail.com>, "David Cournapeau" wrote: > On Jan 5, 2008 1:30 AM, Charles R Harris wrote: > > > > > I like Mercurial and use it a lot, but I'm not convinced we have enough > > developers and code to justify the pain of changing the VCS at this time. > > I don't understand the number of developers argument: on most of the > projects I am working on, I am the only developer, and I much prefer > bzr to svn, although for reasons which are not really relevant to a > project like numpy/scipy. > > > SVN g!enerally works well and has good support on Windows through tortoise. > That's where I don't agree: I don't think svn works really well. As > long as you use it as an history backup, it works ok, but that's it. > The non functional merge makes branching almost useless, reverting > back in time is extremely cumbersome, I am a bit puzzled by the vitriol about merging with svn. svn's built in merge is a joke but svnmerge.py works reasonably well (especially newer versions of svnmerge.py; I use rev 26317 and the version included in the current svn 1.4.6 should be even more recent) I agree that reverting a file to an older versions is clumsy using svn. -- Russell From fperez.net at gmail.com Fri Jan 4 14:52:04 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 4 Jan 2008 12:52:04 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: On Jan 4, 2008 12:21 PM, David Cournapeau wrote: > I understand the "sumpy uses it" reason, it is definitely a factor. > But I would rather have a more thorough study on the merits of each > system. For example, being a user of bzr for a year and a half now, I > think I have a pretty good idea on how it works, and its advantages. > We could then decide on a set of attributes to compare, and people who > knows about one tool could then tell about it. > > Performances-wise, hg and bzr really are comparable nowadays for > common, local operations. I don't think it is a relevant parameter for > the hg vs bzr choice anymor, specially for scipy/numpy which are small > projects (I have bzr imports of scipy and scikits, so I can give some > numbers if you need them). Third party tools, special abilities (svn > import, storage efficiency, special commands, etc...) are more > important I think Absolutely. That's why I said above "when the choice is a sound one on technical merit alone". At the time (for sage/sympy) the bzr/hg choice was unmistakably in favor of hg. Things might be different today. Incidentally, the emacs guys seem to be worrying about the same thing: http://thread.gmane.org/gmane.emacs.devel/85893 If they actually do the work of comparing tools, that work may be useful for us. I'm pretty sure that any tool that can handle the entire history of emacs can chew on numpy/scipy/ipython/matplotlib *combined* for breakfast. Cheers, f From charlesr.harris at gmail.com Fri Jan 4 14:57:51 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 12:57:51 -0700 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: <477E88A6.2070600@astraw.com> References: <477D3292.80702@astraw.com> <477E88A6.2070600@astraw.com> Message-ID: On Jan 4, 2008 12:27 PM, Andrew Straw wrote: > I have added a page to the wiki describing this issue: > > http://scipy.org/numpy_warts_and_gotchas > > I'll link it into the main documentation pages over the next few days, > but I ask for a review the following text for correctness and clarity: > (You can simply edit the wiki page or post your reply here and I'll do > it.) > > Like most of numpy, allclose() uses the broadcasting rules when > performing its operation. This leads to the following behavior: > > >>> a=32 > >>> b=numpy.array([]) > >>> numpy.allclose(a,b) > True > > Upon closer inspection, we can see that the broadcasting rules cause a > to become a zero-dimensional array like b. The default truth value of a It is not the dimensions, it's the fact that the array is empty, so that anything said about it's non-existent elements is true, i.e., x in empty -> anything is always true because x in empty is always false. So we also have the following: In [1]: allclose(32,[[]]) Out[1]: True The other reason is that because of broadcasting, the shape of the arrays may be immaterial. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Jan 4 15:20:11 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 15:20:11 -0500 Subject: [Numpy-discussion] MaskedArray and the min, max, sum, prod Methods In-Reply-To: <525f23e80801040901p257d1eeftda36e1f9cc13e3ce@mail.gmail.com> References: <525f23e80801031249i433937f3n885658625dd3fd0c@mail.gmail.com> <200801041042.54533.pgmdevlist@gmail.com> <525f23e80801040901p257d1eeftda36e1f9cc13e3ce@mail.gmail.com> Message-ID: <200801041520.13144.pgmdevlist@gmail.com> On Friday 04 January 2008 12:01:14 Alexander Michael wrote: > Concretely, this is how I would write my own prod function: > > def ma_prod(a): > p = 1.0 > for x in a.data[~a.mask]: > p *= x > return p Which would work only in 1D, as you mentioned. > All the CS mumbo-jumbo aside, the real question appears to be are fully > masked arrays empty or undefined? I would consider them conceptually empty, > if only because it makes for a convenient continuity as the number of > masked elements increases from one to the whole of the array. That's a way to see it. Personally, I'm happy with this behavior: getting masked as the result of .sum() tells me that all the initial values were masked, which is not the same thing as getting 0. > Thanks for the hard work on the new MaskedArray. I'm attempting to be > constructive by raising these issues, so please let me know if my comments > are otherwise. Oh please, thanks a lot for your contributions ! Another issue is that we need to ensure compatibility with the previous version of maskedarray (numpy.core.ma). At the end, that's what matters. From ivilata at carabos.com Fri Jan 4 15:29:54 2008 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Fri, 4 Jan 2008 21:29:54 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477E1754.6030907@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> Message-ID: <20080104202954.GA23858@tardis.terramar.selidor.net> David Cournapeau (el 2008-01-04 a les 20:24:04 +0900) va dir:: >[...] > Integration with trac is the real problem, I think. According to one bzr > developer, trac model (0.10, the last released one) is really based > around subversion notion of repository, which does not fit well with > mercurial and bzr. I don't know if this is true for the not yet released > 0.11. If bzr is considered a possible candidate, I can get more > informations from bzr developers. >[...] My main concern about Trac and bzr integration is that (please correct me if I'm wrong or outdated) Trac doesn't seem to support multiple bzr branches, even if they are under the same shared repository. This would limit Trac to following only one bzr branch, say the trunk (leaving tags and branches out of its control). If this was to be avoided by creating a single bzr branch with all branches and tags, we may be facing a size problem, since bzr lacks cheap copying operations right now (but support is planned). Anyway, this last approach isn't very appropriate with a DVCS. (Multiple bzr branch support in Trac would be really wonderful!) :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: Digital signature URL: From charlesr.harris at gmail.com Fri Jan 4 15:36:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 13:36:09 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: On Jan 4, 2008 12:52 PM, Fernando Perez wrote: > On Jan 4, 2008 12:21 PM, David Cournapeau wrote: > > > I understand the "sumpy uses it" reason, it is definitely a factor. > > But I would rather have a more thorough study on the merits of each > > system. For example, being a user of bzr for a year and a half now, I > > think I have a pretty good idea on how it works, and its advantages. > > We could then decide on a set of attributes to compare, and people who > > knows about one tool could then tell about it. > > > > Performances-wise, hg and bzr really are comparable nowadays for > > common, local operations. I don't think it is a relevant parameter for > > the hg vs bzr choice anymor, specially for scipy/numpy which are small > > projects (I have bzr imports of scipy and scikits, so I can give some > > numbers if you need them). Third party tools, special abilities (svn > > import, storage efficiency, special commands, etc...) are more > > important I think > > Absolutely. That's why I said above "when the choice is a sound one > on technical merit alone". At the time (for sage/sympy) the bzr/hg > choice was unmistakably in favor of hg. Things might be different > today. > > Incidentally, the emacs guys seem to be worrying about the same thing: > > http://thread.gmane.org/gmane.emacs.devel/85893 > > If they actually do the work of comparing tools, that work may be > useful for us. I'm pretty sure that any tool that can handle the > entire history of emacs can chew on numpy/scipy/ipython/matplotlib > *combined* for breakfast. > A quick google for benchmarks show that a year ago, hg was a bit faster and generated smaller repositories than bzr, but I don't think the difference is enough to matter. Git is 10-20 times faster than either for a lot of things, but Linus was definitely focused on speed, which is easy to understand if you look at the churn in the kernel. Anyway, I suspect that, technically, both bzr and hg are suitable choices. I'm not sure esr correct that it is unlikely that both are going to last long term, bazaar (the ancestor of bzr) is used for Ubuntu. But the two are similar and fill the same niche, so I expect that one or the other will become dominant in the wild. And hg seems to have the advantage of a head start and not being as tightly tied to Linux. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jan 4 15:37:57 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 14:37:57 -0600 Subject: [Numpy-discussion] PyArray_FromAny does not accept a generator? In-Reply-To: References: Message-ID: <477E9925.8020407@gmail.com> Neal Becker wrote: > It seems that PyArray_FromAny does not accept a generator? > > Seems like this would be useful. It's difficult to do all the magical interpretation that PyArray_FromAny() does with a iterator of unknown length. In Python, we have fromiter() which will consume an iterator to make (only) a rank-1 array and thus sidestep a big chunk of the magical interpretation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Jan 4 15:41:16 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 14:41:16 -0600 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477E813C.8060803@enthought.com> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> <477E813C.8060803@enthought.com> Message-ID: <477E99EC.4000505@gmail.com> Travis E. Oliphant wrote: > I like the depth option, and that could be used with the current > interface without too much difficulty, I think. > > This question does come up quite often and should probably be addressed. Alternatively, a shape= argument could be used. You can use -1 entries if you want array() to just figure out the exact dimensions like reshape() does. The shape= argument lets you give array() a bit more information, which might be useful. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Fri Jan 4 15:45:53 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 05:45:53 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> Message-ID: <5b8d13220801041245w25eeecbdy1da6c4de877f6c28@mail.gmail.com> On Jan 5, 2008 4:51 AM, Russell E. Owen wrote: > In article > > I am a bit puzzled by the vitriol about merging with svn. svn's built in > merge is a joke but svnmerge.py works reasonably well (especially newer > versions of svnmerge.py; I use rev 26317 and the version included in the > current svn 1.4.6 should be even more recent) I don't know which version I used. But svnmerge is not that practical: you have to explicitely say which repository you want to track down, and merge almost alway fails. Several times, I had conflicts with files I did not touch, which does not make any sense to me. Merging a file modified in my branch and in another branch almost always gave me a conflict. In bzr (and this is really similar in hg at least), when you merge something, you just say bzr merge SOURCE, and that's it. No svnsync init, no svnmerge avail / svnmerge merge cycles, etc... David From strawman at astraw.com Fri Jan 4 16:00:39 2008 From: strawman at astraw.com (Andrew Straw) Date: Fri, 04 Jan 2008 13:00:39 -0800 Subject: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array) In-Reply-To: References: <477D3292.80702@astraw.com> <477E88A6.2070600@astraw.com> Message-ID: <477E9E77.7000507@astraw.com> Thanks, I updated the page. Charles R Harris wrote: > > > On Jan 4, 2008 12:27 PM, Andrew Straw > wrote: > > I have added a page to the wiki describing this issue: > > http://scipy.org/numpy_warts_and_gotchas > > I'll link it into the main documentation pages over the next few > days, > but I ask for a review the following text for correctness and clarity: > (You can simply edit the wiki page or post your reply here and > I'll do it.) > > Like most of numpy, allclose() uses the broadcasting rules when > performing its operation. This leads to the following behavior: > > >>> a=32 > >>> b=numpy.array([]) > >>> numpy.allclose(a,b) > True > > Upon closer inspection, we can see that the broadcasting rules > cause a > to become a zero-dimensional array like b. The default truth value > of a > > > It is not the dimensions, it's the fact that the array is empty, so > that anything said about it's non-existent elements is true, i.e., x > in empty -> anything is always true because x in empty is always > false. So we also have the following: > > In [1]: allclose(32,[[]]) > Out[1]: True > > The other reason is that because of broadcasting, the shape of the > arrays may be immaterial. > > Chuck > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Fri Jan 4 16:02:16 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 14:02:16 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: On Jan 4, 2008 12:52 PM, Fernando Perez wrote: > On Jan 4, 2008 12:21 PM, David Cournapeau wrote: > > > I understand the "sumpy uses it" reason, it is definitely a factor. > > But I would rather have a more thorough study on the merits of each > > system. For example, being a user of bzr for a year and a half now, I > > think I have a pretty good idea on how it works, and its advantages. > > We could then decide on a set of attributes to compare, and people who > > knows about one tool could then tell about it. > > > > Performances-wise, hg and bzr really are comparable nowadays for > > common, local operations. I don't think it is a relevant parameter for > > the hg vs bzr choice anymor, specially for scipy/numpy which are small > > projects (I have bzr imports of scipy and scikits, so I can give some > > numbers if you need them). Third party tools, special abilities (svn > > import, storage efficiency, special commands, etc...) are more > > important I think > > Absolutely. That's why I said above "when the choice is a sound one > on technical merit alone". At the time (for sage/sympy) the bzr/hg > choice was unmistakably in favor of hg. Things might be different > today. > Sometimes it is the little niggling things that matter, in this case line breaks. Hg (and probably bzr), store everything as binary, so if someone uses an editor that breaks line with CR or LF+CR instead of the good 'ol unixy LF, there might be a lot whitespace updates coming in to the repository. I wonder if there is a way to put an automatic filter in place for that sort of thing? Chuck > > Incidentally, the emacs guys seem to be worrying about the same thing: > > http://thread.gmane.org/gmane.emacs.devel/85893 > > If they actually do the work of comparing tools, that work may be > useful for us. I'm pretty sure that any tool that can handle the > entire history of emacs can chew on numpy/scipy/ipython/matplotlib > *combined* for breakfast. > > Cheers, > > f > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Jan 4 16:05:45 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 06:05:45 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> On Jan 5, 2008 5:36 AM, Charles R Harris wrote: > > > A quick google for benchmarks show that a year ago, hg was a bit faster and > generated smaller repositories than bzr, but I don't think the difference is > enough to matter. Forget a year ago, because as far as bzr is concerned, they got much faster (several times faster for common operations like commit/branch/log/merge). > but Linus was definitely focused on speed, which is easy to understand if > you look at the churn in the kernel. Anyway, I suspect that, technically, > both bzr and hg are suitable choices. I'm not sure esr correct that it is > unlikely that both are going to last long term, bazaar (the ancestor of bzr) > is used for Ubuntu. But the two are similar and fill the same niche, so I > expect that one or the other will become dominant in the wild. And hg seems > to have the advantage of a head start and not being as tightly tied to > Linux. bzr is not tied to linux. They always have win32 binaries, TortoiseBzr has a longer history than the mercurial one, and as said previously, one developer of bzr at least is mainly a windows user. I don't want to sound like I defend bzr, because honestly, I don't care about which one is used, but so far, the arguments I heard against bzr do not reflect my experience at all. One thing that bzr tries hard is the general UI, and the explicit support for several workflows (with moderately advanced concepts such as shared repositories, bound branches: for example, with a branch A bound to branch B, a commit is first pushed on branch B, and if successfull, applied to A; for centralized worflows, this makes things easier). I honestly do not know if this is significant. bzr claims its merge capability is better: I do not know if this is true, or if that matters at all. I would rather discuss those than "bzr is tied to linux", because I don't think they are based on accurate or recent informations. As I said, I have bzr imports of scipy and scikits, and I could easily to the same for hg, make them available for everybody to play with. David From sdb at cloud9.net Fri Jan 4 16:08:54 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 16:08:54 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays Message-ID: NumPy gurus -- I just discovered this today. It looks like a bug to me. Please flame me mercilessly if I am wrong! :-) Sometimes you need to initialize an array using zeros() before doing an assignment to it in a loop. If you assign a complex value to the initialized array, the imaginary part of the array is dropped. Does NumPy do a silent type-cast which causes this behavior? Is this typecast a feature? Below I attach a session log showing the bug. Note that I have boiled down my complex code to this simple case for ease of comprehension. [1] I will also input this bug into the tracking system. By the way, this is NumPy 1.0.4: In [39]: numpy.__version__ Out[39]: '1.0.4' Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ ---------------------- -------------------- In [29]: A = numpy.random.rand(4) + 1j*numpy.random.rand(4) In [30]: B = numpy.zeros((4)) In [31]: In [31]: for i in range(4): ....: B[i] = A[i] ....: In [32]: A Out[32]: array([ 0.12150180+0.00577893j, 0.39792515+0.03607227j, 0.61933379+0.04506978j, 0.56751678+0.24576083j]) In [33]: B Out[33]: array([ 0.1215018 , 0.39792515, 0.61933379, 0.56751678]) ----------------------- ------------------- [1] Yes, I know that I should use vectorized code as often as possible, and that this example is not vectorized. This is a simple example illustrating the problem. Moreover, many times the computation you wish to perform can't easily be vectorized, leaving the nasty old for loop as the only choice...... From ondrej at certik.cz Fri Jan 4 16:17:33 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 4 Jan 2008 22:17:33 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> Message-ID: <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> On Jan 4, 2008 10:05 PM, David Cournapeau wrote: > On Jan 5, 2008 5:36 AM, Charles R Harris wrote: > > > > > > > > A quick google for benchmarks show that a year ago, hg was a bit faster and > > generated smaller repositories than bzr, but I don't think the difference is > > enough to matter. > > Forget a year ago, because as far as bzr is concerned, they got much > faster (several times faster for common operations like > commit/branch/log/merge). > > > but Linus was definitely focused on speed, which is easy to understand if > > you look at the churn in the kernel. Anyway, I suspect that, technically, > > both bzr and hg are suitable choices. I'm not sure esr correct that it is > > unlikely that both are going to last long term, bazaar (the ancestor of bzr) > > is used for Ubuntu. But the two are similar and fill the same niche, so I > > expect that one or the other will become dominant in the wild. And hg seems > > to have the advantage of a head start and not being as tightly tied to > > Linux. > > bzr is not tied to linux. They always have win32 binaries, TortoiseBzr > has a longer history than the mercurial one, and as said previously, > one developer of bzr at least is mainly a windows user. I don't want > to sound like I defend bzr, because honestly, I don't care about which > one is used, but so far, the arguments I heard against bzr do not > reflect my experience at all. > > One thing that bzr tries hard is the general UI, and the explicit > support for several workflows (with moderately advanced concepts such > as shared repositories, bound branches: for example, with a branch A > bound to branch B, a commit is first pushed on branch B, and if > successfull, applied to A; for centralized worflows, this makes things > easier). I honestly do not know if this is significant. bzr claims its > merge capability is better: I do not know if this is true, or if that > matters at all. > > I would rather discuss those than "bzr is tied to linux", because I > don't think they are based on accurate or recent informations. As I > said, I have bzr imports of scipy and scikits, and I could easily to > the same for hg, make them available for everybody to play with. Instead of devising our own arguments, read this: http://bazaar-vcs.org/BzrVsHg and the mercurial response therein. Ondrej From pgmdevlist at gmail.com Fri Jan 4 16:26:00 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 16:26:00 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: <200801041626.00682.pgmdevlist@gmail.com> On Friday 04 January 2008 16:08:54 Stuart Brorson wrote: > Sometimes you need to initialize an array using zeros() before doing > an assignment to it in a loop. If you assign a complex value to the > initialized array, the imaginary part of the array is dropped. Does > NumPy do a silent type-cast which causes this behavior? Is this > typecast a feature? By default, empty arrays are initialized as float. If you try to force a complex into it, yes, the imaginary part will be dropped. Use the dtype argument of zeros or ones to specify the type of your array, for example: B = numpy.zeros((4,), dtype=numpy.complex_) From sdb at cloud9.net Fri Jan 4 16:30:25 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 16:30:25 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: > I just discovered this today. It looks like a bug to me. Please > flame me mercilessly if I am wrong! :-) Hmmmm.... after a little more playing around, I think it's indeed true that NumPy does a typecast to make the resulting assignment have the same type as the LHS, regardless of the type of the RHS. Below I attach another example, which shows this behavior. As a naive user, I would not expect that my variables would get silently typecast in an assignment like this. But is this behavior Pythonic? I'm not sure..... Normally, the Pythonic thing to do when assigning non-conforming variables is to barf -- throw an exception. At least that's what I would expect. Comments? Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ ----------------------- --------------------- In [77]: A = 10*numpy.random.rand(4) In [78]: B = numpy.zeros((4)) In [79]: B.dtype='int64' In [80]: In [80]: for i in range(4): ....: B[i] = A[i] ....: In [81]: A Out[81]: array([ 1.71327285, 3.48128855, 7.51404178, 8.96775254]) In [82]: B Out[82]: array([1, 3, 7, 8], dtype=int64) ----------------------- --------------------- From pgmdevlist at gmail.com Fri Jan 4 16:30:47 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Jan 2008 16:30:47 -0500 Subject: [Numpy-discussion] branches/maskedarray patch + lost password Message-ID: <200801041630.47698.pgmdevlist@gmail.com> All, Here's a patch to the current version of svn/numpy/branches/maskedarray, that corrects several bugs. I tried to commit it to the server itself, but I unfortunately lost my password (or maybe I never had a password for numpy branches in the first place). Could anybody in charge of the SVN contact me offlist ? Thanks an awful lot in advance. The revision info is below. Sorry for the noise, and thanks again. P. fix_invalid : change the default to copy=True _MaskedUnaryOperation : make sure the result gets updated from the input (for subclasses) _MaskedBinaryOperation : make sure the result gets updated from the inputs (for subclasses) _DomainedBinaryOperation : make sure the result gets updated from the inputs (for subclasses) MaskedArray.__new__ : added the ndmin keyword MaskedArray.__getitem__ : works with fields, using the global mask MaskedArray.__setitem__ : works with fields. The global mask is NOT updated MaskedArray.ids : fixed when nomask MaskedArray.min/.max : force masked to be returned when the whole array is masked array : added the ndmin keyword -------------- next part -------------- A non-text attachment was scrubbed... Name: maskedarray_core.patch Type: text/x-diff Size: 13628 bytes Desc: not available URL: From efiring at hawaii.edu Fri Jan 4 16:31:14 2008 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 04 Jan 2008 11:31:14 -1000 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> Message-ID: <477EA5A2.4060502@hawaii.edu> I have been using mercurial for some time now. I just discovered that the introductory documentation has been improved and consolidated in an online book-in-progress: http://hgbook.red-bean.com/hgbook.html Eric David Cournapeau wrote: > On Jan 5, 2008 5:36 AM, Charles R Harris wrote: > > >> A quick google for benchmarks show that a year ago, hg was a bit faster and >> generated smaller repositories than bzr, but I don't think the difference is >> enough to matter. > > Forget a year ago, because as far as bzr is concerned, they got much > faster (several times faster for common operations like > commit/branch/log/merge). > >> but Linus was definitely focused on speed, which is easy to understand if >> you look at the churn in the kernel. Anyway, I suspect that, technically, >> both bzr and hg are suitable choices. I'm not sure esr correct that it is >> unlikely that both are going to last long term, bazaar (the ancestor of bzr) >> is used for Ubuntu. But the two are similar and fill the same niche, so I >> expect that one or the other will become dominant in the wild. And hg seems >> to have the advantage of a head start and not being as tightly tied to >> Linux. > > bzr is not tied to linux. They always have win32 binaries, TortoiseBzr > has a longer history than the mercurial one, and as said previously, > one developer of bzr at least is mainly a windows user. I don't want > to sound like I defend bzr, because honestly, I don't care about which > one is used, but so far, the arguments I heard against bzr do not > reflect my experience at all. > > One thing that bzr tries hard is the general UI, and the explicit > support for several workflows (with moderately advanced concepts such > as shared repositories, bound branches: for example, with a branch A > bound to branch B, a commit is first pushed on branch B, and if > successfull, applied to A; for centralized worflows, this makes things > easier). I honestly do not know if this is significant. bzr claims its > merge capability is better: I do not know if this is true, or if that > matters at all. > > I would rather discuss those than "bzr is tied to linux", because I > don't think they are based on accurate or recent informations. As I > said, I have bzr imports of scipy and scikits, and I could easily to > the same for hg, make them available for everybody to play with. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Fri Jan 4 16:35:29 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 06:35:29 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> Message-ID: <5b8d13220801041335h159e3351r2a723fe18863fbc0@mail.gmail.com> On Jan 5, 2008 6:17 AM, Ondrej Certik wrote: > > On Jan 4, 2008 10:05 PM, David Cournapeau wrote: > > On Jan 5, 2008 5:36 AM, Charles R Harris wrote: > > > > > > > > > > > > > A quick google for benchmarks show that a year ago, hg was a bit faster and > > > generated smaller repositories than bzr, but I don't think the difference is > > > enough to matter. > > > > Forget a year ago, because as far as bzr is concerned, they got much > > faster (several times faster for common operations like > > commit/branch/log/merge). > > > > > but Linus was definitely focused on speed, which is easy to understand if > > > you look at the churn in the kernel. Anyway, I suspect that, technically, > > > both bzr and hg are suitable choices. I'm not sure esr correct that it is > > > unlikely that both are going to last long term, bazaar (the ancestor of bzr) > > > is used for Ubuntu. But the two are similar and fill the same niche, so I > > > expect that one or the other will become dominant in the wild. And hg seems > > > to have the advantage of a head start and not being as tightly tied to > > > Linux. > > > > bzr is not tied to linux. They always have win32 binaries, TortoiseBzr > > has a longer history than the mercurial one, and as said previously, > > one developer of bzr at least is mainly a windows user. I don't want > > to sound like I defend bzr, because honestly, I don't care about which > > one is used, but so far, the arguments I heard against bzr do not > > reflect my experience at all. > > > > One thing that bzr tries hard is the general UI, and the explicit > > support for several workflows (with moderately advanced concepts such > > as shared repositories, bound branches: for example, with a branch A > > bound to branch B, a commit is first pushed on branch B, and if > > successfull, applied to A; for centralized worflows, this makes things > > easier). I honestly do not know if this is significant. bzr claims its > > merge capability is better: I do not know if this is true, or if that > > matters at all. > > > > I would rather discuss those than "bzr is tied to linux", because I > > don't think they are based on accurate or recent informations. As I > > said, I have bzr imports of scipy and scikits, and I could easily to > > the same for hg, make them available for everybody to play with. > > Instead of devising our own arguments, read this: > > http://bazaar-vcs.org/BzrVsHg > > and the mercurial response therein. I personally do not find those pages (both mercurial and bzr) really informative. Bzr page sounds too much like advertisement, and the answer from mercurial's team, logically, sounds like an agressive defense. David > > Ondrej > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Fri Jan 4 16:36:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 14:36:21 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> Message-ID: On Jan 4, 2008 2:05 PM, David Cournapeau wrote: > On Jan 5, 2008 5:36 AM, Charles R Harris > wrote: > > > > > > > > A quick google for benchmarks show that a year ago, hg was a bit faster > and > > generated smaller repositories than bzr, but I don't think the > difference is > > enough to matter. > > Forget a year ago, because as far as bzr is concerned, they got much > faster (several times faster for common operations like > commit/branch/log/merge). Sure, that's why I mentioned the time. Bzr used to claim better directory renames than hg, but that is not the case since version 9.4. So on and so forth. They are both moving targets. bzr is not tied to linux. It is, in that development is funded by Canonical, but I haven't used either on windows, so don't have any idea how they compare in that regard. They always have win32 binaries, TortoiseBzr > has a longer history than the mercurial one, and as said previously, > one developer of bzr at least is mainly a windows user. I don't want > to sound like I defend bzr, because honestly, I don't care about which > one is used, but so far, the arguments I heard against bzr do not > reflect my experience at all. > > One thing that bzr tries hard is the general UI, and the explicit > support for several workflows (with moderately advanced concepts such > as shared repositories, bound branches: for example, with a branch A > bound to branch B, a commit is first pushed on branch B, and if > successfull, applied to A; for centralized worflows, this makes things > easier). Hg has always recommended a similar process: clone the repository, push your changes to the clone, fix what needs fixing, and commit. It's not an atomic operation, though. I don't know where things are in that regard at the moment. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdb at cloud9.net Fri Jan 4 16:37:39 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 16:37:39 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: On Fri, 4 Jan 2008, Stuart Brorson wrote: >> I just discovered this today. It looks like a bug to me. Please >> flame me mercilessly if I am wrong! :-) FWIW, here's what Matlab does: >> A = rand(1, 4) + rand(1, 4)*i A = Columns 1 through 3 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i Column 4 0.5678 + 0.0503i >> B = zeros(1, 4) B = 0 0 0 0 >> for idx=1:4; B(idx) = A(idx); end >> B B = Columns 1 through 3 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i Column 4 0.5678 + 0.0503i I realize NumPy != Matlab, but I'd wager that most users would think that this is the natural behavior...... Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From charlesr.harris at gmail.com Fri Jan 4 16:41:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 14:41:56 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> Message-ID: On Jan 4, 2008 2:17 PM, Ondrej Certik wrote: > > > Instead of devising our own arguments, read this: > > http://bazaar-vcs.org/BzrVsHg > > and the mercurial response therein. I saw that, but thought it is more marketing than technical. Turned me off, actually, last thing I want is someone selling me toothpaste recommended by nine out of ten dentists. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Jan 4 17:02:15 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 4 Jan 2008 17:02:15 -0500 Subject: [Numpy-discussion] pre-initialized arrays In-Reply-To: References: Message-ID: On Fri, 4 Jan 2008, Stuart Brorson apparently wrote: > I realize NumPy != Matlab, but I'd wager that most users > would think that this is the natural behavior. I would not find it "natural" that elements of my float array could be assigned complex values. How could it be a fixed chunk of memory and do such things, unless it would always waste enough memory to hold the biggest possible subsequent data type? Cheers, Alan Isaac (user) From sdb at cloud9.net Fri Jan 4 17:03:10 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 17:03:10 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <200801041626.00682.pgmdevlist@gmail.com> References: <200801041626.00682.pgmdevlist@gmail.com> Message-ID: >> Sometimes you need to initialize an array using zeros() before doing >> an assignment to it in a loop. If you assign a complex value to the >> initialized array, the imaginary part of the array is dropped. Does >> NumPy do a silent type-cast which causes this behavior? Is this >> typecast a feature? > > By default, empty arrays are initialized as float. If you try to force a > complex into it, yes, the imaginary part will be dropped. Use the dtype > argument of zeros or ones to specify the type of your array, for example: > > B = numpy.zeros((4,), dtype=numpy.complex_) Hmmmm, I'll try this. Thanks! I still think that dumb users like me are more likely to assume that the destination array will become complex when you assign complex to it. Is there an articulated philosophy of how assignments (and any accompanying typecasting) should work in Numpy? Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From robert.kern at gmail.com Fri Jan 4 17:09:08 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 16:09:08 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> Message-ID: <477EAE84.2030600@gmail.com> Charles R Harris wrote: > On Jan 4, 2008 2:05 PM, David Cournapeau > wrote: > bzr is not tied to linux. > > It is, in that development is funded by Canonical, but I haven't used > either on windows, so don't have any idea how they compare in that regard. In that sense, it's all pretty much a wash between the three. Selenic initially developed Mercurial in the aftermath of the Linux kernel Bitkeeper foofoorah, and they continue to use it to manage their kernel modules. If we want to talk about Windows support, we should stick to more concrete facts (like the availability of Windows shell integration, etc.) instead of vague inferences. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Fri Jan 4 17:09:20 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 07:09:20 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> Message-ID: <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> On Jan 5, 2008 6:41 AM, Charles R Harris wrote: > > > On Jan 4, 2008 2:17 PM, Ondrej Certik wrote: > > > > > > > > > > > > > > Instead of devising our own arguments, read this: > > > > http://bazaar-vcs.org/BzrVsHg > > > > and the mercurial response therein. > > I saw that, but thought it is more marketing than technical. Turned me off, > actually, last thing I want is someone selling me toothpaste recommended by > nine out of ten dentists. > I agree. I find those pages to be really bad, actually. To have better informations, you should get into the mailing list of the respective projects. David From sdb at cloud9.net Fri Jan 4 17:10:16 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 17:10:16 -0500 (EST) Subject: [Numpy-discussion] pre-initialized arrays In-Reply-To: References: Message-ID: >> I realize NumPy != Matlab, but I'd wager that most users >> would think that this is the natural behavior. > > I would not find it "natural" that elements of my float > array could be assigned complex values. OK, then NumPy should throw an exception if you try to make the assignemnt. I tried it out. NumPy does the right thing in this case: In [10]: A = numpy.zeros([3, 3]) In [11]: A[1, 1] = 1+2j --------------------------------------------------------------------------- Traceback (most recent call last) /fs/home/sdb/ in () : can't convert complex to float; use abs(z) However, not in this case: In [12]: B = 10*numpy.random.rand(3, 3) + 1j*numpy.random.rand(3, 3) In [13]: B[1, 1] Out[13]: (7.15013107181+0.383220559014j) In [14]: A[1, 1] = B[1, 1] In [15]: So the bug is that assignment doesn't do value checking in every case. > How could it be > a fixed chunk of memory and do such things, unless it would > always waste enough memory to hold the biggest possible > subsequent data type? Fair question. Matlab does a realloc if you assign complex values to an initially real array. It can take a long time if your matrices are large. Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From robert.kern at gmail.com Fri Jan 4 17:10:49 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 16:10:49 -0600 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: <477EAEE9.1090501@gmail.com> Stuart Brorson wrote: > On Fri, 4 Jan 2008, Stuart Brorson wrote: > >>> I just discovered this today. It looks like a bug to me. Please >>> flame me mercilessly if I am wrong! :-) > > FWIW, here's what Matlab does: > >>> A = rand(1, 4) + rand(1, 4)*i > > A = > > Columns 1 through 3 > > 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i > > Column 4 > > 0.5678 + 0.0503i > >>> B = zeros(1, 4) > > B = > > 0 0 0 0 > >>> for idx=1:4; B(idx) = A(idx); end >>> B > > B = > > Columns 1 through 3 > > 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i > > Column 4 > > 0.5678 + 0.0503i > > > I realize NumPy != Matlab, but I'd wager that most users would think > that this is the natural behavior...... Well, that behavior won't happen. We won't mutate the dtype of the array because of assignment. Matlab has copy(-on-write) semantics for things like slices while we have view semantics. We can't safely do the reallocation of memory [1]. [1] Well, we do have a .resize() method which will do the reallocation and raise an exception if there are views lingering about. However, this is only done when explicitly asked for because this is a feature that is useful in a limited number of situations. We will not allow it to be done implicitly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthew.brett at gmail.com Fri Jan 4 17:15:33 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Jan 2008 14:15:33 -0800 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> Message-ID: <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> > I agree. I find those pages to be really bad, actually. To have better > informations, you should get into the mailing list of the respective > projects. Just to extend this holiday special: I found the mozilla DVCS discussion informative: http://weblogs.mozillazine.org/preed/2007/04/version_control_system_shootou_1.html (but graphically embarrassing) Matthew From cournape at gmail.com Fri Jan 4 17:16:04 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 07:16:04 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> Message-ID: <5b8d13220801041416r595cd332l2e059f1893422f2@mail.gmail.com> On Jan 5, 2008 6:36 AM, Charles R Harris wrote: > > > > On Jan 4, 2008 2:05 PM, David Cournapeau wrote: > > > > On Jan 5, 2008 5:36 AM, Charles R Harris > wrote: > > > > > > > > > > > > > A quick google for benchmarks show that a year ago, hg was a bit faster > and > > > generated smaller repositories than bzr, but I don't think the > difference is > > > enough to matter. > > > > Forget a year ago, because as far as bzr is concerned, they got much > > faster (several times faster for common operations like > > commit/branch/log/merge). > > Sure, that's why I mentioned the time. Bzr used to claim better directory > renames than hg, but that is not the case since version 9.4. So on and so > forth. They are both moving targets. Yes, I agree on the moving targets. > > > > bzr is not tied to linux. > > It is, in that development is funded by Canonical, but I haven't used either > on windows, so don't have any idea how they compare in that regard. > Being funded by Canonical does not mean it is tied to linux. For example, some people are working on handling case insensitive fs, which is not something a tied-to-linux tool would care about. That's not something we care about, though. Both hg and bzr are developed mainly on linux, by linux developers (typically, the fact that none of them have an official, complete GUI shows that they were not born on windows). bzr works as well on windows as it does on linux (I use bzr on windows for the numscons project, for example), but AFAIK, so does mercurial, so this is not an important point. > > > > Hg has always recommended a similar process: clone the repository, push your > changes to the clone, fix what needs fixing, and commit. It's not an atomic > operation, though. I don't know where things are in that regard at the > moment. The point is really about the one operation; otherwise, any DVCS can do it. Honestly, I do not find it such a useful feature, but maybe it shows when used by many different people ? IMHO, the only really important points are how to convert the current history, and trac integration. All other differences look quite minor to me. David From sdb at cloud9.net Fri Jan 4 17:17:56 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 17:17:56 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <477EAEE9.1090501@gmail.com> References: <477EAEE9.1090501@gmail.com> Message-ID: >> I realize NumPy != Matlab, but I'd wager that most users would think >> that this is the natural behavior...... > > Well, that behavior won't happen. We won't mutate the dtype of the array because > of assignment. Matlab has copy(-on-write) semantics for things like slices while > we have view semantics. We can't safely do the reallocation of memory [1]. That's fair enough. But then I think NumPy should consistently typecheck all assignmetns and throw an exception if the user attempts an assignment which looses information. If you point me to a file where assignments are done (particularly from array elements to array elements) I can see if I can figure out how to fix it & then submit a patch. But I won't promise anything! My brain hurts already after analyzing this "feature"..... :-) Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From cournape at gmail.com Fri Jan 4 17:25:14 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Jan 2008 07:25:14 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> Message-ID: <5b8d13220801041425o62c33e16j964d15c7fb23ced@mail.gmail.com> On Jan 5, 2008 7:15 AM, Matthew Brett wrote: > > I agree. I find those pages to be really bad, actually. To have better > > informations, you should get into the mailing list of the respective > > projects. > > Just to extend this holiday special: > > I found the mozilla DVCS discussion informative: > > http://weblogs.mozillazine.org/preed/2007/04/version_control_system_shootou_1.html > > (but graphically embarrassing) > The open solaris project documented their choice, too: http://www.opensolaris.org/os/community/tools/scm/history/ Contrary to mozilla, solaris is using hg as the main VCS. Again, a thing to keep in mind with the mozilla thing is that bzr improved a lot speed-wise, and that mozilla is several order of magnitudes bigger than scipy will ever be. To test the svn import by hg, svnsync seems to be necessary: I don't go anywhere by using hg convert directly on scipy repository, whereas bzr-svn does. cheers, David > Matthew > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From sransom at nrao.edu Fri Jan 4 17:28:07 2008 From: sransom at nrao.edu (Scott Ransom) Date: Fri, 4 Jan 2008 17:28:07 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> Message-ID: <200801041728.07703.sransom@nrao.edu> On Friday 04 January 2008 05:17:56 pm Stuart Brorson wrote: > >> I realize NumPy != Matlab, but I'd wager that most users would > >> think that this is the natural behavior...... > > > > Well, that behavior won't happen. We won't mutate the dtype of the > > array because of assignment. Matlab has copy(-on-write) semantics > > for things like slices while we have view semantics. We can't > > safely do the reallocation of memory [1]. > > That's fair enough. But then I think NumPy should consistently > typecheck all assignmetns and throw an exception if the user attempts > an assignment which looses information. > > If you point me to a file where assignments are done (particularly > from array elements to array elements) I can see if I can figure out > how to fix it & then submit a patch. But I won't promise anything! > My brain hurts already after analyzing this "feature"..... :-) There is a long history in numeric/numarray/numpy about this "feature". And for many of us, it really is a feature -- it prevents the automatic upcasting of arrays, which is very important if your arrays are huge (i.e. comparable in size to your system memory). For instance in astronomy, where very large 16-bit integer or 32-bit float images or data-cubes are common, if you upcast your 32-bit floats accidentally because you are doing double precision math (i.e. the default in Python) near them, that can cause the program to swap out or die horribly. In fact, this exact example is one of the reasons why the Space Telescope people initially developed numarray. numpy has kept that model. I agree, though, that when using very mixed types (i.e. complex and ints, for example), the results can be confusing. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From charlesr.harris at gmail.com Fri Jan 4 17:53:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2008 15:53:41 -0700 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: On Jan 4, 2008 2:37 PM, Stuart Brorson wrote: > On Fri, 4 Jan 2008, Stuart Brorson wrote: > > > > I realize NumPy != Matlab, but I'd wager that most users would think > that this is the natural behavior...... Matlab support for different types was sort of kludged on. Matlab was intended for computational convenience, not control of data types, and started out as pretty much all doubles and double complex, it didn't (doesn't?) even have integers. NumPy is a bit closer to the metal and support for different data types has been there from the start, but that extra power means you have to devote a bit more thought to what you are doing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Fri Jan 4 18:22:10 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 5 Jan 2008 00:22:10 +0100 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: Message-ID: <20080104232210.GA14286@clipper.ens.fr> On Fri, Jan 04, 2008 at 03:53:41PM -0700, Charles R Harris wrote: > Matlab support for different types was sort of kludged on. Matlab was > intended for computational convenience, not control of data types, and > started out as pretty much all doubles and double complex, it didn't > (doesn't?) even have integers. In has integers arrays, and does not upcast to double arrays. My experience is that you are better off avoiding these (unless you talk to hardware, which is why I use them) because many things won't work with them. Cheers, Ga?l From tim.hochberg at ieee.org Fri Jan 4 18:31:53 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 4 Jan 2008 16:31:53 -0700 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <200801041728.07703.sransom@nrao.edu> References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> Message-ID: On Jan 4, 2008 3:28 PM, Scott Ransom wrote: > On Friday 04 January 2008 05:17:56 pm Stuart Brorson wrote: > > >> I realize NumPy != Matlab, but I'd wager that most users would > > >> think that this is the natural behavior...... > > > > > > Well, that behavior won't happen. We won't mutate the dtype of the > > > array because of assignment. Matlab has copy(-on-write) semantics > > > for things like slices while we have view semantics. We can't > > > safely do the reallocation of memory [1]. > > > > That's fair enough. But then I think NumPy should consistently > > typecheck all assignmetns and throw an exception if the user attempts > > an assignment which looses information. > > > > If you point me to a file where assignments are done (particularly > > from array elements to array elements) I can see if I can figure out > > how to fix it & then submit a patch. But I won't promise anything! > > My brain hurts already after analyzing this "feature"..... :-) > > There is a long history in numeric/numarray/numpy about this "feature". > And for many of us, it really is a feature -- it prevents the automatic > upcasting of arrays, which is very important if your arrays are huge > (i.e. comparable in size to your system memory). > > For instance in astronomy, where very large 16-bit integer or 32-bit > float images or data-cubes are common, if you upcast your 32-bit floats > accidentally because you are doing double precision math (i.e. the > default in Python) near them, that can cause the program to swap out or > die horribly. In fact, this exact example is one of the reasons why > the Space Telescope people initially developed numarray. numpy has > kept that model. I agree, though, that when using very mixed types > (i.e. complex and ints, for example), the results can be confusing. > This isn't a very compelling argument in this case. The concern the numarray people were addressing was the upcasting of precision. However, there are two related hierarchies in numpy, one is the kind[1] of data, roughly: bool, int, float, complex. Each kind has various precisions. The numarray folks were concerned with avoiding upcasting of precision, not with avoiding upcasting up kinds. And, I can't see much (any?) justification for allowing automagic downcasting of kind, complex->float being the most egregious, other than backwards compatibility. This is clearly an opportunity for confusion and likely a magnet for bug. And, I've yet to see any useful examples to support this behaviour. I imagine that their are some benifits, but I doubt that they are compelling enough to justify the current behaviour. [1] I can't recall if this is the official terminology; I'm away from my home computer at the moment and it's hard for me to check. The idea should be clear however, -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdb at cloud9.net Fri Jan 4 19:45:19 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Fri, 4 Jan 2008 19:45:19 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <200801041728.07703.sransom@nrao.edu> References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> Message-ID: >>>> I realize NumPy != Matlab, but I'd wager that most users would >>>> think that this is the natural behavior...... >>> >>> Well, that behavior won't happen. We won't mutate the dtype of the >>> array because of assignment. Matlab has copy(-on-write) semantics >>> for things like slices while we have view semantics. We can't >>> safely do the reallocation of memory [1]. >> >> That's fair enough. But then I think NumPy should consistently >> typecheck all assignmetns and throw an exception if the user attempts >> an assignment which looses information. >> > There is a long history in numeric/numarray/numpy about this "feature". > And for many of us, it really is a feature -- it prevents the automatic > upcasting of arrays, which is very important if your arrays are huge > (i.e. comparable in size to your system memory). That's well and good. But NumPy should *never* automatically -- and silently -- chop the imaginary part off your complex array elements, particularly if you are just doing an innocent assignment! Doing something drastic like silently throwing half your data away can lead to all kinds of bugs in code written by somebody who is unaware of this behavior (i.e. most people)! It sounds to me like the right thing is to throw an exception instead of "downcasting" a data object. Stuart From zpincus at stanford.edu Fri Jan 4 20:15:53 2008 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri, 4 Jan 2008 20:15:53 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> Message-ID: Hello all, > That's well and good. But NumPy should *never* automatically -- and > silently -- chop the imaginary part off your complex array elements, > particularly if you are just doing an innocent assignment! > Doing something drastic like silently throwing half your data away can > lead to all kinds of bugs in code written by somebody who is unaware > of this behavior (i.e. most people)! > > It sounds to me like the right thing is to throw an exception instead > of "downcasting" a data object. I'm not sure that I agree! I'd rather not have to litter my code with "casting" operations every time I wanted to down-convert data types (creating yet more temporary arrays...) via assignment. e.g.: A[i] = calculate(B).astype(A.dtype) vs. A[i] = calculate(B) Further, writing functions to operate on generic user-provided output arrays (or arrays of user-provided dtype; both of which are common e.g. in scipy.ndimage) becomes more bug-prone, as every assignment would need to be modified as above. This change would break a lot of the image-processing code I've written (where one often does computations in float or even double, and then re-scales and down-converts the result to integer for display), for example. I guess that this could be rolled in via the geterr/seterr mechanism, and could by default just print warnings. I agree that silent truncation can be troublesome, but not having to spell every conversion out in explicit ( and error-prone) detail is also pretty useful. (And arguably more pythonic.) Zach From robert.kern at gmail.com Fri Jan 4 20:36:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 04 Jan 2008 19:36:30 -0600 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> Message-ID: <477EDF1E.901@gmail.com> Zachary Pincus wrote: > Hello all, > >> That's well and good. But NumPy should *never* automatically -- and >> silently -- chop the imaginary part off your complex array elements, >> particularly if you are just doing an innocent assignment! >> Doing something drastic like silently throwing half your data away can >> lead to all kinds of bugs in code written by somebody who is unaware >> of this behavior (i.e. most people)! >> >> It sounds to me like the right thing is to throw an exception instead >> of "downcasting" a data object. > > I'm not sure that I agree! I'd rather not have to litter my code with > "casting" operations every time I wanted to down-convert data types > (creating yet more temporary arrays...) via assignment. e.g.: > > A[i] = calculate(B).astype(A.dtype) > vs. > A[i] = calculate(B) > > Further, writing functions to operate on generic user-provided output > arrays (or arrays of user-provided dtype; both of which are common > e.g. in scipy.ndimage) becomes more bug-prone, as every assignment > would need to be modified as above. > > This change would break a lot of the image-processing code I've > written (where one often does computations in float or even double, > and then re-scales and down-converts the result to integer for > display), for example. > > I guess that this could be rolled in via the geterr/seterr mechanism, > and could by default just print warnings. I agree that silent > truncation can be troublesome, but not having to spell every > conversion out in explicit ( and error-prone) detail is also pretty > useful. (And arguably more pythonic.) There's some confusion in the conversation here. Tim already identified it, but since it's continuing, I'm going to reiterate. There are two related hierarchies of datatypes: different kinds (integer, floating point, complex floating point) and different precisions within a given kind (int8, int16, int32, int64). The term "downcasting" should probably be reserved for the latter only. It seems to me that Zach and Scott are defending downcasting of precisions within a given kind. It does not necessarily follow that the behavior we choose for dtypes within a given kind should be the behavior when we are dealing with dtypes across different kinds. We can keep the precision downcasting behavior that you want while raising an error when one attempts to assign a complex number into a floating point array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sransom at nrao.edu Fri Jan 4 20:42:16 2008 From: sransom at nrao.edu (Scott Ransom) Date: Fri, 4 Jan 2008 20:42:16 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> Message-ID: <20080105014216.GA32661@ssh.cv.nrao.edu> On Fri, Jan 04, 2008 at 04:31:53PM -0700, Timothy Hochberg wrote: > On Jan 4, 2008 3:28 PM, Scott Ransom wrote: > > > On Friday 04 January 2008 05:17:56 pm Stuart Brorson wrote: > > > >> I realize NumPy != Matlab, but I'd wager that most users would > > > >> think that this is the natural behavior...... > > > > > > > > Well, that behavior won't happen. We won't mutate the dtype of the > > > > array because of assignment. Matlab has copy(-on-write) semantics > > > > for things like slices while we have view semantics. We can't > > > > safely do the reallocation of memory [1]. > > > > > > That's fair enough. But then I think NumPy should consistently > > > typecheck all assignmetns and throw an exception if the user attempts > > > an assignment which looses information. > > > > > > If you point me to a file where assignments are done (particularly > > > from array elements to array elements) I can see if I can figure out > > > how to fix it & then submit a patch. But I won't promise anything! > > > My brain hurts already after analyzing this "feature"..... :-) > > > > There is a long history in numeric/numarray/numpy about this "feature". > > And for many of us, it really is a feature -- it prevents the automatic > > upcasting of arrays, which is very important if your arrays are huge > > (i.e. comparable in size to your system memory). > > > > For instance in astronomy, where very large 16-bit integer or 32-bit > > float images or data-cubes are common, if you upcast your 32-bit floats > > accidentally because you are doing double precision math (i.e. the > > default in Python) near them, that can cause the program to swap out or > > die horribly. In fact, this exact example is one of the reasons why > > the Space Telescope people initially developed numarray. numpy has > > kept that model. I agree, though, that when using very mixed types > > (i.e. complex and ints, for example), the results can be confusing. > > > > This isn't a very compelling argument in this case. The concern the numarray > people were addressing was the upcasting of precision. However, there are > two related hierarchies in numpy, one is the kind[1] of data, roughly: bool, > int, float, complex. Each kind has various precisions. The numarray folks > were concerned with avoiding upcasting of precision, not with avoiding > upcasting up kinds. And, I can't see much (any?) justification for allowing > automagic downcasting of kind, complex->float being the most egregious, > other than backwards compatibility. This is clearly an opportunity for > confusion and likely a magnet for bug. And, I've yet to see any useful > examples to support this behaviour. I imagine that their are some benifits, > but I doubt that they are compelling enough to justify the current > behaviour. I wasn't at all arguing that having complex data chopped and downcast into an int or float container was the right thing to do. What I was trying to address was that preventing automatic upcasting of the "kind" of data that you have is often very useful and in fact was one of the driving reasons behind numarray. For this particular complex-type example I think that an exception would be the proper thing, since if you really want to throw away the imaginary part it is easy enough to specify x.real. But the float->int situation is different. I can see good reasons for both upcast and downcast. This is one of those situations where the programmer had better be sure that they know what they are doing (and being explicit in the code would be better than implicit). Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From Chris.Barker at noaa.gov Sat Jan 5 02:17:51 2008 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri, 04 Jan 2008 23:17:51 -0800 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <20080105014216.GA32661@ssh.cv.nrao.edu> References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> <20080105014216.GA32661@ssh.cv.nrao.edu> Message-ID: <477F2F1F.5010700@noaa.gov> Scott Ransom wrote: > I wasn't at all arguing that having complex data chopped and > downcast into an int or float container was the right thing to do. indeed, it is an clearly bad thing to do -- but a bug magnet? I'm not so sure, surely, anyone that had any idea at all what they were doing with complex numbers would notice it right away. To the OP: Did you waste any time thinking this was working right? Or was your time spent figuring out why numpy wold do such a thing? which is wasted time none the less. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Sat Jan 5 02:21:19 2008 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri, 04 Jan 2008 23:21:19 -0800 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> Message-ID: <477F2FEF.5050005@noaa.gov> hmmm. Everyone posting so far seems to be positive on this idea, but I'm not so sure. A few thoughts: 1) change is bad. It may be worth it, but this decision needs to be made very differently than if we were starting from scratch. 2) apparently svn merge sucks compared to other merge technology. svn (and cvs before it) is the only system I'm used, and yes, merging is painful, but I have to say that it appeared to be painful because it's a very hard problem. Can anyone comment on why these other systems seem so much better? Does it have anything to do with Centralized vs. Distributed at all? 3) I read Linus' post -- he's quite articulate. However, it seems that most of his arguments really applied primarily to large projects -- where there really will be a lot of different "central" versions. This is very, very, important to the Linux kernel, and probably good for kde, but scipy is a monstrously smaller community. And it's not a question of number of devs -- but rather number of versions. This makes me thing it really comes down to a better merge -- is there a way to address that problem with svn? maybe the svnmerge.py that Russel suggested? 4) SVN is very, very, popular. Lots of folks use it, they use it on all common platforms, and there are tons of clients for it. I work with a bunch of folks that really don't like a command line (for programmers, I think that's just plain weird, but there you go). I could never sell a VCS that didn't have a decent GUI client on Windows and OS-X. Charles R Harris wrote: > Sometimes it is the little niggling things that matter, in this case > line breaks. Hg (and probably bzr), store everything as binary, so if > someone uses an editor that breaks line with CR or LF+CR instead of the > good 'ol unixy LF, there might be a lot whitespace updates coming in to > the repository. Good point. In fact, line break translation was one of the features I loved about cvs -- too bad svn doesn't do it by default, but it can be made to. This is definitely one of the niggling things that matter! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Sat Jan 5 02:37:34 2008 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri, 04 Jan 2008 23:37:34 -0800 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477E813C.8060803@enthought.com> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> <477E813C.8060803@enthought.com> Message-ID: <477F33BE.8020404@noaa.gov> Travis E. Oliphant wrote: > I like the depth option, and that could be used with the current > interface without too much difficulty, I think. > > This question does come up quite often and should probably be addressed. Yes, it has, but the depth option was discussed too, and while it might solve some problems, it would still be ambiguous: What if your "objects" were nested sequences, and you wanted to partly flatten them -- which would you flatten? Maybe that's a weird enough case that the depth option would work most of the time, but I'm not sure. I think a "shape" option might cover this better. It seems the case at hand is how to make a custom object that can tell numpy how it should be put into an object array -- that may be an even rarer use case, but it seems like a good idea to support it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Sat Jan 5 02:38:01 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 05 Jan 2008 01:38:01 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477F2FEF.5050005@noaa.gov> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> <477F2FEF.5050005@noaa.gov> Message-ID: <477F33D9.4020705@gmail.com> Chris Barker wrote: > hmmm. Everyone posting so far seems to be positive on this idea, but I'm > not so sure. A few thoughts: > > 1) change is bad. It may be worth it, but this decision needs to be made > very differently than if we were starting from scratch. > > 2) apparently svn merge sucks compared to other merge technology. svn > (and cvs before it) is the only system I'm used, and yes, merging is > painful, but I have to say that it appeared to be painful because it's a > very hard problem. Can anyone comment on why these other systems seem so > much better? Does it have anything to do with Centralized vs. > Distributed at all? Tangentially, yes. DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. It's worth noting that SVN 1.5 will be tracking such information. But that release is a long ways off. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Jan 5 02:45:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 05 Jan 2008 01:45:45 -0600 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477F33BE.8020404@noaa.gov> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> <477E813C.8060803@enthought.com> <477F33BE.8020404@noaa.gov> Message-ID: <477F35A9.9040000@gmail.com> Chris Barker wrote: > Travis E. Oliphant wrote: > >> I like the depth option, and that could be used with the current >> interface without too much difficulty, I think. >> >> This question does come up quite often and should probably be addressed. > > Yes, it has, but the depth option was discussed too, and while it might > solve some problems, it would still be ambiguous: > > What if your "objects" were nested sequences, and you wanted to partly > flatten them -- which would you flatten? I'm pretty sure that that is exactly the ambiguity that the depth option resolves. Can you give me an example where it's still ambiguous with the depth information provided? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sdb at cloud9.net Sat Jan 5 07:25:14 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Sat, 5 Jan 2008 07:25:14 -0500 (EST) Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <477F2F1F.5010700@noaa.gov> References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> <20080105014216.GA32661@ssh.cv.nrao.edu> <477F2F1F.5010700@noaa.gov> Message-ID: >> I wasn't at all arguing that having complex data chopped and >> downcast into an int or float container was the right thing to do. > > indeed, it is an clearly bad thing to do -- but a bug magnet? I'm not so > sure, surely, anyone that had any idea at all what they were doing with > complex numbers would notice it right away. > > To the OP: Did you waste any time thinking this was working right? Or > was your time spent figuring out why numpy wold do such a thing? which > is wasted time none the less. Thanks for the question. I spent about 1/2 hour looking at my other code trying to figure out why I was getting strange results. Of course, I suspected my own code to be the culpret, since NumPy is a mature package, right?. Then, when I looked closely at the return array given by NumPy, I noticed that it was real, when I was working with complex numbers. I said to myself "WTF?". Then a little more investigating revealed that NumPy was silently truncating my complex array to real. I respectfully submit that silently chopping the imaginary part *is* a magnet for bugs, since many dumb NumPy users (like me) aren't necessarily aware of the typecasting rules. We're only thinking about doing math, after all! Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From zpincus at stanford.edu Sat Jan 5 12:56:52 2008 From: zpincus at stanford.edu (Zachary Pincus) Date: Sat, 5 Jan 2008 12:56:52 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <477EDF1E.901@gmail.com> References: <477EAEE9.1090501@gmail.com> <200801041728.07703.sransom@nrao.edu> <477EDF1E.901@gmail.com> Message-ID: > There are two related hierarchies of datatypes: different kinds > (integer, > floating point, complex floating point) and different precisions > within a given > kind (int8, int16, int32, int64). The term "downcasting" should > probably be > reserved for the latter only. > > It seems to me that Zach and Scott are defending downcasting of > precisions > within a given kind. It does not necessarily follow that the > behavior we choose > for dtypes within a given kind should be the behavior when we are > dealing with > dtypes across different kinds. We can keep the precision > downcasting behavior > that you want while raising an error when one attempts to assign a > complex > number into a floating point array. Actually, my points were explicitly about cross-kind conversions! A lot of image-processing code features heavy use of converting integer images to float for intermediate calculations, and then rescaling and down-converting back to the original type for display, etc. In fact, scipy.ndimage makes use of this, allowing for operations with output into user-specified arrays, or arrays of user-specified dtype, while (I believe) carrying out some of the intermediate calculations at higher precision. As such, pretty much any code that takes a user-specified array and assigns to it the result of a (potentially) mixed-mode calculation would need to be changed from: A[i] = calculate(B) to A[i] = calculate(B).astype(A.dtype) with the proliferation of temp arrays that that entails. I think, but am not sure, that there is a lot of code out there that does this, intentionally, which would be broken by this change. Explicit is indeed better than implicit, but in this case, finding all of the places where mixed-mode conversions happen and tracking them down could be a pain on the same scale as const chasing in C++, where fixing the error in one place makes it reappear elsewhere in the chain of usage, leading to long, painful, and often totally pointless debugging sessions. In the specific case of converting from complex to anything else, I can see the desire for a warning to prevent data loss. But converting from float to integer is a lot more routine and is used a lot in the kind of work that I do, at least. Zach From cookedm at physics.mcmaster.ca Sat Jan 5 13:44:54 2008 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sat, 5 Jan 2008 13:44:54 -0500 Subject: [Numpy-discussion] Error importing from numpy.matlib In-Reply-To: References: Message-ID: <9445228F-5C9B-4A80-872B-BE7602449C3B@physics.mcmaster.ca> On Jan 3, 2008, at 18:29 , Steve Lianoglou wrote: > > Anyway, somewhere in my codebase (for a long time now) I'm doing: > > from numpy.matlib import * > > Now, when I try to use this code, or just type that in the > interpreter, I get this message: > > AttributeError: 'module' object has no attribute 'pkgload' > > This doesn't happen when I do: > > import numpy.matlib as M > > Anyway, can anyone shed some light on this? This is the behaviour you'd get if the module's __all__ attribute lists objects which don't exist in the module. Looks like a regression in r4659; fixed in SVN now as r4674. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Sat Jan 5 14:08:46 2008 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sat, 5 Jan 2008 14:08:46 -0500 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: > My vote so far is for hg, for performance reasons but also partly > because sage and sympy already use it, two projects I'm likely to > interact a lot with and that are squarely in line with the > ipython/numpy/scipy/matplotlib world. Since they went first and made > the choice, I'm happy to let that be a factor in my decision. I'd > rather use a tool that others in the same community are also using, > especially when the choice is a sound one on technical merit alone. > > Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). One thing that can help is an official Mercurial mirror of the subversion repository. IIRC, sharing changegroups or pulling patches between hg repos requires that they have a common ancestor repo (as opposed to two developers independently converting the svn repo). -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From fperez.net at gmail.com Sat Jan 5 14:15:41 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 5 Jan 2008 12:15:41 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: On Jan 5, 2008 12:08 PM, David M. Cooke wrote: > On Jan 4, 2008, at 13:58 , Fernando Perez wrote: > > > My vote so far is for hg, for performance reasons but also partly > > because sage and sympy already use it, two projects I'm likely to > > interact a lot with and that are squarely in line with the > > ipython/numpy/scipy/matplotlib world. Since they went first and made > > the choice, I'm happy to let that be a factor in my decision. I'd > > rather use a tool that others in the same community are also using, > > especially when the choice is a sound one on technical merit alone. > > > > Just my 1e-2... > > > +1 on mercurial. It's what I use these days (previously, I used darcs, > which I still like for its patch-handling semantics, but its > dependence on Haskell, and the dreaded exponential-time merge are a > bit of a pain). Regarding the 'record' capapbilities of darcs which were indeed very nice, here's something that was recently mentioned on the sage list: """ I noticed that Mercurial 0.9.5 has a "record" extension that mimics the darcs record functionality of interactively asking what changes you want to commit out of a file. I know there was discussion of this a while ago. Reference: http://www.selenic.com/pipermail/mercurial/2007-October/015150.html under the New extensions heading. See also http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension Anyways, I'm just posting this as an FYI. It might be nice to expose this functionality to sage, if we haven't already. Thanks, Jason """ Cheers, f From ondrej at certik.cz Sat Jan 5 14:59:50 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 5 Jan 2008 20:59:50 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: <85b5c3130801051159j1166a494neb154e7692c35b6c@mail.gmail.com> On Jan 5, 2008 8:15 PM, Fernando Perez wrote: > On Jan 5, 2008 12:08 PM, David M. Cooke wrote: > > On Jan 4, 2008, at 13:58 , Fernando Perez wrote: > > > > > My vote so far is for hg, for performance reasons but also partly > > > because sage and sympy already use it, two projects I'm likely to > > > interact a lot with and that are squarely in line with the > > > ipython/numpy/scipy/matplotlib world. Since they went first and made > > > the choice, I'm happy to let that be a factor in my decision. I'd > > > rather use a tool that others in the same community are also using, > > > especially when the choice is a sound one on technical merit alone. > > > > > > Just my 1e-2... > > > > > > +1 on mercurial. It's what I use these days (previously, I used darcs, > > which I still like for its patch-handling semantics, but its > > dependence on Haskell, and the dreaded exponential-time merge are a > > bit of a pain). > > Regarding the 'record' capapbilities of darcs which were indeed very > nice, here's something that was recently mentioned on the sage list: > > """ > I noticed that Mercurial 0.9.5 has a "record" extension that mimics the > darcs record functionality of interactively asking what changes you want > to commit out of a file. I know there was discussion of this a while ago. > > Reference: > > http://www.selenic.com/pipermail/mercurial/2007-October/015150.html > under the New extensions heading. See also > http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension > > Anyways, I'm just posting this as an FYI. It might be nice to expose > this functionality to sage, if we haven't already. > > Thanks, > > Jason > """ Kirill (a sympy developer) has also sent patches for qrecord (record for mercurial queues) http://www.selenic.com/pipermail/mercurial-devel/2007-December/003953.html Ondrej From oliphant at enthought.com Sat Jan 5 16:00:21 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 05 Jan 2008 15:00:21 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> Message-ID: <477FEFE5.5050706@enthought.com> David M. Cooke wrote: > On Jan 4, 2008, at 13:58 , Fernando Perez wrote: > > >> My vote so far is for hg, for performance reasons but also partly >> because sage and sympy already use it, two projects I'm likely to >> interact a lot with and that are squarely in line with the >> ipython/numpy/scipy/matplotlib world. Since they went first and made >> the choice, I'm happy to let that be a factor in my decision. I'd >> rather use a tool that others in the same community are also using, >> especially when the choice is a sound one on technical merit alone. >> >> Just my 1e-2... >> > > > +1 on mercurial. It's what I use these days (previously, I used darcs, > which I still like for its patch-handling semantics, but its > dependence on Haskell, and the dreaded exponential-time merge are a > bit of a pain). > I don't think it is time to move wholesale to something like Mercurial or bzr. I would prefer it if all of the Enthought-hosted projects moved to the (new) system at once, which is not going to happen in the short term (but long term of course it's an open question). But, having an official mirror is a useful thing to explore. I suspect there are others with serious reservations about jumping off of SVN just now (just when a lot of people have finally figured out how to use it). If there is an Hg mirror that allows other to use mercurial at the same time then that sounds like a good idea to me. -Travis O. From lists.steve at arachnedesign.net Sat Jan 5 16:14:15 2008 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 5 Jan 2008 16:14:15 -0500 Subject: [Numpy-discussion] Error importing from numpy.matlib In-Reply-To: <9445228F-5C9B-4A80-872B-BE7602449C3B@physics.mcmaster.ca> References: <9445228F-5C9B-4A80-872B-BE7602449C3B@physics.mcmaster.ca> Message-ID: <218478D6-A302-467B-BA5B-2FA43877D62E@arachnedesign.net> > This is the behaviour you'd get if the module's __all__ attribute > lists objects which don't exist in the module. Looks like a regression > in r4659; fixed in SVN now as r4674. Thanks! -steve From stefan at sun.ac.za Sat Jan 5 18:25:00 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 6 Jan 2008 01:25:00 +0200 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477FEFE5.5050706@enthought.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> Message-ID: <20080105232500.GI7649@mentat.za.net> On Sat, Jan 05, 2008 at 03:00:21PM -0600, Travis E. Oliphant wrote: > I suspect there are others with serious reservations about jumping off > of SVN just now (just when a lot of people have finally figured out how > to use it). I recall something you said to David last week, regarding merges with SVN: that a person never knows how to do it until *after* you've done it! We often make branches in scipy and numpy, and stand a lot to gain from a distributed RCS. Once a person knows how to use SVN, it doesn't take much effort at all to learn bzr or hg (even the commands are often the same). The main change is a mind-shift: that branches are now a lot friendlier, and that they are accessable to everybody. At the end of 2005, back when I was still working with Octave, we had a discussion on the merits of switching over to Subversion. That conversation never went anywhere, which is the reason you can still obtain Octave today using cvs -d :ext:anoncvs at www.octave.org:/cvs I know there are reservations about doing the switch *right now*, which is fine -- we must just not wait too long. Regards St?fan From fperez.net at gmail.com Sat Jan 5 18:47:38 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 5 Jan 2008 16:47:38 -0700 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <20080105232500.GI7649@mentat.za.net> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <20080105232500.GI7649@mentat.za.net> Message-ID: I'd like to briefly provide a different perspective on this question, which is not a technical one but a more social/process one. It seems to me (but I could be wrong; this is opinion, not research!) that a DVCS encourages a more open participation model for newcomers. Since anyone with a checkout has the same tree, there is no more 'us vs. them' in the sense of 'developers vs users'. Yes, with SVN anyone can track trunk or branches and submit a patch, but there's a distinct asymmetry in the process that DVCS remove (bviously even with a DVCS model there always be a canonical repository that is considered official, and to which only a group with commit rights can push changes). In addition, DVCS allow more easily the creation of subgroups of parallel developers who share their branches and explore ideas, subprojects, optimizations, etc. With a DVCS, anyone can join such a subgroup, contribute, and if that idea bears fruit, it's easy to fold it back into the official trunk. SVN doesn't really lend itself well at all to this type of approach, and I think it therefore tends to lower the amount of intellectual exploration a project is likely to do during its lifetime. So I'd venture that a DVCS can benefit a project in the long run by lowering the tunneling energy required to make the user->developer transition. Given how users who make this transition are the life and blood of any open source project, I'd argue that anything that helps this is worth considering. Obviously the above is not an argument for doing anything *now*, as for many reasons now may not be the right time. But it is to me a compelling argument for taking the step, leaving only the when and which specific tool as decisions to be appropriately determined. Of course, I could be fully wrong, since the above is little more than common-sense-sounding speculation. Cheers, f From wbaxter at gmail.com Sat Jan 5 18:55:00 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 6 Jan 2008 08:55:00 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <20080105232500.GI7649@mentat.za.net> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <20080105232500.GI7649@mentat.za.net> Message-ID: On Jan 6, 2008 8:25 AM, Stefan van der Walt wrote: > I recall something you said to David last week, regarding merges with > SVN: that a person never knows how to do it until *after* you've done > it! We often make branches in scipy and numpy, and stand a lot to > gain from a distributed RCS. > > Once a person knows how to use SVN, it doesn't take much effort at all > to learn bzr or hg (even the commands are often the same). The main > change is a mind-shift: that branches are now a lot friendlier, and > that they are accessable to everybody. I understand that DVCS's do merging better. But what I don't really understand is why this is an inherent advantage of DVCS. Isnt it just a matter of the current crop of DVCS's implementing a better merge algorithm than SVN? The SVN guys seem to be competent, so if you just give them time surely they will eventually incorporate these better merging algorithms into SVN. Who wouldn't want better merging? --bb From ondrej at certik.cz Sat Jan 5 19:02:40 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 6 Jan 2008 01:02:40 +0100 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <20080105232500.GI7649@mentat.za.net> Message-ID: <85b5c3130801051602n443d7ff7refdb299bd5729675@mail.gmail.com> On Jan 6, 2008 12:55 AM, Bill Baxter wrote: > On Jan 6, 2008 8:25 AM, Stefan van der Walt wrote: > > I recall something you said to David last week, regarding merges with > > SVN: that a person never knows how to do it until *after* you've done > > it! We often make branches in scipy and numpy, and stand a lot to > > gain from a distributed RCS. > > > > Once a person knows how to use SVN, it doesn't take much effort at all > > to learn bzr or hg (even the commands are often the same). The main > > change is a mind-shift: that branches are now a lot friendlier, and > > that they are accessable to everybody. > > I understand that DVCS's do merging better. But what I don't really > understand is why this is an inherent advantage of DVCS. Isnt it > just a matter of the current crop of DVCS's implementing a better > merge algorithm than SVN? The SVN guys seem to be competent, so if > you just give them time surely they will eventually incorporate these > better merging algorithms into SVN. Who wouldn't want better merging? It's not just about merging. But anyway, all arguments were already said in this thread. I fully agree with both Davids and Fernando. So let's setup an official mercurial mirror, that will automatically download all svn commits. That way, we can easily work with Mercurial, clone the repos, browse history, everything. Review patches. And then, when the patches are reviewed, instead of pushing them to the Mercurial repo, they will be committed using svn. No big deal, everyone is happy. We did it too in sympy, at the beginning, because we were afraid of switching (it was mainly me, who was afraid, because I am very conservative, that's why I use Debian:). But then, once you try DVCS, you never want to come back. Ondrej From robert.kern at gmail.com Sat Jan 5 19:56:21 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 05 Jan 2008 18:56:21 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <20080105232500.GI7649@mentat.za.net> Message-ID: <47802735.9070908@gmail.com> Bill Baxter wrote: > On Jan 6, 2008 8:25 AM, Stefan van der Walt wrote: >> I recall something you said to David last week, regarding merges with >> SVN: that a person never knows how to do it until *after* you've done >> it! We often make branches in scipy and numpy, and stand a lot to >> gain from a distributed RCS. >> >> Once a person knows how to use SVN, it doesn't take much effort at all >> to learn bzr or hg (even the commands are often the same). The main >> change is a mind-shift: that branches are now a lot friendlier, and >> that they are accessable to everybody. > > I understand that DVCS's do merging better. But what I don't really > understand is why this is an inherent advantage of DVCS. Isnt it > just a matter of the current crop of DVCS's implementing a better > merge algorithm than SVN? No, it's not the algorithm itself. It's the information that the VCS tracks about files and revisions. > The SVN guys seem to be competent, so if > you just give them time surely they will eventually incorporate these > better merging algorithms into SVN. Who wouldn't want better merging? Yes, such support is on its way in 1.5. Unfortunately, that release is most likely years away. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Sat Jan 5 20:08:55 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 6 Jan 2008 03:08:55 +0200 Subject: [Numpy-discussion] branches/maskedarray patch + lost password In-Reply-To: <200801041630.47698.pgmdevlist@gmail.com> References: <200801041630.47698.pgmdevlist@gmail.com> Message-ID: <20080106010855.GJ7649@mentat.za.net> Hi Pierre On Fri, Jan 04, 2008 at 04:30:47PM -0500, Pierre GM wrote: > Here's a patch to the current version of svn/numpy/branches/maskedarray, that > corrects several bugs. Those tests at the end of core.py should be included in the test suite before we commit (otherwise they won't be ran by numpy.test()). Regards St?fan From keramida at ceid.upatras.gr Sat Jan 5 22:27:00 2008 From: keramida at ceid.upatras.gr (Giorgos Keramidas) Date: Sun, 6 Jan 2008 03:27:00 +0000 (UTC) Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> Message-ID: Fernando Perez gmail.com> writes: > Incidentally, the emacs guys seem to be worrying about the same thing: > > http://thread.gmane.org/gmane.emacs.devel/85893 > > If they actually do the work of comparing tools, that work may be > useful for us. I'm pretty sure that any tool that can handle the > entire history of emacs can chew on numpy/scipy/ipython/matplotlib > *combined* for breakfast. The discussions on emacs-devel are interesting indeed. I've been keeping my own local Emacs patches (to make it build on FreeBSD as a port) in a Mercurial repository for a fair amount of time now, and it seems to work "ok". One interesting datapoint is that the entire history of the "HEAD" branch of the repository of Emacs can fit in 141 MB, which is smaller than a full checkout of Emacs plus object code after a build, and it takes less than 2 seconds to look at the log of the first commit done in CVS ever: kobe % du -sk ~/tmp/emacs-src 187436 /home/keramida/tmp/emacs-src kobe % pwd /home/keramida/hg/emacs/gnu kobe % du -sk .hg 141620 .hg kobe % /usr/bin/time hg log -r0 changeset: 0:c67c006134ec user: jimb date: Thu Apr 18 00:48:29 1985 +0000 summary: entered into RCS 1.72 real 1.40 user 0.29 sys kobe % I'm sure that this is far from a "killer feature", but at least it shows that there's no huge obstacle in using Hg to check-out and browse the history of a repository the size of Emacs' CVS tree. I don't know if Emacs will use Mercurial or some other DVCS, but it should be certainly "do-able". Just my two cents, Giorgos From cournape at gmail.com Sat Jan 5 23:54:26 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 6 Jan 2008 13:54:26 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477F2FEF.5050005@noaa.gov> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> <477F2FEF.5050005@noaa.gov> Message-ID: <5b8d13220801052054i6c6f006frbcdd40f7121a3a56@mail.gmail.com> On Jan 5, 2008 4:21 PM, Chris Barker wrote: > hmmm. Everyone posting so far seems to be positive on this idea, but I'm > not so sure. A few thoughts: > > 1) change is bad. It may be worth it, but this decision needs to be made > very differently than if we were starting from scratch. Change for the sake of change is bad. I thought I highlighted in my email that the difficult point was how to make the change (transition, importing the history, etc...), but instead, it quickly slipped to using mercurial. I would have prefered to see what people thought was important on how to proceed, but we all prefer to speak about which tool to use instead :) > > 2) apparently svn merge sucks compared to other merge technology. svn > (and cvs before it) is the only system I'm used, and yes, merging is > painful, but I have to say that it appeared to be painful because it's a > very hard problem. Can anyone comment on why these other systems seem so > much better? Does it have anything to do with Centralized vs. > Distributed at all? > Merge is a hard problem, but DVCS have to solve it to be of any use. > 3) I read Linus' post -- he's quite articulate. However, it seems that > most of his arguments really applied primarily to large projects -- > where there really will be a lot of different "central" versions. This > is very, very, important to the Linux kernel, and probably good for kde, > but scipy is a monstrously smaller community. And it's not a question of > number of devs -- but rather number of versions. > > This makes me thing it really comes down to a better merge -- is there a > way to address that problem with svn? maybe the svnmerge.py that Russel > suggested? svnmerge just does not cut it, when I was saying that merge in svn does not work, I was not even considering basic svn merge, but svnmerge. svnmerge does not change much: merge still fails more often than not, and you have to do a lot of manual things. In DVCS, merge is one command, you do not need to initialize anything when you start. But DVCS is much more than better merge. And has nothing to do with the size of the project.As I said, the whole concept of sandbox, trying new things, is made harder by using svn; it really goes in the way, instead of helping us. > > 4) SVN is very, very, popular. Lots of folks use it, they use it on all > common platforms, and there are tons of clients for it. I work with a > bunch of folks that really don't like a command line (for programmers, I > think that's just plain weird, but there you go). I could never sell a > VCS that didn't have a decent GUI client on Windows and OS-X. > I don't understand this argument: do your co-workers use scipy now but would not if the code source would be kept under a VCS which has no GUI ? scipy and python are fundamentally command line tools, and you cannot contribute to scipy without using the command line. We do not require python, distutils to have a gui ? cheers, David From cournape at gmail.com Sat Jan 5 23:56:29 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 6 Jan 2008 13:56:29 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477F33D9.4020705@gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> <477F2FEF.5050005@noaa.gov> <477F33D9.4020705@gmail.com> Message-ID: <5b8d13220801052056m3653fca4w14591058f08b23e8@mail.gmail.com> On Jan 5, 2008 4:38 PM, Robert Kern wrote: > Chris Barker wrote: > > hmmm. Everyone posting so far seems to be positive on this idea, but I'm > > not so sure. A few thoughts: > > > > 1) change is bad. It may be worth it, but this decision needs to be made > > very differently than if we were starting from scratch. > > > > 2) apparently svn merge sucks compared to other merge technology. svn > > (and cvs before it) is the only system I'm used, and yes, merging is > > painful, but I have to say that it appeared to be painful because it's a > > very hard problem. Can anyone comment on why these other systems seem so > > much better? Does it have anything to do with Centralized vs. > > Distributed at all? > > Tangentially, yes. DVCSes need to keep track of more information in order to be > distributed. That information is extremely useful for managing merges properly. > Centralized systems could track this information, but they don't *need* to in > order to be functional, so they mostly haven't, yet. > > For each revision, the DVCS knows what revisions it derives from. SVN does not > keep this information. SVN primarily just knows the text diffs from revision to > revision. In particular, if I have a long-lived branch, I am going to merge in > changes from the trunk while I'm working on it. When I go to merge the branch > back into the trunk, I need to know which trunk-revisions I've already merged > into the branch. SVN does not track this information. Tools like svnmerge.py > track some of this information at the expense of some added clumsiness. > > It's worth noting that SVN 1.5 will be tracking such information. But that > release is a long ways off. > My understanding, but I do not follow svn much, is that in 1.5, you will only get what svnmerge gives you today. David From robert.kern at gmail.com Sun Jan 6 00:34:59 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 05 Jan 2008 23:34:59 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801052056m3653fca4w14591058f08b23e8@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> <477F2FEF.5050005@noaa.gov> <477F33D9.4020705@gmail.com> <5b8d13220801052056m3653fca4w14591058f08b23e8@mail.gmail.com> Message-ID: <47806883.9030708@gmail.com> David Cournapeau wrote: > On Jan 5, 2008 4:38 PM, Robert Kern wrote: >> It's worth noting that SVN 1.5 will be tracking such information. But that >> release is a long ways off. >> > My understanding, but I do not follow svn much, is that in 1.5, you > will only get what svnmerge gives you today. I suspect that much of the remaining fragility when using svnmerge is linked to the fact that it must exist outside of the core. It needs to manage its information through SVN properties which are themselves revision-controlled and occasionally need merging themselves if there are multiple branches being tracked. If this information were integrated into the core of SVN itself, I conjecture that as far as merging is concerned, SVN 1.5 will be mostly on par with the average DVCS. For a brief rundown of the differences of 1.5's merge tracking and svnmerge: http://blogs.open.collab.net/svn/2007/10/subversion-15-m.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Sun Jan 6 02:11:11 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 05 Jan 2008 23:11:11 -0800 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801052054i6c6f006frbcdd40f7121a3a56@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <20080104152202.GB7649@mentat.za.net> <5b8d13220801040745s41e180a4q1e87d380b8606194@mail.gmail.com> <477F2FEF.5050005@noaa.gov> <5b8d13220801052054i6c6f006frbcdd40f7121a3a56@mail.gmail.com> Message-ID: <47807F0F.4030906@noaa.gov> David Cournapeau wrote: >> 4) SVN is very, very, popular. Lots of folks use it, they use it on all >> common platforms, and there are tons of clients for it. I work with a >> bunch of folks that really don't like a command line (for programmers, I >> think that's just plain weird, but there you go). I could never sell a >> VCS that didn't have a decent GUI client on Windows and OS-X. >> > I don't understand this argument: do your co-workers use scipy now but > would not if the code source would be kept under a VCS which has no > GUI ? Sorry, that wasn't the point. My co-workers (and most folks) use numpy/scipy by downloading binaries, or maybe tarballs, so what VCS is used is, indeed, pretty irrelevant to most users. Would good multi-platform GUI support of the VCS make for more contributers to numpy/scipy? I don't know. At least at the numpy level the kinds of folks that are likely to contribute are probably command-line savvy, etc (though the command line on Windows really, really sucks!) My point was that there is a benefit to using a widely known and used tool that lots of folks are using for their own projects as well. I use SVN every day, and I'm far more likely to go get and look at SVN head of some open source project because of that. I'm kind of dreading that I'll soon need to figure out svn, mercurial, and who knows what other VCSs in order to keep up with the various projects I like to keep up with. The discussion here has been compelling, however. I see these benefits: 1) Better merge. svn merge does suck, so that would be great. It sounds like it may well get better, but no time soon. 2) Easier to to create sub-versions of the main tree -- experimental projects that groups of folks can work on. I started out, from reading Linus' note, thinking that that was really only beneficial for kernel-scale projects, but I now see that it could be nice for much smaller scale projects as well. I do still think scipy could probably support the few of those that will really happen with svn branches, but then we're back to that merge problem again! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From david at ar.media.kyoto-u.ac.jp Sun Jan 6 02:20:50 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 06 Jan 2008 16:20:50 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477FEFE5.5050706@enthought.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> Message-ID: <47808152.5040404@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > David M. Cooke wrote: >> On Jan 4, 2008, at 13:58 , Fernando Perez wrote: >> >> >>> My vote so far is for hg, for performance reasons but also partly >>> because sage and sympy already use it, two projects I'm likely to >>> interact a lot with and that are squarely in line with the >>> ipython/numpy/scipy/matplotlib world. Since they went first and made >>> the choice, I'm happy to let that be a factor in my decision. I'd >>> rather use a tool that others in the same community are also using, >>> especially when the choice is a sound one on technical merit alone. >>> >>> Just my 1e-2... >>> >> >> +1 on mercurial. It's what I use these days (previously, I used darcs, >> which I still like for its patch-handling semantics, but its >> dependence on Haskell, and the dreaded exponential-time merge are a >> bit of a pain). >> > > I don't think it is time to move wholesale to something like Mercurial > or bzr. I would prefer it if all of the Enthought-hosted projects > moved to the (new) system at once, which is not going to happen in the > short term (but long term of course it's an open question). > How would you define short term and long term ? I understand the need to plan carefully things, and this takes time. But I am not sure I understand why not moving now, in the sense what will change in the future which will make the change better then than now ? I thought about targeting the change for 1.1, because it gives one precise time target, and gives use around 6 months, but this can be later if it takes more time. I have not thought about all the details, but I had something like the following plan in mind: - first, importing the different trunks into the different contenders - making read-only mirrors of those import so that people can try it out - making tutorials so that people who do not know about the tools can start in a few minutes - Having a clear idea on the difference between the tools, and makes a list of the different requirements (speed, availability on platforms, 3-party tools, GUI, etc...). It seems mercurial is the tool of choice for almost everybody; I don't have any problem with that, except that I would like to see more arguments than just "I am using mercurial and it works" (the fact that most contributors knows mercurial certainly is an argument in favor of it, though). Having a list of requirements and how each tool fulfill each of them would be helpful, no ? I am certainly willing to do all the above for bzr, and it seems doing it for mercurial won't be any problem since so many people already know it. > But, having an official mirror is a useful thing to explore. > > I suspect there are others with serious reservations about jumping off > of SVN just now (just when a lot of people have finally figured out how > to use it). Using bzr is easier than svn IMHO, and knowing bzr, I knew how to use mercurial for basic things in 5 minutes (by basic I mean checking out code, branching, committing and getting patches): bzr co http://bzr.scipy.org/bzr/numpy -> get the code bzr st -> see the changes bzr diff -> get a patch bzr ci -> commit Basically, you have to change svn to bzr :) And this is really similar in mercurial. Things like getting through the history, merging is a bit more complicated because there is no notion of global revision anymore, but this won't matter to most people ? > If there is an Hg mirror that allows other to use mercurial > at the same time then that sounds like a good idea to me. Is it possible to get the svn dump for people willing to do the import (doing it from the import is not the only way, but is certainly the fastest) ? I can do the import for bzr; I can do it in mercurial too if nobody else jumps in, but since I am less familiar with mercurial, it would be better for someone else to do it. cheers, David From Chris.Barker at noaa.gov Sun Jan 6 03:01:27 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sun, 06 Jan 2008 00:01:27 -0800 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <477F35A9.9040000@gmail.com> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> <477E813C.8060803@enthought.com> <477F33BE.8020404@noaa.gov> <477F35A9.9040000@gmail.com> Message-ID: <47808AD7.9070102@noaa.gov> Robert Kern wrote: > Chris Barker wrote: >> What if your "objects" were nested sequences, and you wanted to partly >> flatten them -- which would you flatten? > > I'm pretty sure that that is exactly the ambiguity that the depth option > resolves. Can you give me an example where it's still ambiguous with the depth > information provided? I was imagining: Say you have a bunch of nested sequences that, fully expanded, would yield a (i,j,k) rank-3 array. If you wanted a rank-2 array, you could have either: a (i,j) array with each element being a length-k sequence or: a (i,k) array, with each element being a length-j sequence. This is quite trivial with numpy n-d arrays. However, while you could build either of those from nested sequences, there isn't, in fact, and object in there that is that length-j sequence. I guess this really points to the power of n-d arrays! Here's a simple example: t = (((1,2),(3,4)),((5,6),(7,8))) depth 1 -- (4,): array([(1, 2), (3, 4), (5, 6), (7, 8)], dtype=object) >>> a[0] (1, 2) depth 1 -- (2,) >>> a array([(1, 2, 3, 4), (5, 6, 7, 8)], dtype=object) >>> a[0] (1, 2, 3, 4) But since there was no object in the original sequence that actually had a 4-tuple, maybe no one would ever be looking for that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Sun Jan 6 03:05:25 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 06 Jan 2008 02:05:25 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <477FEFE5.5050706@enthought.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> Message-ID: <47808BC5.8070401@gmail.com> Travis E. Oliphant wrote: > I don't think it is time to move wholesale to something like Mercurial > or bzr. I would prefer it if all of the Enthought-hosted projects > moved to the (new) system at once, which is not going to happen in the > short term (but long term of course it's an open question). I think that's irrelevant. There is absolutely no reason that we should force all of the Enthought-hosted projects to move in sync. We would have reason if we were being asked to host a different centralized VCS with a complicated server, but hosting Mercurial or bzr is nearly trivial. We already do it for me: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi The remaining thing we would have to support is the Trac integration. While not as trivial as simply hosting the repositories, it's not a very large commitment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sun Jan 6 03:22:02 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 06 Jan 2008 02:22:02 -0600 Subject: [Numpy-discussion] how to create an array of objects that are sequences? In-Reply-To: <47808AD7.9070102@noaa.gov> References: <63267.85.166.31.187.1199467430.squirrel@cens.ioc.ee> <477E6E00.4070402@enthought.com> <50445.85.166.31.187.1199470407.squirrel@cens.ioc.ee> <477E813C.8060803@enthought.com> <477F33BE.8020404@noaa.gov> <477F35A9.9040000@gmail.com> <47808AD7.9070102@noaa.gov> Message-ID: <47808FAA.1000601@gmail.com> Christopher Barker wrote: > Robert Kern wrote: >> Chris Barker wrote: >>> What if your "objects" were nested sequences, and you wanted to partly >>> flatten them -- which would you flatten? >> I'm pretty sure that that is exactly the ambiguity that the depth option >> resolves. Can you give me an example where it's still ambiguous with the depth >> information provided? > > I was imagining: > > Say you have a bunch of nested sequences that, fully expanded, would > yield a (i,j,k) rank-3 array. If you wanted a rank-2 array, you could > have either: > a (i,j) array with each element being a length-k sequence > or: > a (i,k) array, with each element being a length-j sequence. > > This is quite trivial with numpy n-d arrays. > > However, while you could build either of those from nested sequences, > there isn't, in fact, and object in there that is that length-j sequence. And I think that's the key difference. I don't think array() should be responsible for *arbitrarily* flattening nested sequences in order to reinterpret structure. Instead, I think we really only need array() to be able to answer the question, "is this object an atom or a sequence I need to descend into?" Essentially, for all valid index-tuples (say, [i,j,k]) in the array: arr[i,j,k] == original[i,j,k] > I guess this really points to the power of n-d arrays! Yes. Since one can get the (i,k) array after getting the (i,j,k) array and reshaping, I don't think we need to support it in array(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Sun Jan 6 04:34:25 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 6 Jan 2008 18:34:25 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <47808BC5.8070401@gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> Message-ID: http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram This is a bit puzzling. I understand better merging isn't the only reason to choose DVCS, but the above page basically says that Mercurial just uses whatever external merge program it can find. So the file-level merging sounds like it must really be no different from other VCSs. So it is really just proper merging of directory renames and the like that make it superior? --bb From robert.kern at gmail.com Sun Jan 6 04:38:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 06 Jan 2008 03:38:44 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> Message-ID: <4780A1A4.1020506@gmail.com> Bill Baxter wrote: > http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram > > This is a bit puzzling. I understand better merging isn't the only > reason to choose DVCS, but the above page basically says that > Mercurial just uses whatever external merge program it can find. So > the file-level merging sounds like it must really be no different from > other VCSs. > > So it is really just proper merging of directory renames and the like > that make it superior? No. If you'll pardon my repeating myself: """ DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. """ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Jan 6 04:38:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 06 Jan 2008 18:38:56 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <4780A1A4.1020506@gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> <4780A1A4.1020506@gmail.com> Message-ID: <4780A1B0.6050003@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > Bill Baxter wrote: > >> http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram >> >> This is a bit puzzling. I understand better merging isn't the only >> reason to choose DVCS, but the above page basically says that >> Mercurial just uses whatever external merge program it can find. So >> the file-level merging sounds like it must really be no different from >> other VCSs. >> >> So it is really just proper merging of directory renames and the like >> that make it superior? >> > > No. If you'll pardon my repeating myself: > > """ > DVCSes need to keep track of more information in order to be > distributed. That information is extremely useful for managing merges properly. > Centralized systems could track this information, but they don't *need* to in > order to be functional, so they mostly haven't, yet. > > For each revision, the DVCS knows what revisions it derives from. SVN does not > keep this information. SVN primarily just knows the text diffs from revision to > revision. In particular, if I have a long-lived branch, I am going to merge in > changes from the trunk while I'm working on it. When I go to merge the branch > back into the trunk, I need to know which trunk-revisions I've already merged > into the branch. SVN does not track this information. Tools like svnmerge.py > track some of this information at the expense of some added clumsiness. > """ > > Does good merging only depends on the above ? Martin Pool, one of the bzr programmer, wrote this article two years ago: http://sourcefrog.net/weblog/software/vc/derivatives.html which I found both enlightening and easy to follow. cheers, David From robert.kern at gmail.com Sun Jan 6 05:09:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 06 Jan 2008 04:09:17 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <4780A1B0.6050003@ar.media.kyoto-u.ac.jp> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> <4780A1A4.1020506@gmail.com> <4780A1B0.6050003@ar.media.kyoto-u.ac.jp> Message-ID: <4780A8CD.2070600@gmail.com> David Cournapeau wrote: > Does good merging only depends on the above ? Martin Pool, one of the > bzr programmer, wrote this article two years ago: > > http://sourcefrog.net/weblog/software/vc/derivatives.html > > which I found both enlightening and easy to follow. My terminology was fuzzy/incorrect. By "revision," I meant "changeset" rather than "snapshot." The main thrust of that article is about the value of viewing VCS history as a sequence of changesets rather than a sequence of snapshots. FWIW, svnmerge and SVN 1.5 Merge Tracking are changeset-oriented rather than snapshot-oriented. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Sun Jan 6 06:03:49 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 6 Jan 2008 20:03:49 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <4780A1A4.1020506@gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> <4780A1A4.1020506@gmail.com> Message-ID: On Jan 6, 2008 6:38 PM, Robert Kern wrote: > > Bill Baxter wrote: > > http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram > > > > This is a bit puzzling. I understand better merging isn't the only > > reason to choose DVCS, but the above page basically says that > > Mercurial just uses whatever external merge program it can find. So > > the file-level merging sounds like it must really be no different from > > other VCSs. > > > > So it is really just proper merging of directory renames and the like > > that make it superior? > > No. If you'll pardon my repeating myself: > > """ > DVCSes need to keep track of more information in order to be > distributed. That information is extremely useful for managing merges properly. > Centralized systems could track this information, but they don't *need* to in > order to be functional, so they mostly haven't, yet. > > For each revision, the DVCS knows what revisions it derives from. SVN does not > keep this information. SVN primarily just knows the text diffs from revision to > revision. In particular, if I have a long-lived branch, I am going to merge in > changes from the trunk while I'm working on it. When I go to merge the branch > back into the trunk, I need to know which trunk-revisions I've already merged > into the branch. SVN does not track this information. Tools like svnmerge.py > track some of this information at the expense of some added clumsiness. > > """ Ok. Sorry for not reading that closer. So what you're saying is that the magic is in the deciding of exactly which revisions of which files to run the merge program on? --bb From robert.kern at gmail.com Sun Jan 6 06:10:11 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 06 Jan 2008 05:10:11 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> <4780A1A4.1020506@gmail.com> Message-ID: <4780B713.9050005@gmail.com> Bill Baxter wrote: > On Jan 6, 2008 6:38 PM, Robert Kern wrote: >> Bill Baxter wrote: >>> http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram >>> >>> This is a bit puzzling. I understand better merging isn't the only >>> reason to choose DVCS, but the above page basically says that >>> Mercurial just uses whatever external merge program it can find. So >>> the file-level merging sounds like it must really be no different from >>> other VCSs. >>> >>> So it is really just proper merging of directory renames and the like >>> that make it superior? >> No. If you'll pardon my repeating myself: >> >> """ >> DVCSes need to keep track of more information in order to be >> distributed. That information is extremely useful for managing merges properly. >> Centralized systems could track this information, but they don't *need* to in >> order to be functional, so they mostly haven't, yet. >> >> For each revision, the DVCS knows what revisions it derives from. SVN does not >> keep this information. SVN primarily just knows the text diffs from revision to >> revision. In particular, if I have a long-lived branch, I am going to merge in >> changes from the trunk while I'm working on it. When I go to merge the branch >> back into the trunk, I need to know which trunk-revisions I've already merged >> into the branch. SVN does not track this information. Tools like svnmerge.py >> track some of this information at the expense of some added clumsiness. >> >> """ > > Ok. Sorry for not reading that closer. So what you're saying is that > the magic is in the deciding of exactly which revisions of which files > to run the merge program on? That's the main idea. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndbecker2 at gmail.com Sun Jan 6 06:20:32 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 06 Jan 2008 06:20:32 -0500 Subject: [Numpy-discussion] PyArray_FromAny does not accept a generator? References: <477E9925.8020407@gmail.com> Message-ID: Robert Kern wrote: > Neal Becker wrote: >> It seems that PyArray_FromAny does not accept a generator? >> >> Seems like this would be useful. > > It's difficult to do all the magical interpretation that PyArray_FromAny() > does with a iterator of unknown length. In Python, we have fromiter() > which will consume an iterator to make (only) a rank-1 array and thus > sidestep a big chunk of the magical interpretation. > I have previously created a wrapper for python around boost::ublas vector. Basically it does (in pseudocode): vector_from_object(obj): if (obj is a vector): return a copy elif (obj has __len__): return vector_from_sequence elif (obj has __iter__): return vector_from_iter vector_from_iter(iter): copy_iter_to_new_temp_container (in this case, we use std::vector with push_back) return new vector from temp container From wbaxter at gmail.com Sun Jan 6 11:32:22 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 7 Jan 2008 01:32:22 +0900 Subject: [Numpy-discussion] Initial Hg experiences on Windows Message-ID: I've been playing around with Hg on windows for an hour or so now. My overall impression is that the installation process isn't quite there yet. The basic binary installer goes very smoothly, and after that I was able to open up a prompt and type hg commands right away. But going through the tutorial I ran into some issues: *) After running the binary installer, apparently you're supposed to go edit some .ini file to specify your username. It seems it will work ok even if you don't set your username, but since it is apparently highly recommended, the installer just should ask you as part of the install process. *) Despite all this talk about great merging capabilities, for Windows there is *no* merge functionality installed by default. The merge page (http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram) explains how to get merging working on Windows. But what it gives you is a confusing list of alternatives without offering any guidance as to what the trade-offs are. It mentions "batteries included" binary distributions as one solution without giving any link. It offers an hgmerge.py script as a solution but then is wishy-washy about how you specify it in the Mercurial.ini file (just "hgmerge.py" or "python \path\to\hgmerge.py"?). The hgmerge.py script itself relies on pywin32 for some of its functionality, but that is not mentioned anywhere (well, it is now because I edited the wiki...) Without that it fails to find some installed merge tools but catches the import exceptions and reports no error. The page also fails to give a clear recommendation as to which merge tool to use, and they are *not* all created equal. It managed to find DiffMerge already installed on my system, but DiffMerge does a terrible job of automatic merging. It couldn't properly merge two lines added to different parts of a hello_world.c. *) After failing to run any merge program, I got the hgmerge.py script installed and tried to redo the merge. No dice "error: uncomitted merge pending". But I didn't merge anything yet! A hint about how to rerun or undo the failed merge would have been a little more user friendly. I fortunately stumbled across a web page telling me about hg rollback, and that seemed to do the trick. So I think the installation on Windows is a ways away still from what I get from installing TortoiseSVN, particularly the merge thing. Hg/windows really really needs to come bundled with at least a rudimentary merge program that's all configured properly. Hope this is useful info for someone. --bb From cournape at gmail.com Sun Jan 6 11:51:46 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Jan 2008 01:51:46 +0900 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: References: Message-ID: <5b8d13220801060851m2e192415y93ad56789f286c79@mail.gmail.com> On Jan 7, 2008 1:32 AM, Bill Baxter wrote: > I've been playing around with Hg on windows for an hour or so now. My > overall impression is that the installation process isn't quite there > yet. > > The basic binary installer goes very smoothly, and after that I was > able to open up a prompt and type hg commands right away. But going > through the tutorial I ran into some issues: > > *) After running the binary installer, apparently you're supposed to > go edit some .ini file to specify your username. It seems it will > work ok even if you don't set your username, but since it is > apparently highly recommended, the installer just should ask you as > part of the install process. > > *) Despite all this talk about great merging capabilities, for Windows > there is *no* merge functionality installed by default. The merge > page (http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram) > explains how to get merging working on Windows. But what it gives you > is a confusing list of alternatives without offering any guidance as > to what the trade-offs are. It mentions "batteries included" binary > distributions as one solution without giving any link. It offers an > hgmerge.py script as a solution but then is wishy-washy about how you > specify it in the Mercurial.ini file (just "hgmerge.py" or "python > \path\to\hgmerge.py"?). The hgmerge.py script itself relies on > pywin32 for some of its functionality, but that is not mentioned > anywhere (well, it is now because I edited the wiki...) Without that > it fails to find some installed merge tools but catches the import > exceptions and reports no error. The page also fails to give a clear > recommendation as to which merge tool to use, and they are *not* all > created equal. It managed to find DiffMerge already installed on my > system, but DiffMerge does a terrible job of automatic merging. It > couldn't properly merge two lines added to different parts of a > hello_world.c. > > *) After failing to run any merge program, I got the hgmerge.py script > installed and tried to redo the merge. No dice "error: uncomitted > merge pending". But I didn't merge anything yet! A hint about how to > rerun or undo the failed merge would have been a little more user > friendly. I fortunately stumbled across a web page telling me about > hg rollback, and that seemed to do the trick. > > So I think the installation on Windows is a ways away still from what > I get from installing TortoiseSVN, particularly the merge thing. > Hg/windows really really needs to come bundled with at least a > rudimentary merge program that's all configured properly. > I don't think that any DVCS is where svn is in this regard; mercurial does not seem to be, bzr is not either, and git certainly is not. As far as command line is concerned, I know for a fact that the official bzr installer does work on windows (I use bzr on windows, in particular for my work related to scons integration with distutils). I think that at least wrt windows support, bzr is today better than hg, if only for the fact that some bzr developers are primely windows developers, with the "windows culture". David From pachi at rvburke.com Sun Jan 6 11:49:34 2008 From: pachi at rvburke.com (Rafael Villar Burke) Date: Sun, 6 Jan 2008 16:49:34 +0000 (UTC) Subject: [Numpy-discussion] Moving away from svn ? References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801041121v2e3c7e1q7f0743a532bdc5e7@mail.gmail.com> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> <5b8d13220801041425o62c33e16j964d15c7fb23ced@mail.gmail.com> Message-ID: David Cournapeau gmail.com> writes: > The open solaris project documented their choice, too: > > http://www.opensolaris.org/os/community/tools/scm/history/ > > Contrary to mozilla, solaris is using hg as the main VCS. Mozilla will be using mercurial (hg) too, but decided to do the full transition after the next release. AFAICT, they have fully synched copies of their code in a mercurial repository. Regards, Rafael From ndbecker2 at gmail.com Sun Jan 6 12:06:13 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 06 Jan 2008 12:06:13 -0500 Subject: [Numpy-discussion] fromiter doc incorrect Message-ID: In numpy book, it says: fromiter (iter or gen, dtype=None) but that's not true: Help on built-in function fromiter in module numpy.core.multiarray: fromiter(...) fromiter(iterable, dtype, count=-1) dtype is required From cournape at gmail.com Sun Jan 6 12:22:46 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Jan 2008 02:22:46 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> <5b8d13220801041425o62c33e16j964d15c7fb23ced@mail.gmail.com> Message-ID: <5b8d13220801060922o4217e37ex45daabab64df0d9b@mail.gmail.com> On Jan 7, 2008 1:49 AM, Rafael Villar Burke wrote: > David Cournapeau gmail.com> writes: > > > The open solaris project documented their choice, too: > > > > http://www.opensolaris.org/os/community/tools/scm/history/ > > > > Contrary to mozilla, solaris is using hg as the main VCS. > > Mozilla will be using mercurial (hg) too, but decided to do the full transition > after the next release. > > AFAICT, they have fully synched copies of their code in a mercurial repository. Yes. When I said that open solaris was using hg as their main VCS, I meant today. Mozilla does not use yet mercurial; note that when solaris (and mozilla ?) made their choice, bzr had problems with big code size, which is less of a problem now (bzr heavily focused on performances in the last 6 months, and it showed). I have imported scikits and scipy in both mercurial and bzr, and for some operations, bzr is now faster (in my limited experience). To be frank, I did not realize that mercurial was that popular (which makes it more of an argument than I initially thought: I assumed - wrongly it seems - that both had a similar user-base) cheers, David From matthieu.brucher at gmail.com Sun Jan 6 12:24:25 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 6 Jan 2008 18:24:25 +0100 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: <5b8d13220801060851m2e192415y93ad56789f286c79@mail.gmail.com> References: <5b8d13220801060851m2e192415y93ad56789f286c79@mail.gmail.com> Message-ID: > > > *) After running the binary installer, apparently you're supposed to > > go edit some .ini file to specify your username. It seems it will > > work ok even if you don't set your username, but since it is > > apparently highly recommended, the installer just should ask you as > > part of the install process. For bzr, it is a known instruction, whoami, that specifies the username. > I don't think that any DVCS is where svn is in this regard; mercurial > does not seem to be, bzr is not either, and git certainly is not. Unfortunately :( As far as command line is concerned, I know for a fact that the > official bzr installer does work on windows (I use bzr on windows, in > particular for my work related to scons integration with distutils). I can confirm that it works too, and besides for the BZR_SSH environment variable if there is a bzr+ssh connection, I had no trouble with bzr. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From corrada at cs.umass.edu Sun Jan 6 12:26:10 2008 From: corrada at cs.umass.edu (Andres Corrada-Emmanuel) Date: Sun, 06 Jan 2008 12:26:10 -0500 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: References: Message-ID: <47810F32.9070001@cs.umass.edu> I also have been playing with Hg on Windows apropos discussions on the list. I plan to move repositories at my lab to Hg. I second everything Bill Baxter has said. I plan to use Hg myself and to continue exploring its use but TortoiseHg is not yet at the level of ease of use that TortoiseSVN is for people who just want it to work transparently. This will delay our move to Hg from SVN. Bill Baxter wrote: > I've been playing around with Hg on windows for an hour or so now. My > overall impression is that the installation process isn't quite there > yet. > > The basic binary installer goes very smoothly, and after that I was > able to open up a prompt and type hg commands right away. But going > through the tutorial I ran into some issues: > > *) After running the binary installer, apparently you're supposed to > go edit some .ini file to specify your username. It seems it will > work ok even if you don't set your username, but since it is > apparently highly recommended, the installer just should ask you as > part of the install process. > > *) Despite all this talk about great merging capabilities, for Windows > there is *no* merge functionality installed by default. The merge > page (http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram) > explains how to get merging working on Windows. But what it gives you > is a confusing list of alternatives without offering any guidance as > to what the trade-offs are. It mentions "batteries included" binary > distributions as one solution without giving any link. It offers an > hgmerge.py script as a solution but then is wishy-washy about how you > specify it in the Mercurial.ini file (just "hgmerge.py" or "python > \path\to\hgmerge.py"?). The hgmerge.py script itself relies on > pywin32 for some of its functionality, but that is not mentioned > anywhere (well, it is now because I edited the wiki...) Without that > it fails to find some installed merge tools but catches the import > exceptions and reports no error. The page also fails to give a clear > recommendation as to which merge tool to use, and they are *not* all > created equal. It managed to find DiffMerge already installed on my > system, but DiffMerge does a terrible job of automatic merging. It > couldn't properly merge two lines added to different parts of a > hello_world.c. > > *) After failing to run any merge program, I got the hgmerge.py script > installed and tried to redo the merge. No dice "error: uncomitted > merge pending". But I didn't merge anything yet! A hint about how to > rerun or undo the failed merge would have been a little more user > friendly. I fortunately stumbled across a web page telling me about > hg rollback, and that seemed to do the trick. > > So I think the installation on Windows is a ways away still from what > I get from installing TortoiseSVN, particularly the merge thing. > Hg/windows really really needs to come bundled with at least a > rudimentary merge program that's all configured properly. > > Hope this is useful info for someone. > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Andres Corrada-Emmanuel Research Fellow Aerial Imaging and Remote Sensing Lab Computer Science Department University of Massachusetts at Amherst Blog: www.corrada.com/blog From cournape at gmail.com Sun Jan 6 12:30:16 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Jan 2008 02:30:16 +0900 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: References: Message-ID: <5b8d13220801060930x608aeedfue28cc64e9e512935@mail.gmail.com> On Jan 7, 2008 1:32 AM, Bill Baxter wrote: > I've been playing around with Hg on windows for an hour or so now. My > overall impression is that the installation process isn't quite there > yet. > > The basic binary installer goes very smoothly, and after that I was > able to open up a prompt and type hg commands right away. But going > through the tutorial I ran into some issues: > > *) After running the binary installer, apparently you're supposed to > go edit some .ini file to specify your username. It seems it will > work ok even if you don't set your username, but since it is > apparently highly recommended, the installer just should ask you as > part of the install process. > > *) Despite all this talk about great merging capabilities, for Windows > there is *no* merge functionality installed by default. The merge > page (http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram) > explains how to get merging working on Windows. But what it gives you > is a confusing list of alternatives without offering any guidance as > to what the trade-offs are. It mentions "batteries included" binary > distributions as one solution without giving any link. FIY, it seems you can find it here (I have not tried it): http://qct.sourceforge.net/Mercurial-BI.html cheers, David From efiring at hawaii.edu Sun Jan 6 13:11:02 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 06 Jan 2008 08:11:02 -1000 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <5b8d13220801060922o4217e37ex45daabab64df0d9b@mail.gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801041305o12d5ac5bl4e34c09594f8613e@mail.gmail.com> <85b5c3130801041317pedd565bpa02b683ef71d7429@mail.gmail.com> <5b8d13220801041409i491732c0t6ea990ab75a4dbd6@mail.gmail.com> <1e2af89e0801041415o555675d1w2dc06b10c52d087@mail.gmail.com> <5b8d13220801041425o62c33e16j964d15c7fb23ced@mail.gmail.com> <5b8d13220801060922o4217e37ex45daabab64df0d9b@mail.gmail.com> Message-ID: <478119B6.9040906@hawaii.edu> David Cournapeau wrote: [...] > To be frank, I did not realize that mercurial was that popular (which > makes it more of an argument than I initially thought: I assumed - > wrongly it seems - that both had a similar user-base) David, One reason that they apparently do not is that mercurial has been more stable and usable for a longer period. It was more than two years ago that I decided to begin using a VCS for local projects. I surveyed the field, concluded I wanted a DVCS, and gradually got the sense that mercurial was already usable without too much risk of ending up stranded by major design changes or death of the project. Bzr was in the early development stages, breaking away from the original bazaar which was being abandoned, so it was simply not a reasonable choice at the time. Eric From ndbecker2 at gmail.com Sun Jan 6 13:36:07 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 06 Jan 2008 13:36:07 -0500 Subject: [Numpy-discussion] Initial Hg experiences on Windows References: Message-ID: Did you look at TourtiseHg? From oliphant at enthought.com Sun Jan 6 14:41:06 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sun, 06 Jan 2008 13:41:06 -0600 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <47808BC5.8070401@gmail.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> Message-ID: <47812ED2.1020303@enthought.com> Robert Kern wrote: > Travis E. Oliphant wrote: > > >> I don't think it is time to move wholesale to something like Mercurial >> or bzr. I would prefer it if all of the Enthought-hosted projects >> moved to the (new) system at once, which is not going to happen in the >> short term (but long term of course it's an open question). >> > > I think that's irrelevant. There is absolutely no reason that we should force > all of the Enthought-hosted projects to move in sync. It is relevant, because I have my hands in all of them. I don't want to have to keep up with too many different systems at once. Perhaps it's selfish and immaterial to others, but it is relevant to me. > We would have reason if we > were being asked to host a different centralized VCS with a complicated server, > but hosting Mercurial or bzr is nearly trivial. We already do it for me: > > http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi > > The remaining thing we would have to support is the Trac integration. While not > as trivial as simply hosting the repositories, it's not a very large commitment. > > Trac integration is exactly what I'm thinking about. If it is easy, then it is not a big deal, but I have not seen arguments or evidence that it is the case (or suggestions of something to use other than Trac -- which is also a bigger deal for moving everything at once). -Travis O. From wbaxter at gmail.com Sun Jan 6 18:00:58 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 7 Jan 2008 08:00:58 +0900 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: <5b8d13220801060930x608aeedfue28cc64e9e512935@mail.gmail.com> References: <5b8d13220801060930x608aeedfue28cc64e9e512935@mail.gmail.com> Message-ID: On Jan 7, 2008 2:30 AM, David Cournapeau wrote: > > to what the trade-offs are. It mentions "batteries included" binary > > distributions as one solution without giving any link. > > FIY, it seems you can find it here (I have not tried it): > > http://qct.sourceforge.net/Mercurial-BI.html Thanks. I added the link to the Mercurial merge page. --bb From fperez.net at gmail.com Mon Jan 7 02:17:33 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Jan 2008 00:17:33 -0700 Subject: [Numpy-discussion] Sage/Scipy Days 8 at Enthought: Feb 29/March 4 2008 Message-ID: Hi all, below is the full text of the announcement, which has also been posted here: http://wiki.sagemath.org/days8 Many thanks to Enthought for the generous support they've offered! We really look forward to this meeting being a great opportunity for collaboration between the Scipy and Sage teams. Cheers, Travis Oliphant William Stein Fernando Perez ================================ Sage/Scipy Days 8 at Enthought ================================ ------------------------------------------------------- Connecting Pure Mathematics With Scientific Computation ------------------------------------------------------- .. Contents:: .. 1 Introduction 2 Location 3 Costs and Funding 4 Contacts and further information 5 Preliminary agenda 5.1 Friday 29: Talks 5.2 Saturday 1: More talks/code planning/code 5.3 Sunday 2: Coding 5.4 Monday 3: Coding 5.5 Tuesday 4: Wrapup .. Introduction ============ The Sage_ and Scipy_ teams and `Enthought Inc.`_ are pleased to announce the first collaborative meeting for Sage/Scipy joint development, to be held from February 29 until March 4, 2007 at the Enthought headquarters in Austin, Texas. The purpose of this meeting is to gather developers for these projects in a friendly atmosphere for a few days of technical talks and active development work. It should be clear to those interested in attending that this is *not* an academic conference, but instead an opportunity for the two teams to find common points for collaboration, joint work, better integration between the projects and future directions. The focus of the workshop will be to actually *implement* such ideas, not just to plan for them. **Sage** is a Python-based system which aims at providing an open source, free alternative to existing propietary mathematical software and does so by integrating multiple open source projects, as well as providing its own native functionality in many areas. It includes by default the NumPy and SciPy packages. **NumPy and SciPy** are Python libraries whose focus is high-performance numerical computing, and they are widely accepted as the foundation of most Python-based scientific computing projects. **Enthought** is a scientific computing company that produces Python-based tools for many application-specific domains. Enthought has a strong commitment to open source development: it provides support and hosting for Numpy, Scipy, and many other Python scientific projects and many of its tools_ are freely available. The theme of the workshop is finding ways to best combine our strengths to create something that is significantly better than anything ever done so far. .. _Sage: http://sagemath.org .. _Scipy: http://scipy.org .. _`Enthought Inc.`: http://enthought.com .. _tools: http://code.enthought.com Location ======== The workshop will be held at the headquarters of `Enthought Inc.`_:: Suite 2100 515 Congress Ave. Austin, TX 78701 (512) 536-1057, voice (512) 536-1059, fax .. _`Enthought Inc.`: http://enthought.com Costs and Funding ================= We can accomodate a total of 30 attendees at Enthought's headquarters for the meeting. There is a $100 registration fee, which will be used to cover coffee, snacks and lunches for all the days of the meeting, plus one group dinner outing. Attendees can use the wiki_ to coordinate room/car rental sharing if they so desire. Thanks to Enthought's generous offer of support, we'll be able to cover the above costs for 15 attendees, in addition to offering them housing and transportation. Please note that housing will be provided at Enthought's personal residences, so remember to bring your clean pajamas. We are currently looking into the possibility of additional funding to cover the registration fee for all attendees, and will update the wiki accordingly if that becomes possible. If you plan on coming please email Fernando.Perez at colorado.edu to let us know of your intent so we can have a better idea of the total numbers. Please indicate if you could only come under the condition that you can be hosted. We will try to offer hosting to as many of the Sage and Scipy developers as possible, but if you can fund your own expenses, this may open a slot for someone with limited funds. If the total attendance is below 15, we will offer hosting to everyone. We will close registration for those requesting hosting by Sunday, February 3, 2008. If we actually fill up all 30 available slots we will announce it, otherwise you are free to attend by letting us know anytime before the meeting, though past Feb. 1 you will be required to pay the registration fee of $100. .. _wiki: http://wiki.sagemath.org/days8 Contacts and further information ================================ For further information, you can either contact one of the following people (in parenthesis we note the topic most likely to be relevant to them): - William Stein (Sage): wstein at gmail.org - Fernando Perez (Scipy): Fernando.Perez at colorado.edu - Travis Oliphant (Enthought): oliphant at enthought.com or you can go to our wiki_ for up to date details. Preliminary agenda ================== Friday 29: Talks ---------------- This is a rough cut of suggested topics, along with a few notes on possible details that might be of interest. The actual material in the talks will be up to the presenters, of course. Some of these topics might just become projects to work on rather than actual talks, if we don't have a speaker available but have interested parties who wish to focus on the problem. Speakers are asked to include a slide of where they see any chances for better collaboration between the various projects (to the best of their knowledge). There will be a note-taker during the day who will try to keep tabs on this information and will summarize it as starting material for the joint work panel discussion to be held on Saturday (FPerez volunteers for this task if needed). - Numpy internal architecture and type system. - Sage internal type system, with emphasis on its number type system. - A clarification of where the 'sage language' goes beyond python. Things like ``A\b`` are valid in the CLI but not the notebook. Scipy is pure python, so it would help the scipy team better understand the boundaries between the two. - Special methods used by Sage (foo._magical_sage_method_)? If some of these make sense, we might want to agree on common protocols for numpy/scipy/sage objects to honor. - Sage usage of numpy, sage.matrix vs numpy.arrays. Smoother integration of numpy arrays/sage matrices and vectors. - Extended precision LAPACK. Integration in numpy/sage. The extended precision work LAPACK work was done by Y. Hida and J. Demmel at UC Berkeley. - Distributed/Parallel computing: DSage, ipython, Brian Granger's work on Global arrays for NASA... - Scikits: these are 'toolkits' that use numpy/scipy and can contain GPL code (details of how these will work are being firmed up in the scipy lists, and will be settled by the workshop). Perhaps some of SAGE's library wrappers (like GMP, MPFR or GSL) could become scikits? - Cython: status (inclusion in py2.6?), overview, opportunities for better numpy integration and usage. - Enthought technologies: Traits, TVTK, Mayavi, Chaco, Envisage. - User interface collaboration: 'sage-lite'/pylab/ipython code sharing possibilities? Saturday 1: More talks/code planning/coding ------------------------------------------- 9-11 am: Any remaining talks that didn't fit on Friday. Only if needed. 11-12: panel for specific coding projects and ideas, spill over into lunch time. 12-1: lunch. Rest of day: start coding! Organize in teams according to the plans made earlier and code away... Sunday 2: Coding ---------------- Work on projects decided above. 5-6pm: brief (5-10 minutes) status updates from coding teams. Problems encountered, progress, suggestions for adjustment. Monday 3: Coding ---------------- Same as Sunday. Tuesday 4: Wrapup ----------------- 9-11 am: Wrapup sessions with summary from coding projects. 11-12 am: Panel discussion on future joint work options. Afternoon: anyone left around can continue to code! From Matts.Bjorck at gmail.com Mon Jan 7 04:00:56 2008 From: Matts.Bjorck at gmail.com (Matts Bjoerck) Date: Mon, 7 Jan 2008 09:00:56 +0000 (UTC) Subject: [Numpy-discussion] Bugs using complex192 Message-ID: Hi, I've started using complex192 for some calculations and came across two things that seems to be bugs: In [1]: sqrt(array([-1.0],dtype = complex192)) Out[1]: array([0.0+-6.1646549e-4703j], dtype=complex192) In [2]: sqrt(array([-1.0],dtype = complex128)) Out[2]: array([ 0.+1.j]) In [3]: x Out[3]: array([0.0+0.0j, 1.0+0.0j], dtype=complex192) In [4]: -x Out[4]: array([3.3621031e-4932+0.0012454e-5119j, 1.6794099e-4932+0.0011717e-5119j], dtype=complex192) So, sqrt and using a negation does not seem to work. Have anyone else came across this as well? using numpy 1.0.3 on ubuntu Any help appriciated /Matts From matthieu.brucher at gmail.com Mon Jan 7 04:10:50 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 7 Jan 2008 10:10:50 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: References: Message-ID: i, I managed to reproduce your bugs on a FC6 box : >>> import numpy as n >>> n.sqrt(n.array([-1.0],dtype = n.complex192)) array([0.0+9.2747134e+492j], dtype=complex192) >>> n.sqrt(n.array([-1.0],dtype = n.complex128)) array([ 0.+1.j]) >>> x=n.array([0.0+0.0j, 1.0+0.0j], dtype=n.complex192) >>> x array([0.0+0.0j, 1.0+0.0j], dtype=complex192) >>> -x array([3.3621031e-4932+-1.0204727e+2057j, 1.6794099e-4932+5.5355029e-4930j], dtype=complex192) >>> n.__version__ '1.0.5.dev4494' Matthieu 2008/1/7, Matts Bjoerck : > > Hi, > > I've started using complex192 for some calculations and came across two > things > that seems to be bugs: > > In [1]: sqrt(array([-1.0],dtype = complex192)) > Out[1]: array([0.0+-6.1646549e-4703j], dtype=complex192) > > In [2]: sqrt(array([-1.0],dtype = complex128)) > Out[2]: array([ 0.+1.j]) > > In [3]: x > Out[3]: array([0.0+0.0j, 1.0+0.0j], dtype=complex192) > > In [4]: -x > Out[4]: array([3.3621031e-4932+0.0012454e-5119j, > 1.6794099e-4932+0.0011717e-5119j], dtype=complex192) > > So, sqrt and using a negation does not seem to work. Have anyone else came > across this as well? > > using numpy 1.0.3 on ubuntu > > Any help appriciated > > /Matts > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Mon Jan 7 04:17:08 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 7 Jan 2008 10:17:08 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: References: Message-ID: <80c99e790801070117q692619e0s45fe1134e5e00a84@mail.gmail.com> It doesn't work on Windows, either. In [35]: numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex192)) Out[35]: array([0.0+2.9996087e-305j], dtype=complex192) In [36]: numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex128)) Out[36]: array([ 0.+1.j]) In [37]: numpy.__version__ Out[37]: '1.0.5.dev4567' hth, L. On 1/7/08, Matthieu Brucher wrote: > > i, > > I managed to reproduce your bugs on a FC6 box : > >>> import numpy as n > > >>> n.sqrt(n.array([-1.0],dtype = n.complex192)) > array([0.0+9.2747134e+492j], dtype=complex192) > > >>> n.sqrt(n.array([-1.0],dtype = n.complex128)) > array([ 0.+1.j]) > > >>> x=n.array([0.0+0.0j, 1.0+0.0j], dtype=n.complex192) > >>> x > array([0.0+0.0j, 1.0+0.0j], dtype=complex192) > > >>> -x > array([3.3621031e-4932+-1.0204727e+2057j, 1.6794099e-4932+5.5355029e-4930j], > dtype=complex192) > > >>> n.__version__ > '1.0.5.dev4494' > > Matthieu > > 2008/1/7, Matts Bjoerck < Matts.Bjorck at gmail.com>: > > > > Hi, > > > > I've started using complex192 for some calculations and came across two > > things > > that seems to be bugs: > > > > In [1]: sqrt(array([-1.0],dtype = complex192)) > > Out[1]: array([0.0+-6.1646549e-4703j], dtype=complex192) > > > > In [2]: sqrt(array([-1.0],dtype = complex128)) > > Out[2]: array([ 0.+1.j]) > > > > In [3]: x > > Out[3]: array([0.0+0.0j, 1.0+0.0j], dtype=complex192) > > > > In [4]: -x > > Out[4]: array([3.3621031e-4932+0.0012454e-5119j, > > 1.6794099e-4932+0.0011717e-5119j], dtype=complex192) > > > > So, sqrt and using a negation does not seem to work. Have anyone else > > came > > across this as well? > > > > using numpy 1.0.3 on ubuntu > > > > Any help appriciated > > > > /Matts > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Matts.Bjorck at gmail.com Mon Jan 7 05:04:43 2008 From: Matts.Bjorck at gmail.com (Matts Bjoerck) Date: Mon, 7 Jan 2008 10:04:43 +0000 (UTC) Subject: [Numpy-discussion] Bugs using complex192 References: <80c99e790801070117q692619e0s45fe1134e5e00a84@mail.gmail.com> Message-ID: Thanks for the fast replies, now I know it's not my machine that gives me trouble. In the meantime I tested a couple of other functions. It seems that all of them fail with complex192. /Matts In [19]: x192 = arange(0,2,0.5,dtype = complex192)*pi In [20]: x128 = arange(0,2,0.5,dtype = complex128)*pi In [21]: sin(x192) Out[21]: array([0.0+0.0j, 1.6794099e-4932+0.0j, 1.5925715e-4932+4.2044193e+3906j, 1.6794099e-4932+2.3484019e-4941j], dtype=complex192) In [22]: sin(x128) Out[22]: array([ 0.00000000e+00+0.j, 1.00000000e+00+0.j, 1.22460635e-16-0.j, -1.00000000e+00-0.j]) In [23]: cos(x192) Out[23]: array([1.6794099e-4932+-2.7540704e+667j, 1.5909299e-4932+0e-5119j, 1.6794099e-4932+0.0000074e-5119j, 1.5934769e-4932+0e-5119j], dtype=complex192) In [24]: cos(x128) Out[24]: array([ 1.00000000e+00-0.j, 6.12303177e-17-0.j, -1.00000000e+00-0.j, -1.83690953e-16+0.j]) In [25]: exp(x192) Out[25]: array([1.6794099e-4932+0.0j, 1.6830259e-4932+1.5656013e-4940j, 1.6867092e-4932+1.741791e-2343j, 1.6904736e-4932+0e-5119j], dtype=complex192) In [26]: exp(x128) Out[26]: array([ 1. +0.j, 4.81047738+0.j, 23.14069263+0.j, 111.31777849+0.j]) In [27]: log10(x192) Warning: divide by zero encountered in log10 Out[27]: array([3.3604615e-4932+0.0j, 1.675419e-4932+1.5656013e-4940j, 1.6777496e-4932+7.1009005e-2357j, 1.6783371e-4932+0.0017686e-5119j], dtype=complex192) In [28]: log10(x128) Warning: divide by zero encountered in log10 Out[28]: array([ -inf+0.j, 1.96119877e-001+0.j, 4.97149873e-001+0.j, 6.73241132e-001+0.j]) In [29]: log(x192) Warning: divide by zero encountered in log Out[29]: array([3.3604615e-4932+-2.7540703e+667j, 1.6774503e-4932+0.0007074e-5119j, 1.6796475e-4932+7.0728523e-2357j, 1.6803131e-4932+1.5656013e-4940j], dtype=complex192) In [30]: log(x128) Warning: divide by zero encountered in log Out[30]: array([ -inf+0.j, 4.51582705e-001+0.j, 1.14472989e+000+0.j, 1.55019499e+000+0.j]) From cournape at gmail.com Mon Jan 7 07:59:14 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Jan 2008 21:59:14 +0900 Subject: [Numpy-discussion] Moving away from svn ? In-Reply-To: <47812ED2.1020303@enthought.com> References: <477E1754.6030907@ar.media.kyoto-u.ac.jp> <5b8d13220801040856n85b7f5fv29c608cb95d022a5@mail.gmail.com> <85b5c3130801041045k535d33b8w3a4c1cc3012fdbfb@mail.gmail.com> <477FEFE5.5050706@enthought.com> <47808BC5.8070401@gmail.com> <47812ED2.1020303@enthought.com> Message-ID: <5b8d13220801070459p781b8314lcc14bd044840e7c7@mail.gmail.com> On Jan 7, 2008 4:41 AM, Travis E. Oliphant wrote: > Robert Kern wrote: > > Travis E. Oliphant wrote: > > > > > >> I don't think it is time to move wholesale to something like Mercurial > >> or bzr. I would prefer it if all of the Enthought-hosted projects > >> moved to the (new) system at once, which is not going to happen in the > >> short term (but long term of course it's an open question). > >> > > > > I think that's irrelevant. There is absolutely no reason that we should force > > all of the Enthought-hosted projects to move in sync. > It is relevant, because I have my hands in all of them. I don't want to > have to keep up with too many different systems at once. Perhaps it's > selfish and immaterial to others, but it is relevant to me. I can understand the reluctance, but neither mercurial or bzr are difficult to use. The time required to learn one of them would be quickly negligeable compared to the time gained. > > We would have reason if we > > were being asked to host a different centralized VCS with a complicated server, > > but hosting Mercurial or bzr is nearly trivial. We already do it for me: > > > > http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi > > > > The remaining thing we would have to support is the Trac integration. While not > > as trivial as simply hosting the repositories, it's not a very large commitment. > > > > > Trac integration is exactly what I'm thinking about. If it is easy, > then it is not a big deal, but I have not seen arguments or evidence > that it is the case (or suggestions of something to use other than Trac > -- which is also a bigger deal for moving everything at once). There are trac plugins for both bzr and mercurial. For bzr, I was pointed the following project using trac (10.4) plus trac+bzr: http://bugs.bitlbee.org/bitlbee/browser You seemed to be open on the idea of a mirror: what about a mercurial/bzr mirror of numpy and scipy, + some kind of trac mirror (I don't think it would be a problem to have a mirror to see the branches; having a read only ticket viewer maybe more difficult) ? It is just a matter of having 1/2 volunteers + some help from people having access to the servers (to get svn dump and co). cheers, David From rmay at ou.edu Mon Jan 7 10:47:04 2008 From: rmay at ou.edu (Ryan May) Date: Mon, 07 Jan 2008 09:47:04 -0600 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> Message-ID: <47824978.2050601@ou.edu> Stuart Brorson wrote: >>> I realize NumPy != Matlab, but I'd wager that most users would think >>> that this is the natural behavior...... >> Well, that behavior won't happen. We won't mutate the dtype of the array because >> of assignment. Matlab has copy(-on-write) semantics for things like slices while >> we have view semantics. We can't safely do the reallocation of memory [1]. > > That's fair enough. But then I think NumPy should consistently > typecheck all assignmetns and throw an exception if the user attempts > an assignment which looses information. > Yeah, there's no doubt in my mind that this is a bug, if for no other reason than this inconsistency: >>> import numpy as N >>> a = N.zeros((2,2)) >>> a[0,0]=3+4j Traceback (most recent call last): File "", line 1, in TypeError: can't convert complex to float; use abs(z) >>> b=N.array([3+4j]) >>> a[0,0]=b[0] >>> a array([[ 3., 0.], [ 0., 0.]]) I've been bitten by this myself and would have very much appreciated an exception rather than incorrect answers. It would seem that this behavior violates "In the face of ambiguity, refuse the temptation to guess." The question to me is how much do people rely on this "feature" of the API and does that dictate putting the change off until Numpy 1.1 or beyond? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From charlesr.harris at gmail.com Mon Jan 7 12:36:58 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 10:36:58 -0700 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <47824978.2050601@ou.edu> References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> Message-ID: On Jan 7, 2008 8:47 AM, Ryan May wrote: > Stuart Brorson wrote: > >>> I realize NumPy != Matlab, but I'd wager that most users would think > >>> that this is the natural behavior...... > >> Well, that behavior won't happen. We won't mutate the dtype of the > array because > >> of assignment. Matlab has copy(-on-write) semantics for things like > slices while > >> we have view semantics. We can't safely do the reallocation of memory > [1]. > > > > That's fair enough. But then I think NumPy should consistently > > typecheck all assignmetns and throw an exception if the user attempts > > an assignment which looses information. > > > > Yeah, there's no doubt in my mind that this is a bug, if for no other > reason than this inconsistency: > One place where Numpy differs from MatLab is the way memory is handled. MatLab is always generating new arrays, so for efficiency it is worth preallocating arrays and then filling in the parts. This is not the case in Numpy where lists can be used for things that grow and subarrays are views. Consequently, preallocating arrays in Numpy should be rare and used when either the values have to be generated explicitly, which is what you see when using the indexes in your first example. As to assignment between arrays, it is a mixed question. The problem again is memory usage. For large arrays, it makes since to do automatic conversions, as is also the case in functions taking output arrays, because the typecast can be pushed down into C where it is time and space efficient, whereas explicitly converting the array uses up temporary space. However, I can imagine an explicit typecast function, something like a[...] = typecast(b) that would replace the current behavior. I think the typecast function could be implemented by returning a view of b with a castable flag set to true, that should supply enough information for the assignment operator to do its job. This might be a good addition for Numpy 1.1. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Mon Jan 7 12:57:40 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 07 Jan 2008 18:57:40 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: References: Message-ID: On Mon, 7 Jan 2008 10:10:50 +0100 "Matthieu Brucher" wrote: > i, > > I managed to reproduce your bugs on a FC6 box : >>>> import numpy as n > >>>> n.sqrt(n.array([-1.0],dtype = n.complex192)) > array([0.0+9.2747134e+492j], dtype=complex192) > >>>> n.sqrt(n.array([-1.0],dtype = n.complex128)) > array([ 0.+1.j]) > >>>> x=n.array([0.0+0.0j, 1.0+0.0j], dtype=n.complex192) >>>> x > array([0.0+0.0j, 1.0+0.0j], dtype=complex192) > >>>> -x > array([3.3621031e-4932+-1.0204727e+2057j, >1.6794099e-4932+5.5355029e-4930j], > dtype=complex192) > >>>> n.__version__ > '1.0.5.dev4494' > > Matthieu > > 2008/1/7, Matts Bjoerck : >> >> Hi, >> >> I've started using complex192 for some calculations and >>came across two >> things >> that seems to be bugs: >> >> In [1]: sqrt(array([-1.0],dtype = complex192)) >> Out[1]: array([0.0+-6.1646549e-4703j], dtype=complex192) >> >> In [2]: sqrt(array([-1.0],dtype = complex128)) >> Out[2]: array([ 0.+1.j]) >> >> In [3]: x >> Out[3]: array([0.0+0.0j, 1.0+0.0j], dtype=complex192) >> >> In [4]: -x >> Out[4]: array([3.3621031e-4932+0.0012454e-5119j, >> 1.6794099e-4932+0.0011717e-5119j], dtype=complex192) >> >> So, sqrt and using a negation does not seem to work. >>Have anyone else came >> across this as well? >> >> using numpy 1.0.3 on ubuntu >> >> Any help appriciated >> >> /Matts >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- >French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and >http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher >>> numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex192)) Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'complex192' >>> numpy.__version__ '1.0.5.dev4673' Nils From rmay at ou.edu Mon Jan 7 13:01:04 2008 From: rmay at ou.edu (Ryan May) Date: Mon, 07 Jan 2008 12:01:04 -0600 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> Message-ID: <478268E0.4000804@ou.edu> Charles R Harris wrote: > > > On Jan 7, 2008 8:47 AM, Ryan May > wrote: > > Stuart Brorson wrote: > >>> I realize NumPy != Matlab, but I'd wager that most users would think > >>> that this is the natural behavior...... > >> Well, that behavior won't happen. We won't mutate the dtype of > the array because > >> of assignment. Matlab has copy(-on-write) semantics for things > like slices while > >> we have view semantics. We can't safely do the reallocation of > memory [1]. > > > > That's fair enough. But then I think NumPy should consistently > > typecheck all assignmetns and throw an exception if the user attempts > > an assignment which looses information. > > > > Yeah, there's no doubt in my mind that this is a bug, if for no other > reason than this inconsistency: > > > One place where Numpy differs from MatLab is the way memory is handled. > MatLab is always generating new arrays, so for efficiency it is worth > preallocating arrays and then filling in the parts. This is not the case > in Numpy where lists can be used for things that grow and subarrays are > views. Consequently, preallocating arrays in Numpy should be rare and > used when either the values have to be generated explicitly, which is > what you see when using the indexes in your first example. As to > assignment between arrays, it is a mixed question. The problem again is > memory usage. For large arrays, it makes since to do automatic > conversions, as is also the case in functions taking output arrays, > because the typecast can be pushed down into C where it is time and > space efficient, whereas explicitly converting the array uses up > temporary space. However, I can imagine an explicit typecast function, > something like > > a[...] = typecast(b) > > that would replace the current behavior. I think the typecast function > could be implemented by returning a view of b with a castable flag set > to true, that should supply enough information for the assignment > operator to do its job. This might be a good addition for Numpy 1.1. While that seems like an ok idea, I'm still not sure what's wrong with raising an exception when there will be information loss. The exception is already raised with standard python complex objects. I can think of many times in my code where explicit looping is a necessity, so pre-allocating the array is the only way to go. -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From oliphant at enthought.com Mon Jan 7 13:31:26 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 12:31:26 -0600 Subject: [Numpy-discussion] Overloading sqrt(5.5)*myvector In-Reply-To: <4771F0A8.8070304@ncsu.edu> References: <476AA418.4080108@ncsu.edu> <4771F0A8.8070304@ncsu.edu> Message-ID: <47826FFE.4020207@enthought.com> Bruce Sherwood wrote: > Sorry to repeat myself and be insistent, but could someone please at > least comment on whether I'm doing anything obviously wrong, even if you > don't immediately have a solution to my serious problem? There was no > response to my question (see copy below) which I sent to both the numpy > and Boost mailing lists. > > To the numpy experts: Is there something wrong, or something I > could/should change in how I'm trying to overload multiplication of a > numpy square root (or other numpy function) times my own "vector" > object? I'm seeing a huge performance hit in going from Numeric to numpy > because Numeric sqrt returned float whereas numpy sqrt returns > numpy.float64, so that the result is not one of my vector objects. I > don't have a problem with myvector*sqrt(5.5). > I'm not sure how to help explicitly as I do not know Boost very well and so am not clear on what specifically was done that is now not working (or where the problem lies). I can, however, explain the numpy.float64 Python object. This Python object is binary-compatible with a Python float object (and in fact is a C subtype of it). It is a simple wrapper around a regular C-double. There are lots of possibilities. The most likely issue in my mind is a coercion issue. Whereas the Python float probably bailed when it saw myvector as the other argument, the numpy.float64 is seeing myvector as something that can be converted into a NumPy array and therefore going ahead and doing the conversion and performing the multiplication (which will involve all the overhead for general ufuncs). Thus, there needs to be a way to signal to numpy.float64 that it should let the other object handle it. I'm not sure what the right solution is here, but I think that is the problem. The good news is that we can change it if we figure out what the right thing to do is. Best regards, -Travis O. From faltet at carabos.com Mon Jan 7 13:42:40 2008 From: faltet at carabos.com (Francesc Altet) Date: Mon, 7 Jan 2008 19:42:40 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: References: Message-ID: <200801071942.40634.faltet@carabos.com> A Monday 07 January 2008, Nils Wagner escrigu?: > >>> numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex192)) > > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute > 'complex192' > > >>> numpy.__version__ > > '1.0.5.dev4673' It seems like you are using a 64-bit platform, and they tend to have complex256 (quad-precision) types instead of complex192 (extended-precision) typical in 32-bit platforms. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From zpincus at stanford.edu Mon Jan 7 13:47:52 2008 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 7 Jan 2008 13:47:52 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <478268E0.4000804@ou.edu> References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> <478268E0.4000804@ou.edu> Message-ID: <8C0A64C3-8BEE-42AE-9F1A-EF8B1892C9E8@stanford.edu> >> For large arrays, it makes since to do automatic >> conversions, as is also the case in functions taking output arrays, >> because the typecast can be pushed down into C where it is time and >> space efficient, whereas explicitly converting the array uses up >> temporary space. However, I can imagine an explicit typecast >> function, >> something like >> >> a[...] = typecast(b) >> >> that would replace the current behavior. I think the typecast >> function >> could be implemented by returning a view of b with a castable flag >> set >> to true, that should supply enough information for the assignment >> operator to do its job. This might be a good addition for Numpy 1.1. > > While that seems like an ok idea, I'm still not sure what's wrong with > raising an exception when there will be information loss. The > exception > is already raised with standard python complex objects. I can > think of > many times in my code where explicit looping is a necessity, so > pre-allocating the array is the only way to go. The issue Charles is dealing with here is how to *suppress* the proposed exception in cases (as the several that I described) where the information loss is explicitly desired. With what's currently in numpy now, you would have to do something like this: A[...] = B.astype(A.dtype) to set a portion of A to B, unless you are *certain* that A and B are of compatible types. This is ugly and also bug-prone, seeing as how there's some violation of the don't-repeat-yourself principle. (I.e. A is mentioned twice, and to change the code to use a different array, you need to change the variable name A twice.) Moreover, and worse, the phrase 'A = B.astype(A.dtype)' creates and destroys a temporary array the same size as B. It's equivalent to: temp = B.astype(A.dtype) A[...] = temp Which is no good if B is very large. Currently, the type conversion in 'A[...] = B' cases is handled implicitly, deep down in the C code where it is very very fast, and no temporary array is made. Charles suggests a 'typecast' operator that would set a flag on the array B so that trying to convert it would *not* raise an exception, allowing for the fast, C-level conversion. (This assumes your proposed change in which by default such conversions would raise exceptions.) This 'typecast' operation wouldn't do anything but set a flag, so it doesn't create a temporary array or do any extra work. But still, every time that you are not *certain* what the type of a result from a given operation is, any code that looks like: A[i] = calculate(...) will need to look like this instead: A[i] = typecast(calculate(...)) I agree with others that such assignments aren't highly common, but there will be broken code from this. And as Charles demonstrates, getting the details right of how to implement such a change is non- trivial. Zach From charlesr.harris at gmail.com Mon Jan 7 13:54:47 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 11:54:47 -0700 Subject: [Numpy-discussion] CASTABLE flag Message-ID: Hi All, I'm thinking that one way to make the automatic type conversion a bit safer to use would be to add a CASTABLE flag to arrays. Then we could write something like a[...] = typecast(b) where typecast returns a view of b with the CASTABLE flag set so that the assignment operator can check whether to implement the current behavior or to raise an error. Maybe this could even be part of the dtype scalar, although that might cause a lot of problems with the current default conversions. What do folks think? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Jan 7 13:57:37 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 12:57:37 -0600 Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector In-Reply-To: <4776D1A5.60106@ncsu.edu> References: <4771F0AE.40406@ncsu.edu> <7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com> <47732B84.70400@ncsu.edu> <4773325D.3040907@gmail.com> <4773BD97.1000703@ncsu.edu> <4773C17B.4050802@ncsu.edu> <4775DF5D.3030400@ncsu.edu> <7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com> <4776B111.9020607@ncsu.edu> <4776C3FC.7030101@ncsu.edu> <4776D1A5.60106@ncsu.edu> Message-ID: <47827621.3090906@enthought.com> Bruce Sherwood wrote: > Okay, I've implemented the scheme below that was proposed by Scott > Daniels on the VPython mailing list, and it solves my problem. It's also > much faster than using numpy directly: even with the "def "and "if" > overhead: sqrt(scalar) is over 3 times faster than the numpy sqrt, and > sqrt(array) is very nearly as fast as the numpy sqrt. > Using math.sqrt short-circuits the ufunc approach of returning numpy scalars (with all the methods and attributes of 0-d arrays --- which is really their primary reason for being). It is also faster because it avoids the numpy ufunc machinery (which is some overhead --- the error setting control and broadcasting facility doesn't happen for free). > Thanks to those who made suggestions. There remains the question of why > operator overloading of the kind I've described worked with Numeric and > Boost but not with numpy and Boost. Basically, it boils down to the fact that I took a shortcut with implementing generic multiplication (which all the scalars use for now) on numpy scalars so that the multiplication ufunc is called when they are encountered. Thus, when the numpy.float64 is first (its multiply implementation gets called first) and it uses the equivalent of ufunc.multiply(scalar, vector) I suspect that because your vector can be converted to an array, this procedure works (albeit more slowly than you would like), and so the vector object never gets a chance to try. A quick fix is to make it so that ufunc.multiply(scalar, vector) raises NotImplemented which may not be desireable, either. Alternatively, the generic scalar operations should probably not be so "inclusive" and should allow the other object a chance to perform the operation more often (by returning NotImplemented). -Travis O. From oliphant at enthought.com Mon Jan 7 14:00:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 13:00:24 -0600 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: References: Message-ID: <478276C8.2090806@enthought.com> Charles R Harris wrote: > Hi All, > > I'm thinking that one way to make the automatic type conversion a bit > safer to use would be to add a CASTABLE flag to arrays. Then we could > write something like > > a[...] = typecast(b) > > where typecast returns a view of b with the CASTABLE flag set so that > the assignment operator can check whether to implement the current > behavior or to raise an error. Maybe this could even be part of the > dtype scalar, although that might cause a lot of problems with the > current default conversions. What do folks think? That is an interesting approach. The issue raised of having to convert lines of code that currently work (which does implicit casting) would still be there (like ndimage), but it would not cause unnecessary data copying, and would help with this complaint (that I've heard before and have some sympathy towards). I'm intrigued. -Travis O. From nwagner at iam.uni-stuttgart.de Mon Jan 7 14:00:34 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 07 Jan 2008 20:00:34 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: <200801071942.40634.faltet@carabos.com> References: <200801071942.40634.faltet@carabos.com> Message-ID: On Mon, 7 Jan 2008 19:42:40 +0100 Francesc Altet wrote: > A Monday 07 January 2008, Nils Wagner escrigu?: >> >>> numpy.sqrt(numpy.array([-1.0], >>dtype=numpy.complex192)) >> >> Traceback (most recent call last): >> File "", line 1, in >> AttributeError: 'module' object has no attribute >> 'complex192' >> >> >>> numpy.__version__ >> >> '1.0.5.dev4673' > > It seems like you are using a 64-bit platform, and they >tend to have > complex256 (quad-precision) types instead of complex192 > (extended-precision) typical in 32-bit platforms. > > Cheers, > > -- >>0,0< Francesc Altet http://www.carabos.com/ > V V C?rabos Coop. V. Enjoy Data > "-" > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion >>> import numpy >>> numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex256)) Segmentation fault Shall I file a bug report ? Nils From rmay at ou.edu Mon Jan 7 14:07:35 2008 From: rmay at ou.edu (Ryan May) Date: Mon, 07 Jan 2008 13:07:35 -0600 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: <8C0A64C3-8BEE-42AE-9F1A-EF8B1892C9E8@stanford.edu> References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> <478268E0.4000804@ou.edu> <8C0A64C3-8BEE-42AE-9F1A-EF8B1892C9E8@stanford.edu> Message-ID: <47827877.6000308@ou.edu> Zachary Pincus wrote: >>> For large arrays, it makes since to do automatic >>> conversions, as is also the case in functions taking output arrays, >>> because the typecast can be pushed down into C where it is time and >>> space efficient, whereas explicitly converting the array uses up >>> temporary space. However, I can imagine an explicit typecast >>> function, >>> something like >>> >>> a[...] = typecast(b) >>> >>> that would replace the current behavior. I think the typecast >>> function >>> could be implemented by returning a view of b with a castable flag >>> set >>> to true, that should supply enough information for the assignment >>> operator to do its job. This might be a good addition for Numpy 1.1. >> While that seems like an ok idea, I'm still not sure what's wrong with >> raising an exception when there will be information loss. The >> exception >> is already raised with standard python complex objects. I can >> think of >> many times in my code where explicit looping is a necessity, so >> pre-allocating the array is the only way to go. > > The issue Charles is dealing with here is how to *suppress* the > proposed exception in cases (as the several that I described) where > the information loss is explicitly desired. > > With what's currently in numpy now, you would have to do something > like this: > A[...] = B.astype(A.dtype) > to set a portion of A to B, unless you are *certain* that A and B are > of compatible types. > > This is ugly and also bug-prone, seeing as how there's some violation > of the don't-repeat-yourself principle. (I.e. A is mentioned twice, > and to change the code to use a different array, you need to change > the variable name A twice.) > > Moreover, and worse, the phrase 'A = B.astype(A.dtype)' creates and > destroys a temporary array the same size as B. It's equivalent to: > temp = B.astype(A.dtype) > A[...] = temp > > Which is no good if B is very large. Currently, the type conversion > in 'A[...] = B' cases is handled implicitly, deep down in the C code > where it is very very fast, and no temporary array is made. > > Charles suggests a 'typecast' operator that would set a flag on the > array B so that trying to convert it would *not* raise an exception, > allowing for the fast, C-level conversion. (This assumes your > proposed change in which by default such conversions would raise > exceptions.) This 'typecast' operation wouldn't do anything but set a > flag, so it doesn't create a temporary array or do any extra work. > > But still, every time that you are not *certain* what the type of a > result from a given operation is, any code that looks like: > A[i] = calculate(...) > will need to look like this instead: > A[i] = typecast(calculate(...)) > > I agree with others that such assignments aren't highly common, but > there will be broken code from this. And as Charles demonstrates, > getting the details right of how to implement such a change is non- > trivial. > I agree that some of the other options with typecast/astype look horrible, as well as that unnecessary temporaries are bad. Maybe I'm too focused on just the complex case, where all you really need is to use .real to get only the real part (and coincidentally works with float and int arrays). The nice part about using .real is that it explicitly states what you're looking for. I'm not personally interested in anything other than this silent complex->float conversion. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From peridot.faceted at gmail.com Mon Jan 7 14:08:19 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 7 Jan 2008 14:08:19 -0500 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> Message-ID: On 07/01/2008, Charles R Harris wrote: > > One place where Numpy differs from MatLab is the way memory is handled. > MatLab is always generating new arrays, so for efficiency it is worth > preallocating arrays and then filling in the parts. This is not the case in > Numpy where lists can be used for things that grow and subarrays are views. > Consequently, preallocating arrays in Numpy should be rare and used when > either the values have to be generated explicitly, which is what you see > when using the indexes in your first example. As to assignment between > arrays, it is a mixed question. The problem again is memory usage. For large > arrays, it makes since to do automatic conversions, as is also the case in > functions taking output arrays, because the typecast can be pushed down into > C where it is time and space efficient, whereas explicitly converting the > array uses up temporary space. However, I can imagine an explicit typecast > function, something like > > a[...] = typecast(b) > > that would replace the current behavior. I think the typecast function could > be implemented by returning a view of b with a castable flag set to true, > that should supply enough information for the assignment operator to do its > job. This might be a good addition for Numpy 1.1. This is introducing a fairly complex mechanism to cover, as far as I can see, two cases: conversion of complex to real, and conversion of float to integer. Conversion of complex to real can already be done explicitly without a temporary: a[...] = b.real That leaves only conversion of float to integer. Does this single case cause enough confusion to warrant an exception and a way around it? Anne From charlesr.harris at gmail.com Mon Jan 7 14:13:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 12:13:56 -0700 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: <478276C8.2090806@enthought.com> References: <478276C8.2090806@enthought.com> Message-ID: On Jan 7, 2008 12:00 PM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > Hi All, > > > > I'm thinking that one way to make the automatic type conversion a bit > > safer to use would be to add a CASTABLE flag to arrays. Then we could > > write something like > > > > a[...] = typecast(b) > > > > where typecast returns a view of b with the CASTABLE flag set so that > > the assignment operator can check whether to implement the current > > behavior or to raise an error. Maybe this could even be part of the > > dtype scalar, although that might cause a lot of problems with the > > current default conversions. What do folks think? > > That is an interesting approach. The issue raised of having to > convert lines of code that currently work (which does implicit casting) > would still be there (like ndimage), but it would not cause unnecessary > data copying, and would help with this complaint (that I've heard before > and have some sympathy towards). > > I'm intrigued. Maybe we could also set a global flag, typesafe, that in the current Numpy version would default to false, giving current behavior, but could be set true to get the new behavior. Then when Numpy 1.1 comes out we could make the global default true. That way folks could keep the current code working until they are ready to use the typesafe feature. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jan 7 14:25:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 12:25:04 -0700 Subject: [Numpy-discussion] Nasty bug using pre-initialized arrays In-Reply-To: References: <477EAEE9.1090501@gmail.com> <47824978.2050601@ou.edu> Message-ID: On Jan 7, 2008 12:08 PM, Anne Archibald wrote: > On 07/01/2008, Charles R Harris wrote: > > > > One place where Numpy differs from MatLab is the way memory is handled. > > MatLab is always generating new arrays, so for efficiency it is worth > > preallocating arrays and then filling in the parts. This is not the case > in > > Numpy where lists can be used for things that grow and subarrays are > views. > > Consequently, preallocating arrays in Numpy should be rare and used when > > either the values have to be generated explicitly, which is what you see > > when using the indexes in your first example. As to assignment between > > arrays, it is a mixed question. The problem again is memory usage. For > large > > arrays, it makes since to do automatic conversions, as is also the case > in > > functions taking output arrays, because the typecast can be pushed down > into > > C where it is time and space efficient, whereas explicitly converting > the > > array uses up temporary space. However, I can imagine an explicit > typecast > > function, something like > > > > a[...] = typecast(b) > > > > that would replace the current behavior. I think the typecast function > could > > be implemented by returning a view of b with a castable flag set to > true, > > that should supply enough information for the assignment operator to do > its > > job. This might be a good addition for Numpy 1.1. > > This is introducing a fairly complex mechanism to cover, as far as I > can see, two cases: conversion of complex to real, and conversion of > float to integer. Conversion of complex to real can already be done > explicitly without a temporary: > > a[...] = b.real > > That leaves only conversion of float to integer. > > Does this single case cause enough confusion to warrant an exception > and a way around it? I think it could be something like C++ where implicit downcasts in general are flagged. If we went that way, conversions like uint32 to int32 would also be flagged. The question is how much care we want to take with types. Since Numpy is much more type oriented than, say, MatLab, such type safety might be a useful addition. Python scalar types would remain castable, so you wouldn't have to typecast every darn thing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From strawman at astraw.com Mon Jan 7 14:25:31 2008 From: strawman at astraw.com (Andrew Straw) Date: Mon, 07 Jan 2008 11:25:31 -0800 Subject: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars? Message-ID: <47827CAB.7080607@astraw.com> Hi, I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is having troubles accessing a numpy scalar with the __array_interface__. Is this supposed to work? Or should __array_interface__ trigger an AttributeError on a numpy scalar? Note that I haven't done any digging on this myself... -Andrew -------------- next part -------------- An embedded message was scrubbed... From: "SourceForge.net" Subject: [PyOpenGL-Devel] [ pyopengl-Bugs-1827190 ] Default (Numpy) array return values not accepted Date: Mon, 07 Jan 2008 10:32:51 -0800 Size: 6481 URL: From sransom at nrao.edu Mon Jan 7 14:51:21 2008 From: sransom at nrao.edu (Scott Ransom) Date: Mon, 7 Jan 2008 14:51:21 -0500 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: References: <478276C8.2090806@enthought.com> Message-ID: <200801071451.21339.sransom@nrao.edu> On Monday 07 January 2008 02:13:56 pm Charles R Harris wrote: > On Jan 7, 2008 12:00 PM, Travis E. Oliphant wrote: > > Charles R Harris wrote: > > > Hi All, > > > > > > I'm thinking that one way to make the automatic type conversion a > > > bit safer to use would be to add a CASTABLE flag to arrays. Then > > > we could write something like > > > > > > a[...] = typecast(b) > > > > > > where typecast returns a view of b with the CASTABLE flag set so > > > that the assignment operator can check whether to implement the > > > current behavior or to raise an error. Maybe this could even be > > > part of the dtype scalar, although that might cause a lot of > > > problems with the current default conversions. What do folks > > > think? > > > > That is an interesting approach. The issue raised of having to > > convert lines of code that currently work (which does implicit > > casting) would still be there (like ndimage), but it would not > > cause unnecessary data copying, and would help with this complaint > > (that I've heard before and have some sympathy towards). > > > > I'm intrigued. > > Maybe we could also set a global flag, typesafe, that in the current > Numpy version would default to false, giving current behavior, but > could be set true to get the new behavior. Then when Numpy 1.1 comes > out we could make the global default true. That way folks could keep > the current code working until they are ready to use the typesafe > feature. I'm a bit confused as to which types of casting you are proposing to change. As has been pointed out by several people, users very often _want_ to "lose information". And as I pointed out, it is one of the reasons why we are all using numpy today as opposed to numeric! I'd bet that the vast majority of the people on this list believe that the OPs problem of complex numbers being automatically cast into floats is a real problem. Fine. We should be able to raise an exception in that case. However, two other very common cases of "lost information" are not obviously a problems, and are (for many of us) the _preferred_ actions. These examples are: 1. Performing floating point math in higher precision, but casting to a lower-precision float if that float is on the lhs of the assignment. For example: In [22]: a = arange(5, dtype=float32) In [23]: a += arange(5.0) In [24]: a Out[24]: array([ 0., 2., 4., 6., 8.], dtype=float32) To me, that is fantastic. I've obviously explicitly requested that I want "a" to hold 32-bit floats. And if I'm careful and use in-place math, I get 32-bit floats at the end (and no problems with large temporaries or memory doubling by an automatic cast to float64). In [25]: a = a + arange(5.0) In [26]: a Out[26]: array([ 0., 3., 6., 9., 12.]) In this case, I'm reassigning "a" from 32-bits to 64-bits because I'm not using in-place math. The temporary array created on the rhs defines the type of the new assignment. Once again, I think this is good. 2. Similarly, if I want to stuff floats into a int array: In [28]: a Out[28]: array([0, 1, 2, 3, 4]) In [29]: a += 2.5 In [30]: a Out[30]: array([2, 3, 4, 5, 6]) Here, I get C-like rounding/casting of my originally integer array because I'm using in-place math. This is often a very useful behavior. In [31]: a = a + 2.5 In [32]: a Out[32]: array([ 4.5, 5.5, 6.5, 7.5, 8.5]) But here, without the in-place math, a gets converted to doubles. I can certainly say that in my code (which is used by a fair number of people in my field), each of these use cases are common. And I think they are one of the _strengths_ of numpy. I will be very disappointed if this default behavior changes. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From darren.dale at cornell.edu Mon Jan 7 15:00:21 2008 From: darren.dale at cornell.edu (Darren Dale) Date: Mon, 7 Jan 2008 15:00:21 -0500 Subject: [Numpy-discussion] Does float16 exist? Message-ID: <200801071500.21275.darren.dale@cornell.edu> One of my collaborators would like to use 16bit float arrays. According to http://www.scipy.org/Tentative_NumPy_Tutorial, and references to float16 in numpy.core.numerictypes, it appears that this should be possible, but the following doesnt work: a=arange(10, dtype='float16') TypeError: data type not understood Can anyone offer some advice? Thanks, Darren -- Darren S. Dale, Ph.D. Staff Scientist Cornell High Energy Synchrotron Source Cornell University 275 Wilson Lab Rt. 366 & Pine Tree Road Ithaca, NY 14853 darren.dale at cornell.edu office: (607) 255-3819 fax: (607) 255-9001 http://www.chess.cornell.edu From oliphant at enthought.com Mon Jan 7 15:06:38 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 14:06:38 -0600 Subject: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars? In-Reply-To: <47827CAB.7080607@astraw.com> References: <47827CAB.7080607@astraw.com> Message-ID: <4782864E.3020703@enthought.com> Andrew Straw wrote: > Hi, > > I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is > having troubles accessing a numpy scalar with the __array_interface__. > Is this supposed to work? Or should __array_interface__ trigger an > AttributeError on a numpy scalar? Note that I haven't done any digging > on this myself... > Yes, the __array_interface__ approach works (but read-only). In fact, what happens is that a 0-d array is created and you are given the information for it. -Travis O. From matthieu.brucher at gmail.com Mon Jan 7 15:06:54 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 7 Jan 2008 21:06:54 +0100 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <200801071500.21275.darren.dale@cornell.edu> References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: float16 is not defined in my version of Numpy :( Matthieu 2008/1/7, Darren Dale : > > One of my collaborators would like to use 16bit float arrays. According to > http://www.scipy.org/Tentative_NumPy_Tutorial, and references to float16 > in > numpy.core.numerictypes, it appears that this should be possible, but the > following doesnt work: > > a=arange(10, dtype='float16') > TypeError: data type not understood > > Can anyone offer some advice? > > Thanks, > Darren > > -- > Darren S. Dale, Ph.D. > Staff Scientist > Cornell High Energy Synchrotron Source > Cornell University > 275 Wilson Lab > Rt. 366 & Pine Tree Road > Ithaca, NY 14853 > > darren.dale at cornell.edu > office: (607) 255-3819 > fax: (607) 255-9001 > http://www.chess.cornell.edu > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Jan 7 15:09:33 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 14:09:33 -0600 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <200801071500.21275.darren.dale@cornell.edu> References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: <478286FD.7080006@enthought.com> Darren Dale wrote: > One of my collaborators would like to use 16bit float arrays. According to > http://www.scipy.org/Tentative_NumPy_Tutorial, and references to float16 in > numpy.core.numerictypes, it appears that this should be possible, but the > following doesnt work: > No, it's only possible, if the C-implementation NumPy was compiled with has 16-bits for its float scalar. Only float, double, and long double are implemented. These translate to various bit-widths on different platforms. numerictypes is overly aggressive at guessing what they might translate to. -Travis O. From dmitrey.kroshko at scipy.org Mon Jan 7 15:13:08 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 07 Jan 2008 22:13:08 +0200 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? Message-ID: <478287D4.5020704@scipy.org> Some days ago there was mentioned a parallel numpy that is developed by Brian Granger. Does the project have any blog or website? Has it any description about API and abilities? When 1st release is intended? Regards, D From faltet at carabos.com Mon Jan 7 15:16:17 2008 From: faltet at carabos.com (Francesc Altet) Date: Mon, 7 Jan 2008 21:16:17 +0100 Subject: [Numpy-discussion] Bugs using complex192 In-Reply-To: References: <200801071942.40634.faltet@carabos.com> Message-ID: <200801072116.18057.faltet@carabos.com> A Monday 07 January 2008, Nils Wagner escrigu?: > On Mon, 7 Jan 2008 19:42:40 +0100 > > Francesc Altet wrote: > > A Monday 07 January 2008, Nils Wagner escrigu?: > >> >>> numpy.sqrt(numpy.array([-1.0], > >> > >>dtype=numpy.complex192)) > >> > >> Traceback (most recent call last): > >> File "", line 1, in > >> AttributeError: 'module' object has no attribute > >> 'complex192' > >> > >> >>> numpy.__version__ > >> > >> '1.0.5.dev4673' > > > > It seems like you are using a 64-bit platform, and they > >tend to have > > complex256 (quad-precision) types instead of complex192 > > (extended-precision) typical in 32-bit platforms. > > > > Cheers, > > > > -- > > > >>0,0< Francesc Altet http://www.carabos.com/ > > > > V V C?rabos Coop. V. Enjoy Data > > "-" > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > >>> import numpy > >>> numpy.sqrt(numpy.array([-1.0], dtype=numpy.complex256)) > > Segmentation fault > > Shall I file a bug report ? Yes, I think so. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tim.hochberg at ieee.org Mon Jan 7 15:16:46 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 7 Jan 2008 13:16:46 -0700 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: <200801071451.21339.sransom@nrao.edu> References: <478276C8.2090806@enthought.com> <200801071451.21339.sransom@nrao.edu> Message-ID: Another possible approach is to treat downcasting similar to underflow. That is give it it's own flag in the errstate and people can set it to ignore, warn or raise on downcasting as desired. One could potentially have two flags, one for downcasting across kinds (float->int, int->bool) and one for downcasting within kinds (float64->float32). In this case, I personally would set the first to raise and the second to ignore and would suggest that as the default. IMO: 1. It's a no brainer to raise and exception when assigning a complex value to a float or integer target. Using "Z.real" or "Z.imag" is clearer and has the same performance. 2. I'm fairly dubious about assigning float to ints as is. First off it looks like a bug magnet to me due to accidentally assigning a floating point value to a target that one believes to be float but is in fact integer. Second, C-style rounding is pretty evil; it's not always consistent across platforms, so relying on it for anything other than truncating already integral values is asking for trouble. 3. Downcasting within kinds seems much less hazardous than downcasting across kinds, although I'd still be happy to be able regulate it with errstate. -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From strawman at astraw.com Mon Jan 7 15:19:01 2008 From: strawman at astraw.com (Andrew Straw) Date: Mon, 07 Jan 2008 12:19:01 -0800 Subject: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars? In-Reply-To: <4782864E.3020703@enthought.com> References: <47827CAB.7080607@astraw.com> <4782864E.3020703@enthought.com> Message-ID: <47828935.6050903@astraw.com> Travis E. Oliphant wrote: > Andrew Straw wrote: >> Hi, >> >> I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is >> having troubles accessing a numpy scalar with the __array_interface__. >> Is this supposed to work? Or should __array_interface__ trigger an >> AttributeError on a numpy scalar? Note that I haven't done any digging >> on this myself... >> > > Yes, the __array_interface__ approach works (but read-only). In fact, > what happens is that a 0-d array is created and you are given the > information for it. > Thanks. How is the array reference kept alive? Maybe that's Mike's issue? From peridot.faceted at gmail.com Mon Jan 7 15:24:17 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 7 Jan 2008 15:24:17 -0500 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: References: <478276C8.2090806@enthought.com> <200801071451.21339.sransom@nrao.edu> Message-ID: On 07/01/2008, Timothy Hochberg wrote: > I'm fairly dubious about assigning float to ints as is. First off it looks > like a bug magnet to me due to accidentally assigning a floating point value > to a target that one believes to be float but is in fact integer. Second, > C-style rounding is pretty evil; it's not always consistent across > platforms, so relying on it for anything other than truncating already > integral values is asking for trouble. I'm not sure I agree that this is a bug magnet: if your array is integer and you think it's float, you're going to get burned sooner rather than later, whether you assign to it or not. The only case I can see would be a problem would be if zeros() and ones() created integers - which they don't. Anne From robert.kern at gmail.com Mon Jan 7 15:27:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 07 Jan 2008 14:27:37 -0600 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478287D4.5020704@scipy.org> References: <478287D4.5020704@scipy.org> Message-ID: <47828B39.4080808@gmail.com> dmitrey wrote: > Some days ago there was mentioned a parallel numpy that is developed by > Brian Granger. > > Does the project have any blog or website? Has it any description about > API and abilities? When 1st release is intended? It is just starting development. There is nothing to see, yet, as far as I know. When Brian has something to show, I am sure he will announce it here. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zpincus at stanford.edu Mon Jan 7 15:29:42 2008 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 7 Jan 2008 15:29:42 -0500 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: <200801071451.21339.sransom@nrao.edu> References: <478276C8.2090806@enthought.com> <200801071451.21339.sransom@nrao.edu> Message-ID: <531BD2E2-AE6E-4395-9378-BB2E7700C651@stanford.edu> Hello all, In order to help make things regarding this casting issue more explicit, let me present the following table of potential "down-casts". (Also, for the record, nobody is proposing automatic up-casting of any kind. The proposals on the table focus on preventing some or all implicit down-casting.) (1) complex -> anything else: Data is lost wholesale. (2) large float -> smaller float (also large complex -> smaller complex): Precision is lost. (3) float -> integer: Precision is lost, to a much greater degree than (2). (4) large integer -> smaller integer (also signed/unsigned conversions): Data gets lost/mangled/wrapped around. The original requests for exceptions to be raised focused on case (1), where it is most clear that loss-of-data is happening in a way that is unlikely to be intentional. Robert Kern suggested that exceptions might be raised for cases (1) and (3), which are cross-kind casts, but not for within-kind conversions like (2) and (4). However, I personally don't think that down-converting from float to int is any more or less fraught than converting from int32 to int8: if you need a warning/exception in one case, you'd need it in the rest. Moreover, there's the principle of least surprise, which would suggest that complex rules for when exceptions get raised based on the kind of conversion being made is just asking for trouble. So, as a poll, if you are in favor of exceptions instead of automatic down-conversion, where do you draw the line? What causes an error? Robert seemed to be in favor of (1) and (3). Anne seemed to think that only (1) was problematic enough to worry about. I am personally cool toward the exceptions, but I think that case (4) is just as "bad" as case (3) in terms of data-loss, though I agree that case (1) seems the worst (and I don't really find any of them particularly bad, though case (1) is something of an oddity for newcomers, I agree.) Finally, if people really want these sort of warnings, I would suggest that they be rolled into the get/setoptions mechanism, so that there's a fine-grained mechanism for turning them to off/warn/ raise exception. Zach From charlesr.harris at gmail.com Mon Jan 7 15:53:06 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 13:53:06 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <200801071500.21275.darren.dale@cornell.edu> References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: Hi, On Jan 7, 2008 1:00 PM, Darren Dale wrote: > One of my collaborators would like to use 16bit float arrays. According to > http://www.scipy.org/Tentative_NumPy_Tutorial, and references to float16 > in > numpy.core.numerictypes, it appears that this should be possible, but the > following doesnt work: > > a=arange(10, dtype='float16') > TypeError: data type not understood > > Can anyone offer some advice? > Does he care about speed? I think some graphics cards might support float16, but any normal implementation in C would reguire software floating point, a new type, and you couldn't rely on the normal operators. It might be doable in C++ with operator overloading and templates, but doing it in C would be a real hassle. > Thanks, > Darren Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jan 7 16:00:42 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Jan 2008 14:00:42 -0700 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: References: <478276C8.2090806@enthought.com> <200801071451.21339.sransom@nrao.edu> Message-ID: Hi, On Jan 7, 2008 1:16 PM, Timothy Hochberg wrote: > > Another possible approach is to treat downcasting similar to underflow. > That is give it it's own flag in the errstate and people can set it to > ignore, warn or raise on downcasting as desired. One could potentially have > two flags, one for downcasting across kinds (float->int, int->bool) and one > for downcasting within kinds (float64->float32). In this case, I personally > would set the first to raise and the second to ignore and would suggest that > as the default. > > IMO: > > 1. It's a no brainer to raise and exception when assigning a complex > value to a float or integer target. Using "Z.real" or "Z.imag" is > clearer and has the same performance. > 2. I'm fairly dubious about assigning float to ints as is. First off > it looks like a bug magnet to me due to accidentally assigning a floating > point value to a target that one believes to be float but is in fact > integer. Second, C-style rounding is pretty evil; it's not always consistent > across platforms, so relying on it for anything other than truncating > already integral values is asking for trouble. > 3. Downcasting within kinds seems much less hazardous than > downcasting across kinds, although I'd still be happy to be able regulate it > with errstate. > > Maybe a combination of a typecast function and the errstate would work well. The typecast function would provide a clear local override of the default errstate flags, while the user would have the option to specify what sort of behavior they care about in general. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Jan 7 16:01:45 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 15:01:45 -0600 Subject: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars? In-Reply-To: <47828935.6050903@astraw.com> References: <47827CAB.7080607@astraw.com> <4782864E.3020703@enthought.com> <47828935.6050903@astraw.com> Message-ID: <47829339.6090305@enthought.com> Andrew Straw wrote: > Travis E. Oliphant wrote: > >> Andrew Straw wrote: >> >>> Hi, >>> >>> I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is >>> having troubles accessing a numpy scalar with the __array_interface__. >>> Is this supposed to work? Or should __array_interface__ trigger an >>> AttributeError on a numpy scalar? Note that I haven't done any digging >>> on this myself... >>> >>> >> Yes, the __array_interface__ approach works (but read-only). In fact, >> what happens is that a 0-d array is created and you are given the >> information for it. >> >> > > Thanks. How is the array reference kept alive? Maybe that's Mike's issue? > Hmm... I thought that I thought through this one. But, as I look at the code now, I don't think it is kept alive. I suspect that is the problem. -Travis > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From zpincus at stanford.edu Mon Jan 7 16:03:14 2008 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 7 Jan 2008 16:03:14 -0500 Subject: [Numpy-discussion] Unexpected integer overflow handling Message-ID: Hello all, On my (older) version of numpy (1.0.4.dev3896), I found several oddities in the handling of assignment of long-integer values to integer arrays: In : numpy.array([2**31], dtype=numpy.int8) ------------------------------------------------------------------------ --- ValueError Traceback (most recent call last) /Users/zpincus/ ValueError: setting an array element with a sequence. While this might be reasonable to be an error condition, the precise error raised seems not quite right! But not all overflow errors are caught in this way: In : numpy.array([2**31-1], dtype=numpy.int8) Out: array([-1], dtype=int8) As above, numpy is quite happy allowing overflows; it just breaks when doing a python long-int to int conversion. The conversion from numpy-long-int to int does the "right thing" though (if by "right thing" you mean "allows silent overflow", which is a matter of discussion elsewhere right now...): In : numpy.array(numpy.array([2**31], dtype=numpy.int64), dtype=numpy.int8) Out: array([0], dtype=int8) At least on item assignment, the overflow exception is less odd: In : a = numpy.empty(shape=(1,), dtype=numpy.int8) In : a[0] = 2**31 ------------------------------------------------------------------------ --- OverflowError Traceback (most recent call last) /Users/zpincus/ OverflowError: long int too large to convert to int Things work right with array element assignment: In : a[0] = numpy.array([2**31], dtype=numpy.int64)[0] But break again with array scalars, and with the strange ValueError again! In : a[0] = numpy.array(2**31, dtype=numpy.int64) ------------------------------------------------------------------------ --- ValueError Traceback (most recent call last) /Users/zpincus/ ValueError: setting an array element with a sequence. Note that non-long-int-to-int array scalar conversions work: In : a[0] = numpy.array(2**31-1, dtype=numpy.int64) Is this still the case for the current version of numpy? Best, Zach From darren.dale at cornell.edu Mon Jan 7 16:07:56 2008 From: darren.dale at cornell.edu (Darren Dale) Date: Mon, 7 Jan 2008 16:07:56 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: <200801071607.56454.darren.dale@cornell.edu> On Monday 07 January 2008 03:53:06 pm Charles R Harris wrote: > Hi, > > On Jan 7, 2008 1:00 PM, Darren Dale wrote: > > One of my collaborators would like to use 16bit float arrays. According > > to http://www.scipy.org/Tentative_NumPy_Tutorial, and references to > > float16 in > > numpy.core.numerictypes, it appears that this should be possible, but the > > following doesnt work: > > > > a=arange(10, dtype='float16') > > TypeError: data type not understood > > > > Can anyone offer some advice? > > Does he care about speed? I think some graphics cards might support > float16, but any normal implementation in C would reguire software floating > point, a new type, and you couldn't rely on the normal operators. It might > be doable in C++ with operator overloading and templates, but doing it in C > would be a real hassle. He is analyzing many sets of 2D data, each array is 3450x3450 pixels. I think memory/space is a greater concern at the moment than speed, 24 vs 48 megabytes. They are more concerned with processing these arrays than viewing them. Darren From ellisonbg.net at gmail.com Mon Jan 7 16:10:10 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Mon, 7 Jan 2008 14:10:10 -0700 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478287D4.5020704@scipy.org> References: <478287D4.5020704@scipy.org> Message-ID: <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> Dmitrey, This work is being funded by a new NASA grant that I have at Tech-X Corporation where I work. The grant officially begins as of Jan 18th, so not much has been done as of this point. We have however been having some design discussions with various people. Here is a broad overview of the proposed work: 1) Distributed arrays (dense) based on numpy+pyrex+mpi 2) Parallel algorithms that work with those arrays (ffts, linear algebra, ufuncs, stats, etc) The code will be BSD and will end up mostly in the ipython1 project, but parts of it might end up also in numpy and mpi4py as well. I am more than willing to share more details about the work if you are interested. But, I will surely post to the numpy list as things move forward. Brian On Jan 7, 2008 1:13 PM, dmitrey wrote: > Some days ago there was mentioned a parallel numpy that is developed by > Brian Granger. > > Does the project have any blog or website? Has it any description about > API and abilities? When 1st release is intended? > > Regards, D > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From darren.dale at cornell.edu Mon Jan 7 16:13:04 2008 From: darren.dale at cornell.edu (Darren Dale) Date: Mon, 7 Jan 2008 16:13:04 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478286FD.7080006@enthought.com> References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: <200801071613.04753.darren.dale@cornell.edu> On Monday 07 January 2008 03:09:33 pm Travis E. Oliphant wrote: > Darren Dale wrote: > > One of my collaborators would like to use 16bit float arrays. According > > to http://www.scipy.org/Tentative_NumPy_Tutorial, and references to > > float16 in numpy.core.numerictypes, it appears that this should be > > possible, but the following doesnt work: > > No, it's only possible, if the C-implementation NumPy was compiled with > has 16-bits for its float scalar. > > Only float, double, and long double are implemented. These translate to > various bit-widths on different platforms. numerictypes is overly > aggressive at guessing what they might translate to. Thanks for the clarification, Travis. Darren From oliphant at enthought.com Mon Jan 7 16:18:51 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Jan 2008 15:18:51 -0600 Subject: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars? In-Reply-To: <47827CAB.7080607@astraw.com> References: <47827CAB.7080607@astraw.com> Message-ID: <4782973B.9050902@enthought.com> Andrew Straw wrote: > Hi, > > I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is > having troubles accessing a numpy scalar with the __array_interface__. > Is this supposed to work? Or should __array_interface__ trigger an > AttributeError on a numpy scalar? Note that I haven't done any digging > on this myself... > The other approach would be to get the data attribute directly (which returns a pointer to a read-only buffer object containing the data). I just fixed the reference problem with the scalar array_interface, though. -Travis From dmitrey.kroshko at scipy.org Mon Jan 7 16:26:18 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 07 Jan 2008 23:26:18 +0200 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> Message-ID: <478298FA.1070701@scipy.org> The only one thing I'm very interested in for now - why the most simplest matrix operations are not implemented to be parallel in numpy yet (for several-CPU computers, like my AMD Athlon X2). First of all it's related to matrix multiplication and devision, either point or matrix (i.e. like A\B, A*B, dot(A,B)). Another one highly convenient and rather simple thing to be implemented is direct Decart multiplication like it is mentioned in pyslice (IIRC I had some troubles with installing the one) http://scipy.org/Topical_Software#head-cf472934357fda4558aafdf558a977c4d59baecb I guess for ~95% of users it will be enough, and only 5% will require message-pass between subprocesses etc. BTW, IIRC latest MATLAB can uses 2-processors CPU already, and next version is promised to handle 4-processors as well. Regards, D. Brian Granger wrote: > Dmitrey, > > This work is being funded by a new NASA grant that I have at Tech-X > Corporation where I work. The grant officially begins as of Jan 18th, > so not much has been done as of this point. We have however been > having some design discussions with various people. > > Here is a broad overview of the proposed work: > > 1) Distributed arrays (dense) based on numpy+pyrex+mpi > > 2) Parallel algorithms that work with those arrays (ffts, linear > algebra, ufuncs, stats, etc) > > The code will be BSD and will end up mostly in the ipython1 project, > but parts of it might end up also in numpy and mpi4py as well. > > I am more than willing to share more details about the work if you are > interested. But, I will surely post to the numpy list as things move > forward. > > Brian > > On Jan 7, 2008 1:13 PM, dmitrey wrote: > >> Some days ago there was mentioned a parallel numpy that is developed by >> Brian Granger. >> >> Does the project have any blog or website? Has it any description about >> API and abilities? When 1st release is intended? >> >> Regards, D >> >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > From robert.kern at gmail.com Mon Jan 7 16:32:43 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 07 Jan 2008 15:32:43 -0600 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478298FA.1070701@scipy.org> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> Message-ID: <47829A7B.7010505@gmail.com> dmitrey wrote: > The only one thing I'm very interested in for now - why the most > simplest matrix operations are not implemented to be parallel in numpy > yet (for several-CPU computers, like my AMD Athlon X2). First of all > it's related to matrix multiplication and devision, either point or > matrix (i.e. like A\B, A*B, dot(A,B)). Eric Jones has made an attempt. http://svn.scipy.org/svn/numpy/branches/multicore/ Unfortunately, the overhead of starting the threads and acquiring/releasing thread locks wipes out most of the performance gains until you get fairly large arrays. It is possible that this comes from the particular implementation, rather than being intrinsic to the problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Mon Jan 7 16:33:25 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 7 Jan 2008 22:33:25 +0100 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478298FA.1070701@scipy.org> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> Message-ID: 2008/1/7, dmitrey : > > The only one thing I'm very interested in for now - why the most > simplest matrix operations are not implemented to be parallel in numpy > yet (for several-CPU computers, like my AMD Athlon X2). First of all > it's related to matrix multiplication and devision, either point or > matrix (i.e. like A\B, A*B, dot(A,B)). > Another one highly convenient and rather simple thing to be implemented > is direct Decart multiplication like it is mentioned in > pyslice (IIRC I had some troubles with installing the one) > > http://scipy.org/Topical_Software#head-cf472934357fda4558aafdf558a977c4d59baecb > I guess for ~95% of users it will be enough, and only 5% will require > message-pass between subprocesses etc. > BTW, IIRC latest MATLAB can uses 2-processors CPU already, and next > version is promised to handle 4-processors as well. > Regards, D. > Matlab surely relies on MKL to do this (Matlab ships with MKL or ACML now). The latest Intel library handles multiprocessing, so if you want to use multithreading, use MKL (and it can handle quad-cores with no sweat). So Numpy is multithreaded. On a side note, one of my friends published an article on a markovian predictor for prefetching data objects over a network to speed up computations. It was implemented on Java (search google for Jackal and Java), but it could help in the long term if it is managable. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Mon Jan 7 18:34:05 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 7 Jan 2008 16:34:05 -0700 Subject: [Numpy-discussion] CASTABLE flag In-Reply-To: References: <478276C8.2090806@enthought.com> <200801071451.21339.sransom@nrao.edu> Message-ID: On Jan 7, 2008 2:00 PM, Charles R Harris wrote: > Hi, > > On Jan 7, 2008 1:16 PM, Timothy Hochberg wrote: > > > > > Another possible approach is to treat downcasting similar to underflow. > > That is give it it's own flag in the errstate and people can set it to > > ignore, warn or raise on downcasting as desired. One could potentially have > > two flags, one for downcasting across kinds (float->int, int->bool) and one > > for downcasting within kinds (float64->float32). In this case, I personally > > would set the first to raise and the second to ignore and would suggest that > > as the default. > > > > IMO: > > > > 1. It's a no brainer to raise and exception when assigning a > > complex value to a float or integer target. Using "Z.real" or " > > Z.imag" is clearer and has the same performance. > > 2. I'm fairly dubious about assigning float to ints as is. First > > off it looks like a bug magnet to me due to accidentally assigning a > > floating point value to a target that one believes to be float but is in > > fact integer. Second, C-style rounding is pretty evil; it's not always > > consistent across platforms, so relying on it for anything other than > > truncating already integral values is asking for trouble. > > 3. Downcasting within kinds seems much less hazardous than > > downcasting across kinds, although I'd still be happy to be able regulate it > > with errstate. > > > > Maybe a combination of a typecast function and the errstate would work > well. The typecast function would provide a clear local override of the > default errstate flags, while the user would have the option to specify what > sort of behavior they care about in general. > Note that using the 'with' statement, you can have reasonably lightweight local control using errstate. For example, assuming the hypothetical dowcasting flag was named 'downcast': with errstate(downcast='ignore'): anint[:] = afloat That would be local enough for me although it may not be to everyone's taste. If we were to go the "a[:] = typecast(b)" route, I have a hankering for some more fine grained control of the rounding. Something like "a[:] = lazy_floor(b)", "a[:] = lazy_truncate(b)" or "a[:] = lazy_ceil(b)", only with better names. However, I admit that it's not obvious how to implement that. Yet another approach, is to add an 'out' argument to asstype such as already exists for many of the other methods. Then "a.astype(int, out=b)", will efficiently stick an integerized version of a into b. FWIW, I suspect that the usefulness of casting in avoiding of temporaries is probably overstated. If you really want to avoid temporaries, you either need to program in a very tedious fashion, suitable only for very localized hot spots, or you need to use something like numexpr. If someone comes back with some measurements from real code that show a big time hit, I'll concede the case, but so far it all sounds like guessing and my guess is that it'll rarely make a difference. (And, the cases where it does make a difference will be places your already doing crazy things to optimize the snot out of the code and an extra astype or such won't matter) -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at enthought.com Mon Jan 7 19:02:25 2008 From: eric at enthought.com (eric jones) Date: Mon, 07 Jan 2008 18:02:25 -0600 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <47829A7B.7010505@gmail.com> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47829A7B.7010505@gmail.com> Message-ID: <4782BD91.5060308@enthought.com> Robert Kern wrote: > dmitrey wrote: > >> The only one thing I'm very interested in for now - why the most >> simplest matrix operations are not implemented to be parallel in numpy >> yet (for several-CPU computers, like my AMD Athlon X2). First of all >> it's related to matrix multiplication and devision, either point or >> matrix (i.e. like A\B, A*B, dot(A,B)). >> > > Eric Jones has made an attempt. > > http://svn.scipy.org/svn/numpy/branches/multicore/ > > Unfortunately, the overhead of starting the threads and acquiring/releasing > thread locks wipes out most of the performance gains until you get fairly large > arrays. It is possible that this comes from the particular implementation, > rather than being intrinsic to the problem. > > Yes, the problem in this implementation is that it uses pthreads for synchronization instead of spin locks with a work pool implementation tailored to numpy. The thread synchronization overhead is horrible (300,000-400,000 clock cycles) and swamps anything other than very large arrays. I have played with spin-lock based solutions that cut this to, on average 3000-4000 cylces. With that, arrays of 1e3-1e4 elements can start seeing parallelization benefits. However, this code hasn't passed the mad-scientist tinkering stage... I haven't touched it in at least 6 months, and I doubt I'll get back to it very soon (go Brian!). It did look promising for up to 4 processors on some operations (sin, cos, etc.) and worth-it-but-less-beneficial on simple operations (+,-,*, etc.). Coupled with something like weave.blitz or numexpr that can (or could) compile multiple binary operations into a single kernel, the scaling for expressions with multiple simple operations would scale very well. My tinkering was aimed at a framework that would allow you to write little computational kernels in a prescribed way, and then let a numpy load-balancer automatically split the work up between worker threads that execute these little kernels. Ideally, this load-balancer would be pluggable. The inner loop for numpy's universal functions is probably very close or exactly the interface for these little kernels. Also, it be nice to couple this with weave so that kernels written with weave could execute in parallel without user effort. (This is all like the "map" part of map-reduce architecture... The reduce part also need to fit in the architecture to generically handle things like sum, etc.) Getting it to work with all flavors of numpy arrays (contiguous, non-contiguous, buffered, etc.) is quite a chore, but the contiguous arrays (and perhaps some non-contiguous) offer some relatively low hanging fruit. Here's to hoping Brian's project bears fruit. I haven't thought about matrix ops much, so I don't know if they would fit this (minimally) described architecture. I am sure that they would scale well. eric From strawman at astraw.com Mon Jan 7 23:49:44 2008 From: strawman at astraw.com (Andrew Straw) Date: Mon, 07 Jan 2008 20:49:44 -0800 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478298FA.1070701@scipy.org> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> Message-ID: <478300E8.9030008@astraw.com> dmitrey wrote: > The only one thing I'm very interested in for now - why the most > simplest matrix operations are not implemented to be parallel in numpy > yet (for several-CPU computers, like my AMD Athlon X2). For what it's worth, sometimes I *want* my numpy operations to happen only on one core. (So that I can do more important stuff with the others.) From david at ar.media.kyoto-u.ac.jp Mon Jan 7 23:58:11 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 13:58:11 +0900 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478300E8.9030008@astraw.com> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <478300E8.9030008@astraw.com> Message-ID: <478302E3.9020602@ar.media.kyoto-u.ac.jp> Andrew Straw wrote: > dmitrey wrote: > >> The only one thing I'm very interested in for now - why the most >> simplest matrix operations are not implemented to be parallel in numpy >> yet (for several-CPU computers, like my AMD Athlon X2). >> > For what it's worth, sometimes I *want* my numpy operations to happen > only on one core. (So that I can do more important stuff with the others.) > I have not investigated really deeply, but matlab, at least the version I tried, had the choice between multi-core and mono-core (at runtime). Personally, I have not been really impressed by the speed enhancement: nothing scientific, but I tried it on several of my old matlab scripts (I rarely use matlab since I am using numpy), and for some things it was a bit faster, for some a bit slower. But the multi-core support was said to be experimental for that version (it was not enabled by default for sure). cheers, David From david at ar.media.kyoto-u.ac.jp Tue Jan 8 00:41:58 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 14:41:58 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? Message-ID: <47830D26.9020700@ar.media.kyoto-u.ac.jp> Hi, for my work related on scons, I have a branch build_with_scons in the numpy trunk, which I have initialized exactly as documented on the numpy wiki (http://projects.scipy.org/scipy/numpy/wiki/MakingBranches). When I try to update my branch with the trunk, I got suprising merge request, in perticular, it tried to merge all trunk revision up to 2871, whereas I created my branch with a copy from the trunk at revision 4676. Am I missing something ? Shouldn't it try to merge from the revision I started the branch (since this revision is a common ancestor) ? cheers, David From fperez.net at gmail.com Tue Jan 8 01:01:12 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Jan 2008 23:01:12 -0700 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47830D26.9020700@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 7, 2008 10:41 PM, David Cournapeau wrote: > Hi, > > for my work related on scons, I have a branch build_with_scons in > the numpy trunk, which I have initialized exactly as documented on the > numpy wiki (http://projects.scipy.org/scipy/numpy/wiki/MakingBranches). > When I try to update my branch with the trunk, I got suprising merge > request, in perticular, it tried to merge all trunk revision up to 2871, > whereas I created my branch with a copy from the trunk at revision 4676. > Am I missing something ? Shouldn't it try to merge from the revision I > started the branch (since this revision is a common ancestor) ? AFAIK, the merge command must ALWAYS be given with explicit revision brackets, since this is precisely the information SVN does not track at all. Quoting: http://svnbook.red-bean.com/en/1.0/ch04s04.html """ But as discussed in the section called "Best Practices for Merging", you don't want to merge the changes you've already merged before; you only want to merge everything "new" on your branch since the last time you merged. The trick is to figure out what's new. The first step is to run svn log on the trunk, and look for a log message about the last time you merged from the branch: $ cd calc/trunk $ svn log ? ------------------------------------------------------------------------ r406 | user | 2004-02-08 11:17:26 -0600 (Sun, 08 Feb 2004) | 1 line Merged my-calc-branch changes r341:405 into the trunk. ------------------------------------------------------------------------ ? Aha! Since all branch-changes that happened between revisions 341 and 405 were previously merged to the trunk as revision 406, you now know that you want to merge only the branch changes after that?by comparing revisions 406 and HEAD. """ As you can see, they recommend that you indicate in that particular log message the revision brackets of what you've already merged, so you can find that information easily later. Once you have this information on record, you start doing all your future updates on the branch with specific revision ranges, and that seems to work reasonably well. HTH, f From david at ar.media.kyoto-u.ac.jp Tue Jan 8 00:54:47 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 14:54:47 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> Message-ID: <47831027.3040205@ar.media.kyoto-u.ac.jp> Fernando Perez wrote: > On Jan 7, 2008 10:41 PM, David Cournapeau wrote: > >> Hi, >> >> for my work related on scons, I have a branch build_with_scons in >> the numpy trunk, which I have initialized exactly as documented on the >> numpy wiki (http://projects.scipy.org/scipy/numpy/wiki/MakingBranches). >> When I try to update my branch with the trunk, I got suprising merge >> request, in perticular, it tried to merge all trunk revision up to 2871, >> whereas I created my branch with a copy from the trunk at revision 4676. >> Am I missing something ? Shouldn't it try to merge from the revision I >> started the branch (since this revision is a common ancestor) ? >> > > AFAIK, the merge command must ALWAYS be given with explicit revision > brackets, since this is precisely the information SVN does not track > at all. Quoting: > > http://svnbook.red-bean.com/en/1.0/ch04s04.html > """ > But as discussed in the section called "Best Practices for Merging", > you don't want to merge the changes you've already merged before; you > only want to merge everything "new" on your branch since the last time > you merged. The trick is to figure out what's new. > > The first step is to run svn log on the trunk, and look for a log > message about the last time you merged from the branch: > > $ cd calc/trunk > $ svn log > ? > ------------------------------------------------------------------------ > r406 | user | 2004-02-08 11:17:26 -0600 (Sun, 08 Feb 2004) | 1 line > > Merged my-calc-branch changes r341:405 into the trunk. > ------------------------------------------------------------------------ > ? > > Aha! Since all branch-changes that happened between revisions 341 and > 405 were previously merged to the trunk as revision 406, you now know > that you want to merge only the branch changes after that?by comparing > revisions 406 and HEAD. > """ > > As you can see, they recommend that you indicate in that particular > log message the revision brackets of what you've already merged, so > you can find that information easily later. Once you have this > information on record, you start doing all your future updates on the > branch with specific revision ranges, and that seems to work > reasonably well. > > I understand this if doing the merge at hand with svn merge (that's what I did previously), but I am using svnmerge, which is supposed to avoid all this (I find the whole process extremely error-prone). More specifically, I am surprised by the svnmerge starting revisions, cheers, David From fperez.net at gmail.com Tue Jan 8 01:11:04 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Jan 2008 23:11:04 -0700 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47831027.3040205@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 7, 2008 10:54 PM, David Cournapeau wrote: > I understand this if doing the merge at hand with svn merge (that's what > I did previously), but I am using svnmerge, which is supposed to avoid > all this (I find the whole process extremely error-prone). More > specifically, I am surprised by the svnmerge starting revisions, OK, I've always just used manual revision tracking, since I basically don't trust svnmerge. Seems I was right, since the manual process at least works, even if it's clunky and requires lots of care. Maybe you should be using a DVCS. Oh wait... ;-) Cheers, f From david at ar.media.kyoto-u.ac.jp Tue Jan 8 01:07:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 15:07:21 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> Message-ID: <47831319.9070907@ar.media.kyoto-u.ac.jp> Fernando Perez wrote: > On Jan 7, 2008 10:54 PM, David Cournapeau wrote: > > >> I understand this if doing the merge at hand with svn merge (that's what >> I did previously), but I am using svnmerge, which is supposed to avoid >> all this (I find the whole process extremely error-prone). More >> specifically, I am surprised by the svnmerge starting revisions, >> > > OK, I've always just used manual revision tracking, since I basically > don't trust svnmerge. Seems I was right, since the manual process at > least works, even if it's clunky and requires lots of care. > Between a working clunky solution and a clunky non working solution, the choice is not difficult :) I just wanted to check whether I did something wrong or if svnmerge was fragile (I was a bit surprised, because I could swear I managed to make it work at some point in another branch....) > Maybe you should be using a DVCS. Oh wait... ;-) > Don't worry, I have never used subversion for my own projects, I've never understood the complexity of the thing :) cheers, David From matthieu.brucher at gmail.com Tue Jan 8 01:36:22 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 07:36:22 +0100 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47831027.3040205@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> Message-ID: Hi David, How did you initialize svnmerge ? Matthieu 2008/1/8, David Cournapeau : > > Fernando Perez wrote: > > On Jan 7, 2008 10:41 PM, David Cournapeau > wrote: > > > >> Hi, > >> > >> for my work related on scons, I have a branch build_with_scons in > >> the numpy trunk, which I have initialized exactly as documented on the > >> numpy wiki (http://projects.scipy.org/scipy/numpy/wiki/MakingBranches). > >> When I try to update my branch with the trunk, I got suprising merge > >> request, in perticular, it tried to merge all trunk revision up to > 2871, > >> whereas I created my branch with a copy from the trunk at revision > 4676. > >> Am I missing something ? Shouldn't it try to merge from the revision I > >> started the branch (since this revision is a common ancestor) ? > >> > > > > AFAIK, the merge command must ALWAYS be given with explicit revision > > brackets, since this is precisely the information SVN does not track > > at all. Quoting: > > > > http://svnbook.red-bean.com/en/1.0/ch04s04.html > > """ > > But as discussed in the section called "Best Practices for Merging", > > you don't want to merge the changes you've already merged before; you > > only want to merge everything "new" on your branch since the last time > > you merged. The trick is to figure out what's new. > > > > The first step is to run svn log on the trunk, and look for a log > > message about the last time you merged from the branch: > > > > $ cd calc/trunk > > $ svn log > > ? > > ------------------------------------------------------------------------ > > r406 | user | 2004-02-08 11:17:26 -0600 (Sun, 08 Feb 2004) | 1 line > > > > Merged my-calc-branch changes r341:405 into the trunk. > > ------------------------------------------------------------------------ > > ? > > > > Aha! Since all branch-changes that happened between revisions 341 and > > 405 were previously merged to the trunk as revision 406, you now know > > that you want to merge only the branch changes after that?by comparing > > revisions 406 and HEAD. > > """ > > > > As you can see, they recommend that you indicate in that particular > > log message the revision brackets of what you've already merged, so > > you can find that information easily later. Once you have this > > information on record, you start doing all your future updates on the > > branch with specific revision ranges, and that seems to work > > reasonably well. > > > > > I understand this if doing the merge at hand with svn merge (that's what > I did previously), but I am using svnmerge, which is supposed to avoid > all this (I find the whole process extremely error-prone). More > specifically, I am surprised by the svnmerge starting revisions, > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 01:40:01 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 15:40:01 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> Message-ID: <47831AC1.6050808@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > Hi David, > > How did you initialize svnmerge ? As said in the numpy wiki. More precisely: - In a svn checkout of the trunk, do svn up to be up to date - svn copy TRUNK MY_BRANCH - use svnmerge init MY_BRANCH - svn ci -F svnmerge-commit.txt - svn switch MY_BRANCH - svnmerge init TRUNK - svn ci -F svnmerge-commit.txt One thing which is strange is that in the numpy trunk, you can see the following as svnmerge-integrated: Property svnmerge-integrated set to /branches/build_with_scons:1-4676 /branches/cleanconfig_rtm:1-4610 /branches/distutils-revamp:1-2752 /branches/distutils_scons_command:1-4619 /branches/multicore:1-3687 /branches/numpy.scons:1-4484 /trunk:1-2871 Does that mean that you cannot use svnmerge with several branches at the same time ? cheers, David From matthieu.brucher at gmail.com Tue Jan 8 01:56:07 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 07:56:07 +0100 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47831AC1.6050808@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> Message-ID: 2008/1/8, David Cournapeau : > > Matthieu Brucher wrote: > > Hi David, > > > > How did you initialize svnmerge ? > As said in the numpy wiki. More precisely: > - In a svn checkout of the trunk, do svn up to be up to date > - svn copy TRUNK MY_BRANCH > - use svnmerge init MY_BRANCH > - svn ci -F svnmerge-commit.txt > - svn switch MY_BRANCH > - svnmerge init TRUNK > - svn ci -F svnmerge-commit.txt > > One thing which is strange is that in the numpy trunk, you can see the > following as svnmerge-integrated: > > Property svnmerge-integrated set to /branches/build_with_scons:1-4676 > /branches/cleanconfig_rtm:1-4610 /branches/distutils-revamp:1-2752 > /branches/distutils_scons_command:1-4619 /branches/multicore:1-3687 > /branches/numpy.scons:1-4484 /trunk:1-2871 > > Does that mean that you cannot use svnmerge with several branches at the > same time ? No, this is what I expected. Could you send us the content of svnmerge-integrated for the build_with_scons branch before the merge ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Jan 8 01:56:54 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 07:56:54 +0100 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> Message-ID: 2008/1/8, Matthieu Brucher : > > > > 2008/1/8, David Cournapeau : > > > > Matthieu Brucher wrote: > > > Hi David, > > > > > > How did you initialize svnmerge ? > > As said in the numpy wiki. More precisely: > > - In a svn checkout of the trunk, do svn up to be up to date > > - svn copy TRUNK MY_BRANCH > > - use svnmerge init MY_BRANCH > > - svn ci -F svnmerge-commit.txt > > - svn switch MY_BRANCH > > - svnmerge init TRUNK > > - svn ci -F svnmerge-commit.txt > > > > One thing which is strange is that in the numpy trunk, you can see the > > following as svnmerge-integrated: > > > > Property svnmerge-integrated set to /branches/build_with_scons:1-4676 > > /branches/cleanconfig_rtm:1-4610 /branches/distutils-revamp:1-2752 > > /branches/distutils_scons_command:1-4619 /branches/multicore:1-3687 > > /branches/numpy.scons:1-4484 /trunk:1-2871 > > > > Does that mean that you cannot use svnmerge with several branches at the > > same time ? > > > No, this is what I expected. Could you send us the content of > svnmerge-integrated for the build_with_scons branch before the merge ? > > Matthieu Oups, safe for the "/trunk:1-2871" part. This should be deleted before a commit to the trunk, I think. -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 01:56:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 15:56:05 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> Message-ID: <47831E85.3050001@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > > 2008/1/8, Matthieu Brucher >: > > > > 2008/1/8, David Cournapeau >: > > Matthieu Brucher wrote: > > Hi David, > > > > How did you initialize svnmerge ? > As said in the numpy wiki. More precisely: > - In a svn checkout of the trunk, do svn up to be up to date > - svn copy TRUNK MY_BRANCH > - use svnmerge init MY_BRANCH > - svn ci -F svnmerge-commit.txt > - svn switch MY_BRANCH > - svnmerge init TRUNK > - svn ci -F svnmerge-commit.txt > > One thing which is strange is that in the numpy trunk, you can > see the > following as svnmerge-integrated: > > Property svnmerge-integrated set to > /branches/build_with_scons:1-4676 > /branches/cleanconfig_rtm:1-4610 /branches/distutils-revamp:1-2752 > /branches/distutils_scons_command:1-4619 > /branches/multicore:1-3687 > /branches/numpy.scons:1-4484 /trunk:1-2871 > > Does that mean that you cannot use svnmerge with several > branches at the > same time ? > > > No, this is what I expected. Could you send us the content of > svnmerge-integrated for the build_with_scons branch before the > merge ? > > Matthieu > > > Oups, safe for the "/trunk:1-2871" part. This should be deleted before > a commit to the trunk, I think. Yes, that's what I (quite unclearly) meant: since revision numbers are per- repository in svn, I don't understand the point of tracking trunk revisions: I would think that tracking the last merged version for each branch to be enough (for the kind of merge svn does, at least). If trunk version are tracked, then I would expect two branches using svnmerge to clash each other, cheers, David From matthieu.brucher at gmail.com Tue Jan 8 02:15:35 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 08:15:35 +0100 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47831E85.3050001@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> <47831E85.3050001@ar.media.kyoto-u.ac.jp> Message-ID: > > > Oups, safe for the "/trunk:1-2871" part. This should be deleted before > > a commit to the trunk, I think. > Yes, that's what I (quite unclearly) meant: since revision numbers are > per- repository in svn, I don't understand the point of tracking trunk > revisions: I would think that tracking the last merged version for each > branch to be enough (for the kind of merge svn does, at least). If trunk > version are tracked, then I would expect two branches using svnmerge to > clash each other, > In fact, the trunk should be tracked from all the branches, although there will be the problem with merging the different branches (I did not have many troubles with that, but I only tried with a few differences) into the trunk. I don't think only one branch wants to be up to date with the trunk ;). But each time you create a branch, you must delete the svnmerge_integrated property and set it correctly. Or you can merge by hand the trunk into each branch, but I don't know if it's hard or not. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Tue Jan 8 03:45:41 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Tue, 08 Jan 2008 10:45:41 +0200 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> Message-ID: <47833835.7060605@scipy.org> Matthieu Brucher wrote: > Matlab surely relies on MKL to do this (Matlab ships with MKL or ACML > now). The latest Intel library handles multiprocessing, so if you want > to use multithreading, use MKL (and it can handle quad-cores with no > sweat). So Numpy is multithreaded. I have AMD processor so I guess I should use ACML somehow instead. However, at 1st I would prefer my code to be platform-independent, and at 2nd unfortunately I haven't encountered in numpy documentation (in website scipy.org and numpy.scipy.org) any mention about how to use numpy multithreading at all (neither MKL nor ACML). Also, I intended to try using numpy multithreading on our icyb cluster (IIRC made of intel processors) from world top 500 (however, currently connect to other subsets of processors from other cities have been organized, some of them are AMD). Would 100-200 processors (I don't remember how many have the one) yield at least 2x...3x speedup on some of my test cases, it would be a good deal and something to report in my graduation work. As my chief informed me, people here are fond of the cluster, mentioning the magical word (in my work) would fill them with respect :) Regards, D. From ondrej at certik.cz Tue Jan 8 04:04:43 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 8 Jan 2008 10:04:43 +0100 Subject: [Numpy-discussion] how to work with mercurial and numpy right now Message-ID: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> Hi, if you want to play with Mercurial now (without forcing everyone else to leave svn), I suggest this: http://cheeseshop.python.org/pypi/hgsvn I tried that and it works. It's a very easy way to create a hg mirror at your computer. And then you can take this as the official upstream repository (which you don't have write access to). Whenever somone commits to the svn, you just do hgpullsvn and it updates your mercurial repo. Then you just clone it and create branches, for example the scons branch can be easily managed like this. Then you prepare patches, against your "official local mercurial mirror", using for example "hg export", or something, those patches should be possible to apply against the svn repository as well. You sent them for review and then (you or someone else) commit them using svn, then you'll "hgpullsvn" your local mercurial mirror and merge the changes to all your other branches. Ondrej From david at ar.media.kyoto-u.ac.jp Tue Jan 8 04:36:01 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 18:36:01 +0900 Subject: [Numpy-discussion] how to work with mercurial and numpy right now In-Reply-To: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> References: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> Message-ID: <47834401.60303@ar.media.kyoto-u.ac.jp> Ondrej Certik wrote: > Hi, > > if you want to play with Mercurial now (without forcing everyone else > to leave svn), I suggest this: > > http://cheeseshop.python.org/pypi/hgsvn > > I tried that and it works. It's a very easy way to create a hg mirror > at your computer. And then you can take this > as the official upstream repository (which you don't have write access > to). Whenever somone commits > to the svn, you just do hgpullsvn and it updates your mercurial repo. > > Then you just clone it and create branches, for example the scons > branch can be easily managed like this. > Then you prepare patches, against your "official local mercurial > mirror", using for example > "hg export", or something, those patches should be possible to apply > against the svn repository as well. > You sent them for review and then (you or someone else) commit them > using svn, then you'll "hgpullsvn" your local mercurial mirror and > merge the changes to all your other branches. > The main problem if this approach is that it is quite heavy on the svn server; that's why it would be better if the mirrors are done only once, and are publicly available, I think. Besides, it is easier (and faster) to do the mirrors locally (or from the file:// method, or from a svn dump; both mercurial and bzr have methods to import from those) cheers, David From david at ar.media.kyoto-u.ac.jp Tue Jan 8 04:42:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 18:42:00 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> <47831E85.3050001@ar.media.kyoto-u.ac.jp> Message-ID: <47834568.3010908@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > > Oups, safe for the "/trunk:1-2871" part. This should be deleted > before > > a commit to the trunk, I think. > Yes, that's what I (quite unclearly) meant: since revision numbers are > per- repository in svn, I don't understand the point of tracking trunk > revisions: I would think that tracking the last merged version for > each > branch to be enough (for the kind of merge svn does, at least). If > trunk > version are tracked, then I would expect two branches using > svnmerge to > clash each other, > > > In fact, the trunk should be tracked from all the branches, although > there will be the problem with merging the different branches (I did > not have many troubles with that, but I only tried with a few > differences) into the trunk. I don't think only one branch wants to be > up to date with the trunk ;). Sure, that's what I was trying to do (tracking the trunk). But if svnmerge needs the information from the trunk, this means each branch will need a different value, whereas only one value is possible... I think I give up, I will use svn merge manually, cheers, David From matthieu.brucher at gmail.com Tue Jan 8 05:16:09 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 11:16:09 +0100 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47834568.3010908@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> <47831E85.3050001@ar.media.kyoto-u.ac.jp> <47834568.3010908@ar.media.kyoto-u.ac.jp> Message-ID: > > > In fact, the trunk should be tracked from all the branches, although > > there will be the problem with merging the different branches (I did > > not have many troubles with that, but I only tried with a few > > differences) into the trunk. I don't think only one branch wants to be > > up to date with the trunk ;). > Sure, that's what I was trying to do (tracking the trunk). But if > svnmerge needs the information from the trunk, this means each branch > will need a different value, whereas only one value is possible... In fact everything is stored in this single property and it is enough : svnmerge extracts the revision not merged according to the branch you want to merge in the trunk (in this case), the other indications for the other branches are not touched. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Jan 8 05:33:23 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 11:33:23 +0100 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <47833835.7060605@scipy.org> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47833835.7060605@scipy.org> Message-ID: > > I have AMD processor so I guess I should use ACML somehow instead. > However, at 1st I would prefer my code to be platform-independent, and > at 2nd unfortunately I haven't encountered in numpy documentation (in > website scipy.org and numpy.scipy.org) any mention about how to use > numpy multithreading at all (neither MKL nor ACML). MKL does the multithreading on its own for level 3 BLAS instructions (OpenMP). For ACML, the problem is that AMD does not provide a CBLAS interface and is not interested in doing so. With ACML, the compilation fails with the current Numpy, but hopefully with Scons it will work, at least for the LAPACK part. But I don't think that ACML is parallel. I think that using multithreaded libraries is far more interesting and easy to do than using distributed memory systems. This is due to the fact that Python can use some help to enable multi-processing (not GIL), for instance like Java and Jackal. After some readings, I think this means that the core Python should be updated. Also, I intended to try using numpy multithreading on our icyb cluster > (IIRC made of intel processors) from world top 500 (however, currently > connect to other subsets of processors from other cities have been > organized, some of them are AMD). Would 100-200 processors (I don't > remember how many have the one) yield at least 2x...3x speedup on some > of my test cases, it would be a good deal and something to report in my > graduation work. If you have access to Intel Quad-Core processors with the latest MKL and if you intensively use matrix multiplications, you will have those results. But if you speak at your graduation that using 100 or 200 processors and say that it only yields a 2 or 3 time speedup factor, I think the jury will not appreciate. As my chief informed me, people here are fond of the cluster, mentioning > the magical word (in my work) would fill them with respect :) Then you should first start by looking how to make your algorithms parallel. Just throwing a number of processors will not yield a good speedup per processor, and this is what people are looking for : good scalability. Then you must use tools like the processing module, MPI, ... Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From konrad.hinsen at laposte.net Tue Jan 8 06:17:46 2008 From: konrad.hinsen at laposte.net (Konrad Hinsen) Date: Tue, 8 Jan 2008 12:17:46 +0100 Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector In-Reply-To: <47827621.3090906@enthought.com> References: <4771F0AE.40406@ncsu.edu> <7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com> <47732B84.70400@ncsu.edu> <4773325D.3040907@gmail.com> <4773BD97.1000703@ncsu.edu> <4773C17B.4050802@ncsu.edu> <4775DF5D.3030400@ncsu.edu> <7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com> <4776B111.9020607@ncsu.edu> <4776C3FC.7030101@ncsu.edu> <4776D1A5.60106@ncsu.edu> <47827621.3090906@enthought.com> Message-ID: On Jan 7, 2008, at 19:57, Travis E. Oliphant wrote: > Alternatively, the generic scalar operations should probably not be so > "inclusive" and should allow the other object a chance to perform the > operation more often (by returning NotImplemented). That would be great. In fact, this has been (and still is) the #1 problem for me in moving from Numeric to NumPy: as soon as I have NumPy scalar objects circulating in my programs, everything that behaves like a sequence gets converted to an array in math operations. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From cookedm at physics.mcmaster.ca Tue Jan 8 07:18:05 2008 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 8 Jan 2008 07:18:05 -0500 Subject: [Numpy-discussion] how to work with mercurial and numpy right now In-Reply-To: <47834401.60303@ar.media.kyoto-u.ac.jp> References: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> <47834401.60303@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008, at 04:36 , David Cournapeau wrote: > Ondrej Certik wrote: >> Hi, >> >> if you want to play with Mercurial now (without forcing everyone else >> to leave svn), I suggest this: >> >> http://cheeseshop.python.org/pypi/hgsvn >> >> I tried that and it works. It's a very easy way to create a hg mirror >> at your computer. And then you can take this >> as the official upstream repository (which you don't have write >> access >> to). Whenever somone commits >> to the svn, you just do hgpullsvn and it updates your mercurial repo. >> >> Then you just clone it and create branches, for example the scons >> branch can be easily managed like this. >> Then you prepare patches, against your "official local mercurial >> mirror", using for example >> "hg export", or something, those patches should be possible to apply >> against the svn repository as well. >> You sent them for review and then (you or someone else) commit them >> using svn, then you'll "hgpullsvn" your local mercurial mirror and >> merge the changes to all your other branches. >> > The main problem if this approach is that it is quite heavy on the svn > server; that's why it would be better if the mirrors are done only > once, > and are publicly available, I think. Besides, it is easier (and > faster) > to do the mirrors locally (or from the file:// method, or from a svn > dump; both mercurial and bzr have methods to import from those) At least for mercurial's convert command, it's a one-time thing -- you can't update a created repo from svn. AFAIK, all the tools can specify a svn revision to start from, if you don't need history (or just recent history). -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From david at ar.media.kyoto-u.ac.jp Tue Jan 8 07:16:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 08 Jan 2008 21:16:29 +0900 Subject: [Numpy-discussion] how to work with mercurial and numpy right now In-Reply-To: References: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> <47834401.60303@ar.media.kyoto-u.ac.jp> Message-ID: <4783699D.9060802@ar.media.kyoto-u.ac.jp> David M. Cooke wrote: > On Jan 8, 2008, at 04:36 , David Cournapeau wrote: > > >> Ondrej Certik wrote: >> >>> Hi, >>> >>> if you want to play with Mercurial now (without forcing everyone else >>> to leave svn), I suggest this: >>> >>> http://cheeseshop.python.org/pypi/hgsvn >>> >>> I tried that and it works. It's a very easy way to create a hg mirror >>> at your computer. And then you can take this >>> as the official upstream repository (which you don't have write >>> access >>> to). Whenever somone commits >>> to the svn, you just do hgpullsvn and it updates your mercurial repo. >>> >>> Then you just clone it and create branches, for example the scons >>> branch can be easily managed like this. >>> Then you prepare patches, against your "official local mercurial >>> mirror", using for example >>> "hg export", or something, those patches should be possible to apply >>> against the svn repository as well. >>> You sent them for review and then (you or someone else) commit them >>> using svn, then you'll "hgpullsvn" your local mercurial mirror and >>> merge the changes to all your other branches. >>> >>> >> The main problem if this approach is that it is quite heavy on the svn >> server; that's why it would be better if the mirrors are done only >> once, >> and are publicly available, I think. Besides, it is easier (and >> faster) >> to do the mirrors locally (or from the file:// method, or from a svn >> dump; both mercurial and bzr have methods to import from those) >> > > > At least for mercurial's convert command, it's a one-time thing It's a one-time thing per person. If many people do it, I am just afraid it will overload the servers. Since several people seemed interested in mercurial, it may make sense to have a public repository. > AFAIK, all the tools can specify a svn revision to start from, if you > don't need history (or just recent history). > > Are you sure ? bzr-svn does not do it (logically, since bzr-svn can pull/push), and I don't see any option from the convert extension from mercurial. I don't have hgpullsvn at hand, I don't remember having seen the option either. David From cookedm at physics.mcmaster.ca Tue Jan 8 09:44:21 2008 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 8 Jan 2008 09:44:21 -0500 Subject: [Numpy-discussion] how to work with mercurial and numpy right now In-Reply-To: <4783699D.9060802@ar.media.kyoto-u.ac.jp> References: <85b5c3130801080104i2b669430yb8247234aba6e428@mail.gmail.com> <47834401.60303@ar.media.kyoto-u.ac.jp> <4783699D.9060802@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008, at 07:16 , David Cournapeau wrote: > David M. Cooke wrote: >> AFAIK, all the tools can specify a svn revision to start from, if you >> don't need history (or just recent history). >> > Are you sure ? bzr-svn does not do it (logically, since bzr-svn can > pull/push), and I don't see any option from the convert extension from > mercurial. I don't have hgpullsvn at hand, I don't remember having > seen > the option either. Thought they did; hgimportsvn from hgsvn does, and so does tailor. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From bioinformed at gmail.com Tue Jan 8 10:22:57 2008 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Tue, 8 Jan 2008 10:22:57 -0500 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47833835.7060605@scipy.org> Message-ID: <2e1434c10801080722y12515f9q546bac41915ee15@mail.gmail.com> On 1/8/08, Matthieu Brucher wrote: > > I have AMD processor so I guess I should use ACML somehow instead. > > However, at 1st I would prefer my code to be platform-independent, and > > at 2nd unfortunately I haven't encountered in numpy documentation (in > > website scipy.org and numpy.scipy.org) any mention about how to use > > numpy multithreading at all (neither MKL nor ACML). > > > > MKL does the multithreading on its own for level 3 BLAS instructions > (OpenMP). For ACML, the problem is that AMD does not provide a CBLAS > interface and is not interested in doing so. With ACML, the compilation > fails with the current Numpy, but hopefully with Scons it will work, at > least for the LAPACK part. > [..] That is news to me. I compile NumPy with ACML regularly. I haven't tried with the current trunk but it worked recently. I have not tried the parallel versions, though. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From subscriber100 at rjs.org Tue Jan 8 10:31:11 2008 From: subscriber100 at rjs.org (Ray Schumacher) Date: Tue, 08 Jan 2008 07:31:11 -0800 Subject: [Numpy-discussion] parallel numpy - any info? In-Reply-To: References: Message-ID: <6.2.3.4.2.20080108070109.03542dc0@rjs.org> At 04:27 AM 1/8/2008, you wrote: > 4. Re: parallel numpy (by Brian Granger) - any info? > (Matthieu Brucher) >From: "Matthieu Brucher" > >MKL does the multithreading on its own for level 3 BLAS instructions >(OpenMP). There was brief debate yesterday among the Pythonians in the lab as to whether the MKL operates outside of the GIL, but it was general understanding that it does. It is still unclear to me whether Python/numpy compiled with MKL would be freely re-distributable, as the MSVC version is. >Then you must use tools like the processing module, MPI, ... I have been using shared memory and numpy with good results lately on Win32 with tagged memory. http://projects.scipy.org/pipermail/numpy-discussion/2007-October/029505.html On *NIX, arrayfrombuffer(), mmap and MAP_SHARED, or POSH http://poshmodule.sourceforge.net/ seem like the ticket. Whatever happened to pyGA? GA http://www.emsl.pnl.gov/docs/global/ is still around. http://www.ece.lsu.edu/jxr/pohll-02/papers/jarek.pdf Best, Ray Schumacher -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.13/1211 - Release Date: 1/6/2008 11:57 AM From matthieu.brucher at gmail.com Tue Jan 8 10:30:34 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 16:30:34 +0100 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <2e1434c10801080722y12515f9q546bac41915ee15@mail.gmail.com> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47833835.7060605@scipy.org> <2e1434c10801080722y12515f9q546bac41915ee15@mail.gmail.com> Message-ID: > > MKL does the multithreading on its own for level 3 BLAS instructions > > (OpenMP). For ACML, the problem is that AMD does not provide a CBLAS > > interface and is not interested in doing so. With ACML, the compilation > > fails with the current Numpy, but hopefully with Scons it will work, at > > least for the LAPACK part. > > > > [..] > > That is news to me. I compile NumPy with ACML regularly. I haven't tried > with the current trunk but it worked recently. I have not tried the > parallel versions, though. > Could you share with us your setup.cfg and what you did to make Numpy work with ACML ? Last time I tried (one month ago), I had several unresolved externals (and I'm sure ACML does not provide the CBLAS interface which is needed by Numpy). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Jan 8 10:43:47 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 8 Jan 2008 16:43:47 +0100 Subject: [Numpy-discussion] parallel numpy - any info? In-Reply-To: <6.2.3.4.2.20080108070109.03542dc0@rjs.org> References: <6.2.3.4.2.20080108070109.03542dc0@rjs.org> Message-ID: 2008/1/8, Ray Schumacher : > > At 04:27 AM 1/8/2008, you wrote: > > 4. Re: parallel numpy (by Brian Granger) - any info? > > (Matthieu Brucher) > >From: "Matthieu Brucher" > > > >MKL does the multithreading on its own for level 3 BLAS instructions > >(OpenMP). > > There was brief debate yesterday among the Pythonians in the lab as > to whether the MKL operates outside of the GIL, but it was general > understanding that it does. As Numpy releases the GIL, it does ;) It is still unclear to me whether Python/numpy compiled with MKL > would be freely re-distributable, as the MSVC version is. > > >Then you must use tools like the processing module, MPI, ... > > I have been using shared memory and numpy with good results lately on > Win32 with tagged memory. > > http://projects.scipy.org/pipermail/numpy-discussion/2007-October/029505.html > > On *NIX, arrayfrombuffer(), mmap and MAP_SHARED, or POSH > http://poshmodule.sourceforge.net/ > seem like the ticket. I hope that one day, there will be a Python version that will be able to handle distributed computers, that is distributing one thread per computer (or core). It could be interesting to use if the algorithm was parallelized. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 10:42:48 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 00:42:48 +0900 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47833835.7060605@scipy.org> <2e1434c10801080722y12515f9q546bac41915ee15@mail.gmail.com> Message-ID: <478399F8.5020303@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > MKL does the multithreading on its own for level 3 BLAS > instructions (OpenMP). For ACML, the problem is that AMD does > not provide a CBLAS interface and is not interested in doing > so. With ACML, the compilation fails with the current Numpy, > but hopefully with Scons it will work, at least for the LAPACK > part. > > > [..] > > That is news to me. I compile NumPy with ACML regularly. I > haven't tried with the current trunk but it worked recently. I > have not tried the parallel versions, though. > > > Could you share with us your setup.cfg and what you did to make Numpy > work with ACML ? Last time I tried (one month ago), I had several > unresolved externals (and I'm sure ACML does not provide the CBLAS > interface which is needed by Numpy). The CBLAS is not needed by numpy, only the fortran interface is [1]. But some functions need the CBLAS interface to be optimized, like numpy.dot. cheers, David [1] actually, no blas is needed for numpy, but to be usable with scipy, you need one. I think, without being sure, that this is so such as numpy can be built wo any fortran compiler. From ellisonbg.net at gmail.com Tue Jan 8 12:57:22 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jan 2008 10:57:22 -0700 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <478298FA.1070701@scipy.org> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> Message-ID: <6ce0ac130801080957g1838e960n123d3b3ffd835a4c@mail.gmail.com> As others have mentioned, the quickest and easiest way of getting these things is to build numpy against a LAPACK/BLAS that has threading support enabled. I have not played with this, but there is no reason it shouldn't work out of the box. On Jan 7, 2008 2:26 PM, dmitrey wrote: > The only one thing I'm very interested in for now - why the most > simplest matrix operations are not implemented to be parallel in numpy > yet (for several-CPU computers, like my AMD Athlon X2). First of all > it's related to matrix multiplication and devision, either point or > matrix (i.e. like A\B, A*B, dot(A,B)). > Another one highly convenient and rather simple thing to be implemented > is direct Decart multiplication like it is mentioned in > pyslice (IIRC I had some troubles with installing the one) > http://scipy.org/Topical_Software#head-cf472934357fda4558aafdf558a977c4d59baecb > I guess for ~95% of users it will be enough, and only 5% will require > message-pass between subprocesses etc. > BTW, IIRC latest MATLAB can uses 2-processors CPU already, and next > version is promised to handle 4-processors as well. > Regards, D. > > > Brian Granger wrote: > > Dmitrey, > > > > This work is being funded by a new NASA grant that I have at Tech-X > > Corporation where I work. The grant officially begins as of Jan 18th, > > so not much has been done as of this point. We have however been > > having some design discussions with various people. > > > > Here is a broad overview of the proposed work: > > > > 1) Distributed arrays (dense) based on numpy+pyrex+mpi > > > > 2) Parallel algorithms that work with those arrays (ffts, linear > > algebra, ufuncs, stats, etc) > > > > The code will be BSD and will end up mostly in the ipython1 project, > > but parts of it might end up also in numpy and mpi4py as well. > > > > I am more than willing to share more details about the work if you are > > interested. But, I will surely post to the numpy list as things move > > forward. > > > > Brian > > > > On Jan 7, 2008 1:13 PM, dmitrey wrote: > > > >> Some days ago there was mentioned a parallel numpy that is developed by > >> Brian Granger. > >> > >> Does the project have any blog or website? Has it any description about > >> API and abilities? When 1st release is intended? > >> > >> Regards, D > >> > >> > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at scipy.org > >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ellisonbg.net at gmail.com Tue Jan 8 13:10:56 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jan 2008 11:10:56 -0700 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: <4782BD91.5060308@enthought.com> References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47829A7B.7010505@gmail.com> <4782BD91.5060308@enthought.com> Message-ID: <6ce0ac130801081010h6cc177e7j98960ceb40300745@mail.gmail.com> > Yes, the problem in this implementation is that it uses pthreads for > synchronization instead of spin locks with a work pool implementation > tailored to numpy. The thread synchronization overhead is horrible > (300,000-400,000 clock cycles) and swamps anything other than very large > arrays. I have played with spin-lock based solutions that cut this to, > on average 3000-4000 cylces. With that, arrays of 1e3-1e4 elements can > start seeing parallelization benefits. However, this code hasn't passed > the mad-scientist tinkering stage... I haven't touched it in at least 6 > months, and I doubt I'll get back to it very soon (go Brian!). It did > look promising for up to 4 processors on some operations (sin, cos, > etc.) and worth-it-but-less-beneficial on simple operations (+,-,*, > etc.). Coupled with something like weave.blitz or numexpr that can (or > could) compile multiple binary operations into a single kernel, the > scaling for expressions with multiple simple operations would scale very > well. The distributed array stuff that I will be doing will really focus on scaling across multiple processes. The design I have in mind would not use threads for multi-core parallelization. Instead, my assumption is that as time goes on the lower-level building blocks that I will be using (numpy/blas/lapack) will themselves have threading support. I am going to make sure as much as possible that I defer actual calculations to numpy/blas/lapack so we get threading support for free. So, I am afraid that my project won't really further Eric's threaded numpy efforts. But, it will definitely help the multicore situation if you have an algorithm that 1) doesn't require message passing or 2) is not communications bound. The other thing to keep in mind is GPUs. This makes things even more complicated. Eventually, I think we will be using a very layered approach with GPUs + CPU threads at the lowest level and message passing across multiple processes at a higher level. Because the NASA grant is focused on deploying things onto NASA supercomputers (where message passing is essentially required) my focus for now will be on that side of things. > My tinkering was aimed at a framework that would allow you to write > little computational kernels in a prescribed way, and then let a numpy > load-balancer automatically split the work up between worker threads > that execute these little kernels. Ideally, this load-balancer would be > pluggable. The inner loop for numpy's universal functions is probably > very close or exactly the interface for these little kernels. Also, it > be nice to couple this with weave so that kernels written with weave > could execute in parallel without user effort. (This is all like the > "map" part of map-reduce architecture... The reduce part also need to > fit in the architecture to generically handle things like sum, etc.) It would be truly fantastic if such a framework could handle all levels of parallelization from the GPU+threads all the way up to message passing. Sounds like we have a lot to do :) > Getting it to work with all flavors of numpy arrays (contiguous, > non-contiguous, buffered, etc.) is quite a chore, but the contiguous > arrays (and perhaps some non-contiguous) offer some relatively low > hanging fruit. Here's to hoping Brian's project bears fruit. The distributed arrays that I am building will store their local data in a numpy array - at this point, I don't think it will require local data to be contiguous. But for shared memory+threading situations, this will be more of an issue. > I haven't thought about matrix ops much, so I don't know if they would > fit this (minimally) described architecture. I am sure that they would > scale well. Here I think the best approach is to defer to threaded enabled BLAS/LAPACK packages. > > eric > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Jan 8 13:13:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 11:13:46 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478286FD.7080006@enthought.com> References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 7, 2008 1:09 PM, Travis E. Oliphant wrote: > Darren Dale wrote: > > One of my collaborators would like to use 16bit float arrays. According > to > > http://www.scipy.org/Tentative_NumPy_Tutorial, and references to float16 > in > > numpy.core.numerictypes, it appears that this should be possible, but > the > > following doesnt work: > > > > No, it's only possible, if the C-implementation NumPy was compiled with > has 16-bits for its float scalar. > > Only float, double, and long double are implemented. These translate to > various bit-widths on different platforms. numerictypes is overly > aggressive at guessing what they might translate to. > I'm starting to get interested in implementing float16 support ;) My tentative program goes something like this: 1) Add the operators to the scalar type. This will give sorting, basic printing, addition, etc. 2) Add conversions to other types. This will allow promotion of data to currently supported types. 3) Unoptimized BLAS or ATLAS additions for the type. This would give array operations. 4) Figure out what the precedence relations are. Can you add some details to this roadmap? It also strikes me that this exercise can be repeated for quad precision floats, so might be a good preparation for future IEEE types like the proposed decimal floating types. Note that we are going to need some way to distinguish quads and extended precision types on 64bit platforms, as both will both register as float128 in the current notation. Decimal floats can be added with the notation decimal64, etc. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Tue Jan 8 13:18:20 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jan 2008 11:18:20 -0700 Subject: [Numpy-discussion] parallel numpy (by Brian Granger) - any info? In-Reply-To: References: <478287D4.5020704@scipy.org> <6ce0ac130801071310k44516415j4cdd78f8791ebf29@mail.gmail.com> <478298FA.1070701@scipy.org> <47833835.7060605@scipy.org> Message-ID: <6ce0ac130801081018vf79697aja0825861f61148c1@mail.gmail.com> On Jan 8, 2008 3:33 AM, Matthieu Brucher wrote: > > > I have AMD processor so I guess I should use ACML somehow instead. > > However, at 1st I would prefer my code to be platform-independent, and > > at 2nd unfortunately I haven't encountered in numpy documentation (in > > website scipy.org and numpy.scipy.org) any mention about how to use > > numpy multithreading at all (neither MKL nor ACML). > > > MKL does the multithreading on its own for level 3 BLAS instructions > (OpenMP). For ACML, the problem is that AMD does not provide a CBLAS > interface and is not interested in doing so. With ACML, the compilation > fails with the current Numpy, but hopefully with Scons it will work, at > least for the LAPACK part. But I don't think that ACML is parallel. > I think that using multithreaded libraries is far more interesting and easy > to do than using distributed memory systems. This is due to the fact that > Python can use some help to enable multi-processing (not GIL), for instance > like Java and Jackal. After some readings, I think this means that the core > Python should be updated. Definitely, the easiest route on a multicore machine is to find a library that is already multithreaded. As long as that library is "below" the GIL you will see a good speedup. But, because of the GIL, distributed memory/message passing applications in Python are much easier to write than shared mem/threaded. Especially with packages like mpi4py, message passing in python is extremely easy and even pretty fast. It is also very simple to call mpi directly from within pyrex. > > > Also, I intended to try using numpy multithreading on our icyb cluster > > (IIRC made of intel processors) from world top 500 (however, currently > > connect to other subsets of processors from other cities have been > > organized, some of them are AMD). Would 100-200 processors (I don't > > remember how many have the one) yield at least 2x...3x speedup on some > > of my test cases, it would be a good deal and something to report in my > > graduation work. If you have a multiprocessor cluster you really should look at using mpi4py. It handles numpy arrays very efficiently and performs extremely well. Threading won't help you a bit in this case. > > If you have access to Intel Quad-Core processors with the latest MKL and if > you intensively use matrix multiplications, you will have those results. But > if you speak at your graduation that using 100 or 200 processors and say > that it only yields a 2 or 3 time speedup factor, I think the jury will not > appreciate. > > > > As my chief informed me, people here are fond of the cluster, mentioning > > the magical word (in my work) would fill them with respect :) > > Then you should first start by looking how to make your algorithms parallel. > Just throwing a number of processors will not yield a good speedup per > processor, and this is what people are looking for : good scalability. Then > you must use tools like the processing module, MPI, ... This is always true. It is shocking how easy it is to write parallel code that is slow as hell. It is very difficult to write parallel code that is fast and scales well. Brian > > > Matthieu > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From peridot.faceted at gmail.com Tue Jan 8 14:03:13 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 8 Jan 2008 14:03:13 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On 08/01/2008, Charles R Harris wrote: > I'm starting to get interested in implementing float16 support ;) My > tentative program goes something like this: > > 1) Add the operators to the scalar type. This will give sorting, basic > printing, addition, etc. > 2) Add conversions to other types. This will allow promotion of data to > currently supported types. > 3) Unoptimized BLAS or ATLAS additions for the type. This would give array > operations. > 4) Figure out what the precedence relations are. > > Can you add some details to this roadmap? It also strikes me that this > exercise can be repeated for quad precision floats, so might be a good > preparation for future IEEE types like the proposed decimal floating types. > Note that we are going to need some way to distinguish quads and extended > precision types on 64bit platforms, as both will both register as float128 > in the current notation. Decimal floats can be added with the notation > decimal64, etc. How were you planning to implement the basic mathematics (addition, multiplication, trig, what have you)? Were you planning to code all that from scratch or is there a 16-bit float math library? Or were you planning to make this work only on machines with 16-bit math hardware? Or ship it out to a graphics card? Without hardware support, people may be happier using 16-bit floats only for on-disk storage... Anne From charlesr.harris at gmail.com Tue Jan 8 14:29:42 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 12:29:42 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 8, 2008 12:03 PM, Anne Archibald wrote: > On 08/01/2008, Charles R Harris wrote: > > > I'm starting to get interested in implementing float16 support ;) My > > tentative program goes something like this: > > > > 1) Add the operators to the scalar type. This will give sorting, basic > > printing, addition, etc. > > 2) Add conversions to other types. This will allow promotion of data to > > currently supported types. > > 3) Unoptimized BLAS or ATLAS additions for the type. This would give > array > > operations. > > 4) Figure out what the precedence relations are. > > > > Can you add some details to this roadmap? It also strikes me that this > > exercise can be repeated for quad precision floats, so might be a good > > preparation for future IEEE types like the proposed decimal floating > types. > > Note that we are going to need some way to distinguish quads and > extended > > precision types on 64bit platforms, as both will both register as > float128 > > in the current notation. Decimal floats can be added with the notation > > decimal64, etc. > > How were you planning to implement the basic mathematics (addition, > multiplication, trig, what have you)? Were you planning to code all Basic math isn't bad and I expect that there are already emulations out there. Comparisons look easy for all the floating types. Type conversions are going to be a mess if we get into the decimal floats. And support for ufuncs is probably pointless for the type as is. I will probably look into using up/down conversions on an element by element basis if I do go that route. Logic operators shouldn't be a problem, avg and such already accumulate in greater precision, so there is probably a subset of the ufuncts that should go right over. > that from scratch or is there a 16-bit float math library? Or were you > planning to make this work only on machines with 16-bit math hardware? > Or ship it out to a graphics card? > > Without hardware support, people may be happier using 16-bit floats > only for on-disk storage... Well, at a minimum people will want to read, write, print, and promote them. That would at least let people work with the numbers, and since my understanding is that the main virtue of the format is compactness for storage and communication, a basic need will be filled right there. One potential problem I see is handling +/-inf and nans, tests for these should probably be built into the type. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 8 14:45:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 13:45:41 -0600 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <47834568.3010908@ar.media.kyoto-u.ac.jp> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> <47831E85.3050001@ar.media.kyoto-u.ac.jp> <47834568.3010908@ar.media.kyoto-u.ac.jp> Message-ID: <4783D2E5.10602@gmail.com> David Cournapeau wrote: > Matthieu Brucher wrote: >> > Oups, safe for the "/trunk:1-2871" part. This should be deleted >> before >> > a commit to the trunk, I think. >> Yes, that's what I (quite unclearly) meant: since revision numbers are >> per- repository in svn, I don't understand the point of tracking trunk >> revisions: I would think that tracking the last merged version for >> each >> branch to be enough (for the kind of merge svn does, at least). If >> trunk >> version are tracked, then I would expect two branches using >> svnmerge to >> clash each other, >> >> >> In fact, the trunk should be tracked from all the branches, although >> there will be the problem with merging the different branches (I did >> not have many troubles with that, but I only tried with a few >> differences) into the trunk. I don't think only one branch wants to be >> up to date with the trunk ;). > Sure, that's what I was trying to do (tracking the trunk). But if > svnmerge needs the information from the trunk, this means each branch > will need a different value, whereas only one value is possible... IIRC, its presence in the trunk/'s svnmerge-integrated property is a mistake. It belongs in each of the branches independently. For some reason, it got merged into the trunk from one of the branches by mistake, possibly because someone started merge tracking with svnmerge then another used "svn merge". It should be removed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Tue Jan 8 15:09:53 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 8 Jan 2008 15:09:53 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On 08/01/2008, Charles R Harris wrote: > Well, at a minimum people will want to read, write, print, and promote them. > That would at least let people work with the numbers, and since my > understanding is that the main virtue of the format is compactness for > storage and communication, a basic need will be filled right there. One > potential problem I see is handling +/-inf and nans, tests for these should > probably be built into the type. The el-cheapo solution to this simply provides two functions: take an int16 array (which actually contains float16) and produce a float32 array, and vice versa. Then people do all their work in float32 (or float64 is float32 doesn't have inf/nan, I don't remember) but can read and write float16. Of course it would be nicer to use flaot16 natively, more or less, but without all the math that's going to be a frustrating experience. Anne From charlesr.harris at gmail.com Tue Jan 8 15:24:49 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 13:24:49 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 8, 2008 1:09 PM, Anne Archibald wrote: > On 08/01/2008, Charles R Harris wrote: > > > Well, at a minimum people will want to read, write, print, and promote > them. > > That would at least let people work with the numbers, and since my > > understanding is that the main virtue of the format is compactness for > > storage and communication, a basic need will be filled right there. One > > potential problem I see is handling +/-inf and nans, tests for these > should > > probably be built into the type. > > The el-cheapo solution to this simply provides two functions: take an > int16 array (which actually contains float16) and produce a float32 > array, and vice versa. Then people do all their work in float32 (or > float64 is float32 doesn't have inf/nan, I don't remember) but can > read and write float16. > Sure, but where's the fun in that? Besides, I think that adding a data type might be an opportunity to generate a detailed road map for future projects that might actually matter (quads and decimal floats for money stuff), and might provide a chance to revisit some code and see if it can be simplified. It's tough to get up the motivation to do that without some other prod. Besides, it's new and I have a weakness for new stuff. > Of course it would be nicer to use flaot16 natively, more or less, but > without all the math that's going to be a frustrating experience. I would plan on at least arithmetic. Adding special functions probably ain't worth it and even now a lot of things are done by promoting things to floats or doubles and calling routines in LAPACK. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From darren.dale at cornell.edu Tue Jan 8 15:43:56 2008 From: darren.dale at cornell.edu (Darren Dale) Date: Tue, 8 Jan 2008 15:43:56 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: <200801081543.57133.darren.dale@cornell.edu> On Tuesday 08 January 2008 03:24:49 pm Charles R Harris wrote: > On Jan 8, 2008 1:09 PM, Anne Archibald wrote: > > On 08/01/2008, Charles R Harris wrote: > > > Well, at a minimum people will want to read, write, print, and promote > > > > them. > > > > > That would at least let people work with the numbers, and since my > > > understanding is that the main virtue of the format is compactness for > > > storage and communication, a basic need will be filled right there. One > > > potential problem I see is handling +/-inf and nans, tests for these > > > > should > > > > > probably be built into the type. > > > > The el-cheapo solution to this simply provides two functions: take an > > int16 array (which actually contains float16) and produce a float32 > > array, and vice versa. Then people do all their work in float32 (or > > float64 is float32 doesn't have inf/nan, I don't remember) but can > > read and write float16. > > Sure, but where's the fun in that? Besides, I think that adding a data type > might be an opportunity to generate a detailed road map for future projects > that might actually matter (quads and decimal floats for money stuff), and > might provide a chance to revisit some code and see if it can be > simplified. It's tough to get up the motivation to do that without some > other prod. Besides, it's new and I have a weakness for new stuff. > > > Of course it would be nicer to use flaot16 natively, more or less, but > > without all the math that's going to be a frustrating experience. > > I would plan on at least arithmetic. Adding special functions probably > ain't worth it and even now a lot of things are done by promoting things to > floats or doubles and calling routines in LAPACK. For what its worth, the people I work with were interested in using float16's for their work too (not just storage), but since they are not available, they will use float32 and are investigating pytables to cut back on memory and disk use. -- Darren S. Dale, Ph.D. Staff Scientist Cornell High Energy Synchrotron Source Cornell University 275 Wilson Lab Rt. 366 & Pine Tree Road Ithaca, NY 14853 darren.dale at cornell.edu office: (607) 255-3819 fax: (607) 255-9001 http://www.chess.cornell.edu From wbaxter at gmail.com Tue Jan 8 15:58:05 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 9 Jan 2008 05:58:05 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: If you're really going to try to do it, Charles, there's an implementation of float16 in the OpenEXR toolkit. http://www.openexr.com/ Or more precisely it's in the files in the Half/ directory of this: http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz I don't know if it's IEEE conformant or not (especially w.r.t. NaN's and such) but it should be a good start. The code seems to be well documented. --bb On Jan 9, 2008 5:24 AM, Charles R Harris wrote: > > > > On Jan 8, 2008 1:09 PM, Anne Archibald wrote: > > > > On 08/01/2008, Charles R Harris wrote: > > > > > > > Well, at a minimum people will want to read, write, print, and promote > them. > > > That would at least let people work with the numbers, and since my > > > understanding is that the main virtue of the format is compactness for > > > storage and communication, a basic need will be filled right there. One > > > potential problem I see is handling +/-inf and nans, tests for these > should > > > probably be built into the type. > > > > The el-cheapo solution to this simply provides two functions: take an > > int16 array (which actually contains float16) and produce a float32 > > array, and vice versa. Then people do all their work in float32 (or > > float64 is float32 doesn't have inf/nan, I don't remember) but can > > read and write float16. > > > > Sure, but where's the fun in that? Besides, I think that adding a data type > might be an opportunity to generate a detailed road map for future projects > that might actually matter (quads and decimal floats for money stuff), and > might provide a chance to revisit some code and see if it can be simplified. > It's tough to get up the motivation to do that without some other prod. > Besides, it's new and I have a weakness for new stuff. > > > > > > Of course it would be nicer to use flaot16 natively, more or less, but > > without all the math that's going to be a frustrating experience. > > I would plan on at least arithmetic. Adding special functions probably ain't > worth it and even now a lot of things are done by promoting things to floats > or doubles and calling routines in LAPACK. > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From jdh2358 at gmail.com Tue Jan 8 17:23:00 2008 From: jdh2358 at gmail.com (John Hunter) Date: Tue, 8 Jan 2008 14:23:00 -0800 Subject: [Numpy-discussion] major changes in matplotlib svn Message-ID: <88e473830801081423r27f11642h36a7d2493a15fb5a@mail.gmail.com> Apologies for the off-topic post to the numpy list, but we have just committed some potentially code-breaking changes to the matplotlib svn repository, and we want to gve as wide a notification to people as possible. Please do not reply to the numpy list, but rather to a matplotlib mailing list . Migrating to the new matplotlib codebase ======================================== Michael Droettboom has spent the last several months working on the "transforms branch" of matplotlib, in which he rewrote from the ground up the transformation infrastructure in matplotlib, which many found unintuitive and hard to extend. In addition to a cleaner code base, the reorganization allows you to define your own transformations and projections (e.g. map projections) within matplotlib. He has merged his work into the HEAD of the svn trunk, and this will be the basis for future matplotlib releases. If you are a svn user, we encourage you to continue using the trunk as before, but with the understanding that you are now truly on the bleeding edge. Michael has made sure all the examples still pass with the new code base, so for the vast majority of you, I expect to see few problems. But we need to get as many people as possible using the new code base so we can find and fix the remaining problems. We have take the svn code used in the last stable release in the 0.91 series, and made it a maintenance branch so we can still fix bugs and support people who are not ready to migrate to the new transformation infrastructure but nonetheless need access to svn bug fixes. Using the new code ================== To check out the trunk with the latest transforms changes: > svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/matplotlib If you already have a working copy of the trunk, your next "svn up" will include the latest transforms changes. Before installing, make sure you completely remove the old matplotlib build and install directories, eg: > cd matplotlib > sudo rm -rf build > sudo rm -rf /usr/local/lib/python2.5/site-packages/matplotlib > sudo python setup.py install Using the old svn code ====================== To check out the maintenance branch, in order to commit bugfixes to 0.91.x: > svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v0_91_maint matplotlib_0_91_maint Any applicable bugfixes on the 0.91.x maintenance branch should be merged into the trunk so they are fixed there as well. Svnmerge.py makes this process rather straightforward, but you may also manually merge if you prefer. Merging bugfixes on the maint branch to the trunk using svnmerge.py ------------------------------------------------------------------- Download svnmerge.py from here: http://www.orcaware.com/svn/wiki/Svnmerge.py >From the working copy of the *trunk* (svnmerge.py always pulls *to* the current working copy), so > svnmerge.py merge to pull in changes from the maintenance branch. Manually resolve any conflicts, if necessary, test them, and then commit with > svn commit -F svnmerge-commit-message.txt (Note the above will stop working when the maintenance branch is abandoned.) API CHANGES in the new transformation infrastructure ==================================================== While Michael worked hard to keep the API mostly unchanged while performing what has been called "open heart surgery on matplotlib", there have been some changes, as discussed below. The primary goal of these changes was to make it easier to extend matplotlib to support new kinds of projections. This is primarily an internal improvement, and the possible user-visible changes it allows are yet to come. These changes are detailed in the API_CHANGES document. From charlesr.harris at gmail.com Tue Jan 8 18:03:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 16:03:21 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 8, 2008 1:58 PM, Bill Baxter wrote: > If you're really going to try to do it, Charles, there's an > implementation of float16 in the OpenEXR toolkit. > http://www.openexr.com/ > > Or more precisely it's in the files in the Half/ directory of this: > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > and such) but it should be a good start. The code seems to be well > documented. The license looks good, essentially BSD. The code is all C++, which is the obvious way to go for this sort of thing, and I would like to stick with it, but that could lead to build/compatibility problems. I think NumPy itself should really be in C++. Maybe scons can help with the build. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Tue Jan 8 19:01:55 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 9 Jan 2008 09:01:55 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 9, 2008 8:03 AM, Charles R Harris wrote: > On Jan 8, 2008 1:58 PM, Bill Baxter wrote: > > If you're really going to try to do it, Charles, there's an > > implementation of float16 in the OpenEXR toolkit. > > http://www.openexr.com/ > > > > Or more precisely it's in the files in the Half/ directory of this: > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > > > I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > > and such) but it should be a good start. The code seems to be well > > documented. > > The license looks good, essentially BSD. The code is all C++, which is the > obvious way to go for this sort of thing, and I would like to stick with it, > but that could lead to build/compatibility problems. I think NumPy itself > should really be in C++. Maybe scons can help with the build. Yeh, I was just thinking you'd rip out and C-ify the main algorithms rather than trying to wrap it as-is. --bb From charlesr.harris at gmail.com Tue Jan 8 19:18:23 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 17:18:23 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: On Jan 8, 2008 5:01 PM, Bill Baxter wrote: > On Jan 9, 2008 8:03 AM, Charles R Harris > wrote: > > On Jan 8, 2008 1:58 PM, Bill Baxter wrote: > > > If you're really going to try to do it, Charles, there's an > > > implementation of float16 in the OpenEXR toolkit. > > > http://www.openexr.com/ > > > > > > Or more precisely it's in the files in the Half/ directory of this: > > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > > > > > I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > > > and such) but it should be a good start. The code seems to be well > > > documented. > > > > The license looks good, essentially BSD. The code is all C++, which is > the > > obvious way to go for this sort of thing, and I would like to stick with > it, > > but that could lead to build/compatibility problems. I think NumPy > itself > > should really be in C++. Maybe scons can help with the build. > > Yeh, I was just thinking you'd rip out and C-ify the main algorithms > rather than trying to wrap it as-is. I'd rather not C-ify the thing, I'd rather C++-ify parts of NumPy. I note that MatPlotLab uses C++, so some of the problems must have been solved. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Tue Jan 8 19:49:45 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 9 Jan 2008 09:49:45 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: On Jan 9, 2008 9:18 AM, Charles R Harris wrote: > On Jan 8, 2008 5:01 PM, Bill Baxter wrote: > > On Jan 9, 2008 8:03 AM, Charles R Harris > wrote: > > > On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com> wrote: > > > > If you're really going to try to do it, Charles, there's an > > > > implementation of float16 in the OpenEXR toolkit. > > > > http://www.openexr.com/ > > > > > > > > Or more precisely it's in the files in the Half/ directory of this: > > > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > > > > > > > I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > > > > and such) but it should be a good start. The code seems to be well > > > > documented. > > > > > > The license looks good, essentially BSD. The code is all C++, which is > the > > > obvious way to go for this sort of thing, and I would like to stick with > it, > > > but that could lead to build/compatibility problems. I think NumPy > itself > > > should really be in C++. Maybe scons can help with the build. > > > > Yeh, I was just thinking you'd rip out and C-ify the main algorithms > > rather than trying to wrap it as-is. > > I'd rather not C-ify the thing, I'd rather C++-ify parts of NumPy. I note > that MatPlotLab uses C++, so some of the problems must have been solved. If you think that's easier then go for it. As far as classes go, though, you pretty much won't find anything easier than this to C-ify. Or just to wrap in C. No templates. No inheritance. Just a simple value-type struct that supports a small set of operations. --bb From ndbecker2 at gmail.com Tue Jan 8 20:26:42 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 08 Jan 2008 20:26:42 -0500 Subject: [Numpy-discussion] def of var of complex Message-ID: I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: var () -> (close to 1.0) var () -> (close to 1.0) but var (complex array) -> (close to complex 0) Is that not a strange definition? From efiring at hawaii.edu Tue Jan 8 20:49:15 2008 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 08 Jan 2008 15:49:15 -1000 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> Message-ID: <4784281B.3060900@hawaii.edu> Bill Baxter wrote: > On Jan 9, 2008 9:18 AM, Charles R Harris wrote: >> On Jan 8, 2008 5:01 PM, Bill Baxter wrote: >>> On Jan 9, 2008 8:03 AM, Charles R Harris >> wrote: >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com> wrote: >>>>> If you're really going to try to do it, Charles, there's an >>>>> implementation of float16 in the OpenEXR toolkit. >>>>> http://www.openexr.com/ >>>>> >>>>> Or more precisely it's in the files in the Half/ directory of this: >>>>> >> http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz >>>>> I don't know if it's IEEE conformant or not (especially w.r.t. NaN's >>>>> and such) but it should be a good start. The code seems to be well >>>>> documented. >>>> The license looks good, essentially BSD. The code is all C++, which is >> the >>>> obvious way to go for this sort of thing, and I would like to stick with >> it, >>>> but that could lead to build/compatibility problems. I think NumPy >> itself >>>> should really be in C++. Maybe scons can help with the build. >>> Yeh, I was just thinking you'd rip out and C-ify the main algorithms >>> rather than trying to wrap it as-is. >> I'd rather not C-ify the thing, I'd rather C++-ify parts of NumPy. I note >> that MatPlotLab uses C++, so some of the problems must have been solved. A big chunk of C++ in matplotlib has just been replaced, largely because it was so hard to understand and extend for everyone but its author. There is still C++ code as well as C code in mpl. Personally, I prefer the C. > > If you think that's easier then go for it. The opinion that C++ would improve numpy is not universal, and has been discussed. http://article.gmane.org/gmane.comp.python.numeric.general/13244 Eric > > As far as classes go, though, you pretty much won't find anything > easier than this to C-ify. Or just to wrap in C. No templates. No > inheritance. Just a simple value-type struct that supports a small > set of operations. > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Tue Jan 8 20:54:07 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 19:54:07 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: References: Message-ID: <4784293F.5090705@gmail.com> Neal Becker wrote: > I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: > > var () -> (close to 1.0) > var () -> (close to 1.0) > > but > > var (complex array) -> (close to complex 0) > > Is that not a strange definition? There is some discussion on this in the tracker. http://projects.scipy.org/scipy/numpy/ticket/638 The current state of affairs is that the implementation of var() just naively applies the standard formula for real numbers. mean((x - mean(x)) ** 2) I think this is pretty obviously wrong prima facie. AFAIK, no one considers this a valid definition of variance for complex RVs or in fact a useful value. I think we should change this. Unfortunately, there is no single alternative but several. 1. Punt. Complex numbers are inherently multidimensional, and a single scale parameter doesn't really describe most distributions of complex numbers. Instead, you need a real covariance matrix which you can get with cov([z.real, z.imag]). This estimates the covariance matrix of a 2-D Gaussian distribution over RR^2 (interpreted as CC). 2. Take a slightly less naive formula for the variance which seems to show up in some texts: mean(absolute(z - mean(z)) ** 2) This estimates the single parameter of a circular Gaussian over RR^2 (interpreted as CC). It is also the trace of the covariance matrix above. 3. Take the variances of the real and imaginary components independently. This is equivalent to taking the diagonal of the covariance matrix above. This wouldn't be the definition of "*the* complex variance" that anyone else uses, but rather another form of punting. "There isn't a single complex variance to give you, but in the spirit of broadcasting, we'll compute the marginal variances of each dimension independently." Personally, I like 1 a lot. I'm hesitant to support 2 until I've seen an actual application of that definition. The references I have been given in the ticket comments are all early parts of books where the authors are laying out definitions without applications. Personally, it feels to me like the authors are just sticking in the absolute()'s ex post facto just so they can extend the definition they already have to complex numbers. I'm also not a fan of the expectation-centric treatments of random variables. IMO, the variance of an arbitrary RV isn't an especially important quantity. It's a parameter of a Gaussian distribution, and in this case, I see no reason to favor circular Gaussians in CC over general ones. But if someone shows me an actual application of the definition, I can amend my view. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From simonpy2008 at gmail.com Tue Jan 8 20:49:23 2008 From: simonpy2008 at gmail.com (Simon) Date: Wed, 9 Jan 2008 01:49:23 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy=2Elinalg=2Eeigvals_crashes_whn?= =?utf-8?q?_calling_lapack=5Flite=2Epyd?= Message-ID: Newbie here. Trying to generate eigenvalues from a matrix using: print numpy.linalg.eigvals(matrix) This works with small matrices, say 5 x 5, but causes python to crash on larger matrices, say 136 x 136, which is not really very large. Setup: Win XP SP2 Python 2.5.1 (from .msi) numpy 1.0.4 (from .msi) pywin32-210 (from .exe installer) When running from either the command line or the Pythonwin IDE, python.exe crashes. The info in the microsoft error reporting thingy is: AppName: python.exe ModName: lapack_lite.pyd Offset: 000b7434 Stepping through linalg.py using Pythonwin, I get as far as line 418 (in the eigvals function): results = lapack_routine('N', 'N', n, a, n, wr, wi, dummy, 1, dummy, 1, work, lwork, 0) and then python.exe crashes. That's the extent of my troubleshooting skills at this stage. I haven't worked out if there is a specific matrix size where this starts occurring. Where to now? I can send the actual data for the matrix if need be, but as it's very large I thought it would mess up the list if I posted it here. Simon From ndbecker2 at gmail.com Tue Jan 8 21:34:24 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 08 Jan 2008 21:34:24 -0500 Subject: [Numpy-discussion] def of var of complex References: <4784293F.5090705@gmail.com> Message-ID: Robert Kern wrote: > Neal Becker wrote: >> I noticed that if I generate complex rv i.i.d. with var=1, that numpy >> says: >> >> var () -> (close to 1.0) >> var () -> (close to 1.0) >> >> but >> >> var (complex array) -> (close to complex 0) >> >> Is that not a strange definition? > > There is some discussion on this in the tracker. > > http://projects.scipy.org/scipy/numpy/ticket/638 > > The current state of affairs is that the implementation of var() just > naively applies the standard formula for real numbers. > > mean((x - mean(x)) ** 2) > > I think this is pretty obviously wrong prima facie. AFAIK, no one > considers this a valid definition of variance for complex RVs or in fact a > useful value. I think we should change this. Unfortunately, there is no > single alternative but several. > > 1. Punt. Complex numbers are inherently multidimensional, and a single > scale parameter doesn't really describe most distributions of complex > numbers. Instead, you need a real covariance matrix which you can get with > cov([z.real, z.imag]). This estimates the covariance matrix of a 2-D > Gaussian distribution over RR^2 (interpreted as CC). > > 2. Take a slightly less naive formula for the variance which seems to show > up in some texts: > > mean(absolute(z - mean(z)) ** 2) > > This estimates the single parameter of a circular Gaussian over RR^2 > (interpreted as CC). It is also the trace of the covariance matrix above. > > 3. Take the variances of the real and imaginary components independently. > This is equivalent to taking the diagonal of the covariance matrix above. > This wouldn't be the definition of "*the* complex variance" that anyone > else uses, but rather another form of punting. "There isn't a single > complex variance to give you, but in the spirit of broadcasting, we'll > compute the marginal variances of each dimension independently." > > Personally, I like 1 a lot. I'm hesitant to support 2 until I've seen an > actual application of that definition. The references I have been given in > the ticket comments are all early parts of books where the authors are > laying out definitions without applications. Personally, it feels to me > like the authors are just sticking in the absolute()'s ex post facto just > so they can extend the definition they already have to complex numbers. > I'm also not a fan of the expectation-centric treatments of random > variables. IMO, the variance of an arbitrary RV isn't an especially > important quantity. It's a parameter of a Gaussian distribution, and in > this case, I see no reason to favor circular Gaussians in CC over general > ones. > > But if someone shows me an actual application of the definition, I can > amend my view. > 2 is what I expected. Suppose I have a complex signal x, with additive Gaussian noise (i.i.d, real and imag are independent). y = x + n Consider an estimate \hat{x} = y. What is the mean-squared-error E[(y - x)^2] ? Definition 2 is consistent with that, and gets my vote. From charlesr.harris at gmail.com Tue Jan 8 21:36:18 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 19:36:18 -0700 Subject: [Numpy-discussion] def of var of complex In-Reply-To: <4784293F.5090705@gmail.com> References: <4784293F.5090705@gmail.com> Message-ID: On Jan 8, 2008 6:54 PM, Robert Kern wrote: > Neal Becker wrote: > > I noticed that if I generate complex rv i.i.d. with var=1, that numpy > says: > > > > var () -> (close to 1.0) > > var () -> (close to 1.0) > > > > but > > > > var (complex array) -> (close to complex 0) > > > > Is that not a strange definition? > > There is some discussion on this in the tracker. > > http://projects.scipy.org/scipy/numpy/ticket/638 > > The current state of affairs is that the implementation of var() just > naively > applies the standard formula for real numbers. > > mean((x - mean(x)) ** 2) > > I think this is pretty obviously wrong prima facie. AFAIK, no one > considers this > a valid definition of variance for complex RVs or in fact a useful value. > I > think we should change this. Unfortunately, there is no single alternative > but > several. > > 1. Punt. Complex numbers are inherently multidimensional, and a single > scale > parameter doesn't really describe most distributions of complex numbers. > Instead, you need a real covariance matrix which you can get with cov([ > z.real, > z.imag]). This estimates the covariance matrix of a 2-D Gaussian > distribution > over RR^2 (interpreted as CC). > > 2. Take a slightly less naive formula for the variance which seems to show > up in > some texts: > > mean(absolute(z - mean(z)) ** 2) > > This estimates the single parameter of a circular Gaussian over RR^2 > (interpreted as CC). It is also the trace of the covariance matrix above. > > 3. Take the variances of the real and imaginary components independently. > This > is equivalent to taking the diagonal of the covariance matrix above. This > wouldn't be the definition of "*the* complex variance" that anyone else > uses, > but rather another form of punting. "There isn't a single complex variance > to > give you, but in the spirit of broadcasting, we'll compute the marginal > variances of each dimension independently." > > Personally, I like 1 a lot. I'm hesitant to support 2 until I've seen an > actual > application of that definition. The references I have been given in the > ticket > comments are all early parts of books where the authors are laying out > definitions without applications. Personally, it feels to me like the > authors > are just sticking in the absolute()'s ex post facto just so they can > extend the > definition they already have to complex numbers. I'm also not a fan of the > expectation-centric treatments of random variables. IMO, the variance of > an > arbitrary RV isn't an especially important quantity. It's a parameter of a > Gaussian distribution, and in this case, I see no reason to favor circular > Gaussians in CC over general ones. > > But if someone shows me an actual application of the definition, I can > amend my > view. > Suppose you have a set of z_i and want to choose z to minimize the average square error $ \sum_i |z_i - z|^2 $. The solution is that $z=\mean{z_i}$ and the resulting average error is given by 2). Note that I didn't mention Gaussians anywhere. No distribution is needed to justify the argument, just the idea of minimizing the squared distance. Leaving out the ^2 would yield another metric, or one could ask for a minmax solution. It is a question of the distance function, not probability. Anyway, that is one justification for the approach in 2) and it is one that makes a lot of applied math simple. Whether of not a least squares fit is useful is different question. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 8 21:48:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 20:48:37 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: References: <4784293F.5090705@gmail.com> Message-ID: <47843605.7090109@gmail.com> Charles R Harris wrote: > Suppose you have a set of z_i and want to choose z to minimize the > average square error $ \sum_i |z_i - z|^2 $. The solution is that > $z=\mean{z_i}$ and the resulting average error is given by 2). Note that > I didn't mention Gaussians anywhere. No distribution is needed to > justify the argument, just the idea of minimizing the squared distance. > Leaving out the ^2 would yield another metric, or one could ask for a > minmax solution. It is a question of the distance function, not > probability. Anyway, that is one justification for the approach in 2) > and it is one that makes a lot of applied math simple. Whether of not a > least squares fit is useful is different question. If you're not doing probability, then what are you using var() for? I can accept that the quantity is meaningful for your problem, but I'm not convinced it's a variance. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Jan 8 21:56:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 20:56:10 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: References: <4784293F.5090705@gmail.com> Message-ID: <478437CA.8080903@gmail.com> Neal Becker wrote: > 2 is what I expected. Suppose I have a complex signal x, with additive > Gaussian noise (i.i.d, real and imag are independent). > y = x + n Not only do the real and imag marginal distributions have to be independent, but also of the same scale, i.e. Re(n) ~ Gaussian(0, sigma) and Im(n) ~ Gaussian(0, sigma) for the same sigma. > Consider an estimate \hat{x} = y. > > What is the mean-squared-error E[(y - x)^2] ? > > Definition 2 is consistent with that, and gets my vote. Ah, you have to be careful. What you wrote is what is implemented. Definition 2 is consistent with this, instead: E[|y - x|^2] But like I said, I see no particular reason to favor circular Gaussians over the general form for the implementation of numpy.var(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Tue Jan 8 21:56:34 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 11:56:34 +0900 Subject: [Numpy-discussion] Using svnmerge on numpy: am I missing something ? In-Reply-To: <4783D2E5.10602@gmail.com> References: <47830D26.9020700@ar.media.kyoto-u.ac.jp> <47831027.3040205@ar.media.kyoto-u.ac.jp> <47831AC1.6050808@ar.media.kyoto-u.ac.jp> <47831E85.3050001@ar.media.kyoto-u.ac.jp> <47834568.3010908@ar.media.kyoto-u.ac.jp> <4783D2E5.10602@gmail.com> Message-ID: <478437E2.90606@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > David Cournapeau wrote: > >> Matthieu Brucher wrote: >> >>> > Oups, safe for the "/trunk:1-2871" part. This should be deleted >>> before >>> > a commit to the trunk, I think. >>> Yes, that's what I (quite unclearly) meant: since revision numbers are >>> per- repository in svn, I don't understand the point of tracking trunk >>> revisions: I would think that tracking the last merged version for >>> each >>> branch to be enough (for the kind of merge svn does, at least). If >>> trunk >>> version are tracked, then I would expect two branches using >>> svnmerge to >>> clash each other, >>> >>> >>> In fact, the trunk should be tracked from all the branches, although >>> there will be the problem with merging the different branches (I did >>> not have many troubles with that, but I only tried with a few >>> differences) into the trunk. I don't think only one branch wants to be >>> up to date with the trunk ;). >>> >> Sure, that's what I was trying to do (tracking the trunk). But if >> svnmerge needs the information from the trunk, this means each branch >> will need a different value, whereas only one value is possible... >> > > IIRC, its presence in the trunk/'s svnmerge-integrated property is a mistake. It > belongs in each of the branches independently. For some reason, it got merged > into the trunk from one of the branches by mistake, possibly because someone > started merge tracking with svnmerge then another used "svn merge". It should be > removed. > > I have to confess I may be the one who screwed up when creating one of the sconc branch (I normally take care to never commit to the numpy trunk, but I may have missed it using svnmerge). cheers, David From charlesr.harris at gmail.com Tue Jan 8 22:20:43 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 20:20:43 -0700 Subject: [Numpy-discussion] def of var of complex In-Reply-To: <47843605.7090109@gmail.com> References: <4784293F.5090705@gmail.com> <47843605.7090109@gmail.com> Message-ID: On Jan 8, 2008 7:48 PM, Robert Kern wrote: > Charles R Harris wrote: > > > Suppose you have a set of z_i and want to choose z to minimize the > > average square error $ \sum_i |z_i - z|^2 $. The solution is that > > $z=\mean{z_i}$ and the resulting average error is given by 2). Note that > > I didn't mention Gaussians anywhere. No distribution is needed to > > justify the argument, just the idea of minimizing the squared distance. > > Leaving out the ^2 would yield another metric, or one could ask for a > > minmax solution. It is a question of the distance function, not > > probability. Anyway, that is one justification for the approach in 2) > > and it is one that makes a lot of applied math simple. Whether of not a > > least squares fit is useful is different question. > > If you're not doing probability, then what are you using var() for? I can > accept > that the quantity is meaningful for your problem, but I'm not convinced > it's a > variance. > Lots of fits don't involve probability distributions. For instance, one might want to fit a polynomial to a mathematical curve. This sort of distinction between probability and distance goes back to Gauss himself, although not in his original work on least squares. Whether or not variance implies probability is a semantic question. I think if we are going to compute a single number, 2) is as good as anything even if it doesn't capture the shape of the scatter plot. A 2D covariance wouldn't necessarily capture the shape either. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 22:14:17 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 12:14:17 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <478286FD.7080006@enthought.com> Message-ID: <47843C09.4040908@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 8, 2008 1:58 PM, Bill Baxter > wrote: > > If you're really going to try to do it, Charles, there's an > implementation of float16 in the OpenEXR toolkit. > http://www.openexr.com/ > > Or more precisely it's in the files in the Half/ directory of this: > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > and such) but it should be a good start. The code seems to be well > documented. > > > The license looks good, essentially BSD. The code is all C++, which is > the obvious way to go for this sort of thing, and I would like to > stick with it, but that could lead to build/compatibility problems. I > think NumPy itself should really be in C++. Maybe scons can help with > the build. Why do you think scons would help ? You can build C++ code today with distutils; but I am not sure it is ok to use C++ in numpy (e.g. adding a C++ compiler dependency). Also, note that C++ is really a mess to support on multi-platform because of compiler incompabilities (at the source and at the binary level). cheers, David From charlesr.harris at gmail.com Tue Jan 8 22:36:34 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 20:36:34 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784281B.3060900@hawaii.edu> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> Message-ID: On Jan 8, 2008 6:49 PM, Eric Firing wrote: > Bill Baxter wrote: > > On Jan 9, 2008 9:18 AM, Charles R Harris > wrote: > >> On Jan 8, 2008 5:01 PM, Bill Baxter wrote: > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > >> wrote: > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com> wrote: > >>>>> If you're really going to try to do it, Charles, there's an > >>>>> implementation of float16 in the OpenEXR toolkit. > >>>>> http://www.openexr.com/ > >>>>> > >>>>> Or more precisely it's in the files in the Half/ directory of this: > >>>>> > >> > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > >>>>> I don't know if it's IEEE conformant or not (especially w.r.t. NaN's > >>>>> and such) but it should be a good start. The code seems to be well > >>>>> documented. > >>>> The license looks good, essentially BSD. The code is all C++, which > is > >> the > >>>> obvious way to go for this sort of thing, and I would like to stick > with > >> it, > >>>> but that could lead to build/compatibility problems. I think NumPy > >> itself > >>>> should really be in C++. Maybe scons can help with the build. > >>> Yeh, I was just thinking you'd rip out and C-ify the main algorithms > >>> rather than trying to wrap it as-is. > >> I'd rather not C-ify the thing, I'd rather C++-ify parts of NumPy. I > note > >> that MatPlotLab uses C++, so some of the problems must have been > solved. > > A big chunk of C++ in matplotlib has just been replaced, largely because > it was so hard to understand and extend for everyone but its author. > There is still C++ code as well as C code in mpl. Personally, I prefer > the C. > > > > > If you think that's easier then go for it. > > The opinion that C++ would improve numpy is not universal, and has been > discussed. > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > Ah, the trick to interfacing C to C++ is to derive the C++ classes from the numpy structures and keep the function pointers. Then all the offsets would work. I would think the main advantage of C++ would be in arithmetic operator overloading and using template classes while carefully restricting such things to numbers. Mostly, I would use C++ inline class methods as shorthand that would compile to what the C approach would do. I suppose we could write a python preprocessor that would do essentially the same thing; C++ started that way. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 22:42:07 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 12:42:07 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> Message-ID: <4784428F.7070803@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 8, 2008 6:49 PM, Eric Firing > wrote: > > Bill Baxter wrote: > > On Jan 9, 2008 9:18 AM, Charles R Harris > > wrote: > >> On Jan 8, 2008 5:01 PM, Bill Baxter < wbaxter at gmail.com > > wrote: > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > > > >> wrote: > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com > > wrote: > >>>>> If you're really going to try to do it, Charles, there's an > >>>>> implementation of float16 in the OpenEXR toolkit. > >>>>> http://www.openexr.com/ > >>>>> > >>>>> Or more precisely it's in the files in the Half/ directory > of this: > >>>>> > >> > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > >>>>> I don't know if it's IEEE conformant or not (especially > w.r.t. NaN's > >>>>> and such) but it should be a good start. The code seems to > be well > >>>>> documented. > >>>> The license looks good, essentially BSD. The code is all C++, > which is > >> the > >>>> obvious way to go for this sort of thing, and I would like to > stick with > >> it, > >>>> but that could lead to build/compatibility problems. I think > NumPy > >> itself > >>>> should really be in C++. Maybe scons can help with the build. > >>> Yeh, I was just thinking you'd rip out and C-ify the main > algorithms > >>> rather than trying to wrap it as-is. > >> I'd rather not C-ify the thing, I'd rather C++-ify parts of > NumPy. I note > >> that MatPlotLab uses C++, so some of the problems must have > been solved. > > A big chunk of C++ in matplotlib has just been replaced, largely > because > it was so hard to understand and extend for everyone but its author. > There is still C++ code as well as C code in mpl. Personally, I > prefer > the C. > > > > > If you think that's easier then go for it. > > The opinion that C++ would improve numpy is not universal, and has > been > discussed. > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > > > > Ah, the trick to interfacing C to C++ is to derive the C++ classes > from the numpy structures and keep the function pointers. Then all the > offsets would work. I would think the main advantage of C++ would be > in arithmetic operator overloading and using template classes while > carefully restricting such things to numbers. Mostly, I would use C++ > inline class methods as shorthand that would compile to what the C > approach would do. I suppose we could write a python preprocessor that > would do essentially the same thing; C++ started that way. Are you suggesting to write a compiler to be able to use operator overloading ? :) More seriously, the problem of C/C++ is not only at source level but also at binary level. David From charlesr.harris at gmail.com Tue Jan 8 22:55:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 20:55:46 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784428F.7070803@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008 8:42 PM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > On Jan 8, 2008 6:49 PM, Eric Firing > > wrote: > > > > Bill Baxter wrote: > > > On Jan 9, 2008 9:18 AM, Charles R Harris > > > > wrote: > > >> On Jan 8, 2008 5:01 PM, Bill Baxter < wbaxter at gmail.com > > > wrote: > > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > > > > > >> wrote: > > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com > > > wrote: > > >>>>> If you're really going to try to do it, Charles, there's an > > >>>>> implementation of float16 in the OpenEXR toolkit. > > >>>>> http://www.openexr.com/ > > >>>>> > > >>>>> Or more precisely it's in the files in the Half/ directory > > of this: > > >>>>> > > >> > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > >>>>> I don't know if it's IEEE conformant or not (especially > > w.r.t. NaN's > > >>>>> and such) but it should be a good start. The code seems to > > be well > > >>>>> documented. > > >>>> The license looks good, essentially BSD. The code is all C++, > > which is > > >> the > > >>>> obvious way to go for this sort of thing, and I would like to > > stick with > > >> it, > > >>>> but that could lead to build/compatibility problems. I think > > NumPy > > >> itself > > >>>> should really be in C++. Maybe scons can help with the build. > > >>> Yeh, I was just thinking you'd rip out and C-ify the main > > algorithms > > >>> rather than trying to wrap it as-is. > > >> I'd rather not C-ify the thing, I'd rather C++-ify parts of > > NumPy. I note > > >> that MatPlotLab uses C++, so some of the problems must have > > been solved. > > > > A big chunk of C++ in matplotlib has just been replaced, largely > > because > > it was so hard to understand and extend for everyone but its author. > > There is still C++ code as well as C code in mpl. Personally, I > > prefer > > the C. > > > > > > > > If you think that's easier then go for it. > > > > The opinion that C++ would improve numpy is not universal, and has > > been > > discussed. > > > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > > > > > > > > Ah, the trick to interfacing C to C++ is to derive the C++ classes > > from the numpy structures and keep the function pointers. Then all the > > offsets would work. I would think the main advantage of C++ would be > > in arithmetic operator overloading and using template classes while > > carefully restricting such things to numbers. Mostly, I would use C++ > > inline class methods as shorthand that would compile to what the C > > approach would do. I suppose we could write a python preprocessor that > > would do essentially the same thing; C++ started that way. > Are you suggesting to write a compiler to be able to use operator > overloading ? :) More seriously, the problem of C/C++ is not only at > source level but also at binary level. > As I tried to say, there are ways of dealing with the compatibility problems at the binary level. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 8 22:55:30 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 12:55:30 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> Message-ID: <478445B2.5000403@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 8, 2008 8:42 PM, David Cournapeau > wrote: > > Charles R Harris wrote: > > > > > > On Jan 8, 2008 6:49 PM, Eric Firing > > >> wrote: > > > > Bill Baxter wrote: > > > On Jan 9, 2008 9:18 AM, Charles R Harris > > > >> wrote: > > >> On Jan 8, 2008 5:01 PM, Bill Baxter < wbaxter at gmail.com > > > >> wrote: > > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > > < charlesr.harris at gmail.com > > >> > > >> wrote: > > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com > > > >> wrote: > > >>>>> If you're really going to try to do it, Charles, > there's an > > >>>>> implementation of float16 in the OpenEXR toolkit. > > >>>>> http://www.openexr.com/ > > >>>>> > > >>>>> Or more precisely it's in the files in the Half/ directory > > of this: > > >>>>> > > >> > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > >>>>> I don't know if it's IEEE conformant or not (especially > > w.r.t. NaN's > > >>>>> and such) but it should be a good start. The code > seems to > > be well > > >>>>> documented. > > >>>> The license looks good, essentially BSD. The code is > all C++, > > which is > > >> the > > >>>> obvious way to go for this sort of thing, and I would > like to > > stick with > > >> it, > > >>>> but that could lead to build/compatibility problems. I > think > > NumPy > > >> itself > > >>>> should really be in C++. Maybe scons can help with the > build. > > >>> Yeh, I was just thinking you'd rip out and C-ify the main > > algorithms > > >>> rather than trying to wrap it as-is. > > >> I'd rather not C-ify the thing, I'd rather C++-ify parts of > > NumPy. I note > > >> that MatPlotLab uses C++, so some of the problems must have > > been solved. > > > > A big chunk of C++ in matplotlib has just been replaced, largely > > because > > it was so hard to understand and extend for everyone but its > author. > > There is still C++ code as well as C code in mpl. Personally, I > > prefer > > the C. > > > > > > > > If you think that's easier then go for it. > > > > The opinion that C++ would improve numpy is not universal, > and has > > been > > discussed. > > > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > > < > http://article.gmane.org/gmane.comp.python.numeric.general/13244> > > > > > > Ah, the trick to interfacing C to C++ is to derive the C++ classes > > from the numpy structures and keep the function pointers. Then > all the > > offsets would work. I would think the main advantage of C++ would be > > in arithmetic operator overloading and using template classes while > > carefully restricting such things to numbers. Mostly, I would > use C++ > > inline class methods as shorthand that would compile to what the C > > approach would do. I suppose we could write a python > preprocessor that > > would do essentially the same thing; C++ started that way. > Are you suggesting to write a compiler to be able to use operator > overloading ? :) More seriously, the problem of C/C++ is not only at > source level but also at binary level. > > > As I tried to say, there are ways of dealing with the compatibility > problems at the binary level. Sorry, I was not precise enough: I talked at the binary level wrt compilers. C++ object code compiled by different compilers are not always compatible, without talking about known linking problems. I am afraid this would quickly become a headache to support, specially when you start using external C++ libraries (if the float16 library is BSD, the source code could be incorporated directly into numpy, though, but then compiling the code may be an issue wrt distutils: I don't know how flexible C++ support is in distutils). This is such a big issue (ABI issues) with C++ that some open source projects incorporate the libraries they used in their source tree. David From charlesr.harris at gmail.com Tue Jan 8 23:50:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 21:50:08 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478445B2.5000403@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008 8:55 PM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > On Jan 8, 2008 8:42 PM, David Cournapeau > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > On Jan 8, 2008 6:49 PM, Eric Firing > > > > >> wrote: > > > > > > Bill Baxter wrote: > > > > On Jan 9, 2008 9:18 AM, Charles R Harris > > > > > > > >> wrote: > > > >> On Jan 8, 2008 5:01 PM, Bill Baxter < wbaxter at gmail.com > > > > > >> wrote: > > > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > > > < charlesr.harris at gmail.com > > > > >>> > > > >> wrote: > > > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < wbaxter at gmail.com > > > > > >> wrote: > > > >>>>> If you're really going to try to do it, Charles, > > there's an > > > >>>>> implementation of float16 in the OpenEXR toolkit. > > > >>>>> http://www.openexr.com/ > > > >>>>> > > > >>>>> Or more precisely it's in the files in the Half/ > directory > > > of this: > > > >>>>> > > > >> > > > > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > > >>>>> I don't know if it's IEEE conformant or not (especially > > > w.r.t. NaN's > > > >>>>> and such) but it should be a good start. The code > > seems to > > > be well > > > >>>>> documented. > > > >>>> The license looks good, essentially BSD. The code is > > all C++, > > > which is > > > >> the > > > >>>> obvious way to go for this sort of thing, and I would > > like to > > > stick with > > > >> it, > > > >>>> but that could lead to build/compatibility problems. I > > think > > > NumPy > > > >> itself > > > >>>> should really be in C++. Maybe scons can help with the > > build. > > > >>> Yeh, I was just thinking you'd rip out and C-ify the main > > > algorithms > > > >>> rather than trying to wrap it as-is. > > > >> I'd rather not C-ify the thing, I'd rather C++-ify parts of > > > NumPy. I note > > > >> that MatPlotLab uses C++, so some of the problems must have > > > been solved. > > > > > > A big chunk of C++ in matplotlib has just been replaced, > largely > > > because > > > it was so hard to understand and extend for everyone but its > > author. > > > There is still C++ code as well as C code in mpl. Personally, > I > > > prefer > > > the C. > > > > > > > > > > > If you think that's easier then go for it. > > > > > > The opinion that C++ would improve numpy is not universal, > > and has > > > been > > > discussed. > > > > > > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > > > < > > http://article.gmane.org/gmane.comp.python.numeric.general/13244> > > > > > > > > > Ah, the trick to interfacing C to C++ is to derive the C++ classes > > > from the numpy structures and keep the function pointers. Then > > all the > > > offsets would work. I would think the main advantage of C++ would > be > > > in arithmetic operator overloading and using template classes > while > > > carefully restricting such things to numbers. Mostly, I would > > use C++ > > > inline class methods as shorthand that would compile to what the C > > > approach would do. I suppose we could write a python > > preprocessor that > > > would do essentially the same thing; C++ started that way. > > Are you suggesting to write a compiler to be able to use operator > > overloading ? :) More seriously, the problem of C/C++ is not only at > > source level but also at binary level. > > > > > > As I tried to say, there are ways of dealing with the compatibility > > problems at the binary level. > Sorry, I was not precise enough: I talked at the binary level wrt > compilers. C++ object code compiled by different compilers are not > always compatible, without talking about known linking problems. I am This can also be true of C code unless you use compilers in the same family. The C++ name mangling can be worked around. I'm just thinking out loud about these things. Note that numpy already uses what are essentially templates to generate C code using a Python preprocessor, although perhaps the design could be a bit cleaner and maybe a bit more general so as not to be limited to the C builtin types. What I want is a clean way to get around the limitations of the builtin C numeric types. The ability is there in the Python numeric type, which is function pointer based, so the rest just comes down to ease of coding and trying to make the result efficient. The last is where operator overloading and inline code might come in so as to avoid always calling through a pointer, but it might not be worth the hassle and the returns might be too small to take that last step. I see that there are already a number of parsers available for Python, SPARK, for instance is included in the 2.5.1 distribution. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Jan 8 23:59:46 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 08 Jan 2008 22:59:46 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: <4784293F.5090705@gmail.com> References: <4784293F.5090705@gmail.com> Message-ID: <478454C2.7030106@enthought.com> Robert Kern wrote: > Neal Becker wrote: > >> I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: >> >> var () -> (close to 1.0) >> var () -> (close to 1.0) >> >> but >> >> var (complex array) -> (close to complex 0) >> >> Is that not a strange definition? >> > > > > 2. Take a slightly less naive formula for the variance which seems to show up in > some texts: > > mean(absolute(z - mean(z)) ** 2) > > This estimates the single parameter of a circular Gaussian over RR^2 > (interpreted as CC). It is also the trace of the covariance matrix above. > I tend to favor this interpretation because it is used quite heavily in signal processing applications where "circular" Gaussian random variables show up quite a bit --- so much so, in fact, that most EE folks would expect this as the output and you would have to explain to them why there may be other choices that make sense. So, #2 is kind of a nod to the signal-processing community (especially the communication section). But, there is also merit to me in #3 (although it may be harder to explain why the variance returns a complex number --- if that is what you meant). -Travis O. > 3. Take the variances of the real and imaginary components independently. This > is equivalent to taking the diagonal of the covariance matrix above. This > wouldn't be the definition of "*the* complex variance" that anyone else uses, > but rather another form of punting. "There isn't a single complex variance to > give you, but in the spirit of broadcasting, we'll compute the marginal > variances of each dimension independently." > > Personally, I like 1 a lot. I'm hesitant to support 2 until I've seen an actual > application of that definition. The references I have been given in the ticket > comments are all early parts of books where the authors are laying out > definitions without applications. Personally, it feels to me like the authors > are just sticking in the absolute()'s ex post facto just so they can extend the > definition they already have to complex numbers. I'm also not a fan of the > expectation-centric treatments of random variables. IMO, the variance of an > arbitrary RV isn't an especially important quantity. It's a parameter of a > Gaussian distribution, and in this case, I see no reason to favor circular > Gaussians in CC over general ones. > > But if someone shows me an actual application of the definition, I can amend my > view. > > From robert.kern at gmail.com Wed Jan 9 00:00:42 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 23:00:42 -0600 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> Message-ID: <478454FA.9090701@gmail.com> Charles R Harris wrote: > I see that there are already a number of parsers available for Python, > SPARK, for instance is included in the 2.5.1 distribution. No, it isn't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Jan 9 00:02:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 23:02:05 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: References: <4784293F.5090705@gmail.com> <47843605.7090109@gmail.com> Message-ID: <4784554D.4050308@gmail.com> Charles R Harris wrote: > > > On Jan 8, 2008 7:48 PM, Robert Kern > wrote: > > Charles R Harris wrote: > > > Suppose you have a set of z_i and want to choose z to minimize the > > average square error $ \sum_i |z_i - z|^2 $. The solution is that > > $z=\mean{z_i}$ and the resulting average error is given by 2). > Note that > > I didn't mention Gaussians anywhere. No distribution is needed to > > justify the argument, just the idea of minimizing the squared > distance. > > Leaving out the ^2 would yield another metric, or one could ask > for a > > minmax solution. It is a question of the distance function, not > > probability. Anyway, that is one justification for the approach in 2) > > and it is one that makes a lot of applied math simple. Whether of > not a > > least squares fit is useful is different question. > > If you're not doing probability, then what are you using var() for? > I can accept > that the quantity is meaningful for your problem, but I'm not > convinced it's a > variance. > > > Lots of fits don't involve probability distributions. For instance, one > might want to fit a polynomial to a mathematical curve. This sort of > distinction between probability and distance goes back to Gauss himself, > although not in his original work on least squares. Whether or not > variance implies probability is a semantic question. Well, the problem in front of us is entirely semantics: What does the string "var(z)" mean? Are we going to choose an mechanistic definition: "var(z) is implemented in such and such a way and interpretations are left open"? In that case, why are we using the string "var(z)" rather than something else? We're also still left with the question as to which such and such implementation to use. Alternatively, we can look at what people call "variances" and try to implement the calculation of such. In that case, the term "variance" tends to crop up (and in my experience *only* crop up) in statistics and probability. Certain implementations of the calculations of such quantities have cognates elsewhere, but those cognates are not themselves called variances. My question to you is, is "the resulting average error" a variance? I.e., do people call it a variance outside of S&P? There are any number of computations that are useful but are not variances, and I don't think we should make "var(z)" implement them. In S&P, the single quantity "variance" is well defined for real RVs, even if you step away from Gaussians. It's the second central moment of the PDF of the RV. When you move up to CC (or RR^2), the definition of "moment" changes. It's no longer a real number or even a scalar; the second central moment is a covariance matrix. If we're going to call something "the variance", that's it. The circularly symmetric forms are special cases. Although option #2 is a useful quantity to calculate in some circumstances, I think it's bogus to give it a special status. > I think if we are > going to compute a single number, 2) is as good as anything even if it > doesn't capture the shape of the scatter plot. A 2D covariance wouldn't > necessarily capture the shape either. True, but it is clear exactly what it is. The function is named "cov()", and it computes covariances. It's not called "shape_of_2D_pdf()". Whether or not one ought to compute a covariance is not "cov()"'s problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Jan 9 00:09:37 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 22:09:37 -0700 Subject: [Numpy-discussion] numpy.linalg.eigvals crashes whn calling lapack_lite.pyd In-Reply-To: References: Message-ID: On Jan 8, 2008 6:49 PM, Simon wrote: > Newbie here. Trying to generate eigenvalues from a matrix using: > > print numpy.linalg.eigvals(matrix) > > This works with small matrices, say 5 x 5, but causes python to crash on > larger > matrices, say 136 x 136, which is not really very large. > > Setup: > > Win XP SP2 > Python 2.5.1 (from .msi) > numpy 1.0.4 (from .msi) > pywin32-210 (from .exe installer) > > > When running from either the command line or the Pythonwin IDE, python.exe > crashes. The info in the microsoft error reporting thingy is: > > AppName: python.exe > ModName: lapack_lite.pyd > Offset: 000b7434 > > Stepping through linalg.py using Pythonwin, I get as far as line 418 (in > the > eigvals function): > > results = lapack_routine('N', 'N', n, a, n, wr, wi, > dummy, 1, dummy, 1, work, lwork, 0) > > and then python.exe crashes. > > That's the extent of my troubleshooting skills at this stage. I haven't > worked > out if there is a specific matrix size where this starts occurring. Where > to now? > Probably just a different execution path depending on matrix size. But I am not that familiar with lapack_lite. > > I can send the actual data for the matrix if need be, but as it's very > large I > thought it would mess up the list if I posted it here. > This sounds like a compiler and/or architecture incompatibility since ATLAS doesn't seem to be part of the mix. What is your hardware? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jan 9 00:12:38 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jan 2008 23:12:38 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: <478454C2.7030106@enthought.com> References: <4784293F.5090705@gmail.com> <478454C2.7030106@enthought.com> Message-ID: <478457C6.8070802@gmail.com> Travis E. Oliphant wrote: > Robert Kern wrote: >> Neal Becker wrote: >> >>> I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: >>> >>> var () -> (close to 1.0) >>> var () -> (close to 1.0) >>> >>> but >>> >>> var (complex array) -> (close to complex 0) >>> >>> Is that not a strange definition? >>> >> >> >> 2. Take a slightly less naive formula for the variance which seems to show up in >> some texts: >> >> mean(absolute(z - mean(z)) ** 2) >> >> This estimates the single parameter of a circular Gaussian over RR^2 >> (interpreted as CC). It is also the trace of the covariance matrix above. >> > > I tend to favor this interpretation because it is used quite heavily in > signal processing applications where "circular" Gaussian random > variables show up quite a bit --- so much so, in fact, that most EE > folks would expect this as the output and you would have to explain to > them why there may be other choices that make sense. > > So, #2 is kind of a nod to the signal-processing community (especially > the communication section). Fair enough. I relent. You implement it; I'll document the correct^Wcov() alternative. :-) > But, there is also merit to me in #3 (although it may be harder to > explain why the variance returns a complex number --- if that is what > you meant). Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Jan 9 00:14:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Jan 2008 22:14:08 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478454FA.9090701@gmail.com> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <478454FA.9090701@gmail.com> Message-ID: On Jan 8, 2008 10:00 PM, Robert Kern wrote: > Charles R Harris wrote: > > > I see that there are already a number of parsers available for Python, > > SPARK, for instance is included in the 2.5.1 distribution. > > No, it isn't. Oops, so it isn't. Looks like this news item at the spark site is incorrect. 26 June 2007: John DeGood pointed out that SPARK is in the Python distribution now ? yay! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Wed Jan 9 01:23:20 2008 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 9 Jan 2008 08:23:20 +0200 Subject: [Numpy-discussion] parallel numpy - any info? In-Reply-To: <6.2.3.4.2.20080108070109.03542dc0@rjs.org> References: <6.2.3.4.2.20080108070109.03542dc0@rjs.org> Message-ID: <5eec5f300801082223s4d99482dvab0dddf0b6b072d8@mail.gmail.com> Hello On Jan 8, 2008 5:31 PM, Ray Schumacher wrote: > At 04:27 AM 1/8/2008, you wrote: > > 4. Re: parallel numpy (by Brian Granger) - any info? > > (Matthieu Brucher) > >From: "Matthieu Brucher" > > > >MKL does the multithreading on its own for level 3 BLAS instructions > >(OpenMP). > > There was brief debate yesterday among the Pythonians in the lab as > to whether the MKL operates outside of the GIL, but it was general > understanding that it does. > > It is still unclear to me whether Python/numpy compiled with MKL > would be freely re-distributable, as the MSVC version is. Read the License Agreement on Intel's site. My interpretation is that it would be redistributable. http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm In a related matter, Intel MKL on a MP Xeon with OMP_NUM_THREADS=8 flies on large matrices. Cheers, Albert From david at ar.media.kyoto-u.ac.jp Wed Jan 9 01:26:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 15:26:21 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> Message-ID: <4784690D.1080103@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 8, 2008 8:55 PM, David Cournapeau > wrote: > > Charles R Harris wrote: > > > > > > On Jan 8, 2008 8:42 PM, David Cournapeau > > > >> wrote: > > > > Charles R Harris wrote: > > > > > > > > > On Jan 8, 2008 6:49 PM, Eric Firing < efiring at hawaii.edu > > > > > > > > >>> wrote: > > > > > > Bill Baxter wrote: > > > > On Jan 9, 2008 9:18 AM, Charles R Harris > > > > > > > > > > >>> wrote: > > > >> On Jan 8, 2008 5:01 PM, Bill Baxter < > wbaxter at gmail.com > > > > > > > >>> wrote: > > > >>> On Jan 9, 2008 8:03 AM, Charles R Harris > > > < charlesr.harris at gmail.com > > > > > > > >>> > > > >> wrote: > > > >>>> On Jan 8, 2008 1:58 PM, Bill Baxter < > wbaxter at gmail.com > > > > > > > >>> wrote: > > > >>>>> If you're really going to try to do it, Charles, > > there's an > > > >>>>> implementation of float16 in the OpenEXR toolkit. > > > >>>>> http://www.openexr.com/ > > > >>>>> > > > >>>>> Or more precisely it's in the files in the Half/ > directory > > > of this: > > > >>>>> > > > >> > > > > > > http://download.savannah.nongnu.org/releases/openexr/ilmbase-1.0.1.tar.gz > > > >>>>> I don't know if it's IEEE conformant or not > (especially > > > w.r.t. NaN's > > > >>>>> and such) but it should be a good start. The code > > seems to > > > be well > > > >>>>> documented. > > > >>>> The license looks good, essentially BSD. The code is > > all C++, > > > which is > > > >> the > > > >>>> obvious way to go for this sort of thing, and I > would > > like to > > > stick with > > > >> it, > > > >>>> but that could lead to build/compatibility > problems. I > > think > > > NumPy > > > >> itself > > > >>>> should really be in C++. Maybe scons can help > with the > > build. > > > >>> Yeh, I was just thinking you'd rip out and C-ify > the main > > > algorithms > > > >>> rather than trying to wrap it as-is. > > > >> I'd rather not C-ify the thing, I'd rather C++-ify > parts of > > > NumPy. I note > > > >> that MatPlotLab uses C++, so some of the problems > must have > > > been solved. > > > > > > A big chunk of C++ in matplotlib has just been > replaced, largely > > > because > > > it was so hard to understand and extend for everyone > but its > > author. > > > There is still C++ code as well as C code in mpl. > Personally, I > > > prefer > > > the C. > > > > > > > > > > > If you think that's easier then go for it. > > > > > > The opinion that C++ would improve numpy is not > universal, > > and has > > > been > > > discussed. > > > > > > > http://article.gmane.org/gmane.comp.python.numeric.general/13244 > > > > < > > > http://article.gmane.org/gmane.comp.python.numeric.general/13244> > > > > > > > > > Ah, the trick to interfacing C to C++ is to derive the C++ > classes > > > from the numpy structures and keep the function pointers. Then > > all the > > > offsets would work. I would think the main advantage of > C++ would be > > > in arithmetic operator overloading and using template > classes while > > > carefully restricting such things to numbers. Mostly, I would > > use C++ > > > inline class methods as shorthand that would compile to > what the C > > > approach would do. I suppose we could write a python > > preprocessor that > > > would do essentially the same thing; C++ started that way. > > Are you suggesting to write a compiler to be able to use > operator > > overloading ? :) More seriously, the problem of C/C++ is not > only at > > source level but also at binary level. > > > > > > As I tried to say, there are ways of dealing with the > compatibility > > problems at the binary level. > Sorry, I was not precise enough: I talked at the binary level wrt > compilers. C++ object code compiled by different compilers are not > always compatible, without talking about known linking problems. I am > > > This can also be true of C code unless you use compilers in the same > family. There are also issues, but they are much simpler. > The C++ name mangling can be worked around. name mangling is just the top of the iceberg. There are problems wrt to static initialization, exception, etc...; ABI compatibility is much easier to break in C++ than in C. And then, there is the problem of C++ ability of compilers. http://developer.mozilla.org/en/docs/C%2B%2B_Portability_Guide http://techbase.kde.org/index.php?title=Policies/Binary_Compatibility_Issues_With_C%2B%2B I am not saying it is not possible, or that we should not do it: just that it has a high cost. IMHO, the benefits would have to be high. > I'm just thinking out loud about these things. Note that numpy already > uses what are essentially templates to generate C code using a Python > preprocessor, although perhaps the design could be a bit cleaner and > maybe a bit more general so as not to be limited to the C builtin types. I agree on this point (custom code generators use). Note that there are standard tools to generate simple templates in C (autogen). If what you want to do is to be able to use simple templates (by simple, I mean template function parametrized with the numerical type), C++ is overkill IMHO. cheers, David From matthieu.brucher at gmail.com Wed Jan 9 01:50:19 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 9 Jan 2008 07:50:19 +0100 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784690D.1080103@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> Message-ID: > > > This can also be true of C code unless you use compilers in the same > > family. > There are also issues, but they are much simpler. > > The C++ name mangling can be worked around. > name mangling is just the top of the iceberg. There are problems wrt to > static initialization, exception, etc...; ABI compatibility is much > easier to break in C++ than in C. And then, there is the problem of C++ > ability of compilers. This is no longer the case, sincerely. I use C++ compilers from different vendors for some time, and I had almost no problem safe from some template depth issues. http://developer.mozilla.org/en/docs/C%2B%2B_Portability_Guide This is... well... outdated. I'm sorry, but the issues mentionned there are pre- 2003 standard and since then, the C++ compilers follow the standard. http://techbase.kde.org/index.php?title=Policies/Binary_Compatibility_Issues_With_C%2B%2B KDE wants to maintain binary compatibility across versions. I don't know if Numpy wants to stay compatible this way, but even there, I don't think Numpy will use many virtual functions (too slow), and for the change of the signature of a function, it is logical that it should not work. I am not saying it is not possible, or that we should not do it: just > that it has a high cost. IMHO, the benefits would have to be high. I agree. > I'm just thinking out loud about these things. Note that numpy already > > uses what are essentially templates to generate C code using a Python > > preprocessor, although perhaps the design could be a bit cleaner and > > maybe a bit more general so as not to be limited to the C builtin types. > I agree on this point (custom code generators use). Note that there are > standard tools to generate simple templates in C (autogen). > > If what you want to do is to be able to use simple templates (by simple, > I mean template function parametrized with the numerical type), C++ is > overkill IMHO. There are some other stuff that are interesting, but I don't think it would be easy to use, like expression templates, but some operator overloading are very simple to use (like specializing a template multplication to use CBLAS if available). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Jan 9 02:11:51 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 16:11:51 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> Message-ID: <478473B7.4080807@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > > This can also be true of C code unless you use compilers in the same > > family. > There are also issues, but they are much simpler. > > The C++ name mangling can be worked around. > name mangling is just the top of the iceberg. There are problems > wrt to > static initialization, exception, etc...; ABI compatibility is much > easier to break in C++ than in C. And then, there is the problem > of C++ > ability of compilers. > > > > This is no longer the case, sincerely. I use C++ compilers from > different vendors for some time, and I had almost no problem safe from > some template depth issues. C++ ability is not so much a problem with recent compilers, I agree. But not all platforms are or can use a recent C++ compiler. In particular, proprietary UNIX. Now, maybe that's something we do not want to support. If we limit ourselves to gcc 4 and recent intel/MS compilers, then this issue indeed is moot, at least for basic things. cheers, David From simonpy2008 at gmail.com Wed Jan 9 02:24:50 2008 From: simonpy2008 at gmail.com (Simon) Date: Wed, 9 Jan 2008 07:24:50 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy=2Elinalg=2Eeigvals_crashes_whn?= =?utf-8?q?_calling=09lapack=5Flite=2Epyd?= References: Message-ID: Charles R Harris gmail.com> writes: > > > On Jan 8, 2008 6:49 PM, Simon gmail.com> wrote: > Newbie here. Trying to generate eigenvalues from a matrix using:print numpy.linalg.eigvals(matrix)This works with small matrices, say 5 x 5, but causes python to crash on largermatrices, say 136 x 136, which is not really very large. > Setup:Win XP SP2Python 2.5.1 (from .msi)numpy 1.0.4 (from .msi)pywin32-210 (from .exe installer)When running from either the command line or the Pythonwin IDE, python.execrashes. The info in the microsoft error reporting thingy is: > AppName: python.exeModName: lapack_lite.pydOffset: 000b7434Stepping through linalg.py using Pythonwin, I get as far as line 418 (in theeigvals function): results = lapack_routine('N', 'N', n, a, n, wr, wi, > dummy, 1, dummy, 1, work, lwork, 0)and then python.exe crashes.That's the extent of my troubleshooting skills at this stage. I haven't workedout if there is a specific matrix size where this starts occurring. Where to now? > > > > Probably just a different execution path depending on matrix size. But I am not that familiar with lapack_lite. > > > I can send the actual data for the matrix if need be, but as it's very large Ithought it would mess up the list if I posted it here. > > > This sounds like a compiler and/or architecture incompatibility since ATLAS doesn't seem to be part of the mix. What is your hardware? > Chuck > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Xeon 5150 in Dell 490. I've also tried it on a couple of other boxes with different hardware since my first post, but don't have the details in front of me. Might try a compile from source on a unix box tomorrow, unless someone comes up with a solution before then. Simon Simon From matthieu.brucher at gmail.com Wed Jan 9 02:27:44 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 9 Jan 2008 08:27:44 +0100 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478473B7.4080807@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> <478473B7.4080807@ar.media.kyoto-u.ac.jp> Message-ID: > > > This is no longer the case, sincerely. I use C++ compilers from > > different vendors for some time, and I had almost no problem safe from > > some template depth issues. > C++ ability is not so much a problem with recent compilers, I agree. But > not all platforms are or can use a recent C++ compiler. In particular, > proprietary UNIX. Now, maybe that's something we do not want to support. > If we limit ourselves to gcc 4 and recent intel/MS compilers, then this > issue indeed is moot, at least for basic things. Well, even for basic stuff, I have to say that GCC 4.0 can mysteriously crash (the SFINAE principle is not respected, 4.1 is fine in that matter), so using C++ is not worth the trouble now (and besides, I don't know what we could use right now that cannot be done "easily" in C) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Jan 9 02:50:35 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 16:50:35 +0900 Subject: [Numpy-discussion] def of var of complex In-Reply-To: <478457C6.8070802@gmail.com> References: <4784293F.5090705@gmail.com> <478454C2.7030106@enthought.com> <478457C6.8070802@gmail.com> Message-ID: <47847CCB.4080705@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > Travis E. Oliphant wrote: > >> Robert Kern wrote: >> >>> Neal Becker wrote: >>> >>> >>>> I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: >>>> >>>> var () -> (close to 1.0) >>>> var () -> (close to 1.0) >>>> >>>> but >>>> >>>> var (complex array) -> (close to complex 0) >>>> >>>> Is that not a strange definition? >>>> >>>> >>> 2. Take a slightly less naive formula for the variance which seems to show up in >>> some texts: >>> >>> mean(absolute(z - mean(z)) ** 2) >>> >>> This estimates the single parameter of a circular Gaussian over RR^2 >>> (interpreted as CC). It is also the trace of the covariance matrix above. >>> >>> >> I tend to favor this interpretation because it is used quite heavily in >> signal processing applications where "circular" Gaussian random >> variables show up quite a bit --- so much so, in fact, that most EE >> folks would expect this as the output and you would have to explain to >> them why there may be other choices that make sense. >> >> So, #2 is kind of a nod to the signal-processing community (especially >> the communication section). >> > > Fair enough. I relent. You implement it; I'll document the correct^Wcov() > alternative. :-) > > Not that I find the argument pertinent most of the time, but if there is no clear argument in favor of one formula, would following matlab conventions be ok ? To me, the definition 2 makes more sense, as a perticular case of the correlation between two different complex random variables: \mathbb{E}[X \bar{Y}], such as it keeps the nice properties of scalar product. cheers, David From charlesr.harris at gmail.com Wed Jan 9 06:00:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Jan 2008 04:00:56 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784690D.1080103@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008 11:26 PM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > > > The C++ name mangling can be worked around. > name mangling is just the top of the iceberg. There are problems wrt to > static initialization, exception, etc...; ABI compatibility is much > easier to break in C++ than in C. And then, there is the problem of C++ > ability of compilers. One would naturally have to be very careful about which features to use. In particular, don't use anything that throws, and I suppose one would want to avoid having to link to the stdc++ library also. I don't know if that is possible. > I agree on this point (custom code generators use). Note that there are > standard tools to generate simple templates in C (autogen). Unfortunately, autogen is GPL and I suspect NumPy would be regarded as a derived work. Are you familiar with any other standard, maintained program of that sort with a BSD style license? > If what you want to do is to be able to use simple templates (by simple, > I mean template function parametrized with the numerical type), C++ is > overkill IMHO. > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 9 06:08:42 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Jan 2008 04:08:42 -0700 Subject: [Numpy-discussion] numpy.linalg.eigvals crashes whn calling lapack_lite.pyd In-Reply-To: References: Message-ID: On Jan 9, 2008 12:24 AM, Simon wrote: > Charles R Harris gmail.com> writes: > > > > > > > On Jan 8, 2008 6:49 PM, Simon gmail.com> wrote: > > Newbie here. Trying to generate eigenvalues from a matrix using:print > numpy.linalg.eigvals(matrix)This works with small matrices, say 5 x 5, but > causes python to crash on largermatrices, say 136 x 136, which is not > really > very large. > > Setup:Win XP SP2Python 2.5.1 (from .msi)numpy 1.0.4 (from > .msi)pywin32-210 > (from .exe installer)When running from either the command line or the > Pythonwin > IDE, python.execrashes. The info in the microsoft error reporting thingy > is: > > AppName: python.exeModName: lapack_lite.pydOffset: 000b7434Stepping > through > linalg.py using Pythonwin, I get as far as line 418 (in theeigvals > function): > results = lapack_routine('N', 'N', n, a, n, wr, wi, > > dummy, 1, dummy, 1, work, lwork, 0)and > then > python.exe crashes.That's the extent of my troubleshooting skills at this > stage. > I haven't workedout if there is a specific matrix size where this starts > occurring. Where to now? > > > > > > > > Probably just a different execution path depending on matrix size. But I > am > not that familiar with lapack_lite. > > > > > > I can send the actual data for the matrix if need be, but as it's very > large > Ithought it would mess up the list if I posted it here. > > > > > > This sounds like a compiler and/or architecture incompatibility since > ATLAS > doesn't seem to be part of the mix. What is your hardware? > > > Xeon 5150 in Dell 490. > > I've also tried it on a couple of other boxes with different hardware > since my > first post, but don't have the details in front of me. > > Might try a compile from source on a unix box tomorrow, unless someone > comes up > with a solution before then. > Which specific version of numpy for 2.5 did you use? Can you try one of the other versions on the sourceforge site? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Jan 9 07:43:15 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 09 Jan 2008 21:43:15 +0900 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> Message-ID: <4784C163.9060201@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 8, 2008 11:26 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > > > > > > > > > The C++ name mangling can be worked around. > name mangling is just the top of the iceberg. There are problems > wrt to > static initialization, exception, etc...; ABI compatibility is much > easier to break in C++ than in C. And then, there is the problem > of C++ > ability of compilers. > > > One would naturally have to be very careful about which features to > use. In particular, don't use anything that throws, and I suppose one > would want to avoid having to link to the stdc++ library also. I don't > know if that is possible. Avoiding linking to the stdc++ would be quite difficult, I think (from the build process, at least). Certainly doable, but difficult. > > > > > I agree on this point (custom code generators use). Note that > there are > standard tools to generate simple templates in C (autogen). > > > Unfortunately, autogen is GPL and I suspect NumPy would be regarded as > a derived work. Are you familiar with any other standard, maintained > program of that sort with a BSD style license? I don't think generated source files should be considered as GPL (otherwise, you could not build non GPL source with gcc either: I don't see where the difference would be between object code generated by the compiler and generated source by autogen). If you read the point 0 of the GPL, it looks quite clear to me: "Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does." Once you get the generated source file, you do not need autogen at all anymore, which is a pretty strong hint that it is not derived work, IMHO. Of course, IANAL, blah blah blah. cheers, David From matthieu.brucher at gmail.com Wed Jan 9 08:00:03 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 9 Jan 2008 14:00:03 +0100 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784C163.9060201@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> <4784C163.9060201@ar.media.kyoto-u.ac.jp> Message-ID: > > > One would naturally have to be very careful about which features to > > use. In particular, don't use anything that throws, and I suppose one > > would want to avoid having to link to the stdc++ library also. I don't > > know if that is possible. > Avoiding linking to the stdc++ would be quite difficult, I think (from > the build process, at least). Certainly doable, but difficult. > Why would you want to avoid linking to the stdc++ ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 9 08:38:19 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Jan 2008 06:38:19 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: References: <200801071500.21275.darren.dale@cornell.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> <4784C163.9060201@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 9, 2008 6:00 AM, Matthieu Brucher wrote: > > One would naturally have to be very careful about which features to > > > use. In particular, don't use anything that throws, and I suppose one > > > would want to avoid having to link to the stdc++ library also. I don't > > > know if that is possible. > > Avoiding linking to the stdc++ would be quite difficult, I think (from > > the build process, at least). Certainly doable, but difficult. > > > > Why would you want to avoid linking to the stdc++ ? > Because I was sleep deprived. That's my story and I'm sticking to it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Wed Jan 9 09:41:24 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 9 Jan 2008 09:41:24 -0500 Subject: [Numpy-discussion] Failing to understand vectorize behavior Message-ID: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> Hi all, I'm having trouble understanding the behavior of vectorize on the following example: >>> import string >>> from numpy import vectorize >>> vstrip = vectorize(string.strip) >>> s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' cccccccccccccccccc '] >>> vstrip(s) array(['aaaaaaaa', 'bbbbbbbb', 'cccccccc'], dtype='|S8') where all strings are cropped to 8 characters. Is this expected ? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Wed Jan 9 09:52:07 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 9 Jan 2008 15:52:07 +0100 Subject: [Numpy-discussion] Failing to understand vectorize behavior In-Reply-To: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> References: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> Message-ID: <80c99e790801090652u56eaba65kb400bf4c614f979d@mail.gmail.com> I don't think it's expected: mine are cropped to 4 characters! In [101]: vstrip = vectorize( string.strip) In [102]: s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' cccccccccccccccccc '] In [103]: vstrip(s) Out[103]: array(['aaaa', 'bbbb', 'cccc'], dtype='|S4') You can obviously use "map", instead. In [104]: map(string.strip, s) Out[104]: ['aaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbb', 'cccccccccccccccccc'] hth, L. On 1/9/08, David Huard wrote: > > Hi all, > > I'm having trouble understanding the behavior of vectorize on the > following example: > > >>> import string > >>> from numpy import vectorize > > >>> vstrip = vectorize( string.strip) > >>> s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > cccccccccccccccccc '] > >>> vstrip(s) > array(['aaaaaaaa', 'bbbbbbbb', 'cccccccc'], > dtype='|S8') > > where all strings are cropped to 8 characters. > Is this expected ? > > > Thanks, > > David > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Wed Jan 9 10:13:50 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 9 Jan 2008 10:13:50 -0500 Subject: [Numpy-discussion] Failing to understand vectorize behavior In-Reply-To: <80c99e790801090652u56eaba65kb400bf4c614f979d@mail.gmail.com> References: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> <80c99e790801090652u56eaba65kb400bf4c614f979d@mail.gmail.com> Message-ID: <91cf711d0801090713x41144c25y3ec9bbcf29c976f7@mail.gmail.com> Lorenzo, 2008/1/9, lorenzo bolla : > > I don't think it's expected: mine are cropped to 4 characters! > I am on a 64 bit machine. Are you on a 32 bit one ? In [101]: vstrip = vectorize( string.strip) > In [102]: s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > cccccccccccccccccc '] > In [103]: > vstrip(s) > > Out[103]: > > array(['aaaa', 'bbbb', > 'cccc'], > > dtype='|S4') > > > You can obviously use "map", instead. > I'll do that, Thanks, David In [104]: map(string.strip, s) > Out[104]: ['aaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbb', > 'cccccccccccccccccc'] > > > hth, > L. > > > On 1/9/08, David Huard wrote: > > > Hi all, > > > > I'm having trouble understanding the behavior of vectorize on the > > following example: > > > > >>> import string > > >>> from numpy import vectorize > > > > >>> vstrip = vectorize( string.strip) > > >>> s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > > cccccccccccccccccc '] > > >>> vstrip(s) > > array(['aaaaaaaa', 'bbbbbbbb', 'cccccccc'], > > dtype='|S8') > > > > where all strings are cropped to 8 characters. > > Is this expected ? > > > > > > Thanks, > > > > David > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From subscriber100 at rjs.org Wed Jan 9 10:23:25 2008 From: subscriber100 at rjs.org (Ray Schumacher) Date: Wed, 09 Jan 2008 07:23:25 -0800 Subject: [Numpy-discussion] Intel MKL - was: parallel numpy - any info? In-Reply-To: References: Message-ID: <6.2.3.4.2.20080109070201.041afac0@rjs.org> At 10:38 PM 1/8/2008, Albert Strasheim wrote: > > It is still unclear to me whether Python/numpy compiled with MKL > > would be freely re-distributable, as the MSVC version is. > >Read the License Agreement on Intel's site. My interpretation is that >it would be redistributable. > >http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm That was my thought also, but IANAL. Given the number of people who've evidently compiledPython/numeric with it I was left wondering why there are no binaries to be found. We've been using the Enthought distros, and our company now has (finally) decided to purchase the ICC and MKL with the intention of compiling for P4 and Core2 targets. The binaries could then go up on the company's web site. Thanks, Ray Schumacher -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.13/1214 - Release Date: 1/8/2008 1:38 PM From niki.spahiev at gmail.com Mon Jan 7 07:24:59 2008 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Mon, 07 Jan 2008 14:24:59 +0200 Subject: [Numpy-discussion] Initial Hg experiences on Windows In-Reply-To: <5b8d13220801060851m2e192415y93ad56789f286c79@mail.gmail.com> References: <5b8d13220801060851m2e192415y93ad56789f286c79@mail.gmail.com> Message-ID: <47821A1B.8070801@gmail.com> David Cournapeau wrote: > > I think that at least wrt windows support, bzr is today better than > hg, if only for the fact that some bzr developers are primely windows > developers, with the "windows culture". bzr GUI - olive works great on windows Niki Spahiev From cookedm at physics.mcmaster.ca Wed Jan 9 11:23:10 2008 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 9 Jan 2008 11:23:10 -0500 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <478454FA.9090701@gmail.com> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <478454FA.9090701@gmail.com> Message-ID: <42FE20E2-1D99-41EC-9A88-82BF08765E00@physics.mcmaster.ca> On Jan 9, 2008, at 00:00 , Robert Kern wrote: > Charles R Harris wrote: > >> I see that there are already a number of parsers available for >> Python, >> SPARK, for instance is included in the 2.5.1 distribution. > > No, it isn't. It's used to generate the new parser in 2.5 (Parser/spark.py). It's just not included when installed :) -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From lbolla at gmail.com Wed Jan 9 11:27:24 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 9 Jan 2008 17:27:24 +0100 Subject: [Numpy-discussion] Failing to understand vectorize behavior In-Reply-To: <91cf711d0801090713x41144c25y3ec9bbcf29c976f7@mail.gmail.com> References: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> <80c99e790801090652u56eaba65kb400bf4c614f979d@mail.gmail.com> <91cf711d0801090713x41144c25y3ec9bbcf29c976f7@mail.gmail.com> Message-ID: <80c99e790801090827v54583a7dxf297f4f14cb9fb02@mail.gmail.com> Yes, 32 bits. On a 64 bits machine, I get 8 characters long strings like you. L. On 1/9/08, David Huard wrote: > > Lorenzo, > > 2008/1/9, lorenzo bolla : > > > > I don't think it's expected: mine are cropped to 4 characters! > > > > > I am on a 64 bit machine. Are you on a 32 bit one ? > > In [101]: vstrip = vectorize( string.strip) > > > > In [102]: s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > > cccccccccccccccccc '] > > In [103]: > > vstrip(s) > > > > Out[103]: > > > > array(['aaaa', 'bbbb', > > 'cccc'], > > > > dtype='|S4') > > > > > > You can obviously use "map", instead. > > > > I'll do that, > > Thanks, > > > David > > In [104]: map(string.strip, > > s) > > Out[104]: ['aaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbb', > > 'cccccccccccccccccc'] > > > > > > hth, > > L. > > > > > > On 1/9/08, David Huard < david.huard at gmail.com> wrote: > > > > > Hi all, > > > > > > I'm having trouble understanding the behavior of vectorize on the > > > following example: > > > > > > >>> import string > > > >>> from numpy import vectorize > > > > > > >>> vstrip = vectorize( string.strip) > > > >>> s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > > > cccccccccccccccccc '] > > > >>> vstrip(s) > > > array(['aaaaaaaa', 'bbbbbbbb', 'cccccccc'], > > > dtype='|S8') > > > > > > where all strings are cropped to 8 characters. > > > Is this expected ? > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > -- > > Lorenzo Bolla > > lbolla at gmail.com > > http://lorenzobolla.emurse.com/ > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at enthought.com Wed Jan 9 11:32:07 2008 From: bryanv at enthought.com (Bryan Van de Ven) Date: Wed, 09 Jan 2008 10:32:07 -0600 Subject: [Numpy-discussion] Failing to understand vectorize behavior In-Reply-To: <80c99e790801090827v54583a7dxf297f4f14cb9fb02@mail.gmail.com> References: <91cf711d0801090641t3b115a8ic6a0dc81072291ab@mail.gmail.com> <80c99e790801090652u56eaba65kb400bf4c614f979d@mail.gmail.com> <91cf711d0801090713x41144c25y3ec9bbcf29c976f7@mail.gmail.com> <80c99e790801090827v54583a7dxf297f4f14cb9fb02@mail.gmail.com> Message-ID: <4784F707.5010406@enthought.com> otypes needs to be set for this to work. In [19]: vstrip = vectorize( string.strip, otypes=[object]) In [20]: s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' cccccccccccccccccc '] In [21]: vstrip(s) Out[21]: array([aaaaaaaaaaaaaa, bbbbbbbbbbbbbbbbbb, cccccccccccccccccc], dtype=object) lorenzo bolla wrote: > Yes, 32 bits. > On a 64 bits machine, I get 8 characters long strings like you. > L. > > > On 1/9/08, *David Huard* > wrote: > > Lorenzo, > > 2008/1/9, lorenzo bolla >: > > I don't think it's expected: mine are cropped to 4 characters! > > > > I am on a 64 bit machine. Are you on a 32 bit one ? > > In [101]: vstrip = vectorize( > string.strip) > > > In [102]: s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', > ' cccccccccccccccccc '] > In [103]: > vstrip(s) > > Out[103]: > > array(['aaaa', 'bbbb', > 'cccc'], > > dtype='|S4') > > > You can obviously use "map", instead. > > > I'll do that, > > Thanks, > > > David > > In [104]: map(string.strip, > s) > Out[104]: ['aaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbb', > 'cccccccccccccccccc'] > > > hth, > L. > > > On 1/9/08, *David Huard* < david.huard at gmail.com > > wrote: > > Hi all, > > I'm having trouble understanding the behavior of vectorize > on the following example: > > > >> import string > > >> from numpy import vectorize > > > >> vstrip = vectorize( string.strip) > > >> s = [' aaaaaaaaaaaaaa ' , ' bbbbbbbbbbbbbbbbbb ', ' > cccccccccccccccccc '] > > >> vstrip(s) > array(['aaaaaaaa', 'bbbbbbbb', 'cccccccc'], > dtype='|S8') > > where all strings are cropped to 8 characters. > Is this expected ? > > > Thanks, > > David > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From simonpy2008 at gmail.com Wed Jan 9 16:52:40 2008 From: simonpy2008 at gmail.com (Simon) Date: Wed, 9 Jan 2008 21:52:40 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy=2Elinalg=2Eeigvals_crashes_whn?= =?utf-8?q?_calling=09lapack=5Flite=2Epyd?= References: Message-ID: Charles R Harris gmail.com> writes: [snip] > > Which specific version of numpy for 2.5 did you use? Can you try one of the other versions on the sourceforge site?Chuck > > I have just tried: numpy-1.0.3.1.win32-py2.5.exe and it works! I was using: numpy-1.0.4.win32-py2.5.msi Have no idea what a .egg file is, and I have no access to a compiler on windows, so can't use the source for 1.0.4 (and don't want to go down the cygwin pathway again). However, last night I got hold of a mac and compiled numpy 1.0.4 from source. My code works fine with: OS X 10.4.11 Python 2.4.3 numpy 1.0.4 although this isn't python 2.5, there's clearly something up with my Windows install of numpy 1.0.4. I'll stick with numpy 1.0.3.1 for now. If anybody wants to chase down the problem with 1.0.4, let me know and I'll provide what assistance that I can. Simon From charlesr.harris at gmail.com Wed Jan 9 19:30:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Jan 2008 17:30:08 -0700 Subject: [Numpy-discussion] Does float16 exist? In-Reply-To: <4784690D.1080103@ar.media.kyoto-u.ac.jp> References: <200801071500.21275.darren.dale@cornell.edu> <4784281B.3060900@hawaii.edu> <4784428F.7070803@ar.media.kyoto-u.ac.jp> <478445B2.5000403@ar.media.kyoto-u.ac.jp> <4784690D.1080103@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 8, 2008 11:26 PM, David Cournapeau wrote: If what you want to do is to be able to use simple templates (by simple, > I mean template function parametrized with the numerical type), C++ is > overkill IMHO. > So I pulled the template subsystem out of Django and I already like it a lot. It seems to work well with some quick C code templates that I wrote and the markup looks decent. The template language itself has some built in control tags (if, for loops, etc), templates have inheritance, and you can invent your own tags. It is all done in Python and the license is essentially BSD. Note that the template examples on the Django site generate html code, so all those html tags are just a distraction. There is also a Django book , so it is documented. Chuck > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Jan 9 23:56:28 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 10 Jan 2008 13:56:28 +0900 Subject: [Numpy-discussion] Intel MKL - was: parallel numpy - any info? In-Reply-To: <6.2.3.4.2.20080109070201.041afac0@rjs.org> References: <6.2.3.4.2.20080109070201.041afac0@rjs.org> Message-ID: <4785A57C.80801@ar.media.kyoto-u.ac.jp> Ray Schumacher wrote: > At 10:38 PM 1/8/2008, Albert Strasheim wrote: > > >>> It is still unclear to me whether Python/numpy compiled with MKL >>> would be freely re-distributable, as the MSVC version is. >>> >> Read the License Agreement on Intel's site. My interpretation is that >> it would be redistributable. >> >> http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm >> > > That was my thought also, but IANAL. Given the number of people > who've evidently compiledPython/numeric with it I was left wondering > why there are no binaries to be found. > We've been using the Enthought distros, and our company now has > (finally) decided to purchase the ICC and MKL with the intention of > compiling for P4 and Core2 targets. The binaries could then go up on > the company's web site. > This issue was already discussed a few weeks ago (December 2007), and the last word was although MKL would not be used for the default binary, a link to a binary built by someone else could be added to the scipy download page. Of course, you better have to make sure the license agreement does permit what you think it says, since your company would bear the responsibility for it as far as I understand it. The one thing which I am not sure about is: say one MKL binary does not work, and say I (or anyone outside your company) build numpy with the MKL ro debug it, can I redistribute a new binary, even if it is just for testing purpose ? cheers, David From david at ar.media.kyoto-u.ac.jp Thu Jan 10 00:55:58 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 10 Jan 2008 14:55:58 +0900 Subject: [Numpy-discussion] BzrMirror page on numpy wiki Message-ID: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> Hi, I started a page on how to use bzr, with a link to an import I have done on the numpy trunk: http://scipy.org/scipy/numpy/wiki/BzrMirror I thought it would be useful for people not familiar with DVCS, and are curious to try. In perticular, since I have done the svn->bzr import, the distributed bzr binary will work out of the box on windows and max os X (leopard). For now, I only wrote how to do basic things like getting the code/log/annotation/status, nothing related to actual hacking of the code inside the numpy code. Also, the mirror is pure http, which means it will be pretty slow to get the code the first time (I don't have access to more than bare http). For information, a full checkout (with all the history since 2001, trunk only) takes 30 Mb, without counting the working tree. cheers, David From basilisk96 at gmail.com Thu Jan 10 01:08:03 2008 From: basilisk96 at gmail.com (Basilisk96) Date: Wed, 9 Jan 2008 22:08:03 -0800 (PST) Subject: [Numpy-discussion] subclassing matrix Message-ID: Hello folks, In the course of a project that involved heavy use of geometry and linear algebra, I found it useful to create a Vector subclass of numpy.matrix (represented as a column vector in my case). I'd like to hear comments about my use of this "class promotion" statement in __new__: ret.__class__ = cls It seems to me that it is hackish to just change an instance's class on the fly, so perhaps someone could clue me in on a better practice. Here is my reason for doing this: Many applications of this code involve operations between instances of numpy.matrix and instances of Vector, such as applying a linear- operator matrix on a vector. If I omit that "class promotion" statement, then the results of such operations cannot be instantiated as Vector types: >>> from vector import Vector >>> import numpy >>> u = Vector('1 2 3') >>> A = numpy.matrix('2 0 0; 0 2 0; 0 0 2') >>> p = Vector(A * u) >>> p.__class__ This is undesirable because the calculation result loses the custom Vector methods and attributes that I want to use. However, if I use that "class promotion" statement, the p.__class__ lookup returns what I want: >>> p.__class__ Is there a better way to achieve that? Here is the partial subclass code: #---------- vector.py import numpy as _N import math as _M #default tolerance for equality tests TOL_EQ = 1e-6 #default format for pretty-printing Vector instances FMT_VECTOR_DEFAULT = "%+.5f" class Vector(_N.matrix): """ 2D/3D vector class that supports numpy matrix operations and more. Examples: u = Vector([1,2,3]) v = Vector('3 4 5') w = Vector([1, 2]) """ def __new__(cls, data="0. 0. 0.", dtype=_N.float64): """ Subclass instance constructor. If data is not specified, a zero Vector is constructed. The constructor always returns a Vector instance. The instance gets a customizable Format attribute, which controls the printing precision. """ ret = super(Vector, cls).__new__(cls, data, dtype=dtype) #promote the instance to cls type. ret.__class__ = cls assert ret.size in (2, 3), 'Vector must have either two or three components' if ret.shape[0] == 1: ret = ret.T assert ret.shape == (ret.shape[0], 1), 'could not express Vector as a Mx1 matrix' if ret.shape[0] == 2: ret = _N.vstack((ret, 0.)) ret.Format = FMT_VECTOR_DEFAULT return ret def __str__(self): fmt = getattr(self, "Format", FMT_VECTOR_DEFAULT) fmt = ', '.join([fmt]*3) return ''.join(["(", fmt, ")"]) % (self.X, self.Y, self.Z) def __repr__(self): fmt = ', '.join(['%s']*3) return ''.join(["%s([", fmt, "])"]) % (self.__class__.__name__, self.X, self.Y, self.Z) #### the remaining methods are Vector-specific math operations, including the X,Y,Z properties... Cheers, -Basilisk96 From efiring at hawaii.edu Thu Jan 10 01:31:03 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 09 Jan 2008 20:31:03 -1000 Subject: [Numpy-discussion] BzrMirror page on numpy wiki In-Reply-To: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> References: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> Message-ID: <4785BBA7.9000108@hawaii.edu> David, Thanks, that's great. I am curious: how did you do the conversion? Did you work directly from the existing svn repo or did you make a local mirror first? Eric David Cournapeau wrote: > Hi, > > I started a page on how to use bzr, with a link to an import I have > done on the numpy trunk: > > http://scipy.org/scipy/numpy/wiki/BzrMirror > > I thought it would be useful for people not familiar with DVCS, and are > curious to try. In perticular, since I have done the svn->bzr import, > the distributed bzr binary will work out of the box on windows and max > os X (leopard). > > For now, I only wrote how to do basic things like getting the > code/log/annotation/status, nothing related to actual hacking of the > code inside the numpy code. Also, the mirror is pure http, which means > it will be pretty slow to get the code the first time (I don't have > access to more than bare http). For information, a full checkout (with > all the history since 2001, trunk only) takes 30 Mb, without counting > the working tree. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From fullung at gmail.com Thu Jan 10 02:08:33 2008 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 10 Jan 2008 09:08:33 +0200 Subject: [Numpy-discussion] Intel MKL - was: parallel numpy - any info? In-Reply-To: <4785A57C.80801@ar.media.kyoto-u.ac.jp> References: <6.2.3.4.2.20080109070201.041afac0@rjs.org> <4785A57C.80801@ar.media.kyoto-u.ac.jp> Message-ID: <5eec5f300801092308i6eee1d4kd5932019375bbfa7@mail.gmail.com> Hello On Jan 10, 2008 6:56 AM, David Cournapeau wrote: > The one thing which I am not sure about is: say one MKL binary does not > work, and say I (or anyone outside your company) build numpy with the > MKL ro debug it, can I redistribute a new binary, even if it is just for > testing purpose ? Let's say Ray's company buys one copy of Intel MKL. This gives them the Intel MKL DLL and link libraries (.libs). Now they compile NumPy and link it against Intel MKL. They can then (as I understand the license agreement, IANAL, etc.) distribute this binary and the Intel MKL DLLs. They may *not* distribute the link libraries. This means that there is no easy way for anyone else to build a new executable that is linked against MKL. Cheers, Albert From david at ar.media.kyoto-u.ac.jp Thu Jan 10 05:40:14 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 10 Jan 2008 19:40:14 +0900 Subject: [Numpy-discussion] BzrMirror page on numpy wiki In-Reply-To: <4785BBA7.9000108@hawaii.edu> References: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> <4785BBA7.9000108@hawaii.edu> Message-ID: <4785F60E.6040104@ar.media.kyoto-u.ac.jp> Eric Firing wrote: > David, > > Thanks, that's great. > > I am curious: how did you do the conversion? Did you work directly from > the existing svn repo or did you make a local mirror first? > I did not manage to make a mirror first, but that's definitely what you would want to do (if you want to do it for another repository). Importing numpy took around 3-4 hours, that is 3-4 seconds / revision, mostly because of network latency. With matplotlib, which I tried to, I first did a svn mirror, and then, it can import around 4-5 revision / second on average (of course, it depends on the changeset for each revision). For bzr, once you manage to install bzr-svn, importing is transparent: bzr branch http://blabla/trunk You can also import the whole repository (including branches and tags), using the svn-import command, but that's a bit more difficult, specially if the repository does not follow the trunk/branches/tags convention of subversion. cheers, David From david at ar.media.kyoto-u.ac.jp Thu Jan 10 05:42:36 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 10 Jan 2008 19:42:36 +0900 Subject: [Numpy-discussion] Intel MKL - was: parallel numpy - any info? In-Reply-To: <5eec5f300801092308i6eee1d4kd5932019375bbfa7@mail.gmail.com> References: <6.2.3.4.2.20080109070201.041afac0@rjs.org> <4785A57C.80801@ar.media.kyoto-u.ac.jp> <5eec5f300801092308i6eee1d4kd5932019375bbfa7@mail.gmail.com> Message-ID: <4785F69C.7030404@ar.media.kyoto-u.ac.jp> Albert Strasheim wrote: > Hello > > On Jan 10, 2008 6:56 AM, David Cournapeau wrote: > >> The one thing which I am not sure about is: say one MKL binary does not >> work, and say I (or anyone outside your company) build numpy with the >> MKL ro debug it, can I redistribute a new binary, even if it is just for >> testing purpose ? >> > > Let's say Ray's company buys one copy of Intel MKL. This gives them > the Intel MKL DLL and link libraries (.libs). > > Now they compile NumPy and link it against Intel MKL. They can then > (as I understand the license agreement, IANAL, etc.) distribute this > binary and the Intel MKL DLLs. > > They may *not* distribute the link libraries. This means that there is > no easy way for anyone else to build a new executable that is linked > against MKL. > Ah yes, I forgot about this windows idiosyncrasy of import libraries. In this case, you have a clear technical difference between a shared library for distribution and for developers. But anyway, I now realize that the MKL is not free on windows anyway, so this is a non problem. cheers, David From ndbecker2 at gmail.com Thu Jan 10 08:54:39 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 10 Jan 2008 08:54:39 -0500 Subject: [Numpy-discussion] example of ufunc c-code? Message-ID: Where can I find a (hopefully simple) example of c-code for a ufunc? From oliphant at enthought.com Thu Jan 10 09:27:18 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Jan 2008 08:27:18 -0600 Subject: [Numpy-discussion] example of ufunc c-code? In-Reply-To: References: Message-ID: <47862B46.4020103@enthought.com> Neal Becker wrote: > Where can I find a (hopefully simple) example of c-code for a ufunc? > > Probably in scipy.special and or umathmodule.c in numpy/core/src. The key is to register a 1-d "inner" loop that performs the basic calculation for a particular data-type. You also need to provide the C-function and arrays that indicate the data-types that the C-function accepts to ufunc. The umathmodule.c is harder to track because it is "auto-generated", so the scipy.special (cephesmodule.c) may be easier to understand. Best regards, -Travis O. From david at ar.media.kyoto-u.ac.jp Thu Jan 10 09:37:54 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 10 Jan 2008 23:37:54 +0900 Subject: [Numpy-discussion] BzrMirror page on numpy wiki In-Reply-To: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> References: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> Message-ID: <47862DC2.2060206@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Hi, > > I started a page on how to use bzr, with a link to an import I have > done on the numpy trunk: > > http://scipy.org/scipy/numpy/wiki/BzrMirror > > I thought it would be useful for people not familiar with DVCS, and are > curious to try. In perticular, since I have done the svn->bzr import, > the distributed bzr binary will work out of the box on windows and max > os X (leopard). > > For now, I only wrote how to do basic things like getting the > code/log/annotation/status, nothing related to actual hacking of the > code inside the numpy code. Also, the mirror is pure http, which means > it will be pretty slow to get the code the first time (I don't have > access to more than bare http). For information, a full checkout (with > all the history since 2001, trunk only) takes 30 Mb, without counting > the working tree. > > I upgraded the information with more advanced concepts (repository vs branch vs checkout, committing, and merging). I tried to be as concise and as simple as possible, so that people who may wonder what we are talking about when we speak about hg/bzr/git can see what it is about. I also tried to show that at least as far as bzr is concerned, most commands (the one most people will use) are really similar to svn, for people who wonder about the learning curve. cheers, David From cjw at sympatico.ca Thu Jan 10 10:03:43 2008 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu, 10 Jan 2008 10:03:43 -0500 Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: Message-ID: Basilisk96 wrote: > Hello folks, > > In the course of a project that involved heavy use of geometry and > linear algebra, I found it useful to create a Vector subclass of > numpy.matrix (represented as a column vector in my case). Why not consider a matrix with a shape of (1, n) as a row vector and one with (n, 1) as a column vector? Then you can simply write A * u or u.T * A. Does this not meet the need? You could add methods isRowVector and isColumnVector to the Matrix class. Colin W. > > I'd like to hear comments about my use of this "class promotion" > statement in __new__: > ret.__class__ = cls > > It seems to me that it is hackish to just change an instance's class > on the fly, so perhaps someone could clue me in on a better practice. > Here is my reason for doing this: > Many applications of this code involve operations between instances of > numpy.matrix and instances of Vector, such as applying a linear- > operator matrix on a vector. If I omit that "class promotion" > statement, then the results of such operations cannot be instantiated > as Vector types: > >>> from vector import Vector > >>> import numpy > >>> u = Vector('1 2 3') > >>> A = numpy.matrix('2 0 0; 0 2 0; 0 0 2') > >>> p = Vector(A * u) > >>> p.__class__ > > > This is undesirable because the calculation result loses the custom > Vector methods and attributes that I want to use. However, if I use > that "class promotion" statement, the p.__class__ lookup returns what > I want: > >>> p.__class__ > > > Is there a better way to achieve that? > > Here is the partial subclass code: > #---------- vector.py > import numpy as _N > import math as _M > #default tolerance for equality tests > TOL_EQ = 1e-6 > #default format for pretty-printing Vector instances > FMT_VECTOR_DEFAULT = "%+.5f" > > class Vector(_N.matrix): > """ > 2D/3D vector class that supports numpy matrix operations and more. > > Examples: > u = Vector([1,2,3]) > v = Vector('3 4 5') > w = Vector([1, 2]) > """ > def __new__(cls, data="0. 0. 0.", dtype=_N.float64): > """ > Subclass instance constructor. > > If data is not specified, a zero Vector is constructed. > The constructor always returns a Vector instance. > The instance gets a customizable Format attribute, which > controls the printing precision. > """ > ret = super(Vector, cls).__new__(cls, data, dtype=dtype) > #promote the instance to cls type. > ret.__class__ = cls > assert ret.size in (2, 3), 'Vector must have either two or > three components' > if ret.shape[0] == 1: > ret = ret.T > assert ret.shape == (ret.shape[0], 1), 'could not express > Vector as a Mx1 matrix' > if ret.shape[0] == 2: > ret = _N.vstack((ret, 0.)) > ret.Format = FMT_VECTOR_DEFAULT > return ret > > def __str__(self): > fmt = getattr(self, "Format", FMT_VECTOR_DEFAULT) > fmt = ', '.join([fmt]*3) > return ''.join(["(", fmt, ")"]) % (self.X, self.Y, self.Z) > > def __repr__(self): > fmt = ', '.join(['%s']*3) > return ''.join(["%s([", fmt, "])"]) % > (self.__class__.__name__, self.X, self.Y, self.Z) > > #### the remaining methods are Vector-specific math operations, > including the X,Y,Z properties... > > > Cheers, > -Basilisk96 From bacmsantos at gmail.com Thu Jan 10 08:54:36 2008 From: bacmsantos at gmail.com (bacmsantos at gmail.com) Date: Thu, 10 Jan 2008 13:54:36 +0000 Subject: [Numpy-discussion] Torna-te um pioneiro do Portal Emprego.pt. Message-ID: <062ccfa20a9ba555f8ecfa28cc63a84c@emprego.pt> To view the message, please use an HTML compatible email viewer! -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.j.mclay at gmail.com Thu Jan 10 10:43:59 2008 From: michael.j.mclay at gmail.com (Michael McLay) Date: Thu, 10 Jan 2008 10:43:59 -0500 Subject: [Numpy-discussion] BzrMirror page on numpy wiki In-Reply-To: <47862DC2.2060206@ar.media.kyoto-u.ac.jp> References: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> <47862DC2.2060206@ar.media.kyoto-u.ac.jp> Message-ID: <683f83340801100743m25543ecfgd30f8ea045297222@mail.gmail.com> Thanks for the explanation of bzr. I'd like to give it a try on another project. How did you create the repository at http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy.trunk repository? Is this bzr mirror of the numpy svn repository automatically updated to follow the commits to the numpy svn repository? Also, what tool was used to draw the DAG in the screenshot? The screenshot was very helpful. On Jan 10, 2008 9:37 AM, David Cournapeau wrote: > David Cournapeau wrote: > > Hi, > > > > I started a page on how to use bzr, with a link to an import I have > > done on the numpy trunk: > > > > http://scipy.org/scipy/numpy/wiki/BzrMirror > > > > I thought it would be useful for people not familiar with DVCS, and are > > curious to try. In perticular, since I have done the svn->bzr import, > > the distributed bzr binary will work out of the box on windows and max > > os X (leopard). > > > > For now, I only wrote how to do basic things like getting the > > code/log/annotation/status, nothing related to actual hacking of the > > code inside the numpy code. Also, the mirror is pure http, which means > > it will be pretty slow to get the code the first time (I don't have > > access to more than bare http). For information, a full checkout (with > > all the history since 2001, trunk only) takes 30 Mb, without counting > > the working tree. > > > > > I upgraded the information with more advanced concepts (repository vs > branch vs checkout, committing, and merging). > > I tried to be as concise and as simple as possible, so that people who > may wonder what we are talking about when we speak about hg/bzr/git can > see what it is about. > > I also tried to show that at least as far as bzr is concerned, most > commands (the one most people will use) are really similar to svn, for > people who wonder about the learning curve. > > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From david at ar.media.kyoto-u.ac.jp Thu Jan 10 10:46:41 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 11 Jan 2008 00:46:41 +0900 Subject: [Numpy-discussion] BzrMirror page on numpy wiki In-Reply-To: <683f83340801100743m25543ecfgd30f8ea045297222@mail.gmail.com> References: <4785B36E.6010403@ar.media.kyoto-u.ac.jp> <47862DC2.2060206@ar.media.kyoto-u.ac.jp> <683f83340801100743m25543ecfgd30f8ea045297222@mail.gmail.com> Message-ID: <47863DE1.2080904@ar.media.kyoto-u.ac.jp> Michael McLay wrote: > Thanks for the explanation of bzr. I'd like to give it a try on > another project. How did you create the repository at > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy.trunk > repository? Is this bzr mirror of the numpy svn repository > automatically updated to follow the commits to the numpy svn > repository? > No, this is just a "demo". I don't have access to the server, except to push files to it, and also, I don't want that too many people to access to it. This is something which at some point should be done on scipy.org (but now, there are some problems with the scipy.org servers which need to be solved/upgrade, as I understand, and a bzr mirror is not a priority, obviously). This way, it could be easily and regularly updated through a cron job, or whatever. > Also, what tool was used to draw the DAG in the screenshot? The > screenshot was very helpful. > It is bzr-gtk, which is a graphical front end to bzr *commands*, that is it is not an integral GUI, just a graphical version of every command. For the log and annotation, I find it useful (as well as for the diff, in some particular cases). The bzr-gtk plugin is not included by default in the binaries (yet, at least). cheers, David From stefan at sun.ac.za Thu Jan 10 10:53:59 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 10 Jan 2008 17:53:59 +0200 Subject: [Numpy-discussion] Setting WRITEABLE flag on array scalar Message-ID: <20080110155359.GH12476@mentat.za.net> Hi all, We currently use an array scalar of value False as the mask in MaskedArray. I would like to make sure that the mask value cannot be modified, but when I try import numpy as np x = np.bool_(False) x.flags['WRITEABLE'] = False I am warned that you "Cannot set flags on array scalars.". Is there any reason why this is prohibited? Which is the best way around it? Thanks for any feedback. St?fan From tim.hochberg at ieee.org Thu Jan 10 11:13:36 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 10 Jan 2008 09:13:36 -0700 Subject: [Numpy-discussion] Setting WRITEABLE flag on array scalar In-Reply-To: <20080110155359.GH12476@mentat.za.net> References: <20080110155359.GH12476@mentat.za.net> Message-ID: On Jan 10, 2008 8:53 AM, Stefan van der Walt wrote: > Hi all, > > We currently use an array scalar of value False as the mask in > MaskedArray. I would like to make sure that the mask value cannot be > modified, but when I try > > import numpy as np > x = np.bool_(False) > x.flags['WRITEABLE'] = False > > I am warned that you "Cannot set flags on array scalars.". Is there > any reason why this is prohibited? Which is the best way around it? When I try this, the WRITEABLE flag is False to begin with. Are you seeing it as True? If not it wouldn't seem like there's any reason to do anything in your case. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Thu Jan 10 11:21:28 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Jan 2008 10:21:28 -0600 Subject: [Numpy-discussion] Setting WRITEABLE flag on array scalar In-Reply-To: <20080110155359.GH12476@mentat.za.net> References: <20080110155359.GH12476@mentat.za.net> Message-ID: <47864608.6010100@enthought.com> Stefan van der Walt wrote: > Hi all, > > We currently use an array scalar of value False as the mask in > MaskedArray. I would like to make sure that the mask value cannot be > modified, but when I try > > import numpy as np > x = np.bool_(False) > x.flags['WRITEABLE'] = False > > I am warned that you "Cannot set flags on array scalars.". Is there > any reason why this is prohibited? Which is the best way around it? > You can't do it, because there is no place for the information to go. The array scalars are always read-only anyway. So, there should be no reason to set this flag. -Travis O. From stefan at sun.ac.za Thu Jan 10 14:02:52 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 10 Jan 2008 21:02:52 +0200 Subject: [Numpy-discussion] Setting WRITEABLE flag on array scalar In-Reply-To: <47864608.6010100@enthought.com> References: <20080110155359.GH12476@mentat.za.net> <47864608.6010100@enthought.com> Message-ID: <20080110190252.GI12476@mentat.za.net> Hi, On Thu, Jan 10, 2008 at 10:21:28AM -0600, Travis E. Oliphant wrote: > You can't do it, because there is no place for the information to go. > The array scalars are always read-only anyway. So, there should be no > reason to set this flag. Right, I was confused because I saw: In [1]: x = N.bool_(3) In [2]: x Out[2]: True In [3]: x += 1 In [4]: x Out[4]: 2 But now I see that iadd does not actually change the original value. Thanks St?fan From basilisk96 at gmail.com Thu Jan 10 19:55:46 2008 From: basilisk96 at gmail.com (Basilisk96) Date: Thu, 10 Jan 2008 16:55:46 -0800 (PST) Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: Message-ID: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> Yes, that certainly meets the need. In previous applications, I never thought to use Vector because it was sufficient for me to adopt the convention of a column vector to represent XY points. That way, I could do things like c * R * u + t. Isn't that simple? Not only is it easy on the eyes, but it follows most academic definitions that I know. Furthermore, I was able to apply linear operators to a whole set of XY points by doing something to the tune of: u = numpy.hstack(tuple_of_points_from_ascii_file) # u.shape is (2, n) R = some_2d_numpy_rotation_matrix # R.shape is (2, 2) t = numpy.mat("1.; 3.") v = R * u + t # v.shape is the same as u.shape So, the need arose for a self-contained Vector class that inherently supports a number of common geometric operations, and uses numpy for them. It seemed natural to subclass matrix because numpy.array is a general-case array; numpy.matrix restricts that to an array of rank 2; and Vector further restricts that to a *column* array of rank 2. Now, if I were to write a custom library of my common vector operations using module functions, I'd have to always keep in mind that they must operate on either column or row vectors, but not both. Instead, I chose to write a subclass of a well-tested class and explicitly forced all Vector instances to be columns. Vector has a number of methods that wrap numpy functions within the Vector context. For example: def cross(self, other): """Cross product of this vector and another vector""" if isinstance(other, self.__class__): return Vector(_N.cross(self, other, axis=0)) else: raise TypeError('Both arguments must be of type Vector') I think that a call like w = u.cross(v) is less verbose than having to remember the `axis` argument in numpy.cross(u, v, axis=0). Yes, I realize that it would not be required if the vectors were row matrices, but then I'd have to live with using u.T * A, which I don't particularly like either. But now I can do things like: >>> from vector import Vector >>> u = Vector('1 2 3') >>> v = Vector('3 2 1') >>> w = u.cross(v) >>> print w (-4.00000, +8.00000, -4.00000) >>> alpha = u.angle(w) >>> print alpha 1.57079632679 ..and so on. However, my original question still stands: is ret.__class__ = cls a good idea? Cheers, -Basilisk96 On Jan 10, 10:03 am, "Colin J. Williams" wrote: > Basilisk96 wrote: > > Hello folks, > > > In the course of a project that involved heavy use of geometry and > > linear algebra, I found it useful to create a Vector subclass of > > numpy.matrix (represented as a column vector in my case). > > Why not consider a matrix with a shape > of (1, n) as a row vector and > one with (n, 1) as a column vector? > > Then you can simply write A * u or u.T * A. > > Does this not meet the need? > > You could add methods isRowVector and > isColumnVector to the Matrix class. > > Colin W. > > > > > > > I'd like to hear comments about my use of this "class promotion" > > statement in __new__: > > ret.__class__ = cls > > > It seems to me that it is hackish to just change an instance's class > > on the fly, so perhaps someone could clue me in on a better practice. > > Here is my reason for doing this: > > Many applications of this code involve operations between instances of > > numpy.matrix and instances of Vector, such as applying a linear- > > operator matrix on a vector. If I omit that "class promotion" > > statement, then the results of such operations cannot be instantiated > > as Vector types: > > >>> from vector import Vector > > >>> import numpy > > >>> u = Vector('1 2 3') > > >>> A = numpy.matrix('2 0 0; 0 2 0; 0 0 2') > > >>> p = Vector(A * u) > > >>> p.__class__ > > > > > This is undesirable because the calculation result loses the custom > > Vector methods and attributes that I want to use. However, if I use > > that "class promotion" statement, the p.__class__ lookup returns what > > I want: > > >>> p.__class__ > > > > > Is there a better way to achieve that? > > > Here is the partial subclass code: > > #---------- vector.py > > import numpy as _N > > import math as _M > > #default tolerance for equality tests > > TOL_EQ = 1e-6 > > #default format for pretty-printing Vector instances > > FMT_VECTOR_DEFAULT = "%+.5f" > > > class Vector(_N.matrix): > > """ > > 2D/3D vector class that supports numpy matrix operations and more. > > > Examples: > > u = Vector([1,2,3]) > > v = Vector('3 4 5') > > w = Vector([1, 2]) > > """ > > def __new__(cls, data="0. 0. 0.", dtype=_N.float64): > > """ > > Subclass instance constructor. > > > If data is not specified, a zero Vector is constructed. > > The constructor always returns a Vector instance. > > The instance gets a customizable Format attribute, which > > controls the printing precision. > > """ > > ret = super(Vector, cls).__new__(cls, data, dtype=dtype) > > #promote the instance to cls type. > > ret.__class__ = cls > > assert ret.size in (2, 3), 'Vector must have either two or > > three components' > > if ret.shape[0] == 1: > > ret = ret.T > > assert ret.shape == (ret.shape[0], 1), 'could not express > > Vector as a Mx1 matrix' > > if ret.shape[0] == 2: > > ret = _N.vstack((ret, 0.)) > > ret.Format = FMT_VECTOR_DEFAULT > > return ret > > > def __str__(self): > > fmt = getattr(self, "Format", FMT_VECTOR_DEFAULT) > > fmt = ', '.join([fmt]*3) > > return ''.join(["(", fmt, ")"]) % (self.X, self.Y, self.Z) > > > def __repr__(self): > > fmt = ', '.join(['%s']*3) > > return ''.join(["%s([", fmt, "])"]) % > > (self.__class__.__name__, self.X, self.Y, self.Z) > > > #### the remaining methods are Vector-specific math operations, > > including the X,Y,Z properties... > > > Cheers, > > -Basilisk96 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Jan 11 11:47:33 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Jan 2008 09:47:33 -0700 Subject: [Numpy-discussion] Template system Message-ID: Hi All, I'm thinking of adding the template system I pulled out of Django to Numpy to use in place of the current code generator. Its advantages are documentation, flexibility, and maintainance. The code totals about 470 KB, comes with a BSD license, and is compatible with Python >= 2.3. I want to bring this up on the list because I'm sure there will be objections along the lines of "why do we need that"? So I want to see how strenuous the objections are. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Jan 11 12:45:21 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 11 Jan 2008 12:45:21 -0500 Subject: [Numpy-discussion] example of ufunc c-code? References: Message-ID: Neal Becker wrote: > Where can I find a (hopefully simple) example of c-code for a ufunc? Wow, that was easy. Maybe you can use this as an example: -------------- next part -------------- A non-text attachment was scrubbed... Name: mag_sqr.cc Type: text/x-c++src Size: 1407 bytes Desc: not available URL: From robert.kern at gmail.com Fri Jan 11 16:58:36 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 11 Jan 2008 15:58:36 -0600 Subject: [Numpy-discussion] Template system In-Reply-To: References: Message-ID: <4787E68C.1020505@gmail.com> Charles R Harris wrote: > Hi All, > > I'm thinking of adding the template system I pulled out of Django to > Numpy to use in place of the current code generator. Its advantages are > documentation, flexibility, and maintainance. The code totals about 470 > KB, comes with a BSD license, and is compatible with Python >= 2.3. I > want to bring this up on the list because I'm sure there will be > objections along the lines of "why do we need that"? So I want to see > how strenuous the objections are. It's too big. If you want a better template system than the current one, there are a few which are contained in a single file. That would be fine to include, but I object strongly to incorporating Jinja. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Jan 11 20:16:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Jan 2008 18:16:22 -0700 Subject: [Numpy-discussion] Template system In-Reply-To: <4787E68C.1020505@gmail.com> References: <4787E68C.1020505@gmail.com> Message-ID: On Jan 11, 2008 2:58 PM, Robert Kern wrote: > Charles R Harris wrote: > > Hi All, > > > > I'm thinking of adding the template system I pulled out of Django to > > Numpy to use in place of the current code generator. Its advantages are > > documentation, flexibility, and maintainance. The code totals about 470 > > KB, comes with a BSD license, and is compatible with Python >= 2.3. I > > want to bring this up on the list because I'm sure there will be > > objections along the lines of "why do we need that"? So I want to see > > how strenuous the objections are. > > It's too big. If you want a better template system than the current one, > there > are a few which are contained in a single file. That would be fine to > include, > but I object strongly to incorporating Jinja. > Ummm, harsh. I'm certainly open to suggestions. I took a closer look at the django system and reduced it to 308K, 70K zipped. There was a lot of space taken up by the .svn directories inherited from django and *.pyc files. I could probably knock another 50K off by removing some functionality we don't really need: html, http, dates, time, and maybe template inheritance. Just for comparison, here are some things that are in a fresh checkout of numpy, including the .svn files Numpy 23 MB f2py 5.4 MB doc/swig 1.5 MB code_generators 192K So adding the template subsystem would increase the repository size by about 1.4%. Do you have any objections in addition to the size? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mani.sabri at gmail.com Fri Jan 11 20:35:10 2008 From: mani.sabri at gmail.com (mani sabri) Date: Sat, 12 Jan 2008 05:05:10 +0330 Subject: [Numpy-discussion] wrap Python/numpy to C/C++. mingw. elmer. Message-ID: <478819a2.096c100a.16a8.626f@mx.google.com> Hi Sorry for irrelevant subject. I found elmer when I was googling for something to wrap my python/numpy code to C/C++ automatically because I want a dll for some application and I know nothing about python Api (shame on me). Does anybody successfully compiled elmer on windows with mingw? Or is it anything better than elmer? (mingwISHER!/windowsISHER!) I will appreciate any suggestion. Regards Mani From robert.kern at gmail.com Fri Jan 11 21:02:22 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 11 Jan 2008 20:02:22 -0600 Subject: [Numpy-discussion] Template system In-Reply-To: References: <4787E68C.1020505@gmail.com> Message-ID: <47881FAE.9000308@gmail.com> Charles R Harris wrote: > > > On Jan 11, 2008 2:58 PM, Robert Kern > wrote: > > Charles R Harris wrote: > > Hi All, > > > > I'm thinking of adding the template system I pulled out of Django to > > Numpy to use in place of the current code generator. Its > advantages are > > documentation, flexibility, and maintainance. The code totals > about 470 > > KB, comes with a BSD license, and is compatible with Python >= 2.3. I > > want to bring this up on the list because I'm sure there will be > > objections along the lines of "why do we need that"? So I want > to see > > how strenuous the objections are. > > It's too big. If you want a better template system than the current > one, there > are a few which are contained in a single file. That would be fine > to include, > but I object strongly to incorporating Jinja. > > > Ummm, harsh. I'm certainly open to suggestions. I took a closer look at > the django system and reduced it to 308K, 70K zipped. There was a lot of > space taken up by the .svn directories inherited from django and *.pyc > files. I could probably knock another 50K off by removing some > functionality we don't really need: html, http, dates, time, and maybe > template inheritance. Just for comparison, here are some things that are > in a fresh checkout of numpy, including the .svn files > > Numpy 23 MB > f2py 5.4 MB > doc/swig 1.5 MB > code_generators 192K > > So adding the template subsystem would increase the repository size by > about 1.4%. Do you have any objections in addition to the size? Yes. I'm not convinced that we need anything better than we have now. Show me how using Jinja templates makes the current code better. Check the one-file templating modules (like YAPTU[1] or Cog[2]) out there to see if those are sufficient. Jinja and its kin (Mako, Cheetah, etc.) are intended for a very different use case, and their design, implementation, and footprint are optimized for that. [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52305 [2] http://nedbatchelder.com/code/cog/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Jan 11 21:24:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Jan 2008 19:24:04 -0700 Subject: [Numpy-discussion] Template system In-Reply-To: <47881FAE.9000308@gmail.com> References: <4787E68C.1020505@gmail.com> <47881FAE.9000308@gmail.com> Message-ID: On Jan 11, 2008 7:02 PM, Robert Kern wrote: > Charles R Harris wrote: > > > > > > On Jan 11, 2008 2:58 PM, Robert Kern > > wrote: > > > > Charles R Harris wrote: > > > Hi All, > > > > > > I'm thinking of adding the template system I pulled out of Django > to > > > Numpy to use in place of the current code generator. Its > > advantages are > > > documentation, flexibility, and maintainance. The code totals > > about 470 > > > KB, comes with a BSD license, and is compatible with Python >= > 2.3. I > > > want to bring this up on the list because I'm sure there will be > > > objections along the lines of "why do we need that"? So I want > > to see > > > how strenuous the objections are. > > > > It's too big. If you want a better template system than the current > > one, there > > are a few which are contained in a single file. That would be fine > > to include, > > but I object strongly to incorporating Jinja. > > > > > > Ummm, harsh. I'm certainly open to suggestions. I took a closer look at > > the django system and reduced it to 308K, 70K zipped. There was a lot of > > space taken up by the .svn directories inherited from django and *.pyc > > files. I could probably knock another 50K off by removing some > > functionality we don't really need: html, http, dates, time, and maybe > > template inheritance. Just for comparison, here are some things that are > > in a fresh checkout of numpy, including the .svn files > > > > Numpy 23 MB > > f2py 5.4 MB > > doc/swig 1.5 MB > > code_generators 192K > > > > So adding the template subsystem would increase the repository size by > > about 1.4%. Do you have any objections in addition to the size? > > Yes. I'm not convinced that we need anything better than we have now. Show > me > how using Jinja templates makes the current code better. Check the > one-file > templating modules (like YAPTU[1] or Cog[2]) out there to see if those are > sufficient. Jinja and its kin (Mako, Cheetah, etc.) are intended for a > very YAPTU lacks any sort of control structures in the markup, in fact, it doesn't *have* markup. This has several consequences. 1) You have to write code with all the loops and substitutions. That's work. 2) Someone has to document the markup they decide to use. 3) It won't be obvious from looking at the template what is going to happen. YAPTU is a step down from what we have now. Indeed, what we have now has something like YAPTU built in. Cog is 109KB and has the same problems as YAPTU, except the code is now part of the template, totally messing up the separation of view and model, and it is excessively ugly to look at (IMHO). I suspect we could find many uses for a template capability because folks haven't been thinking that way. I wonder if f2py couldn't make use of such a facility? As to the usual use of Cheetah and such in web serving, common problems lead to common solutions. It's an example of convergent evolution, not an insult. I picked Django over Cheetah as a starting point because I thought the markup was much better looking. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at enthought.com Fri Jan 11 23:09:50 2008 From: eric at enthought.com (eric jones) Date: Fri, 11 Jan 2008 22:09:50 -0600 Subject: [Numpy-discussion] wrap Python/numpy to C/C++. mingw. elmer. In-Reply-To: <478819a2.096c100a.16a8.626f@mx.google.com> References: <478819a2.096c100a.16a8.626f@mx.google.com> Message-ID: <47883D8E.5060205@enthought.com> Hey Mani, I've included Rick Ratzel on this since he is the author of elmer and may have more guidance. eric mani sabri wrote: > Hi > Sorry for irrelevant subject. > I found elmer when I was googling for something to wrap my python/numpy code > to C/C++ automatically because I want a dll for some application and I know > nothing about python Api (shame on me). Does anybody successfully compiled > elmer on windows with mingw? Or is it anything better than elmer? > (mingwISHER!/windowsISHER!) I will appreciate any suggestion. > > Regards > Mani > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From basilisk96 at gmail.com Fri Jan 11 23:59:55 2008 From: basilisk96 at gmail.com (Basilisk96) Date: Fri, 11 Jan 2008 20:59:55 -0800 (PST) Subject: [Numpy-discussion] subclassing matrix In-Reply-To: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> Message-ID: <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> On Jan 11, 2008, Colin J. Williams wrote: > You make a good case that it's good not > to need to ponder what sort of > vector you are dealing with. > > My guess is that the answer to your > question is "no" but I would need to > play with your code to see that. My > feeling is that, at the bottom of > the __new__ module, the returned object > should be an instance of the > Vector class. > > It's been a while since I've worked with > numpy and so I'll look at it > and hope that someone gives you a > definitive answer before I sort it out. > > Colin W. Well, let's say that I get rid of that class promotion line. When the input object to the constructor is a string or a tuple such as Vector('1 2 3') or Vector([1,2,3]), then the returned object is always an instance of Vector. However, when the input object is a numpy.matrix instance, the returned object remains a numpy.matrix instance! So by doing that little hack, I promote it to Vector. BUT... It seems that I have solved only half of my problem here. The other half rears its ugly head when I perform operations between instances of numpy.matrix and Vector. The result ends up returning a matrix, which is bad because it has no knowledge of any custom Vector attributes. Here's a simple case: u = Vector('1 2 3') #Vector instance P = numpy.mat(numpy.eye(3)) #matrix instance u_new = P*u #matrix instance, not desirable! u_new_as_Vector = Vector(P*u) #Vector instance I'd rather not have to remember to re-instantiate the result in client code. I think I understand why this is happening - the code in numpy.core.defmatrix.matrix.__mul__ goes like this: def __mul__(self, other): if isinstance(other,(N.ndarray, list, tuple)) : # This promotes 1-D vectors to row vectors return N.dot(self, asmatrix(other)) if N.isscalar(other) or not hasattr(other, '__rmul__') : return N.dot(self, other) return NotImplemented It passes the first condition: isinstance(other,(N.ndarray)) is true; and so the return value becomes a matrix. Bummer. Do I also need to override a few overloaded methods like __mul__, __rmul__, etc. to make this work? Cheers, -Basilisk96 From charlesr.harris at gmail.com Sat Jan 12 00:05:43 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Jan 2008 22:05:43 -0700 Subject: [Numpy-discussion] Template system In-Reply-To: <47881FAE.9000308@gmail.com> References: <4787E68C.1020505@gmail.com> <47881FAE.9000308@gmail.com> Message-ID: OK, so far I've knocked it down to 35KB by removing stuff I'm not interested in. It is now smaller than Cog, and 7x larger than the file we now use to do the same job. I'm pretty sure I can make it leaner than that. It remains extensible. On Jan 11, 2008 7:02 PM, Robert Kern wrote: > Charles R Harris wrote: > > > > > > On Jan 11, 2008 2:58 PM, Robert Kern > > wrote: > > > > Charles R Harris wrote: > > > Hi All, > > > > > > I'm thinking of adding the template system I pulled out of Django > to > > > Numpy to use in place of the current code generator. Its > > advantages are > > > documentation, flexibility, and maintainance. The code totals > > about 470 > > > KB, comes with a BSD license, and is compatible with Python >= > 2.3. I > > > want to bring this up on the list because I'm sure there will be > > > objections along the lines of "why do we need that"? So I want > > to see > > > how strenuous the objections are. > > > > It's too big. If you want a better template system than the current > > one, there > > are a few which are contained in a single file. That would be fine > > to include, > > but I object strongly to incorporating Jinja. > > > > > > Ummm, harsh. I'm certainly open to suggestions. I took a closer look at > > the django system and reduced it to 308K, 70K zipped. There was a lot of > > space taken up by the .svn directories inherited from django and *.pyc > > files. I could probably knock another 50K off by removing some > > functionality we don't really need: html, http, dates, time, and maybe > > template inheritance. Just for comparison, here are some things that are > > in a fresh checkout of numpy, including the .svn files > > > > Numpy 23 MB > > f2py 5.4 MB > > doc/swig 1.5 MB > > code_generators 192K > > > > So adding the template subsystem would increase the repository size by > > about 1.4%. Do you have any objections in addition to the size? > > Yes. I'm not convinced that we need anything better than we have now. Show > me > how using Jinja templates makes the current code better. Check the > one-file > templating modules (like YAPTU[1] or Cog[2]) out there to see if those are > sufficient. Jinja and its kin (Mako, Cheetah, etc.) are intended for a > very > different use case, and their design, implementation, and footprint are > optimized for that. > > [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52305 > [2] http://nedbatchelder.com/code/cog/ > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Sat Jan 12 01:36:15 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 11 Jan 2008 23:36:15 -0700 Subject: [Numpy-discussion] subclassing matrix In-Reply-To: <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> Message-ID: On Jan 11, 2008 9:59 PM, Basilisk96 wrote: > On Jan 11, 2008, Colin J. Williams wrote: > > > You make a good case that it's good not > > to need to ponder what sort of > > vector you are dealing with. > > > > My guess is that the answer to your > > question is "no" but I would need to > > play with your code to see that. My > > feeling is that, at the bottom of > > the __new__ module, the returned object > > should be an instance of the > > Vector class. > > > > It's been a while since I've worked with > > numpy and so I'll look at it > > and hope that someone gives you a > > definitive answer before I sort it out. > > > > Colin W. > > Well, let's say that I get rid of that class promotion line. When the > input object to the constructor is a string or a tuple > such as Vector('1 2 3') or Vector([1,2,3]), then the returned object > is always an instance of Vector. However, when the input object is a > numpy.matrix instance, the returned object remains a numpy.matrix > instance! So by doing that little hack, I promote it to Vector. > > BUT... > > It seems that I have solved only half of my problem here. The other > half rears its ugly head when I perform operations between instances > of numpy.matrix and Vector. The result ends up returning a matrix, > which is bad because it has no knowledge of any custom Vector > attributes. Here's a simple case: > > u = Vector('1 2 3') #Vector instance > P = numpy.mat(numpy.eye(3)) #matrix instance > u_new = P*u #matrix instance, not desirable! > u_new_as_Vector = Vector(P*u) #Vector instance > > I'd rather not have to remember to re-instantiate the result in client > code. I think I understand why this is happening - the code in > numpy.core.defmatrix.matrix.__mul__ goes like this: > > def __mul__(self, other): > if isinstance(other,(N.ndarray, list, tuple)) : > # This promotes 1-D vectors to row vectors > return N.dot(self, asmatrix(other)) > if N.isscalar(other) or not hasattr(other, '__rmul__') : > return N.dot(self, other) > return NotImplemented > > It passes the first condition: isinstance(other,(N.ndarray)) is true; > and so the return value becomes a matrix. > > Bummer. > Do I also need to override a few overloaded methods like __mul__, > __rmul__, etc. to make this work? I believe that you need to look at __array_finalize__ and __array_priority__ (and there may be one other thing as well, I can't remember; it's late). Search for __array_finalize__ and that will probably help get you started. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Jan 12 02:00:11 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 12 Jan 2008 16:00:11 +0900 Subject: [Numpy-discussion] Template system In-Reply-To: References: <4787E68C.1020505@gmail.com> <47881FAE.9000308@gmail.com> Message-ID: <4788657B.4070000@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > OK, so far I've knocked it down to 35KB by removing stuff I'm not > interested in. It is now smaller than Cog, and 7x larger than the file > we now use to do the same job. I'm pretty sure I can make it leaner > than that. It remains extensible. One good way to see whereas it is a worthwhile addition would be to give an example of one source file using this new template. cheers, David From david at ar.media.kyoto-u.ac.jp Sat Jan 12 03:53:45 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 12 Jan 2008 17:53:45 +0900 Subject: [Numpy-discussion] Update on using scons to build numpy Message-ID: <47888019.1090007@ar.media.kyoto-u.ac.jp> Hi there, A quick email to give an update on my work to build numpy with scons. I've finished a few days ago to make my former work a separate package from numpy: it was more work than I expected because of bootstrapping issues, but I can now build numpy again with the new package on Linux. The separate package is called numscons, and is developed using launchpad for bug tracking and source code repository: https://code.launchpad.net/numpy.scons.support You can find tarballs for releases there (you want the last one, that is 0.2.2 at that time): https://launchpad.net/numpy.scons.support/+download And you need to use the branch build_with_scons to build numpy with numscons (the way to build numpy with scons is as before: you use setupscons.py instead of setup.py). I am now working on cleaning some things in the code (in particular, all the changes I had to do to scons itself are now being worked on to be integrated upstream, so that upstream scons will be used instead of our own cannibalized copy). cheers, David From robert.kern at gmail.com Sat Jan 12 06:12:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 12 Jan 2008 05:12:52 -0600 Subject: [Numpy-discussion] Template system In-Reply-To: References: <4787E68C.1020505@gmail.com> <47881FAE.9000308@gmail.com> Message-ID: <4788A0B4.4060107@gmail.com> Charles R Harris wrote: > OK, so far I've knocked it down to 35KB by removing stuff I'm not > interested in. It is now smaller than Cog, and 7x larger than the file > we now use to do the same job. I'm pretty sure I can make it leaner than > that. It remains extensible. Can you put a tarball up somewhere? Django generally requires environment variables to configure itself. There is a way to configure these from code, but it looks like that pulls in django.conf. This is my primary source of discomfort with using Django templates. My experience with using Django components outside of a Django web app has been frustrating. The more modifications we make, the more we have to maintain. And if any of it causes problems for someone trying to build numpy, that's more questions we need to answer. I want to answer numpy questions, not Django questions. I would be much, much more comfortable with an implementation that can simply be dropped in without modification. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Jan 12 11:12:26 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 12 Jan 2008 09:12:26 -0700 Subject: [Numpy-discussion] Template system In-Reply-To: <4788A0B4.4060107@gmail.com> References: <4787E68C.1020505@gmail.com> <47881FAE.9000308@gmail.com> <4788A0B4.4060107@gmail.com> Message-ID: On Jan 12, 2008 4:12 AM, Robert Kern wrote: > Charles R Harris wrote: > > OK, so far I've knocked it down to 35KB by removing stuff I'm not > > interested in. It is now smaller than Cog, and 7x larger than the file > > we now use to do the same job. I'm pretty sure I can make it leaner than > > that. It remains extensible. > > Can you put a tarball up somewhere? Django generally requires environment > variables to configure itself. There is a way to configure these from > code, but > it looks like that pulls in django.conf. This is my primary source of > discomfort Yeah, I had that problem running Django straight from the box. Pain.In.The.Ass. However, the easiest way to just get the template subsystem out is to delete a bunch of files and rename global_settings to settings. Then I also changed the name django to something else and put some imports in the __init__, but you don't have to do that. Easily done with a script. > with using Django templates. My experience with using Django components > outside > of a Django web app has been frustrating. The more modifications we make, > the > more we have to maintain. And if any of it causes problems for someone > trying to > build numpy, that's more questions we need to answer. I want to answer > numpy > questions, not Django questions. > > I would be much, much more comfortable with an implementation that can > simply be > dropped in without modification. > Well, now I'm all hot to see how small I can make it while keeping the ability to add bits back in. You've got me digging through the code with the usual results. The configuration file is gone entirely, I'm cleaning up other bits, and getting rid of automatic html escaping, caching, and other such serving nonsense because, well, I don't need it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From basilisk96 at gmail.com Sat Jan 12 16:24:31 2008 From: basilisk96 at gmail.com (Basilisk96) Date: Sat, 12 Jan 2008 13:24:31 -0800 (PST) Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> Message-ID: On Jan 12, 1:36 am, "Timothy Hochberg" wrote: > I believe that you need to look at __array_finalize__ and __array_priority__ > (and there may be one other thing as well, I can't remember; it's late). > Search for __array_finalize__ and that will probably help get you started. > Well sonovagun! I removed the hack. Then just by setting __array_priority__ = 20.0 in the class body, things are magically working almost as I expect. I say "almost" because of this custom method: def cross(self, other): """Cross product of this vector and another vector""" return _N.cross(self, other, axis=0) That call to numpy.cross returns a numpy.ndarray. Unless I do return Vector(_N.cross(self, other, axis=0)), I get problems downstream. When is __array_finalize__ called? By adding some print traces, I can see it's called every time an array is modified in any way i.e., reshaped, transposed, etc., and also during operations like u+v, u-v, A*u. But it's not called during the call to numpy.cross. Why? Cheers, -Basilisk96 From stefan at sun.ac.za Sat Jan 12 18:13:49 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 13 Jan 2008 01:13:49 +0200 Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> Message-ID: <20080112231349.GD15436@mentat.za.net> On Sat, Jan 12, 2008 at 01:24:31PM -0800, Basilisk96 wrote: [...] > When is __array_finalize__ called? By adding some print traces, I can > see it's called every time an array is modified in any way i.e., > reshaped, transposed, etc., and also during operations like u+v, u-v, > A*u. But it's not called during the call to numpy.cross. Why? Take a look at http://www.scipy.org/Subclasses for more detail. Regards St?fan From cjw at sympatico.ca Sat Jan 12 19:43:56 2008 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat, 12 Jan 2008 19:43:56 -0500 Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> Message-ID: <47895ECC.8040700@sympatico.ca> Basilisk96 wrote: > On Jan 12, 1:36 am, "Timothy Hochberg" wrote: >> I believe that you need to look at __array_finalize__ and __array_priority__ >> (and there may be one other thing as well, I can't remember; it's late). >> Search for __array_finalize__ and that will probably help get you started. >> > > Well sonovagun! > I removed the hack. > Then just by setting __array_priority__ = 20.0 in the class body, > things are magically working almost as I expect. I say "almost" > because of this custom method: > > def cross(self, other): > """Cross product of this vector and another vector""" > return _N.cross(self, other, axis=0) > > That call to numpy.cross returns a numpy.ndarray. Unless I do return > Vector(_N.cross(self, other, axis=0)), I get problems downstream. > > When is __array_finalize__ called? By adding some print traces, I can > see it's called every time an array is modified in any way i.e., > reshaped, transposed, etc., and also during operations like u+v, u-v, > A*u. But it's not called during the call to numpy.cross. Why? > > Cheers, > -Basilisk96 This may help. It is based on your initial script. The Vectors are considered as columns but presented as rows. This adds a complication which is not resolved. Colin W. #---------- vector.py import numpy as _N import math as _M #default tolerance for equality tests TOL_EQ = 1e-6 #default format for pretty-printing Vector instances FMT_VECTOR_DEFAULT = "%+.5f" class Vector(_N.matrix): """ 2D/3D vector class that supports numpy matrix operations and more. Examples: u = Vector([1,2,3]) v = Vector('3 4 5') w = Vector([1, 2]) """ def __new__(cls, data="0. 0. 0.", dtype=_N.float64): """ Subclass instance constructor. If data is not specified, a zero Vector is constructed. The constructor always returns a Vector instance. The instance gets a customizable Format attribute, which controls the printing precision. """ data= [1, 2, 3] ret= _N.matrix(data, dtype) ## ret = super(Vector, cls).__new__(cls, data, dtype=dtype) ## #promote the instance to cls type. ## ret.__class__ = cls assert ret.size in (2, 3), 'Vector must have either two or three components' if ret.shape[0] == 1: ret = ret.T assert ret.shape == (ret.shape[0], 1), 'could not express Vector as a Mx1 matrix' if ret.shape[0] == 2: ret = _N.vstack((ret, 0.)) ret.Format = FMT_VECTOR_DEFAULT ret= _N.ndarray.__new__(cls, ret.shape, dtype, buffer=ret.data) return ret def __str__(self): fmt = getattr(self, "Format", FMT_VECTOR_DEFAULT) fmt = ', '.join([fmt]*3) return ''.join(["(", fmt, ")"]) % tuple(self.T.tolist()[0]) def __repr__(self): fmt = ', '.join(['%s']*3) return ''.join(["%s([", fmt, "])"]) % tuple([self.__class__.__name__] + self.T.tolist()[0]) def __mul__(self, mult): ''' self * multiplicand ''' if isinstance(mult, _N.matrix): return _N.dot(self, mult) else: raise DataError, 'multiplicand must be a Vector or a matrix' def __rmul__(self, mult): ''' multiplier * self.__mul__ ''' if isinstance(mult, _N.matrix): return Vector(_N.dot(mult, self)) else: raise DataError, 'multiplier must be a Vector or a matrix' #### the remaining methods are Vector-specific math operations, including the X,Y,Z properties... if __name__ == '__main__': u = Vector('1 2 3') print str(u) print repr(u) A = _N.matrix('2 0 0; 0 2 0; 0 0 2') print A p = A * u print p print p.__class__ q= u.T * A try: print q except: print "we don't allow for the display of row vectors" print q.A, q.T print q.__class__ From cjw at sympatico.ca Sat Jan 12 19:43:56 2008 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat, 12 Jan 2008 19:43:56 -0500 Subject: [Numpy-discussion] subclassing matrix In-Reply-To: References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> Message-ID: <47895ECC.8040700@sympatico.ca> Basilisk96 wrote: > On Jan 12, 1:36 am, "Timothy Hochberg" wrote: >> I believe that you need to look at __array_finalize__ and __array_priority__ >> (and there may be one other thing as well, I can't remember; it's late). >> Search for __array_finalize__ and that will probably help get you started. >> > > Well sonovagun! > I removed the hack. > Then just by setting __array_priority__ = 20.0 in the class body, > things are magically working almost as I expect. I say "almost" > because of this custom method: > > def cross(self, other): > """Cross product of this vector and another vector""" > return _N.cross(self, other, axis=0) > > That call to numpy.cross returns a numpy.ndarray. Unless I do return > Vector(_N.cross(self, other, axis=0)), I get problems downstream. > > When is __array_finalize__ called? By adding some print traces, I can > see it's called every time an array is modified in any way i.e., > reshaped, transposed, etc., and also during operations like u+v, u-v, > A*u. But it's not called during the call to numpy.cross. Why? > > Cheers, > -Basilisk96 This may help. It is based on your initial script. The Vectors are considered as columns but presented as rows. This adds a complication which is not resolved. Colin W. #---------- vector.py import numpy as _N import math as _M #default tolerance for equality tests TOL_EQ = 1e-6 #default format for pretty-printing Vector instances FMT_VECTOR_DEFAULT = "%+.5f" class Vector(_N.matrix): """ 2D/3D vector class that supports numpy matrix operations and more. Examples: u = Vector([1,2,3]) v = Vector('3 4 5') w = Vector([1, 2]) """ def __new__(cls, data="0. 0. 0.", dtype=_N.float64): """ Subclass instance constructor. If data is not specified, a zero Vector is constructed. The constructor always returns a Vector instance. The instance gets a customizable Format attribute, which controls the printing precision. """ data= [1, 2, 3] ret= _N.matrix(data, dtype) ## ret = super(Vector, cls).__new__(cls, data, dtype=dtype) ## #promote the instance to cls type. ## ret.__class__ = cls assert ret.size in (2, 3), 'Vector must have either two or three components' if ret.shape[0] == 1: ret = ret.T assert ret.shape == (ret.shape[0], 1), 'could not express Vector as a Mx1 matrix' if ret.shape[0] == 2: ret = _N.vstack((ret, 0.)) ret.Format = FMT_VECTOR_DEFAULT ret= _N.ndarray.__new__(cls, ret.shape, dtype, buffer=ret.data) return ret def __str__(self): fmt = getattr(self, "Format", FMT_VECTOR_DEFAULT) fmt = ', '.join([fmt]*3) return ''.join(["(", fmt, ")"]) % tuple(self.T.tolist()[0]) def __repr__(self): fmt = ', '.join(['%s']*3) return ''.join(["%s([", fmt, "])"]) % tuple([self.__class__.__name__] + self.T.tolist()[0]) def __mul__(self, mult): ''' self * multiplicand ''' if isinstance(mult, _N.matrix): return _N.dot(self, mult) else: raise DataError, 'multiplicand must be a Vector or a matrix' def __rmul__(self, mult): ''' multiplier * self.__mul__ ''' if isinstance(mult, _N.matrix): return Vector(_N.dot(mult, self)) else: raise DataError, 'multiplier must be a Vector or a matrix' #### the remaining methods are Vector-specific math operations, including the X,Y,Z properties... if __name__ == '__main__': u = Vector('1 2 3') print str(u) print repr(u) A = _N.matrix('2 0 0; 0 2 0; 0 0 2') print A p = A * u print p print p.__class__ q= u.T * A try: print q except: print "we don't allow for the display of row vectors" print q.A, q.T print q.__class__ From basilisk96 at gmail.com Sat Jan 12 22:59:20 2008 From: basilisk96 at gmail.com (Basilisk96) Date: Sat, 12 Jan 2008 19:59:20 -0800 (PST) Subject: [Numpy-discussion] subclassing matrix In-Reply-To: <47895ECC.8040700@sympatico.ca> References: <33921db4-3df7-4b75-8b62-c23863bd3609@k39g2000hsf.googlegroups.com> <68f480d5-3823-45a1-958d-844bc07efc1c@e6g2000prf.googlegroups.com> <47895ECC.8040700@sympatico.ca> Message-ID: <04b25e82-6c44-4492-a28d-4d3c42e75b42@v29g2000hsf.googlegroups.com> Thanks Stefan and Colin, The subclass documentation made this a little clearer now. Instead of using a super() call in __new__, I now do this: #construct a matrix based on the input ret = _N.matrix(data, dtype=dtype) #promote it to Vector ret = ret.view(cls) The second statement there also invokes a call to __array_finalize__, which ensures that the Format attribute is attached to the instance. Then I validate and manipulate the shape of the matrix so that it is a (3,1) vector. Now the instance builder is proper. my __array_finalize__ looks like this: def __array_finalize__(self, obj): self.Format = getattr(obj, "Format", FMT_VECTOR_DEFAULT) return Also, after learning about the view method, I am now doing the following in Vector.cross: def cross(self, other): """Cross product of this vector and another vector""" #calling the result's view method promotes it to Vector result = _N.cross(self, other, axis=0) result = result.view(type(self)) return result It also works if I just use Vector(...), but it is my gut feeling that invoking a view vs. instantiating a new Vector instance is cheaper? Now I can add more unit tests. Cheers, -Basilisk96 From matthew.brett at gmail.com Sun Jan 13 14:03:00 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Jan 2008 19:03:00 +0000 Subject: [Numpy-discussion] Update on using scons to build numpy In-Reply-To: <47888019.1090007@ar.media.kyoto-u.ac.jp> References: <47888019.1090007@ar.media.kyoto-u.ac.jp> Message-ID: <1e2af89e0801131103s4147b195lededb65934773eea@mail.gmail.com> Hi, > A quick email to give an update on my work to build numpy with > scons. I've finished a few days ago to make my former work a separate > package from numpy: it was more work than I expected because of > bootstrapping issues, but I can now build numpy again with the new > package on Linux. Just to thank you very much for your work on this. Matthew From ndbecker2 at gmail.com Sun Jan 13 18:14:29 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 13 Jan 2008 18:14:29 -0500 Subject: [Numpy-discussion] casting Message-ID: numpy frequently refers to 'casting'. I'm not sure if that term is ever defined. I believe it has the same meaning as in C. In that case, it is unfortunately used to mean 2 different things. There are casts that do not change the underlying bits (such as a pointer cast), and there are casts that actually convert to different bits (such as float -> double). I think numpy means the latter. When an array where the underlying data is one type, a cast to another type means actually reallocating and converting the data. It often occurs that I have an algorithm that can take any integral type, because it is written with c++ templates. In that case, I don't want to use PyArray_FROMANY, because I don't want to unecessarily convert the array data. Instead, I'd like to inquire what is the preferred type of the data. The solution I'm exploring is to use a function I call 'preferred_array_type'. This uses the __array_struct__ interface to find the native data type. I chose to use this interface, because then it will work with both numpy arrays and other array-like types. Any thoughts on all of this? Particularly, my observation that numpy seems to want me to tell it what data type of array my algorithm wants, but doesn't seem to provide a good mechanism to allow me to inquire what is the native type to avoid conversion. From robert.kern at gmail.com Sun Jan 13 22:23:55 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 13 Jan 2008 21:23:55 -0600 Subject: [Numpy-discussion] casting In-Reply-To: References: Message-ID: <478AD5CB.6000800@gmail.com> Neal Becker wrote: > numpy frequently refers to 'casting'. I'm not sure if that term is ever > defined. I believe it has the same meaning as in C. In that case, it is > unfortunately used to mean 2 different things. There are casts that do not > change the underlying bits (such as a pointer cast), and there are casts > that actually convert to different bits (such as float -> double). > > I think numpy means the latter. When an array where the underlying data is > one type, a cast to another type means actually reallocating and converting > the data. Yes, that is usually what people mean when they use _casting_ in the context of numpy. It is the more frequently performed operation of the two. The former can be accomplished with the .view(dtype) method of ndarrays. > It often occurs that I have an algorithm that can take any integral type, > because it is written with c++ templates. In that case, I don't want to > use PyArray_FROMANY, because I don't want to unecessarily convert the array > data. Instead, I'd like to inquire what is the preferred type of the data. > > The solution I'm exploring is to use a function I > call 'preferred_array_type'. This uses the __array_struct__ interface to > find the native data type. I chose to use this interface, because then it > will work with both numpy arrays and other array-like types. > > Any thoughts on all of this? I'm not sure what you mean by "preferred type of the data". Do you mean the dtype of the array as it comes in? There are several functions and function macros in the numpy C API which take differing amounts of information. For example, * PyArray_FROM_O(PyObject*onj) just takes an object. * PyArray_FROM_OF(PyObject* obj, int req) takes an object and flags like NPY_CONTIGUOUS. * PyArray_FROM_OT(PyObject* obj, int typenum) takes an object and a type number. * PyArray_FROM_OTF(PyObject* obj, int typenum, int req) takes an object, type, and flags. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Mon Jan 14 05:55:08 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Jan 2008 19:55:08 +0900 Subject: [Numpy-discussion] Update on using scons to build numpy In-Reply-To: <1e2af89e0801131103s4147b195lededb65934773eea@mail.gmail.com> References: <47888019.1090007@ar.media.kyoto-u.ac.jp> <1e2af89e0801131103s4147b195lededb65934773eea@mail.gmail.com> Message-ID: <478B3F8C.3040209@ar.media.kyoto-u.ac.jp> Matthew Brett wrote: > Hi, > > >> A quick email to give an update on my work to build numpy with >> scons. I've finished a few days ago to make my former work a separate >> package from numpy: it was more work than I expected because of >> bootstrapping issues, but I can now build numpy again with the new >> package on Linux. >> > > Just to thank you very much for your work on this. > Thanks. If you are willing to test, please submit any bugs in launchpad, or on this ML, cheers, David From ndbecker2 at gmail.com Mon Jan 14 07:08:52 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 14 Jan 2008 07:08:52 -0500 Subject: [Numpy-discussion] casting References: <478AD5CB.6000800@gmail.com> Message-ID: Robert Kern wrote: > Neal Becker wrote: >> numpy frequently refers to 'casting'. I'm not sure if that term is ever >> defined. I believe it has the same meaning as in C. In that case, it is >> unfortunately used to mean 2 different things. There are casts that do >> not change the underlying bits (such as a pointer cast), and there are >> casts that actually convert to different bits (such as float -> double). >> >> I think numpy means the latter. When an array where the underlying data >> is one type, a cast to another type means actually reallocating and >> converting the data. > > Yes, that is usually what people mean when they use _casting_ in the > context of numpy. It is the more frequently performed operation of the > two. The former can be accomplished with the .view(dtype) method of > ndarrays. > >> It often occurs that I have an algorithm that can take any integral type, >> because it is written with c++ templates. In that case, I don't want to >> use PyArray_FROMANY, because I don't want to unecessarily convert the >> array >> data. Instead, I'd like to inquire what is the preferred type of the >> data. >> >> The solution I'm exploring is to use a function I >> call 'preferred_array_type'. This uses the __array_struct__ interface to >> find the native data type. I chose to use this interface, because then >> it will work with both numpy arrays and other array-like types. >> >> Any thoughts on all of this? > > I'm not sure what you mean by "preferred type of the data". Do you mean > the dtype of the array as it comes in? There are several functions and > function macros in the numpy C API which take differing amounts of > information. For example, > > * PyArray_FROM_O(PyObject*onj) just takes an object. > * PyArray_FROM_OF(PyObject* obj, int req) takes an object and flags like > NPY_CONTIGUOUS. > * PyArray_FROM_OT(PyObject* obj, int typenum) takes an object and a type > number. > * PyArray_FROM_OTF(PyObject* obj, int typenum, int req) takes an object, > type, > and flags. > Let me try again to explain. I don't want to convert to some type first - that would be a waste. I need to find out what is the native data type of the input array first. Also, I'd like to allow that the input is not a PyArray, but could be something conforming to __array_struct__ interface. So, I need to find the native data type _first_, _then_ call the appropriate PyArray_FROM... Further, I don't believe this requirement is unique. I would think it would be needed for any time when a user wants to create a function that can accept a numpy array, and would like to avoid unnecessary data conversion. This is particularly true when the underlying function is using c++ templates to allow the data type to be a template parameter (and so can operate on any - or a range - of data types). From matthew.brett at gmail.com Mon Jan 14 07:21:00 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Jan 2008 12:21:00 +0000 Subject: [Numpy-discussion] Nose testing for numpy Message-ID: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> Hi, I've just finished moving the scipy tests over to nose. Thinking about it, it seems to me to be a good idea to do the same for numpy. The advantages of doing this now are that numpy and scipy would be in parallel, that we can continue to have one testing system for both, and that it would be clear to both numpy and scipy developers that they should not use NumpyTest but the nose test framework. At the moment, you can still find yourself using the numpy test framework in scipy, and the tests will work - but it should be deprecated. Ideally, to make a clean switch, I think using numpytest should raise an error. One issue is that the scipy nose test labels use decorators, for which the standard syntax requires python 2.4. To avoid this, we can just apply the decorators with the def test_func(): pass test_func = dec.slow(test_func) syntax. It's a rather moot point, as the decorators are mainly used to define test level, there's only one extra test found for numpy.test(10) compared to numpy.test(). I think I could do the port in quite a short time. Any thoughts? Matthew From robert.kern at gmail.com Mon Jan 14 11:07:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Jan 2008 10:07:30 -0600 Subject: [Numpy-discussion] casting In-Reply-To: References: <478AD5CB.6000800@gmail.com> Message-ID: <478B88C2.7040609@gmail.com> Neal Becker wrote: > Robert Kern wrote: > >> Neal Becker wrote: >>> numpy frequently refers to 'casting'. I'm not sure if that term is ever >>> defined. I believe it has the same meaning as in C. In that case, it is >>> unfortunately used to mean 2 different things. There are casts that do >>> not change the underlying bits (such as a pointer cast), and there are >>> casts that actually convert to different bits (such as float -> double). >>> >>> I think numpy means the latter. When an array where the underlying data >>> is one type, a cast to another type means actually reallocating and >>> converting the data. >> Yes, that is usually what people mean when they use _casting_ in the >> context of numpy. It is the more frequently performed operation of the >> two. The former can be accomplished with the .view(dtype) method of >> ndarrays. >> >>> It often occurs that I have an algorithm that can take any integral type, >>> because it is written with c++ templates. In that case, I don't want to >>> use PyArray_FROMANY, because I don't want to unecessarily convert the >>> array >>> data. Instead, I'd like to inquire what is the preferred type of the >>> data. >>> >>> The solution I'm exploring is to use a function I >>> call 'preferred_array_type'. This uses the __array_struct__ interface to >>> find the native data type. I chose to use this interface, because then >>> it will work with both numpy arrays and other array-like types. >>> >>> Any thoughts on all of this? >> I'm not sure what you mean by "preferred type of the data". Do you mean >> the dtype of the array as it comes in? There are several functions and >> function macros in the numpy C API which take differing amounts of >> information. For example, >> >> * PyArray_FROM_O(PyObject*onj) just takes an object. >> * PyArray_FROM_OF(PyObject* obj, int req) takes an object and flags like >> NPY_CONTIGUOUS. >> * PyArray_FROM_OT(PyObject* obj, int typenum) takes an object and a type >> number. >> * PyArray_FROM_OTF(PyObject* obj, int typenum, int req) takes an object, >> type, >> and flags. >> > > Let me try again to explain. I don't want to convert to some type first - > that would be a waste. I need to find out what is the native data type of > the input array first. Also, I'd like to allow that the input is not a > PyArray, but could be something conforming to __array_struct__ interface. > So, I need to find the native data type _first_, _then_ call the > appropriate PyArray_FROM... I'm sorry, I still think we're talking past each other. What do you mean by "native data type"? If you just want to get an ndarray without specifying a type, use PyArray_FROM_O(). That's what it's for. You don't need to know the data type beforehand. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wright at esrf.fr Mon Jan 14 11:58:50 2008 From: wright at esrf.fr (Jon Wright) Date: Mon, 14 Jan 2008 17:58:50 +0100 Subject: [Numpy-discussion] casting In-Reply-To: <478B88C2.7040609@gmail.com> References: <478AD5CB.6000800@gmail.com> <478B88C2.7040609@gmail.com> Message-ID: <478B94CA.6020507@esrf.fr> > I'm sorry, I still think we're talking past each other. What do you mean by > "native data type"? If you just want to get an ndarray without specifying a > type, use PyArray_FROM_O(). That's what it's for. You don't need to know the > data type beforehand. What I have wanted in the past (and what I thought Neal was after) is a way to choose which function to call according to the typecode of the data as it is currently in memory. I don't want to convert (or cast or even touch the data) but just call a type specific function instead. C++ templates can take some of the tedium out of that, but in some cases algorithms may be different too. Guessing which sort algorithm to use springs to mind. Rather than saying "give me the right kind of array", I think there is an interest in saying "choose which function is the best for this data". Something like: PyArrayObject* array = PyArray_FROM_O( (PyObject*) O ); type = array -> descr -> type_num ; switch (type){ case NPY_BYTE : signed_func(array); case NPY_UBYTE : unsigned_func(array); // etc It sort of implies having a C++ type hierarchy for numpy arrays and casting array to be a PyFloatArray or PyDoubleArray etc? The extra confusion might be due to the way arrays can be laid out in memory - indexing into array slices is not always obvious. Also if you want to make sure your inner loop goes over the "fast" index you might want an algorithm which reads the strides when it runs. Sorry if I've only added to the confusion. Cheers, Jon From ndbecker2 at gmail.com Mon Jan 14 13:59:15 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 14 Jan 2008 13:59:15 -0500 Subject: [Numpy-discussion] casting References: <478AD5CB.6000800@gmail.com> <478B88C2.7040609@gmail.com> <478B94CA.6020507@esrf.fr> Message-ID: Jon Wright wrote: >> I'm sorry, I still think we're talking past each other. What do you mean >> by "native data type"? If you just want to get an ndarray without >> specifying a type, use PyArray_FROM_O(). That's what it's for. You don't >> need to know the data type beforehand. > > What I have wanted in the past (and what I thought Neal was after) is a > way to choose which function to call according to the typecode of the > data as it is currently in memory. I don't want to convert (or cast or > even touch the data) but just call a type specific function instead. C++ > templates can take some of the tedium out of that, but in some cases > algorithms may be different too. Guessing which sort algorithm to use > springs to mind. > > Rather than saying "give me the right kind of array", I think there is > an interest in saying "choose which function is the best for this data". > Something like: > > PyArrayObject* array = PyArray_FROM_O( (PyObject*) O ); > type = array -> descr -> type_num ; > switch (type){ > case NPY_BYTE : signed_func(array); > case NPY_UBYTE : unsigned_func(array); > // etc > > It sort of implies having a C++ type hierarchy for numpy arrays and > casting array to be a PyFloatArray or PyDoubleArray etc? > > The extra confusion might be due to the way arrays can be laid out in > memory - indexing into array slices is not always obvious. Also if you > want to make sure your inner loop goes over the "fast" index you might > want an algorithm which reads the strides when it runs. > > Sorry if I've only added to the confusion. > > Cheers, > > Jon This is close to what I'm doing. If I really can handle any type, then FROM_O is fine. Commonly, it's a little more complicated. Here is an example, in pseudo-code (real code is in c++): if (native_type_of (x) is int or long): do_something_with (convert_to_long_array(x)) elif (native_type_of (x) is complex): do_something_with (convert_to_complex_array (x)) In the above, native_type_of means: if (hasattr (x, "__array_struct__")): array_if = PyCObject_AsVoidPtr (x.__array_struct__) return (map_array_if_to_typenum (array_if)) So this means for any numpy array or any type supporting __array_struct__ protocol, find out the native data type. I don't want to use FROM_O here, because I really can only handle certain types. If I used FROM_O, then after calling FROM_O, if the type was not one I could handle, I'd have to call FromAny and convert it. Or, in the case above, if I was given an array of int32 which I'd want to handle as long (int64), I'd have to convert it again. From fperez.net at gmail.com Mon Jan 14 14:36:06 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 14 Jan 2008 12:36:06 -0700 Subject: [Numpy-discussion] Nose testing for numpy In-Reply-To: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> References: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> Message-ID: On Jan 14, 2008 5:21 AM, Matthew Brett wrote: > Hi, > > I've just finished moving the scipy tests over to nose. > > Thinking about it, it seems to me to be a good idea to do the same for numpy. > Any thoughts? A big +1 from me. Cheers, f From ndbecker2 at gmail.com Mon Jan 14 14:37:54 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 14 Jan 2008 14:37:54 -0500 Subject: [Numpy-discussion] RFC: out of range slice indexes Message-ID: I've never liked that python silently ignores slices with out of range indexes. I believe this is a source of bugs (it has been for me). It goes completely counter to the python philosophy. I vote to ban them from numpy. >>> from numpy import array >>> x = array (xrange (10)) >>> x[11] Traceback (most recent call last): File "", line 1, in IndexError: index out of bounds >>> x[:12] = 2 >>> x array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>> len (x) 10 Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm not advocating that position). I believe that out of bounds indexes should always throw IndexError. We can't change that in Python now, but maybe we can in numpy. From robert.kern at gmail.com Mon Jan 14 14:44:19 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Jan 2008 13:44:19 -0600 Subject: [Numpy-discussion] RFC: out of range slice indexes In-Reply-To: References: Message-ID: <478BBB93.9090206@gmail.com> Neal Becker wrote: > I've never liked that python silently ignores slices with out of range > indexes. I believe this is a source of bugs (it has been for me). It goes > completely counter to the python philosophy. > > I vote to ban them from numpy. >>>> from numpy import array >>>> x = array (xrange (10)) >>>> x[11] > Traceback (most recent call last): > File "", line 1, in > IndexError: index out of bounds >>>> x[:12] = 2 >>>> x > array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>>> len (x) > 10 > > Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to > _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm > not advocating that position). > > I believe that out of bounds indexes should always throw IndexError. We > can't change that in Python now, but maybe we can in numpy. -1. Regardless of the merits if we had a blank slate, there is code that depends on this, specifically my code. It simplifies certain operations that would otherwise need tedious special case-handling. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthew.brett at gmail.com Mon Jan 14 14:49:14 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Jan 2008 11:49:14 -0800 Subject: [Numpy-discussion] Nose testing for numpy In-Reply-To: References: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> Message-ID: <1e2af89e0801141149v63106af8p89c259cd7c84dbdf@mail.gmail.com> An added advantage is that is makes it much easier to run doctests: numpy.test(doctests=True) On Jan 14, 2008 11:36 AM, Fernando Perez wrote: > On Jan 14, 2008 5:21 AM, Matthew Brett wrote: > > Hi, > > > > I've just finished moving the scipy tests over to nose. > > > > Thinking about it, it seems to me to be a good idea to do the same for numpy. > > > Any thoughts? > > A big +1 from me. > > Cheers, > > f > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Mon Jan 14 14:54:04 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 14 Jan 2008 14:54:04 -0500 Subject: [Numpy-discussion] RFC: out of range slice indexes References: <478BBB93.9090206@gmail.com> Message-ID: Robert Kern wrote: > Neal Becker wrote: >> I've never liked that python silently ignores slices with out of range >> indexes. I believe this is a source of bugs (it has been for me). It >> goes completely counter to the python philosophy. >> >> I vote to ban them from numpy. >>>>> from numpy import array >>>>> x = array (xrange (10)) >>>>> x[11] >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: index out of bounds >>>>> x[:12] = 2 >>>>> x >> array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>>>> len (x) >> 10 >> >> Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to >> _extend_ x to have lenght 12, at least _that_ would be reasonable (but >> I'm not advocating that position). >> >> I believe that out of bounds indexes should always throw IndexError. We >> can't change that in Python now, but maybe we can in numpy. > > -1. Regardless of the merits if we had a blank slate, there is code that > depends on this, specifically my code. It simplifies certain operations > that would otherwise need tedious special case-handling. > For example? From oliphant at enthought.com Mon Jan 14 14:59:20 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 14 Jan 2008 13:59:20 -0600 Subject: [Numpy-discussion] Nose testing for numpy In-Reply-To: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> References: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> Message-ID: <478BBF18.9010903@enthought.com> Matthew Brett wrote: > Hi, > > I've just finished moving the scipy tests over to nose. > > Thinking about it, it seems to me to be a good idea to do the same for numpy. > We talked about this at the SciPy Sprint. Eventually, we will get there. However, if we do it before 1.0.5, it will require nose to run the NumPy tests. I'm concerned to make this kind of change, prior to 1.1 So, the strategy is to support what scipy needs for nose testing inside of NumPy for now and wait until 1.1 to move over to requiring nose for all of NumPy tests. Yes, it is not as clean, but I really hesitate making a whole-sale switch at version 1.0.5 -Travis O. From tim.hochberg at ieee.org Mon Jan 14 15:03:51 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 14 Jan 2008 13:03:51 -0700 Subject: [Numpy-discussion] RFC: out of range slice indexes In-Reply-To: References: Message-ID: On Jan 14, 2008 12:37 PM, Neal Becker wrote: > I've never liked that python silently ignores slices with out of range > indexes. I believe this is a source of bugs (it has been for me). It > goes > completely counter to the python philosophy. > > I vote to ban them from numpy. > >>> from numpy import array > >>> x = array (xrange (10)) > >>> x[11] > Traceback (most recent call last): > File "", line 1, in > IndexError: index out of bounds > >>> x[:12] = 2 > >>> x > array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) > >>> len (x) > 10 > > Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to > _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm > not advocating that position). > > I believe that out of bounds indexes should always throw IndexError. We > can't change that in Python now, but maybe we can in numpy. > -1. Regardless of the possible merit of this on it's face, I think this is an area we should maintain compatibility with Python sequences. Not to mention that it would likely break a bunch of code, including my own. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jan 14 15:04:20 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Jan 2008 14:04:20 -0600 Subject: [Numpy-discussion] RFC: out of range slice indexes In-Reply-To: References: <478BBB93.9090206@gmail.com> Message-ID: <478BC044.9080107@gmail.com> Neal Becker wrote: > Robert Kern wrote: > >> Neal Becker wrote: >>> I've never liked that python silently ignores slices with out of range >>> indexes. I believe this is a source of bugs (it has been for me). It >>> goes completely counter to the python philosophy. >>> >>> I vote to ban them from numpy. >>>>>> from numpy import array >>>>>> x = array (xrange (10)) >>>>>> x[11] >>> Traceback (most recent call last): >>> File "", line 1, in >>> IndexError: index out of bounds >>>>>> x[:12] = 2 >>>>>> x >>> array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>>>>> len (x) >>> 10 >>> >>> Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to >>> _extend_ x to have lenght 12, at least _that_ would be reasonable (but >>> I'm not advocating that position). >>> >>> I believe that out of bounds indexes should always throw IndexError. We >>> can't change that in Python now, but maybe we can in numpy. >> -1. Regardless of the merits if we had a blank slate, there is code that >> depends on this, specifically my code. It simplifies certain operations >> that would otherwise need tedious special case-handling. > > For example? def ichunk(arr, chunk_size=10): for i in range(0, len(arr), chunk_size): yield arr[i:i+chunk_size] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthew.brett at gmail.com Mon Jan 14 15:38:05 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Jan 2008 12:38:05 -0800 Subject: [Numpy-discussion] Nose testing for numpy In-Reply-To: <478BBF18.9010903@enthought.com> References: <1e2af89e0801140421v292bf49aj5afa49488832e6a9@mail.gmail.com> <478BBF18.9010903@enthought.com> Message-ID: <1e2af89e0801141238p294136abpc3a95de58a85466e@mail.gmail.com> Hi, > We talked about this at the SciPy Sprint. Eventually, we will get > there. However, if we do it before 1.0.5, it will require nose to run > the NumPy tests. I'm concerned to make this kind of change, prior to 1.1 Ah, sorry, I heard of the conclusion, but had thought it was due to the 2.4 dependency. Matthew From robert.kern at gmail.com Mon Jan 14 16:36:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Jan 2008 15:36:06 -0600 Subject: [Numpy-discussion] casting In-Reply-To: References: <478AD5CB.6000800@gmail.com> <478B88C2.7040609@gmail.com> <478B94CA.6020507@esrf.fr> Message-ID: <478BD5C6.6060607@gmail.com> Neal Becker wrote: > Jon Wright wrote: > >>> I'm sorry, I still think we're talking past each other. What do you mean >>> by "native data type"? If you just want to get an ndarray without >>> specifying a type, use PyArray_FROM_O(). That's what it's for. You don't >>> need to know the data type beforehand. >> What I have wanted in the past (and what I thought Neal was after) is a >> way to choose which function to call according to the typecode of the >> data as it is currently in memory. I don't want to convert (or cast or >> even touch the data) but just call a type specific function instead. C++ >> templates can take some of the tedium out of that, but in some cases >> algorithms may be different too. Guessing which sort algorithm to use >> springs to mind. >> >> Rather than saying "give me the right kind of array", I think there is >> an interest in saying "choose which function is the best for this data". >> Something like: >> >> PyArrayObject* array = PyArray_FROM_O( (PyObject*) O ); >> type = array -> descr -> type_num ; >> switch (type){ >> case NPY_BYTE : signed_func(array); >> case NPY_UBYTE : unsigned_func(array); >> // etc >> >> It sort of implies having a C++ type hierarchy for numpy arrays and >> casting array to be a PyFloatArray or PyDoubleArray etc? >> >> The extra confusion might be due to the way arrays can be laid out in >> memory - indexing into array slices is not always obvious. Also if you >> want to make sure your inner loop goes over the "fast" index you might >> want an algorithm which reads the strides when it runs. >> >> Sorry if I've only added to the confusion. >> >> Cheers, >> >> Jon > > This is close to what I'm doing. If I really can handle any type, then > FROM_O is fine. > > Commonly, it's a little more complicated. > > Here is an example, in pseudo-code (real code is in c++): > > if (native_type_of (x) is int or long): > do_something_with (convert_to_long_array(x)) > elif (native_type_of (x) is complex): > do_something_with (convert_to_complex_array (x)) > > In the above, native_type_of means: > if (hasattr (x, "__array_struct__")): > array_if = PyCObject_AsVoidPtr (x.__array_struct__) > return (map_array_if_to_typenum (array_if)) > > So this means for any numpy array or any type supporting __array_struct__ > protocol, find out the native data type. > > I don't want to use FROM_O here, because I really can only handle certain > types. If I used FROM_O, then after calling FROM_O, if the type was not > one I could handle, I'd have to call FromAny and convert it. Or, in the > case above, if I was given an array of int32 which I'd want to handle as > long (int64), I'd have to convert it again. Okay, I think I see now. I'm not sure what numpy could do to make your code more elegant. I would recommend just using PyArray_FROM_O() or PyArray_EnsureArray() to get a real ndarray object. They should not copy when the object is already an ndarray or if it satisfies the array interface; however, it will also handle the cases when you have a Python-level __array_interface__ or just nested sequences, too. You can look at the PyArray_Descr directly, dispatch the types that you can handle directly, and then convert for the types which you can't. You will not get any extraneous conversions or copies of data. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From william.ratcliff at gmail.com Tue Jan 15 05:40:00 2008 From: william.ratcliff at gmail.com (william ratcliff) Date: Tue, 15 Jan 2008 05:40:00 -0500 Subject: [Numpy-discussion] A question about ctypes and numpy Message-ID: <827183970801150240j3645316dqa117060f5b6dd7d0@mail.gmail.com> Hi! I have been having some difficulty understanding how to deal with multidimensional ndarrays in structs with ctypes. Namely, I want to the define numpy arrays in python and have them operated on in C, and have them updated on the python side. Here is a toy example: c code fragment: foo.c typedef struct { double X[4][4]; double Y[4][4]; } matrices; //there could be other elements in the struct between the arrays, after them, etc. int correct(matrices* pt){ int i=2,j=1; pt->Y[i][j]=5.0; } python: import numpy as N import ctypes xdata=N.zeros((4,4),dtype='float64') ydata=N.ones((4,4),dtype='float64') class fiduc(ctypes.Structure): _fields_=[("X",some_ctype), ("Y",some_ctype)] pt=fiduc() pt.X=xdata.ctypes.data pt.Y=ydata.ctypes.data foo = N.ctypeslib.load_library('foo.dll', '.') foo.correct(ctypes.byref(pt)) print xdata print ydata My question is: If I want to define a class in python and later to have my c code operate on an instance of it, then I need to know the c_type representation for the ndarray. I can't simply blast the data into the structure because due to memory alignment, structs in C are not guaranteed to have contiguous elements in memory--I also don't want to take a risk with for example my mingw generated dll having a different alignment for a structure than what a ctypes made using msvc expects. Thanks!!!!! Cheers, William -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Tue Jan 15 09:11:01 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 15 Jan 2008 09:11:01 -0500 Subject: [Numpy-discussion] Warning: PyUFunc_FromFuncAndData: types, funcs, Message-ID: AFAICT, the 'types' and 'funcs' arguments to PyUFunc_FromFuncAndData are not copied. There is one example in numpy book showing them pointing to static storage, but the doc of PyUFunc_FromFuncAndData fails to point this out. From david.huard at gmail.com Tue Jan 15 16:29:22 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 15 Jan 2008 16:29:22 -0500 Subject: [Numpy-discussion] maskedarray branch Message-ID: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> Hi all, I am trying to install the maskedarray numpy branch for use with the timeseries package and I get the following error when importing numpy: /usr/local/lib64/python2.5/site-packages/numpy/__init__.py in () 45 import random 46 import ctypeslib ---> 47 import ma 48 49 # Make these accessible from numpy name-space /usr/local/lib64/python2.5/site-packages/numpy/ma/__init__.py in () 12 __date__ = '$Date: 2007-10-29 17:18:13 +0200 (Mon, 29 Oct 2007) $' 13 ---> 14 import core 15 from core import * 16 /usr/local/lib64/python2.5/site-packages/numpy/ma/core.py in () 3073 return numpy.inner(fa, fb).view(MaskedArray) 3074 inner.__doc__ = numpy.inner.__doc__ -> 3075 inner.__doc__ += "\n*Notes*:\n Masked values are replaced by 0." 3076 innerproduct = inner 3077 TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' Everything being fresh from svn, I tried installing the maskedarray living in the sandbox, but I had no success either with the installation of timeseries. Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Wed Jan 16 06:26:20 2008 From: robince at gmail.com (Robin) Date: Wed, 16 Jan 2008 11:26:20 +0000 Subject: [Numpy-discussion] Test failure with latest SVN Message-ID: I am on OS X 10.5.1 and get a test failure with the latest numpy svn. (Previously ran with no errors with svn as of a few weeks ago). test_zero_probability (numpy.tests.test_random.TestMultinomial) ... ok ====================================================================== ERROR: Ticket #396 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line 600, in check_poly1d_nan_roots self.failUnlessRaises(np.linalg.LinAlgError,getattr,p,"r") File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 623, in __getattr__ return roots(self.coeffs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 124, in roots roots = _eigvals(A) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 37, in _eigvals return eigvals(arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", line 378, in eigvals return eig(a,b=b,left=0,right=0,overwrite_a=overwrite_a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", line 128, in eig a1 = asarray_chkfinite(a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/function_base.py", line 398, in asarray_chkfinite raise ValueError, "array must not contain infs or NaNs" ValueError: array must not contain infs or NaNs ---------------------------------------------------------------------- Ran 713 tests in 0.697s FAILED (errors=1) Out[3]: Robin From ndbecker2 at gmail.com Wed Jan 16 07:09:38 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 16 Jan 2008 07:09:38 -0500 Subject: [Numpy-discussion] RFC: out of range slice indexes References: Message-ID: Neal Becker wrote: > I've never liked that python silently ignores slices with out of range > indexes. I believe this is a source of bugs (it has been for me). It > goes completely counter to the python philosophy. > > I vote to ban them from numpy. >>>> from numpy import array >>>> x = array (xrange (10)) >>>> x[11] > Traceback (most recent call last): > File "", line 1, in > IndexError: index out of bounds >>>> x[:12] = 2 >>>> x > array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) >>>> len (x) > 10 > > Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to > _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm > not advocating that position). > > I believe that out of bounds indexes should always throw IndexError. We > can't change that in Python now, but maybe we can in numpy. OK, here's another idea: How about an _optional_ warning, which relies on numpy.warn setting? Could be a big aid in debugging. From meine at informatik.uni-hamburg.de Wed Jan 16 11:27:21 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 16 Jan 2008 17:27:21 +0100 Subject: [Numpy-discussion] casting In-Reply-To: References: <478B94CA.6020507@esrf.fr> Message-ID: <200801161727.22162.meine@informatik.uni-hamburg.de> Am Montag, 14. Januar 2008 19:59:15 schrieb Neal Becker: > I don't want to use FROM_O here, because I really can only handle certain > types. If I used FROM_O, then after calling FROM_O, if the type was not > one I could handle, I'd have to call FromAny and convert it. What is the problem with that? Is there any, apart from the code not being as elegant as one might want it to be? AFAICS, it would be nice to have a function that takes a PyObject and a list of supported types and would perform the minimal needed conversion. OTOH, if your code is templatized, it is maybe already written in an STL-style way, using iterators? Then, it would be very simple to provide a set of iterators that e.g. promote unsigned short->int such that your algorithm can work on the original data without creating a copy. Note that internally, ufuncs already do what you want. I have not yet found the perfect way for a NumPy/C++ integration (which I am interested in, possibly using boost::python which I have been doing for the last years), but for element-wise operators, it would be very cool to be able to re-use the ufunc mechanism (i.e. only implement the inner loop, not caring about non-contiguous arrays etc.). Ciao, / / /--/ / / ANS From denisbz at t-online.de Wed Jan 16 11:49:03 2008 From: denisbz at t-online.de (denis bzowy) Date: Wed, 16 Jan 2008 17:49:03 +0100 Subject: [Numpy-discussion] gcc not g++ for build numpy on mac 10.4 ? Message-ID: <478E357F.1060503@t-online.de> An HTML attachment was scrubbed... URL: From kchristman54 at yahoo.com Wed Jan 16 12:46:01 2008 From: kchristman54 at yahoo.com (Kevin Christman) Date: Wed, 16 Jan 2008 09:46:01 -0800 (PST) Subject: [Numpy-discussion] numpy quit working Message-ID: <825370.690.qm@web38709.mail.mud.yahoo.com> I've started to use numpy (1.0.4) and matplotlib but it suddenly quit working. Now when I import a simple test script: import numpy numpy.linspace(0,1) it gives the following error: >>> Traceback (most recent call last): File "D:/anothertest.py", line 1, in import numpy File "C:\Python25\lib\site-packages\numpy\__init__.py", line 37, in import testing File "C:\Python25\lib\site-packages\numpy\testing\__init__.py", line 3, in from numpytest import * File "C:\Python25\lib\site-packages\numpy\testing\numpytest.py", line 19, in from numpy.distutils.exec_command import splitcmdline File "C:\Python25\lib\site-packages\numpy\distutils\__init__.py", line 6, in import ccompiler File "C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py", line 46, in replace_method(CCompiler, 'spawn', CCompiler_spawn) File "C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py", line 24, in replace_method m = new.instancemethod(func, None, klass) AttributeError: 'module' object has no attribute 'instancemethod' >>> However, other import statements, like "import time" work ok. The problem first started when a script would only run if on the root of the drive (e.g. D:\), but now even if I run it from there it doesn't work. What's going on? Kevin ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ From robert.kern at gmail.com Wed Jan 16 13:01:13 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Jan 2008 12:01:13 -0600 Subject: [Numpy-discussion] numpy quit working In-Reply-To: <825370.690.qm@web38709.mail.mud.yahoo.com> References: <825370.690.qm@web38709.mail.mud.yahoo.com> Message-ID: <478E4669.1090301@gmail.com> Kevin Christman wrote: > I've started to use numpy (1.0.4) and matplotlib but it suddenly quit working. Now when I import a simple test script: > import numpy > numpy.linspace(0,1) > > it gives the following error: > > Traceback (most recent call last): > File "D:/anothertest.py", line 1, in > import numpy > File "C:\Python25\lib\site-packages\numpy\__init__.py", line 37, in > import testing > File "C:\Python25\lib\site-packages\numpy\testing\__init__.py", line 3, in > from numpytest import * > File "C:\Python25\lib\site-packages\numpy\testing\numpytest.py", line 19, in > from numpy.distutils.exec_command import splitcmdline > File "C:\Python25\lib\site-packages\numpy\distutils\__init__.py", line 6, in > import ccompiler > File "C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py", line 46, in > replace_method(CCompiler, 'spawn', CCompiler_spawn) > File "C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py", line 24, in replace_method > m = new.instancemethod(func, None, klass) > AttributeError: 'module' object has no attribute 'instancemethod' > > However, other import statements, like "import time" work ok. > The problem first started when a script would only run if on the root of the drive (e.g. D:\), but now even if I run it from there it doesn't work. What's going on? Do you have a module called new.py somewhere? If so, then it is overriding the standard library's "new" module which we use. >>> import new >>> new If that filename does not point to 'C:\Python25\lib\new.pyc', then you have a problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndbecker2 at gmail.com Wed Jan 16 13:30:03 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 16 Jan 2008 13:30:03 -0500 Subject: [Numpy-discussion] casting References: <478B94CA.6020507@esrf.fr> <200801161727.22162.meine@informatik.uni-hamburg.de> Message-ID: Hans Meine wrote: > Am Montag, 14. Januar 2008 19:59:15 schrieb Neal Becker: >> I don't want to use FROM_O here, because I really can only handle certain >> types. If I used FROM_O, then after calling FROM_O, if the type was not >> one I could handle, I'd have to call FromAny and convert it. > What is the problem with that? Is there any, apart from the code not > being as elegant as one might want it to be? > > AFAICS, it would be nice to have a function that takes a PyObject and a > list > of supported types and would perform the minimal needed conversion. OTOH, > if your code is templatized, it is maybe already written in an STL-style > way, > using iterators? Then, it would be very simple to provide a set of > iterators that e.g. promote unsigned short->int such that your algorithm > can work on the original data without creating a copy. > > Note that internally, ufuncs already do what you want. I have not yet > found the perfect way for a NumPy/C++ integration (which I am interested > in, possibly using boost::python which I have been doing for the last > years), but for element-wise operators, it would be very cool to be able > to re-use the ufunc mechanism (i.e. only implement the inner loop, not > caring about non-contiguous arrays etc.). > I have been making some good progress on numpy/c++ integration. I can share some examples if there is interest. From kchristman54 at yahoo.com Wed Jan 16 13:46:12 2008 From: kchristman54 at yahoo.com (Kevin Christman) Date: Wed, 16 Jan 2008 10:46:12 -0800 (PST) Subject: [Numpy-discussion] numpy quit workinge Message-ID: <591650.97374.qm@web38703.mail.mud.yahoo.com> That's it. I had named one of my files new.py. Thanks so much; now I have one less evidence that I'm going crazy. Kevin ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From charlesr.harris at gmail.com Wed Jan 16 14:47:05 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Jan 2008 12:47:05 -0700 Subject: [Numpy-discussion] casting In-Reply-To: References: <478B94CA.6020507@esrf.fr> <200801161727.22162.meine@informatik.uni-hamburg.de> Message-ID: On Jan 16, 2008 11:30 AM, Neal Becker wrote: > Hans Meine wrote: > > > Am Montag, 14. Januar 2008 19:59:15 schrieb Neal Becker: > >> I don't want to use FROM_O here, because I really can only handle > certain > >> types. If I used FROM_O, then after calling FROM_O, if the type was > not > >> one I could handle, I'd have to call FromAny and convert it. > > What is the problem with that? Is there any, apart from the code not > > being as elegant as one might want it to be? > > > > AFAICS, it would be nice to have a function that takes a PyObject and a > > list > > of supported types and would perform the minimal needed conversion. > OTOH, > > if your code is templatized, it is maybe already written in an STL-style > > way, > > using iterators? Then, it would be very simple to provide a set of > > iterators that e.g. promote unsigned short->int such that your algorithm > > can work on the original data without creating a copy. > > > > Note that internally, ufuncs already do what you want. I have not yet > > found the perfect way for a NumPy/C++ integration (which I am interested > > in, possibly using boost::python which I have been doing for the last > > years), but for element-wise operators, it would be very cool to be able > > to re-use the ufunc mechanism (i.e. only implement the inner loop, not > > caring about non-contiguous arrays etc.). > > > > I have been making some good progress on numpy/c++ integration. I can > share > some examples if there is interest. > I would be interested in that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From HansJoachim.Ehlers at eumetsat.int Thu Jan 17 15:25:41 2008 From: HansJoachim.Ehlers at eumetsat.int (Hans-Joachim Ehlers) Date: Thu, 17 Jan 2008 21:25:41 +0100 Subject: [Numpy-discussion] Problem during build of scipy: - TypeError: can only concatenate list (not "str") to list Message-ID: <478FC7C6.8350.00AE.3@eumetsat.int> Given: AIX 5.3 xlc v8 xlf v10 essl v4.2 Build and installed: numpy 1.0.4 Build and installed: lapack 3.1.1 with CCI and essl support Task: Trying to build scipy 0.6.0 Get the following error which seems to be numpy related: .... extending extension 'scipy.linsolve._csuperlu' defined_macros with [('USE_VENDOR_BLAS', 1)] extending extension 'scipy.linsolve._ssuperlu' defined_macros with [('USE_VENDOR_BLAS', 1)] customize UnixCCompiler customize UnixCCompiler using build_ext Traceback (most recent call last): File "setup.py", line 53, in setup_package() File "setup.py", line 45, in setup_package configuration=configuration ) File "/usr/local/lib/python2.5/site-packages/numpy/distutils/core.py", line 176, in setup return old_setup(**new_attr) File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/usr/local/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/usr/local/lib/python2.5/site-packages/numpy/distutils/command/build_ext.py", line 166, in run self._cxx_compiler = compiler.cxx_compiler() File "/usr/local/lib/python2.5/site-packages/numpy/distutils/ccompiler.py", line 303, in CCompiler_cxx_compiler + cxx.linker_so[2:] TypeError: can only concatenate list (not "str") to list Any tips ? tia Hajo From ndbecker2 at gmail.com Thu Jan 17 16:23:41 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 17 Jan 2008 16:23:41 -0500 Subject: [Numpy-discussion] casting References: <478B94CA.6020507@esrf.fr> <200801161727.22162.meine@informatik.uni-hamburg.de> Message-ID: Charles R Harris wrote: > On Jan 16, 2008 11:30 AM, Neal Becker wrote: > >> Hans Meine wrote: >> >> > Am Montag, 14. Januar 2008 19:59:15 schrieb Neal Becker: >> >> I don't want to use FROM_O here, because I really can only handle >> certain >> >> types. If I used FROM_O, then after calling FROM_O, if the type was >> not >> >> one I could handle, I'd have to call FromAny and convert it. >> > What is the problem with that? Is there any, apart from the code not >> > being as elegant as one might want it to be? >> > >> > AFAICS, it would be nice to have a function that takes a PyObject and a >> > list >> > of supported types and would perform the minimal needed conversion. >> OTOH, >> > if your code is templatized, it is maybe already written in an >> > STL-style way, >> > using iterators? Then, it would be very simple to provide a set of >> > iterators that e.g. promote unsigned short->int such that your >> > algorithm can work on the original data without creating a copy. >> > >> > Note that internally, ufuncs already do what you want. I have not yet >> > found the perfect way for a NumPy/C++ integration (which I am >> > interested in, possibly using boost::python which I have been doing for >> > the last years), but for element-wise operators, it would be very cool >> > to be able to re-use the ufunc mechanism (i.e. only implement the inner >> > loop, not caring about non-contiguous arrays etc.). >> > >> >> I have been making some good progress on numpy/c++ integration. I can >> share >> some examples if there is interest. >> > > I would be interested in that. > > Chuck I placed a little test code and a quick design doc here: https://nbecker.dyndns.org/numpy.ext.tar.bz2 From subscriber100 at rjs.org Thu Jan 17 22:29:50 2008 From: subscriber100 at rjs.org (Ray Schumacher) Date: Thu, 17 Jan 2008 19:29:50 -0800 Subject: [Numpy-discussion] numpy Intel ICC build help? In-Reply-To: References: Message-ID: <6.2.3.4.2.20080117192350.03530280@rjs.org> We've just built Python 2.4 in the lab with Intel ICC and MS VS 2005, but had problems building 2.5, and also numpy, with the MKL. Would someone be willing to share their build experience or project files? Ray -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.19.2/1222 - Release Date: 1/13/2008 12:23 PM From michael.abshoff at googlemail.com Thu Jan 17 22:51:27 2008 From: michael.abshoff at googlemail.com (Michael.Abshoff) Date: Fri, 18 Jan 2008 04:51:27 +0100 Subject: [Numpy-discussion] numpy Intel ICC build help? In-Reply-To: <6.2.3.4.2.20080117192350.03530280@rjs.org> References: <6.2.3.4.2.20080117192350.03530280@rjs.org> Message-ID: <4790223F.3080304@gmail.com> Ray Schumacher wrote: > We've just built Python 2.4 in the lab with Intel ICC and MS VS 2005, > but had problems building 2.5, and also numpy, with the MKL. > Would someone be willing to share their build experience or project files? Which version of the MKL did you use? For example: 9.0 is very different from 9.1. not only in the name of the libraries, but also which libraries you need to link. What was the exact problem? > Ray > Cheers, Michael From stefan at sun.ac.za Fri Jan 18 02:36:43 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 18 Jan 2008 09:36:43 +0200 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> Message-ID: <20080118073643.GL28865@mentat.za.net> Hi David On Tue, Jan 15, 2008 at 04:29:22PM -0500, David Huard wrote: > I am trying to install the maskedarray numpy branch for use with the timeseries > package and I get the following error when importing numpy: Before I try to find the problem: did you remove the previous numpy installation and build directories before installing? Regards St?fan From david at ar.media.kyoto-u.ac.jp Fri Jan 18 04:06:28 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Jan 2008 18:06:28 +0900 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available Message-ID: <47906C14.10104@ar.media.kyoto-u.ac.jp> Hi there, I got a mercurial mirror of numpy available. I put some basic info on the wiki http://scipy.org/scipy/numpy/wiki/HgMirror I don't know mercurial so much, so I only put some instructions to get started. It will force me to get more familiar with the mercurial features which are different from bzr, too (I will put more info then; in particular, merging works a bit differently). Some info: - the import was done with hgsvn. - The mirror contains only the trunk, and not from the start: it looks like hgsvn does not follow svn copy, and as such, it started importing from the scipy_core -> numpy rename. Still, there is the whole trunk history from 2 years. - The .hg directory only takes 17 Mb. With the working tree, it takes 27 Mb, that is only 3 Mb less than a recent svn checkout. - for some reasons, when you clone the hg repository, the tags with svn revisions are lost. If anyone knows how to avoid that (it is really useful to find a hg revision corresponding to a svn revision) cheers, David From ndbecker2 at gmail.com Fri Jan 18 06:35:31 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Jan 2008 06:35:31 -0500 Subject: [Numpy-discussion] PyArray_FromAny min/max dims Message-ID: Why are 0 values of min_depth/max_depth ignored? Why not allow 0, which is a scalar array, and use -1 to mean ignore? From david.huard at gmail.com Fri Jan 18 09:53:33 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 18 Jan 2008 09:53:33 -0500 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <20080118073643.GL28865@mentat.za.net> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> <20080118073643.GL28865@mentat.za.net> Message-ID: <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> 2008/1/18, Stefan van der Walt : > > Hi David > > On Tue, Jan 15, 2008 at 04:29:22PM -0500, David Huard wrote: > > I am trying to install the maskedarray numpy branch for use with the > timeseries > > package and I get the following error when importing numpy: > > Before I try to find the problem: did you remove the previous numpy > installation and build directories before installing? Yes. I am not familiar with the subtleties of module importing, but here would be my guess. The error implies that numpy.inner.__doc__ is None. Now numpy's inner function seems to be defined in the multiarray.c module, and it's doc in add_newdocs.py. So is it possible that importation of the ma module happens before add_newdocs is called ? David Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Jan 18 09:58:44 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 18 Jan 2008 09:58:44 -0500 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> <20080118073643.GL28865@mentat.za.net> <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> Message-ID: <91cf711d0801180658q54963007mc1d66300f9c4cc66@mail.gmail.com> Putting import add_newdocs before import ma seems to fix the ImportError, but I'm not sure if there could be undesirable side effects. David 2008/1/18, David Huard : > > > 2008/1/18, Stefan van der Walt : > > > > Hi David > > > > On Tue, Jan 15, 2008 at 04:29:22PM -0500, David Huard wrote: > > > I am trying to install the maskedarray numpy branch for use with the > > timeseries > > > package and I get the following error when importing numpy: > > > > Before I try to find the problem: did you remove the previous numpy > > installation and build directories before installing? > > > Yes. I am not familiar with the subtleties of module importing, but here > would be my guess. The error implies that numpy.inner.__doc__ is None. Now > numpy's inner function seems to be defined in the multiarray.c module, and > it's doc in add_newdocs.py. So is it possible that importation of the ma > module happens before add_newdocs is called ? > > David > > > Regards > > St?fan > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Jan 18 10:53:28 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 18 Jan 2008 10:53:28 -0500 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <91cf711d0801180658q54963007mc1d66300f9c4cc66@mail.gmail.com> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> <20080118073643.GL28865@mentat.za.net> <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> <91cf711d0801180658q54963007mc1d66300f9c4cc66@mail.gmail.com> Message-ID: <91cf711d0801180753m2f9b11b8yb7ca56135be342e1@mail.gmail.com> Stefan, It seems that the current maskedarray branch is not compatible with the current scipy trunk. Cheers, David 2008/1/18, David Huard : > > Putting > import add_newdocs > before > import ma > > seems to fix the ImportError, but I'm not sure if there could be > undesirable side effects. > > David > > 2008/1/18, David Huard < david.huard at gmail.com>: > > > > > > 2008/1/18, Stefan van der Walt : > > > > > > Hi David > > > > > > On Tue, Jan 15, 2008 at 04:29:22PM -0500, David Huard wrote: > > > > I am trying to install the maskedarray numpy branch for use with the > > > timeseries > > > > package and I get the following error when importing numpy: > > > > > > Before I try to find the problem: did you remove the previous numpy > > > installation and build directories before installing? > > > > > > Yes. I am not familiar with the subtleties of module importing, but here > > would be my guess. The error implies that numpy.inner.__doc__ is None. > > Now numpy's inner function seems to be defined in the multiarray.cmodule, and it's doc in add_newdocs.py. So is it possible that importation > > of the ma module happens before add_newdocs is called ? > > > > David > > > > > > Regards > > > St?fan > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Jan 18 13:35:24 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 18 Jan 2008 20:35:24 +0200 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <47906C14.10104@ar.media.kyoto-u.ac.jp> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> Message-ID: <1200681324.7361.5.camel@localhost.localdomain> pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: > Hi there, > > I got a mercurial mirror of numpy available. I put some basic info > on the wiki > > http://scipy.org/scipy/numpy/wiki/HgMirror Nice! > I don't know mercurial so much, so I only put some instructions to get > started. It will force me to get more familiar with the mercurial > features which are different from bzr, too (I will put more info then; > in particular, merging works a bit differently). > Some info: > - the import was done with hgsvn. > - The mirror contains only the trunk, and not from the start: it > looks like hgsvn does not follow svn copy, and as such, it started > importing from the scipy_core -> numpy rename. Still, there is the whole > trunk history from 2 years. > - The .hg directory only takes 17 Mb. With the working tree, it > takes 27 Mb, that is only 3 Mb less than a recent svn checkout. > - for some reasons, when you clone the hg repository, the tags with > svn revisions are lost. If anyone knows how to avoid that (it is really > useful to find a hg revision corresponding to a svn revision) hgsvn makes only repository-local svn tags. They go in .hg/localtags, from where they can be copied manually. I don't think hgsvn has an option to put the tags into .hgtags (from where they would be passed on automatically when the repository is cloned). -- Pauli Virtanen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digitaalisesti allekirjoitettu viestin osa URL: From rmay at ou.edu Fri Jan 18 13:49:09 2008 From: rmay at ou.edu (Ryan May) Date: Fri, 18 Jan 2008 12:49:09 -0600 Subject: [Numpy-discussion] Ctypes reference counting Message-ID: <4790F4A5.1040808@ou.edu> Hi, Can someone explain the reference counting wrt using ctypes and numpy. Specifically, I have code like: from ctypes import * import numpy as N class Data(Structure): _fields_=[('var',POINTER(c_float))] d = Data() d.var = N.arange(100., dtype=N.float32).ctypes.data_as(POINTER(c_float)) print d.var[5] #Should be 5.0 Instead it prints 0.0. (On my bigger code, it just segfaults.) Obviously, the array going in get's collected b/c its reference count goes to 0. Why does getting the data as a pointer not add a reference? Is there a "correct" way to do what I'm attempting? Overall, I'm trying to create arrays using Numpy that are saved into structures and passed into ctypes-wrapped C code that does the heavy lifting. However, this heavy lifting is done in a different (ie. separate and higher) scope than the code that creates the arrays. Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From oliphant at enthought.com Fri Jan 18 17:13:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 18 Jan 2008 16:13:56 -0600 Subject: [Numpy-discussion] Ctypes reference counting In-Reply-To: <4790F4A5.1040808@ou.edu> References: <4790F4A5.1040808@ou.edu> Message-ID: <479124A4.5040105@enthought.com> Ryan May wrote: > Hi, > > Can someone explain the reference counting wrt using ctypes and numpy. > Specifically, I have code like: > > from ctypes import * > import numpy as N > > class Data(Structure): > _fields_=[('var',POINTER(c_float))] > > d = Data() > d.var = N.arange(100., dtype=N.float32).ctypes.data_as(POINTER(c_float)) > print d.var[5] #Should be 5.0 > > Instead it prints 0.0. (On my bigger code, it just segfaults.) > Obviously, the array going in get's collected b/c its reference count > goes to 0. Why does getting the data as a pointer not add a reference? > The problem is that there is no way to keep a reference to some "memory" in the ctypes "POINTER" object, that I know of. So, when using POINTERS to memory like this you've got to make sure the array does not disappear before you start using it (i.e you have to keep references around yourself). So, the solution is to: 1) Keep track of the memory yourself (by keeping a reference to the array while you call the "heavy-lifting" code). 2) Let another part of the C-code, allocate and free memory and then you just wrap it with NumPy array's for manipulation in Python (i.e. using frombuffer or the ndarray constructor). > Is there a "correct" way to do what I'm attempting? > > Overall, I'm trying to create arrays using Numpy that are saved into > structures and passed into ctypes-wrapped C code that does the heavy > lifting. However, this heavy lifting is done in a different (ie. > separate and higher) scope than the code that creates the arrays. > > I'm not sure what "higher" in this context means. Specific to your example, I would do the following: from ctypes import * import numpy as np d = Data() d.a = np.arange(100.,dtype=np.float32) d.var = d.a.ctypes.data_as(POINTER(c_float)) d.var[5] Now, the memory for a will not go away until you delete d (or d.a) -Travis O. From stefan at sun.ac.za Fri Jan 18 17:23:58 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 19 Jan 2008 00:23:58 +0200 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <91cf711d0801180753m2f9b11b8yb7ca56135be342e1@mail.gmail.com> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> <20080118073643.GL28865@mentat.za.net> <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> <91cf711d0801180658q54963007mc1d66300f9c4cc66@mail.gmail.com> <91cf711d0801180753m2f9b11b8yb7ca56135be342e1@mail.gmail.com> Message-ID: <20080118222358.GD19790@mentat.za.net> Hi David On Fri, Jan 18, 2008 at 10:53:28AM -0500, David Huard wrote: > Stefan, > It seems that the current maskedarray branch is not compatible with the current > scipy trunk. Would you mind expanding on that? Thanks St?fan From ryanlists at gmail.com Fri Jan 18 18:35:13 2008 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri, 18 Jan 2008 17:35:13 -0600 Subject: [Numpy-discussion] Test failure with latest SVN In-Reply-To: References: Message-ID: I just built from svn in Ubuntu and get this same error. Ryan On Jan 16, 2008 5:26 AM, Robin wrote: > I am on OS X 10.5.1 and get a test failure with the latest numpy svn. > (Previously ran with no errors with svn as of a few weeks ago). > > test_zero_probability (numpy.tests.test_random.TestMultinomial) ... ok > > ====================================================================== > ERROR: Ticket #396 > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", > line 600, in check_poly1d_nan_roots > self.failUnlessRaises(np.linalg.LinAlgError,getattr,p,"r") > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/unittest.py", > line 320, in failUnlessRaises > callableObj(*args, **kwargs) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", > line 623, in __getattr__ > return roots(self.coeffs) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", > line 124, in roots > roots = _eigvals(A) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", > line 37, in _eigvals > return eigvals(arg) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", > line 378, in eigvals > return eig(a,b=b,left=0,right=0,overwrite_a=overwrite_a) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", > line 128, in eig > a1 = asarray_chkfinite(a) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/function_base.py", > line 398, in asarray_chkfinite > raise ValueError, "array must not contain infs or NaNs" > ValueError: array must not contain infs or NaNs > > ---------------------------------------------------------------------- > Ran 713 tests in 0.697s > > FAILED (errors=1) > Out[3]: > > Robin > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Fri Jan 18 20:01:58 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Jan 2008 20:01:58 -0500 Subject: [Numpy-discussion] numpy c++ integration Message-ID: If anyone is interested in this subject, I have various tests I've been working on, available via this mercurial repo: https://nbecker.dyndns.org/hg/numpy/ From david at ar.media.kyoto-u.ac.jp Sat Jan 19 00:15:01 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 19 Jan 2008 14:15:01 +0900 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <1200681324.7361.5.camel@localhost.localdomain> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> <1200681324.7361.5.camel@localhost.localdomain> Message-ID: <47918755.6070704@ar.media.kyoto-u.ac.jp> Pauli Virtanen wrote: > pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: >> Hi there, >> >> I got a mercurial mirror of numpy available. I put some basic info >> on the wiki >> >> http://scipy.org/scipy/numpy/wiki/HgMirror > > Nice! Don't hesitate to put more info there. Although I would consider myself relatively proficient with bzr, mercurial has some few important differences that I am still not sure to understand (in particular the lack of "one branch is one directory" concept) > > hgsvn makes only repository-local svn tags. They go in .hg/localtags, > from where they can be copied manually. I don't think hgsvn has an > option to put the tags into .hgtags (from where they would be passed on > automatically when the repository is cloned). Since tags seem to be handled through plain files in mercurial, would copying localtags into .hgtags work ? cheers, David From barrywark at gmail.com Sat Jan 19 02:17:00 2008 From: barrywark at gmail.com (Barry Wark) Date: Fri, 18 Jan 2008 23:17:00 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: Jarrod, I promise: last change. I changed the URL to http://physionconsulting.blogspot.com/search/label/scipy. My wife said physion consultants is a crappy name. Oh well :) barry On Jan 1, 2008 6:35 PM, Jarrod Millman wrote: > On Jan 1, 2008 6:18 PM, Barry Wark wrote: > > I'd like to submit my blog for inclusion too... > > > > http://softwareforscientists.blogspot.com/search/label/scipy > > Hmm. I added you, but while I was adding you it looks like your blog > disappeared. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Sat Jan 19 15:02:24 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 19 Jan 2008 13:02:24 -0700 Subject: [Numpy-discussion] Scipy page Message-ID: Hi All, Just two comments on the current scipy top page: 1) I really miss the recent changes link, it was the only way I could keep current on the Wiki contents. It was also a good way to track spam. 2) The addition of the bug reporting link is good, but it is hard to notice. How about putting it up top using one of the neat python logos with a bug addition? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sswatson at olemiss.edu Sat Jan 19 15:19:05 2008 From: sswatson at olemiss.edu (Sam Watson) Date: Sat, 19 Jan 2008 14:19:05 -0600 Subject: [Numpy-discussion] Problems installing on OS X Message-ID: <540FE9B5-395E-4004-A95A-F9DD2AC07352@olemiss.edu> Dear all, I am running OS X 10.4.11, and when I run gcc -v, I get: Using built-in specs. Target: powerpc-apple-darwin8 Configured with: /private/var/tmp/gcc/gcc-5026.obj~19/src/configure -- disable-checking --prefix=/usr --mandir=/share/man --enable- languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^+.-]*$/ s/$/-4.0/ --with-gxx-include-dir=/include/gcc/darwin/4.0/c++ -- build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 -- target=powerpc-apple-darwin8 Thread model: posix gcc version 4.0.0 (Apple Computer, Inc. build 5026) However, when I move into the numpy-1.0.4 directory and run python setup.py install, I get hundreds of error messages which appear to relate to my C compiler. Here is the output with lots of similar C messages in the middle omitted: non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2_4422 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/ vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands -- compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands -- fcompiler options running build_src building py_modules sources building extension "numpy.core.multiarray" sources Generating build/src.macosx-10.3-fat-2.5/numpy/core/config.h customize NAGFCompiler Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Found executable /sw/bin/f77 absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Found executable /sw/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/ MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 compile options: '-I/Library/Frameworks/Python.framework/Versions/2.5/ include/python2.5 -Inumpy/core/src -Inumpy/core/include -I/Library/ Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c In file included from _configtest.c:2: /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:18:20: error: limits.h: No such file or directory /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:21:2: error: #error "Something's broken. UCHAR_MAX should be defined in limits.h." /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:25:2: error: #error "Python's source code assumes C's unsigned char is an 8-bit type." /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:32:19: error: stdio.h: No such file or directory /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:34:5: error: #error "Python.h requires that stdio.h define NULL." /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:37:20: error: string.h: No such file or directory /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:39:19: error: errno.h: No such file or directory /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:41:20: error: stdlib.h: No such file or directory ... /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ pystrtod.h:11: error: parse error before 'size_t' In file included from _configtest.c:2: /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:134: error: parse error before '*' token /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:134: error: parse error before '*' token /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ Python.h:134: warning: data definition has no type or storage class _configtest.c: In function 'main': _configtest.c:9: error: 'FILE' undeclared (first use in this function) _configtest.c:9: error: (Each undeclared identifier is reported only once _configtest.c:9: error: for each function it appears in.) _configtest.c:9: error: 'fp' undeclared (first use in this function) _configtest.c:17: warning: incompatible implicit declaration of built- in function 'fprintf' lipo: can't figure out the architecture type of: /var/tmp//ccPjVTJJ.out failure. removing: _configtest.c _configtest.o I know that my C compiler works, because I do my research in C on a regular basis. Does anybody have any ideas? Thanks, Sam Watson From efiring at hawaii.edu Sat Jan 19 16:25:02 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 19 Jan 2008 11:25:02 -1000 Subject: [Numpy-discussion] r4730 broke numpy in maskedarray branch Message-ID: <47926AAE.6060407@hawaii.edu> Stefan, The renaming of concatenator to AxisConcatenator in r4730 in the maskedaray branch needs to be propagated into ma/extras.py; I don't know whether there are other places that similar changes are needed. As it stands, after 4730, numpy cannot be imported. Eric From pgmdevlist at gmail.com Sat Jan 19 16:37:06 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 19 Jan 2008 16:37:06 -0500 Subject: [Numpy-discussion] r4730 broke numpy in maskedarray branch In-Reply-To: <47926AAE.6060407@hawaii.edu> References: <47926AAE.6060407@hawaii.edu> Message-ID: <200801191637.08301.pgmdevlist@gmail.com> Eric, My bad, I discovered that eralier this week, created a patch and didn't pass it around. Here it is. Sorry for the inconvenience, Pierre > The renaming of concatenator to AxisConcatenator in r4730 in the > maskedaray branch needs to be propagated into ma/extras.py; I don't know > whether there are other places that similar changes are needed. As it > stands, after 4730, numpy cannot be imported. > > Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: ma_extras_patch Type: text/x-diff Size: 1646 bytes Desc: not available URL: From chanley at stsci.edu Sat Jan 19 21:58:42 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Sat, 19 Jan 2008 21:58:42 -0500 Subject: [Numpy-discussion] Problems installing on OS X In-Reply-To: <540FE9B5-395E-4004-A95A-F9DD2AC07352@olemiss.edu> References: <540FE9B5-395E-4004-A95A-F9DD2AC07352@olemiss.edu> Message-ID: <4792B8E2.8060401@stsci.edu> Greetings, I am by no means an installation expert. However, looking over your build log it looks like the Python you are using is the one that came with Mac OS X. My experience has been that it is necessary to install the Mac OS X Python version available at http://www.python.org/download/. The one that comes installed with OS X seems to be a custom job by Apple. However, I am sure that the true installation experts on the list will have better information or advice for you. Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 Sam Watson wrote: > Dear all, > > I am running OS X 10.4.11, and when I run gcc -v, I get: > > Using built-in specs. > Target: powerpc-apple-darwin8 > Configured with: /private/var/tmp/gcc/gcc-5026.obj~19/src/configure -- > disable-checking --prefix=/usr --mandir=/share/man --enable- > languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^+.-]*$/ > s/$/-4.0/ --with-gxx-include-dir=/include/gcc/darwin/4.0/c++ -- > build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 -- > target=powerpc-apple-darwin8 > Thread model: posix > gcc version 4.0.0 (Apple Computer, Inc. build 5026) > > However, when I move into the numpy-1.0.4 directory and run python > setup.py install, I get hundreds of error messages which appear to > relate to my C compiler. Here is the output with lots of similar C > messages in the middle omitted: > > non-existing path in 'numpy/distutils': 'site.cfg' > F2PY Version 2_4422 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/ > vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running install > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands -- > compiler options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands -- > fcompiler options > running build_src > building py_modules sources > building extension "numpy.core.multiarray" sources > Generating build/src.macosx-10.3-fat-2.5/numpy/core/config.h > customize NAGFCompiler > Could not locate executable f95 > customize AbsoftFCompiler > Could not locate executable f90 > Found executable /sw/bin/f77 > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > customize IBMFCompiler > Could not locate executable xlf90 > Could not locate executable xlf > customize IntelFCompiler > Could not locate executable ifort > Could not locate executable ifc > customize GnuFCompiler > Found executable /sw/bin/g77 > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize GnuFCompiler > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize GnuFCompiler using config > C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/ > MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp > -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 > > compile options: '-I/Library/Frameworks/Python.framework/Versions/2.5/ > include/python2.5 -Inumpy/core/src -Inumpy/core/include -I/Library/ > Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' > gcc: _configtest.c > In file included from _configtest.c:2: > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:18:20: error: limits.h: No such file or directory > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:21:2: error: #error "Something's broken. UCHAR_MAX should > be defined in limits.h." > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:25:2: error: #error "Python's source code assumes C's > unsigned char is an 8-bit type." > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:32:19: error: stdio.h: No such file or directory > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:34:5: error: #error "Python.h requires that stdio.h define > NULL." > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:37:20: error: string.h: No such file or directory > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:39:19: error: errno.h: No such file or directory > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:41:20: error: stdlib.h: No such file or directory > ... > > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > pystrtod.h:11: error: parse error before 'size_t' > In file included from _configtest.c:2: > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:134: error: parse error before '*' token > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:134: error: parse error before '*' token > /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/ > Python.h:134: warning: data definition has no type or storage class > _configtest.c: In function 'main': > _configtest.c:9: error: 'FILE' undeclared (first use in this function) > _configtest.c:9: error: (Each undeclared identifier is reported only > once > _configtest.c:9: error: for each function it appears in.) > _configtest.c:9: error: 'fp' undeclared (first use in this function) > _configtest.c:17: warning: incompatible implicit declaration of built- > in function 'fprintf' > lipo: can't figure out the architecture type of: /var/tmp//ccPjVTJJ.out > failure. > removing: _configtest.c _configtest.o > > > > > I know that my C compiler works, because I do my research in C on a > regular basis. Does anybody have any ideas? > > Thanks, > Sam Watson > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Sat Jan 19 22:23:13 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 19 Jan 2008 19:23:13 -0800 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: On Jan 19, 2008 12:02 PM, Charles R Harris wrote: > 1) I really miss the recent changes link, it was the only way I could keep > current on the Wiki contents. It was also a good way to track spam. The link just adds more clutter to the front page, which should be aimed at new users. I think that the front page is still way too busy. The page is still available. If you want to see the recent changes just go to: http://www.scipy.org/RecentChanges If you need to find it, you can bookmark it, remember the URL, do a search for "recent changes", add it to your navibar, or get the RSS feed. > 2) The addition of the bug reporting link is good, but it is hard to notice. > How about putting it up top using one of the neat python logos with a bug > addition? +1. If you (or someone else) make a new icon for bugs, I will happily add it. I'll give it a shot as well, but I am sure whatever I do will be easy to improve on. By the way, I would like to thank who ever made the existing icons; they make the front page much more attractive. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Sat Jan 19 23:35:57 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 19 Jan 2008 20:35:57 -0800 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: On Jan 19, 2008 7:23 PM, Jarrod Millman wrote: > On Jan 19, 2008 12:02 PM, Charles R Harris wrote: > > 2) The addition of the bug reporting link is good, but it is hard to notice. > > How about putting it up top using one of the neat python logos with a bug > > addition? > > +1. If you (or someone else) make a new icon for bugs, I will happily > add it. I'll give it a shot as well, but I am sure whatever I do will > be easy to improve on. By the way, I would like to thank who ever > made the existing icons; they make the front page much more > attractive. I went ahead and took a shot at it: www.scipy.org I also mocked up an all black bug: http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm5b.png If anyone can do a better job (and surely someone can), please go ahead. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david.huard at gmail.com Sat Jan 19 23:45:43 2008 From: david.huard at gmail.com (David Huard) Date: Sat, 19 Jan 2008 23:45:43 -0500 Subject: [Numpy-discussion] maskedarray branch In-Reply-To: <20080118222358.GD19790@mentat.za.net> References: <91cf711d0801151329r50535eb1v246eab7ce0645635@mail.gmail.com> <20080118073643.GL28865@mentat.za.net> <91cf711d0801180653m247f2fdbt826f6c219da7890@mail.gmail.com> <91cf711d0801180658q54963007mc1d66300f9c4cc66@mail.gmail.com> <91cf711d0801180753m2f9b11b8yb7ca56135be342e1@mail.gmail.com> <20080118222358.GD19790@mentat.za.net> Message-ID: <91cf711d0801192045x1f0edeffy6df4569b1cba5f45@mail.gmail.com> 2008/1/18, Stefan van der Walt : > > Hi David > > On Fri, Jan 18, 2008 at 10:53:28AM -0500, David Huard wrote: > > Stefan, > > It seems that the current maskedarray branch is not compatible with the > current > > scipy trunk. > > Would you mind expanding on that? >From memory it had something to do with a function in numpy, imported by scipy, called deprecate, that changed to deprecate_doc (or vice-versa). Thanks > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 20 04:06:31 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Jan 2008 02:06:31 -0700 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: On Jan 19, 2008 9:35 PM, Jarrod Millman wrote: > On Jan 19, 2008 7:23 PM, Jarrod Millman wrote: > > On Jan 19, 2008 12:02 PM, Charles R Harris > wrote: > > > 2) The addition of the bug reporting link is good, but it is hard to > notice. > > > How about putting it up top using one of the neat python logos with a > bug > > > addition? > > > > +1. If you (or someone else) make a new icon for bugs, I will happily > > add it. I'll give it a shot as well, but I am sure whatever I do will > > be easy to improve on. By the way, I would like to thank who ever > > made the existing icons; they make the front page much more > > attractive. > > I went ahead and took a shot at it: www.scipy.org > I also mocked up an all black bug: > > http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm5b.png > > If anyone can do a better job (and surely someone can), please go ahead. > > Thanks, Looks pretty good to me. I like the red spotted bug better than the black one. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholasinparis at gmail.com Sun Jan 20 11:06:45 2008 From: nicholasinparis at gmail.com (Nicholas) Date: Sun, 20 Jan 2008 17:06:45 +0100 Subject: [Numpy-discussion] creating rec arrays Message-ID: I am trying to package a set of vector arrays into a single rec array with column titles. c = numpy.rec.fromarrays([a,b],names='a,b') the problem is there is no 'copy' argument in .fromarrays which I can set to False. i.e. is there a way to do this without the overhead of copying? Thanks Nicholas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sun Jan 20 14:15:48 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 20 Jan 2008 12:15:48 -0700 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: On Jan 20, 2008 2:06 AM, Charles R Harris wrote: > > > > On Jan 19, 2008 9:35 PM, Jarrod Millman wrote: > > > > On Jan 19, 2008 7:23 PM, Jarrod Millman wrote: > > > On Jan 19, 2008 12:02 PM, Charles R Harris < charlesr.harris at gmail.com> > wrote: > > > > > > 2) The addition of the bug reporting link is good, but it is hard to > notice. > > > > How about putting it up top using one of the neat python logos with a > bug > > > > addition? > > > > > > +1. If you (or someone else) make a new icon for bugs, I will happily > > > add it. I'll give it a shot as well, but I am sure whatever I do will > > > be easy to improve on. By the way, I would like to thank who ever > > > made the existing icons; they make the front page much more > > > attractive. > > > > I went ahead and took a shot at it: www.scipy.org > > I also mocked up an all black bug: > > > http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm5b.png > > > > If anyone can do a better job (and surely someone can), please go ahead. > > > > Thanks, > > Looks pretty good to me. I like the red spotted bug better than the black > one. Yup, this one is my favorite: http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm4b.png Thanks! f From stefan at sun.ac.za Sun Jan 20 15:37:22 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 20 Jan 2008 22:37:22 +0200 Subject: [Numpy-discussion] r4730 broke numpy in maskedarray branch In-Reply-To: <47926AAE.6060407@hawaii.edu> References: <47926AAE.6060407@hawaii.edu> Message-ID: <20080120203722.GG19790@mentat.za.net> Hi Eric Sorry for the inconvenience -- I applied Pierre's patch. Thanks, St?fan On Sat, Jan 19, 2008 at 11:25:02AM -1000, Eric Firing wrote: > Stefan, > > The renaming of concatenator to AxisConcatenator in r4730 in the maskedaray > branch needs to be propagated into ma/extras.py; I don't know whether there > are other places that similar changes are needed. As it stands, after > 4730, numpy cannot be imported. > > Eric From travis at enthought.com Sun Jan 20 15:52:14 2008 From: travis at enthought.com (Travis Vaught) Date: Sun, 20 Jan 2008 14:52:14 -0600 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: Greetings, On Jan 20, 2008, at 1:15 PM, Fernando Perez wrote: > On Jan 20, 2008 2:06 AM, Charles R Harris > wrote: >> >> >> >> On Jan 19, 2008 9:35 PM, Jarrod Millman wrote: >>> >>> On Jan 19, 2008 7:23 PM, Jarrod Millman >>> wrote: >>>> On Jan 19, 2008 12:02 PM, Charles R Harris < charlesr.harris at gmail.com >>>> > >> wrote: >>> >>>>> 2) The addition of the bug reporting link is good, but it is >>>>> hard to >> notice. >>>>> How about putting it up top using one of the neat python logos >>>>> with a >> bug >>>>> addition? >>>> >>>> +1. If you (or someone else) make a new icon for bugs, I will >>>> happily >>>> add it. I'll give it a shot as well, but I am sure whatever I do >>>> will >>>> be easy to improve on. By the way, I would like to thank who ever >>>> made the existing icons; they make the front page much more >>>> attractive. >>> >>> I went ahead and took a shot at it: www.scipy.org >>> I also mocked up an all black bug: >>> >> http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm5b.png >>> >>> If anyone can do a better job (and surely someone can), please go >>> ahead. >>> >>> Thanks, >> >> Looks pretty good to me. I like the red spotted bug better than the >> black >> one. > > Yup, this one is my favorite: > > http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm4b.png > > Thanks! > > f Thanks for the attention to this...I think we need to seek continual improvement to the web site (numpy's default pages, again, could use some TLC). I've tweaked the bug a bit using the original photoshop scipy logo. Anyone who'd like the source images in photoshop format (or any other) let me know. Hopefully a distributed approach to the graphics and site design will exceed my limited skill. Thanks, Travis (V.) From efiring at hawaii.edu Sun Jan 20 16:00:53 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 20 Jan 2008 11:00:53 -1000 Subject: [Numpy-discussion] r4730 broke numpy in maskedarray branch In-Reply-To: <20080120203722.GG19790@mentat.za.net> References: <47926AAE.6060407@hawaii.edu> <20080120203722.GG19790@mentat.za.net> Message-ID: <4793B685.9030308@hawaii.edu> Stefan van der Walt wrote: > Hi Eric > > Sorry for the inconvenience -- I applied Pierre's patch. No problem, and thanks to both of you. I am glad to see maskedarray making progress towards official inclusion. Eric > > Thanks, > St?fan > > On Sat, Jan 19, 2008 at 11:25:02AM -1000, Eric Firing wrote: >> Stefan, >> >> The renaming of concatenator to AxisConcatenator in r4730 in the maskedaray >> branch needs to be propagated into ma/extras.py; I don't know whether there >> are other places that similar changes are needed. As it stands, after >> 4730, numpy cannot be imported. >> >> Eric From charlesr.harris at gmail.com Sun Jan 20 16:30:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Jan 2008 14:30:14 -0700 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: On Jan 20, 2008 1:52 PM, Travis Vaught wrote: > Greetings, > > On Jan 20, 2008, at 1:15 PM, Fernando Perez wrote: > > > On Jan 20, 2008 2:06 AM, Charles R Harris > > wrote: > >> > >> > >> > >> On Jan 19, 2008 9:35 PM, Jarrod Millman wrote: > >>> > >>> On Jan 19, 2008 7:23 PM, Jarrod Millman > >>> wrote: > >>>> On Jan 19, 2008 12:02 PM, Charles R Harris < > charlesr.harris at gmail.com > >>>> > > >> wrote: > >>> > >>>>> 2) The addition of the bug reporting link is good, but it is > >>>>> hard to > >> notice. > >>>>> How about putting it up top using one of the neat python logos > >>>>> with a > >> bug > >>>>> addition? > >>>> > >>>> +1. If you (or someone else) make a new icon for bugs, I will > >>>> happily > >>>> add it. I'll give it a shot as well, but I am sure whatever I do > >>>> will > >>>> be easy to improve on. By the way, I would like to thank who ever > >>>> made the existing icons; they make the front page much more > >>>> attractive. > >>> > >>> I went ahead and took a shot at it: www.scipy.org > >>> I also mocked up an all black bug: > >>> > >> > http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm5b.png > >>> > >>> If anyone can do a better job (and surely someone can), please go > >>> ahead. > >>> > >>> Thanks, > >> > >> Looks pretty good to me. I like the red spotted bug better than the > >> black > >> one. > > > > Yup, this one is my favorite: > > > > > http://www.scipy.org/SciPy?action=AttachFile&do=view&target=scipybuglogosm4b.png > > > > Thanks! > > > > f > > Thanks for the attention to this...I think we need to seek continual > improvement to the web site (numpy's default pages, again, could use > some TLC). I've tweaked the bug a bit using the original photoshop > scipy logo. Anyone who'd like the source images in photoshop format The bigger size is good, but I think the head needs to be green or yellow so that it stands out against the background. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Sun Jan 20 16:32:40 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 20 Jan 2008 11:32:40 -1000 Subject: [Numpy-discussion] maskedarray bug? Message-ID: <4793BDF8.3050902@hawaii.edu> Pierre, The attached script shows how one can make a masked array with different dimensions for the mask than for the data. I presume this is a bug. It is triggered by the present code in the matplotlib quiver function. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: maskdimbug.py Type: text/x-python Size: 198 bytes Desc: not available URL: From travis at enthought.com Sun Jan 20 17:15:24 2008 From: travis at enthought.com (Travis Vaught) Date: Sun, 20 Jan 2008 16:15:24 -0600 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: Message-ID: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> On Jan 20, 2008, at 3:30 PM, Charles R Harris wrote: > > ... > > Thanks for the attention to this...I think we need to seek continual > improvement to the web site (numpy's default pages, again, could use > some TLC). I've tweaked the bug a bit using the original photoshop > scipy logo. Anyone who'd like the source images in photoshop format > > The bigger size is good, but I think the head needs to be green or > yellow so that it stands out against the background. > > Chuck > Better now? -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 20 17:35:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Jan 2008 15:35:13 -0700 Subject: [Numpy-discussion] Scipy page In-Reply-To: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> References: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> Message-ID: On Jan 20, 2008 3:15 PM, Travis Vaught wrote: > > On Jan 20, 2008, at 3:30 PM, Charles R Harris wrote: > > > ... > > > > Thanks for the attention to this...I think we need to seek continual > > improvement to the web site (numpy's default pages, again, could use > > some TLC). I've tweaked the bug a bit using the original photoshop > > scipy logo. Anyone who'd like the source images in photoshop format > > > > The bigger size is good, but I think the head needs to be green or yellow > so that it stands out against the background. > > Chuck > > > Better now? > Ah, yes, good. Perhaps some color for the antennae too? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 20 18:25:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Jan 2008 16:25:22 -0700 Subject: [Numpy-discussion] Scipy page In-Reply-To: References: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> Message-ID: On Jan 20, 2008 3:35 PM, Charles R Harris wrote: > > > On Jan 20, 2008 3:15 PM, Travis Vaught wrote: > > > > > On Jan 20, 2008, at 3:30 PM, Charles R Harris wrote: > > > > > > ... > > > > > > Thanks for the attention to this...I think we need to seek continual > > > improvement to the web site (numpy's default pages, again, could use > > > some TLC). I've tweaked the bug a bit using the original photoshop > > > scipy logo. Anyone who'd like the source images in photoshop format > > > > > > > The bigger size is good, but I think the head needs to be green or > > yellow so that it stands out against the background. > > > > Chuck > > > > > > Better now? > > > > Ah, yes, good. Perhaps some color for the antennae too? > Excellent! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Jan 21 01:44:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 21 Jan 2008 15:44:05 +0900 Subject: [Numpy-discussion] numpy.distutils does not output compilation warning on win32 ? Message-ID: <47943F35.5030504@ar.media.kyoto-u.ac.jp> Hi, I noticed a strange behaviour with distutils: when compiling C code on windows (using mingw), the compilation warning are not output on the console. For example, umathmodule.c compilation emits the following warnings: numpy\core\src\umathmodule.c.src:128: warning: static declaration of 'acoshf' follows non-static declaration numpy\core\src\umathmodule.c.src:136: warning: static declaration of 'asinhf' follows non-static declaration I think this is coming from distutils because when I execute the exact same command on the shell, I get those warnings. If there is an error (by inserting #error), then all the warnings appear also when using distutils, which would suggest that distutils checks the return status of gcc to decide when to output should be sent on the shell ? Anyway, if we can do something about it, I think it would be better to always output warnings (it took me quite a while to understand why I got warnings with scons and not with distutils...). cheers, David From faltet at carabos.com Mon Jan 21 03:58:25 2008 From: faltet at carabos.com (Francesc Altet) Date: Mon, 21 Jan 2008 09:58:25 +0100 Subject: [Numpy-discussion] creating rec arrays In-Reply-To: References: Message-ID: <200801210958.25819.faltet@carabos.com> A Sunday 20 January 2008, Nicholas escrigu?: > I am trying to package a set of vector arrays into a single rec array > with column titles. > c = numpy.rec.fromarrays([a,b],names='a,b') > the problem is there is no 'copy' argument in .fromarrays which I can > set to False. i.e. is there a way to do this without the overhead of > copying? You can't because of implementation issues: a recarray is implemented as a row-wise object, so the copy is absolutely needed. BTW, perhaps column-wise recarray would be a great addition to NumPy, although I'm not sure whether this object would be much better than a dictionary of homogeneous arrays. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From meine at informatik.uni-hamburg.de Mon Jan 21 04:19:21 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 21 Jan 2008 10:19:21 +0100 Subject: [Numpy-discussion] Scipy page In-Reply-To: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> References: <85259A69-FB03-4D86-B95D-6C9786A4A558@enthought.com> Message-ID: <200801211019.21717.meine@informatik.uni-hamburg.de> Am Sonntag, 20. Januar 2008 23:15:24 schrieb Travis Vaught: > On Jan 20, 2008, at 3:30 PM, Charles R Harris wrote: > > Thanks for the attention to this...I think we need to seek continual > > improvement to the web site (numpy's default pages, again, could use > > some TLC). I've tweaked the bug a bit using the original photoshop > > scipy logo. Anyone who'd like the source images in photoshop format > > > > The bigger size is good, but I think the head needs to be green or > > yellow so that it stands out against the background. > > > > Chuck > > Better now? Nice. In order to have better contrast around the legs, one could use the same white "shadow"/aura as in the download icon. (++consistency) -- Ciao, / / /--/ / / ANS From pearu at cens.ioc.ee Mon Jan 21 07:13:00 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 21 Jan 2008 14:13:00 +0200 (EET) Subject: [Numpy-discussion] numpy.distutils does not output compilation warning on win32 ? In-Reply-To: <47943F35.5030504@ar.media.kyoto-u.ac.jp> References: <47943F35.5030504@ar.media.kyoto-u.ac.jp> Message-ID: <59858.85.166.31.187.1200917580.squirrel@cens.ioc.ee> Hi, If I remember correctly then the warnings were disabled because when compiling numpy/scipy on windows there were *lots* of warnings, especially for pyrex generated sources. When there is an error, all warnings will be shown. Hmm, and on linux the warnings should also be shown (this actually depends on what Python distutils one is using as the distutils logging interface has been changed between Python versions). In any case, we can enable all the warnigs again with no problems (may be only windows guys will not be so happy about it). Regards, Pearu On Mon, January 21, 2008 8:44 am, David Cournapeau wrote: > Hi, > > I noticed a strange behaviour with distutils: when compiling C code > on windows (using mingw), the compilation warning are not output on the > console. For example, umathmodule.c compilation emits the following > warnings: > > numpy\core\src\umathmodule.c.src:128: warning: static declaration of > 'acoshf' follows non-static declaration > numpy\core\src\umathmodule.c.src:136: warning: static declaration of > 'asinhf' follows non-static declaration > > I think this is coming from distutils because when I execute the exact > same command on the shell, I get those warnings. If there is an error > (by inserting #error), then all the warnings appear also when using > distutils, which would suggest that distutils checks the return status > of gcc to decide when to output should be sent on the shell ? Anyway, if > we can do something about it, I think it would be better to always > output warnings (it took me quite a while to understand why I got > warnings with scons and not with distutils...). > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Mon Jan 21 08:01:30 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 21 Jan 2008 08:01:30 -0500 Subject: [Numpy-discussion] def of var of complex References: Message-ID: Neal Becker wrote: > I noticed that if I generate complex rv i.i.d. with var=1, that numpy > says: > > var () -> (close to 1.0) > var () -> (close to 1.0) > > but > > var (complex array) -> (close to complex 0) > > Is that not a strange definition? I don't think there is any ambiguity about the definition of the variance of complex. Var(x) = E{(x-E[x])^2} = E{x}^2 - E{x} An estimator for this: E[x^n] \approx (1/M)\sum(x^n) From tteststudent at gmail.com Mon Jan 21 09:39:21 2008 From: tteststudent at gmail.com (theodore test) Date: Mon, 21 Jan 2008 09:39:21 -0500 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) Message-ID: Hello all, I'm scratching my head over how to make this image color space conversion from "RGB" to "HSV" quicker. It requires input from all three bands of a given pixel at each pixel, and thus can't be easily flattened. I've already implemented psyco and removed the extra step of calling colorsys's rgb_to_hsv function by adapting the code into my primary for loop. Right now it takes around 9 seconds for a single 1600x1200 RGB image, a conversion that I've seen implemented more or less instantly in, say, ImageJ. What can I do to make this conversion more efficient? Thank you in advance, Theodore Test ------------ #Necessary imports and hand-waving at psyco usage. import numpy from numpy import * from scipy.misc import pilutil import psyco psyco.full() # Read image file, cram it into a normalized array with one row per pixel # with each column holding the given pixel's R,G, or B band-value. s = pilutil.imread('A_Biggish_Image.tif') r,c,d = s.shape l = r*c t = t.reshape(l,3) t = t/255.0 # Cycle through all of the pixels, converting them to HSV space and putting them # back into the array. This is the big time-waster. # # The conversion is adapted directly from colorsys.rgb_to_hsv to reduce call time for x in range(0,l): tr = float(t[x,0]) tg = float(t[x,1]) tb = float(t[x,2]) maxc = max(tr, tg, tb) minc = min(tr, tg, tb) v = maxc if minc == maxc: t[x,0],t[x,1],t[x,2] = 0.0, 0.0, v else: s = (maxc-minc) / maxc rc = (maxc-tr) / (maxc-minc) gc = (maxc-tg) / (maxc-minc) bc = (maxc-tb) / (maxc-minc) if tr == maxc: h = bc-gc elif tg == maxc: h = 2.0+rc-bc else: h = 4.0+gc-rc h = (h/6.0) % 1.0 t[x,0],t[x,1],t[x,2] = h, s, v # Renormalize shape and contents to image-file standards. t = t*255.0 t = t.astype(uint8) t = t.reshape(r,c,d) finaloutputarray = t -------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Mon Jan 21 10:25:54 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 21 Jan 2008 10:25:54 -0500 Subject: [Numpy-discussion] Bus Error with object arrays on big endian system Message-ID: <4794B982.5070101@stsci.edu> Hi, The following will cause a bus error on a big endian machine (Solaris 10 Sun in this case): > Python 2.5.1 (r251:54863, Jun 29 2007, 15:29:55) [C] on sunos5 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy >>>> o = numpy.ndarray(shape=3,dtype=[('SEGMENT', '|S4'), ('SPEC_FOUND', '|i1')]) >>>> o1 = o.getfield(numpy.dtype('|S4'),0) >>>> print o1[0] > UX? >>>> print o1[1] > 4 >>>> print o1[2] > NT >>>> print o1 > Bus error (core dumped) There are no issues on Linux or Mac OS X Intel based systems. This example was done on the latest svn version of numpy (r1.0.5.dev47360). Does anyone have an idea of what may have broken? Thank you for your time and help, Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From meine at informatik.uni-hamburg.de Mon Jan 21 10:57:14 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 21 Jan 2008 16:57:14 +0100 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) In-Reply-To: References: Message-ID: <200801211657.15228.meine@informatik.uni-hamburg.de> Am Montag, 21. Januar 2008 15:39:21 schrieb theodore test: > Right now it takes around 9 seconds for a single 1600x1200 RGB image, a > conversion that I've seen implemented more or less instantly in, say, > ImageJ. What can I do to make this conversion more efficient? You should "just remove" the loop. Try to convert all operations to elementwise array/matrix-operations, which are very fast. IOW, make the algorithm work on all pixels in parallel. The only tricky part is the conditional operation, but you can emulate that with the binary operators and e.g. put(). -- Ciao, / / /--/ / / ANS From david at ar.media.kyoto-u.ac.jp Mon Jan 21 10:49:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 22 Jan 2008 00:49:13 +0900 Subject: [Numpy-discussion] numpy.distutils does not output compilation warning on win32 ? In-Reply-To: <59858.85.166.31.187.1200917580.squirrel@cens.ioc.ee> References: <47943F35.5030504@ar.media.kyoto-u.ac.jp> <59858.85.166.31.187.1200917580.squirrel@cens.ioc.ee> Message-ID: <4794BEF9.6060705@ar.media.kyoto-u.ac.jp> Pearu Peterson wrote: > Hi, > > If I remember correctly then the warnings were disabled because > when compiling numpy/scipy on windows there were *lots* of > warnings, especially for pyrex generated sources. > Yes, that's certainly one of the thing I intend to improve with numscons :) (as an aside, I also hope that if numscons is adopted, we will be able to set more warnings for many extensions). Thanks for the info. cheers, David From jbednar at inf.ed.ac.uk Mon Jan 21 11:00:28 2008 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Mon, 21 Jan 2008 16:00:28 +0000 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) In-Reply-To: References: Message-ID: <18324.49564.818348.73175@lodestar.inf.ed.ac.uk> | Date: Mon, 21 Jan 2008 09:39:21 -0500 | From: "theodore test" | | Hello all, | | I'm scratching my head over how to make this image color space | conversion from "RGB" to "HSV" quicker. It requires input from all | three bands of a given pixel at each pixel, and thus can't be | easily flattened. I've already implemented psyco and removed the | extra step of calling colorsys's rgb_to_hsv function by adapting | the code into my primary for loop. | | Right now it takes around 9 seconds for a single 1600x1200 RGB image, a | conversion that I've seen implemented more or less instantly in, say, | ImageJ. What can I do to make this conversion more efficient? I would love to see a decent implementation of that! We have versions quite similar to yours, instead. We've had a to-do list item to reimplement rgb_to_hsv (and its converse hsv_to_rgb) in C using weave, but haven't ever gotten around to it. I believe the best solution would be for PIL to be modified to accept an image of type HSV; it already accepts four others, including the non-obvious CMYK: m.mode => string Image mode. This is a string specifying the pixel format used by the image. Typical values are "1", "L", "RGB", or "CMYK." Adding CSV to this list would seem quite reasonable, and would allow simple conversion between the various color spaces. But I don't know anything about the internals of the PIL code, to know whether that would indeed make sense. Jim From chan_dhf at yahoo.de Mon Jan 21 11:26:03 2008 From: chan_dhf at yahoo.de (Danny Chan) Date: Mon, 21 Jan 2008 17:26:03 +0100 (CET) Subject: [Numpy-discussion] compiling c extension Message-ID: <924930.14099.qm@web26205.mail.ukl.yahoo.com> Hi! I am trying to compile a c extension that uses numpy arrays on windows. I can compile the extension file, but once I get to the linking stage I get unresolved references to PyArray_API and import_array. Can anyone help me with the correct linker flags? Thanks, Danny running build running build_py file libaid.py (for module libaid) not found file libaid.py (for module libaid) not found running build_ext building '_libaid' extension writing build\temp.win32-2.5\Release\.\python\_libaid.def x:\eda\mingw\5.1.3\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release\.\python\libaid_wrap.o build\temp.win32-2.5\Release\.\python\_libaid.def -L../../build/win32/bin -Lx:\eda\python\windows\libs -Lx:\eda\python\windows\PCBuild -laid -lpython25 -lmsvcr71 -o build\lib.win32-2.5\_libaid.pyd build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x2602): undefined reference to `PyArray_API' build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x2635): undefined reference to `PyArray_API' build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x264f): undefined reference to `PyArray_API' build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x2709): undefined reference to `PyArray_API' build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x2752): undefined reference to `PyArray_API' build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0x2784): more undefined references to `PyArray_API' follow build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0xc216): undefined reference to `import_array' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 --------------------------------- Beginnen Sie den Tag mit den neuesten Nachrichten. Machen Sie Yahoo! zu Ihrer Startseite! -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Jan 21 11:41:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 22 Jan 2008 01:41:04 +0900 Subject: [Numpy-discussion] Do not use -fomit-frame-pointer with mingw ! Message-ID: <4794CB20.8050809@ar.media.kyoto-u.ac.jp> Hi, I don't know if this is known, but since I wasted half a day on it, I thought I could avoid the pain for someone else: do not build numpy with -fomit-frame-pointer with mingw (windows), it will crash. I don't know if this just cannot be used, if this is a mingw bug, or something on our side. But you do not want to look for a bug caused by using this option. cheers, David From lou_boog2000 at yahoo.com Mon Jan 21 12:03:29 2008 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 21 Jan 2008 09:03:29 -0800 (PST) Subject: [Numpy-discussion] compiling c extension In-Reply-To: <924930.14099.qm@web26205.mail.ukl.yahoo.com> Message-ID: <195621.19830.qm@web34411.mail.mud.yahoo.com> Did you include arrayobject.h and call import_array() in the initialization function, after the call to Py_InitModule() ? --- Danny Chan wrote: > Hi! > I am trying to compile a c extension that uses numpy > arrays on windows. I can compile the extension file, > but once I get to the linking stage I get unresolved > references to PyArray_API and import_array. Can > anyone help me with the correct linker flags? > > Thanks, Danny [cut] > more undefined references to `PyArray_API' follow > build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0xc216): > undefined reference to `import_array' > collect2: ld returned 1 exit status > error: command 'gcc' failed with exit status 1 -- Lou Pecora, my views are my own. ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping From robert.kern at gmail.com Mon Jan 21 12:05:18 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 21 Jan 2008 11:05:18 -0600 Subject: [Numpy-discussion] def of var of complex In-Reply-To: References: Message-ID: <4794D0CE.404@gmail.com> Neal Becker wrote: > Neal Becker wrote: > >> I noticed that if I generate complex rv i.i.d. with var=1, that numpy >> says: >> >> var () -> (close to 1.0) >> var () -> (close to 1.0) >> >> but >> >> var (complex array) -> (close to complex 0) >> >> Is that not a strange definition? > > I don't think there is any ambiguity about the definition of the variance of > complex. > > Var(x) = E{(x-E[x])^2} = E{x}^2 - E{x} That's currently what's implemented, but there is simply no evidence that anyone ever uses this definition for complex random variables. Note that for real variables, E{(x-E[x])^2} = E{|x-E[x]|^2} but for complex variables, there is a large difference. Since the || are superfluous with real variables, probability texts rarely include them unless if they are then going on to talk about complex variables. If you want to extend the definition for real variables to complex variables, that is an ambiguity you have to resolve. There is, apparently, a large body of statistical signal processing literature that defines the complex variance as E{|z-E[z]|^2} so, I (now) believe that this is what should be implemented. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bsouthey at gmail.com Mon Jan 21 12:33:20 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 21 Jan 2008 11:33:20 -0600 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) In-Reply-To: References: Message-ID: Hi, I don't know the area but following your code I would suggest the completely untested code. I am not sure about the conditions to get h so I leave you get to the correct h and transform it appropriately. Bruce # First create 2 dimensional array/matrix (number of pixels by three): 'rows' are pixels and one 'columns' for each of r, g, b values, say RGB from the image # Compute v ie maximum across r, g and b for each pixel v=RGB.max(axis=1) #compute range between high and low values for each pixel maxmin=v-RGB.min(axis=1) #compute s s=maxmin/v #Compute h hr=(RGB[:,1]-RGB[:,2])/maxmin hg=2.0+(RGB[:,2]-RGB[:,0])/maxmin hb=4.0+(RGB[:,0]-RGB[:,1])/maxmin #Some code to get h #Final output HSV=255.0*numpy.column_stack(h,s,v) On Jan 21, 2008 8:39 AM, theodore test wrote: > Hello all, > > I'm scratching my head over how to make this image color space conversion > from "RGB" to "HSV" quicker. It requires input from all three bands of a > given pixel at each pixel, and thus can't be easily flattened. I've already > implemented psyco and removed the extra step of calling colorsys's > rgb_to_hsv function by adapting the code into my primary for loop. > > Right now it takes around 9 seconds for a single 1600x1200 RGB image, a > conversion that I've seen implemented more or less instantly in, say, > ImageJ. What can I do to make this conversion more efficient? > > Thank you in advance, > Theodore Test > > ------------ > > > #Necessary imports and hand-waving at psyco usage. > import numpy > from numpy import * > from scipy.misc import pilutil > > import psyco > psyco.full() > > > # Read image file, cram it into a normalized array with one row per pixel > # with each column holding the given pixel's R,G, or B band-value. > s = pilutil.imread('A_Biggish_Image.tif') > r,c,d = s.shape > > l = r*c > t = t.reshape(l,3) > t = t/255.0 > > > # Cycle through all of the pixels, converting them to HSV space and putting > them > # back into the array. This is the big time-waster. > # > # The conversion is adapted directly from colorsys.rgb_to_hsv to reduce call > time > for x in range(0,l): > tr = float(t[x,0]) > tg = float(t[x,1]) > tb = float(t[x,2]) > maxc = max(tr, tg, tb) > minc = min(tr, tg, tb) > v = maxc > if minc == maxc: > t[x,0],t[x,1],t[x,2] = 0.0, 0.0, v > else: > s = (maxc-minc) / maxc > rc = (maxc-tr) / (maxc-minc) > gc = (maxc-tg) / (maxc-minc) > bc = (maxc-tb) / (maxc-minc) > if tr == maxc: h = bc-gc > elif tg == maxc: h = 2.0+rc-bc > else: h = 4.0+gc-rc > h = (h/6.0) % 1.0 > t[x,0],t[x,1],t[x,2] = h, s, v > > # Renormalize shape and contents to image-file standards. > t = t*255.0 > t = t.astype(uint8) > t = t.reshape(r,c,d) > > finaloutputarray = t > > -------- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From pgmdevlist at gmail.com Mon Jan 21 15:35:37 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 21 Jan 2008 15:35:37 -0500 Subject: [Numpy-discussion] maskedarray bug? In-Reply-To: <4793BDF8.3050902@hawaii.edu> References: <4793BDF8.3050902@hawaii.edu> Message-ID: <200801211535.37916.pgmdevlist@gmail.com> On Sunday 20 January 2008 16:32:40 you wrote: > Pierre, > > The attached script shows how one can make a masked array with different > dimensions for the mask than for the data. I presume this is a bug. It > is triggered by the present code in the matplotlib quiver function. Yep, bug indeed. Thanks for pointing that out ! The following patch should take care of the problem. (In short, a getmask function was used instead of getmaskarray). Note that the patch takes also into account elements I had sent to Stefan 2 weeks ago, that were not ported yet to the SVN: I still can't commit to the numpy/branches/maskedarray. Let me know how it works, and thanks again for your time. P. -------------- next part -------------- A non-text attachment was scrubbed... Name: ma_v20080121.patch Type: text/x-diff Size: 16498 bytes Desc: not available URL: From matthew.brett at gmail.com Mon Jan 21 17:03:17 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 21 Jan 2008 22:03:17 +0000 Subject: [Numpy-discussion] Docstring page, out of date? Message-ID: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Hi, Search for the docstring standard, I hit this: http://www.scipy.org/DocstringStandard but I think the current thinking is this: http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines Is that correct? Does the first page apply to matplotlib in some way? Should we change the first page to match the second now? Matthew From arnar.flatberg at gmail.com Mon Jan 21 17:17:38 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Mon, 21 Jan 2008 23:17:38 +0100 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) In-Reply-To: References: Message-ID: <5d3194020801211417t44469324xf1b812e2053670b@mail.gmail.com> Hi Theodore Probably not the fastest, but a full example of how you may vectorize your loop. Ran just one test, and that speeded up the original code. Example: ---------------- from numpy import empty_like def vectorized_rgb2hsv(im): "im is a (m, n, 3) array.""" im = im/255. out = empty_like(im) im_max = im.max(-1) delta = im.ptp(-1) s = delta/im_max s[delta==0] = 0 index = im[:,:,0] == im_max # red is max out[index, 0] = (im[index, 1] - im[index, 2] ) / delta[index] index = im[:,:,1] == im_max # green is max out[index, 0] = 2 + (im[index, 2] - im[index, 0] ) / delta[index] index = im[:,:,2] == im_max # blue is max out[index, 0] = 4 + (im[index, 0] - im[index, 1] ) / delta[index] out[:,:,0] = (out[:,:,0]/6.0) % 1.0 out[:,:,1] = s out[:,:,2] = im_max out = (255.*out).astype(uint8) return out Timings (shape: 1202, 800, 3): ----------------- code in first post: %prun a = rgb2hsv.rgb2hsv(s) 1923207 function calls in 18.423 CPU seconds removing loop: %prun b = rgb2hsv.vectorized_rgb2hsv(s) 8 function calls in 4.630 CPU seconds Arnar From millman at berkeley.edu Mon Jan 21 20:09:27 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 21 Jan 2008 17:09:27 -0800 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: On Jan 21, 2008 2:03 PM, Matthew Brett wrote: > Search for the docstring standard, I hit this: > > http://www.scipy.org/DocstringStandard Good catch, I didn't know this page existed. If I recall correctly, Keir Mierle showed up on the mailing list around the time we were discussing the docstring standard. He proposed to do some work, but then must have gotten busy with something else. In particular, I believe he was interested in seeing a unified docstring standard for numpy, scipy, and matplotlib. I guess he put this page up during that period. I went ahead and deleted it, since it conflicts with the official docstring standard. > but I think the current thinking is this: > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > Is that correct? Does the first page apply to matplotlib in some way? > Should we change the first page to match the second now? Yes. That page is autogenerated from the official coding standard that is in the numpy trunk. One of the nice features of trac is that it can render restructured text from the svn repository. This helps us keep from having duplicate information that needs to be kept in sync by hand. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Mon Jan 21 22:38:38 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Jan 2008 20:38:38 -0700 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: On Jan 21, 2008 6:09 PM, Jarrod Millman wrote: > On Jan 21, 2008 2:03 PM, Matthew Brett wrote: > > Search for the docstring standard, I hit this: > > > > http://www.scipy.org/DocstringStandard > > Good catch, I didn't know this page existed. If I recall correctly, > Keir Mierle showed up on the mailing list around the time we were > discussing the docstring standard. He proposed to do some work, but > then must have gotten busy with something else. In particular, I > believe he was interested in seeing a unified docstring standard for > numpy, scipy, and matplotlib. I guess he put this page up during that > period. I went ahead and deleted it, since it conflicts with the > official docstring standard. > > > but I think the current thinking is this: > > > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > > > Is that correct? Does the first page apply to matplotlib in some way? > > Should we change the first page to match the second now? > > Yes. That page is autogenerated from the official coding standard > that is in the numpy trunk. One of the nice features of trac is that > it can render restructured text from the svn repository. This helps > us keep from having duplicate information that needs to be kept in > sync by hand. > If I hit the source code link in the generated html, it looks like that page was generated from the old document format. The new document format doesn't produce output that looks anything like that and epydoc generates a couple of warnings: | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in | example.foo | Warning: Line 24: Wrong underline character for heading. | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph | indentation. The new document format requires a preprocessor that has yet to be written. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Mon Jan 21 22:54:55 2008 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 21 Jan 2008 17:54:55 -1000 Subject: [Numpy-discussion] maskedarray bug? In-Reply-To: <200801211535.37916.pgmdevlist@gmail.com> References: <4793BDF8.3050902@hawaii.edu> <200801211535.37916.pgmdevlist@gmail.com> Message-ID: <4795690F.1020407@hawaii.edu> Pierre GM wrote: > On Sunday 20 January 2008 16:32:40 you wrote: >> Pierre, >> >> The attached script shows how one can make a masked array with different >> dimensions for the mask than for the data. I presume this is a bug. It >> is triggered by the present code in the matplotlib quiver function. > > Yep, bug indeed. Thanks for pointing that out ! > The following patch should take care of the problem. (In short, a getmask > function was used instead of getmaskarray). > Note that the patch takes also into account elements I had sent to Stefan 2 > weeks ago, that were not ported yet to the SVN: I still can't commit to the > numpy/branches/maskedarray. > Let me know how it works, and thanks again for your time. Pierre, Thank you, it works fine. Eric From charlesr.harris at gmail.com Mon Jan 21 22:56:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Jan 2008 20:56:48 -0700 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: On Jan 21, 2008 8:38 PM, Charles R Harris wrote: > > > On Jan 21, 2008 6:09 PM, Jarrod Millman wrote: > > > On Jan 21, 2008 2:03 PM, Matthew Brett wrote: > > > Search for the docstring standard, I hit this: > > > > > > http://www.scipy.org/DocstringStandard > > > > Good catch, I didn't know this page existed. If I recall correctly, > > Keir Mierle showed up on the mailing list around the time we were > > discussing the docstring standard. He proposed to do some work, but > > then must have gotten busy with something else. In particular, I > > believe he was interested in seeing a unified docstring standard for > > numpy, scipy, and matplotlib. I guess he put this page up during that > > period. I went ahead and deleted it, since it conflicts with the > > official docstring standard. > > > > > but I think the current thinking is this: > > > > > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > > > > > Is that correct? Does the first page apply to matplotlib in some way? > > > Should we change the first page to match the second now? > > > > Yes. That page is autogenerated from the official coding standard > > that is in the numpy trunk. One of the nice features of trac is that > > it can render restructured text from the svn repository. This helps > > us keep from having duplicate information that needs to be kept in > > sync by hand. > > > > If I hit the source code link in the generated html, it looks like that > page was generated from the old document format. The new document format > doesn't produce output that looks anything like that and epydoc generates a > couple of warnings: > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > | example.foo > | Warning: Line 24: Wrong underline character for heading. > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph > | indentation. > > The new document format requires a preprocessor that has yet to be > written. > Since epydoc also works for compiled modules, I think the easiest way to do that is to fork epydoc. The syntax of the new markup is context sensitive - single types are no longer in {} - so parsing will be a bit more complicated. As ReST is not part of the current document markup, we might consider removing the parts of the document documentation that refer to it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Jan 22 00:58:43 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Jan 2008 14:58:43 +0900 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: <5b8d13220801212158p5b16c055ga2985e9198cbb7f7@mail.gmail.com> On Jan 22, 2008 12:56 PM, Charles R Harris wrote: > > > > > On Jan 21, 2008 8:38 PM, Charles R Harris wrote: > > > > > > > > > > On Jan 21, 2008 6:09 PM, Jarrod Millman wrote: > > > > > > > > On Jan 21, 2008 2:03 PM, Matthew Brett wrote: > > > > Search for the docstring standard, I hit this: > > > > > > > > http://www.scipy.org/DocstringStandard > > > > > > Good catch, I didn't know this page existed. If I recall correctly, > > > Keir Mierle showed up on the mailing list around the time we were > > > discussing the docstring standard. He proposed to do some work, but > > > then must have gotten busy with something else. In particular, I > > > believe he was interested in seeing a unified docstring standard for > > > numpy, scipy, and matplotlib. I guess he put this page up during that > > > period. I went ahead and deleted it, since it conflicts with the > > > official docstring standard. > > > > > > > > > > but I think the current thinking is this: > > > > > > > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > > > > > > > Is that correct? Does the first page apply to matplotlib in some way? > > > > Should we change the first page to match the second now? > > > > > > Yes. That page is autogenerated from the official coding standard > > > that is in the numpy trunk. One of the nice features of trac is that > > > it can render restructured text from the svn repository. This helps > > > us keep from having duplicate information that needs to be kept in > > > sync by hand. > > > > > > > > > If I hit the source code link in the generated html, it looks like that > page was generated from the old document format. The new document format > doesn't produce output that looks anything like that and epydoc generates a > couple of warnings: > > > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > > | example.foo > > | Warning: Line 24: Wrong underline character for heading. > > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph > > | indentation. > > > > > > The new document format requires a preprocessor that has yet to be > written. > > > > Since epydoc also works for compiled modules, I think the easiest way to do > that is to fork epydoc. The syntax of the new markup is context sensitive - > single types are no longer in {} - so parsing will be a bit more > complicated. As ReST is not part of the current document markup, we might > consider removing the parts of the document documentation that refer to it. > I must confess I don't get those discussion on docstring - at all. I think there has been some discussion on this for almost one year, and I have never managed to "compile" a full doc for numpy or scipy. Nobody likes to write doc, specially if you have to follow many rules. Not being able to see the result does not help. I don't know much about the current documation tools situation with python, but I have seen good, automatically generated doc for python modules: for example, the current developement doc of python (http://docs.python.org/dev/) looks pretty good to me, and using the same tools as python makes more sense than forking our own, no ? Or am I missing something fundamental ? cheers, David From robert.kern at gmail.com Tue Jan 22 01:08:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Jan 2008 00:08:14 -0600 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: <5b8d13220801212158p5b16c055ga2985e9198cbb7f7@mail.gmail.com> References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> <5b8d13220801212158p5b16c055ga2985e9198cbb7f7@mail.gmail.com> Message-ID: <4795884E.4090300@gmail.com> David Cournapeau wrote: > Nobody likes to write doc, specially if you have to follow many rules. > Not being able to see the result does not help. I don't know much > about the current documation tools situation with python, but I have > seen good, automatically generated doc for python modules: for > example, the current developement doc of python > (http://docs.python.org/dev/) looks pretty good to me, and using the > same tools as python makes more sense than forking our own, no ? Or am > I missing something fundamental ? Yes. Sphinx does not do automatically generated documentation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Tue Jan 22 01:18:17 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 21 Jan 2008 22:18:17 -0800 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: On Jan 21, 2008 7:56 PM, Charles R Harris wrote: > On Jan 21, 2008 8:38 PM, Charles R Harris wrote: > > If I hit the source code link in the generated html, it looks like that > page was generated from the old document format. The new document format > doesn't produce output that looks anything like that and epydoc generates a > couple of warnings: > > > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > > | example.foo > > | Warning: Line 24: Wrong underline character for heading. > > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph > > | indentation. > > > > The new document format requires a preprocessor that has yet to be > written. > > Since epydoc also works for compiled modules, I think the easiest way to do > that is to fork epydoc. The syntax of the new markup is context sensitive - > single types are no longer in {} - so parsing will be a bit more > complicated. As ReST is not part of the current document markup, we might > consider removing the parts of the document documentation that refer to it. Hey Chuck, I am sorry that there has been so much confusion over the docstring standards. As you know, the problems you are pointing out arose from the changes Travis made in December. I added a warning to the top of the coding style guideline explaining the situation: http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines Once Travis writes a tool to render the docstrings, we can go ahead and update the instructions. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Tue Jan 22 01:27:19 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Jan 2008 00:27:19 -0600 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: <47958CC7.5050005@gmail.com> Charles R Harris wrote: > > > On Jan 21, 2008 6:09 PM, Jarrod Millman > wrote: > > On Jan 21, 2008 2:03 PM, Matthew Brett > wrote: > > Search for the docstring standard, I hit this: > > > > http://www.scipy.org/DocstringStandard > > Good catch, I didn't know this page existed. If I recall correctly, > Keir Mierle showed up on the mailing list around the time we were > discussing the docstring standard. He proposed to do some work, but > then must have gotten busy with something else. In particular, I > believe he was interested in seeing a unified docstring standard for > numpy, scipy, and matplotlib. I guess he put this page up during that > period. I went ahead and deleted it, since it conflicts with the > official docstring standard. > > > but I think the current thinking is this: > > > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > > > > Is that correct? Does the first page apply to matplotlib in some way? > > Should we change the first page to match the second now? > > Yes. That page is autogenerated from the official coding standard > that is in the numpy trunk. One of the nice features of trac is that > it can render restructured text from the svn repository. This helps > us keep from having duplicate information that needs to be kept in > sync by hand. > > > If I hit the source code link in the generated html, it looks like that > page was generated from the old document format. The new document format > doesn't produce output that looks anything like that and epydoc > generates a couple of warnings: > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > | example.foo > | Warning: Line 24: Wrong underline character for heading. > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph > | indentation. > > The new document format requires a preprocessor that has yet to be written. epydoc from SVN works just fine once the following line is added at the top. __docformat__ = "restructuredtext en" -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jussi.enkovaara at csc.fi Tue Jan 22 01:44:26 2008 From: jussi.enkovaara at csc.fi (Jussi Enkovaara) Date: Tue, 22 Jan 2008 08:44:26 +0200 Subject: [Numpy-discussion] Building a static libnumpy Message-ID: <479590CA.9040404@csc.fi> Hi, I need to use numpy in an environment which does not support shared libraries. Previously, I have used the old Numeric where I created a Makefile to build a static Numeric library which was later on linked to the python interpreter. As far as I understood, numpy uses sort of extended distutils. I was wondering if this extended distutils enables building of static libraries or do I have to go the cumbersome Makefile-way again? Regards, Jussi Enkovaara From matthieu.brucher at gmail.com Tue Jan 22 01:47:26 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 22 Jan 2008 07:47:26 +0100 Subject: [Numpy-discussion] Building a static libnumpy In-Reply-To: <479590CA.9040404@csc.fi> References: <479590CA.9040404@csc.fi> Message-ID: Hi, I think the main problem would be that some parts of Numpy are coded in C and that they must be compiled as a shared library so that Python can access them. Matthieu 2008/1/22, Jussi Enkovaara : > > Hi, > I need to use numpy in an environment which does not support shared > libraries. > Previously, I have used the old Numeric where I created a Makefile to > build a > static Numeric library which was later on linked to the python > interpreter. > > As far as I understood, numpy uses sort of extended distutils. I was > wondering if > this extended distutils enables building of static libraries or do I have > to go > the cumbersome Makefile-way again? > > Regards, > Jussi Enkovaara > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Jan 22 01:49:04 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 21 Jan 2008 23:49:04 -0700 Subject: [Numpy-discussion] Sage/Scipy Days 8 reminder: Feb 29-March 4. Message-ID: Hi all, Just a quick reminder for all about the upcoming Sage/Scipy Days 8 at Enthought collaborative meeting: http://wiki.sagemath.org/days8 Email me directly (Fernando.Perez at Colorado.edu) if you plan on coming, so we can have a proper count and plan accordingly. Cheers, f From charlesr.harris at gmail.com Tue Jan 22 01:49:24 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Jan 2008 23:49:24 -0700 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: <47958CC7.5050005@gmail.com> References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> <47958CC7.5050005@gmail.com> Message-ID: On Jan 21, 2008 11:27 PM, Robert Kern wrote: > Charles R Harris wrote: > > > > > > On Jan 21, 2008 6:09 PM, Jarrod Millman > > wrote: > > > > On Jan 21, 2008 2:03 PM, Matthew Brett > > wrote: > > > Search for the docstring standard, I hit this: > > > > > > http://www.scipy.org/DocstringStandard > > > > Good catch, I didn't know this page existed. If I recall correctly, > > Keir Mierle showed up on the mailing list around the time we were > > discussing the docstring standard. He proposed to do some work, but > > then must have gotten busy with something else. In particular, I > > believe he was interested in seeing a unified docstring standard for > > numpy, scipy, and matplotlib. I guess he put this page up during > that > > period. I went ahead and deleted it, since it conflicts with the > > official docstring standard. > > > > > but I think the current thinking is this: > > > > > > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > > > > > > > > Is that correct? Does the first page apply to matplotlib in some > way? > > > Should we change the first page to match the second now? > > > > Yes. That page is autogenerated from the official coding standard > > that is in the numpy trunk. One of the nice features of trac is > that > > it can render restructured text from the svn repository. This helps > > us keep from having duplicate information that needs to be kept in > > sync by hand. > > > > > > If I hit the source code link in the generated html, it looks like that > > page was generated from the old document format. The new document format > > doesn't produce output that looks anything like that and epydoc > > generates a couple of warnings: > > > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > > | example.foo > > | Warning: Line 24: Wrong underline character for heading. > > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper > paragraph > > | indentation. > > > > The new document format requires a preprocessor that has yet to be > written. > > epydoc from SVN works just fine once the following line is added at the > top. > > __docformat__ = "restructuredtext en" > So it does, sorry for the noise. Does it work with the docstrings for the C code also? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 22 01:50:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Jan 2008 00:50:05 -0600 Subject: [Numpy-discussion] Building a static libnumpy In-Reply-To: <479590CA.9040404@csc.fi> References: <479590CA.9040404@csc.fi> Message-ID: <4795921D.9020509@gmail.com> Jussi Enkovaara wrote: > Hi, > I need to use numpy in an environment which does not support shared libraries. > Previously, I have used the old Numeric where I created a Makefile to build a > static Numeric library which was later on linked to the python interpreter. > > As far as I understood, numpy uses sort of extended distutils. I was wondering if > this extended distutils enables building of static libraries or do I have to go > the cumbersome Makefile-way again? Cumbersome Makefile, sorry. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Tue Jan 22 02:32:38 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 22 Jan 2008 09:32:38 +0200 Subject: [Numpy-discussion] maskedarray bug? In-Reply-To: <200801211535.37916.pgmdevlist@gmail.com> References: <4793BDF8.3050902@hawaii.edu> <200801211535.37916.pgmdevlist@gmail.com> Message-ID: <20080122073238.GD31735@mentat.za.net> Hi Pierre, Eric On Mon, Jan 21, 2008 at 03:35:37PM -0500, Pierre GM wrote: > Yep, bug indeed. Thanks for pointing that out ! > The following patch should take care of the problem. (In short, a getmask > function was used instead of getmaskarray). > Note that the patch takes also into account elements I had sent to Stefan 2 > weeks ago, that were not ported yet to the SVN: I still can't commit to the > numpy/branches/maskedarray. The patch you refer to was applied in r4718 on 10 January: http://projects.scipy.org/scipy/numpy/changeset/4718 I hope there weren't others I missed. I shall work in your latest patch as soon as I can access the repo again. Regards St?fan From cournape at gmail.com Tue Jan 22 03:12:32 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Jan 2008 17:12:32 +0900 Subject: [Numpy-discussion] Building a static libnumpy In-Reply-To: <479590CA.9040404@csc.fi> References: <479590CA.9040404@csc.fi> Message-ID: <5b8d13220801220012p6ac1b15er606e583c2e27af57@mail.gmail.com> On Jan 22, 2008 3:44 PM, Jussi Enkovaara wrote: > Hi, > I need to use numpy in an environment which does not support shared libraries. > Previously, I have used the old Numeric where I created a Makefile to build a > static Numeric library which was later on linked to the python interpreter. > Interesting, did not know it was possible. How did you do it ? I am working on an alternative build system for numpy, based on scons, and depending on the necessary steps, I could add this feature to the build system. scons itself certainly support static libraries, but I have never built static python extensions. cheers, David From cournape at gmail.com Tue Jan 22 03:33:08 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Jan 2008 17:33:08 +0900 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> Message-ID: <5b8d13220801220033o2b5f5d0dgaadd70d6db18b5c@mail.gmail.com> On Jan 22, 2008 3:18 PM, Jarrod Millman wrote: > On Jan 21, 2008 7:56 PM, Charles R Harris wrote: > > On Jan 21, 2008 8:38 PM, Charles R Harris wrote: > > > If I hit the source code link in the generated html, it looks like that > > page was generated from the old document format. The new document format > > doesn't produce output that looks anything like that and epydoc generates a > > couple of warnings: > > > > > > | File /home/charris/workspace/numpy/numpy/doc/example.py, line 19, in > > > | example.foo > > > | Warning: Line 24: Wrong underline character for heading. > > > | Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph > > > | indentation. > > > > > > The new document format requires a preprocessor that has yet to be > > written. > > > > Since epydoc also works for compiled modules, I think the easiest way to do > > that is to fork epydoc. The syntax of the new markup is context sensitive - > > single types are no longer in {} - so parsing will be a bit more > > complicated. As ReST is not part of the current document markup, we might > > consider removing the parts of the document documentation that refer to it. > > Hey Chuck, > > I am sorry that there has been so much confusion over the docstring > standards. As you know, the problems you are pointing out arose from > the changes Travis made in December. I added a warning to the top of > the coding style guideline explaining the situation: > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > Once Travis writes a tool to render the docstrings, we can go ahead > and update the instructions. > Thanks for the update in svn. It makes the current situation much clearer, I think. cheers, David From chan_dhf at yahoo.de Tue Jan 22 03:46:41 2008 From: chan_dhf at yahoo.de (Danny Chan) Date: Tue, 22 Jan 2008 09:46:41 +0100 (CET) Subject: [Numpy-discussion] compiling c extension In-Reply-To: <195621.19830.qm@web34411.mail.mud.yahoo.com> Message-ID: <504595.9740.qm@web26205.mail.ukl.yahoo.com> Yes, that is not the problem. The compiling of the source file works fine, it is when the linker tries to create the shared library file that something blows up. Lou Pecora schrieb: Did you include arrayobject.h and call import_array() in the initialization function, after the call to Py_InitModule() ? --- Danny Chan wrote: > Hi! > I am trying to compile a c extension that uses numpy > arrays on windows. I can compile the extension file, > but once I get to the linking stage I get unresolved > references to PyArray_API and import_array. Can > anyone help me with the correct linker flags? > > Thanks, Danny [cut] > more undefined references to `PyArray_API' follow > build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0xc216): > undefined reference to `import_array' > collect2: ld returned 1 exit status > error: command 'gcc' failed with exit status 1 -- Lou Pecora, my views are my own. ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion --------------------------------- Heute schon einen Blick in die Zukunft von E-Mails wagen? Versuchen Sie?s mit dem neuen Yahoo! Mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Tue Jan 22 04:13:07 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Tue, 22 Jan 2008 10:13:07 +0100 Subject: [Numpy-discussion] Docstring page, out of date? In-Reply-To: <47958CC7.5050005@gmail.com> References: <1e2af89e0801211403x437d30cfic7aa74e9b8fa6ea7@mail.gmail.com> <47958CC7.5050005@gmail.com> Message-ID: <200801221013.08170.meine@informatik.uni-hamburg.de> Am Dienstag, 22. Januar 2008 07:27:19 schrieb Robert Kern: > > The new document format requires a preprocessor that has yet to be > > written. > > epydoc from SVN works just fine once the following line is added at the > top. > > __docformat__ = "restructuredtext en" No need to clutter the files, just use "epydoc --docformat restructuredtext" Ciao, / / /--/ / / ANS From chan_dhf at yahoo.de Tue Jan 22 04:18:09 2008 From: chan_dhf at yahoo.de (Danny Chan) Date: Tue, 22 Jan 2008 10:18:09 +0100 (CET) Subject: [Numpy-discussion] compiling c extension In-Reply-To: <504595.9740.qm@web26205.mail.ukl.yahoo.com> Message-ID: <502826.86666.qm@web26210.mail.ukl.yahoo.com> Ha, found the problem! I thought I had included arrayobject.h and import_array correctly, but due to the preprocessor it wasn't defined correctly! Danny Chan schrieb: Yes, that is not the problem. The compiling of the source file works fine, it is when the linker tries to create the shared library file that something blows up. Lou Pecora schrieb: Did you include arrayobject.h and call import_array() in the initialization function, after the call to Py_InitModule() ? --- Danny Chan wrote: > Hi! > I am trying to compile a c extension that uses numpy > arrays on windows. I can compile the extension file, > but once I get to the linking stage I get unresolved > references to PyArray_API and import_array. Can > anyone help me with the correct linker flags? > > Thanks, Danny [cut] > more undefined references to `PyArray_API' follow > build\temp.win32-2.5\Release\.\python\libaid_wrap.o:libaid_wrap.c:(.text+0xc216): > undefined reference to `import_array' > collect2: ld returned 1 exit status > error: command 'gcc' failed with exit status 1 -- Lou Pecora, my views are my own. ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion --------------------------------- Heute schon einen Blick in die Zukunft von E-Mails wagen? Versuchen Sie?s mit dem neuen Yahoo! Mail. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion --------------------------------- Heute schon einen Blick in die Zukunft von E-Mails wagen? Versuchen Sie?s mit dem neuen Yahoo! Mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Jan 22 04:52:21 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 22 Jan 2008 11:52:21 +0200 Subject: [Numpy-discussion] Building a static libnumpy In-Reply-To: <5b8d13220801220012p6ac1b15er606e583c2e27af57@mail.gmail.com> References: <479590CA.9040404@csc.fi> <5b8d13220801220012p6ac1b15er606e583c2e27af57@mail.gmail.com> Message-ID: <20080122095221.GE31735@mentat.za.net> On Tue, Jan 22, 2008 at 05:12:32PM +0900, David Cournapeau wrote: > On Jan 22, 2008 3:44 PM, Jussi Enkovaara wrote: > > Hi, > > I need to use numpy in an environment which does not support shared libraries. > > Previously, I have used the old Numeric where I created a Makefile to build a > > static Numeric library which was later on linked to the python interpreter. > > > Interesting, did not know it was possible. How did you do it ? > > I am working on an alternative build system for numpy, based on scons, > and depending on the necessary steps, I could add this feature to the > build system. scons itself certainly support static libraries, but I > have never built static python extensions. I would also be interested in making such a build, since it would allow profiling using gcov. Cheers St?fan From jussi.enkovaara at csc.fi Tue Jan 22 05:52:34 2008 From: jussi.enkovaara at csc.fi (Jussi Enkovaara) Date: Tue, 22 Jan 2008 12:52:34 +0200 Subject: [Numpy-discussion] Building a static libnumpy In-Reply-To: <5b8d13220801220012p6ac1b15er606e583c2e27af57@mail.gmail.com> References: <479590CA.9040404@csc.fi> <5b8d13220801220012p6ac1b15er606e583c2e27af57@mail.gmail.com> Message-ID: <4795CAF2.70404@csc.fi> David Cournapeau wrote: > On Jan 22, 2008 3:44 PM, Jussi Enkovaara wrote: >> Hi, >> I need to use numpy in an environment which does not support shared libraries. >> Previously, I have used the old Numeric where I created a Makefile to build a >> static Numeric library which was later on linked to the python interpreter. >> > Interesting, did not know it was possible. How did you do it ? > > I am working on an alternative build system for numpy, based on scons, > and depending on the necessary steps, I could add this feature to the > build system. scons itself certainly support static libraries, but I > have never built static python extensions. The tricky part is the creation of the static library, with Numeric I just put all the object files to a library with 'ar', e.g. ar cr libnumpy.a $(OBJ) When having the static library, one can specify the static modules in the file Modules/Setup in Python source tree. The file is well documented and contains detailed instructions for building static modules. In the case of Numeric, I had the following lines in Setup: NUMPY = ${HOME}/Numeric-23.8/libnumpy.a _numpy $(NUMPY) arrayfns $(NUMPY) ranlib $(NUMPY) multiarray $(NUMPY) umath $(NUMPY) lapack_lite $(NUMPY) fftpack $(NUMPY) It is of course very cumbersome as one has to specify all the modules which are written in C before compiling the actual interpreter. I think that the whole procedure cannot be automatized, but it should be possible to have distutils to create the static library and produce maybe the lines to be included in Modules/Setup. Thus, one should first create a minimal interpreter, then build the necessary extensions statically with this minimal interpreter and distutils, and at the end create the full featured python interpreter. At some point I tried to look the distutils source but I did not have the time to understand it properly so that I could make the necessary modifications. Regards, Jussi Enkovaara From bsouthey at gmail.com Tue Jan 22 09:21:11 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 22 Jan 2008 08:21:11 -0600 Subject: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion) In-Reply-To: <5d3194020801211417t44469324xf1b812e2053670b@mail.gmail.com> References: <5d3194020801211417t44469324xf1b812e2053670b@mail.gmail.com> Message-ID: Hi, Also try NumExpr (http://www.scipy.org/SciPyPackages/NumExpr): "The scipy.sandbox.numexpr package supplies routines for the fast evaluation of array expressions elementwise by using a vector-based virtual machine. " However, this page is probably a little out of date. Regards Bruce On Jan 21, 2008 4:17 PM, Arnar Flatberg wrote: > Hi Theodore > > Probably not the fastest, but a full example of how you may vectorize your loop. > Ran just one test, and that speeded up the original code. > > Example: > ---------------- > > from numpy import empty_like > > def vectorized_rgb2hsv(im): > "im is a (m, n, 3) array.""" > im = im/255. > out = empty_like(im) > im_max = im.max(-1) > delta = im.ptp(-1) > s = delta/im_max > s[delta==0] = 0 > index = im[:,:,0] == im_max # red is max > out[index, 0] = (im[index, 1] - im[index, 2] ) / delta[index] > index = im[:,:,1] == im_max # green is max > out[index, 0] = 2 + (im[index, 2] - im[index, 0] ) / delta[index] > index = im[:,:,2] == im_max # blue is max > out[index, 0] = 4 + (im[index, 0] - im[index, 1] ) / delta[index] > out[:,:,0] = (out[:,:,0]/6.0) % 1.0 > out[:,:,1] = s > out[:,:,2] = im_max > out = (255.*out).astype(uint8) > return out > > > Timings (shape: 1202, 800, 3): > ----------------- > code in first post: > %prun a = rgb2hsv.rgb2hsv(s) > 1923207 function calls in 18.423 CPU seconds > > removing loop: > %prun b = rgb2hsv.vectorized_rgb2hsv(s) > 8 function calls in 4.630 CPU seconds > > > Arnar > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From gary.pajer at gmail.com Tue Jan 22 11:18:01 2008 From: gary.pajer at gmail.com (Gary Pajer) Date: Tue, 22 Jan 2008 11:18:01 -0500 Subject: [Numpy-discussion] data type specifications Message-ID: <88fe22a0801220818x189e5da2j220fdbea59fd86d8@mail.gmail.com> Occasionally I find myself poking into docstrings and googling randomly trying to find the proper way to specify a data type, or trying to remind myself just what a "float" is. I can never find the info easily. Preferable: Is there a docstring somewhere that lists the data types and acceptable ways to specify them? Less Preferable: Is there a web page? Least Preferable: I have The Book, but it's usually not accessible when I need it for this kind of on-the-fly question. thanks, gary From than at pixar.com Tue Jan 22 13:22:39 2008 From: than at pixar.com (Peter Ward) Date: Tue, 22 Jan 2008 10:22:39 -0800 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? Message-ID: <6D90660F-1E31-492A-BAF1-F9CD88214320@pixar.com> Hi, I have been trying to build numpy on an opteron based Solaris 10 machine, with poor results. After skimming over the discussion archives I noticed a branch for numpy.sunperf in the svn repository. Unfortunately this is no longer available, so I was curious as to the best approach to building numpy on my problem platform. Thanks, Pete From oliphant at enthought.com Tue Jan 22 14:49:20 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 13:49:20 -0600 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? In-Reply-To: <6D90660F-1E31-492A-BAF1-F9CD88214320@pixar.com> References: <6D90660F-1E31-492A-BAF1-F9CD88214320@pixar.com> Message-ID: <479648C0.1000103@enthought.com> Peter Ward wrote: > Hi, > I have been trying to build numpy on an opteron based Solaris 10 > machine, with poor results. > After skimming over the discussion archives I noticed a branch for > numpy.sunperf in the svn repository. > Unfortunately this is no longer available, so I was curious as to the > best approach to building > numpy on my problem platform. > In general, the key to building on a platform is the config.h file that numpy generates. It knows how to do it for a few platforms, but for others it may not work. In particular, we need to know how to find: 1) float and long double versions of functions 2) whether or not and how IEEE 754 floating point signalling is handled on your platform. There are a few other things that must be done. You can generate the config.h file by hand and then figure out how to get the build scripts to do it automatically. Start with a config.h file from another platform and start tracking through the error messages. It is a tedious process. If you post error messages (or file them on the Trac page for NumPy), we may be able to help. -Travis O. From tclarke at ball.com Tue Jan 22 15:02:48 2008 From: tclarke at ball.com (Clarke, Trevor) Date: Tue, 22 Jan 2008 15:02:48 -0500 Subject: [Numpy-discussion] Looking to access C array in numpy. Message-ID: <873043AF54FE794B912B616ABA7E9FE503D6668F@daytonmsg2k3.AERO.BALL.COM> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm calling some python code from a C++ app via an intermediary library (i.e. I can't directly create Python C objects like Buffers). I'm passing a void* (cast to a long) to the python method and I'd like to use numpy to access that memory as an array. I'll know what the C data type is and the minimum length of the data. It's a continuous array but is multiple dimensions (either 2 dimensional or 3 dimensional BIP). I was able to find a may to create an array accessor over a python buffer but I'm not sure how to back an array with a void* (as a long) or create a buffer object for that memory inside python. Could someone point me in the right direction? - ------------------------------ Trevor R.H. Clarke tclarke at ball.com Ball Aerospace & Technologies Corp GPG key available on random.sks.keyserver.penguin.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (MingW32) iD8DBQFHlkvo+xUTKUxH/LkRAuzCAJ4ojQRgMgQj6vbhnSU3pxfiilIRYACePqiR hZyD05z1CgJuOtzw7pdJ9TQ= =H+HS -----END PGP SIGNATURE----- This message and any enclosures are intended only for the addressee. Please notify the sender by email if you are not the intended recipient. If you are not the intended recipient, you may not use, copy, disclose, or distribute this message or its contents or enclosures to any other person and any such actions may be unlawful. Ball reserves the right to monitor and review all messages and enclosures sent to or from this email address. From oliphant at enthought.com Tue Jan 22 15:10:41 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 14:10:41 -0600 Subject: [Numpy-discussion] SciPy/NumPy Doc-day on Friday Jan 25th Message-ID: <47964DC1.7020404@enthought.com> It's time for another doc-day for SciPy/NumPy. We will convene on IRC during the day on Friday. I will try and spend some time in the afternoon/evening on Friday night (Central Standard Time), but will be logged on to IRC during the rest of the day. Come join us at irc.freenode.net (channel scipy). We may update the list of priorities which is still located on the SciPy Trac Wiki: http://projects.scipy.org/scipy/scipy/wiki/DocDays I look forward to "irc-eeing" as many of you as can participate on Friday-Saturday (depending on your timezone). -Travis O. From oliphant at enthought.com Tue Jan 22 15:30:41 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 14:30:41 -0600 Subject: [Numpy-discussion] Looking to access C array in numpy. In-Reply-To: <873043AF54FE794B912B616ABA7E9FE503D6668F@daytonmsg2k3.AERO.BALL.COM> References: <873043AF54FE794B912B616ABA7E9FE503D6668F@daytonmsg2k3.AERO.BALL.COM> Message-ID: <47965271.9090202@enthought.com> Clarke, Trevor wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm calling some python code from a C++ app via an intermediary library > (i.e. I can't directly create Python C objects like Buffers). I'm > passing a void* (cast to a long) to the python method and I'd like to > use numpy to access that memory as an array. I'll know what the C data > type is and the minimum length of the data. It's a continuous array but > is multiple dimensions (either 2 dimensional or 3 dimensional BIP). I > was able to find a may to create an array accessor over a python buffer > but I'm not sure how to back an array with a void* (as a long) or create > a buffer object for that memory inside python. Could someone point me in > the right direction? > A couple of options that I see: 1) Use ctypes to wrap your void* into an object that can be passed into numpy.frombuffer. 2) Use the "intentionally not well publicisized function" numpy.core.multiarray.int_asbuffer The latter function is not well publicisized because I think solution #1 if you can figure it out may be better long term. But, #2 has some interesting features like being able to classify the memory as read-only and being able to check to see if de-referencing the beginning and end of the region causes a segfault (which is caught on some platforms) and handled gracefully with a Python error. The calling syntax is int_asbuffer(mem=, size=, readonly=, check=) where mem: long integer representing a void * size: The size of the memory region being converted to a buffer object. readonly: Should this memory object be readonly or not (default FALSE) check: Should the routine try to de-reference the first and last byte of the memory to catch any segfault? (default TRUE). -Travis O. From theller at ctypes.org Tue Jan 22 15:57:34 2008 From: theller at ctypes.org (Thomas Heller) Date: Tue, 22 Jan 2008 21:57:34 +0100 Subject: [Numpy-discussion] __array_interface__ / __array_struct__ Message-ID: I am experimenting with implementing __array_interface__ and/or __array_struct__ properties for ctypes instances, and have problems to create numpy arrays from them that share the memory. Probably I'm doing something wrong; what is the correct function in numpy to create these shared objects? I am using numpy.core.multiarray.array(ctypes-object), is that correct? Thanks, Thomas From oliphant at enthought.com Tue Jan 22 16:03:54 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 15:03:54 -0600 Subject: [Numpy-discussion] Looking to access C array in numpy. In-Reply-To: <47965271.9090202@enthought.com> References: <873043AF54FE794B912B616ABA7E9FE503D6668F@daytonmsg2k3.AERO.BALL.COM> <47965271.9090202@enthought.com> Message-ID: <47965A3A.9020106@enthought.com> Travis E. Oliphant wrote: > Clarke, Trevor wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I'm calling some python code from a C++ app via an intermediary library >> (i.e. I can't directly create Python C objects like Buffers). I'm >> passing a void* (cast to a long) to the python method and I'd like to >> use numpy to access that memory as an array. I'll know what the C data >> type is and the minimum length of the data. It's a continuous array but >> is multiple dimensions (either 2 dimensional or 3 dimensional BIP). I >> was able to find a may to create an array accessor over a python buffer >> but I'm not sure how to back an array with a void* (as a long) or create >> a buffer object for that memory inside python. Could someone point me in >> the right direction? >> >> > A couple of options that I see: > > 1) Use ctypes to wrap your void* into an object that can be passed into > numpy.frombuffer. > Specifically: res = ctypes.ARRAY(ctypes.c_bytes, sizeofbuffer).from_address(void_ptr_as_long) array = np.frombuffer(res, dtype=mydtype) > 2) Use the "intentionally not well publicisized function" > numpy.core.multiarray.int_asbuffer > > res = numpy.core.multiarray.int_asbuffer(void_ptr_as_long, sizeofbuffer, readonly=False, check=False) array = np.frombuffer(res, dtype=mydtype) Have fun! -Travis O. From oliphant at enthought.com Tue Jan 22 16:05:52 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 15:05:52 -0600 Subject: [Numpy-discussion] __array_interface__ / __array_struct__ In-Reply-To: References: Message-ID: <47965AB0.6080604@enthought.com> Thomas Heller wrote: > I am experimenting with implementing __array_interface__ and/or __array_struct__ > properties for ctypes instances, and have problems to create numpy arrays > from them that share the memory. Probably I'm doing something wrong; > what is the correct function in numpy to create these shared objects? > > I am using numpy.core.multiarray.array(ctypes-object), is that correct? > Yes, this should work, as the array function goes through several checks including looking for the __array_struct__ and/or __array_interface__ attributes. If you can point me to the code, I can probably help. -Travis O. From theller at ctypes.org Tue Jan 22 16:15:02 2008 From: theller at ctypes.org (Thomas Heller) Date: Tue, 22 Jan 2008 22:15:02 +0100 Subject: [Numpy-discussion] __array_interface__ / __array_struct__ In-Reply-To: <47965AB0.6080604@enthought.com> References: <47965AB0.6080604@enthought.com> Message-ID: Travis E. Oliphant schrieb: > Thomas Heller wrote: >> I am experimenting with implementing __array_interface__ and/or __array_struct__ >> properties for ctypes instances, and have problems to create numpy arrays >> from them that share the memory. Probably I'm doing something wrong; >> what is the correct function in numpy to create these shared objects? >> >> I am using numpy.core.multiarray.array(ctypes-object), is that correct? >> > Yes, this should work, as the array function goes through several checks > including looking for the __array_struct__ and/or __array_interface__ > attributes. If you can point me to the code, I can probably help. > The pure-python code, using __array_interface__, is here: http://ctypes-stuff.googlecode.com/svn/trunk/numpy/ Use a webbrowser to view or download it, or an svn client to checkout a copy. I use it like this: from ctypes_array mport as_ctypes, as_array c_array = (c_double * 3)() numpy_array = as_array(c_array) or numpy_array = zeros(32) c_array = as_array(numpy_array) In the former example the objects to bot share memory, in the latter example they do. I also have an extension in the works that uses __array_struct__, but this is not yet uploaded. (See also the message titled 'ctypes to numpy and back' that I posted to the ctypes-users list and gmane.python.scientific.users. Probably the wrong lists since you did not see it ;-) Thanks, Thomas From oliphant at enthought.com Tue Jan 22 17:20:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Jan 2008 16:20:56 -0600 Subject: [Numpy-discussion] __array_interface__ / __array_struct__ In-Reply-To: References: <47965AB0.6080604@enthought.com> Message-ID: <47966C48.1050409@enthought.com> Thomas Heller wrote: > Travis E. Oliphant schrieb: > >> Thomas Heller wrote: >> >>> I am experimenting with implementing __array_interface__ and/or __array_struct__ >>> properties for ctypes instances, and have problems to create numpy arrays >>> from them that share the memory. Probably I'm doing something wrong; >>> what is the correct function in numpy to create these shared objects? >>> >>> I am using numpy.core.multiarray.array(ctypes-object), is that correct? >>> >>> >> Yes, this should work, as the array function goes through several checks >> including looking for the __array_struct__ and/or __array_interface__ >> attributes. If you can point me to the code, I can probably help. >> >> > > The pure-python code, using __array_interface__, is here: > > http://ctypes-stuff.googlecode.com/svn/trunk/numpy/ > > Use a webbrowser to view or download it, or an svn client > to checkout a copy. I use it like this: > > from ctypes_array mport as_ctypes, as_array > c_array = (c_double * 3)() > numpy_array = as_array(c_array) > > or > > numpy_array = zeros(32) > c_array = as_array(numpy_array) > > In the former example the objects to bot share memory, in the > latter example they do. > > I also have an extension in the works that uses __array_struct__, > but this is not yet uploaded. > This is all very good. The only thing missing that causes the former to not share memory is you are missing a copy=False argument to the multi_array function. Thus: return multi_array(obj, copy=0) is what you need to use. Also, the address of the memory is also available as arr.ctypes.data if arr is a NumPy array (but this is less general than using the array interface for sure). -Travis O. From mattknox_ca at hotmail.com Tue Jan 22 22:37:29 2008 From: mattknox_ca at hotmail.com (Matt Knox) Date: Tue, 22 Jan 2008 22:37:29 -0500 Subject: [Numpy-discussion] location of ma in maskedarray branch Message-ID: I noticed that the new masked array module resides in numpy/ma in the maskedarray branch as opposed to numpy/core/ma like it does in the current trunk. Was this intentional? Code that explicitly imports ma from the core subfolder will break from this change (like the __init__.py script for the ma subfolder in matplotlib for example). - Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jan 22 22:32:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 23 Jan 2008 12:32:20 +0900 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? In-Reply-To: <479648C0.1000103@enthought.com> References: <6D90660F-1E31-492A-BAF1-F9CD88214320@pixar.com> <479648C0.1000103@enthought.com> Message-ID: <4796B544.9000908@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > Peter Ward wrote: > >> Hi, >> I have been trying to build numpy on an opteron based Solaris 10 >> machine, with poor results. >> After skimming over the discussion archives I noticed a branch for >> numpy.sunperf in the svn repository. >> Unfortunately this is no longer available, so I was curious as to the >> best approach to building >> numpy on my problem platform. >> >> > In general, the key to building on a platform is the config.h file that > numpy generates. It knows how to do it for a few platforms, but for > others it may not work. > > In particular, we need to know how to find: > > 1) float and long double versions of functions > 2) whether or not and how IEEE 754 floating point signalling is handled > on your platform. > > There are a few other things that must be done. You can generate the > config.h file by hand and then figure out how to get the build scripts > to do it automatically. Start with a config.h file from another > platform and start tracking through the error messages. > The current config.h works fine for solaris with Sun compilers, in my experience, so the problem must be somewhere else. Peter, could you post the errors you got ? As an alternative, I am working on an alternative build system for numpy: it should work on solaris (tested on Indiana, and other people reported success on solaris 9 and 10). Unfortunately, I have been redesigning the internals quite a lot lately, and I have not tested the changes on solaris yet. If you are willing to test, it should be easy for me to make it work again on solaris in a few minutes, though. cheers, David From millman at berkeley.edu Wed Jan 23 01:04:28 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 22 Jan 2008 22:04:28 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: On Jan 18, 2008 11:17 PM, Barry Wark wrote: > I promise: last change. I changed the URL to > http://physionconsulting.blogspot.com/search/label/scipy. My wife said > physion consultants is a crappy name. Oh well :) Done. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From efiring at hawaii.edu Wed Jan 23 02:40:02 2008 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 22 Jan 2008 21:40:02 -1000 Subject: [Numpy-discussion] location of ma in maskedarray branch In-Reply-To: References: Message-ID: <4796EF52.4050107@hawaii.edu> Matt Knox wrote: > I noticed that the new masked array module resides in numpy/ma in the > maskedarray branch as opposed to numpy/core/ma like it does in the > current trunk. Was this intentional? Code that explicitly imports ma > from the core subfolder will break from this change (like the > __init__.py script for the ma subfolder in matplotlib for example). I modified mpl to handle this new location, assuming it was intentional and likely to remain. The change is included in release 0.91.2 as well as in the svn trunk (formerly the transforms branch) and the svn maintenance branch derived from 0.91.2). The previous location is still supported also. Eric From stefan at sun.ac.za Wed Jan 23 02:48:17 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 23 Jan 2008 09:48:17 +0200 Subject: [Numpy-discussion] data type specifications In-Reply-To: <88fe22a0801220818x189e5da2j220fdbea59fd86d8@mail.gmail.com> References: <88fe22a0801220818x189e5da2j220fdbea59fd86d8@mail.gmail.com> Message-ID: <20080123074817.GC25779@mentat.za.net> Hi Gary On Tue, Jan 22, 2008 at 11:18:01AM -0500, Gary Pajer wrote: > Occasionally I find myself poking into docstrings and googling > randomly trying to find the proper way to specify a data type, or > trying to remind myself just what a "float" is. I can never find the > info easily. > > Preferable: Is there a docstring somewhere that lists the data types > and acceptable ways to specify them? > Less Preferable: Is there a web page? This should be a good starting place (search for dtype): http://www.scipy.org/Numpy_Example_List Regards St?fan From stefan at sun.ac.za Wed Jan 23 03:05:52 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 23 Jan 2008 10:05:52 +0200 Subject: [Numpy-discussion] location of ma in maskedarray branch In-Reply-To: References: Message-ID: <20080123080552.GD25779@mentat.za.net> Hi Matt On Tue, Jan 22, 2008 at 10:37:29PM -0500, Matt Knox wrote: > I noticed that the new masked array module resides in numpy/ma in the > maskedarray branch as opposed to numpy/core/ma like it does in the current > trunk. Was this intentional? Code that explicitly imports ma from the core > subfolder will break from this change (like the __init__.py script for the ma > subfolder in matplotlib for example). Yes, the move was intentional and was discussed beforehand. I may be mistaken, but as far as I know numpy.core is not a public API, so code should rather do from numpy import ma (that works with both the current trunk and the maskedarray branch). Regards St?fan From cournape at gmail.com Wed Jan 23 03:34:24 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 23 Jan 2008 17:34:24 +0900 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? In-Reply-To: <4796B544.9000908@ar.media.kyoto-u.ac.jp> References: <6D90660F-1E31-492A-BAF1-F9CD88214320@pixar.com> <479648C0.1000103@enthought.com> <4796B544.9000908@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220801230034w4cd7eeake3becbb619f03a25@mail.gmail.com> On Jan 23, 2008 12:32 PM, David Cournapeau wrote: > > The current config.h works fine for solaris with Sun compilers, in my > experience, so the problem must be somewhere else. I've just tried numpy SVN with sun compiler (Sun studio 12) on Indiana, and it works fine, so there is no recent regressions on this. The problem may be because of opteron (64 bits). cheers, David From pearu at cens.ioc.ee Wed Jan 23 06:11:28 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 23 Jan 2008 13:11:28 +0200 (EET) Subject: [Numpy-discussion] ndarray.__mul__ is too gready? Message-ID: <54727.85.166.31.187.1201086688.squirrel@cens.ioc.ee> Hi, Say, one defines a class A that does not like to have numpy arrays in the left-hand-side of an operation, say *, but in the rhs it is expected. For an example, consider A as a huge customized matrix that defines only matrix * vector where vector can be numpy array and scalar * matrix where scalar is python float or int. Users may accidentally hit vector * matrix and this should raise an exception. But when vector is numpy array, then ndarray.__mul__ calls element * A for each array element and here A.__rmul__ gets only the types of array elements (that are floats, for instance) as rhs and hence is not able to decide that it should return NotImplemented. Instead, it computes scalar * matrix and the final result will be array([element1*matrix, element2*matrix, ..]) which is not desired already because of the memory usage and should have raised an exception instead. Here is a small example illustrating the problem: class A: def __rmul__(self, other): if isinstance(other, ndarray): return NotImplemented if isinstance(other, (int, float)): return self # pretend other==1 for an example return NotImplemented def __repr__(self): return 'A()' >>> array([1,2,3]) * A() array([A(), A(), A()], dtype=object) # expected TypeError Is there any workaround to this problem? In other situations/applications the above behavior of ndarray.__mul__ is a feature but is there any way to avoid this feature? Thanks, Pearu From gary.pajer at gmail.com Wed Jan 23 07:00:55 2008 From: gary.pajer at gmail.com (Gary Pajer) Date: Wed, 23 Jan 2008 07:00:55 -0500 Subject: [Numpy-discussion] data type specifications In-Reply-To: <20080123074817.GC25779@mentat.za.net> References: <88fe22a0801220818x189e5da2j220fdbea59fd86d8@mail.gmail.com> <20080123074817.GC25779@mentat.za.net> Message-ID: <88fe22a0801230400g4b6758e9q45bd050d28525e44@mail.gmail.com> On Jan 23, 2008 2:48 AM, Stefan van der Walt wrote: > Hi Gary > > On Tue, Jan 22, 2008 at 11:18:01AM -0500, Gary Pajer wrote: > > Occasionally I find myself poking into docstrings and googling > > randomly trying to find the proper way to specify a data type, or > > trying to remind myself just what a "float" is. I can never find the > > info easily. > > > > Preferable: Is there a docstring somewhere that lists the data types > > and acceptable ways to specify them? > > Less Preferable: Is there a web page? > > This should be a good starting place (search for dtype): > > http://www.scipy.org/Numpy_Example_List Thanks, St?fan. That link led me to the numpy dictionary numpy.typeDict which lists them all. -gary > > Regards > St?fan > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Wed Jan 23 08:55:46 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 23 Jan 2008 15:55:46 +0200 Subject: [Numpy-discussion] Docstring standard: how to specify variable types Message-ID: <20080123135546.GK25779@mentat.za.net> Hi all, The numpy documentation standard example shows: Parameters ---------- var1 : array_like Array_like means all those objects -- lists, nested lists, etc. -- that can be converted to an array. var2 : integer Write out the full type long_variable_name : {'hi', 'ho'}, optional Choices in brackets, default first when optional. I'd like to know: 1. "array_like" describes objects that can be forced to quack like ndarrays. Are there any other such "special" descriptions? 2. How do we specify default values? 3. Why do we need the "optional" keyword (the function signature already indicates that the parameter is optional). 4. Do we really need the "Other Parameters" list? It would make more sense to split positional and keyword arguments, but I'm not even sure that is necessary, since that information is already specified in the function signature. 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a limited number of values? In Python {} is a dictionary, so why not use ('hi','ho') instead? Thanks for your feedback! Regards St?fan From david at ar.media.kyoto-u.ac.jp Wed Jan 23 10:34:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 24 Jan 2008 00:34:56 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release Message-ID: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> Hi, I've just released the 0.3.0 release of numscons, an alternative build system for numpy. The tarballs are available on launchpad. https://launchpad.net/numpy.scons.support/0.3/0.3.0 To use it, you need to get the build_with_scons numpy branch: see http://projects.scipy.org/scipy/numpy/wiki/NumpyScons for more details. This release is an important milestone: - all regressions because of the split from numpy are fixed. - it can build numpy on linux (gcc/intel), mac os X (gcc), windows (mingw) and solaris (Sun compilers). - mkl, sunperf, accelerate/vecLib frameworks and ATLAS should work on the platforms where it makes sense. - a lot of internal changes: some basic unittest, a total revamp of the code to check for performance libraries. - almost all changes necessary to scons code are now included upstream, or pending review. If you test it and has problems building numpy, please submit a bug to launchpad: https://bugs.launchpad.net/numpy.scons.support/0.3/ Thanks, David From charlesr.harris at gmail.com Wed Jan 23 11:34:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Jan 2008 09:34:44 -0700 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <20080123135546.GK25779@mentat.za.net> References: <20080123135546.GK25779@mentat.za.net> Message-ID: On Jan 23, 2008 6:55 AM, Stefan van der Walt wrote: > Hi all, > > The numpy documentation standard example shows: > > Parameters > ---------- > var1 : array_like > Array_like means all those objects -- lists, nested lists, etc. -- > that can be converted to an array. > var2 : integer > Write out the full type > long_variable_name : {'hi', 'ho'}, optional > Choices in brackets, default first when optional. > > I'd like to know: > > 1. "array_like" describes objects that can be forced to quack like > ndarrays. Are there any other such "special" descriptions? > I can't think of any, but that doesn't mean there aren't any. > 2. How do we specify default values? > I like to put them first in the list: {-1, integer} > 3. Why do we need the "optional" keyword (the function signature > already indicates that the parameter is optional). > It's extra information, true, but that isn't always a bad thing. It's like looking up "whole numbers" in a book index and, instead of the page reference, the index directs you to "numbers, whole". Flip, flip, flip. Drives me crazy. Besides, the function signature might be missing. > 4. Do we really need the "Other Parameters" list? It would make more > sense to split positional and keyword arguments, but I'm not even > sure that is necessary, since that information is already specified in > the > function signature. > I agree, but Travis likes this section and I don't feel strongly about it. > > 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a > limited number of values? In Python {} is a dictionary, so why not > use ('hi','ho') instead? > Either would be fine. IIRC, the {} was inherited from epydoc consolidated fields. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpmusu at cc.usu.edu Wed Jan 23 12:07:18 2008 From: mpmusu at cc.usu.edu (Mark.Miller) Date: Wed, 23 Jan 2008 10:07:18 -0700 Subject: [Numpy-discussion] changed behavior of numpy.histogram Message-ID: <47977446.4030407@cc.usu.edu> Greetings: I just noticed a changed behavior of numpy.histogram. I think that a recent 'fix' to the code has changed my ability to use that function (albeit in an unconventional manner). I previously used the histogram function to obtain counts of each unique string within a string array. Again, I recognize that it is not a typical use of the histogram function, but it did work very nicely for me. Here's an example: ###numpy 1.0.3 --works just fine >>> import numpy >>> numpy.__version__ '1.0.3' >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) >>> a array(['atcg', 'atcg', 'aaaa', 'aaaa'], dtype='|S4') >>> b=numpy.unique(a) >>> numpy.histogram(a,b) (array([2, 2]), array(['aaaa', 'atcg'], dtype='|S4')) >>> ###numpy 1.0.4 --no longer functions >>> import numpy >>> numpy.__version__ '1.0.4' >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) >>> a array(['atcg', 'atcg', 'aaaa', 'aaaa'], dtype='|S4') >>> b=numpy.unique(a) >>> numpy.histogram(a,b) Traceback (most recent call last): File "", line 1, in File "/opt/libraries/python/python-2.5.1/numpy-1.0.4-gnu/lib/python2.5/site-packages/numpy/lib/function_base.py", line 154, in histogram if(any(bins[1:]-bins[:-1] < 0)): TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'numpy.ndarray' >>> Is this something that can possibly be fixed (should I submit a ticket)? Or should I revert to some other approaches for implementing the same idea? It really was a nice convenience. Or, alternately, would some sort of new function along the lines of a numpy.countunique() ultimately be useful? Thanks, -Mark From sdb at cloud9.net Wed Jan 23 12:14:17 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Wed, 23 Jan 2008 12:14:17 -0500 (EST) Subject: [Numpy-discussion] changed behavior of numpy.histogram In-Reply-To: <47977446.4030407@cc.usu.edu> References: <47977446.4030407@cc.usu.edu> Message-ID: Hi -- > Greetings: I just noticed a changed behavior of numpy.histogram. I > think that a recent 'fix' to the code has changed my ability to use that > function (albeit in an unconventional manner). You can blame me for this. I submitted a patch which prohibited the user from entering any range into histogram other than a monotonically increasing one. The ticket with the info is: http://scipy.org/scipy/numpy/ticket/586 This patch was apparently applied by the NumPy folks, and it broke your code. Sorry! > Is this something that can possibly be fixed (should I submit a ticket)? > Or should I revert to some other approaches for implementing the same > idea? It really was a nice convenience. Or, alternately, would some > sort of new function along the lines of a numpy.countunique() ultimately > be useful? IMO, your use was invalid (albeit a cute and useful shortcut). Allowing non-monotonically increasing bins can lead to confusing or inexplicable results. On the other hand, I would support the idea of a new function numpy.countunique() as you suggest. Just my two cents... Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From oliphant at enthought.com Wed Jan 23 12:15:15 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 23 Jan 2008 11:15:15 -0600 Subject: [Numpy-discussion] changed behavior of numpy.histogram In-Reply-To: <47977446.4030407@cc.usu.edu> References: <47977446.4030407@cc.usu.edu> Message-ID: <47977623.5000403@enthought.com> Mark.Miller wrote: > Greetings: I just noticed a changed behavior of numpy.histogram. I > think that a recent 'fix' to the code has changed my ability to use that > function (albeit in an unconventional manner). I previously used the > histogram function to obtain counts of each unique string within a > string array. Again, I recognize that it is not a typical use of the > histogram function, but it did work very nicely for me. > > Here's an example: > > ###numpy 1.0.3 --works just fine > >>> import numpy > >>> numpy.__version__ > '1.0.3' > >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) > >>> a > array(['atcg', 'atcg', 'aaaa', 'aaaa'], > dtype='|S4') > >>> b=numpy.unique(a) > >>> numpy.histogram(a,b) > (array([2, 2]), array(['aaaa', 'atcg'], > dtype='|S4')) > >>> > > ###numpy 1.0.4 --no longer functions > >>> import numpy > >>> numpy.__version__ > '1.0.4' > >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) > >>> a > array(['atcg', 'atcg', 'aaaa', 'aaaa'], > dtype='|S4') > >>> b=numpy.unique(a) > >>> numpy.histogram(a,b) > Traceback (most recent call last): > File "", line 1, in > File > "/opt/libraries/python/python-2.5.1/numpy-1.0.4-gnu/lib/python2.5/site-packages/numpy/lib/function_base.py", > line 154, in histogram > if(any(bins[1:]-bins[:-1] < 0)): > TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and > 'numpy.ndarray' > >>> > > Is this something that can possibly be fixed (should I submit a ticket)? > Or should I revert to some other approaches for implementing the same > idea? It really was a nice convenience. Or, alternately, would some > sort of new function along the lines of a numpy.countunique() ultimately > be useful? > It can be and should be fixed. Interesting. I'll have to look back at the changes to see what was done before. The problem is the string array does not support the monoticity check (it is also unnecessary for that use case, I believe). -Travis From sdb at cloud9.net Wed Jan 23 12:47:15 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Wed, 23 Jan 2008 12:47:15 -0500 (EST) Subject: [Numpy-discussion] changed behavior of numpy.histogram In-Reply-To: <47977446.4030407@cc.usu.edu> References: <47977446.4030407@cc.usu.edu> Message-ID: Hi again -- You made me feel guilty about breaking your code. Here's some suggested substitute code : In [10]: import numpy In [11]: a = numpy.array(('atcg', 'aaaa', 'atcg', 'actg', 'aaaa')) In [12]: b = numpy.sort(a) In [13]: c = numpy.unique(b) In [14]: d = numpy.searchsorted(b, c) In [15]: e = numpy.append(d[1:], len(a)) In [16]: f = e - d In [17]: In [17]: print c ['aaaa' 'actg' 'atcg'] In [18]: print f [2 1 2] Note that histogram also uses searchsorted to do its stuff. Personally, I think the way to go is have a "countunique" function which returns a list of unique occurrances of the array elements (regardless of their type), and a list of their count. The above code could be a basis for this fcn. I'm not sure that this should be implemented using histogram, since at least I ordinarily consider histogram as a numeric function. Others may have different opinions. Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ On Wed, 23 Jan 2008, Mark.Miller wrote: > Greetings: I just noticed a changed behavior of numpy.histogram. I > think that a recent 'fix' to the code has changed my ability to use that > function (albeit in an unconventional manner). I previously used the > histogram function to obtain counts of each unique string within a > string array. Again, I recognize that it is not a typical use of the > histogram function, but it did work very nicely for me. > > Here's an example: > > ###numpy 1.0.3 --works just fine > >>> import numpy > >>> numpy.__version__ > '1.0.3' > >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) > >>> a > array(['atcg', 'atcg', 'aaaa', 'aaaa'], > dtype='|S4') > >>> b=numpy.unique(a) > >>> numpy.histogram(a,b) > (array([2, 2]), array(['aaaa', 'atcg'], > dtype='|S4')) > >>> > > ###numpy 1.0.4 --no longer functions > >>> import numpy > >>> numpy.__version__ > '1.0.4' > >>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) > >>> a > array(['atcg', 'atcg', 'aaaa', 'aaaa'], > dtype='|S4') > >>> b=numpy.unique(a) > >>> numpy.histogram(a,b) > Traceback (most recent call last): > File "", line 1, in > File > "/opt/libraries/python/python-2.5.1/numpy-1.0.4-gnu/lib/python2.5/site-packages/numpy/lib/function_base.py", > line 154, in histogram > if(any(bins[1:]-bins[:-1] < 0)): > TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and > 'numpy.ndarray' > >>> > > Is this something that can possibly be fixed (should I submit a ticket)? > Or should I revert to some other approaches for implementing the same > idea? It really was a nice convenience. Or, alternately, would some > sort of new function along the lines of a numpy.countunique() ultimately > be useful? > > Thanks, > > -Mark > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From mpmusu at cc.usu.edu Wed Jan 23 13:03:20 2008 From: mpmusu at cc.usu.edu (Mark.Miller) Date: Wed, 23 Jan 2008 11:03:20 -0700 Subject: [Numpy-discussion] changed behavior of numpy.histogram In-Reply-To: References: <47977446.4030407@cc.usu.edu> Message-ID: <47978168.6010609@cc.usu.edu> Nah...no worries Stuart. Again, I recognize that what I was doing deviated from the likely true intent of the histogram function. But it was a nice convenient bit of code, for sure. I'll take a look at your suggestion...it's different than what I previously used. So, thanks for the input. And yes...your description of a countunique() function is precisely what I had in mind. Might be very useful if it could also work on object arrays. Thanks again, -Mark Stuart Brorson wrote: > Hi again -- > > You made me feel guilty about breaking your code. Here's some > suggested substitute code : > > In [10]: import numpy > > In [11]: a = numpy.array(('atcg', 'aaaa', 'atcg', 'actg', 'aaaa')) > > In [12]: b = numpy.sort(a) > > In [13]: c = numpy.unique(b) > > In [14]: d = numpy.searchsorted(b, c) > > In [15]: e = numpy.append(d[1:], len(a)) > > In [16]: f = e - d > > In [17]: > > In [17]: print c > ['aaaa' 'actg' 'atcg'] > > In [18]: print f > [2 1 2] > > Note that histogram also uses searchsorted to do its stuff. > > Personally, I think the way to go is have a "countunique" function > which returns a list of unique occurrances of the array elements > (regardless of their type), and a list of their count. The above code > could be a basis for this fcn. > > I'm not sure that this should be implemented using histogram, since > at least I ordinarily consider histogram as a numeric function. > Others may have different opinions. > > Cheers, > > Stuart Brorson > Interactive Supercomputing, inc. > 135 Beaver Street | Waltham | MA | 02452 | USA > http://www.interactivesupercomputing.com/ > > > On Wed, 23 Jan 2008, Mark.Miller wrote: > >> Greetings: I just noticed a changed behavior of numpy.histogram. I >> think that a recent 'fix' to the code has changed my ability to use that >> function (albeit in an unconventional manner). I previously used the >> histogram function to obtain counts of each unique string within a >> string array. Again, I recognize that it is not a typical use of the >> histogram function, but it did work very nicely for me. >> >> Here's an example: >> >> ###numpy 1.0.3 --works just fine >>>>> import numpy >>>>> numpy.__version__ >> '1.0.3' >>>>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) >>>>> a >> array(['atcg', 'atcg', 'aaaa', 'aaaa'], >> dtype='|S4') >>>>> b=numpy.unique(a) >>>>> numpy.histogram(a,b) >> (array([2, 2]), array(['aaaa', 'atcg'], >> dtype='|S4')) >> ###numpy 1.0.4 --no longer functions >>>>> import numpy >>>>> numpy.__version__ >> '1.0.4' >>>>> a=numpy.array(('atcg', 'atcg', 'aaaa', 'aaaa')) >>>>> a >> array(['atcg', 'atcg', 'aaaa', 'aaaa'], >> dtype='|S4') >>>>> b=numpy.unique(a) >>>>> numpy.histogram(a,b) >> Traceback (most recent call last): >> File "", line 1, in >> File >> "/opt/libraries/python/python-2.5.1/numpy-1.0.4-gnu/lib/python2.5/site-packages/numpy/lib/function_base.py", >> line 154, in histogram >> if(any(bins[1:]-bins[:-1] < 0)): >> TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and >> 'numpy.ndarray' >> Is this something that can possibly be fixed (should I submit a ticket)? >> Or should I revert to some other approaches for implementing the same >> idea? It really was a nice convenience. Or, alternately, would some >> sort of new function along the lines of a numpy.countunique() ultimately >> be useful? >> >> Thanks, >> >> -Mark >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From than at pixar.com Wed Jan 23 13:06:12 2008 From: than at pixar.com (Peter Ward) Date: Wed, 23 Jan 2008 10:06:12 -0800 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? In-Reply-To: References: Message-ID: <779A7062-4B27-44B5-9B56-1E530B8FF23B@pixar.com> David Cournapeau wrote: > The current config.h works fine for solaris with Sun compilers, in my > experience, so the problem must be somewhere else. > > Peter, could you post the errors you got ? As an alternative, I am > working on an alternative build system for numpy: it should work on > solaris (tested on Indiana, and other people reported success on > solaris > 9 and 10). Unfortunately, I have been redesigning the internals > quite a > lot lately, and I have not tested the changes on solaris yet. If > you are > willing to test, it should be easy for me to make it work again on > solaris in a few minutes, though. > The problems I was having were due to a bad site.cfg initially and then a problem with the python pkg from sunfreeware (ctypes version mismatch). Numpy is now happily installed. If you need someone to test anything new, let me know. Thanks! Pete From cbc at unc.edu Wed Jan 23 14:09:58 2008 From: cbc at unc.edu (Chris Calloway) Date: Wed, 23 Jan 2008 14:09:58 -0500 Subject: [Numpy-discussion] Sage/Scipy Days 8 reminder: Feb 29-March 4. In-Reply-To: References: Message-ID: <47979106.6040901@unc.edu> Fernando Perez wrote: > Just a quick reminder for all about the upcoming Sage/Scipy Days 8 at > Enthought collaborative meeting: > > http://wiki.sagemath.org/days8 This page says "February 29 until March 4, *2007*." -- Sincerely, Chris Calloway http://www.seacoos.org office: 332 Chapman Hall phone: (919) 962-4323 mail: Campus Box #3300, UNC-CH, Chapel Hill, NC 27599 From theller at ctypes.org Wed Jan 23 14:15:38 2008 From: theller at ctypes.org (Thomas Heller) Date: Wed, 23 Jan 2008 20:15:38 +0100 Subject: [Numpy-discussion] __array_interface__ / __array_struct__ In-Reply-To: <47966C48.1050409@enthought.com> References: <47965AB0.6080604@enthought.com> <47966C48.1050409@enthought.com> Message-ID: Travis E. Oliphant schrieb: > Thomas Heller wrote: >> Travis E. Oliphant schrieb: >> >>> Thomas Heller wrote: >>> >>>> I am experimenting with implementing __array_interface__ and/or __array_struct__ >>>> properties for ctypes instances, and have problems to create numpy arrays >>>> from them that share the memory. Probably I'm doing something wrong; >>>> what is the correct function in numpy to create these shared objects? >>>> >>>> I am using numpy.core.multiarray.array(ctypes-object), is that correct? >>>> >>>> >>> Yes, this should work, as the array function goes through several checks >>> including looking for the __array_struct__ and/or __array_interface__ >>> attributes. If you can point me to the code, I can probably help. >> >> The pure-python code, using __array_interface__, is here: >> >> http://ctypes-stuff.googlecode.com/svn/trunk/numpy/ >> > > This is all very good. The only thing missing that causes the former to > not share memory is you are missing a copy=False argument to the > multi_array function. > > Thus: > > return multi_array(obj, copy=0) > > is what you need to use. Cool! Thanks for that, it works now ;-). > Also, the address of the memory is also available as > arr.ctypes.data if arr is a NumPy array (but this is less general than > using the array interface for sure). I'm not completely sure what you are suggesting here; I just use all the info in the numpy array's __array_interface__ property to find the ctypes type to construct. And the address also is available in this property. Thanks, Thomas From fperez.net at gmail.com Wed Jan 23 14:29:42 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Jan 2008 12:29:42 -0700 Subject: [Numpy-discussion] Sage/Scipy Days 8 reminder: Feb 29-March 4. In-Reply-To: <47979106.6040901@unc.edu> References: <47979106.6040901@unc.edu> Message-ID: On Jan 23, 2008 12:09 PM, Chris Calloway wrote: > Fernando Perez wrote: > > Just a quick reminder for all about the upcoming Sage/Scipy Days 8 at > > Enthought collaborative meeting: > > > > http://wiki.sagemath.org/days8 > > This page says "February 29 until March 4, *2007*." Oops, typo. Fixed, thanks. f From efiring at hawaii.edu Wed Jan 23 16:17:51 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 23 Jan 2008 11:17:51 -1000 Subject: [Numpy-discussion] numpy.ma.compress Message-ID: <4797AEFF.3080509@hawaii.edu> Pierre, numpy.compress exists, but numpy.ma.compress does not; is this intentional? Eric From stefan at sun.ac.za Wed Jan 23 17:05:25 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 00:05:25 +0200 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <4797AEFF.3080509@hawaii.edu> References: <4797AEFF.3080509@hawaii.edu> Message-ID: <20080123220525.GA7352@mentat.za.net> On Wed, Jan 23, 2008 at 11:17:51AM -1000, Eric Firing wrote: > Pierre, > > numpy.compress exists, but numpy.ma.compress does not; is this > intentional? Looks like x.compress exists, but it doesn't work as expected: x = N.ma.array([1,2,3],mask=[True,False,Fals]) x.compress(x<2) throws ValueError: total size of new array must be unchanged Regards St?fan From stefan at sun.ac.za Wed Jan 23 17:15:07 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 00:15:07 +0200 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <4797AEFF.3080509@hawaii.edu> References: <4797AEFF.3080509@hawaii.edu> Message-ID: <20080123221507.GB7352@mentat.za.net> Hi Eric, On Wed, Jan 23, 2008 at 11:17:51AM -1000, Eric Firing wrote: > Pierre, > > numpy.compress exists, but numpy.ma.compress does not; is this > intentional? Thanks for the report. I added a simple implementation to SVN for the time being. Regards St?fan From efiring at hawaii.edu Wed Jan 23 17:25:03 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 23 Jan 2008 12:25:03 -1000 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <200801231632.43001.pgmdevlist@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <200801231632.43001.pgmdevlist@gmail.com> Message-ID: <4797BEBF.7010405@hawaii.edu> Pierre GM wrote: > On Wednesday 23 January 2008 16:17:51 you wrote: >> Pierre, >> >> numpy.compress exists, but numpy.ma.compress does not; is this intentional? > > Probably not. I usually don't use this function, preferring to use indexing > instead. If you have a need for it, I can probably come up with something > relatively soon: the basis would be to apply compress first on the ._data > part, return a view with the same class as the original object, and update > the mask with compress as needed. The only reason to add it would be backwards compatibility. Evidently it exists in original numpy.ma. Mike D used it in mpl's path.py until today when someone pointed out that it did not work with the maskedarray branch. (I think I tripped over the same thing a couple days ago, but worked around it at a higher level.) I agree that it is better *not* to use it, and we can easily strip it out of mpl if it occurs anywhere else; but there may be other user code that will trip over its absence when 1.05 comes out, so it might be easier to put it in, preserving similarity to numpy as well as old numpy.ma, than to leave it out and have to field future questions about it. Eric From barrywark at gmail.com Wed Jan 23 17:54:56 2008 From: barrywark at gmail.com (Barry Wark) Date: Wed, 23 Jan 2008 14:54:56 -0800 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: <85b5c3130801011207v61d55a7du8c0a8907dc52011e@mail.gmail.com> Message-ID: thanks. On Jan 22, 2008 10:04 PM, Jarrod Millman wrote: > On Jan 18, 2008 11:17 PM, Barry Wark wrote: > > I promise: last change. I changed the URL to > > http://physionconsulting.blogspot.com/search/label/scipy. My wife said > > physion consultants is a crappy name. Oh well :) > > Done. > > -- > > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Wed Jan 23 18:10:43 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Jan 2008 18:10:43 -0500 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <20080123220525.GA7352@mentat.za.net> References: <4797AEFF.3080509@hawaii.edu> <20080123220525.GA7352@mentat.za.net> Message-ID: <200801231810.44517.pgmdevlist@gmail.com> On Wednesday 23 January 2008 17:05:25 Stefan van der Walt wrote: > On Wed, Jan 23, 2008 at 11:17:51AM -1000, Eric Firing wrote: > > Pierre, > > > > numpy.compress exists, but numpy.ma.compress does not; is this > > intentional? > > Looks like x.compress exists, but it doesn't work as expected: > > x = N.ma.array([1,2,3],mask=[True,False,Fals]) > x.compress(x<2) > > throws > > ValueError: total size of new array must be unchanged Hardly a surprise: .compress is inherited from ndarray, and doesn't know how to handle the mask. We end up with a mask that doesn't have the size of the data, hence the ValueError. Stefan's modification should work for now. I'm working on a version closer to the numpy original (that accepts the other arguments as well), I'll try to commit it later tonight. Thanks again for your help ! From pgmdevlist at gmail.com Wed Jan 23 18:15:25 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Jan 2008 18:15:25 -0500 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <20080123221507.GB7352@mentat.za.net> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> Message-ID: <200801231815.26464.pgmdevlist@gmail.com> Oh, a last comment question: Is the current documentation standard definitive ? It seems to have changed in the last few weeks (I haven't really followed what happened). If yes, I'll try to correct the current SVN version, but that may take a little while... From jbattat at cfa.harvard.edu Wed Jan 23 18:32:06 2008 From: jbattat at cfa.harvard.edu (James Battat) Date: Wed, 23 Jan 2008 18:32:06 -0500 (EST) Subject: [Numpy-discussion] where() on mac/pc return different things In-Reply-To: <200801231815.26464.pgmdevlist@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> <200801231815.26464.pgmdevlist@gmail.com> Message-ID: Hi, numpy.where() returns different things on my windowsXP machine than on my collaborator's mac osx machine. For example, consider: >>> import numpy >>> a = numpy.array([1,2,3,4]) >>> b = numpy.where( a > 2 ) On WindowsXP (python 2.4.2, numpy 1.0.1), I get: >>> print b (array([2, 3]), ) >>> print b[0] [2 3] While in OSX (python 2.4.3, numpy 0.9.6), he gets: >>> print b [2, 3] >>> print b[0] 2 This matters because we use where() to get indices to loop over. On the mac you can then loop through the b array: for item in b: if b > 5: print 'hi' But this approach fails on a pc because: if b > 5: >>> ValueError: The truth value of an array with more than one element >>> is ambiguous. Use a.any() or a.all() I'd like to avoid having to do: if mac: b = numpy.where( a > 5) if pc: b = numpy.where( a > 5)[0] Has anybody else notice dealt with this? Is this a mac/pc difference or a numpy 1.01 vs. numpy 0.9.6 difference? Thanks for the help, James From robert.kern at gmail.com Wed Jan 23 18:40:12 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Jan 2008 17:40:12 -0600 Subject: [Numpy-discussion] where() on mac/pc return different things In-Reply-To: References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> <200801231815.26464.pgmdevlist@gmail.com> Message-ID: <4797D05C.10203@gmail.com> James Battat wrote: > Hi, > > numpy.where() returns different things on my windowsXP machine than on my > collaborator's mac osx machine. > > For example, consider: > >>> import numpy > >>> a = numpy.array([1,2,3,4]) > >>> b = numpy.where( a > 2 ) > > On WindowsXP (python 2.4.2, numpy 1.0.1), I get: > >>> print b > (array([2, 3]), ) > >>> print b[0] > [2 3] > > While in OSX (python 2.4.3, numpy 0.9.6), he gets: > >>> print b > [2, 3] > >>> print b[0] > 2 > > This matters because we use where() to get indices to loop over. On the > mac you can then loop through the b array: > > for item in b: > if b > 5: > print 'hi' > > But this approach fails on a pc because: > if b > 5: > >>> ValueError: The truth value of an array with more than one element > >>> is ambiguous. Use a.any() or a.all() > > > I'd like to avoid having to do: > if mac: > b = numpy.where( a > 5) > if pc: > b = numpy.where( a > 5)[0] > > Has anybody else notice dealt with this? Is this a mac/pc difference or a > numpy 1.01 vs. numpy 0.9.6 difference? The latter. Before the 1.0 release, we tried to make it clear that we were still experimenting with the APIs and that they might change up until 1.0 got released. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jbattat at cfa.harvard.edu Wed Jan 23 18:54:37 2008 From: jbattat at cfa.harvard.edu (James Battat) Date: Wed, 23 Jan 2008 18:54:37 -0500 (EST) Subject: [Numpy-discussion] where() on mac/pc return different things In-Reply-To: <4797D05C.10203@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> <200801231815.26464.pgmdevlist@gmail.com> <4797D05C.10203@gmail.com> Message-ID: Robert, Thanks for solving that puzzle! I'll get our group on the same 1.0.x numpy release. Take care, James ************************** Harvard University Dept. of Astronomy 60 Garden Street MS-10 Cambridge, MA 02138 phone 617.496.5988 lab 617.495.3267 email jbattat at cfa.harvard.edu web http://www.cfa.harvard.edu/~jbattat ************************** On Wed, 23 Jan 2008, Robert Kern wrote: > James Battat wrote: > > Hi, > > > > numpy.where() returns different things on my windowsXP machine than on my > > collaborator's mac osx machine. > > > > For example, consider: > > >>> import numpy > > >>> a = numpy.array([1,2,3,4]) > > >>> b = numpy.where( a > 2 ) > > > > On WindowsXP (python 2.4.2, numpy 1.0.1), I get: > > >>> print b > > (array([2, 3]), ) > > >>> print b[0] > > [2 3] > > > > While in OSX (python 2.4.3, numpy 0.9.6), he gets: > > >>> print b > > [2, 3] > > >>> print b[0] > > 2 > > > > This matters because we use where() to get indices to loop over. On the > > mac you can then loop through the b array: > > > > for item in b: > > if b > 5: > > print 'hi' > > > > But this approach fails on a pc because: > > if b > 5: > > >>> ValueError: The truth value of an array with more than one element > > >>> is ambiguous. Use a.any() or a.all() > > > > > > I'd like to avoid having to do: > > if mac: > > b = numpy.where( a > 5) > > if pc: > > b = numpy.where( a > 5)[0] > > > > Has anybody else notice dealt with this? Is this a mac/pc difference or a > > numpy 1.01 vs. numpy 0.9.6 difference? > > The latter. Before the 1.0 release, we tried to make it clear that we were still > experimenting with the APIs and that they might change up until 1.0 got released. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Wed Jan 23 20:45:26 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Jan 2008 20:45:26 -0500 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <20080123221507.GB7352@mentat.za.net> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> Message-ID: <200801232045.27432.pgmdevlist@gmail.com> All, I just committed a fix on the SVN. Now, the axis keyword should be recognized. Sorry for the delay. Pierre From oliphant at enthought.com Thu Jan 24 00:31:02 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 23 Jan 2008 23:31:02 -0600 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <200801231815.26464.pgmdevlist@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> <200801231815.26464.pgmdevlist@gmail.com> Message-ID: <47982296.5070406@enthought.com> Pierre GM wrote: > Oh, a last comment question: > Is the current documentation standard definitive ? Yes. -Travis From stefan at sun.ac.za Thu Jan 24 04:02:52 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 11:02:52 +0200 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <200801232045.27432.pgmdevlist@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <20080123221507.GB7352@mentat.za.net> <200801232045.27432.pgmdevlist@gmail.com> Message-ID: <20080124090252.GA16470@mentat.za.net> Hi Pierre On Wed, Jan 23, 2008 at 08:45:26PM -0500, Pierre GM wrote: > All, > I just committed a fix on the SVN. Now, the axis keyword should be recognized. > Sorry for the delay. I'm not 100% sure about the new behaviour -- compress now removes masked elements, instead of ignoring them. Whereas a person would have been able to do compress(x,condition).compressed() before, the mask information is now thrown away. The numpy docstring states that compress should be equivalent to a[condition], which is no longer the case. Regards St?fan From stefan at sun.ac.za Thu Jan 24 04:12:58 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 11:12:58 +0200 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: References: <20080123135546.GK25779@mentat.za.net> Message-ID: <20080124091258.GB16470@mentat.za.net> Hi Charles On Wed, Jan 23, 2008 at 09:34:44AM -0700, Charles R Harris wrote: > 2. How do we specify default values? > > I like to put them first in the list: {-1, integer} When exactly is this list used? That should be made clear in the standard as well. > 3. Why do we need the "optional" keyword (the function signature > already indicates that the parameter is optional). > > It's extra information, true, but that isn't always a bad thing. It's like > looking up "whole numbers" in a book index and, instead of the page reference, > the index directs you to "numbers, whole". Flip, flip, flip. Drives me crazy. > Besides, the function signature might be missing. When would the function signature be missing? In C functions we copy the signature into the docstring. I am concerned about duplicating information that may change. > 4. Do we really need the "Other Parameters" list? It would make more > sense to split positional and keyword arguments, but I'm not even > sure that is necessary, since that information is already specified in > the > function signature. > > I agree, but Travis likes this section and I don't feel strongly > about it. If I understood its role properly, I would be happier to use it, but at the moment I'm not convinced that it contributes anything useful. Parameters are normally sorted from most to least valuable anyway. > 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a > limited number of values? In Python {} is a dictionary, so why not > use ('hi','ho') instead? > > Either would be fine. IIRC, the {} was inherited from epydoc consolidated > fields. I thought that, since we threw all epydoc guidelines out the window, this would be a good time talk about it -- the next docday is just around the corner! Thank you very much for your response. Regards St?fan From stefan at sun.ac.za Thu Jan 24 04:16:16 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 11:16:16 +0200 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> Message-ID: <20080124091616.GC16470@mentat.za.net> Hi David On Thu, Jan 24, 2008 at 12:34:56AM +0900, David Cournapeau wrote: > I've just released the 0.3.0 release of numscons, an alternative > build system for numpy. The tarballs are available on launchpad. > > https://launchpad.net/numpy.scons.support/0.3/0.3.0 > > To use it, you need to get the build_with_scons numpy branch: see > http://projects.scipy.org/scipy/numpy/wiki/NumpyScons for more > details. Fantastic job, thank you for investing so much time in improving the build system. I noticed that you were trying to get scipy building -- how is that progressing? Scons should make building on non-*nix platforms a lot easier. Regards St?fan From pgmdevlist at gmail.com Thu Jan 24 04:58:04 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 24 Jan 2008 04:58:04 -0500 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <20080124090252.GA16470@mentat.za.net> References: <4797AEFF.3080509@hawaii.edu> <200801232045.27432.pgmdevlist@gmail.com> <20080124090252.GA16470@mentat.za.net> Message-ID: <200801240458.05771.pgmdevlist@gmail.com> On Thursday 24 January 2008 04:02:52 Stefan van der Walt wrote: > I'm not 100% sure about the new behaviour -- compress now removes > masked elements, instead of ignoring them. Whereas a person would > have been able to do > compress(x,condition).compressed() > before, the mask information is now thrown away. Mmh, OK, I see. The problem arises when the condition is masked. Filling w/ False will get rid of the masked values, filling with True won't work either (cf example below). An option is then to simply take the condition as a ndarray. >>>a=masked_array([1,2,3,4,5],mask=[0,0,1,1,0]) >>>a.compress(a<4) masked_array(data = [1 2], mask = [False False], fill_value=999999) >>>a.compress((a<4).filled(True)) masked_array(data = [1 2 -- --], mask = [False False True True], fill_value=999999) >>>a.compress((a<4).view(ndarray)) masked_array(data = [1 2 --], mask = [False False True], fill_value=999999) >>>a[a<4] masked_array(data = [1 2 --], mask = [False False True], fill_value=999999) > The numpy docstring states that compress should be equivalent to > a[condition], which is no longer the case. Good point... In most cases, as long as an axis is not specified. Note that the equivalence a.compress(condition) and a[condition] is not strictly true even for regular ndarrays: take a look at the example on the scipy site (http://www.scipy.org/Numpy_Example_List_With_Doc) >>>b = array([[10,20,30],[40,50,60]]) >>>b.compress(b.ravel() >= 22) array([30, 40, 50, 60]) >>>b[b.ravel()>=22] IndexError: index (2) out of range (0<=index<=1) in dimension 0 Anyhow, I just commited an update fixing our initial problem (viz, forcing condition to a regular ndarray). From matthew.brett at gmail.com Thu Jan 24 05:00:09 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 24 Jan 2008 10:00:09 +0000 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <20080124091258.GB16470@mentat.za.net> References: <20080123135546.GK25779@mentat.za.net> <20080124091258.GB16470@mentat.za.net> Message-ID: <1e2af89e0801240200ve035fa7le50b7c08f5c16522@mail.gmail.com> Hi, > When would the function signature be missing? In C functions we copy > the signature into the docstring. I am concerned about duplicating > information that may change. I guess that would be def my_func(*args, **kwargs): although, if you're going to document the parameters in detail for such a function, it seems sensible to specify what the args are in the signature. > I thought that, since we threw all epydoc guidelines out the window, > this would be a good time talk about it -- the next docday is just > around the corner! Perhaps we could have that discussion on the IRC channel on Friday at some specified time? Matthew From matthew.brett at gmail.com Thu Jan 24 05:04:56 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 24 Jan 2008 10:04:56 +0000 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <1e2af89e0801240200ve035fa7le50b7c08f5c16522@mail.gmail.com> References: <20080123135546.GK25779@mentat.za.net> <20080124091258.GB16470@mentat.za.net> <1e2af89e0801240200ve035fa7le50b7c08f5c16522@mail.gmail.com> Message-ID: <1e2af89e0801240204i4f6bebebgabfd63fbecae4150@mail.gmail.com> Also, > Perhaps we could have that discussion on the IRC channel on Friday at > some specified time? I suppose it's possible to hack up some processor of the form: doc_me('my_file.py') that can introspect things like argument lists, add boilerplate and process the resulting text according to the given rules. And maybe switch between standards if we change. Matthew From sdb at cloud9.net Thu Jan 24 11:26:02 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Thu, 24 Jan 2008 11:26:02 -0500 (EST) Subject: [Numpy-discussion] numpy.concatenate doesn't check axis for 1D case. Bug or feature? Message-ID: As the subject says, numpy.concatenate doesn't seem to obey -- or even check -- the axis flag when concatenating 1D arrays: ----------------------- ----------------- In [30]: A = numpy.array([1, 2, 3, 4]) In [31]: D = numpy.array([6, 7, 8, 9]) In [32]: numpy.concatenate((A, D)) Out[32]: array([1, 2, 3, 4, 6, 7, 8, 9]) In [33]: numpy.concatenate((A, D), axis=0) Out[33]: array([1, 2, 3, 4, 6, 7, 8, 9]) In [34]: numpy.concatenate((A, D), axis=1) Out[34]: array([1, 2, 3, 4, 6, 7, 8, 9]) In [35]: numpy.concatenate((A, D), axis=2) Out[35]: array([1, 2, 3, 4, 6, 7, 8, 9]) ----------------------- ----------------- However, if you create the same arrays as 2D (1xn) arrays, then numpy checks, and does the right thing: ---------------------- ------------------- In [36]: A = numpy.array([[1, 2, 3, 4]]) In [37]: D = numpy.array([[6, 7, 8, 9]]) In [38]: A.shape Out[38]: (1, 4) In [39]: numpy.concatenate((A, D)) Out[39]: array([[1, 2, 3, 4], [6, 7, 8, 9]]) In [40]: numpy.concatenate((A, D), axis=0) Out[40]: array([[1, 2, 3, 4], [6, 7, 8, 9]]) In [41]: numpy.concatenate((A, D), axis=1) Out[41]: array([[1, 2, 3, 4, 6, 7, 8, 9]]) In [42]: numpy.concatenate((A, D), axis=2) --------------------------------------------------------------------------- Traceback (most recent call last) /fs/home/sdb/ in () : bad axis1 argument to swapaxes ----------------------- ----------------- Question: Is is a bug or a feature? I'd at least think that numpy would check the axis arg in the 1D case, and issue an error if the user tried to do something impossible (e.g. axis=2). Whether numpy would promote a 1D to a 2D array is a different question, and I am agnostic about that one. Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ From robert.kern at gmail.com Thu Jan 24 14:15:13 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Jan 2008 13:15:13 -0600 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <20080124091258.GB16470@mentat.za.net> References: <20080123135546.GK25779@mentat.za.net> <20080124091258.GB16470@mentat.za.net> Message-ID: <4798E3C1.3010309@gmail.com> Stefan van der Walt wrote: > Hi Charles > > On Wed, Jan 23, 2008 at 09:34:44AM -0700, Charles R Harris wrote: >> 2. How do we specify default values? >> >> I like to put them first in the list: {-1, integer} > > When exactly is this list used? That should be made clear in the > standard as well. No special processing is done. I omit it. If I need to talk about the default value, I do it in prose in the argument's description. Note that the default value is not always the same thing as the value in the signature, particularly when kwd=None. But in those cases, I usually need a sentence to describe what's happening, so that's where I always put that information. >> 3. Why do we need the "optional" keyword (the function signature >> already indicates that the parameter is optional). >> >> It's extra information, true, but that isn't always a bad thing. It's like >> looking up "whole numbers" in a book index and, instead of the page reference, >> the index directs you to "numbers, whole". Flip, flip, flip. Drives me crazy. >> Besides, the function signature might be missing. > > When would the function signature be missing? In C functions we copy > the signature into the docstring. I am concerned about duplicating > information that may change. > >> 4. Do we really need the "Other Parameters" list? It would make more >> sense to split positional and keyword arguments, but I'm not even >> sure that is necessary, since that information is already specified in >> the >> function signature. >> >> I agree, but Travis likes this section and I don't feel strongly >> about it. > > If I understood its role properly, I would be happier to use it, but > at the moment I'm not convinced that it contributes anything useful. > Parameters are normally sorted from most to least valuable anyway. Look at the scipy.optimize.fmin*() functions, for instance. If you don't feel like using it, don't. None of the numpy function signatures should be long enough to need it. >> 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a >> limited number of values? In Python {} is a dictionary, so why not >> use ('hi','ho') instead? >> >> Either would be fine. IIRC, the {} was inherited from epydoc consolidated >> fields. > > I thought that, since we threw all epydoc guidelines out the window, > this would be a good time talk about it -- the next docday is just > around the corner! Personally, I don't think it's worth standardizing. If it's readable and valid reST, just do it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bolme1234 at comcast.net Thu Jan 24 15:18:45 2008 From: bolme1234 at comcast.net (David Bolme) Date: Thu, 24 Jan 2008 13:18:45 -0700 Subject: [Numpy-discussion] segfault problem with numpy and pickle Message-ID: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> A am having some trouble when pickling numpy arrays. Basically I use one python script to create the array and then pickle it. When I load the pickled array using a different python script it appears to load fine. When I try to perform a matrix multiply on the array with a vector (using dot) python crashes with a segmentation violation. The array contains 64bit floating point numbers and has a shape of (47, 16384) and the vector has a shape of (16384,1). Other shapes I have tested also have this problem. A workaround is to create a new array that is initialized with the data from the first. For example: a = array(pickle.loads(buffer)) After calling this command the script runs fine. Here is some information on my system setup: OS: Macos X Leopard 64bit Intel numpy.version.version = 1.0.5.dev4624 Below is a simple script that reproduces this error on my system. I would like to get this problem fixed so that I don't have to use this work around every time I load a pickled array. Is this a known bug? I could not find it in the bug tracker. Thanks in advance for the help. Dave import pickle from numpy import array,dot,ones,random # Create the arrays a = random.normal(size=(47,128*128)) b = ones((128*128,2)) c = ones((128*128,1)) # Test dot print "Computing dot product on original arrays..." print dot(a,b).shape # WORKS print dot(a,c).shape # WORKS print "Pickle and unpickel the array with workaround." buffer = pickle.dumps(a) a = array(pickle.loads(buffer)) # Test dot print "Computing dot product with workaround..." print dot(a,b).shape # WORKS print dot(a,c).shape # WORKS print "Pickle and unpickel the array without workaround." a = pickle.loads(buffer) # Test dot print "Computing dot product without workaround..." print dot(a,b).shape # WORKS print dot(a,c).shape # CRASH -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Jan 24 15:53:11 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 22:53:11 +0200 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <4798E3C1.3010309@gmail.com> References: <20080123135546.GK25779@mentat.za.net> <20080124091258.GB16470@mentat.za.net> <4798E3C1.3010309@gmail.com> Message-ID: <20080124205311.GE27699@mentat.za.net> Hi Robert On Thu, Jan 24, 2008 at 01:15:13PM -0600, Robert Kern wrote: > >> 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a > >> limited number of values? In Python {} is a dictionary, so why not > >> use ('hi','ho') instead? > >> > >> Either would be fine. IIRC, the {} was inherited from epydoc consolidated > >> fields. > > > > I thought that, since we threw all epydoc guidelines out the window, > > this would be a good time talk about it -- the next docday is just > > around the corner! > > Personally, I don't think it's worth standardizing. If it's readable > and valid reST, just do it. The attached output shows the difference between using {} and () -- it doesn't seem to make a difference. Since [] or () is normally used to indicate a set, I found it more natural. Either way, I think I'll follow your other suggestion and not use such a list -- rather specify the default value in the prose. Thanks St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Jan 24 15:58:14 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 24 Jan 2008 22:58:14 +0200 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <200801240458.05771.pgmdevlist@gmail.com> References: <4797AEFF.3080509@hawaii.edu> <200801232045.27432.pgmdevlist@gmail.com> <20080124090252.GA16470@mentat.za.net> <200801240458.05771.pgmdevlist@gmail.com> Message-ID: <20080124205814.GF27699@mentat.za.net> Hi Pierre On Thu, Jan 24, 2008 at 04:58:04AM -0500, Pierre GM wrote: > On Thursday 24 January 2008 04:02:52 Stefan van der Walt wrote: > > I'm not 100% sure about the new behaviour -- compress now removes > > masked elements, instead of ignoring them. Whereas a person would > > have been able to do > > compress(x,condition).compressed() > > before, the mask information is now thrown away. > > Mmh, OK, I see. The problem arises when the condition is masked. Filling w/ > False will get rid of the masked values, filling with True won't work either > (cf example below). An option is then to simply take the condition as a > ndarray. How about masking the output where the condition is masked? I.e. keep where condition is True, remove where condition is False and mask where condition is masked. > > The numpy docstring states that compress should be equivalent to > > a[condition], which is no longer the case. > Good point... In most cases, as long as an axis is not specified. Note that > the equivalence a.compress(condition) and a[condition] is not strictly true > even for regular ndarrays: take a look at the example on the scipy site > (http://www.scipy.org/Numpy_Example_List_With_Doc) > > >>>b = array([[10,20,30],[40,50,60]]) > >>>b.compress(b.ravel() >= 22) > array([30, 40, 50, 60]) > >>>b[b.ravel()>=22] > IndexError: index (2) out of range (0<=index<=1) in dimension 0 You're right, better equivalent code would be a.flat[condition] > Anyhow, I just commited an update fixing our initial problem (viz, forcing > condition to a regular ndarray). Thanks! I'll check it out. Regards St?fan From pgmdevlist at gmail.com Thu Jan 24 16:37:42 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 24 Jan 2008 16:37:42 -0500 Subject: [Numpy-discussion] numpy.ma.compress In-Reply-To: <20080124205814.GF27699@mentat.za.net> References: <4797AEFF.3080509@hawaii.edu> <200801240458.05771.pgmdevlist@gmail.com> <20080124205814.GF27699@mentat.za.net> Message-ID: <200801241637.43087.pgmdevlist@gmail.com> On Thursday 24 January 2008 15:58:14 Stefan van der Walt wrote: > How about masking the output where the condition is masked? > I.e. keep where condition is True, remove where condition is False and > mask where condition is masked. Won't systematically do, check one of the examples I gave in a previous emails. >>>a=masked_array([1,2,3,4,5],mask=[0,0,1,1,0]) >>>a.compress((a<4).filled(True)) masked_array(data = [1 2 -- --], ? ? ? mask = [False False ?True ?True], ? ? ? fill_value=999999) Fundamentally, it's a very bad idea to use a masked array as a condition: what should be done when the condition is masked ? Here, we're going against one of the basic principles of Paul Dubois, the original author: we are making assumptions about the masked values of the condition when we consider the condition as a ndarray. In timeseries, we prevent the use of masked arrays as conditions. That's not much of a problem, as we already overloading __getitem__ to a rather ugly extent. I'm a bit reluctant to introduce that feature in numpy.ma, as it would crash the performance. That's something we may want to consider when we'll port numpy.ma to C: a masked condition is undefined, therefore cannot be used in selecting array elements, therefore should be set to False. From filip.wasilewski at gmail.com Thu Jan 24 17:07:55 2008 From: filip.wasilewski at gmail.com (Filip Wasilewski) Date: Thu, 24 Jan 2008 23:07:55 +0100 Subject: [Numpy-discussion] Numpy Example List with Doc updates Message-ID: Hi all, Just a small note. I've updated the `Numpy Example List With Doc`[1] that is linked from the main doc index. Because I'm not always in the loop and the page likes to get out of sync with the base `Numpy Example List`[2] and NumPy, I've posted some instructions[3] that you might find useful when trying to bring it up-to-date :) [1] http://scipy.org/Numpy_Example_List_With_Doc [2] http://scipy.org/Numpy_Example_List [3] http://scipy.org/Numpy_Example_List_With_Doc/script cheers, f -- http://filipwasilewski.pl From charlesr.harris at gmail.com Thu Jan 24 19:22:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Jan 2008 17:22:04 -0700 Subject: [Numpy-discussion] Docstring standard: how to specify variable types In-Reply-To: <20080124205311.GE27699@mentat.za.net> References: <20080123135546.GK25779@mentat.za.net> <20080124091258.GB16470@mentat.za.net> <4798E3C1.3010309@gmail.com> <20080124205311.GE27699@mentat.za.net> Message-ID: On Jan 24, 2008 1:53 PM, Stefan van der Walt wrote: > Hi Robert > > On Thu, Jan 24, 2008 at 01:15:13PM -0600, Robert Kern wrote: > > >> 5. Is the {'hi', 'ho'} syntax used when a parameter can only > assume a > > >> limited number of values? In Python {} is a dictionary, so why > not > > >> use ('hi','ho') instead? > > >> > > >> Either would be fine. IIRC, the {} was inherited from epydoc > consolidated > > >> fields. > > > > > > I thought that, since we threw all epydoc guidelines out the window, > > > this would be a good time talk about it -- the next docday is just > > > around the corner! > > > > Personally, I don't think it's worth standardizing. If it's readable > > and valid reST, just do it. > > The attached output shows the difference between using {} and () -- it > doesn't seem to make a difference. Since [] or () is normally used to > indicate a set, I found it more natural. > I think it best for everyone to settle on one or the other, both for consistency of appearance and later convenience if someone wants to parse the documentation. Using neither {} nor () looks would also be reasonable, but why don't we just go with the current standard? It was a bit of a surprise to me, too. But *having* a working standard is more important than the details. In the end, these things are just useful conventions. Sets are normally defined using {} these days and set union and difference are no longer '+' and '-'. There are old books where Euler angles are defined using left handed coordinate systems. The Russian literature used '+' instead of '-' as the sign in the exponential of the forward Fourier transform. Strangest of all, in tensor analysis the meanings of covariant and contravariant are interchanged. So all of those things could have gone differently. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Jan 24 23:03:10 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 13:03:10 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> Message-ID: <47995F7E.6070509@ar.media.kyoto-u.ac.jp> Matthew Brett wrote: > Hi David, > > Thanks again for doing this. I've been playing with the compilation - > should I email you direct with questions, to the lists? I don't think it would be inappropriate to use the current lists for that. > > Basically, I'm trying to understand the library discovery, linking > steps - ATLAS in particular. Don't trust the included doc: it is not up-to-date, and that's the part which I totally redesigned since I wrote the initial doc. For things like FFT, BLAS, LAPACK, things are a bit complicated. Fortunately, for "normal" libraries, things are much easier: you use the NumpyCheckLibraryAndHeader function http://projects.scipy.org/scipy/numpy/wiki/NumpySconsExtExamples#Checkingforalibrary This function can check for a library, headers, and check for symbol in the library. site.cfg customization is also automatically taken care of. I also intend to add the possibility to check for libraries which use pkgconfig. For BLAS/LAPACK/FFT, the basic principles are: - I make a distinction between "API libraries" (called meta libraries in numscons) and "implementation libraries" (called perflib in numscons). - Meta libraries uses perflibs: FFT/BLAS/LAPACK may use Sunperf/MKL/ATLAS. - For each meta library and each perflib, a Check function is defined: CheckMKL, CheckCBLAS, etc... - For a perflib check to succeed, I try to compile, link and run code snippets. Any failure means the Check* function will return failure. - When a perflib Check* succeeds, the meta library check checks that another code snippet can be compiled/linked/run. For example, if CheckMKL succeeds in CheckCBLAS, I will compile/link/run a cblas program using MKL configuration. - Both perflib and meta libraries are customizable through site.cfg (I tried to keep backward compatibility at this level with numpy.distutils). I still have the feeling that it is over-designed, but BLAS/LAPACK libraries are really a PITA. Every perflib has slightly different conventions, and I wanted to use common code for all of them as much as possible [1]. Also, the configuration is not inside python code anymore: libraries, frameworks, link flags are all defined in perflib.cfg. cheers, David [1] This design also allow some funky stuff: for example, when using sun performance libraries with sun tools, because of something which look like a bug to me, the sunperf are not linked when building a shared library. So I link a code snippet, and parse the linker output to get the necessary libraries; this is totally transparent to BLAS/LAPACK checkers. From david at ar.media.kyoto-u.ac.jp Thu Jan 24 23:45:30 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 13:45:30 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <20080124091616.GC16470@mentat.za.net> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <20080124091616.GC16470@mentat.za.net> Message-ID: <4799696A.6010102@ar.media.kyoto-u.ac.jp> Stefan van der Walt wrote: > Hi David > > On Thu, Jan 24, 2008 at 12:34:56AM +0900, David Cournapeau wrote: >> I've just released the 0.3.0 release of numscons, an alternative >> build system for numpy. The tarballs are available on launchpad. >> >> https://launchpad.net/numpy.scons.support/0.3/0.3.0 >> >> To use it, you need to get the build_with_scons numpy branch: see >> http://projects.scipy.org/scipy/numpy/wiki/NumpyScons for more >> details. > > Fantastic job, thank you for investing so much time in improving the > build system. I noticed that you were trying to get scipy building -- > how is that progressing? Not much, but there is not much left to do: I don't remember exactly, but umfpack was still problematic; but except that, scipy should be able to build with numscons once I change the imports (to cope with the split between scons support and numpy.distutils). Scipy is much easier to build than numpy, because except f2py/fortran related problems, everything needed in scipy is needed by numpy. And the really hard job was to understand distutils anyway :) > Scons should make building on non-*nix > platforms a lot easier. Not so much, because scons does not know how to build python extensions (I had to develop my own builder, which fortunately should end up in scons at some point). Since I started this work, my perception on the advantages have changed: - real dependencies handling. I did not think it would be so interesting, but while using it, I got used to not removing the build directory when I change one configuration. - ability to use different compiler options for different extensions. - easy customization of compiler flags (interesting for debugging, or find the best optimization flags). - checks are logged in config.log files - and of course, the ability to build shared library (c-types extensions), static libraries, etc... which is one of the reason I started this. There are also other points which are less obvious, or for which I am too biased: - the system is simpler: the codebase is 3 times smaller (2500 vs 8000 lines) and there is a clear distinction between configuration and code (flags are not buried in the code anymore, they are defined in separate .cfg files). Also, even if scons is 'unpythonic' for some aspects, the codebase is so much better than distutils in the stdlib. Everytime I touch stdlib distutils, I fear that it will break something in another unrelated place, or that I did not call some functions in the right, platform specific order; I don't have this problem with scons. - the SConscripts also are simpler to read than setup.py I think (specially for numpy.core and other modules with complicated configurations). They are longer, but there is less magic, and the intent is more obvious I think. This is really important, because I think I spent more than 50 % of my time on understanding distutils for this. I hope that nobody will have to do that anymore. cheers, David From stefan at sun.ac.za Fri Jan 25 02:17:41 2008 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 25 Jan 2008 09:17:41 +0200 Subject: [Numpy-discussion] segfault problem with numpy and pickle In-Reply-To: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> References: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> Message-ID: <20080125071741.GB9574@mentat.za.net> Hi David On Thu, Jan 24, 2008 at 01:18:45PM -0700, David Bolme wrote: > A am having some trouble when pickling numpy arrays. Basically I use one > python script to create the array and then pickle it. When I load the pickled > array using a different python script it appears to load fine. When I try to > perform a matrix multiply on the array with a vector (using dot) python crashes > with a segmentation violation. Please see if http://projects.scipy.org/scipy/numpy/ticket/551 matches your problem, and add your comments there. Regards St?fan From haase at msg.ucsf.edu Fri Jan 25 02:55:05 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 25 Jan 2008 08:55:05 +0100 Subject: [Numpy-discussion] Numpy Example List with Doc updates In-Reply-To: References: Message-ID: On Jan 24, 2008 11:07 PM, Filip Wasilewski wrote: > Hi all, > > Just a small note. I've updated the `Numpy Example List With Doc`[1] > that is linked from the main doc index. Because I'm not always in the > loop and the page likes to get out of sync with the base `Numpy > Example List`[2] and NumPy, I've posted some instructions[3] that you > might find useful when trying to bring it up-to-date :) > > [1] http://scipy.org/Numpy_Example_List_With_Doc > [2] http://scipy.org/Numpy_Example_List > [3] http://scipy.org/Numpy_Example_List_With_Doc/script > > Hi, a general note: I like those two pages a lot. Recently my problem has been however, that they have grown so LARGE that, I could not open them anymore in a reasonable time.... I suggest: They would have to get split up - somehow .... Maybe a script like yours might be the solution ! One could keep the "complete" document page. And -- instead of splitting each function into a separate page -- it would be great to sort by "category". (like creation, comparision, scalar, distribution-functions,...) My most common usage of these pages is when I don't know what the right function is, and I'm just browsing through to learn new things. I understand that ironically the complete-page might be best for this, but if cannot get it opened entirely, it doesn't help either. Just some food for thought, Sebsstian Haase From dmitrey.kroshko at scipy.org Fri Jan 25 03:08:42 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 25 Jan 2008 10:08:42 +0200 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799696A.6010102@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <20080124091616.GC16470@mentat.za.net> <4799696A.6010102@ar.media.kyoto-u.ac.jp> Message-ID: <4799990A.2030202@scipy.org> Hi all, I don't know much about what are these scons are, if it's something essential (as it seems to be from amount of mailing list traffic) why can't it be just merged to numpy, w/o making any additional branches? Regards, D. David Cournapeau wrote: > > From robert.kern at gmail.com Fri Jan 25 03:14:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 25 Jan 2008 02:14:41 -0600 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799990A.2030202@scipy.org> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <20080124091616.GC16470@mentat.za.net> <4799696A.6010102@ar.media.kyoto-u.ac.jp> <4799990A.2030202@scipy.org> Message-ID: <47999A71.1060405@gmail.com> dmitrey wrote: > Hi all, > I don't know much about what are these scons are, if it's something > essential (as it seems to be from amount of mailing list traffic) why > can't it be just merged to numpy, w/o making any additional branches? It's a very large, still experimental change to the entire build infrastructure. It requires branches for us to evaluate. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Fri Jan 25 03:17:31 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 25 Jan 2008 09:17:31 +0100 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799990A.2030202@scipy.org> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <20080124091616.GC16470@mentat.za.net> <4799696A.6010102@ar.media.kyoto-u.ac.jp> <4799990A.2030202@scipy.org> Message-ID: 2008/1/25, dmitrey : > > Hi all, > I don't know much about what are these scons are, if it's something > essential (as it seems to be from amount of mailing list traffic) why > can't it be just merged to numpy, w/o making any additional branches? > Scons is a building system, like distutils but much more powerful. It is not simple to replace partially distutils in the numpy construction system. There are a lot of things to test during the build. So while David makes this happen, numpy can't be in an unstable state where the SVN head can't be compiled and used. This branch is thus needed and will be merged when both the trunk and the branch are stable. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Jan 25 03:20:50 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 17:20:50 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799990A.2030202@scipy.org> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <20080124091616.GC16470@mentat.za.net> <4799696A.6010102@ar.media.kyoto-u.ac.jp> <4799990A.2030202@scipy.org> Message-ID: <47999BE2.607@ar.media.kyoto-u.ac.jp> dmitrey wrote: > Hi all, > I don't know much about what are these scons are, if it's something > essential (as it seems to be from amount of mailing list traffic) why > can't it be just merged to numpy, w/o making any additional branches? scons is a build system, like make. The difference being that sconscripts (makefiles for scons) are written in python, and well supported on windows. The standard way to build python extension is to use distutils, but distutils itself is difficult to extend (the code is basically spaghetti, not maintained and not documented), and to be fair, numpy's needs go way beyond the usual python extension: we want to control optimization flags, we need fortran, we depend on a lot of external libraries we need to check, etc... numpy uses distutils extension (numpy.distutils), to support fortran and other goodies. But because it is based on distutils, it has inherited some of the distutils' problems; in particular, it is not possible to build dynamically loaded libraries (necessary for ctypes-based extensions), it is difficult to check for additional libraries, etc... So instead, I plug scons in distutils, so that all C code in numpy is built through sconscripts. But the build system arguably being one of the most crucial part of numpy, it is necessary to do changes incrementally in branches (the code changes are big: several thousand of python code, which is difficult to test by nature). cheers, David From cournape at gmail.com Fri Jan 25 04:22:26 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Jan 2008 18:22:26 +0900 Subject: [Numpy-discussion] How to build on Solaris 10 (x86) using sunperf? In-Reply-To: <779A7062-4B27-44B5-9B56-1E530B8FF23B@pixar.com> References: <779A7062-4B27-44B5-9B56-1E530B8FF23B@pixar.com> Message-ID: <5b8d13220801250122y43c25087t27af295b3dc72ca@mail.gmail.com> On Jan 24, 2008 3:06 AM, Peter Ward wrote: > > The problems I was having were due to a bad site.cfg initially and then > a problem with the python pkg from sunfreeware (ctypes version > mismatch). > > Numpy is now happily installed. > > If you need someone to test anything new, let me know. If you have a few minutes to spend on it, you could try numscons; I don't have access to solaris (I only have access to indiana, that is opensolaris on vmware, 32 bits only). For solaris, the advantages of numscons is native support sunperf, and optimization flags for compilation (only non-arch dependant for now). As I consider it still experimental, you should not use numpy built this way, though. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Jan 25 04:41:14 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 18:41:14 +0900 Subject: [Numpy-discussion] numscons, available as python egg Message-ID: <4799AEBA.3080103@ar.media.kyoto-u.ac.jp> Hi, Sorry for the flooding, but I finally managed to build an egg and put it on the web, so numscons is available as an egg, now. You should be able to install it using easy_install, e.g easy_install numscons should work. cheers, David From matthew.brett at gmail.com Fri Jan 25 04:52:16 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 25 Jan 2008 09:52:16 +0000 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <47995F7E.6070509@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> Message-ID: <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> Hi David, > > Basically, I'm trying to understand the library discovery, linking > > steps - ATLAS in particular. > Don't trust the included doc: it is not up-to-date, and that's the part > which I totally redesigned since I wrote the initial doc. > - For a perflib check to succeed, I try to compile, link and run > code snippets. Any failure means the Check* function will return failure. > - When a perflib Check* succeeds, the meta library check checks that > another code snippet can be compiled/linked/run. Thanks - and I agree, for all the reasons you've given in later emails that this seems like the way to go. But, in the mean time, I've got problems with the perflib stuff. Specifically, the scons build is not picking up the atlas library for blas and lapack that works for the standard build. I'm on a 64 bit system, providing an extra layer of complexity. I've attached the build logs. I noticed that, for atlas, you check for atlas_enum.c - but do you in fact need this for the build? numpy.distutils seemed to be satisfied with cblas.h and clapack.h in /usr/local/include/atlas. It's no big deal to copy it from sources, but was there a reason you chose that file? The test for linking to blas and lapack from atlas fails too - is this a false positive? For both numpy.distutils and numpyscons, default locations of libraries do not include the lib64 libraries like /usr/local/lib64 that us 64 bit people use. Is that easy to fix? Thanks a lot, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: npsons_result.tgz Type: application/x-gzip Size: 8707 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Jan 25 05:21:33 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 19:21:33 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> Message-ID: <4799B82D.6080600@ar.media.kyoto-u.ac.jp> Matthew Brett wrote: > Hi David, > >>> Basically, I'm trying to understand the library discovery, linking >>> steps - ATLAS in particular. >> Don't trust the included doc: it is not up-to-date, and that's the part >> which I totally redesigned since I wrote the initial doc. > >> - For a perflib check to succeed, I try to compile, link and run >> code snippets. Any failure means the Check* function will return failure. >> - When a perflib Check* succeeds, the meta library check checks that >> another code snippet can be compiled/linked/run. > > Thanks - and I agree, for all the reasons you've given in later emails > that this seems like the way to go. > > But, in the mean time, I've got problems with the perflib stuff. > Specifically, the scons build is not picking up the atlas library for > blas and lapack that works for the standard build. I'm on a 64 bit > system, providing an extra layer of complexity. I have never tested the thing on 64 bits (I don't have easy access to 64 bits machine, which should change soon, hopefully), so it is not surprising it does not work now. I have taken this into account for the design, though (see below). > > I've attached the build logs. I noticed that, for atlas, you check > for atlas_enum.c - but do you in fact need this for the build? Now. I just wanted one header specific to atlas. It looks like not all version of ATLAS install this one, unfortunately (3.8, for example). > numpy.distutils seemed to be satisfied with cblas.h and clapack.h in > /usr/local/include/atlas. It's no big deal to copy it from sources, > but was there a reason you chose that file? No reason, but it cannot be cblas.h (it has to be atlas specific; otherwise, it does not make sense). The list of headers to check can be empty, though. > > The test for linking to blas and lapack from atlas fails too - is this > a false positive? Hmm, if atlas check does not work, it sounds like a right negative to me :) If ATLAS is not detected correctly, it won't be used by blas/lapack checkers. Or do you mean something else ? > > For both numpy.distutils and numpyscons, default locations of > libraries do not include the lib64 libraries like /usr/local/lib64 > that us 64 bit people use. Is that easy to fix? Yes, it is easy, in the sense that nothing in the checkers code is harcoded: all the checks internally uses BuildConfig instances, which is like a dictionary with default values and a restricted set of keys (the keys are library path, libraries, headers, etc...). Those BuildConfig instances are created from a config file (perflib.cfg), and should always be customizable from site.cfg The options which can be customized can be found in the perflib.cfg file. For example, having: [atlas] htc = in your site.cfg should say to CheckATLAS to avoid looking for atlas_enum.h To make 64 bits work by default is a bit more complicated. I thought a bit about the problem: that's why the checkers do not use BuildConfig instances directly, but request them through a BuildConfigFactory. One problem is that I don't understand how 64 bits libraries work; more precisely, what is the convention of library path ? Is there a lib64 for each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the standard ones (/lib, /usr/lib) checked at all, after the 64 bits counterpart ? cheers, David From matthew.brett at gmail.com Fri Jan 25 06:03:36 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 25 Jan 2008 11:03:36 +0000 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799B82D.6080600@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> Message-ID: <1e2af89e0801250303y2377bf2dv58fd1e988f7f034d@mail.gmail.com> Hi, > > I've attached the build logs. I noticed that, for atlas, you check > > for atlas_enum.c - but do you in fact need this for the build? > Now. I just wanted one header specific to atlas. It looks like not all > version of ATLAS install this one, unfortunately (3.8, for example). > > numpy.distutils seemed to be satisfied with cblas.h and clapack.h in > > /usr/local/include/atlas. It's no big deal to copy it from sources, > > but was there a reason you chose that file? > No reason, but it cannot be cblas.h (it has to be atlas specific; > otherwise, it does not make sense). The list of headers to check can be > empty, though. I see. You want to be sure from the include check that you have actually discovered ATLAS rather than something else? I guess that numpy.distutils solves this by the directory naming conventions - so - if it finds cblas.h in /usr/local/include/atlas as opposed to somewhere else, then it assumes it has the ATLAS version? Are these files different for ATLAS than for other libraries? If not, do we need to check that they are the ATLAS headers rather than any other? > > The test for linking to blas and lapack from atlas fails too - is this > > a false positive? > Hmm, if atlas check does not work, it sounds like a right negative to me > :) If ATLAS is not detected correctly, it won't be used by blas/lapack > checkers. Or do you mean something else ? I mean false positive in the sense that it appears that numpy can build and pass tests with the ATLAS I have, so excluding it seems too stringent. The tests presumably should correspond to something the numpy code actually needs, rather than parts of ATLAS it can do without. > > For both numpy.distutils and numpyscons, default locations of > > libraries do not include the lib64 libraries like /usr/local/lib64 > > that us 64 bit people use. Is that easy to fix? > Yes, it is easy, in the sense that nothing in the checkers code is > harcoded: all the checks internally uses BuildConfig instances, which is > like a dictionary with default values and a restricted set of keys (the > keys are library path, libraries, headers, etc...). Those BuildConfig > instances are created from a config file (perflib.cfg), and should > always be customizable from site.cfg > > The options which can be customized can be found in the perflib.cfg > file. For example, having: > > [atlas] > htc = > > in your site.cfg should say to CheckATLAS to avoid looking for atlas_enum.h Thanks ... > To make 64 bits work by default is a bit more complicated. I thought a > bit about the problem: that's why the checkers do not use BuildConfig > instances directly, but request them through a BuildConfigFactory. One > problem is that I don't understand how 64 bits libraries work; more > precisely, what is the convention of library path ? Is there a lib64 for > each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the > standard ones (/lib, /usr/lib) checked at all, after the 64 bits > counterpart ? Well, the build works fine, it's just the perflib discovery - but I suppose that's what you meant. I think the convention is that 64 bit libraries do indeed go in /lib64, /usr/lib64, /usr/local/li64. My guess is that only 32-bit libraries should go in /lib, /usr/lib, but I don't think that convention is always followed. fftw, for example, installs 64 bit libraries in /usr/local/lib on my system. The compiler (at least gcc) rejects libraries that are in the wrong format, so I believe that finding 32 bit libraries will just cause a warning and continued search. Thanks again. Herculean work. Matthew From ndbecker2 at gmail.com Fri Jan 25 06:20:20 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 25 Jan 2008 06:20:20 -0500 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> Message-ID: Is numscons specific to numpy/scipy, or is it for building arbitrary python extensions (replacing distutils?). I'm hoping for the latter. From david at ar.media.kyoto-u.ac.jp Fri Jan 25 07:08:22 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 25 Jan 2008 21:08:22 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> Message-ID: <4799D136.80305@ar.media.kyoto-u.ac.jp> Neal Becker wrote: > Is numscons specific to numpy/scipy, or is it for building arbitrary python > extensions (replacing distutils?). I'm hoping for the latter. Actually, another way to answer your question: I am working on a patch such as the part of numscons which takes care of building python extensions in a cross platform way can be included in scons itself (I already have contribute quite a few patches/fix for scons since I started this, since I would rather not depend on in-house changes to scons). cheers, David From oliphant at enthought.com Fri Jan 25 10:39:46 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 25 Jan 2008 09:39:46 -0600 Subject: [Numpy-discussion] Doc-days are today! Come join on irc.freenode.net (#scipy) Message-ID: <479A02C2.2060606@enthought.com> From wfspotz at sandia.gov Fri Jan 25 14:36:33 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Fri, 25 Jan 2008 12:36:33 -0700 Subject: [Numpy-discussion] NPY_FORCECAST Message-ID: <56DF44D1-0F58-4E77-B8EB-61F108E4C21C@sandia.gov> Hi, I am currently using PyArray_FromObject() to convert an input argument to a 32-bit integer array. On a 64-bit architecture, it is easy to create an integer array whose default type is 64-bit. When this is sent to PyArray_FromObject(), it raises an error, saying "array cannot be safely cast to required type", even though all of its elements are representable by 32 bits. I see from the "Guide to NumPy" that I could call PyArray_FromAny() instead, providing the flag NPY_FORCECAST to force the cast to be made. However, this code is within numpy.i, will be used by others, and could lead to unexpected behavior if I were to make the change. How hard would it be to implement a new option (NPY_ATTEMPTCAST?) that attempts to make the cast, but raises an OverflowError if any of the source data is too large for the target array? (Or, alternatively, make this the default behavior if NPY_FORCECAST is false?) Thanks ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From kchristman54 at yahoo.com Fri Jan 25 15:33:04 2008 From: kchristman54 at yahoo.com (Kevin Christman) Date: Fri, 25 Jan 2008 12:33:04 -0800 (PST) Subject: [Numpy-discussion] extending a recarray Message-ID: <582906.78354.qm@web38701.mail.mud.yahoo.com> I am using recarray to store experiment information that is read from a file: self.records = numpy.rec.fromrecords(self.allData,formats='f4,S30,S30,f4,f4,f4,f4,f4,f4,f4,f4,f4,f4',names='MotorNum,MotorType,MotorName,MotorHP, NSync,NShaftFL,NShaft,pLoad,pLoadSlipMethod, pLoadORMEL,etaMeasured,etaORMEL,etaNameplate') Now I would like to add additional "columns" to this array for my calculations/analysis of the data. How is this done? Thanks, Kevin ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping From robert.kern at gmail.com Fri Jan 25 15:55:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 25 Jan 2008 14:55:34 -0600 Subject: [Numpy-discussion] extending a recarray In-Reply-To: <582906.78354.qm@web38701.mail.mud.yahoo.com> References: <582906.78354.qm@web38701.mail.mud.yahoo.com> Message-ID: <479A4CC6.5010800@gmail.com> Kevin Christman wrote: > I am using recarray to store experiment information that is read from a file: > > self.records = numpy.rec.fromrecords(self.allData,formats='f4,S30,S30,f4,f4,f4,f4,f4,f4,f4,f4,f4,f4',names='MotorNum,MotorType,MotorName,MotorHP, NSync,NShaftFL,NShaft,pLoad,pLoadSlipMethod, pLoadORMEL,etaMeasured,etaORMEL,etaNameplate') > > Now I would like to add additional "columns" to this array for my calculations/analysis of the data. How is this done? Thanks, You need to make a new dtype with the new columns. Then make an empty array with the new dtype. Copy in each column from the old array. Add the new column's data. For example: http://projects.scipy.org/pipermail/numpy-discussion/2007-September/029357.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Jan 25 19:04:00 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Jan 2008 17:04:00 -0700 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <4799B82D.6080600@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 25, 2008 3:21 AM, David Cournapeau wrote: > Matthew Brett wrote: > > Hi David, > > > >>> Basically, I'm trying to understand the library discovery, linking > >>> steps - ATLAS in particular. > >> Don't trust the included doc: it is not up-to-date, and that's the part > >> which I totally redesigned since I wrote the initial doc. > > > >> - For a perflib check to succeed, I try to compile, link and run > >> code snippets. Any failure means the Check* function will return > failure. > >> - When a perflib Check* succeeds, the meta library check checks > that > >> another code snippet can be compiled/linked/run. > > > > Thanks - and I agree, for all the reasons you've given in later emails > > that this seems like the way to go. > > > > But, in the mean time, I've got problems with the perflib stuff. > > Specifically, the scons build is not picking up the atlas library for > > blas and lapack that works for the standard build. I'm on a 64 bit > > system, providing an extra layer of complexity. > I have never tested the thing on 64 bits (I don't have easy access to 64 > bits machine, which should change soon, hopefully), so it is not > surprising it does not work now. I have taken this into account for the > design, though (see below). > > > > I've attached the build logs. I noticed that, for atlas, you check > > for atlas_enum.c - but do you in fact need this for the build? > Now. I just wanted one header specific to atlas. It looks like not all > version of ATLAS install this one, unfortunately (3.8, for example). > > numpy.distutils seemed to be satisfied with cblas.h and clapack.h in > > /usr/local/include/atlas. It's no big deal to copy it from sources, > > but was there a reason you chose that file? > No reason, but it cannot be cblas.h (it has to be atlas specific; > otherwise, it does not make sense). The list of headers to check can be > empty, though. > > > > The test for linking to blas and lapack from atlas fails too - is this > > a false positive? > Hmm, if atlas check does not work, it sounds like a right negative to me > :) If ATLAS is not detected correctly, it won't be used by blas/lapack > checkers. Or do you mean something else ? > > > > For both numpy.distutils and numpyscons, default locations of > > libraries do not include the lib64 libraries like /usr/local/lib64 > > that us 64 bit people use. Is that easy to fix? > Yes, it is easy, in the sense that nothing in the checkers code is > harcoded: all the checks internally uses BuildConfig instances, which is > like a dictionary with default values and a restricted set of keys (the > keys are library path, libraries, headers, etc...). Those BuildConfig > instances are created from a config file (perflib.cfg), and should > always be customizable from site.cfg > > The options which can be customized can be found in the perflib.cfg > file. For example, having: > > [atlas] > htc = > > in your site.cfg should say to CheckATLAS to avoid looking for > atlas_enum.h > > To make 64 bits work by default is a bit more complicated. I thought a > bit about the problem: that's why the checkers do not use BuildConfig > instances directly, but request them through a BuildConfigFactory. One > problem is that I don't understand how 64 bits libraries work; more > precisely, what is the convention of library path ? Is there a lib64 for > each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the > standard ones (/lib, /usr/lib) checked at all, after the 64 bits > counterpart ? It varies with the Linux distro. The usual convention (LSB, I think) uses /usr/local/lib64, but Debian and distros derived from Debian use /usr/local/lib instead. That's how it was the last time I checked, anyway. I don't know what Gentoo and all the others do. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Jan 25 22:13:10 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 25 Jan 2008 21:13:10 -0600 Subject: [Numpy-discussion] NPY_FORCECAST In-Reply-To: <56DF44D1-0F58-4E77-B8EB-61F108E4C21C@sandia.gov> References: <56DF44D1-0F58-4E77-B8EB-61F108E4C21C@sandia.gov> Message-ID: <479AA546.2070009@enthought.com> Bill Spotz wrote: > Hi, > > I am currently using PyArray_FromObject() to convert an input > argument to a 32-bit integer array. On a 64-bit architecture, it is > easy to create an integer array whose default type is 64-bit. When > this is sent to PyArray_FromObject(), it raises an error, saying > "array cannot be safely cast to required type", even though all of > its elements are representable by 32 bits. > There is not a per-element check to casting. The casting is done with a C-level coercion which does not check for overflow. > How hard would it be to implement a new option (NPY_ATTEMPTCAST?) > that attempts to make the cast, but raises an OverflowError if any of > the source data is too large for the target array? (Or, > alternatively, make this the default behavior if NPY_FORCECAST is > false?) > This would be a big change because it would have to be implemented (as far as I can tell) with some kind of checked-casting loops at the C-level. So, it could be done, but it would be a fairly large effort (and would impact the size of numpy significantly, I think). -Travis From charlesr.harris at gmail.com Sat Jan 26 01:32:40 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Jan 2008 23:32:40 -0700 Subject: [Numpy-discussion] tensordot bug? Message-ID: Is this a bug? In [21]: a = ones((2,2)) In [22]: b = ones((2,2,2,2)) In [23]: tensordot(a, tensordot(b, a, 2), 2) Out[23]: array(16.0) It seems to me that consistency with the dot product would require a scalar result, not a 0-dim array. Also, do we have a plain old tensor product? Outer flattens the indices. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Jan 26 01:30:55 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 26 Jan 2008 15:30:55 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> Message-ID: <479AD39F.5040304@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 25, 2008 3:21 AM, David Cournapeau > > > wrote: > > Matthew Brett wrote: > > Hi David, > > > >>> Basically, I'm trying to understand the library discovery, linking > >>> steps - ATLAS in particular. > >> Don't trust the included doc: it is not up-to-date, and that's > the part > >> which I totally redesigned since I wrote the initial doc. > > > >> - For a perflib check to succeed, I try to compile, link > and run > >> code snippets. Any failure means the Check* function will > return failure. > >> - When a perflib Check* succeeds, the meta library check > checks that > >> another code snippet can be compiled/linked/run. > > > > Thanks - and I agree, for all the reasons you've given in later > emails > > that this seems like the way to go. > > > > But, in the mean time, I've got problems with the perflib stuff. > > Specifically, the scons build is not picking up the atlas > library for > > blas and lapack that works for the standard build. I'm on a 64 bit > > system, providing an extra layer of complexity. > I have never tested the thing on 64 bits (I don't have easy access > to 64 > bits machine, which should change soon, hopefully), so it is not > surprising it does not work now. I have taken this into account > for the > design, though (see below). > > > > I've attached the build logs. I noticed that, for atlas, you check > > for atlas_enum.c - but do you in fact need this for the build? > Now. I just wanted one header specific to atlas. It looks like not all > version of ATLAS install this one, unfortunately (3.8, for example). > > numpy.distutils seemed to be satisfied with cblas.h and clapack.h in > > /usr/local/include/atlas. It's no big deal to copy it from sources, > > but was there a reason you chose that file? > No reason, but it cannot be cblas.h (it has to be atlas specific; > otherwise, it does not make sense). The list of headers to check > can be > empty, though. > > > > The test for linking to blas and lapack from atlas fails too - > is this > > a false positive? > Hmm, if atlas check does not work, it sounds like a right negative > to me > :) If ATLAS is not detected correctly, it won't be used by blas/lapack > checkers. Or do you mean something else ? > > > > For both numpy.distutils and numpyscons, default locations of > > libraries do not include the lib64 libraries like /usr/local/lib64 > > that us 64 bit people use. Is that easy to fix? > Yes, it is easy, in the sense that nothing in the checkers code is > harcoded: all the checks internally uses BuildConfig instances, > which is > like a dictionary with default values and a restricted set of keys > (the > keys are library path, libraries, headers, etc...). Those BuildConfig > instances are created from a config file (perflib.cfg), and should > always be customizable from site.cfg > > The options which can be customized can be found in the perflib.cfg > file. For example, having: > > [atlas] > htc = > > in your site.cfg should say to CheckATLAS to avoid looking for > atlas_enum.h > > To make 64 bits work by default is a bit more complicated. I thought a > bit about the problem: that's why the checkers do not use BuildConfig > instances directly, but request them through a BuildConfigFactory. One > problem is that I don't understand how 64 bits libraries work; more > precisely, what is the convention of library path ? Is there a > lib64 for > each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the > standard ones (/lib, /usr/lib) checked at all, after the 64 bits > counterpart ? > > > It varies with the Linux distro. The usual convention (LSB, I think) > uses /usr/local/lib64, but Debian and distros derived from Debian use > /usr/local/lib instead. That's how it was the last time I checked, > anyway. I don't know what Gentoo and all the others do. Grrr, that's annoying. Do you know any resource which has clear explanation on that ? (reading LSB documents do not appeal much to me, and I would only do that as a last resort). cheers, David From tjhnson at gmail.com Sat Jan 26 02:39:33 2008 From: tjhnson at gmail.com (Tom Johnson) Date: Fri, 25 Jan 2008 23:39:33 -0800 Subject: [Numpy-discussion] Problems with long Message-ID: Hi, I'm having some troubles with long. >>> from numpy import log >>> log(8463186938969424928L) 43.5822574833 >>> log(10454852688145851272L) : 'long' object has no attribute 'log' Thoughts? From charlesr.harris at gmail.com Sat Jan 26 03:09:25 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Jan 2008 01:09:25 -0700 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <479AD39F.5040304@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> <479AD39F.5040304@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 25, 2008 11:30 PM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > It varies with the Linux distro. The usual convention (LSB, I think) > > uses /usr/local/lib64, but Debian and distros derived from Debian use > > /usr/local/lib instead. That's how it was the last time I checked, > > anyway. I don't know what Gentoo and all the others do. > Grrr, that's annoying. Do you know any resource which has clear > explanation on that ? (reading LSB documents do not appeal much to me, > and I would only do that as a last resort). The Debian folks will tell you that they did it right and everyone else screwed up horribly O_o. Anyway, it looks like they put the 32 bit stuff in lib32, so if a distro has a lib32, then you can figure the 64 bit stuff is in lib. Those are the only two variations I know of. What 64 bit windows does, I have no idea. 32-bit (compat) libraries 64-bit (native) libraries Fedora Core /lib/, /usr/lib/ /lib64/, /usr/lib64/ Debian /lib32/, /usr/lib32/ /lib/, /usr/lib/ Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Jan 26 03:05:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 26 Jan 2008 17:05:13 +0900 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> <479AD39F.5040304@ar.media.kyoto-u.ac.jp> Message-ID: <479AE9B9.3010906@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Jan 25, 2008 11:30 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > > > > > > > It varies with the Linux distro. The usual convention (LSB, I think) > > uses /usr/local/lib64, but Debian and distros derived from > Debian use > > /usr/local/lib instead. That's how it was the last time I checked, > > anyway. I don't know what Gentoo and all the others do. > Grrr, that's annoying. Do you know any resource which has clear > explanation on that ? (reading LSB documents do not appeal much to me, > and I would only do that as a last resort). > > > The Debian folks will tell you that they did it right and everyone > else screwed up horribly O_o. Well, as arrogant as debian folks may be, they are almost always right :) I will take a look at the debian maintainer guide, then. > Anyway, it looks like they put the 32 bit stuff in lib32, so if a > distro has a lib32, then you can figure the 64 bit stuff is in lib. > Those are the only two variations I know of. What 64 bit windows does, > I have no idea. Windows and 64 bits, here is something which scares me. I also have to see how solaris does it. But anyway, since vmware can run 64 bits OS on my 32 bits linux, things should be easier. Thanks for the info, David From charlesr.harris at gmail.com Sat Jan 26 03:30:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Jan 2008 01:30:10 -0700 Subject: [Numpy-discussion] [ANN] numscons 0.3.0 release In-Reply-To: <479AE9B9.3010906@ar.media.kyoto-u.ac.jp> References: <47975EA0.1050504@ar.media.kyoto-u.ac.jp> <1e2af89e0801231447p3d09d610hae994caeda9560ae@mail.gmail.com> <47995F7E.6070509@ar.media.kyoto-u.ac.jp> <1e2af89e0801250152p546a2b6rf80039f546060296@mail.gmail.com> <4799B82D.6080600@ar.media.kyoto-u.ac.jp> <479AD39F.5040304@ar.media.kyoto-u.ac.jp> <479AE9B9.3010906@ar.media.kyoto-u.ac.jp> Message-ID: On Jan 26, 2008 1:05 AM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > On Jan 25, 2008 11:30 PM, David Cournapeau > > > > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > > > > > > It varies with the Linux distro. The usual convention (LSB, I > think) > > > uses /usr/local/lib64, but Debian and distros derived from > > Debian use > > > /usr/local/lib instead. That's how it was the last time I checked, > > > anyway. I don't know what Gentoo and all the others do. > > Grrr, that's annoying. Do you know any resource which has clear > > explanation on that ? (reading LSB documents do not appeal much to > me, > > and I would only do that as a last resort). > > > > > > The Debian folks will tell you that they did it right and everyone > > else screwed up horribly O_o. > Well, as arrogant as debian folks may be, they are almost always right > :) I will take a look at the debian maintainer guide, then. > > Anyway, it looks like they put the 32 bit stuff in lib32, so if a > > distro has a lib32, then you can figure the 64 bit stuff is in lib. > > Those are the only two variations I know of. What 64 bit windows does, > > I have no idea. > Windows and 64 bits, here is something which scares me. I also have to > see how solaris does it. But anyway, since vmware can run 64 bits OS on > my 32 bits linux, things should be easier. ISTR that Solaris does it the Debian way. I wouldn't put money on it without checking, though. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jan 26 03:41:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Jan 2008 01:41:28 -0700 Subject: [Numpy-discussion] Problems with long In-Reply-To: References: Message-ID: On Jan 26, 2008 12:39 AM, Tom Johnson wrote: > Hi, I'm having some troubles with long. > > >>> from numpy import log > >>> log(8463186938969424928L) > 43.5822574833 > >>> log(10454852688145851272L) > : 'long' object has no attribute 'log' > Numpy uses the standard C functions, so can't represent very large integers exactly. However, given the precision of the log function, it might be reasonable to truncate the digits and write the Python long as a float before conversion. That's what Python does. In [6]: import math In [7]: math.log(10454852688145851272L) Out[7]: 43.793597916587672 In [8]: float(10454852688145851272L) Out[8]: 1.045485268814585e+19 Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Jan 26 07:49:06 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 26 Jan 2008 06:49:06 -0600 Subject: [Numpy-discussion] Problems with long In-Reply-To: References: Message-ID: <479B2C42.9040603@enthought.com> Tom Johnson wrote: > Hi, I'm having some troubles with long. > > >>>> from numpy import log >>>> log(8463186938969424928L) >>>> > 43.5822574833 > >>>> log(10454852688145851272L) >>>> > : 'long' object has no attribute 'log' > The problem is that the latter long integer is too big to fit into an int64 (long long) and so it converts it to an object array. The default behavior of log on object arrays is to look for a method on each element of the array called log and call that. Your best bet is to convert to double before calling log log(float(10454852688145851272L)) -Travis O. From pav at iki.fi Sat Jan 26 11:11:50 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Jan 2008 18:11:50 +0200 Subject: [Numpy-discussion] segfault problem with numpy and pickle In-Reply-To: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> References: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> Message-ID: <1201363910.6964.6.camel@localhost.localdomain> to, 2008-01-24 kello 13:18 -0700, David Bolme kirjoitti: > A am having some trouble when pickling numpy arrays. Basically I use > one python script to create the array and then pickle it. When I load > the pickled array using a different python script it appears to load > fine. When I try to perform a matrix multiply on the array with a > vector (using dot) python crashes with a segmentation violation. The > array contains 64bit floating point numbers and has a shape of (47, > 16384) and the vector has a shape of (16384,1). Other shapes I have > tested also have this problem. A workaround is to create a new array > that is initialized with the data from the first. For example: This probably is the same problem as bug #551, http://projects.scipy.org/scipy/numpy/ticket/551 Summary: The cause is not known at this point, but so far appears to have something to do with SSE2 optimized ATLAS library. Are you using Atlas library on Mac OS X, and if yes, which version? Do you have SSE2 optimizations enabled? What happens if you try to use a non-SSE2-optimized Atlas, eg. by pointing LD_LIBRARY_PATH (I'm not sure this works on Mac, but I assume so...) to non-optimized Atlas libraries: LD_LIBRARY_PATH=/usr/lib/atlas python crasher.py -- Pauli Virtanen From bolme1234 at comcast.net Sat Jan 26 17:01:55 2008 From: bolme1234 at comcast.net (David Bolme) Date: Sat, 26 Jan 2008 15:01:55 -0700 Subject: [Numpy-discussion] segfault problem with numpy and pickle In-Reply-To: <1201363910.6964.6.camel@localhost.localdomain> References: <5C3C064A-B9E2-4DF3-B09D-CF6A87FEB462@comcast.net> <1201363910.6964.6.camel@localhost.localdomain> Message-ID: <9521F49B-FBA4-4A9A-8DEE-6B2253321F23@comcast.net> I think you are right. This does seem to be the same bug as 551. I will try a non optimized ATLAS to see if that helps. From efiring at hawaii.edu Sun Jan 27 01:30:01 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 26 Jan 2008 20:30:01 -1000 Subject: [Numpy-discussion] major bug in fromstring, ascii mode Message-ID: <479C24E9.3010602@hawaii.edu> In the course of trying to parse ascii times, I ran into a puzzling bug. Sometimes it works as expected: In [31]:npy.fromstring('23:19:01', dtype=int, sep=':') Out[31]:array([23, 19, 1]) But sometimes it doesn't: In [32]:npy.fromstring('23:09:01', dtype=int, sep=':') Out[32]:array([23, 0]) In [33]:npy.__version__ Out[33]:'1.0.5.dev4742' In [34]:npy.fromstring('23:09:01', dtype=int, sep=':', count=3) Out[34]:array([23, 0, 16]) In [35]:npy.fromstring('23 09 01', dtype=int, sep=' ', count=3) Out[35]:array([23, 0, 9]) In [36]:npy.fromstring('23 09 01', dtype=int, sep=' ') Out[36]:array([23, 0, 9, 1]) Maybe it is a problem specific to int conversion; examples that fail with int work with float: In [37]:npy.fromstring('23 09 01', dtype=float, sep=' ') Out[37]:array([ 23., 9., 1.]) In [38]:npy.fromstring('23:09:01', dtype=float, sep=':') Out[38]:array([ 23., 9., 1.]) Eric From charlesr.harris at gmail.com Sun Jan 27 03:16:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 27 Jan 2008 01:16:36 -0700 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: <479C24E9.3010602@hawaii.edu> References: <479C24E9.3010602@hawaii.edu> Message-ID: On Jan 26, 2008 11:30 PM, Eric Firing wrote: > In the course of trying to parse ascii times, I ran into a puzzling bug. > Sometimes it works as expected: > > In [31]:npy.fromstring('23:19:01', dtype=int, sep=':') > Out[31]:array([23, 19, 1]) > > But sometimes it doesn't: > > In [32]:npy.fromstring('23:09:01', dtype=int, sep=':') > Out[32]:array([23, 0]) > > In [33]:npy.__version__ > Out[33]:'1.0.5.dev4742' > Works here. In [6]: for i in range(100):fromstring('23:19:01', dtype=int, sep=':') In [7]: numpy.__version__ Out[7]: '1.0.5.dev4730' produces no failures. The fact that it fails for you sometimes and not others is very odd. It's like some sort of bizarre race condition or bit flip. What architecture/OS/compiler are you using? Have you tested memory? Chuck > > In [34]:npy.fromstring('23:09:01', dtype=int, sep=':', count=3) > Out[34]:array([23, 0, 16]) > > In [35]:npy.fromstring('23 09 01', dtype=int, sep=' ', count=3) > Out[35]:array([23, 0, 9]) > > In [36]:npy.fromstring('23 09 01', dtype=int, sep=' ') > Out[36]:array([23, 0, 9, 1]) > > Maybe it is a problem specific to int conversion; examples that fail > with int work with float: > > In [37]:npy.fromstring('23 09 01', dtype=float, sep=' ') > Out[37]:array([ 23., 9., 1.]) > > In [38]:npy.fromstring('23:09:01', dtype=float, sep=':') > Out[38]:array([ 23., 9., 1.]) > > > Eric > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sun Jan 27 05:12:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 27 Jan 2008 19:12:38 +0900 Subject: [Numpy-discussion] Which fortran compilers should numpy/scipy support ? Message-ID: <479C5916.1090204@ar.media.kyoto-u.ac.jp> Hi, For fortran support in numscons, I would like to know which fortran compiler we support ? For now, I successfully used the following: - g77 on linux. - mingw g77 on windows. - gfortran on linux and mac os X. - intel compiler on linux. - sun compiler on linux and solaris. I suspect most unix compilers to work ok wo explicit support (the annoying ones are really windows ones), but I would prefer having a list of compilers explicitely supported for my unit tests. cheers, David From pearu at cens.ioc.ee Sun Jan 27 05:49:43 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 27 Jan 2008 12:49:43 +0200 (EET) Subject: [Numpy-discussion] Which fortran compilers should numpy/scipy support ? In-Reply-To: <479C5916.1090204@ar.media.kyoto-u.ac.jp> References: <479C5916.1090204@ar.media.kyoto-u.ac.jp> Message-ID: <49242.85.166.27.136.1201430983.squirrel@cens.ioc.ee> On Sun, January 27, 2008 12:12 pm, David Cournapeau wrote: > Hi, > > For fortran support in numscons, I would like to know which fortran > compiler we support ? For now, I successfully used the following: > - g77 on linux. > - mingw g77 on windows. > - gfortran on linux and mac os X. > - intel compiler on linux. > - sun compiler on linux and solaris. > > I suspect most unix compilers to work ok wo explicit support (the > annoying ones are really windows ones), but I would prefer having a list > of compilers explicitely supported for my unit tests. All compilers defined in numpy/distutils/fcompiler used to work at some point but it is difficult to maintain them when one does not have access to all of these compilers and one must relay on users bug reports. In addition to the above list, the absoft compiler has been quite popular, both on linux and windows. The mips compiler has been used a lot and it used to work fine. The compaq compiler is used by windows users. I haven't seen hpux, vast, lahey, pg compilers used much but all except hpux (iirc) have been worked fine. nag compiler should be easy to support. I think we should support as much compilers as possible provided that any of the numpy/scipy developers have access to these and are willing to run the tests now and then. Pearu From beckers at orn.mpg.de Sun Jan 27 06:14:34 2008 From: beckers at orn.mpg.de (Gabriel J.L. Beckers) Date: Sun, 27 Jan 2008 12:14:34 +0100 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: References: <479C24E9.3010602@hawaii.edu> Message-ID: <1201432474.6587.8.camel@gabriel-desktop> Doesn't work here: In [1]: import numpy as npy In [2]: npy.fromstring('23:09:01', dtype=int, sep=':') Out[2]: array([23, 0]) In [3]: npy.__version__ Out[3]: '1.0.5.dev4722' In [4]: npy.fromstring('23:09:01', dtype=int, sep=':', count=3) Out[4]: array([ 23, 0, 151904160]) ... Pentium Dual Core, Ubuntu Linux 7.10, I have both gcc-3.4 and gcc-4.1 on my system; don't know which one the setup.py uses. Default is 4.1.3 Gabriel On Sun, 2008-01-27 at 01:16 -0700, Charles R Harris wrote: > > > On Jan 26, 2008 11:30 PM, Eric Firing wrote: > In the course of trying to parse ascii times, I ran into a > puzzling bug. > Sometimes it works as expected: > > In [31]:npy.fromstring('23:19:01', dtype=int, sep=':') > Out[31]:array([23, 19, 1]) > > But sometimes it doesn't: > > In [32]:npy.fromstring('23:09:01', dtype=int, sep=':') > Out[32]:array([23, 0]) > > In [33]:npy.__version__ > Out[33]:'1.0.5.dev4742' > > Works here. > > In [6]: for i in range(100):fromstring('23:19:01', dtype=int, sep=':') > > In [7]: numpy.__version__ > Out[7]: '1.0.5.dev4730' > > > produces no failures. The fact that it fails for you sometimes and not > others is very odd. It's like some sort of bizarre race condition or > bit flip. What architecture/OS/compiler are you using? Have you tested > memory? > > Chuck > > > > In [34]:npy.fromstring('23:09:01', dtype=int, sep=':', > count=3) > Out[34]:array([23, 0, 16]) > > In [35]:npy.fromstring('23 09 01', dtype=int, sep=' ', > count=3) > Out[35]:array([23, 0, 9]) > > In [36]:npy.fromstring('23 09 01', dtype=int, sep=' ') > Out[36]:array([23, 0, 9, 1]) > > Maybe it is a problem specific to int conversion; examples > that fail > with int work with float: > > In [37]:npy.fromstring('23 09 01', dtype=float, sep=' ') > Out[37]:array([ 23., 9., 1.]) > > In [38]:npy.fromstring('23:09:01', dtype=float, sep=':') > Out[38]:array([ 23., 9., 1.]) > > > Eric > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Dr. Gabri?l J.L. Beckers Max Planck Institute for Ornithology Group Neurobiology of Behaviour Postfach 1564, 82305 Starnberg, Germany Web: http://www.gbeckers.nl Phone: +49-8157932273, Fax: +49-8157932285 From pav at iki.fi Sun Jan 27 06:50:14 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 27 Jan 2008 13:50:14 +0200 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: References: <479C24E9.3010602@hawaii.edu> Message-ID: <1201434614.6963.4.camel@localhost.localdomain> su, 2008-01-27 kello 01:16 -0700, Charles R Harris kirjoitti: > > > On Jan 26, 2008 11:30 PM, Eric Firing wrote: > In the course of trying to parse ascii times, I ran into a > puzzling bug. > Sometimes it works as expected: > > In [31]:npy.fromstring('23:19:01', dtype=int, sep=':') > Out[31]:array([23, 19, 1]) > > But sometimes it doesn't: > > In [32]:npy.fromstring('23:09:01', dtype=int, sep=':') > Out[32]:array([23, 0]) > > In [33]:npy.__version__ > Out[33]:'1.0.5.dev4742' > > Works here. I think it's that some numbers work, and some don't. Consider: >>> npy.fromstring('23:06:01', dtype=int, sep=':') array([23, 6, 1]) >>> npy.fromstring('23:07:01', dtype=int, sep=':') array([23, 7, 1]) >>> npy.fromstring('23:08:01', dtype=int, sep=':') array([23, 0]) >>> npy.fromstring('23:09:01', dtype=int, sep=':') array([23, 0]) and >>> npy.fromstring('23:010:01', dtype=int, sep=':') array([23, 8, 1]) >>> npy.fromstring('23:011:01', dtype=int, sep=':') array([23, 9, 1]) and >>> npy.fromstring('23:0xff:01', dtype=int, sep=':') array([ 23, 255, 1]) Smells like some scanf function is interpreting numbers beginning with zero as octal, and recognizing also hexadecimals. This is a bit surprising, and whether this is the desired behavior is questionable. -- Pauli Virtanen From david at ar.media.kyoto-u.ac.jp Sun Jan 27 07:04:02 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 27 Jan 2008 21:04:02 +0900 Subject: [Numpy-discussion] Which fortran compilers should numpy/scipy support ? In-Reply-To: <49242.85.166.27.136.1201430983.squirrel@cens.ioc.ee> References: <479C5916.1090204@ar.media.kyoto-u.ac.jp> <49242.85.166.27.136.1201430983.squirrel@cens.ioc.ee> Message-ID: <479C7332.6000205@ar.media.kyoto-u.ac.jp> Pearu Peterson wrote: > On Sun, January 27, 2008 12:12 pm, David Cournapeau wrote: >> Hi, >> >> For fortran support in numscons, I would like to know which fortran >> compiler we support ? For now, I successfully used the following: >> - g77 on linux. >> - mingw g77 on windows. >> - gfortran on linux and mac os X. >> - intel compiler on linux. >> - sun compiler on linux and solaris. >> >> I suspect most unix compilers to work ok wo explicit support (the >> annoying ones are really windows ones), but I would prefer having a list >> of compilers explicitely supported for my unit tests. > > All compilers defined in numpy/distutils/fcompiler used to work > at some point but it is difficult to maintain them when one does > not have access to all of these compilers and one must relay on > users bug reports. Indeed. However, I believe numscons has an advantage here, because most (but not all) informations for fortran compilers are obtained at runtime. Testing that compiling works is hard, but testing that the runtime parser works is easy and testable on every platform. Since my parser is modeled on autoconf AC_F77_LDFLAGS and co, it should work with many compilers (except windows, obviously). > > In addition to the above list, the absoft compiler has been quite > popular, both on linux and windows. The mips compiler has been > used a lot and it used to work fine. The compaq compiler is used > by windows users. I haven't seen hpux, vast, lahey, pg compilers used > much but all except hpux (iirc) have been worked fine. > nag compiler should be easy to support. > > I think we should support as much compilers as possible provided > that any of the numpy/scipy developers have access to these and > are willing to run the tests now and then. If you have a few minutes to spend, and access to compilers not cited in my list, would you mind using some of the small tools I hacked to get informations on fortran compilers ? For example, for g77: - python ac_f77_ldflags.py g77 - python f77link_output.py g77 should give all the necessary informations to link g77 object code and object code compiled by a C compiler. The tools are in numscons/tests/fortran. The first one runs autoconf and get the value from AC_F77_LDFLAGS for a given F77 compiler, whereas the other one just compile an empty fortran program, to get the verbose output of the link step. They should work on any platform (of course, the ac_f77_ldflags won't work in a standard windows environment, since autoconf needs a basic unix environment). The archive is there: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numscons-0.3.4dev.tar.gz thanks, David From pav at iki.fi Sun Jan 27 12:57:45 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 27 Jan 2008 19:57:45 +0200 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <47918755.6070704@ar.media.kyoto-u.ac.jp> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> <1200681324.7361.5.camel@localhost.localdomain> <47918755.6070704@ar.media.kyoto-u.ac.jp> Message-ID: <1201456665.6963.15.camel@localhost.localdomain> la, 2008-01-19 kello 14:15 +0900, David Cournapeau kirjoitti: > Pauli Virtanen wrote: > > pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: > >> Hi there, > >> > >> I got a mercurial mirror of numpy available. I put some basic info > >> on the wiki > >> > >> http://scipy.org/scipy/numpy/wiki/HgMirror > > > > Nice! > > Don't hesitate to put more info there. Although I would consider myself > relatively proficient with bzr, mercurial has some few important > differences that I am still not sure to understand (in particular the > lack of "one branch is one directory" concept) I think I don't have edit access to the Trac Wiki... Anyway, it'd be useful to also have either hg or bzr read-only mirror of scipy too! I'd actually like to see these mirrors hosted at some single "official" place (svn.scipy.org?): If everyone maintains their own SVN mirror, the resulting repositories typically turn out "unrelated" as far as the SCM programs are concerned (because of different start points, different conversion tools etc.), and one cannot easily pull and merge changes from other people's repositories. > > hgsvn makes only repository-local svn tags. They go in .hg/localtags, > > from where they can be copied manually. I don't think hgsvn has an > > option to put the tags into .hgtags (from where they would be passed on > > automatically when the repository is cloned). > > Since tags seem to be handled through plain files in mercurial, would > copying localtags into .hgtags work ? It will work by just pasting the contents of localtags to .hgtags and committing. However, this would need to be done periodically and would pollute the history a bit. -- Pauli Virtanen From efiring at hawaii.edu Sun Jan 27 14:40:18 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 27 Jan 2008 09:40:18 -1000 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: <1201434614.6963.4.camel@localhost.localdomain> References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> Message-ID: <479CDE22.1060903@hawaii.edu> Pauli Virtanen wrote: > su, 2008-01-27 kello 01:16 -0700, Charles R Harris kirjoitti: >> >> On Jan 26, 2008 11:30 PM, Eric Firing wrote: >> In the course of trying to parse ascii times, I ran into a >> puzzling bug. >> Sometimes it works as expected: >> >> In [31]:npy.fromstring('23:19:01', dtype=int, sep=':') >> Out[31]:array([23, 19, 1]) >> >> But sometimes it doesn't: >> >> In [32]:npy.fromstring('23:09:01', dtype=int, sep=':') >> Out[32]:array([23, 0]) >> >> In [33]:npy.__version__ >> Out[33]:'1.0.5.dev4742' >> >> Works here. > > I think it's that some numbers work, and some don't. Consider: > >>>> npy.fromstring('23:06:01', dtype=int, sep=':') > array([23, 6, 1]) >>>> npy.fromstring('23:07:01', dtype=int, sep=':') > array([23, 7, 1]) >>>> npy.fromstring('23:08:01', dtype=int, sep=':') > array([23, 0]) >>>> npy.fromstring('23:09:01', dtype=int, sep=':') > array([23, 0]) > > and > >>>> npy.fromstring('23:010:01', dtype=int, sep=':') > array([23, 8, 1]) >>>> npy.fromstring('23:011:01', dtype=int, sep=':') > array([23, 9, 1]) > > and > >>>> npy.fromstring('23:0xff:01', dtype=int, sep=':') > array([ 23, 255, 1]) > > Smells like some scanf function is interpreting numbers beginning with > zero as octal, and recognizing also hexadecimals. That is it exactly. The code in core/src/arraytypes.inc.src is using scanf, and scanf tries hard to recognize integers specified in different ways. So, what caught me is a feature, not a bug, and I should have recognized it as such right away. The bug was in my expectations, not in the code. > > This is a bit surprising, and whether this is the desired behavior is > questionable. > From a user's standpoint it would be nice to be able to have numbers with leading zeros interpreted as base 10 instead of octal, since this turns up any time one converts date and time-of-day strings, and can occur in many other contexts also. (Outside of computer science octal is rare, as far as I know.) It looks like supporting this would require quite a bit of change in the code, however. I suspect it would have to go in as a kwarg that would be propagated through several layers of C function calls. Otherwise, if octal conversion support were simply dropped, I suspect someone else's code would break, and equally reasonable expectations would be violated. Eric From charlesr.harris at gmail.com Sun Jan 27 15:48:37 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 27 Jan 2008 13:48:37 -0700 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: <479CDE22.1060903@hawaii.edu> References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> Message-ID: On Jan 27, 2008 12:40 PM, Eric Firing wrote: > Pauli Virtanen wrote: > > > That is it exactly. The code in core/src/arraytypes.inc.src is using > scanf, and scanf tries hard to recognize integers specified in different > ways. So, what caught me is a feature, not a bug, and I should have > recognized it as such right away. The bug was in my expectations, not > in the code. > > > > > This is a bit surprising, and whether this is the desired behavior is > > questionable. > > > From a user's standpoint it would be nice to be able to have numbers > with leading zeros interpreted as base 10 instead of octal, since this > turns up any time one converts date and time-of-day strings, and can > occur in many other contexts also. (Outside of computer science octal is > rare, as far as I know.) It looks like supporting this would require > quite a bit of change in the code, however. I suspect it would have to > go in as a kwarg that would be propagated through several layers of C > function calls. Otherwise, if octal conversion support were simply > dropped, I suspect someone else's code would break, and equally > reasonable expectations would be violated. > I don't think the problem is scanf, at least not here. The following code snippet works fine for me. #include int main(int argc, char** argv) { int a,b,c; sscanf(argv[1], "%d :%d :%d", &a, &b, &c); printf("%d %d %d\n", a, b, c); return 0; } $[charris at f8 scratch]$ ./a.out "23:09:01" 23 9 1 Maybe it acts differently for files? $[charris at f8 scratch]$ echo "23:09:01" > tmp $[charris at f8 scratch]$ ./a.out 23 9 1 Nope, that works fine also. Numpy is making a type decision somewhere else. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Jan 27 17:19:47 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 28 Jan 2008 00:19:47 +0200 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> Message-ID: <1201472387.6963.28.camel@localhost.localdomain> su, 2008-01-27 kello 13:48 -0700, Charles R Harris kirjoitti: [clip] > > I don't think the problem is scanf, at least not here. The following code snippet works fine for me. > Reading the code in arraytypes.inc.src and multiarraymodule.c, it appears that numpy is using strtol(str, &tailptr, 0) for the string to integer conversion. Calling strtol with BASE == 0 enables the automatic base detection from the prefix. However, as you say, scanf does not do this. Numpy appears to use fscanf when reading data from files, so there is a discrepancy here: >>> from numpy import fromfile, fromstring >>> f = open('test.dat', 'w') >>> f.write("20:09:21") >>> f.close() >>> fromfile('test.dat', dtype=int, sep=':') array([20, 9, 21]) >>> fromstring('20:09:21', dtype=int, sep=':') array([20, 0]) Also, the following result is quite strange, seems like a silent failure: >>> fromfile('test.dat', dtype=int) array([809119794, 825375289]) I guess some more testcases should be written... -- Pauli Virtanen From charlesr.harris at gmail.com Sun Jan 27 17:43:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 27 Jan 2008 15:43:41 -0700 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: <1201472387.6963.28.camel@localhost.localdomain> References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> <1201472387.6963.28.camel@localhost.localdomain> Message-ID: On Jan 27, 2008 3:19 PM, Pauli Virtanen wrote: > > su, 2008-01-27 kello 13:48 -0700, Charles R Harris kirjoitti: > [clip] > > > > I don't think the problem is scanf, at least not here. The following > code snippet works fine for me. > > > > Reading the code in arraytypes.inc.src and multiarraymodule.c, it > appears that numpy is using strtol(str, &tailptr, 0) for the string to > integer conversion. Calling strtol with BASE == 0 enables the automatic > base detection from the prefix. > > However, as you say, scanf does not do this. Numpy appears to use fscanf > when reading data from files, so there is a discrepancy here: > > >>> from numpy import fromfile, fromstring > >>> f = open('test.dat', 'w') > >>> f.write("20:09:21") > >>> f.close() > > >>> fromfile('test.dat', dtype=int, sep=':') > array([20, 9, 21]) > >>> fromstring('20:09:21', dtype=int, sep=':') > array([20, 0]) > I vote for fromstring working like fromfile. > > Also, the following result is quite strange, seems like a silent > failure: > > >>> fromfile('test.dat', dtype=int) > array([809119794, 825375289]) > The default is to treat the file as containing binary data. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Mon Jan 28 05:17:31 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 28 Jan 2008 11:17:31 +0100 Subject: [Numpy-discussion] tensordot and axes argument Message-ID: <80c99e790801280217w38f37ae0w98e6d083d2e67a5d@mail.gmail.com> Shouldn't the "axes" argument in tensordot be named "axis"? L. -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Mon Jan 28 08:22:15 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 28 Jan 2008 14:22:15 +0100 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <1201456665.6963.15.camel@localhost.localdomain> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> <1200681324.7361.5.camel@localhost.localdomain> <47918755.6070704@ar.media.kyoto-u.ac.jp> <1201456665.6963.15.camel@localhost.localdomain> Message-ID: <85b5c3130801280522i7408cc0p520a5a64bc649a81@mail.gmail.com> On Jan 27, 2008 6:57 PM, Pauli Virtanen wrote: > > la, 2008-01-19 kello 14:15 +0900, David Cournapeau kirjoitti: > > Pauli Virtanen wrote: > > > pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: > > >> Hi there, > > >> > > >> I got a mercurial mirror of numpy available. I put some basic info > > >> on the wiki > > >> > > >> http://scipy.org/scipy/numpy/wiki/HgMirror > > > > > > Nice! > > > > Don't hesitate to put more info there. Although I would consider myself > > relatively proficient with bzr, mercurial has some few important > > differences that I am still not sure to understand (in particular the > > lack of "one branch is one directory" concept) > > I think I don't have edit access to the Trac Wiki... > > Anyway, it'd be useful to also have either hg or bzr read-only mirror of > scipy too! > > I'd actually like to see these mirrors hosted at some single "official" > place (svn.scipy.org?): If everyone maintains their own SVN mirror, the > resulting repositories typically turn out "unrelated" as far as the SCM > programs are concerned (because of different start points, different > conversion tools etc.), and one cannot easily pull and merge changes > from other people's repositories. > > > > hgsvn makes only repository-local svn tags. They go in .hg/localtags, > > > from where they can be copied manually. I don't think hgsvn has an > > > option to put the tags into .hgtags (from where they would be passed on > > > automatically when the repository is cloned). > > > > Since tags seem to be handled through plain files in mercurial, would > > copying localtags into .hgtags work ? > > It will work by just pasting the contents of localtags to .hgtags and > committing. However, this would need to be done periodically and would > pollute the history a bit. Better solution is to keep the tags in a Mercurial Queues patch, as I did. See for yourself: http://hg.certik.cz/numpy-hg/ BTW this mirror is much faster for me. You can browse the full history with tags online, you can clone like this: hg clone http://hg.certik.cz/numpy-hg/ It will convert the MQ patch to a regular commit, so you won't see the difference. But I, on the server, will just do: $ hg qpop Patch queue now empty $ hg pull pulling from /home/ondra/scipy/trunk searching for changes no changes found $ hg qpush applying tags.patch Now at: tags.patch Then I'll copy the updated localtags to .hgtags and do: $ hg qrefresh and that's it. No history pollution. If I decide I don't wont't the tags, I just do "hg qpop" and tags are gone. Ondrej From HansJoachim.Ehlers at eumetsat.int Mon Jan 28 10:26:55 2008 From: HansJoachim.Ehlers at eumetsat.int (Hans-Joachim Ehlers) Date: Mon, 28 Jan 2008 16:26:55 +0100 Subject: [Numpy-discussion] scipy build error - ./numpy/distutils/ccompiler.py bug ? - Was (Numpy-discussion Digest, Vol 16, Issue 59 ) In-Reply-To: References: Message-ID: <479E024A.8350.00AE.3@eumetsat.int> Given: AIX 5.3 xlc v8 xlf v10 essl v4.2 Build and installed: numpy 1.0.4 Build and installed: lapack 3.1.1 with CCI and essl support Task: Trying to build scipy 0.6.0 Error: Get the following error which seems to be numpy related: ... ... File "/usr/local/lib/python2.5/site-packages/numpy/distutils/ccompiler.py", line 303, in CCompiler_cxx_compiler + cxx.linker_so[2:] TypeError: can only concatenate list (not "str") to list ... Current Solution: Changed line 303 in /usr/local/lib/python2.5/site-packages/numpy/distutils/ccompiler.py from: ... cxx.linker_so = [cxx.linker_so[0]] + cxx.compiler_cxx[0] \ + cxx.linker_so[2:] ... to ... cxx.linker_so = [cxx.linker_so[0]] + [cxx.compiler_cxx[0]] \ + cxx.linker_so[2:] ... Is that change OK for the logic ? Must be a bug report opened for that ? BTW: Has somebody build scipy for AIX 5.3 with the xlc & xlf compilers ? If yes - Could she/he provide some information regarding the requirements tia Hajo From charlesr.harris at gmail.com Mon Jan 28 11:32:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Jan 2008 09:32:29 -0700 Subject: [Numpy-discussion] tensordot and axes argument In-Reply-To: <80c99e790801280217w38f37ae0w98e6d083d2e67a5d@mail.gmail.com> References: <80c99e790801280217w38f37ae0w98e6d083d2e67a5d@mail.gmail.com> Message-ID: On Jan 28, 2008 3:17 AM, lorenzo bolla wrote: > Shouldn't the "axes" argument in tensordot be named "axis"? > Axes is the plural of axis. Because tensordot sums over lists of axes, axes is the best word. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From HansJoachim.Ehlers at eumetsat.int Mon Jan 28 11:39:33 2008 From: HansJoachim.Ehlers at eumetsat.int (Hans-Joachim Ehlers) Date: Mon, 28 Jan 2008 17:39:33 +0100 Subject: [Numpy-discussion] Error during scipy.fftpack._fftpack compilation ( AIX/cc_r) In-Reply-To: References: Message-ID: <479E1350.8350.00AE.3@eumetsat.int> Given: AIX 5.3 xlc v8 xlf v10 During the build scipy the following error shows up: ..... building 'scipy.fftpack._fftpack' extension compiling C sources C compiler: cc_r -DNDEBUG -O2 -qmaxmem=-1 ... "scipy/fftpack/src/zfftnd_fftpack.c", line 92.5: 1506-046 (S) Syntax error. "scipy/fftpack/src/zfftnd_fftpack.c", line 93.5: 1506-046 (S) Syntax error. "scipy/fftpack/src/zfftnd_fftpack.c", line 115.13: 1506-046 (S) Syntax error. error: Command "cc_r -DNDEBUG -O2 -qmaxmem=-1 -Ibuild/src.aix-5.3-2.5 -I/usr/local/lib/python2.5/site-packages/numpy/core/include -I/usr/local/include/python2.5 -c scipy/fftpack/src/zfftnd.c -o build/temp.aix-5.3-2.5/scipy/fftpack/src/zfftnd.o" failed with exit status 1 .... The problem comes from C++ comment lines within c-code. The IBM Compiler v8 invoked as cc_r will complain and stop compilation. Solution: 1) Remove C++ comments from source code or 2) add the option "-qcpluscmt" for the cc_r compiler into the /etc/vac.cfg file or build python with this option. ( -qcpluscmt should be the default for cc_r but it does not afaics ) >From xlc ... -qcpluscmt | -qnocpluscmt Permits the usage of "//" to introduce a comment that lasts until the end of the current source line, as in C++. The default is -qcpluscmt when you invoke the compiler with xlc, xlc_r, cc, or cc_r, or when -qlanglvl is set to stdc99 or extc99. Otherwise, the default setting is -qnocpluscmt. .... regards Hajo BTW: Is there any way to add C-compiler options to the scipy build process. I could figure this out for the fortran compiler but for the c compiler i did not. From robert.kern at gmail.com Mon Jan 28 12:03:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 28 Jan 2008 11:03:58 -0600 Subject: [Numpy-discussion] Error during scipy.fftpack._fftpack compilation ( AIX/cc_r) In-Reply-To: <479E1350.8350.00AE.3@eumetsat.int> References: <479E1350.8350.00AE.3@eumetsat.int> Message-ID: <479E0AFE.7050304@gmail.com> Hans-Joachim Ehlers wrote: > Given: > AIX 5.3 > xlc v8 > xlf v10 > > During the build scipy the following error shows up: > > ..... > building 'scipy.fftpack._fftpack' extension > compiling C sources > C compiler: cc_r -DNDEBUG -O2 -qmaxmem=-1 > ... > "scipy/fftpack/src/zfftnd_fftpack.c", line 92.5: 1506-046 (S) Syntax error. > "scipy/fftpack/src/zfftnd_fftpack.c", line 93.5: 1506-046 (S) Syntax error. > "scipy/fftpack/src/zfftnd_fftpack.c", line 115.13: 1506-046 (S) Syntax error. > error: Command "cc_r -DNDEBUG -O2 -qmaxmem=-1 -Ibuild/src.aix-5.3-2.5 -I/usr/local/lib/python2.5/site-packages/numpy/core/include -I/usr/local/include/python2.5 -c scipy/fftpack/src/zfftnd.c -o build/temp.aix-5.3-2.5/scipy/fftpack/src/zfftnd.o" failed with exit status 1 > .... > > The problem comes from C++ comment lines within c-code. The IBM Compiler v8 invoked as cc_r will complain and stop compilation. > > Solution: > 1) Remove C++ comments from source code Fixed in SVN. Thank you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From HansJoachim.Ehlers at eumetsat.int Mon Jan 28 13:34:13 2008 From: HansJoachim.Ehlers at eumetsat.int (Hans-Joachim Ehlers) Date: Mon, 28 Jan 2008 19:34:13 +0100 Subject: [Numpy-discussion] /usr/bin/xlf: 1501-210 command option NO_ATLAS_INFO=1 contains an incorrect subargument In-Reply-To: References: Message-ID: <479E2E30.8350.00AE.3@eumetsat.int> Given: scipy-0.6.4 numpy 1.0.4 python 2.5.1 AIX 5.3 xlc v8 xlf v10 essel 4.2 lapack 3.1.1 with CCI Problem: During the scipy build - python setyp.py build - the following error shows up: .... building 'scipy.lib.blas.fblas' extension compiling C sources C compiler: cc_r -DNDEBUG -O2 -qmaxmem=-1 compile options: '-DNO_ATLAS_INFO=1 -Ibuild/src.aix-5.3-2.5 -I/usr/local/lib/python2.5/site-packages/numpy/core/include -I/usr/local/include/python2.5 -c' compiling Fortran sources Fortran f77 compiler: /usr/bin/xlf -qextname -O5 Fortran f90 compiler: /usr/bin/xlf90 -qextname -O5 Fortran fix compiler: /usr/bin/xlf90 -qfixed -qextname -O5 compile options: '-DNO_ATLAS_INFO=1 -Ibuild/src.aix-5.3-2.5 -I/usr/local/lib/python2.5/site-packages/numpy/core/include -I/usr/local/include/python2.5 -c' xlf:f77: build/src.aix-5.3-2.5/build/src.aix-5.3-2.5/scipy/lib/blas/fblas-f2pywrappers.f /usr/bin/xlf: 1501-210 command option NO_ATLAS_INFO=1 contains an incorrect subargument /usr/bin/xlf: 1501-210 command option NO_ATLAS_INFO=1 contains an incorrect subargument >>>>>>>>>>>>>>>>>> error: Command "/usr/bin/xlf -qextname -O5 -DNO_ATLAS_INFO=1 -Ibuild/src.aix-5.3-2.5 -I/usr/local/lib/python2.5/site-packages/numpy/core/include -I/usr/local/include/python2.5 -c -c build/src.aix-5.3-2.5/build/src.aix-5.3-2.5/scipy/lib/blas/fblas-f2pywrappers.f -o build/temp.aix-5.3-2.5/build/src.aix-5.3-2.5/build/src.aix-5.3-2.5/scipy/lib/blas/fblas-f2pywrappers.o" failed with exit status 40 <<<<<<<<<<<<<<<<<< I found a reference regarding the IBM XLF compiler running on mac http://projects.scipy.org/pipermail/scipy-user/2004-May/002884.html where Pearu says: ... it turns out that -D/-U options to xlf compiler are not for defining cpp macros... The following information from the xlf Compiler says regarding the -D option : ... -D Specifies whether the compiler compiles fixed source form lines with a D in column 1 or treats them as comments. Please note that in order to pass C-style -D macro definitions to the C preprocessor (e.g. compiling a file that ends with .F) use the -W option. For example: -WF,-DDEFINE_THIS -D is the short form of -qdlines ... -U Makes the compiler case-sensitive to identifier names. By default the compiler interprets all names as if they were lower-case. Solution: Changeing the Command line from : /usr/bin/xlf -qextname -O5 -DNO_ATLAS_INFO=1 ....../fblas-f2pywrappers.o to /usr/bin/xlf -qextname -O5 -WF,-DNO_ATLAS_INFO=1 .... /fblas-f2pywrappers.o allowed the command line to succed without an error. But i do not know if the -WF,DNO_ATLAS_INFO=1 will change the program logic nor where to change ( in case the -WF, is the solution ) this permanently. The only reference i found regarding -DNO_ATLAS_INFO=1 was within the file: ../site-packages/numpy/distutils/system_info.py tia Hajo From millman at berkeley.edu Mon Jan 28 17:33:38 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 28 Jan 2008 14:33:38 -0800 Subject: [Numpy-discussion] preparing for 1.0.5 release Message-ID: Hello, I am pushing back the NumPy 1.0.5 release by at least 1 week (http://projects.scipy.org/scipy/numpy/milestone/1.0.5). After speaking with Travis O., we would like to see the following branches merged this week: http://projects.scipy.org/scipy/numpy/browser/branches/maskedarray http://projects.scipy.org/scipy/numpy/browser/branches/build_with_scons/ Once these merges occur we would like to wait at least 1 week, before branching for the 1.0.5 release. This will give us an opportunity to see much wider testing of this work before making the new release. Please let me know if there is any reason that we should delay merging these two branches. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From efiring at hawaii.edu Mon Jan 28 18:09:38 2008 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 28 Jan 2008 13:09:38 -1000 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> <1201472387.6963.28.camel@localhost.localdomain> Message-ID: <479E60B2.2010003@hawaii.edu> Charles R Harris wrote: > Reading the code in arraytypes.inc.src and multiarraymodule.c, it > appears that numpy is using strtol(str, &tailptr, 0) for the string to > integer conversion. Calling strtol with BASE == 0 enables the automatic > base detection from the prefix. > > However, as you say, scanf does not do this. Numpy appears to use fscanf > when reading data from files, so there is a discrepancy here: > > >>> from numpy import fromfile, fromstring > >>> f = open('test.dat', 'w') > >>> f.write("20:09:21") > >>> f.close() > > >>> fromfile('test.dat', dtype=int, sep=':') > array([20, 9, 21]) > >>> fromstring('20:09:21', dtype=int, sep=':') > array([20, 0]) > > > I vote for fromstring working like fromfile. I agree. Can we get this change into 1.05? I could make a patch if that would help. Although I was wrong in calling this a "major bug", I think it is an inconsistency that should be removed. The fromfile and fromstring docstrings could also state explicitly what the behavior is. Eric From charlesr.harris at gmail.com Mon Jan 28 18:56:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Jan 2008 16:56:52 -0700 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: <479E60B2.2010003@hawaii.edu> References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> <1201472387.6963.28.camel@localhost.localdomain> <479E60B2.2010003@hawaii.edu> Message-ID: On Jan 28, 2008 4:09 PM, Eric Firing wrote: > Charles R Harris wrote: > > > Reading the code in arraytypes.inc.src and multiarraymodule.c, it > > appears that numpy is using strtol(str, &tailptr, 0) for the string > to > > integer conversion. Calling strtol with BASE == 0 enables the > automatic > > base detection from the prefix. > > > > However, as you say, scanf does not do this. Numpy appears to use > fscanf > > when reading data from files, so there is a discrepancy here: > > > > >>> from numpy import fromfile, fromstring > > >>> f = open('test.dat', 'w') > > >>> f.write("20:09:21") > > >>> f.close() > > > > >>> fromfile('test.dat', dtype=int, sep=':') > > array([20, 9, 21]) > > >>> fromstring('20:09:21', dtype=int, sep=':') > > array([20, 0]) > > > > > > I vote for fromstring working like fromfile. > > I agree. Can we get this change into 1.05? I could make a patch if > that would help. Although I was wrong in calling this a "major bug", I > think it is an inconsistency that should be removed. The fromfile and > fromstring docstrings could also state explicitly what the behavior is. Your best bet is to file a ticket. I you don't, I will, but I'll wait a bit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Mon Jan 28 20:26:41 2008 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 28 Jan 2008 15:26:41 -1000 Subject: [Numpy-discussion] major bug in fromstring, ascii mode In-Reply-To: References: <479C24E9.3010602@hawaii.edu> <1201434614.6963.4.camel@localhost.localdomain> <479CDE22.1060903@hawaii.edu> <1201472387.6963.28.camel@localhost.localdomain> <479E60B2.2010003@hawaii.edu> Message-ID: <479E80D1.4000103@hawaii.edu> Charles R Harris wrote: > > > On Jan 28, 2008 4:09 PM, Eric Firing > wrote: > > Charles R Harris wrote: > > > Reading the code in arraytypes.inc.src and multiarraymodule.c, it > > appears that numpy is using strtol(str, &tailptr, 0) for the > string to > > integer conversion. Calling strtol with BASE == 0 enables the > automatic > > base detection from the prefix. > > > > However, as you say, scanf does not do this. Numpy appears to > use fscanf > > when reading data from files, so there is a discrepancy here: > > > > >>> from numpy import fromfile, fromstring > > >>> f = open('test.dat', 'w') > > >>> f.write("20:09:21") > > >>> f.close() > > > > >>> fromfile('test.dat', dtype=int, sep=':') > > array([20, 9, 21]) > > >>> fromstring('20:09:21', dtype=int, sep=':') > > array([20, 0]) > > > > > > I vote for fromstring working like fromfile. > > I agree. Can we get this change into 1.05? I could make a patch if > that would help. Although I was wrong in calling this a "major bug", I > think it is an inconsistency that should be removed. The fromfile and > fromstring docstrings could also state explicitly what the behavior is. > > > Your best bet is to file a ticket. I you don't, I will, but I'll wait a bit. It is ticket #650. Eric From david at ar.media.kyoto-u.ac.jp Mon Jan 28 21:58:19 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 29 Jan 2008 11:58:19 +0900 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <85b5c3130801280522i7408cc0p520a5a64bc649a81@mail.gmail.com> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> <1200681324.7361.5.camel@localhost.localdomain> <47918755.6070704@ar.media.kyoto-u.ac.jp> <1201456665.6963.15.camel@localhost.localdomain> <85b5c3130801280522i7408cc0p520a5a64bc649a81@mail.gmail.com> Message-ID: <479E964B.3030807@ar.media.kyoto-u.ac.jp> Ondrej Certik wrote: > On Jan 27, 2008 6:57 PM, Pauli Virtanen wrote: >> la, 2008-01-19 kello 14:15 +0900, David Cournapeau kirjoitti: >>> Pauli Virtanen wrote: >>>> pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: >>>>> Hi there, >>>>> >>>>> I got a mercurial mirror of numpy available. I put some basic info >>>>> on the wiki >>>>> >>>>> http://scipy.org/scipy/numpy/wiki/HgMirror >>>> Nice! >>> Don't hesitate to put more info there. Although I would consider myself >>> relatively proficient with bzr, mercurial has some few important >>> differences that I am still not sure to understand (in particular the >>> lack of "one branch is one directory" concept) >> I think I don't have edit access to the Trac Wiki... >> >> Anyway, it'd be useful to also have either hg or bzr read-only mirror of >> scipy too! >> >> I'd actually like to see these mirrors hosted at some single "official" >> place (svn.scipy.org?): If everyone maintains their own SVN mirror, the >> resulting repositories typically turn out "unrelated" as far as the SCM >> programs are concerned (because of different start points, different >> conversion tools etc.), and one cannot easily pull and merge changes >> from other people's repositories. >> >>>> hgsvn makes only repository-local svn tags. They go in .hg/localtags, >>>> from where they can be copied manually. I don't think hgsvn has an >>>> option to put the tags into .hgtags (from where they would be passed on >>>> automatically when the repository is cloned). >>> Since tags seem to be handled through plain files in mercurial, would >>> copying localtags into .hgtags work ? >> It will work by just pasting the contents of localtags to .hgtags and >> committing. However, this would need to be done periodically and would >> pollute the history a bit. > > Better solution is to keep the tags in a Mercurial Queues patch, as I did. Good. As I said, I know bzr much better than hg, and I did the mirror to get something started (and get used to hg, too). Since you know hg, it is better than you maintain this for a while for people to try it out. > > See for yourself: > > http://hg.certik.cz/numpy-hg/ > > BTW this mirror is much faster for me. > Not surprising: my mirror is pure http (static-http as hg puts it), I can only upload files to it, meaning that I cannot launch any hg server. Same for bzr, by the way. thanks for doing this, David From matthew.yeomans at gmail.com Tue Jan 29 03:05:45 2008 From: matthew.yeomans at gmail.com (matthew yeomans) Date: Tue, 29 Jan 2008 09:05:45 +0100 Subject: [Numpy-discussion] Numpy Message-ID: <4ff732450801290005s21799d11odfb8be4097c9cd15@mail.gmail.com> Hello Dear Members of Numpy. I experienced some problems using numpy when i tried to compile it with py2exe. How this is done? Thanks and Regrads Matthew -- Kollox Ghal Xejn From lbolla at gmail.com Tue Jan 29 05:32:10 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 29 Jan 2008 11:32:10 +0100 Subject: [Numpy-discussion] difference numpy/matlab Message-ID: <80c99e790801290232t6e5c7245s789edd21f26f2ac2@mail.gmail.com> I noticed that: min([1+1j,-1+3j]) gives 1+1j in matlab (where for complex, min(abs) is used) but gives -1+3j in numpy (where lexicographic order is used) shouldn't this be mentioned somewhere in "Numpy for Matlab users" webpage? -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Jan 29 05:41:18 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Jan 2008 10:41:18 +0000 Subject: [Numpy-discussion] preparing for 1.0.5 release In-Reply-To: References: Message-ID: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> Hi, > http://projects.scipy.org/scipy/numpy/browser/branches/build_with_scons/ Excellent idea - and congratulations to David for all his hard work - but this is a large change, and I wonder if we need more time with the scons build system in the svn trunk? Not to say it should not be in 1.0.5, but suggesting a little more delay might be useful. See y'all, Matthew From david at ar.media.kyoto-u.ac.jp Tue Jan 29 05:38:06 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 29 Jan 2008 19:38:06 +0900 Subject: [Numpy-discussion] preparing for 1.0.5 release In-Reply-To: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> References: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> Message-ID: <479F020E.9090405@ar.media.kyoto-u.ac.jp> Matthew Brett wrote: > Hi, > >> http://projects.scipy.org/scipy/numpy/browser/branches/build_with_scons/ > > Excellent idea - and congratulations to David for all his hard work - > but this is a large change, and I wonder if we need more time with the > scons build system in the svn trunk? Not to say it should not be in > 1.0.5, but suggesting a little more delay might be useful. Note that the current build system using numpy.distutils would still be the default, and that the branch does *not* include numscons. IOW, the only things merged in the trunk would be the hook in distutils to call scons, the scons scripts, as well as a few modifications which hopefully should not change anything. Although there is still a lot to do in numscons, I don't expect much changes in the merged code. cheers, David From ovendrell at gmail.com Tue Jan 29 08:58:15 2008 From: ovendrell at gmail.com (Oriol Vendrell) Date: Tue, 29 Jan 2008 14:58:15 +0100 Subject: [Numpy-discussion] argsort memory problem? Message-ID: <20080129135815.GA19022@pci.uni-heidelberg.de> Hi all, I've noticed something that looks like an odd behaviour in array.argsort(). # test1 --------------------- from numpy import array while True: a=array([8.0,7.0,6.0,5.0,4.0,2.0]) i=a.argsort() # --------------------------- # test2 --------------------- from numpy import array a=array([8.0,7.0,6.0,5.0,4.0,2.0]) while True: i=a.argsort() # --------------------------- test1 runs out of memory after a few minutes, it seems that in each cycle some memory is allocated and never returned back. test2 runs fine until killed. I'm unsure if I'm missing something or if this could be a bug. I'm using numpy 1.0.1 with python 2.4.4 in a debian stable system. Any suggestions? Thanks, -- Oriol Vendrell ovendrell at gmail.com From alexandre.fayolle at logilab.fr Tue Jan 29 09:28:35 2008 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Tue, 29 Jan 2008 15:28:35 +0100 Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: <20080129135815.GA19022@pci.uni-heidelberg.de> References: <20080129135815.GA19022@pci.uni-heidelberg.de> Message-ID: <20080129142835.GF32511@logilab.fr> On Tue, Jan 29, 2008 at 02:58:15PM +0100, Oriol Vendrell wrote: > Hi all, > > I've noticed something that looks like an odd behaviour in array.argsort(). > > > # test1 --------------------- > from numpy import array > while True: > a=array([8.0,7.0,6.0,5.0,4.0,2.0]) > i=a.argsort() > # --------------------------- > > # test2 --------------------- > from numpy import array > a=array([8.0,7.0,6.0,5.0,4.0,2.0]) > while True: > i=a.argsort() > # --------------------------- > > > test1 runs out of memory after a few minutes, it seems that in each cycle > some memory is allocated and never returned back. > test2 runs fine until killed. > > I'm unsure if I'm missing something or if this could be a bug. I'm using > numpy 1.0.1 with python 2.4.4 in a debian stable system. Certainly a bug, but it has been fixed and I cannot reproduce in debian sid (using 1.0.4-5) -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: Digital signature URL: From lou_boog2000 at yahoo.com Tue Jan 29 10:13:59 2008 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 29 Jan 2008 07:13:59 -0800 (PST) Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: <20080129142835.GF32511@logilab.fr> Message-ID: <769926.44299.qm@web34408.mail.mud.yahoo.com> This still occurs in numpy 1.0.3.1 so must have been fixed between that and your 1.0.4-5 version. By the way the memory problem crashes my Intel Mac Book Pro (system 10.4.11) with the gray screen and black dialog box telling me to restart my computer. A very UN-unix like and UN-Mac like way of handling a memory problem IMHO. Let us Mac people not be too smug. -- Lou Pecora --- Alexandre Fayolle wrote: > On Tue, Jan 29, 2008 at 02:58:15PM +0100, Oriol > Vendrell wrote: > > Hi all, > > > > I've noticed something that looks like an odd > behaviour in array.argsort(). > > > > > > # test1 --------------------- > > from numpy import array > > while True: > > a=array([8.0,7.0,6.0,5.0,4.0,2.0]) > > i=a.argsort() > > # --------------------------- > > > > # test2 --------------------- > > from numpy import array > > a=array([8.0,7.0,6.0,5.0,4.0,2.0]) > > while True: > > i=a.argsort() > > # --------------------------- > > > > > > test1 runs out of memory after a few minutes, it > seems that in each cycle > > some memory is allocated and never returned back. > > test2 runs fine until killed. > > > > I'm unsure if I'm missing something or if this > could be a bug. I'm using > > numpy 1.0.1 with python 2.4.4 in a debian stable > system. > > Certainly a bug, but it has been fixed and I cannot > reproduce in debian > sid (using 1.0.4-5) > > -- > Alexandre Fayolle ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From ndbecker2 at gmail.com Tue Jan 29 10:49:46 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 29 Jan 2008 10:49:46 -0500 Subject: [Numpy-discussion] difference numpy/matlab References: <80c99e790801290232t6e5c7245s789edd21f26f2ac2@mail.gmail.com> Message-ID: lorenzo bolla wrote: > I noticed that: > > min([1+1j,-1+3j]) > > gives 1+1j in matlab (where for complex, min(abs) is used) > but gives -1+3j in numpy (where lexicographic order is used) > > shouldn't this be mentioned somewhere in "Numpy for Matlab users" webpage? > It should be stated that they're both wrong. From lbolla at gmail.com Tue Jan 29 11:00:49 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 29 Jan 2008 17:00:49 +0100 Subject: [Numpy-discussion] difference numpy/matlab In-Reply-To: References: <80c99e790801290232t6e5c7245s789edd21f26f2ac2@mail.gmail.com> Message-ID: <80c99e790801290800m2aa3b9e7u8c521e841a5428b7@mail.gmail.com> I'd rather say "arbitrary". On 1/29/08, Neal Becker wrote: > > lorenzo bolla wrote: > > > I noticed that: > > > > min([1+1j,-1+3j]) > > > > gives 1+1j in matlab (where for complex, min(abs) is used) > > but gives -1+3j in numpy (where lexicographic order is used) > > > > shouldn't this be mentioned somewhere in "Numpy for Matlab users" > webpage? > > > > It should be stated that they're both wrong. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Jan 29 11:04:12 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Jan 2008 16:04:12 +0000 Subject: [Numpy-discussion] Median again Message-ID: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> Hi, I was wondering whether there was any plan to change the anomalous interface of median, for example compared to 'mean', 'min', 'max' np.mean(a, axis=None, dtype=None, out=None) whereas: np.median(m) 'median(m) returns a median of m along the first dimension of m.' I think it would be a good to harmonize the median call signature and functionality with mean and the rest, but that will obviously break code. I was wondering whether the team would consider a migration plan on the lines of: median moved mediandim0 implementation of medianwithaxis or similar, with same call signature as mean. Deprecation warning for use of median, and return of mediandim0 for now. Eventual move of median to return medianwithaxis. Does that sound sensible? Matthew From oliphant at enthought.com Tue Jan 29 11:27:30 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Jan 2008 10:27:30 -0600 Subject: [Numpy-discussion] preparing for 1.0.5 release In-Reply-To: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> References: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> Message-ID: <479F53F2.4020500@enthought.com> Matthew Brett wrote: > Hi, > > >> http://projects.scipy.org/scipy/numpy/browser/branches/build_with_scons/ >> > > Excellent idea - and congratulations to David for all his hard work - > but this is a large change, and I wonder if we need more time with the > scons build system in the svn trunk? Not to say it should not be in > 1.0.5, but suggesting a little more delay might be useful. > All that is going to be merged are the "hooks" necessary to allow his numscons package to work. This is not a big change, as far as I know. -Travis From Joris.DeRidder at ster.kuleuven.be Tue Jan 29 11:51:07 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Tue, 29 Jan 2008 17:51:07 +0100 Subject: [Numpy-discussion] Median again In-Reply-To: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> Message-ID: <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> This feature request has been repeatedly asked before (e.g. 6 months ago). The relevant ticket (#558, although this only asks for axis support) mentions a milestone 1.1. I would like to ask if it could be moved somewhat higher on the priority list. I provided some code for axis support (see ticket), which I largely copied from the Scipy version back then. I suspect there are many other people who also implemented their own customized median() function that has axis support. > median moved mediandim0 > implementation of medianwithaxis or similar, with same call > signature as mean. > > Deprecation warning for use of median, and return of mediandim0 for > now. Eventual move of median to return medianwithaxis. This would confuse people even more, I'm afraid. First they're said that median() is deprecated, and then later on it becomes the standard function to use. I would actually prefer a short pain rather than a long one. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From sdb at cloud9.net Tue Jan 29 11:52:23 2008 From: sdb at cloud9.net (Stuart Brorson) Date: Tue, 29 Jan 2008 11:52:23 -0500 (EST) Subject: [Numpy-discussion] difference numpy/matlab In-Reply-To: <80c99e790801290800m2aa3b9e7u8c521e841a5428b7@mail.gmail.com> References: <80c99e790801290232t6e5c7245s789edd21f26f2ac2@mail.gmail.com> <80c99e790801290800m2aa3b9e7u8c521e841a5428b7@mail.gmail.com> Message-ID: I have to agree with Lorenzo. There is no natural ordering of the complex numbers. Any way you order them is arbitrary. Accepting this, the question then becomes "what should NumPy do when the user tries to do order comparison operations on complex numbers. The problem is that NumPy is schizophrenic. Watch this: -------------------------- --------------------- In [20]: A = numpy.array([3+1j, 1+3j, -1-3j, -1+3j, -3-1j]) In [21]: B = A[numpy.random.permutation(5)] In [22]: In [22]: A Out[22]: array([ 3.+1.j, 1.+3.j, -1.-3.j, -1.+3.j, -3.-1.j]) In [23]: B Out[23]: array([-1.+3.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) In [24]: numpy.greater(A, B) Out[24]: array([ True, False, False, False, False], dtype=bool) In [25]: numpy.maximum(A, B) Out[25]: array([ 3.+1.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) In [26]: In [26]: 3+1j > -1+3j --------------------------------------------------------------------------- Traceback (most recent call last) /tmp/test/web/ in () : no ordering relation is defined for complex numbers ---------------------------- ---------------------- I can compare complex numbers living in arrays using vectorized functions. However, doing comparison ops on the same complex numbers individually throws an exception. I raised this question some months ago in this thread: http://projects.scipy.org/pipermail/numpy-discussion/2007-September/029289.html Although I was initially confused about what NumPy was supposed to do, I eventually convinced myself that NumPy has contradictory behaviors depending upon exactly which comparison operation you were doing. Personally, I think NumPy should throw an exception whenever the user tries to do a greater-than or less-than type operation involving complex numbers. This would force the user to explicitly take the magnitude or the real & imaginary part of his number before doing any comparisons, thereby eliminating any confusion due to ambiguity. Just my $0.02. Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ On Tue, 29 Jan 2008, lorenzo bolla wrote: > I'd rather say "arbitrary". > > On 1/29/08, Neal Becker wrote: >> >> lorenzo bolla wrote: >> >>> I noticed that: >>> >>> min([1+1j,-1+3j]) >>> >>> gives 1+1j in matlab (where for complex, min(abs) is used) >>> but gives -1+3j in numpy (where lexicographic order is used) >>> >>> shouldn't this be mentioned somewhere in "Numpy for Matlab users" >> webpage? >>> >> >> It should be stated that they're both wrong. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > From robert.kern at gmail.com Tue Jan 29 12:55:16 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 29 Jan 2008 11:55:16 -0600 Subject: [Numpy-discussion] difference numpy/matlab In-Reply-To: References: <80c99e790801290232t6e5c7245s789edd21f26f2ac2@mail.gmail.com> <80c99e790801290800m2aa3b9e7u8c521e841a5428b7@mail.gmail.com> Message-ID: <479F6884.6080108@gmail.com> Stuart Brorson wrote: > I have to agree with Lorenzo. There is no natural ordering of the > complex numbers. Any way you order them is arbitrary. > > Accepting this, the question then becomes "what should NumPy do when > the user tries to do order comparison operations on complex numbers. > The problem is that NumPy is schizophrenic. Watch this: > > -------------------------- --------------------- > > In [20]: A = numpy.array([3+1j, 1+3j, -1-3j, -1+3j, -3-1j]) > > In [21]: B = A[numpy.random.permutation(5)] > > In [22]: > > In [22]: A > Out[22]: array([ 3.+1.j, 1.+3.j, -1.-3.j, -1.+3.j, -3.-1.j]) > > In [23]: B > Out[23]: array([-1.+3.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) > > In [24]: numpy.greater(A, B) > Out[24]: array([ True, False, False, False, False], dtype=bool) > > In [25]: numpy.maximum(A, B) > Out[25]: array([ 3.+1.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) > > In [26]: > > In [26]: 3+1j > -1+3j > --------------------------------------------------------------------------- > Traceback (most recent call > last) > > /tmp/test/web/ in () > > : no ordering relation is defined for > complex numbers > > ---------------------------- ---------------------- No, numpy is entirely consistent. In[26] has Python's complex numbers, not numpy's. In [1]: from numpy import complex64 In [2]: complex64(3+1j) > complex64(-1+3j) Out[2]: True -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Tue Jan 29 13:02:14 2008 From: faltet at carabos.com (Francesc Altet) Date: Tue, 29 Jan 2008 19:02:14 +0100 Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: <769926.44299.qm@web34408.mail.mud.yahoo.com> References: <769926.44299.qm@web34408.mail.mud.yahoo.com> Message-ID: <200801291902.14150.faltet@carabos.com> A Tuesday 29 January 2008, Lou Pecora escrigu?: > This still occurs in numpy 1.0.3.1 so must have been > fixed between that and your 1.0.4-5 version. It works here and I'm using NumPy 1.0.3, Python 2.5.1 on a Ubuntu 7.10 / Pentium4 machine. > By the way the memory problem crashes my Intel Mac > Book Pro (system 10.4.11) with the gray screen and > black dialog box telling me to restart my computer. A > very UN-unix like and UN-Mac like way of handling a > memory problem IMHO. Let us Mac people not be too > smug. Um, it would be nice if some other Mac-user can reproduce your problem. Perhaps you are suffering some other problem that can be exposed by this code snip. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From lou_boog2000 at yahoo.com Tue Jan 29 14:16:17 2008 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 29 Jan 2008 11:16:17 -0800 (PST) Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: <200801291902.14150.faltet@carabos.com> Message-ID: <628841.98517.qm@web34406.mail.mud.yahoo.com> Hmmm... Interesting. I am using Python 2.4.4. It would be nice to have other Mac people with same/other Python and numpy versions try the argsort "bug" code. -- Lou Pecora --- Francesc Altet wrote: > A Tuesday 29 January 2008, Lou Pecora escrigu?: > > This still occurs in numpy 1.0.3.1 so must have > been > > fixed between that and your 1.0.4-5 version. > > It works here and I'm using NumPy 1.0.3, Python > 2.5.1 on a Ubuntu 7.10 / > Pentium4 machine. > > > By the way the memory problem crashes my Intel Mac > > Book Pro (system 10.4.11) with the gray screen and > > black dialog box telling me to restart my > computer. A > > very UN-unix like and UN-Mac like way of handling > a > > memory problem IMHO. Let us Mac people not be too > > smug. > > Um, it would be nice if some other Mac-user can > reproduce your problem. > Perhaps you are suffering some other problem that > can be exposed by > this code snip. > > Cheers, > > -- > >0,0< Francesc Altet ? ? http://www.carabos.com/ -- Lou Pecora, my views are my own. ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From robince at gmail.com Tue Jan 29 14:23:11 2008 From: robince at gmail.com (Robin) Date: Tue, 29 Jan 2008 19:23:11 +0000 Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: <628841.98517.qm@web34406.mail.mud.yahoo.com> References: <200801291902.14150.faltet@carabos.com> <628841.98517.qm@web34406.mail.mud.yahoo.com> Message-ID: On Jan 29, 2008 7:16 PM, Lou Pecora wrote: > Hmmm... Interesting. I am using Python 2.4.4. It > would be nice to have other Mac people with same/other > Python and numpy versions try the argsort "bug" code. I don't see any memory leak with the test code. Mac OS X 10.5.1 Python 2.5.1 (not apple one) Numpy 1.0.5.dev4722 Robin From lxander.m at gmail.com Tue Jan 29 13:16:04 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 29 Jan 2008 13:16:04 -0500 Subject: [Numpy-discussion] Median again In-Reply-To: <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> Message-ID: <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> On Jan 29, 2008, at 11:51 AM, Joris De Ridder wrote: > This feature request has been repeatedly asked before (e.g. 6 months > ago). The relevant ticket (#558, although this only asks for axis > support) mentions a milestone 1.1. I would like to ask if it could be > moved somewhat higher on the priority list. > > I provided some code for axis support (see ticket), which I largely > copied from the Scipy version back then. I suspect there are many > other people who also implemented their own customized median() > function that has axis support. > >> median moved mediandim0 >> implementation of medianwithaxis or similar, with same call >> signature as mean. >> >> Deprecation warning for use of median, and return of mediandim0 for >> now. Eventual move of median to return medianwithaxis. > > This would confuse people even more, I'm afraid. First they're said > that median() is deprecated, and then later on it becomes the standard > function to use. I would actually prefer a short pain rather than a > long one. I would certainly like median to take the axis keyword. The axis keyword (and its friends) could be added to 1.0.5 with the default being 1 instead of None, so that it keeps compatibility with the 1.0 API. Then, with 1.1 (an API-breaking release) the default can be changed to None to restore consistency with mean, etc. From matthew.brett at gmail.com Tue Jan 29 16:31:30 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Jan 2008 21:31:30 +0000 Subject: [Numpy-discussion] Median again In-Reply-To: <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> Message-ID: <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> Hi, > >> median moved mediandim0 > >> implementation of medianwithaxis or similar, with same call > >> signature as mean. > >> > >> Deprecation warning for use of median, and return of mediandim0 for > >> now. Eventual move of median to return medianwithaxis. > > > > This would confuse people even more, I'm afraid. First they're said > > that median() is deprecated, and then later on it becomes the standard > > function to use. I would actually prefer a short pain rather than a > > long one. I was thinking the warning could be something like: "The current and previous version of numpy use a version of median that is not consistent with other summary functions such as mean. The calling convention of median will change in a future version of numpy to match that of the other summary functions. This compatible future version is implemented as medianwithaxis, and will become the default implementation of median. Please change any code using median to call medianwithaxis specifically, to maintain compatibility with future numpy APIs." > I would certainly like median to take the axis keyword. The axis > keyword (and its friends) could be added to 1.0.5 with the default > being 1 instead of None, so that it keeps compatibility with the 1.0 > API. Then, with 1.1 (an API-breaking release) the default can be > changed to None to restore consistency with mean, etc. But that would be very surprising to a new user, and might lead to some hard to track down silent bugs at a later date. Matthew From matthew.brett at gmail.com Tue Jan 29 16:45:16 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Jan 2008 21:45:16 +0000 Subject: [Numpy-discussion] preparing for 1.0.5 release In-Reply-To: <479F53F2.4020500@enthought.com> References: <1e2af89e0801290241n4277c7e0p131d82250f15663d@mail.gmail.com> <479F53F2.4020500@enthought.com> Message-ID: <1e2af89e0801291345q605a068g143303245d2151df@mail.gmail.com> Hi, > All that is going to be merged are the "hooks" necessary to allow his > numscons package to work. > > This is not a big change, as far as I know. Yes, sorry, I misunderstood - and had thought the plan was to move to the numscons build system as the default. Now I've understood it, it sounds like an excellent plan. Matthew From strawman at astraw.com Tue Jan 29 17:07:07 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 29 Jan 2008 14:07:07 -0800 Subject: [Numpy-discussion] Median again In-Reply-To: <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> Message-ID: <479FA38B.5030605@astraw.com> Considering that many of the statistical functions (mean, std, median) must iterate over all the data and that people (or at least myself) typically call them sequentially on the same data, it may make sense to make a super-function with less repetition. Instead of: x_mean = np.mean(x) x_median = np.median(x) x_std = np.std(x) x_min = np.min(x) x_max = np.max(x) We do: x_stats = np.get_descriptive_stats(x, stats=['mean','median','std','min','max'],axis=-1) And x_stats is a dictionary with 'mean','meadian','std','min', 'max' keys. The implementation could reduce the number of iterations over the data in this case. The implementation wouldn't have to be optimized initially, but could be gradually sped up once the interface is in place. I bring this up now to suggest such an idea as a more-general alternative to the "medianwithaxis" function proposed. What do you think? (Perhaps something like this already exists?) And, finally, this all surely belongs in scipy, but we already have stuff in numpy that can't be removed without seriously breaking backwards compatibility... -Andrew Matthew Brett wrote: > Hi, > >>>> median moved mediandim0 >>>> implementation of medianwithaxis or similar, with same call >>>> signature as mean. >>>> >>>> Deprecation warning for use of median, and return of mediandim0 for >>>> now. Eventual move of median to return medianwithaxis. >>>> >>> This would confuse people even more, I'm afraid. First they're said >>> that median() is deprecated, and then later on it becomes the standard >>> function to use. I would actually prefer a short pain rather than a >>> long one. >>> > > I was thinking the warning could be something like: > > "The current and previous version of numpy use a version of median > that is not consistent with other summary functions such as mean. The > calling convention of median will change in a future version of numpy > to match that of the other summary functions. This compatible future > version is implemented as medianwithaxis, and will become the default > implementation of median. Please change any code using median to call > medianwithaxis specifically, to maintain compatibility with future > numpy APIs." > > >> I would certainly like median to take the axis keyword. The axis >> keyword (and its friends) could be added to 1.0.5 with the default >> being 1 instead of None, so that it keeps compatibility with the 1.0 >> API. Then, with 1.1 (an API-breaking release) the default can be >> changed to None to restore consistency with mean, etc. >> > > But that would be very surprising to a new user, and might lead to > some hard to track down silent bugs at a later date. > > Matthew > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From efiring at hawaii.edu Tue Jan 29 17:22:52 2008 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 29 Jan 2008 12:22:52 -1000 Subject: [Numpy-discussion] Median again In-Reply-To: <479FA38B.5030605@astraw.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <479FA38B.5030605@astraw.com> Message-ID: <479FA73C.80501@hawaii.edu> Andrew Straw wrote: > Considering that many of the statistical functions (mean, std, median) > must iterate over all the data and that people (or at least myself) > typically call them sequentially on the same data, it may make sense to > make a super-function with less repetition. http://currents.soest.hawaii.edu/hg/hgwebdir.cgi/pycurrents/file/df129ff36f68/num/stats.py?style=gitweb I have something like that, in the link above (if the mailer does not break the line). I think it is quite flexible and efficient; it calculates only as much as necessary, so, for example, it only calculates the median if you ask for it. In the file that the link points to, you can import numpy.ma as MA to remove its one external dependency. Eric > > Instead of: > x_mean = np.mean(x) > x_median = np.median(x) > x_std = np.std(x) > x_min = np.min(x) > x_max = np.max(x) > > We do: > x_stats = np.get_descriptive_stats(x, > stats=['mean','median','std','min','max'],axis=-1) > And x_stats is a dictionary with 'mean','meadian','std','min', 'max' keys. > > The implementation could reduce the number of iterations over the data > in this case. The implementation wouldn't have to be optimized > initially, but could be gradually sped up once the interface is in > place. I bring this up now to suggest such an idea as a more-general > alternative to the "medianwithaxis" function proposed. What do you > think? (Perhaps something like this already exists?) And, finally, this > all surely belongs in scipy, but we already have stuff in numpy that > can't be removed without seriously breaking backwards compatibility... > > -Andrew > > Matthew Brett wrote: >> Hi, >> >>>>> median moved mediandim0 >>>>> implementation of medianwithaxis or similar, with same call >>>>> signature as mean. >>>>> >>>>> Deprecation warning for use of median, and return of mediandim0 for >>>>> now. Eventual move of median to return medianwithaxis. >>>>> >>>> This would confuse people even more, I'm afraid. First they're said >>>> that median() is deprecated, and then later on it becomes the standard >>>> function to use. I would actually prefer a short pain rather than a >>>> long one. >>>> >> I was thinking the warning could be something like: >> >> "The current and previous version of numpy use a version of median >> that is not consistent with other summary functions such as mean. The >> calling convention of median will change in a future version of numpy >> to match that of the other summary functions. This compatible future >> version is implemented as medianwithaxis, and will become the default >> implementation of median. Please change any code using median to call >> medianwithaxis specifically, to maintain compatibility with future >> numpy APIs." >> >> >>> I would certainly like median to take the axis keyword. The axis >>> keyword (and its friends) could be added to 1.0.5 with the default >>> being 1 instead of None, so that it keeps compatibility with the 1.0 >>> API. Then, with 1.1 (an API-breaking release) the default can be >>> changed to None to restore consistency with mean, etc. >>> >> But that would be very surprising to a new user, and might lead to >> some hard to track down silent bugs at a later date. >> >> Matthew >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Tue Jan 29 17:43:57 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Jan 2008 22:43:57 +0000 Subject: [Numpy-discussion] Median again In-Reply-To: <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> Message-ID: <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> Hi, > > >> median moved mediandim0 > > >> implementation of medianwithaxis or similar, with same call > > >> signature as mean. But - for the median function change - do we agree that this should be changed? I think it is a significant wart in the numpy API, and has caught quite a few people... Matthew From bblais at bryant.edu Tue Jan 29 17:55:18 2008 From: bblais at bryant.edu (Brian Blais) Date: Tue, 29 Jan 2008 17:55:18 -0500 Subject: [Numpy-discussion] load movie frames in python? Message-ID: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> Hello, Is there a way to read frames of a movie in python? Ideally, something as simple as: for frame in movie('mymovie.mov'): pass where frame is either a 2-D list, or a numpy array? The movie format can be anything, because I can probably convert things, but most convenient would be avi, mov, and flv (for youtube videos). thanks, Brian Blais -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Tue Jan 29 18:07:02 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 30 Jan 2008 00:07:02 +0100 Subject: [Numpy-discussion] load movie frames in python? In-Reply-To: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> References: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> Message-ID: <200801300007.02338.meine@informatik.uni-hamburg.de> On Dienstag 29 Januar 2008, Brian Blais wrote: > Is there a way to read frames of a movie in python? Ideally, > something as simple as: > > for frame in movie('mymovie.mov'): > pass > > > where frame is either a 2-D list, or a numpy array? The movie format > can be anything, because I can probably convert things, but most > convenient would be avi, mov, and flv (for youtube videos). I'd look for Gstreamer python bindings. Or run mplayer/mencoder with a "raw" format (e.g. yuv4mpeg or -ovc raw -vf format=bgr24/rgb15/...) using a pipe and processing the frames one by one. OK, that's no iterator-based interface yet, but one could probably hack that together in an hour or so. Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From oliphant at enthought.com Tue Jan 29 18:32:05 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Jan 2008 17:32:05 -0600 Subject: [Numpy-discussion] Median again In-Reply-To: <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> Message-ID: <479FB775.4080204@enthought.com> Matthew Brett wrote: > Hi, > >>>>> median moved mediandim0 >>>>> implementation of medianwithaxis or similar, with same call >>>>> signature as mean. >>>>> > > But - for the median function change - do we agree that this should be > changed? I think it is a significant wart in the numpy API, and has > caught quite a few people... > I'm fine with a median API change for 1.1. We can add the axis keyword for 1.0.5 as long as the default stays the same. We can also add the other keywords as well if appropriate defaults can be determined. -Travis O. From Joris.DeRidder at ster.kuleuven.be Tue Jan 29 19:32:50 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Wed, 30 Jan 2008 01:32:50 +0100 Subject: [Numpy-discussion] Median again In-Reply-To: <479FB775.4080204@enthought.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> <479FB775.4080204@enthought.com> Message-ID: <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> On 30 Jan 2008, at 00:32, Travis E. Oliphant wrote: > Matthew Brett wrote: >> Hi, >> >>>>>> median moved mediandim0 >>>>>> implementation of medianwithaxis or similar, with same call >>>>>> signature as mean. >>>>>> >> >> But - for the median function change - do we agree that this should >> be >> changed? I think it is a significant wart in the numpy API, and has >> caught quite a few people... >> > I'm fine with a median API change for 1.1. > > We can add the axis keyword for 1.0.5 as long as the default stays the > same. We can also add the other keywords as well if appropriate > defaults can be determined. Do you mean creating a median(a, axis=0) for 1.0.5, and changing it to median(a,axis=None) for 1.1? (Modulo other keywords). Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From oliphant at enthought.com Tue Jan 29 19:48:43 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Jan 2008 18:48:43 -0600 Subject: [Numpy-discussion] Median again In-Reply-To: <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> <479FB775.4080204@enthought.com> <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> Message-ID: <479FC96B.7030403@enthought.com> Joris De Ridder wrote: > On 30 Jan 2008, at 00:32, Travis E. Oliphant wrote: > > >> Matthew Brett wrote: >> >>> Hi, >>> >>> >>>>>>> median moved mediandim0 >>>>>>> implementation of medianwithaxis or similar, with same call >>>>>>> signature as mean. >>>>>>> >>>>>>> >>> But - for the median function change - do we agree that this should >>> be >>> changed? I think it is a significant wart in the numpy API, and has >>> caught quite a few people... >>> >>> >> I'm fine with a median API change for 1.1. >> >> We can add the axis keyword for 1.0.5 as long as the default stays the >> same. We can also add the other keywords as well if appropriate >> defaults can be determined. >> > > Do you mean creating a median(a, axis=0) for 1.0.5, and changing it to > median(a,axis=None) for 1.1? (Modulo other keywords). > Yes. That is the approach I prefer. -Travis From strawman at astraw.com Tue Jan 29 20:24:14 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 29 Jan 2008 17:24:14 -0800 Subject: [Numpy-discussion] load movie frames in python? In-Reply-To: <200801300007.02338.meine@informatik.uni-hamburg.de> References: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> <200801300007.02338.meine@informatik.uni-hamburg.de> Message-ID: <479FD1BE.80300@astraw.com> I'd sugget pyglet's avbin library. Hans Meine wrote: > On Dienstag 29 Januar 2008, Brian Blais wrote: > >> Is there a way to read frames of a movie in python? Ideally, >> something as simple as: >> >> for frame in movie('mymovie.mov'): >> pass >> >> >> where frame is either a 2-D list, or a numpy array? The movie format >> can be anything, because I can probably convert things, but most >> convenient would be avi, mov, and flv (for youtube videos). >> > > I'd look for Gstreamer python bindings. Or run mplayer/mencoder with a "raw" > format (e.g. yuv4mpeg or -ovc raw -vf format=bgr24/rgb15/...) using a pipe > and processing the frames one by one. OK, that's no iterator-based interface > yet, but one could probably hack that together in an hour or so. > > Ciao, / / .o. > /--/ ..o > / / ANS ooo > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bblais at bryant.edu Tue Jan 29 21:21:36 2008 From: bblais at bryant.edu (Brian Blais) Date: Tue, 29 Jan 2008 21:21:36 -0500 Subject: [Numpy-discussion] load movie frames in python? In-Reply-To: <479FD1BE.80300@astraw.com> References: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> <200801300007.02338.meine@informatik.uni-hamburg.de> <479FD1BE.80300@astraw.com> Message-ID: <48F6B360-071C-4CAC-95CE-65ABC999A372@bryant.edu> On Jan 29, 2008, at Jan 29:8:24 PM, Andrew Straw wrote: > I'd suggest pyglet's avbin library. > great suggestion! I never would have thought to do that. Do you happen to know how to convert a player.texture into a numpy.array? there is ImageData, but I can't seem to figure out how to do the conversion. bb -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: From strawman at astraw.com Tue Jan 29 21:25:55 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 29 Jan 2008 18:25:55 -0800 Subject: [Numpy-discussion] load movie frames in python? In-Reply-To: <48F6B360-071C-4CAC-95CE-65ABC999A372@bryant.edu> References: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> <200801300007.02338.meine@informatik.uni-hamburg.de> <479FD1BE.80300@astraw.com> <48F6B360-071C-4CAC-95CE-65ABC999A372@bryant.edu> Message-ID: <479FE033.4000207@astraw.com> I'm pretty sure there's code floating around the pyglet mailing list. I'd be happy to add it to http://code.astraw.com/projects/motmot/wiki/pygarrayimage if it seems reasonable. (pygarrayimage goes from numpy array to pyglet texture). Brian Blais wrote: > On Jan 29, 2008, at Jan 29:8:24 PM, Andrew Straw wrote: > >> I'd suggest pyglet's avbin library. >> > > > great suggestion! I never would have thought to do that. Do you > happen to know how to convert a > > player.texture into a numpy.array? > > there is ImageData, but I can't seem to figure out how to do the > conversion. > > > bb > -- > Brian Blais > bblais at bryant.edu > http://web.bryant.edu/~bblais > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From berthe.loic at gmail.com Wed Jan 30 03:49:44 2008 From: berthe.loic at gmail.com (LB) Date: Wed, 30 Jan 2008 00:49:44 -0800 (PST) Subject: [Numpy-discussion] Vectorizing a function Message-ID: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> Hi, I've got some questions on the numpy.vectorize function. Currently, i'm doing this kind of work : [ code] def calc_0d(x, y): """ make complexe calculation using two scalars x and y """ [ ... ] return res1, res2, res 3 # vectorize the function calc = vectorize(calc_0d) res1, res_2, res_3 = calc(array_x, array_y) [/code] This works fine. Really impressive. Good work for this ! My problem is that the complexe calculations made in calc_0d use some parameters, which are currently defined at the head of my python file. This is not very nice and I can't define a module containing theses two functions and call them with different parameters. I would like to make this cleaner and pass theses parameter as keyword argument, but this don't seems to be possible with vectorize. Indeed, some of theses parameters are array parameters and only the x and y arguments should be interpreted with the broadcasting rules.... What is the "good way" for doing this ? Regards, -- LB From yichun.wei at gmail.com Wed Jan 30 04:18:39 2008 From: yichun.wei at gmail.com (YW) Date: Wed, 30 Jan 2008 01:18:39 -0800 (PST) Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> Message-ID: <441052f6-77af-490a-ab53-18cd032f441c@q21g2000hsa.googlegroups.com> Try use a closure. On Jan 30, 12:49?am, LB wrote: > ? ?Hi, > > I've got some questions on the numpy.vectorize ?function. > Currently, i'm doing this kind of work : > > [ code] > def calc_0d(x, y): > ? ? """ make complexe calculation using two scalars x and y """ > ? ? [ ... ] > ? ? return res1, res2, res 3 > > # vectorize the function > calc = vectorize(calc_0d) > > res1, res_2, res_3 = calc(array_x, array_y) > [/code] > > This works fine. Really impressive. Good work for this ! > > My problem is that the complexe calculations made in calc_0d use some > parameters, which are currently defined at the head of my python file. > This is not very nice and I can't define a module containing theses > two functions and call them with different parameters. > > I would like to make this cleaner and pass theses parameter as > keyword ?argument, but this don't seems to be possible with vectorize. > Indeed, some of theses parameters are array parameters and only the x > and y arguments should be interpreted with the broadcasting rules.... > > What is the "good way" for doing this ? > > Regards, > > -- > LB > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From gael.varoquaux at normalesup.org Wed Jan 30 04:22:15 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 30 Jan 2008 10:22:15 +0100 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> Message-ID: <20080130092215.GA18502@phare.normalesup.org> On Wed, Jan 30, 2008 at 12:49:44AM -0800, LB wrote: > My problem is that the complexe calculations made in calc_0d use some > parameters, which are currently defined at the head of my python file. > This is not very nice and I can't define a module containing theses > two functions and call them with different parameters. > I would like to make this cleaner and pass theses parameter as > keyword argument, but this don't seems to be possible with vectorize. > Indeed, some of theses parameters are array parameters and only the x > and y arguments should be interpreted with the broadcasting rules.... > What is the "good way" for doing this ? I don't know what the "good way" is, but you can always use functional programming style (Oh, no, CaML is getting on me !): def calc_0d_params(param1, param2, param3): def calc_0d(x, y): # Here your code making use of param1, param2, param3) ... return calc_0d(x, y) you call the function like this: calc_0d_params(param1, param2, param3)(x, y) To vectorize it you can do: calc_0d_vect = lambda *params: vectorize(calc_0d_params(*params)) This is untested code, but I hope you get the idea. It all about partial evaluation of arguments. By the way, the parameters can now be keyword arguments. HTH, Ga?l From ovendrell at gmail.com Wed Jan 30 04:32:38 2008 From: ovendrell at gmail.com (Oriol Vendrell) Date: Wed, 30 Jan 2008 10:32:38 +0100 Subject: [Numpy-discussion] argsort memory problem? In-Reply-To: References: <200801291902.14150.faltet@carabos.com> <628841.98517.qm@web34406.mail.mud.yahoo.com> Message-ID: <20080130093238.GA6019@pci.uni-heidelberg.de> * Robin [2008-01-29 19:23:11 +0000]: > On Jan 29, 2008 7:16 PM, Lou Pecora wrote: > > Hmmm... Interesting. I am using Python 2.4.4. It > > would be nice to have other Mac people with same/other > > Python and numpy versions try the argsort "bug" code. > > I don't see any memory leak with the test code. > Mac OS X 10.5.1 > Python 2.5.1 (not apple one) > Numpy 1.0.5.dev4722 I have run the test1 code again, this time on my laptop PC (no MAC-user, sorry) using the last stable numpy release. The memory problem does _not_ show up now. I'm running with: - Ubuntu Feisty (kernel 2.6.17-12-generic i686) - python 2.5.1 (Feisty package) - numpy 1.0.4 (compiled with gcc version 4.1.2) However, the memory leak appears on my laptop if I use python 2.5.1 with numpy 1.0.1. At least here, this seems to be an issue dependent only on the numpy version, and a solved one. -- oriol From ondrej at certik.cz Wed Jan 30 04:52:30 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 30 Jan 2008 10:52:30 +0100 Subject: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available In-Reply-To: <479E964B.3030807@ar.media.kyoto-u.ac.jp> References: <47906C14.10104@ar.media.kyoto-u.ac.jp> <1200681324.7361.5.camel@localhost.localdomain> <47918755.6070704@ar.media.kyoto-u.ac.jp> <1201456665.6963.15.camel@localhost.localdomain> <85b5c3130801280522i7408cc0p520a5a64bc649a81@mail.gmail.com> <479E964B.3030807@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130801300152g1b929645g55fb16b516ec13a@mail.gmail.com> > > Better solution is to keep the tags in a Mercurial Queues patch, as I did. > Good. As I said, I know bzr much better than hg, and I did the mirror to > get something started (and get used to hg, too). Since you know hg, it > is better than you maintain this for a while for people to try it out. I also just found that if I serve the hg repository generated by "hgpullsvn" directly, the tags are visible on the web, but not when you clone. I sometimes do hg mirrors of some svn projects just to browse the history conveniently, so this is handy. Ondrej From yichun.wei at gmail.com Wed Jan 30 04:54:42 2008 From: yichun.wei at gmail.com (YW) Date: Wed, 30 Jan 2008 01:54:42 -0800 (PST) Subject: [Numpy-discussion] a view of a numpy object array Message-ID: Hi, How would one deal with the problem of getting a view of a numpy object array? For example, In [49]: b Out[49]: TaskResultArray([[TaskResult[ID:0]:{'out': 22}, TaskResult[ID:1]: {'out': 22}, TaskResult[ID:2]:{'out': 16}, TaskResult[ID:3]:{'out': 16}, TaskResult[ID:4]:{'out': 16}], [TaskResult[ID:5]:{'out': 16}, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None]], dtype=object) In [50]: b.shape Out[50]: (5, 5) In [58]: b[0,0]['out'] Out[58]: 21 What would be the best way without looping and copying, to get a view of this array, say c, so that we have In [69]: c Out[69]: array([[ 21., 22., 16., 16., 16.], [ 16., NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN]]) it is desired that c is a view instead of a copy so no extra care of updating c is needed. Is the tools that does this available in current numpy already? thanks in advance From yichun.wei at gmail.com Wed Jan 30 04:56:32 2008 From: yichun.wei at gmail.com (YW) Date: Wed, 30 Jan 2008 01:56:32 -0800 (PST) Subject: [Numpy-discussion] a view of a numpy object array Message-ID: Hi, How would one deal with the problem of getting a view of a numpy object array? For example, In [49]: b Out[49]: TaskResultArray([[TaskResult[ID:0]:{'out': 22}, TaskResult[ID:1]: {'out': 22}, TaskResult[ID:2]:{'out': 16}, TaskResult[ID:3]:{'out': 16}, TaskResult[ID:4]:{'out': 16}], [TaskResult[ID:5]:{'out': 16}, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None]], dtype=object) In [50]: b.shape Out[50]: (5, 5) In [58]: b[0,0]['out'] Out[58]: 21 What would be the best way without looping and copying, to get a view of this array, say c, so that we have In [69]: c Out[69]: array([[ 21., 22., 16., 16., 16.], [ 16., NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN]]) it is desired that c is a view instead of a copy so no extra care of updating c is needed. Is the tools that does this available in current numpy already? thanks in advance From yichun.wei at gmail.com Wed Jan 30 04:56:32 2008 From: yichun.wei at gmail.com (YW) Date: Wed, 30 Jan 2008 01:56:32 -0800 (PST) Subject: [Numpy-discussion] a view of a numpy object array Message-ID: Hi, How would one deal with the problem of getting a view of a numpy object array? For example, In [49]: b Out[49]: TaskResultArray([[TaskResult[ID:0]:{'out': 22}, TaskResult[ID:1]: {'out': 22}, TaskResult[ID:2]:{'out': 16}, TaskResult[ID:3]:{'out': 16}, TaskResult[ID:4]:{'out': 16}], [TaskResult[ID:5]:{'out': 16}, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None]], dtype=object) In [50]: b.shape Out[50]: (5, 5) In [58]: b[0,0]['out'] Out[58]: 21 What would be the best way without looping and copying, to get a view of this array, say c, so that we have In [69]: c Out[69]: array([[ 21., 22., 16., 16., 16.], [ 16., NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN], [ NaN, NaN, NaN, NaN, NaN]]) it is desired that c is a view instead of a copy so no extra care of updating c is needed. Is the tools that does this available in current numpy already? thanks in advance From sransom at nrao.edu Wed Jan 30 08:20:54 2008 From: sransom at nrao.edu (Scott Ransom) Date: Wed, 30 Jan 2008 08:20:54 -0500 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <20080130092215.GA18502@phare.normalesup.org> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: <20080130132054.GA27229@ssh.cv.nrao.edu> On a side note, given that I've seen quite a few posts about vectorize() over the past several months... I've written hundreds or thousands of functions that are intended to work with numeric/numpy arrays and/or scalars and I've _never_ (not once!) found a need for the vectorize function. Python's duck-typing has almost always allowed things to work without any (or sometimes with only minor) changes to the function. For example: In [12]: def foo(x, y): ....: return 2.0*x + y In [13]: foo(3.0, 5.0) Out[13]: 11.0 In [14]: foo(arange(4), ones(4)+3.0) Out[14]: array([ 4., 6., 8., 10.]) In [15]: foo(arange(4), 3.0) Out[15]: array([ 3., 5., 7., 9.]) That works fine with arrays, scalars, or array/scalar mixes in the calling. I do understand that more complicated functions might require vectorize(), however, I wonder if sometimes it is used when it doesn't need to be? Scott On Wed, Jan 30, 2008 at 10:22:15AM +0100, Gael Varoquaux wrote: > On Wed, Jan 30, 2008 at 12:49:44AM -0800, LB wrote: > > My problem is that the complexe calculations made in calc_0d use some > > parameters, which are currently defined at the head of my python file. > > This is not very nice and I can't define a module containing theses > > two functions and call them with different parameters. > > > I would like to make this cleaner and pass theses parameter as > > keyword argument, but this don't seems to be possible with vectorize. > > Indeed, some of theses parameters are array parameters and only the x > > and y arguments should be interpreted with the broadcasting rules.... > > > What is the "good way" for doing this ? > > I don't know what the "good way" is, but you can always use functional > programming style (Oh, no, CaML is getting on me !): > > def calc_0d_params(param1, param2, param3): > def calc_0d(x, y): > # Here your code making use of param1, param2, param3) > ... > > return calc_0d(x, y) > > you call the function like this: > > calc_0d_params(param1, param2, param3)(x, y) > > To vectorize it you can do: > > calc_0d_vect = lambda *params: vectorize(calc_0d_params(*params)) > > This is untested code, but I hope you get the idea. It all about partial > evaluation of arguments. By the way, the parameters can now be keyword > arguments. > > HTH, > > Ga?l > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From bblais at bryant.edu Wed Jan 30 08:39:57 2008 From: bblais at bryant.edu (Brian Blais) Date: Wed, 30 Jan 2008 08:39:57 -0500 Subject: [Numpy-discussion] load movie frames in python? In-Reply-To: <479FE033.4000207@astraw.com> References: <4697A1F5-6273-477D-8A6C-3BD34786CD64@bryant.edu> <200801300007.02338.meine@informatik.uni-hamburg.de> <479FD1BE.80300@astraw.com> <48F6B360-071C-4CAC-95CE-65ABC999A372@bryant.edu> <479FE033.4000207@astraw.com> Message-ID: <90B788BD-75A5-4EF4-BC4A-6C3C83012C52@bryant.edu> On Jan 29, 2008, at Jan 29:9:25 PM, Andrew Straw wrote: > I'm pretty sure there's code floating around the pyglet mailing list. > I'd be happy to add it to > http://code.astraw.com/projects/motmot/wiki/pygarrayimage if it seems > reasonable. (pygarrayimage goes from numpy array to pyglet texture). > I checked the pyglet users group, and the only mention I see is from: http://groups.google.com/group/pyglet-users/browse_thread/thread/ 5981f764902c7df/888328e19653be1a?lnk=gst&q=numpy#888328e19653be1a there, he has the solution: a = numpy.frombuffer(surf_clip.get_data(), numpy.uint8) a.shape = (sw, sh, 4) a = a[:,:,0:3] #drop alpha channel but when I do it (see code below) the image is clearly munged, so I think the decoding is not quite right. any ideas? thanks, Brian Blais -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais import pylab import numpy from pyglet import media,window win = window.Window(resizable=True) player = media.Player() filename='ica_mvl.avi' # from http://hlab.phys.rug.nl/demos/ica/ jiminy.html source = media.load(filename) player.queue(source) player.play() if True: player.dispatch_events() imdata=player.texture.get_image_data() a = numpy.frombuffer(player.texture.get_image_data().data, numpy.uint8) a.shape = (imdata.width, imdata.height, 4) a = a[:,:,0:3] #drop alpha channel # make gray im=a.sum(axis=1)/3 pylab.imshow(im,cmap=pylab.cm.gray) pylab.show() pylab.draw() -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Wed Jan 30 08:42:20 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 30 Jan 2008 15:42:20 +0200 Subject: [Numpy-discussion] Can not update a submatrix Message-ID: <1201700540.26483.16.camel@nadav.envision.co.il> In the following piece of code: >>> import numpy as N >>> R = N.arange(9).reshape(3,3) >>> ax = [1,2] >>> R array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> R[ax,:][:,ax] = 100 >>> R array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) Why R is not updated? I was expecting: >>> R array([[0, 1, 2], [3, 100, 100], [6, 100, 100]]) I am using numpy 1.04 on gentoo-linux/amd64 Nadav -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Jan 30 10:08:02 2008 From: faltet at carabos.com (Francesc Altet) Date: Wed, 30 Jan 2008 16:08:02 +0100 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <1201700540.26483.16.camel@nadav.envision.co.il> References: <1201700540.26483.16.camel@nadav.envision.co.il> Message-ID: <200801301608.03192.faltet@carabos.com> A Wednesday 30 January 2008, Nadav Horesh escrigu?: > In the following piece of code: > >>> import numpy as N > >>> R = N.arange(9).reshape(3,3) > >>> ax = [1,2] > >>> R > > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > > >>> R[ax,:][:,ax] = 100 > >>> R > > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > > Why R is not updated? Because R[ax] is not a view of R, but another copy of the original object (fancy indexing does return references to different objects). In order to get views, you must specify only a slice of the original array. For example: In [50]: S = R[::2] In [51]: S[:] = 2 In [52]: R Out[52]: array([[2, 2, 2], [3, 4, 5], [2, 2, 2]]) So, what you need is something like: In [68]: R = N.arange(9).reshape(3,3) In [69]: S = R[1:3,:][:,1:3] In [70]: S[:] = 2 In [71]: R Out[71]: array([[0, 1, 2], [3, 2, 2], [6, 2, 2]]) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From nadavh at visionsense.com Wed Jan 30 10:21:40 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 30 Jan 2008 17:21:40 +0200 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <200801301608.03192.faltet@carabos.com> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> Message-ID: <1201706500.7030.3.camel@nadav.envision.co.il> But: >>> R[ax,:] = 100 >>> R array([[ 0, 1, 2], [100, 100, 100], [100, 100, 100]]) >>> R[:,ax] = 200 >>> R array([[ 0, 200, 200], [100, 200, 200], [100, 200, 200]]) Do I get an array view only if the array is contiguous? Nadav. On Wed, 2008-01-30 at 16:08 +0100, Francesc Altet wrote: > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > In the following piece of code: > > >>> import numpy as N > > >>> R = N.arange(9).reshape(3,3) > > >>> ax = [1,2] > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > >>> R[ax,:][:,ax] = 100 > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > Why R is not updated? > > Because R[ax] is not a view of R, but another copy of the original > object (fancy indexing does return references to different objects). > In order to get views, you must specify only a slice of the original > array. For example: > > In [50]: S = R[::2] > In [51]: S[:] = 2 > In [52]: R > Out[52]: > array([[2, 2, 2], > [3, 4, 5], > [2, 2, 2]]) > > So, what you need is something like: > > In [68]: R = N.arange(9).reshape(3,3) > In [69]: S = R[1:3,:][:,1:3] > In [70]: S[:] = 2 > In [71]: R > Out[71]: > array([[0, 1, 2], > [3, 2, 2], > [6, 2, 2]]) > > Cheers, > -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Wed Jan 30 10:26:50 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 30 Jan 2008 16:26:50 +0100 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <1201706500.7030.3.camel@nadav.envision.co.il> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> <1201706500.7030.3.camel@nadav.envision.co.il> Message-ID: <200801301626.50534.meine@informatik.uni-hamburg.de> Am Mittwoch, 30. Januar 2008 16:21:40 schrieb Nadav Horesh: > But: > >>> R[ax,:] = 100 This is calling __setitem__, i.e. does not create either a view or a copy. Non-contiguous views (e.g. using [::2]) are also possible AFAIK, but fancy indexing is something different. -- Ciao, / / /--/ / / ANS From lbolla at gmail.com Wed Jan 30 10:26:57 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 30 Jan 2008 16:26:57 +0100 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <200801301608.03192.faltet@carabos.com> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> Message-ID: <80c99e790801300726g67acdb90yec3bb430b1cb6d8e@mail.gmail.com> you simply need to change the definition of ax: ax = slice(1,3) and all works fine. L. On 1/30/08, Francesc Altet wrote: > > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > In the following piece of code: > > >>> import numpy as N > > >>> R = N.arange(9).reshape(3,3) > > >>> ax = [1,2] > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > >>> R[ax,:][:,ax] = 100 > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > Why R is not updated? > > Because R[ax] is not a view of R, but another copy of the original > object (fancy indexing does return references to different objects). > In order to get views, you must specify only a slice of the original > array. For example: > > In [50]: S = R[::2] > In [51]: S[:] = 2 > In [52]: R > Out[52]: > array([[2, 2, 2], > [3, 4, 5], > [2, 2, 2]]) > > So, what you need is something like: > > In [68]: R = N.arange(9).reshape(3,3) > In [69]: S = R[1:3,:][:,1:3] > In [70]: S[:] = 2 > In [71]: R > Out[71]: > array([[0, 1, 2], > [3, 2, 2], > [6, 2, 2]]) > > Cheers, > > -- > >0,0< Francesc Altet http://www.carabos.com/ > V V C?rabos Coop. V. Enjoy Data > "-" > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 30 10:27:34 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2008 08:27:34 -0700 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <1201706500.7030.3.camel@nadav.envision.co.il> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> <1201706500.7030.3.camel@nadav.envision.co.il> Message-ID: On Jan 30, 2008 8:21 AM, Nadav Horesh wrote: > But: > > > >>> R[ax,:] = 100 > >>> R > array([[ 0, 1, 2], > [100, 100, 100], > [100, 100, 100]]) > >>> R[:,ax] = 200 > >>> R > array([[ 0, 200, 200], > [100, 200, 200], > [100, 200, 200]]) > > Do I get an array view only if the array is contiguous? > You get an array view if the elements can be addressed with offsets, strides, and counts. Think nested for loops. In [1]: a = zeros((4,4)) In [2]: a[::2,::2] = 1 In [3]: a Out[3]: array([[ 1., 0., 1., 0.], [ 0., 0., 0., 0.], [ 1., 0., 1., 0.], [ 0., 0., 0., 0.]]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Wed Jan 30 10:31:36 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 30 Jan 2008 16:31:36 +0100 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <80c99e790801300726g67acdb90yec3bb430b1cb6d8e@mail.gmail.com> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> <80c99e790801300726g67acdb90yec3bb430b1cb6d8e@mail.gmail.com> Message-ID: <80c99e790801300731j304938eaqe0271e0270a3e906@mail.gmail.com> or you can maybe use numpy.ix_: ax = [1,2] R[numpy.ix_(ax,ax)] = 100 hth, L. On 1/30/08, lorenzo bolla wrote: > > you simply need to change the definition of ax: > ax = slice(1,3) > > and all works fine. > L. > > On 1/30/08, Francesc Altet wrote: > > > > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > > In the following piece of code: > > > >>> import numpy as N > > > >>> R = N.arange(9).reshape(3,3) > > > >>> ax = [1,2] > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > >>> R[ax,:][:,ax] = 100 > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > Why R is not updated? > > > > Because R[ax] is not a view of R, but another copy of the original > > object (fancy indexing does return references to different objects). > > In order to get views, you must specify only a slice of the original > > array. For example: > > > > In [50]: S = R[::2] > > In [51]: S[:] = 2 > > In [52]: R > > Out[52]: > > array([[2, 2, 2], > > [3, 4, 5], > > [2, 2, 2]]) > > > > So, what you need is something like: > > > > In [68]: R = N.arange(9).reshape(3,3) > > In [69]: S = R[1:3,:][:,1:3] > > In [70]: S[:] = 2 > > In [71]: R > > Out[71]: > > array([[0, 1, 2], > > [3, 2, 2], > > [6, 2, 2]]) > > > > Cheers, > > > > -- > > >0,0< Francesc Altet http://www.carabos.com/ > > V V C?rabos Coop. V. Enjoy Data > > "-" > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Jan 30 11:30:16 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Jan 2008 11:30:16 -0500 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <20080130132054.GA27229@ssh.cv.nrao.edu> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> <20080130132054.GA27229@ssh.cv.nrao.edu> Message-ID: On 30/01/2008, Scott Ransom wrote: > That works fine with arrays, scalars, or array/scalar mixes in the > calling. I do understand that more complicated functions might > require vectorize(), however, I wonder if sometimes it is used > when it doesn't need to be? It certainly is! I would guess that it gets used for one of two reasons: * People don't realize that it's just a for loop and assume it does numpy magic to make things faster. and * People care more about convenience than speed. I am usually (when I use it) in the second camp. I have some function with a bunch of conditionals - patching up the phase of some inverse trig function, say, or dealing with a few special cases, or cleanly raising exceptions - and it matters more to get the thing written quickly than to have it run quickly. So rather than stop and figure out how to use where() or fancy indexing to handle it efficiently, I just slap a vectorize() on a simple python function. I can always accelerate it later. For the same reason, I often drop into list comprehensions for a few lines, converting back to an array at the end. IMHO, one of the primary values of numpy/scipy is that it lets you express vector algorithms conveniently. When this makes it faster, this is a bonus; when it makes it slower (as sometimes happens with very small or very large arrays) it's a pain. But vectorize() lets me use these vector expressions very readily with functions that are written in a simple python style. Nevertheless, I think (going by some of the questions we get here) a lot of people start using numpy without appreciating that the *point* of using it is that you can use vector expressions; they think, "well, I'm using numbers in python, so I should use numpy", and then they don't take advantage of numpy's speedups. So perhaps we should do some more educating, in tutorials and docstrings, about the limitations of vectorize(). Anyone want to start a wiki page "How to get rid of vectorize()"? Anne From tim.hochberg at ieee.org Wed Jan 30 11:57:40 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 30 Jan 2008 09:57:40 -0700 Subject: [Numpy-discussion] Median again In-Reply-To: <479FC96B.7030403@enthought.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> <479FB775.4080204@enthought.com> <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> <479FC96B.7030403@enthought.com> Message-ID: On Jan 29, 2008 5:48 PM, Travis E. Oliphant wrote: > Joris De Ridder wrote: > > On 30 Jan 2008, at 00:32, Travis E. Oliphant wrote: > > > > > >> Matthew Brett wrote: > >> > >>> Hi, > >>> > >>> > >>>>>>> median moved mediandim0 > >>>>>>> implementation of medianwithaxis or similar, with same call > >>>>>>> signature as mean. > >>>>>>> > >>>>>>> > >>> But - for the median function change - do we agree that this should > >>> be > >>> changed? I think it is a significant wart in the numpy API, and has > >>> caught quite a few people... > >>> > >>> > >> I'm fine with a median API change for 1.1. > >> > >> We can add the axis keyword for 1.0.5 as long as the default stays the > >> same. We can also add the other keywords as well if appropriate > >> defaults can be determined. > >> > > > > Do you mean creating a median(a, axis=0) for 1.0.5, and changing it to > > median(a,axis=None) for 1.1? (Modulo other keywords). > > > > Yes. That is the approach I prefer. I'm all for fixing this, but the prospect of going straight from one default to another makes me nervous. Is there is any prospect we could spit out a warning when an axis is not specified for median starting in 1.05 up till 1.1. It could even be a PendingDeprecationWarning, which by default doesn't print anything I believe, but would allow people to check there code for potential failure points. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 30 12:10:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2008 10:10:29 -0700 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <20080130092215.GA18502@phare.normalesup.org> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: On Jan 30, 2008 2:22 AM, Gael Varoquaux wrote: > On Wed, Jan 30, 2008 at 12:49:44AM -0800, LB wrote: > > My problem is that the complexe calculations made in calc_0d use some > > parameters, which are currently defined at the head of my python file. > > This is not very nice and I can't define a module containing theses > > two functions and call them with different parameters. > > > I would like to make this cleaner and pass theses parameter as > > keyword argument, but this don't seems to be possible with vectorize. > > Indeed, some of theses parameters are array parameters and only the x > > and y arguments should be interpreted with the broadcasting rules.... > > > What is the "good way" for doing this ? > > I don't know what the "good way" is, but you can always use functional > programming style (Oh, no, CaML is getting on me !): > > def calc_0d_params(param1, param2, param3): > def calc_0d(x, y): > # Here your code making use of param1, param2, param3) > ... > > return calc_0d(x, y) > > you call the function like this: > > calc_0d_params(param1, param2, param3)(x, y) > > To vectorize it you can do: > > calc_0d_vect = lambda *params: vectorize(calc_0d_params(*params)) > > This is untested code, but I hope you get the idea. It all about partial > evaluation of arguments. By the way, the parameters can now be keyword > arguments. > IIRC, the way to do closures in Python is something like In [5]: def factory(x) : ...: def f() : ...: print x ...: f.x = x ...: return f ...: In [6]: f = factory("Hello world.") In [7]: f() Hello world. There is a reason to do it that way, but I don't recall what it is. Nor do I know if the result can be vectorized, I've never used vectorize. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Wed Jan 30 12:18:09 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 30 Jan 2008 10:18:09 -0700 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: On Jan 30, 2008 10:10 AM, Charles R Harris wrote: [SNIP] > > IIRC, the way to do closures in Python is something like > > In [5]: def factory(x) : > ...: def f() : > ...: print x > ...: f.x = x > ...: return f > ...: > > In [6]: f = factory("Hello world.") > > In [7]: f() > Hello world. > > There is a reason to do it that way, but I don't recall what it is. > You don't need the "f.x = x" line. It's possible it has some beneficial side effect that I don't recall, but basic closures work fine without out. Nor do I know if the result can be vectorized, I've never used vectorize. > I expect they could be, but I don't use vectorize either. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 30 12:35:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2008 10:35:04 -0700 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: On Jan 30, 2008 10:10 AM, Charles R Harris wrote: > > > On Jan 30, 2008 2:22 AM, Gael Varoquaux > wrote: > > > On Wed, Jan 30, 2008 at 12:49:44AM -0800, LB wrote: > > > My problem is that the complexe calculations made in calc_0d use some > > > parameters, which are currently defined at the head of my python file. > > > This is not very nice and I can't define a module containing theses > > > two functions and call them with different parameters. > > > > > I would like to make this cleaner and pass theses parameter as > > > keyword argument, but this don't seems to be possible with vectorize. > > > Indeed, some of theses parameters are array parameters and only the x > > > and y arguments should be interpreted with the broadcasting rules.... > > > > > What is the "good way" for doing this ? > > > > I don't know what the "good way" is, but you can always use functional > > programming style (Oh, no, CaML is getting on me !): > > > > def calc_0d_params(param1, param2, param3): > > def calc_0d(x, y): > > # Here your code making use of param1, param2, param3) > > ... > > > > return calc_0d(x, y) > > > > you call the function like this: > > > > calc_0d_params(param1, param2, param3)(x, y) > > > > To vectorize it you can do: > > > > calc_0d_vect = lambda *params: vectorize(calc_0d_params(*params)) > > > > This is untested code, but I hope you get the idea. It all about partial > > evaluation of arguments. By the way, the parameters can now be keyword > > arguments. > > > > IIRC, the way to do closures in Python is something like > > In [5]: def factory(x) : > ...: def f() : > ...: print x > ...: f.x = x > ...: return f > ...: > Oops, looks like that needs to be: In [5]: def factory(x) : ...: def f() : ...: print f.x ...: f.x = x ...: return f ...: You can also do something simpler: In [51]: def f() : print f.x ....: In [52]: f.x = "Hello World!" In [53]: f() Hello World! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 30 13:16:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2008 11:16:39 -0700 Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: On Jan 30, 2008 10:18 AM, Timothy Hochberg wrote: > > > On Jan 30, 2008 10:10 AM, Charles R Harris > wrote: > > [SNIP] > > > > > > IIRC, the way to do closures in Python is something like > > > > In [5]: def factory(x) : > > ...: def f() : > > ...: print x > > ...: f.x = x > > ...: return f > > ...: > > > > In [6]: f = factory("Hello world.") > > > > In [7]: f() > > Hello world. > > > > There is a reason to do it that way, but I don't recall what it is. > > > > You don't need the "f.x = x" line. It's possible it has some beneficial > side effect that I don't recall, but basic closures work fine without out. > You're right. It looks like the context gets stored in f.func_closure. In [14]: f.func_closure[0].cell_contents Out[14]: 'Hello world.' Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Wed Jan 30 13:42:40 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 30 Jan 2008 20:42:40 +0200 Subject: [Numpy-discussion] Can not update a submatrix References: <1201700540.26483.16.camel@nadav.envision.co.il><200801301608.03192.faltet@carabos.com><80c99e790801300726g67acdb90yec3bb430b1cb6d8e@mail.gmail.com> <80c99e790801300731j304938eaqe0271e0270a3e906@mail.gmail.com> Message-ID: <710F2847B0018641891D9A21602763600B6EB1@ex3.envision.co.il> Thank you very much, that what I was looking for. Charles made a good point about offsets, counts and strides --- I really should go and reread the documentation. Nadav. -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of lorenzo bolla Sent: Wed 30-Jan-08 17:31 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Can not update a submatrix or you can maybe use numpy.ix_: ax = [1,2] R[numpy.ix_(ax,ax)] = 100 hth, L. On 1/30/08, lorenzo bolla wrote: > > you simply need to change the definition of ax: > ax = slice(1,3) > > and all works fine. > L. > > On 1/30/08, Francesc Altet wrote: > > > > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > > In the following piece of code: > > > >>> import numpy as N > > > >>> R = N.arange(9).reshape(3,3) > > > >>> ax = [1,2] > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > >>> R[ax,:][:,ax] = 100 > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > Why R is not updated? > > > > Because R[ax] is not a view of R, but another copy of the original > > object (fancy indexing does return references to different objects). > > In order to get views, you must specify only a slice of the original > > array. For example: > > > > In [50]: S = R[::2] > > In [51]: S[:] = 2 > > In [52]: R > > Out[52]: > > array([[2, 2, 2], > > [3, 4, 5], > > [2, 2, 2]]) > > > > So, what you need is something like: > > > > In [68]: R = N.arange(9).reshape(3,3) > > In [69]: S = R[1:3,:][:,1:3] > > In [70]: S[:] = 2 > > In [71]: R > > Out[71]: > > array([[0, 1, 2], > > [3, 2, 2], > > [6, 2, 2]]) > > > > Cheers, > > > > -- > > >0,0< Francesc Altet http://www.carabos.com/ > > V V C?rabos Coop. V. Enjoy Data > > "-" > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3798 bytes Desc: not available URL: From oliphant at enthought.com Wed Jan 30 13:41:29 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 30 Jan 2008 12:41:29 -0600 Subject: [Numpy-discussion] Median again In-Reply-To: References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> <479FB775.4080204@enthought.com> <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> <479FC96B.7030403@enthought.com> Message-ID: <47A0C4D9.4030308@enthought.com> Timothy Hochberg wrote: > > > On Jan 29, 2008 5:48 PM, Travis E. Oliphant > wrote: > > Joris De Ridder wrote: > > On 30 Jan 2008, at 00:32, Travis E. Oliphant wrote: > > > > > >> Matthew Brett wrote: > >> > >>> Hi, > >>> > >>> > >>>>>>> median moved mediandim0 > >>>>>>> implementation of medianwithaxis or similar, with same call > >>>>>>> signature as mean. > >>>>>>> > >>>>>>> > >>> But - for the median function change - do we agree that this > should > >>> be > >>> changed? I think it is a significant wart in the numpy API, > and has > >>> caught quite a few people... > >>> > >>> > >> I'm fine with a median API change for 1.1. > >> > >> We can add the axis keyword for 1.0.5 as long as the default > stays the > >> same. We can also add the other keywords as well if appropriate > >> defaults can be determined. > >> > > > > Do you mean creating a median(a, axis=0) for 1.0.5, and changing > it to > > median(a,axis=None) for 1.1? (Modulo other keywords). > > > > Yes. That is the approach I prefer. > > > I'm all for fixing this, but the prospect of going straight from one > default to another makes me nervous. Is there is any prospect we could > spit out a warning when an axis is not specified for median starting > in 1.05 up till 1.1. It could even be a PendingDeprecationWarning, > which by default doesn't print anything I believe, but would allow > people to check there code for potential failure points. Yes, we could start to do that (spit out a warning in 1.0.5). This should definitely be done in 1.0.6 Perhaps we use axis=None to start with and then check for that and spit out the warning (and change axis to 0). Thanks for the pointer to PendingDeprecationWarning. We could also have an APIChangeWarning that we define in NumPy which would allow SciPy to use it as well. -Travis From berthe.loic at gmail.com Wed Jan 30 13:50:24 2008 From: berthe.loic at gmail.com (LB) Date: Wed, 30 Jan 2008 10:50:24 -0800 (PST) Subject: [Numpy-discussion] Vectorizing a function In-Reply-To: <20080130092215.GA18502@phare.normalesup.org> References: <2d31d662-32e2-48c7-897f-7ca46e70aa2a@b2g2000hsg.googlegroups.com> <20080130092215.GA18502@phare.normalesup.org> Message-ID: Thank you Gael, I think this could work for my case. It will be a bit tricky, since calc_0d is already a closure in which I've defined a function : the parameters x and y are to main parameters of an ODE. So calc_0d define a function, integrate it sing scipy.integrate.odeint and returns some characteristics of the solution. Numpy.vectorize allows me to draw very easily map and contours of theses characteristics. So I will have a function, defining a function, defining another one... From peridot.faceted at gmail.com Wed Jan 30 14:43:06 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Jan 2008 14:43:06 -0500 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: <200801301608.03192.faltet@carabos.com> References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> Message-ID: On 30/01/2008, Francesc Altet wrote: > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > In the following piece of code: > > >>> import numpy as N > > >>> R = N.arange(9).reshape(3,3) > > >>> ax = [1,2] > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > >>> R[ax,:][:,ax] = 100 > > >>> R > > > > array([[0, 1, 2], > > [3, 4, 5], > > [6, 7, 8]]) > > > > Why R is not updated? > > Because R[ax] is not a view of R, but another copy of the original > object (fancy indexing does return references to different objects). > In order to get views, you must specify only a slice of the original > array. For example: This is not exactly correct. There are two kinds of fancy indexing in numpy, which behave similarly. There is normal fancy indexing, which produces a new array, not sharing data with the old array: a = N.random.normal(size=10) b = a[a>0] There is also fancy indexing of lvalues (to use a C term): a = N.random.normal(size=10) a[a<0] *= -1 Here no copy is made; instead, the data is modified in-place. This requires some clever hacks under the hood, since a[a<0] *can't* be a view of a. The problem the OP had is that they are going beyond what the clever hacks can deal with. That's why R[ax,:] = 100 works but R[ax,:][:,ax] = 100 doesn't. The problem is that you have two indexing operations in the second situation, and the inplace operators can't deal with that. Solutions include working on a flattened version of the array (for which a single indexing operation suffices), using the (not in-place) where() construct, or using the ix_ function: R[N.ix_(ax,ax)] = 100 N.ix_ is necessary because R[ax,ax] doesn't do what I, for one, would have expected: R[[1,2],[3,4]] yields [R[1,3],R[2,4]], not [[R[1,3],R[1,4]],[R[2,3],R[2,4]]]. But in any case, once you reduce it to a single fancy-indexing operation, you can successfully use it as an lvalue. Striding and views are not really the issue. Anne From tim.hochberg at ieee.org Wed Jan 30 15:35:50 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 30 Jan 2008 13:35:50 -0700 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> Message-ID: On Jan 30, 2008 12:43 PM, Anne Archibald wrote: > On 30/01/2008, Francesc Altet wrote: > > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > > In the following piece of code: > > > >>> import numpy as N > > > >>> R = N.arange(9).reshape(3,3) > > > >>> ax = [1,2] > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > >>> R[ax,:][:,ax] = 100 > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > Why R is not updated? > > > > Because R[ax] is not a view of R, but another copy of the original > > object (fancy indexing does return references to different objects). > > In order to get views, you must specify only a slice of the original > > array. For example: > > This is not exactly correct. > [...a fine explanation by Anne...] Another way to look at this is note that: R[ax,:][:,ax] = 100 is translated to: R.__getitem__((ax, :)).__setitem__((:,ax), 100) '__setitem__' is capable of setting values using fancy indexing, but as has been noted, '__getitem__' makes a copy of R when using fancy indexing, and therefore the original does not get modified. [Technically, ':' gets translated to slice(None,None,None), but I didn't want to write that out]. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Wed Jan 30 18:56:01 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 30 Jan 2008 15:56:01 -0800 Subject: [Numpy-discussion] Median again In-Reply-To: <47A0C4D9.4030308@enthought.com> References: <1e2af89e0801290804x5d73f794sa940daf4417c5e5c@mail.gmail.com> <7D40E37F-91B3-4674-8E03-EDAD0DF80C85@ster.kuleuven.be> <449C3A13-061F-4369-9626-1E72EB84431B@gmail.com> <1e2af89e0801291331kbc68772i50027d34f3e1555@mail.gmail.com> <1e2af89e0801291443s4ad7e1dcqff78e32bbac9306b@mail.gmail.com> <479FB775.4080204@enthought.com> <165E4956-A110-417E-BCD1-621149679651@ster.kuleuven.be> <479FC96B.7030403@enthought.com> <47A0C4D9.4030308@enthought.com> Message-ID: On Jan 30, 2008 10:41 AM, Travis E. Oliphant wrote: > Yes, we could start to do that (spit out a warning in 1.0.5). This > should definitely be done in 1.0.6 > > Perhaps we use axis=None to start with and then check for that and spit > out the warning (and change axis to 0). Thanks for the pointer to > PendingDeprecationWarning. > > We could also have an APIChangeWarning that we define in NumPy which > would allow SciPy to use it as well. +1 I think this sounds very reasonable and will fix a major NumPy wart. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From cournape at gmail.com Thu Jan 31 01:25:26 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 31 Jan 2008 15:25:26 +0900 Subject: [Numpy-discussion] Visual studio with mingw g77 built libraries Message-ID: <5b8d13220801302225h3bbb76a4x87df63379a7afbd0@mail.gmail.com> Hi, I would like to ask some details about the build process of numpy when visual studio is used for C compilation, and g77 for blas/lapack. As I understand it, the current situation consists in using libraries built the "Unix way" (e.g. libblas.a, static library built with ar + ranlib), taking the gcc runtimes on the fly (libgcc and g2c), and put everything together to feed VS compiler. I wonder if we should not use another way, that is use libraries built the "windows way" (dll + lib). I think it is more natural (and would also make my life easier for numscons): - it would means that on windows, all compilers combination would follow a similar process. - dll and lib are more natural for windows developers (who are the ones using VS in the first place). The only difference for users would be in the way blas/lapack should be built. Is there a reason why this was not the chosen path ? Is it because it would have been too difficult to do it with distutils (using dlltool and co) ? Or other reasons I am not aware of ? cheers, David From nadavh at visionsense.com Thu Jan 31 06:21:31 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 31 Jan 2008 13:21:31 +0200 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: References: <1201700540.26483.16.camel@nadav.envision.co.il> <200801301608.03192.faltet@carabos.com> Message-ID: <1201778491.1737.5.camel@nadav.envision.co.il> Thanks for your explanation. It explains also the following: >>> R = N.arange(9).reshape(3,3) >>> ax = [0,2] >>> U = R[ax,:] >>> U array([[0, 1, 2], [6, 7, 8]]) >>> U[:] = 100 >>> U array([[100, 100, 100], [100, 100, 100]]) >>> R # No change since U is a copy array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> R[ax,:] = 100 # Works through __setitem__ >>> R array([[100, 100, 100], [ 3, 4, 5], [100, 100, 100]]) Nadav On Wed, 2008-01-30 at 13:35 -0700, Timothy Hochberg wrote: > > > > On Jan 30, 2008 12:43 PM, Anne Archibald > wrote: > > On 30/01/2008, Francesc Altet wrote: > > A Wednesday 30 January 2008, Nadav Horesh escrigu?: > > > In the following piece of code: > > > >>> import numpy as N > > > >>> R = N.arange(9).reshape(3,3) > > > >>> ax = [1,2] > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > >>> R[ax,:][:,ax] = 100 > > > >>> R > > > > > > array([[0, 1, 2], > > > [3, 4, 5], > > > [6, 7, 8]]) > > > > > > Why R is not updated? > > > > Because R[ax] is not a view of R, but another copy of the > original > > object (fancy indexing does return references to different > objects). > > In order to get views, you must specify only a slice of the > original > > array. For example: > > > > This is not exactly correct. > > > [...a fine explanation by Anne...] > > Another way to look at this is note that: > > > R[ax,:][:,ax] = 100 > > > > is translated to: > > > > R.__getitem__((ax, :)).__setitem__((:,ax), 100) > > > > '__setitem__' is capable of setting values using fancy indexing, but > as has been noted, '__getitem__' makes a copy of R when using fancy > indexing, and therefore the original does not get modified. > > [Technically, ':' gets translated to slice(None,None,None), but I > didn't want to write that out]. > > -- > . __ > . |-\ > . > . tim.hochberg at ieee.org > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Jan 31 07:19:24 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 31 Jan 2008 07:19:24 -0500 Subject: [Numpy-discussion] blcr checkpoint/restart Message-ID: Hi numeric processing fans. I'm pleased to report that you can now have convenient checkpoint/restart, at least if you are running fedora linux. Berkeley Lab Checkpoint/Restart for Linux (BLCR) http://ftg.lbl.gov/CheckpointRestart/CheckpointDownloads.shtml What I've done is: 1) built 2 rpm packages for blcr. First installs everything except kernel module. 2nd is the kernel module, packaged for use with akmods, which will be (I think) the kmod format of choice in rpmfusion. 2) made a python ctypes module to use it. This allows your python program to checkpoint itself. You will need a couple of packages from livna development to use this: rpm -q --whatprovides kmodtool kmodtool-1-7.fc8.noarch rpm -q --whatprovides akmods akmods-0.2.1-1.fc8.noarch You can find this stuff here: http://livna-dl.reloumirrors.net/fedora/development/SRPMS/ I grabbed the development srpms from livna and built them for my F8 machine. My stuff is here: https://nbecker.dyndns.org/RPM/blcr_mod.py https://nbecker.dyndns.org/RPM/blcr-0.6.4-1.src.rpm https://nbecker.dyndns.org/RPM/blcr-kmod-0.6.4-2.fc8.src.rpm From faltet at carabos.com Thu Jan 31 07:23:36 2008 From: faltet at carabos.com (Francesc Altet) Date: Thu, 31 Jan 2008 13:23:36 +0100 Subject: [Numpy-discussion] Can not update a submatrix In-Reply-To: References: <1201700540.26483.16.camel@nadav.envision.co.il> Message-ID: <200801311323.36744.faltet@carabos.com> A Wednesday 30 January 2008, Timothy Hochberg escrigu?: > [...a fine explanation by Anne and Timothy...] Ok. As it seems that this subject has interest enough, I went ahead and created a small document about views vs copies at: http://www.scipy.org/Cookbook/ViewsVsCopies I think it resumes what has been said in this thread, but feel free to update it if you find some inaccuracy or want to complete it more. Although perhaps this is more a FAQ than a recipe, I've added it to the SciPy cookbook because I find it a bit long for a FAQ entry (but I may be wrong). In fact, I feel that the cookbook would need some reorganization, because there are too many different subjects in NumPy/SciPy section of http://www.scipy.org/Cookbook. One first action, IMHO, would be to create separate NumPy/SciPy FAQs and Cookbooks. It's my impression that NumPy is a much more general tool than SciPy with much less requeriments to install, test and use it, and having such a monolithic merge of NumPy and SciPy cookbok could discourage many potential NumPy users that might see NumPy as 'too scientific'. After that, I'd introduce a new subsection called 'Advanced topics' in both NumPy & SciPy FAQ/Cookbook. For example, currently there exists things like "SphericalBesselZeros", "Savitzky Golay filtering of data" in the first section of the cookbook, which is clearly a bit overhelming for the naive user. I'd like to hear some comments about this proposal from the community. At any rate, I've placed the "ViewsVsCopies" entry under a new section that I created in the cookbook, that I called "Advanced topics", but I'm open to suggestions on a better name/place. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From philbinj at gmail.com Thu Jan 31 09:35:25 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 14:35:25 +0000 Subject: [Numpy-discussion] searchsorted bug Message-ID: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> Hi, The following gives the wrong answer: In [2]: A = array(['a','aa','b']) In [3]: B = array(['d','e']) In [4]: A.searchsorted(B) Out[4]: array([3, 0]) The answer should be [3,3]. I've come across this while trying to come up with an ismember function which works for strings (setmember1d doesn't seems to assume numerical arrays). Thanks, James From lbolla at gmail.com Thu Jan 31 09:51:58 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Thu, 31 Jan 2008 15:51:58 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> Message-ID: <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> works fine for me. In [33]: A = numpy.array(['a','aa','b']) In [34]: B = numpy.array(['d','e']) In [35]: A.searchsorted(B) Out[35]: array([3, 3]) In [36]: numpy.__version__ Out[36]: '1.0.5.dev4567' L. On 1/31/08, James Philbin wrote: > > Hi, > > The following gives the wrong answer: > > In [2]: A = array(['a','aa','b']) > > In [3]: B = array(['d','e']) > > In [4]: A.searchsorted(B) > Out[4]: array([3, 0]) > > The answer should be [3,3]. I've come across this while trying to come > up with an ismember function which works for strings (setmember1d > doesn't seems to assume numerical arrays). > > Thanks, > James > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From philbinj at gmail.com Thu Jan 31 09:53:20 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 14:53:20 +0000 Subject: [Numpy-discussion] [numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> Message-ID: <2b1c8c4f0801310653t4a526336vac664704d995c4d0@mail.gmail.com> Hi, The following gives the wrong answer: In [2]: A = array(['a','aa','b']) In [3]: B = array(['d','e']) In [4]: A.searchsorted(B) Out[4]: array([3, 0]) The answer should be [3,3]. I've come across this while trying to come up with an ismember function which works for strings (setmember1d doesn't seems to assume numerical arrays). Thanks, James From philbinj at gmail.com Thu Jan 31 09:55:15 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 14:55:15 +0000 Subject: [Numpy-discussion] [numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310653t4a526336vac664704d995c4d0@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <2b1c8c4f0801310653t4a526336vac664704d995c4d0@mail.gmail.com> Message-ID: <2b1c8c4f0801310655h7cf8033ct69983ad9fa4d2b@mail.gmail.com> Hi, OK, i'm using: In [6]: numpy.__version__ Out[6]: '1.0.3' Should I try the development version? Which version of numpy would people generally recommend? James From nadavh at visionsense.com Thu Jan 31 09:56:56 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 31 Jan 2008 16:56:56 +0200 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> Message-ID: <1201791416.5786.0.camel@nadav.envision.co.il> It works fine also for me (numpy 1.04 gentoo linux on amd64) Nadav On Thu, 2008-01-31 at 15:51 +0100, lorenzo bolla wrote: > works fine for me. > > > In [33]: A = numpy.array(['a','aa','b']) > > In [34]: B = numpy.array(['d','e']) > > In [35]: A.searchsorted(B) > Out[35]: array([3, 3]) > > In [36]: numpy.__version__ > Out[36]: '1.0.5.dev4567' > > > > > > L. > > > > > On 1/31/08, James Philbin wrote: > > Hi, > > The following gives the wrong answer: > > In [2]: A = array(['a','aa','b']) > > In [3]: B = array(['d','e']) > > In [4]: A.searchsorted(B) > Out[4]: array([3, 0]) > > The answer should be [3,3]. I've come across this while trying > to come > up with an ismember function which works for strings > (setmember1d > doesn't seems to assume numerical arrays). > > Thanks, > James > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From philbinj at gmail.com Thu Jan 31 10:01:11 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 15:01:11 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <1201791416.5786.0.camel@nadav.envision.co.il> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> Message-ID: <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> Hmmm. Just downloaded and installed 1.0.4 and i'm still getting this error. Are you guys using the bleeding edge version or the official 1.0.4 tarball from the webpage? James From philbinj at gmail.com Thu Jan 31 10:10:25 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 15:10:25 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> Message-ID: <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> Hi, Just tried with numpy from svn and still get this problem: >>> import numpy >>> numpy.__version__ '1.0.5.dev4763' >>> A = numpy.array(['a','aa','b']) >>> B = numpy.array(['d','e']) >>> A.searchsorted(B) array([3, 0]) I guess this must be a platform-dependent bug. I'm running python version: Python 2.5 (r25:51908, Nov 6 2007, 15:55:44) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 I'm on an Intel Xeon E5345. Any help would be much appreciated. Thanks, James From lbolla at gmail.com Thu Jan 31 10:10:50 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Thu, 31 Jan 2008 16:10:50 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> Message-ID: <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> I use a dev version (1.0.5.dev4567). L. On 1/31/08, James Philbin wrote: > > Hmmm. Just downloaded and installed 1.0.4 and i'm still getting this > error. Are you guys using the bleeding edge version or the official > 1.0.4 tarball from the webpage? > > James > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Jan 31 10:17:01 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 31 Jan 2008 16:17:01 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> Message-ID: No problem for me (also a svn version) : Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 >>> import numpy >>> A = numpy.array(['a','aa','b']) >>> B = numpy.array(['d','e']) >>> A.searchsorted(B) array([3, 3]) Matthieu 2008/1/31, lorenzo bolla : > > I use a dev version (1.0.5.dev4567). > L. > > > On 1/31/08, James Philbin wrote: > > > > Hmmm. Just downloaded and installed 1.0.4 and i'm still getting this > > error. Are you guys using the bleeding edge version or the official > > 1.0.4 tarball from the webpage? > > > > James > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Thu Jan 31 10:23:08 2008 From: robince at gmail.com (Robin) Date: Thu, 31 Jan 2008 15:23:08 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> Message-ID: I do get the problem with a recent(ish) svn, on OS X 10.5.1, python 2.5.1 (from python.org): In [76]: A = array(['a','aa','b']) In [77]: B = array(['d','e']) In [78]: A.searchsorted(B) Out[78]: array([3, 0]) In [79]: numpy.__version__ Out[79]: '1.0.5.dev4722' From dmitrey.kroshko at scipy.org Thu Jan 31 10:35:24 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 31 Jan 2008 17:35:24 +0200 Subject: [Numpy-discussion] matrix <-> ndarray bug Message-ID: <47A1EABC.40500@scipy.org> I don't know, maybe it's already fixed in more recent versions? >>> from numpy import * >>> a=mat('1 2') >>> b = asfarray(a).flatten() >>> print b[0] [[ 1. 2.]] # ^^ I expected getting 1.0 here >>> numpy.version.version '1.0.3' From bsouthey at gmail.com Thu Jan 31 10:41:39 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 31 Jan 2008 09:41:39 -0600 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> Message-ID: Hi, With my system running x86_64 SUSE10.0 AMD opteron: Under Python 2.5.1 (Python 2.5.1 -r251:54863) and numpy 1.0.4 (download of released version) I have the same bug. Under Python 2.4.1 (May 25 2007, 18:41:31) and numpy 1.0.3 I have no problem. Perhaps a 32/64 bit problem? Bruce On Jan 31, 2008 9:17 AM, Matthieu Brucher wrote: > No problem for me (also a svn version) : > > Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11) > [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 > > >>> import numpy > > >>> A = numpy.array(['a','aa','b']) > >>> B = numpy.array(['d','e']) > >>> A.searchsorted(B) > array([3, 3]) > > Matthieu > > 2008/1/31, lorenzo bolla : > > > > I use a dev version (1.0.5.dev4567). > > L. > > > > > > On 1/31/08, James Philbin wrote: > > > > > Hmmm. Just downloaded and installed 1.0.4 and i'm still getting this > > > error. Are you guys using the bleeding edge version or the official > > > 1.0.4 tarball from the webpage? > > > > > > James > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > -- > > Lorenzo Bolla > > lbolla at gmail.com > > http://lorenzobolla.emurse.com/ > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From meine at informatik.uni-hamburg.de Thu Jan 31 10:45:03 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 31 Jan 2008 16:45:03 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> Message-ID: <200801311645.03808.meine@informatik.uni-hamburg.de> Am Donnerstag, 31. Januar 2008 15:35:25 schrieb James Philbin: > The following gives the wrong answer: > > In [2]: A = array(['a','aa','b']) > > In [3]: B = array(['d','e']) > > In [4]: A.searchsorted(B) > Out[4]: array([3, 0]) > > The answer should be [3,3]. Heh, I got both answers in the same session (not 100% reproducable, but several times already)! In [1]: %cpaste Pasting code; enter '--' alone on the line to stop. :>>> import numpy :>>> A = numpy.array(['a','aa','b']) :>>> B = numpy.array(['d','e']) :>>> print A.searchsorted(B) :>>> print numpy.__version__ :-- [3 3] 1.0.5.dev4420 In [2]: In [3]: print A.searchsorted(B) [3 3] In [4]: print A.searchsorted(B) [3 3] In [5]: print A.searchsorted(B) [3 3] In [6]: %cpaste Pasting code; enter '--' alone on the line to stop. :>>> import numpy :>>> A = numpy.array(['a','aa','b']) :>>> B = numpy.array(['d','e']) :>>> print A.searchsorted(B) :>>> print numpy.__version__ :-- [3 0] 1.0.5.dev4420 -- Ciao, / / /--/ / / ANS From aisaac at american.edu Thu Jan 31 10:55:41 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 31 Jan 2008 10:55:41 -0500 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com><80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com><1201791416.5786.0.camel@nadav.envision.co.il><2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com><2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> Message-ID: Problem also with Windows P3 binaries. fwiw, Alan Isaac Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.0.4' >>> A = numpy.array(['a','aa','b']) >>> B = numpy.array(['d','e']) >>> A.searchsorted(B) array([3, 0]) >>> From aisaac at american.edu Thu Jan 31 11:05:02 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 31 Jan 2008 11:05:02 -0500 Subject: [Numpy-discussion] matrix <-> ndarray bug In-Reply-To: <47A1EABC.40500@scipy.org> References: <47A1EABC.40500@scipy.org> Message-ID: On Thu, 31 Jan 2008, dmitrey apparently wrote: > already fixed in more recent versions? Yes, at least it's fixed in my 1.0.4. By the way, do you know about the ``A`` attribute of matrices? Cheers, Alan Isaac From lbolla at gmail.com Thu Jan 31 11:17:56 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Thu, 31 Jan 2008 17:17:56 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> Message-ID: <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> from docstring in multiarraymodule.c /** @brief Use bisection of sorted array to find first entries >= keys. * * For each key use bisection to find the first index i s.t. key <= arr[i]. * When there is no such index i, set i = len(arr). Return the results in ret. * All arrays are assumed contiguous on entry and both arr and key must be of <----- * the same comparable type. <----- * * @param arr contiguous sorted array to be searched. * @param key contiguous array of keys. * @param ret contiguous array of intp for returned indices. * @return void */ static void local_search_left(PyArrayObject *arr, PyArrayObject *key, PyArrayObject *ret) In particular: * All arrays are assumed contiguous on entry and both arr and key must be of <----- * the same comparable type. <----- A and B are not of the same type ('|S2' is not '|S1'). This should be mentioned somewhere more accessible. L. On 1/31/08, Alan G Isaac wrote: > > Problem also with Windows P3 binaries. > fwiw, > Alan Isaac > > Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 > 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.__version__ > '1.0.4' > >>> A = numpy.array(['a','aa','b']) > >>> B = numpy.array(['d','e']) > >>> A.searchsorted(B) > array([3, 0]) > >>> > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Thu Jan 31 11:44:31 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 31 Jan 2008 18:44:31 +0200 Subject: [Numpy-discussion] matrix <-> ndarray bug In-Reply-To: References: <47A1EABC.40500@scipy.org> Message-ID: <47A1FAEF.7040707@scipy.org> Alan G Isaac wrote: > On Thu, 31 Jan 2008, dmitrey apparently wrote: > >> already fixed in more recent versions? >> > > Yes, at least it's fixed in my 1.0.4. > > By the way, do you know about the ``A`` attribute of matrices? > Yes, I know, but numpy.ndarray has no the attribute (I had said it's better to be present, as well as .M for matrices, but numpy developers hadn't paid attention), and since I don't know does user pass ndarray or matrix to a func, it's better using asfarray(a) than a.A (later will fail on ndarrays) Regards, D. From lbolla at gmail.com Thu Jan 31 10:41:49 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Thu, 31 Jan 2008 16:41:49 +0100 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <80c99e790801310710j5efdaaebxdd95cb415c67411e@mail.gmail.com> Message-ID: <80c99e790801310741q7ff96823x606514be4a22880c@mail.gmail.com> oops. it fails also on an SGI Altix with Suse Linux on it: Linux pico 2.6.16.27-0.9-default #1 SMP Tue Feb 13 09:35:18 UTC 2007 ia64 ia64 ia64 GNU/Linux -------------------------------------------- In [33]: A = numpy.array(['a','aa','b']) In [34]: B = numpy.array(['d','e']) In [35]: A.searchsorted(B) Out[35]: array([3, 0]) In [36]: numpy.__version__ Out[36]: '1.0.5.dev4567' -------------------------------------------- The problem seems to be in the different dtypes of A and B. Using the same dtype '|S2' it works fine. -------------------------------------------- In [38]: A = numpy.array(['a','aa','b']) In [39]: A.dtype Out[39]: dtype('|S2') In [40]: B = numpy.array(['d','e']) In [41]: A.searchsorted(B) Out[41]: array([3, 0]) In [42]: B = numpy.array(['d','e'], dtype='|S2') In [43]: A.searchsorted(B) Out[43]: array([3, 3]) -------------------------------------------- L. On 1/31/08, Matthieu Brucher wrote: > > No problem for me (also a svn version) : > > Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11) > [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 > > >>> import numpy > >>> A = numpy.array(['a','aa','b']) > >>> B = numpy.array(['d','e']) > >>> A.searchsorted(B) > array([3, 3]) > > Matthieu > > 2008/1/31, lorenzo bolla : > > > > I use a dev version (1.0.5.dev4567). > > L. > > > > > > On 1/31/08, James Philbin wrote: > > > > > > Hmmm. Just downloaded and installed 1.0.4 and i'm still getting this > > > error. Are you guys using the bleeding edge version or the official > > > 1.0.4 tarball from the webpage? > > > > > > James > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > -- > > Lorenzo Bolla > > lbolla at gmail.com > > http://lorenzobolla.emurse.com/ > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mack at trdlnk.com Thu Jan 31 12:31:44 2008 From: mack at trdlnk.com (Mack Smith) Date: Thu, 31 Jan 2008 11:31:44 -0600 Subject: [Numpy-discussion] Accepted Numpy Core Datatype Formats Message-ID: <47A20600.5010504@trdlnk.com> Where can I find a list of accepted tokens for the Numpy Core formats as used in numpy.core.rec.fromfile for instance. I'm working with a binary file format that includes single byte buffers and other formats that I need to correctly parse. Any help would be appreciated. Thanks in advance. Mack From Chris.Barker at noaa.gov Thu Jan 31 12:33:25 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 31 Jan 2008 09:33:25 -0800 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> Message-ID: <47A20665.7080505@noaa.gov> lorenzo bolla wrote: > * All arrays are assumed contiguous on entry and both arr and key must > be of <----- > * the same comparable type. <----- > > A and B are not of the same type ('|S2' is not '|S1'). > This should be mentioned somewhere more accessible. It should also raise an exception -- something that *sometimes* works, and sometimes doesn't is really asking for trouble! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From philbinj at gmail.com Thu Jan 31 12:33:08 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 17:33:08 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> Message-ID: <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> Hi, > In particular: > > * All arrays are assumed contiguous on entry and both arr and key must be > of <----- > * the same comparable type. <----- In which case, this seems to be an overly strict implementation of searchsorted. Surely all that should be required is that the comparison function can take both types. James From charlesr.harris at gmail.com Thu Jan 31 12:38:01 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Jan 2008 10:38:01 -0700 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> Message-ID: On Jan 31, 2008 9:17 AM, lorenzo bolla wrote: > from docstring in multiarraymodule.c > > /** @brief Use bisection of sorted array to find first entries >= keys. > * > * For each key use bisection to find the first index i s.t. key <= > arr[i]. > * When there is no such index i, set i = len(arr). Return the results in > ret. > * All arrays are assumed contiguous on entry and both arr and key must be > of <----- > * the same comparable type. <----- > * > * @param arr contiguous sorted array to be searched. > * @param key contiguous array of keys. > * @param ret contiguous array of intp for returned indices. > * @return void > */ > static void > local_search_left(PyArrayObject *arr, PyArrayObject *key, PyArrayObject > *ret) > > In particular: > > * All arrays are assumed contiguous on entry and both arr and key must be > of <----- > * the same comparable type. <----- > > Heh. I knew there was a reason I documented that subroutine, I had forgotten about that. Anyway, I think an exception should be thrown in the higher level routine that sets up the call when there is a type mismatch. Some typecasting might also be appropriate. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From philbinj at gmail.com Thu Jan 31 12:49:11 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 17:49:11 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> Message-ID: <2b1c8c4f0801310949q31a80ea4x4fbbf5d15400ba0f@mail.gmail.com> Well, i've digged around in the source code and here is a patch which makes it work for the case I wanted: --- multiarraymodule.c.old 2008-01-31 17:42:32.000000000 +0000 +++ multiarraymodule.c 2008-01-31 17:43:43.000000000 +0000 @@ -2967,7 +2967,10 @@ char *parr = arr->data; char *pkey = key->data; intp *pret = (intp *)ret->data; - int elsize = arr->descr->elsize; + + int elsize1 = arr->descr->elsize; + int elsize2 = key->descr->elsize; + intp i; for(i = 0; i < nkeys; ++i) { @@ -2975,14 +2978,14 @@ intp imax = nelts; while (imin < imax) { intp imid = imin + ((imax - imin) >> 2); - if (compare(parr + elsize*imid, pkey, key) < 0) + if (compare(parr + elsize1*imid, pkey, key) < 0) imin = imid + 1; else imax = imid; } *pret = imin; pret += 1; - pkey += elsize; + pkey += elsize2; } } @@ -3008,7 +3011,10 @@ char *parr = arr->data; char *pkey = key->data; intp *pret = (intp *)ret->data; - int elsize = arr->descr->elsize; + + int elsize1 = arr->descr->elsize; + int elsize2 = key->descr->elsize; + intp i; for(i = 0; i < nkeys; ++i) { @@ -3016,14 +3022,14 @@ intp imax = nelts; while (imin < imax) { intp imid = imin + ((imax - imin) >> 2); - if (compare(parr + elsize*imid, pkey, key) <= 0) + if (compare(parr + elsize1*imid, pkey, key) <= 0) imin = imid + 1; else imax = imid; } *pret = imin; pret += 1; - pkey += elsize; + pkey += elsize2; } } James From charlesr.harris at gmail.com Thu Jan 31 12:50:40 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Jan 2008 10:50:40 -0700 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> Message-ID: On Jan 31, 2008 10:33 AM, James Philbin wrote: > Hi, > > > In particular: > > > > * All arrays are assumed contiguous on entry and both arr and key must > be > > of <----- > > * the same comparable type. <----- > In which case, this seems to be an overly strict implementation of > searchsorted. Surely all that should be required is that the > comparison function can take both types. > True. The problem is knowing when that is the case. The subroutine in question is at the bottom of the heap and don't know nothin'. IIRC, it just sits there and does the comparison by calling through a pointer with char* arguments. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From philbinj at gmail.com Thu Jan 31 12:55:18 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 17:55:18 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> Message-ID: <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> > True. The problem is knowing when that is the case. The subroutine in > question is at the bottom of the heap and don't know nothin'. IIRC, it just > sits there and does the comparison by calling through a pointer with char* > arguments. What does the comparison function actually look like for the case of dtype='|Sn'? Is there no way of sending the underlying types to the comparison, so it can throw an exception if the two data types are not supported? James From charlesr.harris at gmail.com Thu Jan 31 12:57:17 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Jan 2008 10:57:17 -0700 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310949q31a80ea4x4fbbf5d15400ba0f@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310949q31a80ea4x4fbbf5d15400ba0f@mail.gmail.com> Message-ID: On Jan 31, 2008 10:49 AM, James Philbin wrote: > Well, i've digged around in the source code and here is a patch which > makes it work for the case I wanted: > > --- multiarraymodule.c.old 2008-01-31 17:42:32.000000000 +0000 > +++ multiarraymodule.c 2008-01-31 17:43:43.000000000 +0000 > @@ -2967,7 +2967,10 @@ > char *parr = arr->data; > char *pkey = key->data; > intp *pret = (intp *)ret->data; > - int elsize = arr->descr->elsize; > + > + int elsize1 = arr->descr->elsize; > + int elsize2 = key->descr->elsize; > + > intp i; > > for(i = 0; i < nkeys; ++i) { > @@ -2975,14 +2978,14 @@ > intp imax = nelts; > while (imin < imax) { > intp imid = imin + ((imax - imin) >> 2); > - if (compare(parr + elsize*imid, pkey, key) < 0) > + if (compare(parr + elsize1*imid, pkey, key) < 0) > imin = imid + 1; > else > imax = imid; > } > *pret = imin; > pret += 1; > - pkey += elsize; > + pkey += elsize2; > } > } > But is that safe? You have changed the stepping to adjust for the element size, but there is no guarantee that the comparison works. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jan 31 13:02:54 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Jan 2008 11:02:54 -0700 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <80c99e790801310651p190e77c7n108df61afb6084b6@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> Message-ID: On Jan 31, 2008 10:55 AM, James Philbin wrote: > > True. The problem is knowing when that is the case. The subroutine in > > question is at the bottom of the heap and don't know nothin'. IIRC, it > just > > sits there and does the comparison by calling through a pointer with > char* > > arguments. > > What does the comparison function actually look like for the case of > dtype='|Sn'? Is there no way of sending the underlying types to the > comparison, so it can throw an exception if the two data types are not > supported? > There is an upper level routine that parses the keywords and sets things up. There may even be two upper level routines, but I don't recall. The purpose of the two routines you touched was to pull out a small block of code that could be very simple because of its assumptions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From philbinj at gmail.com Thu Jan 31 13:41:00 2008 From: philbinj at gmail.com (James Philbin) Date: Thu, 31 Jan 2008 18:41:00 +0000 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> Message-ID: <2b1c8c4f0801311041t25c2f09ch398e5a6f62af33f6@mail.gmail.com> I can't fathom where the comparison functions exist in the code. It seems that the comparison signature is of the form (void*, void*, PyArrayObject*), so it doesn't seem possible at the moment to specify a compare function which can reason about the underlying types of the two void*'s. However, I think arrays of strings are a common enough use case that they should work as expected - would it be possible to extend the comparison type to accept two integers specifying the types of the arguments? James On Jan 31, 2008 6:02 PM, Charles R Harris wrote: > > > > > On Jan 31, 2008 10:55 AM, James Philbin wrote: > > > > > True. The problem is knowing when that is the case. The subroutine in > > > question is at the bottom of the heap and don't know nothin'. IIRC, it > just > > > sits there and does the comparison by calling through a pointer with > char* > > > arguments. > > > > What does the comparison function actually look like for the case of > > dtype='|Sn'? Is there no way of sending the underlying types to the > > comparison, so it can throw an exception if the two data types are not > > supported? > > > > > > > > > > There is an upper level routine that parses the keywords and sets things up. > There may even be two upper level routines, but I don't recall. The purpose > of the two routines you touched was to pull out a small block of code that > could be very simple because of its assumptions. > > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From oliphant at enthought.com Thu Jan 31 13:53:42 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 31 Jan 2008 12:53:42 -0600 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801311041t25c2f09ch398e5a6f62af33f6@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> <2b1c8c4f0801311041t25c2f09ch398e5a6f62af33f6@mail.gmail.com> Message-ID: <47A21936.7010504@enthought.com> James Philbin wrote: > I can't fathom where the comparison functions exist in the code. It > seems that the comparison signature is of the form (void*, void*, > PyArrayObject*), so it doesn't seem possible at the moment to specify > a compare function which can reason about the underlying types of the > two void*'s. However, I think arrays of strings are a common enough > use case that they should work as expected - would it be possible to > extend the comparison type to accept two integers specifying the types > of the arguments? > The problem is due to the use of an older API in type conversion. I think I can provide a fix in a few minutes. The compare function is type-specific and works for strings but requires the same length string for each argument. It is defined as part of the PyArray_Descr object (defined in arraytypes.inc.src). It may be possible to not require contiguous arrays, but that is a separate issue. -Travis O. From oliphant at enthought.com Thu Jan 31 14:52:14 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 31 Jan 2008 13:52:14 -0600 Subject: [Numpy-discussion] searchsorted bug In-Reply-To: <2b1c8c4f0801311041t25c2f09ch398e5a6f62af33f6@mail.gmail.com> References: <2b1c8c4f0801310635y58700fe3n399d22311bc0441c@mail.gmail.com> <1201791416.5786.0.camel@nadav.envision.co.il> <2b1c8c4f0801310701y4c2d29d3j939c0f1e84c4d44e@mail.gmail.com> <2b1c8c4f0801310710lb6dd039r9101d89f6b76878f@mail.gmail.com> <80c99e790801310817n69680f86u1ec371f4f89ab8c5@mail.gmail.com> <2b1c8c4f0801310933u48d92103q6924fa977078ecd6@mail.gmail.com> <2b1c8c4f0801310955wf3af941g6c88354c813b503e@mail.gmail.com> <2b1c8c4f0801311041t25c2f09ch398e5a6f62af33f6@mail.gmail.com> Message-ID: <47A226EE.2050705@enthought.com> James Philbin wrote: > I can't fathom where the comparison functions exist in the code. It > seems that the comparison signature is of the form (void*, void*, > PyArrayObject*), so it doesn't seem possible at the moment to specify > a compare function which can reason about the underlying types of the > two void*'s. However, I think arrays of strings are a common enough > use case that they should work as expected - would it be possible to > extend the comparison type to accept two integers specifying the types > of the arguments? > Try out latest SVN. It should have this problem fixed. Look at the diff to see the fix (note that copies are made if the types are not "exactly" the same --- meaning the same length if variable types). -Travis O.