Mailman 3 January 2004 - NumPy-Discussion

RE: [Numpy-discussion] efficient sum of "sparse" 2D arrays?
by Kuzminski, Stefan R Jan. 25, 2004

Jan. 25, 2004

Could you use masked arrays more efficiently in this case? If you create the array so that values >255 and <0 are masked, then they will be excluded from the sum ( and from any other operations as well ). Stefan -----Original Message----- From: numpy-discussion-admin(a)lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net] On Behalf Of Konrad Hinsen Sent: Monday, January 26, 2004 12:17 AM To: RJS Cc: numpy-discussion(a)lists.sourceforge.net Subject: Re: [Numpy-discussion] efficient sum of "sparse" 2D arrays? On 26.01.2004, at 07:14, RJS wrote: > The problem: I have a "stack" of 8, 640 x 480 integer image arrays > from a FITS cube concatenated into a 3D array, and I want to sum each > pixel such that the result ignores clipped values (255+); i.e., if two > images have clipped pixels at (x,y) the result along z will be the sum > of the other 6. > Memory doesn't seem critical for such small arrays, so you can just do sum([where(a < 255, a, 0) for a in images]) Konrad. ------------------------------------------------------- The SF.Net email is sponsored by EclipseCon 2004 Premiere Conference on Open Tools Development and Integration See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. http://www.eclipsecon.org/osdn _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

1 0

efficient sum of "sparse" 2D arrays?
by RJS Jan. 25, 2004

Jan. 25, 2004

Hi all, The problem: I have a "stack" of 8, 640 x 480 integer image arrays from a FITS cube concatenated into a 3D array, and I want to sum each pixel such that the result ignores clipped values (255+); i.e., if two images have clipped pixels at (x,y) the result along z will be the sum of the other 6. I'm trying to come up with a pure Numeric way (hopefully so that I can use weave.blitz) to speed up the calculation. I just looked into masked arrays, but I'm not familiar with that module at all. I was guessing someone out there has done this before... Ray

2 1

Numpy capabilities?
by Nancy Keuss Jan. 25, 2004

Jan. 25, 2004

Hi, I will be working a lot with matrices, and I am wondering a few things before I get started with NumPy: 1) Is there a function that performs matrix multiplication? 2) Is there a function that takes a tensor product, or Kronecker product, of two matrices? 3) Is it possible to concatenate two matrices together? 4) Is there a way to insert a matrix into a subsection of an already existing matrix. For instance, to insert a 2x2 matrix into the upper left hand corner of a 4x4 matrix? Thank you very much in advance! Nancy

2 1

numarray0.4->numarray0.8
by René Bastian Jan. 23, 2004

Jan. 23, 2004

I need your help. I tried to update numarray-0.4 to numarray-0.8 I did not get error messages during "install" but lauching python2.3 >>>import numarray I get the message Fatal Python error : Can't import module numarray.libnumarray Uninstall 0.4 (or 0.8) ? How to uninstall numarray ? Thanks for your answers -- René Bastian http://www.musiques-rb.org : Musique en Python

2 1

Payment Past Due, acct Numpy-discussion uxzgqdfj
by Charity Pearce Jan. 23, 2004

Jan. 23, 2004

1 0

Boost Your Car's Gas Mileage 27%+.....ifama
by Joe Manning Jan. 23, 2004

Jan. 23, 2004

FUEL SAVER PRO This revolutionary device Boosts Gas Mileage 27%+ by helping fuel burn better using three patented processes from General Motors. www.xnue.biz?axel=49 PROVEN TECHNOLOGY A certified U.S. Environmental Protection Agency (EPA) laboratory recently completed tests on the new Fuel Saver. The results were astounding! Master Service, a subsidiary of Ford Motor Company, also conducted extensive emissions testing and obtained similar, unheard of results. The achievements of the Fuel Saver is so noteworthy to the environmental community, that Commercial News has featured it as their cover story in their June, 2000 edition. Take a test drive Today - www.xnue.biz?axel=49 No more advertisements, thanks - www.sftwre.biz/gh/r/r.asp bawu iayn slh i eyulrvukbm e y i

1 0

Welcome to www.dumpsmarket.com
by orders＠avahost.net Jan. 23, 2004

Jan. 23, 2004

Welcome to www.dumpsmarket.com the site of stolen credit cards,child porno, fake money, passports and complete info about any US citizen! You can get fresh stolen dumps here: http://www.dumpsmarket.com/forum/viewtopic.php?t=77 Only using our site you can get every detail of any US citizen including SSN number: http://www.dumpsmarket.com/forum/viewtopic.php?t=192 Credit cards with cvv2 information are available here: http://www.dumpsmarket.com/forum/viewtopic.php?t=57 Our site will be usefull for the those who want to wash their money also. If you don't want to pay taxes or you need to buy something illegal like weapons or drugs. Quick contacts: Panther 44007777 Graph 146191522 You can order by phone: 5092757151. Best regards, Vladimir V Panfilovich.

1 0

Welcome to www.dumpsmarket.com
by orders＠avahost.net Jan. 23, 2004

Jan. 23, 2004

Welcome to www.dumpsmarket.com the site of stolen credit cards,child porno, fake money, passports and complete info about any US citizen! You can get fresh stolen dumps here: http://www.dumpsmarket.com/forum/viewtopic.php?t=77 Only using our site you can get every detail of any US citizen including SSN number: http://www.dumpsmarket.com/forum/viewtopic.php?t=192 Credit cards with cvv2 information are available here: http://www.dumpsmarket.com/forum/viewtopic.php?t=57 Our site will be usefull for the those who want to wash their money also. If you don't want to pay taxes or you need to buy something illegal like weapons or drugs. Quick contacts: Panther 44007777 Graph 146191522 You can order by phone: 5092757151. Best regards, Vladimir V Panfilovich.

1 0

Your Computer may need Cleaning...
by Nancy Mcgill Jan. 22, 2004

Jan. 22, 2004

1 0

the direction and pace of development
by Joe Harrington Jan. 22, 2004

Jan. 22, 2004

This is a necessarily long post about the path to an open-source replacement for IDL and Matlab. While I have tried to be fair to those who have contributed much more than I have, I have also tried to be direct about what I see as some fairly fundamental problems in the way we're going about this. I've given it some section titles so you can navigate, but I hope that you will read the whole thing before posting a reply. I fear that this will offend some people, but please know that I value all your efforts, and offense is not my intent. THE PAST VS. NOW While there is significant and dedicated effort going into numeric/numarray/scipy, it's becoming clear that we are not progressing quickly toward a replacement for IDL and Matlab. I have great respect for all those contributing to the code base, but I think the present discussion indicates some deep problems. If we don't identify those problems (easy) and solve them (harder, but not impossible), we will continue not to have the solution so many people want. To be convinced that we are doing something wrong at a fundamental level, consider that Python was the clear choice for a replacement in 1996, when Paul Barrett and I ran a BoF at ADASS VI on interactive data analysis environments. That was over 7 years ago. When people asked at that conference, "what does Python need to replace IDL or Matlab", the answer was clearly "stable interfaces to basic numerics and plotting; then we can build it from there following the open-source model". Work on both these problems was already well underway then. Now, both the numerical and plotting development efforts have branched. There is still no stable base upon which to build. There aren't even packages for popular OSs that people can install and play with. The problem is not that we don't know how to do numerics or graphics; if anything, we know these things too well. In 1996, if anyone had told us that in 2004 there would be no ready-to-go replacement system because of a factor of 4 in small array creation overhead (on computers that ran 100x as fast as those then available) or the lack of interactive editing of plots at video speeds, the response would not have been pretty. How would you have felt? THE PROBLEM We are not following the open-source development model. Rather, we pay lip service to it. Open source's development mantra is "release early, release often". This means release to the public, for use, a package that has core capability and reasonably-defined interfaces. Release it in a way that as many people as possible will get it, install it, use it for real work, and contribute to it. Make the main focus of the core development team the evaluation and inclusion of contributions from others. Develop a common vision for the program, and use that vision to make decisions and keep efforts focused. Include contributing developers in decision making, but do make decisions and move on from them. Instead, there are no packages for general distribution. The basic interfaces are unstable, and not even being publicly debated to decide among them (save for the past 3 days). The core developers seem to spend most of their time developing, mostly out of view of the potential user base. I am asked probably twice a week by different fellow astronomers when an open-source replacement for IDL will be available. They are mostly unaware that this effort even exists. However, this indicates that there are at least hundreds of potential contributors of application code in astronomy alone, as I don't nearly know everyone. The current efforts look rather more like the GNU project than Linux. I'm sorry if that hurts, but it is true. I know that Perry's group at STScI and the fine folks at Enthought will say they have to work on what they are being paid to work on. Both groups should consider the long term cost, in dollars, of spending those development dollars 100% on coding, rather than 50% on coding and 50% on outreach and intake. Linus himself has written only a small fraction of the Linux kernel, and almost none of the applications, yet in much less than 7 years Linux became a viable operating system, something much bigger than what we are attempting here. He couldn't have done that himself, for any amount of money. We all know this. THE PATH Here is what I suggest: 1. We should identify the remaining open interface questions. Not, "why is numeric faster than numarray", but "what should the syntax of creating an array be, and of doing different basic operations". If numeric and numarray are in agreement on these issues, then we can move on, and debate performance and features later. 2. We should identify what we need out of the core plotting capability. Again, not "chaco vs. pyxis", but the list of requirements (as an astronomer, I very much like Perry's list). 3. We should collect or implement a very minimal version of the featureset, and document it well enough that others like us can do simple but real tasks to try it out, without reading source code. That documentation should include lists of things that still need to be done. 4. We should release a stand-alone version of the whole thing in the formats most likely to be installed by users on the four most popular OSs: Linux, Windows, Mac, and Solaris. For Linux, this means .rpm and .deb files for Fedora Core 1 and Debian 3.0r2. Tarballs and CVS checkouts are right out. We have seen that nobody in the real world installs them. To be most portable and robust, it would make sense to include the Python interpreter, named such that it does not stomp on versions of Python in the released operating systems. Static linking likewise solves a host of problems and greatly reduces the number of package variants we will have to maintain. 5. We should advertize and advocate the result at conferences and elsewhere, being sure to label it what it is: a first-cut effort designed to do a few things well and serve as a platform for building on. We should also solicit and encourage people either to work on the included TODO lists or to contribute applications. One item on the TODO list should be code converters from IDL and Matlab to Python, and compatibility libraries. 6. We should then all continue to participate in the discussions and development efforts that appeal to us. We should keep in mind that evaluating and incorporating code that comes in is in the long run much more efficient than writing the universe ourselves. 7. We should cut and package new releases frequently, at least once every six months. It is better to delay a wanted feature by one release than to hold a release for a wanted feature. The mountain is climbed in small steps. The open source model is successful because it follows closely something that has worked for a long time: the scientific method, with its community contributions, peer review, open discussion, and progress mainly in small steps. Once basic capability is out there, we can twiddle with how to improve things behind the scenes. IS SCIPY THE WAY? The recipe above sounds a lot like SciPy. SciPy began as a way to integrate the necessary add-ons to numeric for real work. It was supposed to test, document, and distribute everything together. I am aware that there are people who use it, but the numbers are small and they seem to be tightly connected to Enthought for support and application development. Enthought's focus seems to be on servicing its paying customers rather than on moving SciPy development along, and I fear they are building an installed customer base on interfaces that were not intended to be stable. So, I will raise the question: is SciPy the way? Rather than forking the plotting and numerical efforts from what SciPy is doing, should we not be creating a new effort to do what SciPy has so far not delivered? These are not rhetorical or leading questions. I don't know enough about the motivations, intentions, and resources of the folks at Enthought (and elsewhere) to know the answer. I do think that such a fork will occur unless SciPy's approach changes substantially. The way to decide is for us all to discuss the question openly on these lists, and for those willing to participate and contribute effort to declare so openly. I think all that is needed, either to help SciPy or replace it, is some leadership in the direction outlined above. I would be interested in hearing, perhaps from the folks at Enthought, alternative points of view. Why are there no packages for popular OSs for SciPy 0.2? Why are releases so infrequent? If the folks running the show at scipy.org disagree with many others on these lists, then perhaps those others would like to roll their own. Or, perhaps stable/testing/unstable releases of the whole package are in order. HOW TO CONTRIBUTE? Judging by the number of PhDs in sigs, there are a lot of researchers on this list. I'm one, and I know that our time for doing core development or providing the aforementioned leadership is very limited, if not zero. Later we will be in a much better position to contribute application software. However, there is a way we can contribute to the core effort even if we are not paid, and that is to put budget items in grant and project proposals to support the work of others. Those others could be either our own employees or subcontractors at places like Enthought or STScI. A handful of contributors would be all we'd need to support someone to produce OS packages and tutorial documentation (the stuff core developers find boring) for two releases a year. --jh--

7 11