Mailman 3 August 2009 - NumPy-Discussion

Bug in NoseTester?
by Skipper Seabold 22 Aug '09

22 Aug '09

I'm trying to define a function, so that I don't have to pass the extra_argv==["--exe"] manually to run the tests for my scikits package. To do so, I believe I need to define a function that calls Tester(package=string_of_fullpath).test(extra_argv=["--exe"]) In nosetester.NoseTester, the docstring says that package can be a string, but if it's a string then package_path gets referenced before assignment I believe (line 128) <http://svn.scipy.org/svn/numpy/trunk/numpy/testing/nosetester.py> There needs to be another explicit test to see if it's a string and then assign? Skipper

1 0

Scipy09 Sprints
by John Schulman 22 Aug '09

22 Aug '09

For the numpy/scipyers at Caltech now: I'm wondering what is happening with the scipy09 sprints. According to the website, the sprints start at 8AM in Powell Booth, but that building is locked and there's no sign of life. Best, John

2 1

Anyone coming to the sprints have a >= 1GB memory stick or blank CD?
by David Goldsmith 22 Aug '09

22 Aug '09

If so, and I could use it to try to install Kubuntu tomorrow, I'd really appreciate it if you'd bring it w/ you. Thanks! DG __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

PRNGs and multi-threading
by Sturla Molden 22 Aug '09

22 Aug '09

I am not sure if this is the right place to discuss this issue. However, I problem I keep running into in Monte Carlo simulations is generating pseudorandom numbers with multiple threads. PRNGs such as the Mersenne Twister keep an internal state, which prevents the PRNG from being re-entrant and thread-safe. Possible solutions: 1. Use multiple instances of PRNG states (numpy.random.RandomState), one for each thread. This should give no contention, but is this mathematically acceptable? I don't know. At least I have not seen any proof that it is. 2. Protect the PRNG internally with a spinlock. In Windows lingo, that is: #include <windows.h> static volatile long spinlock = 0; #define ACQUIRE_SPINLOCK while(InterlockedExchangeAcquire(&spinlock, 1)); #define RELEASE_SPINLOCK InterlockedExchangeAcquire(&spinlock, 0); Problem: possible contention between thread, idle work if threads spins a lot. 3. Use a conventional mutex object to protect the PRNG (e.g. threading.Lock in Python or CRITICAL_SECTION in Windows). Problem: contention, context shifting, and mutexes tend to be slow. Possibly the worst solution. 4. Put the PRNG in a dedicated thread, fill up rather big arrays with pseudo-random numbers, and write them to a queue. Problem: Context shifting unless a CPU is dedicated to this task. Unless producing random numbers consitutes a major portion of the simulation, this should not lead to much contention. import threading import Queue import numpy from numpy.random import rand class PRNG(threading.Thread): ''' a thread that generate arrays with random numbers and dumps them to a queue ''' def __init__(self, nthreads, shape): self.shape = shape self.count = numpy.prod(shape) self.queue = Queue.Queue(4 * nthreads) # magic number self.stop_evt = threading.Event() def generate(self, block=True, timeout=None): return self.queue.get(block,timeout) def join(self, timeout=None): self.stop_evt.set() super(PRNG,self).join(timeout) def run(self): tmp = rand(count).reshape(shape) while 1: try: self.queue.put(tmp, block=True, timeout=2.0) except Queue.Full: if self.stop_evt.isSet(): break else: tmp = rand(count).reshape(shape) Do you have any view on this? Is there any way of creating multiple independent random states that will work correctly? I know of SPRNG (Scalable PRNG), but it is made to work with MPI (which I don't use). Regards, Sturla Molden

3 7

GPU Numpy
by Romain Brette 22 Aug '09

22 Aug '09

Hi everyone, I was wondering if you had any plan to incorporate some GPU support to numpy, or perhaps as a separate module. What I have in mind is something that would mimick the syntax of numpy arrays, with a new dtype (gpufloat), like this: from gpunumpy import * x=zeros(100,dtype='gpufloat') # Creates an array of 100 elements on the GPU y=ones(100,dtype='gpufloat') z=exp(2*x+y) # z in on the GPU, all operations on GPU with no transfer z_cpu=array(z,dtype='float') # z is copied to the CPU i=(z>2.3).nonzero()[0] # operation on GPU, returns a CPU integer array I came across a paper about something like that but couldn't find any public release: http://www.tricity.wsu.edu/~bobl/personal/mypubs/2009_gpupy_toms.pdf There is a library named GPULib (http://www.txcorp.com/products/GPULib/) that does similar things, but unfortunately they don't support Python (I think their main Python developer left). I think this would be very useful for many people. For our project (a neural network simulator, http://www.briansimulator.org) we use PyCuda (http://mathema.tician.de/software/pycuda), which is great, but it is mainly for low-level GPU programming. Cheers Romain

10 14

Problems distributing NumPy
by Michael Cooper 21 Aug '09

21 Aug '09

Hi all- I am writing a C++ application with embedded Python scripting. Some of the scripts use NumPy, so I have been working out the best way to distribute NumPy with my software. At the moment, I've got a private folder which I add to the Python path. In this folder, I include all the files which would usually get installed to the "site-packages" folder. To get these files, I've simply unzipped the distutils installer (for NumPy, I am using the "no SSE" version at the moment), and copied the contents of the resulting "PLATLIB" folder to my private folder. For all the other libraries I am using, this method seems to work fine. However, with NumPy, if I do things this way, when I call "import_array()", it jumps back out of the calling function, skipping the rest of that function, and continues from there. If I install NumPy in the usual way, this does not happen. It seems like the NumPy initialization is failing when I install into the private folder, but not if I use the normal installer. Many of the users of my software aren't particularly Python savvy, so having them install everything manually is not an option. I would like to avoid having my own installer call external installers, sine that's confusing for some users. Finally, if possible, I would like to avoid changing the user's "Python26" folder, since I have no way of knowing what else might be relying on its contents. Lots of searching on how to install third-party libraries has led me to the "PLATLIB" method, so I'm at a bit of a loss as to what else to try. Does anyone here know what might be going wrong? I am using Python 2.6, Boost 1.38, and NumPy 1.3.0 on a Windows XP system. The embedding program is written in C++, and compiled using Visual Studio 2005. Thanks, Michael

1 1

itemsize() doesn't work
by Dr. Phillip M. Feldman 21 Aug '09

21 Aug '09

I've been reading the online NumPy tutorial at the following URL: http://numpy.scipy.org/numpydoc/numpy-10.html When I try the following example, I get an error message: In [1]: a=arange(10) In [2]: a.itemsize() --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Python\<ipython console> in <module>() TypeError: 'int' object is not callable -- View this message in context: http://www.nabble.com/itemsize%28%29-doesn%27t-work-tp25072522p25072522.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

5 5

nosetests and permissions
by Chris Colbert 21 Aug '09

21 Aug '09

when I build numpy from source via: python setup.py build sudo python setup.py install the nosetests fail because of permissions: In [5]: np.test() Running unit tests for numpy NumPy version 1.3.0 NumPy is installed in /usr/local/lib/python2.6/dist-packages/numpy Python version 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] nose version 0.10.4 ---------------------------------------------------------------------- Ran 0 tests in 0.007s OK Out[5]: <nose.result.TextTestResult run=0 errors=0 failures=0> The problem I'm running into is I can't do a blanket chmod 664 *.py on the numpy directory because that breaks things. And since I don't which files are nosetests, it's very difficult to change by hand. Is there a workaround for this, or would it more appropriate for the numpy build script to set the permissions of the test file accordingly? Cheers, Chris

4 9

how to find array indices at which a condition is satisfied?
by Dr. Phillip M. Feldman 21 Aug '09

21 Aug '09

I have a 1-D array and would like to generate a list of indices for which a given condition is satisfied. What is the cleanest way to do this? -- View this message in context: http://www.nabble.com/how-to-find-array-indices-at-which-a-condition-is-sat… Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 3

what to do with chararray
by Ralf Gommers 20 Aug '09

20 Aug '09

[this discussion moved here from the SciPy list] On Thu, Aug 20, 2009 at 9:50 AM, Christopher Hanley <chanley(a)stsci.edu>wrote: > Hi, > > I'd like to respectfully request that we move any discussion of what > to do with the numpy.char module to the numpy list. > > I'm a little concerned about some of the assumptions that are being > made about the number of users of the module. My assumption about there being few users of chararray is based on the absence of questions about it being asked on the list, the lack of docs and the apparent lack of a good use-case. Plus "who uses this?" has been asked twice on the list, and in both cases you (Chris) were the only one replying. > I would also like to > better understand the reasons for wanting to dump it. Let me be > clear. I'm not opposed to change. However breaking other people's > code just for the sake of change seems like a poor reason and a mean > thing to do to our customers. That would be a very poor reason, and I don't think that's the case. The reason this question about the future of chararray has now been asked twice is that people are trying to document it, and having trouble. If it stays, it has to be documented (and the bugs found in the process fixed) which costs time as well. It just being there will also cause people to understand / try to use it. Then if it turns out it is useless to those people, it wasted their time. This was the case for me. It would be great if we could a clear use-case and a reason why chararray is in NumPy besides backwards compatibility. Otherwise it should at least be documented as not for new development, and possibly deprecated. Finally, deprecation does not mean that the module disappears tomorrow. It can stick around for years if needed (there are functions in fft for example that have been deprecated for three years), but give a clear message to new users not to bother with it. Best regards, Ralf > > Thank you for your time and help, > Chris > > > -- > Christopher Hanley > Senior Systems Software Engineer > Space Telescope Science Institute > 3700 San Martin Drive > Baltimore MD, 21218 > (410) 338-4338 > > On Aug 20, 2009, at 1:35 AM, Robert Kern wrote: > > > On Wed, Aug 19, 2009 at 20:03, David > > Goldsmith<d_l_goldsmith(a)yahoo.com> wrote: > >> I'm going to take it a step further: "breakage" is always the > >> deterrent to change, and yet "change we must" (i.e., "adapt or > >> die"). It's certainly not without precedent - even within Numpy, I > >> believe - for things (though perhaps not whole namespaces) to be > >> deemed "to-be-deprecated," have a warning to this effect > >> established in one x.[even #].0 release, and then be removed by the > >> x.[even # + 2 or + 4].0 release. How has deprecation in Numpy > >> worked in the past - by dictum, vote, or consensus? > > > > Consensus or dictum without major objection. Voting is pointless > > except to inform one of those. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev(a)scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev >

2 1