NumPy-Discussion September 2007

numpy-discussion@python.org

76 participants
71 discussions

by Charles R Harris

Hi All, While doing documentation, I've run into the need for a simple word to express the fact that a variable can be an array, or anything that can be converted to an array using asarray(). I have been using array_like for this kind of variable, but I am open to suggestions. Also, the current version of epydoc doesn't like None as a default keyword when doing module introspection. On my machine this yields [{'bogomips': '5066.01', 'cache size': '4096 KB', 'clflush siz... as the value. I suspect this can be fixed in epydoc. There are a few other minor glitches as well. Chuck

14 years, 4 months

pycdf probs

by Mathew Yeates

Anybody know how to contact the pycdf author? His name is Gosselin I think. There are hardcoded values that cause pycdf to segfault when using large strings. Mathew

14 years, 4 months

ANN: Reminder - Texas Python Regional Unconference

by Travis Vaught

Greetings, Just a reminder for those in the area... http://pycamp.python.org/Texas/HomePage The Unconference is to be held this weekend (Saturday and Sunday, September 15, 16) at the Texas Learning & Computing Center at the University of Houston main campus. It's free. Sign up by adding your name to the wiki page. Travis

14 years, 4 months

Optimizing similarity matrix algorithm

by Kurdt Bane

Hi to all! For reverse engineering purposes, I need to find where every possible chunk of bytes in file A is contained in file B. Obviously, if a chunk of length n is contained in B, I dont' want my script to recognize also all the subchunks of size < n contained in the chunk. I coded a naive implementation of a similarity algorithm: scan the similarity algorithm and find all the diagonals. Here's the code: file_a = open(argv[1]) file_b = open(argv[2]) a = numpy.fromstring(file_a.read(),'c') b = numpy.fromstring(file_b.read(),'c') tolerance = int(argv[3]) chunks = [] valid = True count_xcent = 0 for x in xrange(len(a)): for y in xrange(len(b)): count = 0 if (a[x] == b[y]): x_cnt, y_cnt = x,y if (a[x_cnt] == b[y_cnt]): try: while (a[x_cnt+1] == b[y_cnt+1]): count += 1 x_cnt += 1 y_cnt += 1 except IndexError: pass if ((count > tolerance) or (count == tolerance)): for tuple in chunks: if (((x >= tuple[0]) and (x_cnt <= tuple[1])) and ((y >= tuple[2]) and (y_cnt <= tuple[3]))): valid = False if __debug__: print "Found an already processed subchunk" break if (valid): chunks.append ((x, x_cnt, y, y_cnt)) print "Corresponding chunk found. List 1: from " + str(x) + " to " + str(x_cnt) +". List 2: from " + str(y) + " to " + str(y_cnt) print "with length of " + str (x_cnt + 1 - x) else: valid = True It simply scans the similarity matrix, finds the diagonals in wich a[x] == a[y] and interprets the diagonal as a chunk. Then, it stores the chunk in a list and determines if it's a subchunk of another greater chunk already found. The problem is: this implementation is very slow, imho because of three factors: 1. I use a nested for loop to scan the matrix 2. When the start of a diagonal is found, the program scans the diagonal with another additional loop. Maybe it would be faster to use a function such as diagonal() (but I can't actually _create_ the boolean similarity matrix, as it gets way too big for files big enough (we're talking about ~ 100 Kb files) - I am forced to compute it "on the way". 3. When I find a chunk, I compute all its subchunks with this approach, and compare them with the "big" chunk I stored in the list. Maybe there's a better algorithmical way. What do you think about these issues? Is there a way to optimize them? And are there other issues I didn't take in account? Thanks in advance, regards, Chris.

14 years, 4 months

how to include numpy headers when building an extension?

by Christopher Barker

Hi all, I'm porting an extension from Numeric. At the top, I've changed: #include <Numeric/arrayobject.h> to #include <numpy/arrayobject.h> But distutils can't find numpy/arrayobject.h -- how do I tell distutils where to look for it? I've gotten it to work by hard-coding the entire path, but that's not very portable. It looks like Numeric used to put its headers in a standard location, but that numpy buries them deeper. thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

14 years, 4 months

odd behavior for numpy.ndarray index?

by Orest Kozyar

In the following output (see below), why would x[1,None] work, but x[1,None,2] or even x[1,2,None] not work? Incidentally, I would be very interested in a solution that allows me to index numpy arrays using a list/iterable that might contain None values. Is there a straightforward way to do so, or should I just use list comprehensions (i.e. [x[a] for a in indices if a])? Thanks! Orest In [122]: x = arange(5) In [123]: x[1] Out[123]: 1 In [124]: x[None] Out[124]: array([[0, 1, 2, 3, 4]]) In [125]: x[1,None] Out[125]: array([1]) In [126]: x[1,None,2] --------------------------------------------------------------------------- <type 'exceptions.IndexError'> Traceback (most recent call last) c:\documents\research\programs\python\epl\src\<ipython console> in <module>() <type 'exceptions.IndexError'>: invalid inde

14 years, 4 months

von mises distribution in numpy.random biased

by killian koepsell

hi, the von mises distribution in numpy.random seems to be biased towards a higher concentration (kappa). given a concentration of 2, it produces data that has a concentration of 2.36. i compared the distribution to the one produced by the CircStats[1] package of R[2] using RPy [3] and created a figure here: http://redwood.berkeley.edu/kilian/vonmises.png the script i used is attached to this email. i don't know what algorithm NumPy uses, so i can't tell if it is a real bug or some sort of rounding error. the CircStats package uses the algorithm by Best and Fisher [4]. kilian [1] http://cran.r-project.org/src/contrib/Descriptions/CircStats.html [2] http://www.r-project.org/ [3] http://rpy.sf.net/ [4] Best, D. and Fisher, N. (1979). Efficient simulation of the von Mises distribution. Applied Statistics, 24, 152-157. ---------------------- vonmises.py ---------------------- from numpy import array,pi,exp,mod,random from pylab import hist,linspace,figure,clf,plot,subplot,xlim,title,legend from rpy import r as R R.library("CircStats") # compute von Mises distribuion fo two different values of kappa size = 1e5 kappa1 = 1 x1 = random.vonmises(0,kappa1,size=size) y1 = mod(R.rvm(size,0,kappa1),2*pi) kappa2 = 2 x2 = random.vonmises(0,kappa2,size=size) y2 = mod(R.rvm(size,0,kappa2),2*pi) def phasedist(x,kappa): ph = linspace(-pi,pi,50) vM = R.dvm(ph,0,kappa) kest = R.A1inv(abs(exp(1j*array(x)).mean())) vMest = R.dvm(ph,0,kest) plot(ph,vM,'r',linewidth=2) plot(ph,vMest,'k:',linewidth=2) legend(['$\\kappa=%d$'%kappa,'$\\hat{\\kappa}=%1.2f$'%kest]) hist(x,ph,normed=1) xlim((-pi,pi)) # plot figure fig=figure(1) clf() ax1=subplot(2,2,1) title('NumPy.random') phasedist(x1,kappa1) ax2=subplot(2,2,3) phasedist(x2,kappa2) subplot(2,2,2,sharey=ax1) title('RPy.CircStats') phasedist(y1,kappa1) subplot(2,2,4,sharey=ax2) phasedist(y2,kappa2)

14 years, 4 months

why system_info.check_libs does not look for dll on windows ?

by David Cournapeau

Hi, I would like to know if there is a reason why system_info does not look for dll on windows ? I think it would make sense to look for dll when you want to use an external lib through ctypes, for example. cheers, David

14 years, 4 months

Importing data from html tables

by ale

Hi, I'm trying to import into array the data contained in a html table. I use BeautifulSoup as html parser html = open('T0015.html','r') bs = BeautifulSoup(html) for tr in bs.findAll('tr')[1:]: table.append([td.p.string for td in tr.findAll('td')]) and I get this: print table [[u'1925', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'105.0'] [u'1926', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'136.0'] [u'1927', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'51.0'] [u'1928', u'--', u'--', u'--', u'nn', u'--', u'--', u'--', u'--', u'104.0'] ,.......and so on] How to put this list of list of strings in a numpy array, and set '--' and 'nn' as NaN? Thank you Alessio

14 years, 4 months

corrcoef

by John Hunter

Is it desirable that numpy.corrcoef for two arrays returns a 2x2 array rather than a scalar In [10]: npy.corrcoef(npy.random.rand(10), npy.random.rand(10)) Out[10]: array([[ 1. , -0.16088728], [-0.16088728, 1. ]]) I always end up extracting the 0,1 element anyway. What is the advantage, aside from backwards compatibility, for returning a 2x2? JDH

14 years, 4 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2007