Mailman 3 May 2010 - NumPy-Discussion

Array input
by lumtegis＠gmail.com May 16, 2010

May 16, 2010

Hi all, Am creating a script to do least square adjustment of levelling data. How do I get user input into a 1D array of intergers. Thanks in advance. Allan maungu.

2 1

EuroScipy abstract submission deadline extended
by Gael Varoquaux May 15, 2010

May 15, 2010

Given that we have been able to turn on registration only very late, the EuroScipy conference committee is extending the deadline for abstract submission for the 2010 EuroScipy conference. On Thursday May 20th, at midnight Samoa time, we will turn off the abstract submission on the conference site. Up to then, you can modify the already-submitted abstract, or submit new abstracts. We are very much looking forward to your submissions to the conference. Ga�l Varoquaux Nicolas Chauvat -- EuroScipy 2010 is the annual conference for scientists using Python. It will be held July 8-11 2010, in ENS, Paris, France. Links: Conference website: http://www.euroscipy.org/conference/euroscipy2010 Call for papers: http://www.euroscipy.org/card/euroscipy2010_call_for_papers Practical information: http://www.euroscipy.org/card/euroscipy2010_practical_information

1 0

Bug with how numpy.distutils.system_info handles the site.cfg
by Chris Colbert May 15, 2010

May 15, 2010

I had this problem back in 2009 when building Enthought Enable, and was happy with a work around. It just bit me again, and I finally got around to drilling down to the problem. On linux, if one uses the numpy/site.cfg [default] section when building from source to specifiy local library directories, the x11 libs won't be found by NumPy. The relevant section of the site.cfg.example reads as follows: # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # #[DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include Now, I build NumPy with Atlas and my Atlas libs are installed in /usr/local, so my [default] section of site.cfg looks like this (as suggested by the site.cfg.example): # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # [DEFAULT] library_dirs = /usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/local/include NumPy builds and works fine with this. The problem occurs when other libraries use numpy.distutils.system_info.get_info('x11') (ala Enthought Enable). That function eventually calls numpy.distutils.system_info.system_info.parse_config_files which has the following definition: def parse_config_files(self): self.cp.read(self.files) if not self.cp.has_section(self.section): if self.section is not None: self.cp.add_section(self.section) When self.cp is instantiated (when looking for the x11 libs), it is provided the following defaults: {'libraries': '', 'src_dirs': '.:/usr/local/src', 'search_static_first': '0', 'library_dirs': '/usr/X11R6/lib64:/usr/X11R6/lib:/usr/X11/lib64:/usr/X11/lib:/usr/lib64:/usr/lib', 'include_dirs': '/usr/X11R6/include:/usr/X11/include:/usr/include'} As is clearly seen, the 'library_dirs' contains the proper paths to find the x11 libs. But since the config file has [default] section, these paths get trampled and replaced with whatever is contained in the site.cfg [default] section. In my case, this is /usr/local/lib:/usr/local/lib/atlas. Thus, my x11 libs aren't found and the Enable build fails. The workaround is to include an [x11] section in site.cfg with the appropriate paths, but I don't really feel this should be necessary. Would the better behavior be to look for a [default] section in the config file in the parse_config_files method and add those paths to the already specified defaults? Changing the site.cfg [default] section to read as follows: [DEFAULT] library_dirs = /usr/lib:/usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/include:/usr/local/include is not an option because then NumPy will find and use the system atlas, which in my case is not threaded nor optimized for my machine. If you want me to patch the parse_config_files method, just let me know. Cheers, Chris

2 2

Problems creating numpy.array with a dtype
by Jimmie Houchin May 15, 2010

May 15, 2010

Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks. I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions. My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...) Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data. I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database. Code slightly modified, not tested. #examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg) def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data) return np.array(wdata,dtype=dtminute) The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer. However changing the getArray() function to this works just fine. def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know. I find no errors in my data length or types. I would thing that either would cause version two to fail also. In help in understanding is greatly appreciated. This is using Numpy 1.4.1. Thanks. Jimmie

4 5

default behavior of argsort
by Dr. Phillip M. Feldman May 15, 2010

May 15, 2010

When operating on an array whose last dimension is unity, the default behavior of argsort is not very useful: |6> x=random.random((4,1)) |7> shape(x) <7> (4, 1) |8> argsort(x) <8> array([[0], [0], [0], [0]]) |9> argsort(x,axis=0) <9> array([[0], [2], [1], [3]]) -- View this message in context: http://old.nabble.com/default-behavior-of-argsort-tp28542476p28542476.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 7

memory leak?
by Brian Blais May 14, 2010

May 14, 2010

Hello, I have the following code, where I noticed a memory leak with +=, but not with + alone. import numpy m=numpy.matrix(numpy.ones((23,23))) for i in range(10000000): m+=0.0 # keeps growing in memory # m=m+0.0 # is stable in memory My version of python is 2.5, numpy 1.3.0, but it also causes memory build-up in 2.6 with numpy 1.4.0, as distributed by the Enthought Python Distribution. It's easy to work around, but could cause someone some problems. Anyone else get this? bb -- Brian Blais bblais(a)bryant.edu http://web.bryant.edu/~bblais http://bblais.blogspot.com/

2 3

recarray question
by Vincent Davis May 14, 2010

May 14, 2010

The setup: >>> Adata array([(1, 24, 'Male', '', 212, 193, 'High Pass'), (2, 26, 'Male', 'Caucasian', 234, 221, 'Honors'), (3, 31, 'Female', 'Caucasian', 182, 189, ''), (4, 27, 'Female', 'Hispanic', 214, 211, 'High Pass'), (5, 27, 'Female', 'Asian', 213, 204, 'Pass'), (6, 29, 'Female', 'Caucasian', 209, -1, 'High Pass'), (7, 26, 'Female', 'Hispanic', 212, -1, 'Honors'), (8, 25, 'Female', 'Caucasian', 230, 238, 'Honors'), (9, 27, 'Female', 'Caucasian', 239, 245, 'Honors'), (10, 26, 'Male', 'Caucasian', 226, -1, 'Honors')], dtype=[('ID', '<i8'), ('Age', '<i8'), ('Gender', '|S6'), ('Ethnicity', '|S12'), ('USMLE_1', '<i8'), ('USMLE_2', '<i8'), ('Medicine_Clerk', '|S9')]) >>> BData array([(1,), (1,), (0,), (0,), (0,), (0,), (0,), (0,), (0,), (1,)], dtype=[('Gender', '<i8')]) How do replace the "Gender" column in Adata with that in BData and update the datatype. >>> Adata['Gender'] = Bdata['Gender'] >>> Adata array([(1, 24, '1', '', 212, 193, 'High Pass'), (2, 26, '1', 'Caucasian', 234, 221, 'Honors'), (3, 31, '0', 'Caucasian', 182, 189, ''), (4, 27, '0', 'Hispanic', 214, 211, 'High Pass'), (5, 27, '0', 'Asian', 213, 204, 'Pass'), (6, 29, '0', 'Caucasian', 209, -1, 'High Pass'), (7, 26, '0', 'Hispanic', 212, -1, 'Honors'), (8, 25, '0', 'Caucasian', 230, 238, 'Honors'), (9, 27, '0', 'Caucasian', 239, 245, 'Honors'), (10, 26, '1', 'Caucasian', 226, -1, 'Honors')], dtype=[('ID', '<i8'), ('Age', '<i8'), ('Gender', '|S6'), ('Ethnicity', '|S12'), ('USMLE_1', '<i8'), ('USMLE_2', '<i8'), ('Medicine_Clerk', '|S9')]) Ok so now I need to change the dtype, not having much luck with that , do I need to copy the array? Is there a nice way to do this? That is swap Adata['Gender'] = Bdata['Gender'] and update the dtype at the same time?

1 0

missing='' not documented in genfromtxt()
by Vincent Davis May 14, 2010

May 14, 2010

Maybe I am missing something but it does not appear that missing='' is documented although it is shown s an argument ? genfromtxt(fname, dtype=float, comments='#', delimiter=None, skiprows=0, skip_header=0, skip_footer=0, converters=None, missing='', missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=None, autostrip=False, case_sensitive=True, defaultfmt="f%i", unpack=None, usemask=False, loose=True, invalid_raise=True):

2 3

ANN: MDP release 2.6 and MDP Sprint 2010
by Tiziano Zito May 14, 2010

May 14, 2010

We are glad to announce release 2.6 of the Modular toolkit for Data Processing (MDP). MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software. The base of available algorithms includes, to name but the most common, Principal Component Analysis (PCA and NIPALS), several Independent Component Analysis algorithms (CuBICA, FastICA, TDSEP, JADE, and XSFA), Slow Feature Analysis, Restricted Boltzmann Machine, and Locally Linear Embedding. What's new in version 2.6? -------------------------- - Several new classifier nodes have been added. - A new node extension mechanism makes it possible to dynamically add methods or attributes for specific features to node classes, enabling aspect-oriented programming in MDP. Several MDP features (like parallelization) are now based on this mechanism, and users can add their own custom node extensions. - BiMDP is a large new package in MDP that introduces bidirectional data flows to MDP, including backpropagation and even loops. BiMDP also enables the transportation of additional data in flows via messages. - BiMDP includes a new flow inspection tool, that runs as as a graphical debugger in the webrowser to step through complex flows. It can be extended by users for the analysis and visualization of intermediate data. - As usual, tons of bug fixes The new additions in the library have been thoroughly tested but, as usual after a public release, we especially welcome user's feedback and bug reports. MDP Sprint 2010 --------------- Following our tradition of sprint-driven development, the team of the core developers decided to organize a programming sprint open to external participants. We invite in particular all users who implemented new algorithms and would like to see them integrated in MDP: you will work together with a core developer! More info: http://sourceforge.net/apps/mediawiki/mdp-toolkit/index.php?title=MDP_Sprin… Resources --------- Download: http://sourceforge.net/projects/mdp-toolkit/files Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://lists.sourceforge.net/mailman/listinfo/mdp-toolkit-users -- Pietro Berkes Volen Center for Complex Systems Brandeis University Waltham, MA, USA Rike-Benjamin Schuppner Berlin, Germany Niko Wilbert Institute for Theoretical Biology Humboldt-University Berlin, Germany Tiziano Zito Modelling of Cognitive Processes Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany

1 0

mac os x installation
by Nicola May 13, 2010

May 13, 2010

Hi. I've to admit that I'm quite new in python and also to numpy. I'm trying to install numpy and scypy on my mac (Mac Os X 10.6.3). I've installed the last version of python (Python 2.6.5 Mac OS X Installer Disk Image<http://www.python.org/ftp/python/2.6.5/python-2.6.5-macosx10.3-2010-03-24.d…> ). After than I've downloaded the latest version of numpy ( numpy-1.4.1-py2.6-python.org.dmg<http://sourceforge.net/projects/numpy/files/NumPy/1.4.1/numpy-1.4.1-py2.6-p…>) and finally also the latest version of scypy ( scipy-0.7.2-py2.6-python.org.dmg<http://sourceforge.net/projects/scipy/files/scipy/0.7.2/scipy-0.7.2-py2.6-p…> ). When I try the test I obtain the following errors Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test('1','10') Running unit tests for numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 326, in test self._show_system_info() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 187, in _show_system_info nose = import_nose() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 69, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose >>> any idea? -- ---------------------------------------------- nicola.vianello(a)gmail.com skype:nicolavianello ----------------------------------------------

2 2