From davide.lasagna at polito.it Tue Mar 1 02:23:57 2011 From: davide.lasagna at polito.it (Davide) Date: Tue, 01 Mar 2011 08:23:57 +0100 Subject: [Numpy-discussion] Grid indexing In-Reply-To: <4D6BB268.90508@gmail.com> References: <4D68BBA0.4060009@gmail.com> <4D6BAF83.1010100@hkl.hms.harvard.edu> <4D6BB268.90508@gmail.com> Message-ID: <4D6C9F0D.1010501@polito.it> I had a similar problem some time ago, which i solved by building my own class containing an n-d array and an n-d interpolator. Then i overrided the __getitem__ special method so that i could use a fancy notation like the one you want. Inside the method i used n-d dimensional interpolation of the nd.array to get function evaluation at the point/s specified. Maybe this can help, Davide >>> I have a n dimensional grid. The grids axes are linear but not >>> intergers. Let's say I want the value in gridcell [3.2,-5.6,0.01]. Is >>> there an easy way to transform the index? Do I have to write my own >>> class to accomplish this. Is there an easy way in numpy to accomplish >>> this. Can I give functions to transform the index somehow to the numpy >>> array? From ndbecker2 at gmail.com Tue Mar 1 07:23:29 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 01 Mar 2011 07:23:29 -0500 Subject: [Numpy-discussion] tensordot? Message-ID: Can tensordot do this (or some other operator?) r[l,i,j] = \sum_k w[i,j,k] y[l,k] From ndbecker2 at gmail.com Tue Mar 1 07:50:06 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 01 Mar 2011 07:50:06 -0500 Subject: [Numpy-discussion] tensordot? References: Message-ID: Neal Becker wrote: > Can tensordot do this (or some other operator?) > > r[l,i,j] = \sum_k w[i,j,k] y[l,k] Looks like r = tensordot (w, y, axes=(2,1)) will produce r[i,j,l], so transpose (r, (2,0,1)) will do it. From jonovik at gmail.com Tue Mar 1 08:20:34 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Tue, 1 Mar 2011 13:20:34 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: <4D6C42B8.7040902@molden.no> Message-ID: Robert Kern gmail.com> writes: > On Mon, Feb 28, 2011 at 18:50, Sturla Molden molden.no> wrote: > > Den 01.03.2011 01:15, skrev Robert Kern: > >> You can have each of those processes memory-map the whole file and > >> just operate on their own slices. Your operating system's virtual > >> memory manager should handle all of the details for you. Wow, I didn't know that. So as long as the ranges touched by each process do not overlap, I'll be safe? If I modify only a few discontiguous chunks in a range, will the virtual memory manager decide whether it is most efficient to write just the chunks or the entire range back to disk? > > Mapping large files from the start will not always work on 32-bit > > systems. That is why mmap.mmap take an offset argument now (Python 2.7 > > and 3.1.) > > > > Making a view np.memmap with slices is useful on 64-bit but not 32-bit > > systems. > > I'm talking about the OP's stated use case where he generates the file > via memory-mapping the whole thing on the same machine. The whole file > does fit into the address space in his use case. > > I'd like to see a real use case where this does not hold. I suspect > that this is not the API we would want for such use cases. Use case: Generate "large" output for "many" parameter scenarios. 1. Preallocate "enormous" output file on disk. 2. Each process fills in part of the output. 3. Analyze, aggregate results, perhaps save to HDF or database, in a sliding- window fashion using a memory-mapped array. The aggregated results fit in memory, even though the raw output doesn't. My real work has been done on a 64-bit cluster running 64-bit Python, but I'd like to have the option of post-processing on my laptop's 32-bit Python (either spending a few hours copying the file to my laptop first, or mounting the remote disk using e.g. ExpanDrive). Maybe that is impossible with 32-bit Python: at least I cannot allocate that big a file on my laptop. >>> m = np.lib.format.open_memmap("c:/temp/temp.npy", "w+", dtype=np.int8, shape=2**33) ------------------------------------------------------------ Traceback (most recent call last): File "", line 1, in File "C:\Python26\lib\site-packages\numpy\lib\format.py", line 563, in open_memmap mode=mode, offset=offset) File "C:\Python26\lib\site-packages\numpy\core\memmap.py", line 221, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start) OverflowError: cannot fit 'long' into an index-sized integer From opossumnano at gmail.com Tue Mar 1 08:41:32 2011 From: opossumnano at gmail.com (Tiziano Zito) Date: Tue, 1 Mar 2011 14:41:32 +0100 (CET) Subject: [Numpy-discussion] =?utf-8?q?=5BANN=5D_Summer_School_=22Advanced_?= =?utf-8?q?Scientific_Programming_in_Python=22_in_St_Andrews=2C_UK?= Message-ID: <20110301134132.80F4E2498CF@mail.bccn-berlin> ?Advanced Scientific Programming in Python ========================================= a Summer School by the G-Node and the School of Psychology, University of St Andrews Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists actually use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques, incorporating theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python is assumed. Participants without any prior experience with Python should work through the proposed introductory materials before the course. Date and Location ================= September 11?16, 2011. St Andrews, UK. Preliminary Program =================== Day 0 (Sun Sept 11) ? Best Programming Practices - Agile development & Extreme Programming - Advanced Python: decorators, generators, context managers - Version control with git Day 1 (Mon Sept 12) ? Software Carpentry - Object-oriented programming & design patterns - Test-driven development, unit testing & quality assurance - Debugging, profiling and benchmarking techniques - Programming in teams Day 2 (Tue Sept 13) ? Scientific Tools for Python - Advanced NumPy - The Quest for Speed (intro): Interfacing to C with Cython - Best practices in data visualization Day 3 (Wed Sept 14) ? The Quest for Speed - Writing parallel applications in Python - Programming project Day 4 (Thu Sept 15) ? Efficient Memory Management - When parallelization does not help: the starving CPUs problem - Data serialization: from pickle to databases - Programming project Day 5 (Fri Sept 16) ? Practical Software Development - Programming project - The Pac-Man Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects. Applications ============ You can apply on-line at http://python.g-node.org Applications must be submitted before May 29, 2011. Notifications of acceptance will be sent by June 19, 2011. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate in past editions was around 30%. Prerequisites: You are supposed to know the basics of Python to participate in the lectures. Please consult the website for a list of introductory material. Faculty ======= - Francesc Alted, author of PyTables, Castell? de la Plana, Spain - Pietro Berkes, Volen Center for Complex Systems, Brandeis University, USA - Valentin Haenel, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany - Zbigniew J?drzejewski-Szmek, Faculty of Physics, University of Warsaw, Poland - Eilif Muller, The Blue Brain Project, Ecole Polytechnique F?d?rale de Lausanne, Switzerland - Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno Kessler and University of Trento, Italy - Rike-Benjamin Schuppner, Bernstein Center for Computational Neuroscience Berlin, Germany - Bartosz Tele?czuk, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany - Bastian Venthur, Berlin Institute of Technology and Bernstein Focus: Neurotechnology, Germany - Pauli Virtanen, Institute for Theoretical Physics and Astrophysics, University of W?rzburg, Germany - Tiziano Zito, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany Organized by Katharina Maria Zeiner and Manuel Spitschan of the School of Psychology, University of St Andrews, and by Zbigniew J?drzejewski-Szmek and Tiziano Zito for the German Neuroinformatics Node of the INCF. Website: http://python.g-node.org Contact: python-info at g-node.org From wesmckinn at gmail.com Tue Mar 1 10:51:10 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 1 Mar 2011 10:51:10 -0500 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses In-Reply-To: References: <28A39E36-034E-4A62-AB24-3D545769C7F6@gmail.com> Message-ID: On Mon, Feb 28, 2011 at 10:52 PM, Wes McKinney wrote: > On Mon, Feb 28, 2011 at 7:24 PM, Pierre GM wrote: >> >> On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: >> >>> On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: >>>> I'm having some trouble with the zeros_like function via np.fix: >>>> >>>> def zeros_like(a): >>>> ? ?if isinstance(a, ndarray): >>>> ? ? ? ?res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) >>>> ? ? ? ?res.fill(0) >>>> ? ? ? ?return res >>>> ? ?try: >>>> ? ? ? ?wrap = a.__array_wrap__ >>>> ? ?except AttributeError: >>>> ? ? ? ?wrap = None >>>> ? ?a = asarray(a) >>>> ? ?res = zeros(a.shape, a.dtype) >>>> ? ?if wrap: >>>> ? ? ? ?res = wrap(res) >>>> ? ?return res >>>> >>>> As you can see this is going to discard any metadata stored in a >>>> subtype. I'm not sure whether this is a bug or a feature but wanted to >>>> bring it up. >>>> >>>> Thanks, >>>> Wes >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >>> I guess this is ticket 929. >>> http://projects.scipy.org/numpy/ticket/929 >>> >>> I was looking at it today but was not sure what is really desired >>> here. I considered that this just meant shape and dtype but not sure >>> about masked or record arrays behavior. So: >>> What is the value of having the metadata? >>> What is the meaning of 'like' here? >> >> Well, that depends on what you wanna do, of course. To handle metadata, I use some kind of dictionary updated in the __array_finalize__. Check numpy.ma.MaskedArray and its subclasses (like scikits.timeseries.TimeSeries) for the details. >> Now that you could store some extra data in the dtype (if I remmbr and understand correctly), it might be worth considering a proper way to deal with that. >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > The ticket is exactly related to the problem at hand-- having > __array_finalize__ defined won't help you as it never gets called. > Looks like this commit fixed the problem, so the ticket can be closed https://github.com/numpy/numpy/commit/c9d1849332ae5bf73299ea1268f6a55f78624688#numpy/core/numeric.py From robert.kern at gmail.com Tue Mar 1 11:07:00 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Mar 2011 10:07:00 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: <4D6C42B8.7040902@molden.no> Message-ID: On Tue, Mar 1, 2011 at 07:20, Jon Olav Vik wrote: > Robert Kern gmail.com> writes: > >> On Mon, Feb 28, 2011 at 18:50, Sturla Molden molden.no> wrote: >> > Den 01.03.2011 01:15, skrev Robert Kern: >> >> You can have each of those processes memory-map the whole file and >> >> just operate on their own slices. Your operating system's virtual >> >> memory manager should handle all of the details for you. > > Wow, I didn't know that. So as long as the ranges touched by each process do > not overlap, I'll be safe? If I modify only a few discontiguous chunks in a > range, will the virtual memory manager decide whether it is most efficient to > write just the chunks or the entire range back to disk? It's up to the virtual memory manager, but usually, it will just load those pages (chunks the size of mmap.PAGESIZE) that are touched by your request and write them back. >> > Mapping large files from the start will not always work on 32-bit >> > systems. That is why mmap.mmap take an offset argument now (Python 2.7 >> > and 3.1.) >> > >> > Making a view np.memmap with slices is useful on 64-bit but not 32-bit >> > systems. >> >> I'm talking about the OP's stated use case where he generates the file >> via memory-mapping the whole thing on the same machine. The whole file >> does fit into the address space in his use case. >> >> I'd like to see a real use case where this does not hold. I suspect >> that this is not the API we would want for such use cases. > > Use case: Generate "large" output for "many" parameter scenarios. > 1. Preallocate "enormous" output file on disk. > 2. Each process fills in part of the output. > 3. Analyze, aggregate results, perhaps save to HDF or database, in a sliding- > window fashion using a memory-mapped array. The aggregated results fit in > memory, even though the raw output doesn't. > > My real work has been done on a 64-bit cluster running 64-bit Python, but I'd > like to have the option of post-processing on my laptop's 32-bit Python (either > spending a few hours copying the file to my laptop first, or mounting the > remote disk using e.g. ExpanDrive). Okay, in this case, I don't think that just adding an offset argument to np.load() is very useful. You will want to read the dtype and shape information from the header, *then* decide what offset and shape to use for the memory-mapped segment. You will want to use the functions read_magic() and read_array_header_1_0() from np.lib.format directly. You can slightly modify the logic in open_memmap(): # Read the header of the file first. fp = open(filename, 'rb') try: version = read_magic(fp) if version != (1, 0): msg = "only support version (1,0) of file format, not %r" raise ValueError(msg % (version,)) shape, fortran_order, dtype = read_array_header_1_0(fp) if dtype.hasobject: msg = "Array can't be memory-mapped: Python objects in dtype." raise ValueError(msg) offset = fp.tell() finally: fp.close() chunk_offset, chunk_shape = decide_offset_shape(dtype, shape, fortran_order, offset) marray = np.memmap(filename, dtype=dtype, shape=chunk_shape, order=('F' if fortran_order else 'C'), mode='r+', offset=chunk_offset) What might help is combining the first stanza of logic together into one read_header() function that returns the usual information and also the offset to the actual data. That lets you avoid replicating the logic for handling different format versions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From lopes80andre at gmail.com Tue Mar 1 11:07:23 2011 From: lopes80andre at gmail.com (Andre Lopes) Date: Tue, 1 Mar 2011 16:07:23 +0000 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? Message-ID: Hi, I'm new to Numpy. I'm doing some tests with some Stock Market Quotes My struggle right now is "how to get the values of the moving averages crosses", I send an image in attach to illustrate what I'm trying to get. I'm using the this computation to get when the moving averages crosses, but when I look at the graph, the values doesn't seem ok. [quote] # Get when the ma20 cross ma50 equal = np.round(ma20,2)==np.round(ma50,2) dates_cross = (dates[equal]) prices_cross = (prices[equal]) [/quote] The full code is this: [quote] # Modules import datetime import numpy as np import matplotlib.finance as finance import matplotlib.mlab as mlab import matplotlib.pyplot as plot # Define quote startdate = datetime.date(2008,10,1) today = enddate = datetime.date.today() ticker = 'uso' # Catch CSV fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) # From CSV to REACARRAY r = mlab.csv2rec(fh); fh.close() # Order by Desc r.sort() ### Methods Begin def moving_average(x, n, type='simple'): """ compute an n period moving average. type is 'simple' | 'exponential' """ x = np.asarray(x) if type=='simple': weights = np.ones(n) else: weights = np.exp(np.linspace(-1., 0., n)) weights /= weights.sum() a = np.convolve(x, weights, mode='full')[:len(x)] a[:n] = a[n] return a ### Methods End prices = r.adj_close dates = r.date ma20 = moving_average(prices, 20, type='simple') ma50 = moving_average(prices, 50, type='simple') # Get when the ma20 cross ma50 equal = np.round(ma20,2)==np.round(ma50,2) dates_cross = (dates[equal]) prices_cross = (prices[equal]) # Ver se a ma20 > ma50 # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) # dates_ma20_greater_than_ma50 = (dates[ma20_greater_than_ma50]) # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) print dates_cross print prices_cross #print dates_ma20_greater_than_ma50 #print prices_ma20_greater_than_ma50 plot.plot(prices) plot.plot(ma20) plot.plot(ma50) plot.show() [/quote] Someone can give me some clues? Best Regards, -------------- next part -------------- A non-text attachment was scrubbed... Name: moving_averages_cross.PNG Type: image/png Size: 28319 bytes Desc: not available URL: From kwgoodman at gmail.com Tue Mar 1 12:03:12 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 1 Mar 2011 09:03:12 -0800 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? In-Reply-To: References: Message-ID: On Tue, Mar 1, 2011 at 8:07 AM, Andre Lopes wrote: > Hi, > > I'm new to Numpy. I'm doing some tests with some Stock Market Quotes > > My struggle right now is "how to get the values of the moving averages > crosses", I send an image in attach to illustrate what I'm trying to > get. > > I'm using the this computation to get when the moving averages > crosses, but when I look at the graph, the values doesn't seem ok. > > [quote] > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross ?= (dates[equal]) > prices_cross = (prices[equal]) > [/quote] > > > The full code is this: > [quote] > # Modules > import datetime > import numpy as np > import matplotlib.finance as finance > import matplotlib.mlab as mlab > import matplotlib.pyplot as plot > > # Define quote > startdate = datetime.date(2008,10,1) > today = enddate = datetime.date.today() > ticker = 'uso' > > # Catch CSV > fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) > > # From CSV to REACARRAY > r = mlab.csv2rec(fh); fh.close() > # Order by Desc > r.sort() > > > ### Methods Begin > def moving_average(x, n, type='simple'): > ? ?""" > ? ?compute an n period moving average. > > ? ?type is 'simple' | 'exponential' > > ? ?""" > ? ?x = np.asarray(x) > ? ?if type=='simple': > ? ? ? ?weights = np.ones(n) > ? ?else: > ? ? ? ?weights = np.exp(np.linspace(-1., 0., n)) > > ? ?weights /= weights.sum() > > > ? ?a = ?np.convolve(x, weights, mode='full')[:len(x)] > ? ?a[:n] = a[n] > ? ?return a > ### Methods End > > > prices = r.adj_close > dates = r.date > ma20 = moving_average(prices, 20, type='simple') > ma50 = moving_average(prices, 50, type='simple') > > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross ?= (dates[equal]) > prices_cross = (prices[equal]) > > # Ver se a ma20 > ma50 > # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) > # dates_ma20_greater_than_ma50 ?= (dates[ma20_greater_than_ma50]) > # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) > > print dates_cross > print prices_cross > #print dates_ma20_greater_than_ma50 > #print prices_ma20_greater_than_ma50 > > > plot.plot(prices) > plot.plot(ma20) > plot.plot(ma50) > plot.show() > [/quote] > > Someone can give me some clues? I gave it a try using a labeled array: >> from la.data.yahoo import quotes >> data = quotes(['uso'], date1=(2008,10,1)) >> close = data.lix[:,['close']] >> ma20 = close.mov_mean(20) >> ma50 = close.mov_mean(50) >> lo = ma20 <= ma50 >> hi = ma20 >= ma50 >> cross_up = lo.lag(1) & hi >> cross_dn = hi.lag(1) & lo Dates when ma20 goes above ma50: >> cross_up[cross_up.x] label_0 2009-03-24 2009-08-14 2009-10-20 2010-01-15 2010-03-09 2010-07-16 2010-10-08 x array([ True, True, True, True, True, True, True], dtype=bool) Dates when it goes below: >> cross_dn[cross_dn.x] label_0 2009-07-17 2009-09-23 2009-12-09 2010-02-04 2010-05-10 2010-08-27 2011-02-04 x array([ True, True, True, True, True, True, True], dtype=bool) I didn't double check my logic, but hopefully it will give you some ideas to play with. From jkington at wisc.edu Tue Mar 1 12:23:10 2011 From: jkington at wisc.edu (Joe Kington) Date: Tue, 01 Mar 2011 11:23:10 -0600 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? In-Reply-To: References: Message-ID: Hi Andre, Assuming that you want the exact point (date and value) where each crossing occurs, you'll need to interpolate where they cross. There are a number of different ways to do so, but assuming you're okay with linear interpolation, and everything's sampled on the same dates, you can simply do something like this: import numpy as np import matplotlib.pyplot as plt def main(): x = np.linspace(0, 2*np.pi, 20) y1 = np.sin(2*x) y2 = np.cos(x) crossings = find_crossings(x, y1, y2) cross_x, cross_y = crossings.T plt.plot(x, y1, 'bx-') plt.plot(x, y2, 'gx-') plt.plot(cross_x, cross_y, 'ro') plt.show() def find_crossings(x, y1, y2): diff = np.diff(np.sign(y1 - y2)) indicies, = np.nonzero(diff) crossings = [interpolate_crossing(i, x, y1, y2) for i in indicies] return np.array(crossings) def interpolate_crossing(i, x, y1, y2): slope = ( (y1[i] - y2[i]) / ((y2[i+1] - y2[i]) - (y1[i+1] - y1[i]))) x = x[i] + slope * (x[i+1] - x[i]) y = y1[i] + slope * (y1[i+1] - y1[i]) return x, y main() [image: VXsqp.png] On Tue, Mar 1, 2011 at 10:07 AM, Andre Lopes wrote: > Hi, > > I'm new to Numpy. I'm doing some tests with some Stock Market Quotes > > My struggle right now is "how to get the values of the moving averages > crosses", I send an image in attach to illustrate what I'm trying to > get. > > I'm using the this computation to get when the moving averages > crosses, but when I look at the graph, the values doesn't seem ok. > > [quote] > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross = (dates[equal]) > prices_cross = (prices[equal]) > [/quote] > > > The full code is this: > [quote] > # Modules > import datetime > import numpy as np > import matplotlib.finance as finance > import matplotlib.mlab as mlab > import matplotlib.pyplot as plot > > # Define quote > startdate = datetime.date(2008,10,1) > today = enddate = datetime.date.today() > ticker = 'uso' > > # Catch CSV > fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) > > # From CSV to REACARRAY > r = mlab.csv2rec(fh); fh.close() > # Order by Desc > r.sort() > > > ### Methods Begin > def moving_average(x, n, type='simple'): > """ > compute an n period moving average. > > type is 'simple' | 'exponential' > > """ > x = np.asarray(x) > if type=='simple': > weights = np.ones(n) > else: > weights = np.exp(np.linspace(-1., 0., n)) > > weights /= weights.sum() > > > a = np.convolve(x, weights, mode='full')[:len(x)] > a[:n] = a[n] > return a > ### Methods End > > > prices = r.adj_close > dates = r.date > ma20 = moving_average(prices, 20, type='simple') > ma50 = moving_average(prices, 50, type='simple') > > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross = (dates[equal]) > prices_cross = (prices[equal]) > > # Ver se a ma20 > ma50 > # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) > # dates_ma20_greater_than_ma50 = (dates[ma20_greater_than_ma50]) > # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) > > print dates_cross > print prices_cross > #print dates_ma20_greater_than_ma50 > #print prices_ma20_greater_than_ma50 > > > plot.plot(prices) > plot.plot(ma20) > plot.plot(ma50) > plot.show() > [/quote] > > Someone can give me some clues? > > Best Regards, > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lopes80andre at gmail.com Tue Mar 1 13:55:54 2011 From: lopes80andre at gmail.com (Andre Lopes) Date: Tue, 1 Mar 2011 18:55:54 +0000 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? In-Reply-To: References: Message-ID: Hi Joe, Thanks for the reply. I'm not okay with linear interpolation. Can you suggest me some books around this subject. I will mainly try to build some indicator for the stock market. If you can give me a clue I would be appreciated. Best Regards, On Tue, Mar 1, 2011 at 5:23 PM, Joe Kington wrote: > Hi Andre, > > Assuming that you want the exact point (date and value) where each crossing > occurs, you'll need to interpolate where they cross. > > There are a number of different ways to do so, but assuming you're okay > with linear interpolation, and everything's sampled on the same dates, you > can simply do something like this: > > import numpy as np > import matplotlib.pyplot as plt > > def main(): > x = np.linspace(0, 2*np.pi, 20) > y1 = np.sin(2*x) > y2 = np.cos(x) > crossings = find_crossings(x, y1, y2) > cross_x, cross_y = crossings.T > plt.plot(x, y1, 'bx-') > plt.plot(x, y2, 'gx-') > plt.plot(cross_x, cross_y, 'ro') > plt.show() > > def find_crossings(x, y1, y2): > diff = np.diff(np.sign(y1 - y2)) > indicies, = np.nonzero(diff) > crossings = [interpolate_crossing(i, x, y1, y2) for i in indicies] > return np.array(crossings) > > def interpolate_crossing(i, x, y1, y2): > slope = ( (y1[i] - y2[i]) > / ((y2[i+1] - y2[i]) - (y1[i+1] - y1[i]))) > x = x[i] + slope * (x[i+1] - x[i]) > y = y1[i] + slope * (y1[i+1] - y1[i]) > return x, y > > main() > > [image: VXsqp.png] > > On Tue, Mar 1, 2011 at 10:07 AM, Andre Lopes wrote: > >> Hi, >> >> I'm new to Numpy. I'm doing some tests with some Stock Market Quotes >> >> My struggle right now is "how to get the values of the moving averages >> crosses", I send an image in attach to illustrate what I'm trying to >> get. >> >> I'm using the this computation to get when the moving averages >> crosses, but when I look at the graph, the values doesn't seem ok. >> >> [quote] >> # Get when the ma20 cross ma50 >> equal = np.round(ma20,2)==np.round(ma50,2) >> dates_cross = (dates[equal]) >> prices_cross = (prices[equal]) >> [/quote] >> >> >> The full code is this: >> [quote] >> # Modules >> import datetime >> import numpy as np >> import matplotlib.finance as finance >> import matplotlib.mlab as mlab >> import matplotlib.pyplot as plot >> >> # Define quote >> startdate = datetime.date(2008,10,1) >> today = enddate = datetime.date.today() >> ticker = 'uso' >> >> # Catch CSV >> fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) >> >> # From CSV to REACARRAY >> r = mlab.csv2rec(fh); fh.close() >> # Order by Desc >> r.sort() >> >> >> ### Methods Begin >> def moving_average(x, n, type='simple'): >> """ >> compute an n period moving average. >> >> type is 'simple' | 'exponential' >> >> """ >> x = np.asarray(x) >> if type=='simple': >> weights = np.ones(n) >> else: >> weights = np.exp(np.linspace(-1., 0., n)) >> >> weights /= weights.sum() >> >> >> a = np.convolve(x, weights, mode='full')[:len(x)] >> a[:n] = a[n] >> return a >> ### Methods End >> >> >> prices = r.adj_close >> dates = r.date >> ma20 = moving_average(prices, 20, type='simple') >> ma50 = moving_average(prices, 50, type='simple') >> >> # Get when the ma20 cross ma50 >> equal = np.round(ma20,2)==np.round(ma50,2) >> dates_cross = (dates[equal]) >> prices_cross = (prices[equal]) >> >> # Ver se a ma20 > ma50 >> # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) >> # dates_ma20_greater_than_ma50 = (dates[ma20_greater_than_ma50]) >> # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) >> >> print dates_cross >> print prices_cross >> #print dates_ma20_greater_than_ma50 >> #print prices_ma20_greater_than_ma50 >> >> >> plot.plot(prices) >> plot.plot(ma20) >> plot.plot(ma50) >> plot.show() >> [/quote] >> >> Someone can give me some clues? >> >> Best Regards, >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 1 14:17:02 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 01 Mar 2011 11:17:02 -0800 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? In-Reply-To: References: Message-ID: <4D6D462E.5000709@noaa.gov> On 3/1/11 10:55 AM, Andre Lopes wrote: > I'm not okay with linear interpolation. Well, odds are that the crossing point won't be exactly at a data point, so you need to do SOME kind of interpolation. You could use a higher order interpolation (cubic spline, etc): see the interpolation routines in scipy. However, for this use, perhaps rather than the crossing point, you could use the next time after the crossing point as your metric. What's appropriate depends entirely on your purpose. Do note that that key here is that you need to find a point where the difference changes sign -- it's not likely to be zero at any point. One more thought -- you're looking at a moving average anyway -- with that smoothing already in place linear interpolation is probably fine, after all the exact crossing point is going to be a function of your smoothing parameters. You'll also want to think about what it means when the prices cross, then cross right back at the next time step, or even match, then the one that was higher goes back up... -Chris > Can you suggest me some books > around this subject. I will mainly try to build some indicator for the > stock market. > > If you can give me a clue I would be appreciated. > > Best Regards, > > > > On Tue, Mar 1, 2011 at 5:23 PM, Joe Kington > wrote: > > Hi Andre, > > Assuming that you want the exact point (date and value) where each > crossing occurs, you'll need to interpolate where they cross. > > There are a number of different ways to do so, but assuming you're > okay with linear interpolation, and everything's sampled on the same > dates, you can simply do something like this: > > import numpy as np > import matplotlib.pyplot as plt > > def main(): > x = np.linspace(0, 2*np.pi, 20) > y1 = np.sin(2*x) > y2 = np.cos(x) > crossings = find_crossings(x, y1, y2) > cross_x, cross_y = crossings.T > plt.plot(x, y1, 'bx-') > plt.plot(x, y2, 'gx-') > plt.plot(cross_x, cross_y, 'ro') > plt.show() > > def find_crossings(x, y1, y2): > diff = np.diff(np.sign(y1 - y2)) > indicies, = np.nonzero(diff) > crossings = [interpolate_crossing(i, x, y1, y2) for i in indicies] > return np.array(crossings) > > def interpolate_crossing(i, x, y1, y2): > slope = ( (y1[i] - y2[i]) > / ((y2[i+1] - y2[i]) - (y1[i+1] - y1[i]))) > x = x[i] + slope * (x[i+1] - x[i]) > y = y1[i] + slope * (y1[i+1] - y1[i]) > return x, y > > main() > > VXsqp.png > > On Tue, Mar 1, 2011 at 10:07 AM, Andre Lopes > wrote: > > Hi, > > I'm new to Numpy. I'm doing some tests with some Stock Market Quotes > > My struggle right now is "how to get the values of the moving > averages > crosses", I send an image in attach to illustrate what I'm trying to > get. > > I'm using the this computation to get when the moving averages > crosses, but when I look at the graph, the values doesn't seem ok. > > [quote] > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross = (dates[equal]) > prices_cross = (prices[equal]) > [/quote] > > > The full code is this: > [quote] > # Modules > import datetime > import numpy as np > import matplotlib.finance as finance > import matplotlib.mlab as mlab > import matplotlib.pyplot as plot > > # Define quote > startdate = datetime.date(2008,10,1) > today = enddate = datetime.date.today() > ticker = 'uso' > > # Catch CSV > fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) > > # From CSV to REACARRAY > r = mlab.csv2rec(fh); fh.close() > # Order by Desc > r.sort() > > > ### Methods Begin > def moving_average(x, n, type='simple'): > """ > compute an n period moving average. > > type is 'simple' | 'exponential' > > """ > x = np.asarray(x) > if type=='simple': > weights = np.ones(n) > else: > weights = np.exp(np.linspace(-1., 0., n)) > > weights /= weights.sum() > > > a = np.convolve(x, weights, mode='full')[:len(x)] > a[:n] = a[n] > return a > ### Methods End > > > prices = r.adj_close > dates = r.date > ma20 = moving_average(prices, 20, type='simple') > ma50 = moving_average(prices, 50, type='simple') > > # Get when the ma20 cross ma50 > equal = np.round(ma20,2)==np.round(ma50,2) > dates_cross = (dates[equal]) > prices_cross = (prices[equal]) > > # Ver se a ma20 > ma50 > # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) > # dates_ma20_greater_than_ma50 = (dates[ma20_greater_than_ma50]) > # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) > > print dates_cross > print prices_cross > #print dates_ma20_greater_than_ma50 > #print prices_ma20_greater_than_ma50 > > > plot.plot(prices) > plot.plot(ma20) > plot.plot(ma50) > plot.show() > [/quote] > > Someone can give me some clues? > > Best Regards, > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From lopes80andre at gmail.com Tue Mar 1 15:31:37 2011 From: lopes80andre at gmail.com (Andre Lopes) Date: Tue, 1 Mar 2011 20:31:37 +0000 Subject: [Numpy-discussion] How to get the prices of Moving Averages Crosses? In-Reply-To: <4D6D462E.5000709@noaa.gov> References: <4D6D462E.5000709@noaa.gov> Message-ID: Hi, Thanks for the reply. I think my problem will be to get some knowledge on some math subjects. What kind of subjects in math I need to get knowledge to be able to work in this field(indicators for financial products)? Best Regards, On Tue, Mar 1, 2011 at 7:17 PM, Christopher Barker wrote: > On 3/1/11 10:55 AM, Andre Lopes wrote: >> I'm not okay with linear interpolation. > > Well, odds are that the crossing point won't be exactly at a data point, > so you need to do SOME kind of interpolation. > > You could use a higher order interpolation (cubic spline, etc): see the > interpolation routines in scipy. > > However, for this use, perhaps rather than the crossing point, you could > use the next time after the crossing point as your metric. > > What's appropriate depends entirely on your purpose. > > Do note that that key here is that you need to find a point where the > difference changes sign -- it's not likely to be zero at any point. > > One more thought -- you're looking at a moving average anyway -- with > that smoothing already in place linear interpolation is probably fine, > after all the exact crossing point is going to be a function of your > smoothing parameters. > > You'll also want to think about what it means when the prices cross, > then cross right back at the next time step, or even match, then the one > that was higher goes back up... > > -Chris > > >> Can you suggest me some books >> around this subject. I will mainly try to build some indicator for the >> stock market. >> >> If you can give me a clue I would be appreciated. >> >> Best Regards, >> >> >> >> On Tue, Mar 1, 2011 at 5:23 PM, Joe Kington > > wrote: >> >> ? ? Hi Andre, >> >> ? ? Assuming that you want the exact point (date and value) where each >> ? ? crossing occurs, you'll need to interpolate where they cross. >> >> ? ? There are a number of different ways to do so, but assuming you're >> ? ? okay with linear interpolation, and everything's sampled on the same >> ? ? dates, you can simply do something like this: >> >> ? ? import numpy as np >> ? ? import matplotlib.pyplot as plt >> >> ? ? def main(): >> ? ? ? ? ?x = np.linspace(0, 2*np.pi, 20) >> ? ? ? ? ?y1 = np.sin(2*x) >> ? ? ? ? ?y2 = np.cos(x) >> ? ? ? ? ?crossings = find_crossings(x, y1, y2) >> ? ? ? ? ?cross_x, cross_y = crossings.T >> ? ? ? ? ?plt.plot(x, y1, 'bx-') >> ? ? ? ? ?plt.plot(x, y2, 'gx-') >> ? ? ? ? ?plt.plot(cross_x, cross_y, 'ro') >> ? ? ? ? ?plt.show() >> >> ? ? def find_crossings(x, y1, y2): >> ? ? ? ? ?diff = np.diff(np.sign(y1 - y2)) >> ? ? ? ? ?indicies, = np.nonzero(diff) >> ? ? ? ? ?crossings = [interpolate_crossing(i, x, y1, y2) for i in indicies] >> ? ? ? ? ?return np.array(crossings) >> >> ? ? def interpolate_crossing(i, x, y1, y2): >> ? ? ? ? ?slope = ( ? ? ? (y1[i] - y2[i]) >> ? ? ? ? ? ? ? ? / ((y2[i+1] - y2[i]) - (y1[i+1] - y1[i]))) >> ? ? ? ? ?x = x[i] + slope * (x[i+1] - x[i]) >> ? ? ? ? ?y = y1[i] + slope * (y1[i+1] - y1[i]) >> ? ? ? ? ?return x, y >> >> ? ? main() >> >> ? ? VXsqp.png >> >> ? ? On Tue, Mar 1, 2011 at 10:07 AM, Andre Lopes > ? ? > wrote: >> >> ? ? ? ? Hi, >> >> ? ? ? ? I'm new to Numpy. I'm doing some tests with some Stock Market Quotes >> >> ? ? ? ? My struggle right now is "how to get the values of the moving >> ? ? ? ? averages >> ? ? ? ? crosses", I send an image in attach to illustrate what I'm trying to >> ? ? ? ? get. >> >> ? ? ? ? I'm using the this computation to get when the moving averages >> ? ? ? ? crosses, but when I look at the graph, the values doesn't seem ok. >> >> ? ? ? ? [quote] >> ? ? ? ? # Get when the ma20 cross ma50 >> ? ? ? ? equal = np.round(ma20,2)==np.round(ma50,2) >> ? ? ? ? dates_cross ?= (dates[equal]) >> ? ? ? ? prices_cross = (prices[equal]) >> ? ? ? ? [/quote] >> >> >> ? ? ? ? The full code is this: >> ? ? ? ? [quote] >> ? ? ? ? # Modules >> ? ? ? ? import datetime >> ? ? ? ? import numpy as np >> ? ? ? ? import matplotlib.finance as finance >> ? ? ? ? import matplotlib.mlab as mlab >> ? ? ? ? import matplotlib.pyplot as plot >> >> ? ? ? ? # Define quote >> ? ? ? ? startdate = datetime.date(2008,10,1) >> ? ? ? ? today = enddate = datetime.date.today() >> ? ? ? ? ticker = 'uso' >> >> ? ? ? ? # Catch CSV >> ? ? ? ? fh = finance.fetch_historical_yahoo(ticker, startdate, enddate) >> >> ? ? ? ? # From CSV to REACARRAY >> ? ? ? ? r = mlab.csv2rec(fh); fh.close() >> ? ? ? ? # Order by Desc >> ? ? ? ? r.sort() >> >> >> ? ? ? ? ### Methods Begin >> ? ? ? ? def moving_average(x, n, type='simple'): >> ? ? ? ? """ >> ? ? ? ? ? ? compute an n period moving average. >> >> ? ? ? ? ? ? type is 'simple' | 'exponential' >> >> ? ? ? ? """ >> ? ? ? ? ? ? x = np.asarray(x) >> ? ? ? ? ? ? if type=='simple': >> ? ? ? ? ? ? ? ? weights = np.ones(n) >> ? ? ? ? ? ? else: >> ? ? ? ? ? ? ? ? weights = np.exp(np.linspace(-1., 0., n)) >> >> ? ? ? ? ? ? weights /= weights.sum() >> >> >> ? ? ? ? ? ? a = ?np.convolve(x, weights, mode='full')[:len(x)] >> ? ? ? ? ? ? a[:n] = a[n] >> ? ? ? ? ? ? return a >> ? ? ? ? ### Methods End >> >> >> ? ? ? ? prices = r.adj_close >> ? ? ? ? dates = r.date >> ? ? ? ? ma20 = moving_average(prices, 20, type='simple') >> ? ? ? ? ma50 = moving_average(prices, 50, type='simple') >> >> ? ? ? ? # Get when the ma20 cross ma50 >> ? ? ? ? equal = np.round(ma20,2)==np.round(ma50,2) >> ? ? ? ? dates_cross ?= (dates[equal]) >> ? ? ? ? prices_cross = (prices[equal]) >> >> ? ? ? ? # Ver se a ma20 > ma50 >> ? ? ? ? # ma20_greater_than_ma50 = np.round(ma20,2) > np.round(ma50,2) >> ? ? ? ? # dates_ma20_greater_than_ma50 ?= (dates[ma20_greater_than_ma50]) >> ? ? ? ? # prices_ma20_greater_than_ma50 = (prices[ma20_greater_than_ma50]) >> >> ? ? ? ? print dates_cross >> ? ? ? ? print prices_cross >> ? ? ? ? #print dates_ma20_greater_than_ma50 >> ? ? ? ? #print prices_ma20_greater_than_ma50 >> >> >> ? ? ? ? plot.plot(prices) >> ? ? ? ? plot.plot(ma20) >> ? ? ? ? plot.plot(ma50) >> ? ? ? ? plot.show() >> ? ? ? ? [/quote] >> >> ? ? ? ? Someone can give me some clues? >> >> ? ? ? ? Best Regards, >> >> ? ? ? ? _______________________________________________ >> ? ? ? ? NumPy-Discussion mailing list >> ? ? ? ? NumPy-Discussion at scipy.org >> ? ? ? ? http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> ? ? _______________________________________________ >> ? ? NumPy-Discussion mailing list >> ? ? NumPy-Discussion at scipy.org >> ? ? http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From amenity at enthought.com Tue Mar 1 16:31:32 2011 From: amenity at enthought.com (Amenity Applewhite) Date: Tue, 1 Mar 2011 15:31:32 -0600 Subject: [Numpy-discussion] =?utf-8?q?Webinar_3/4=3A_How_do_I=E2=80=A6_sol?= =?utf-8?q?ve_ODEs=3F_Part_II?= Message-ID: March EPD Webinar: How do I...solve ODEs? Part II This Friday, Warren Weckesser will present a second installment of his webinars on differential equations. We will explore two Python packages for solving boundary value problems. Both are packaged as scikits: scikits.bvp_solver, written by John Salvatier, is a wrapper of the BVP_SOLVER code by Shampine and Muir; scikits.bvp1lg, written by Pauli Virtanen, is a wrapper of the COLNEW solver developed by Ascher and Bader. Enthought Python Distribution Webinar How do I... solve ODEs? Part II Friday, March 4: 1pm CST/7pm UTC Wait list (for non EPD subscribers): send an email to amenity at enthought.com Thanks! _________________________ Amenity Applewhite Enthought, Inc. Scientific Computing Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonovik at gmail.com Tue Mar 1 16:36:28 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Tue, 1 Mar 2011 21:36:28 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: <4D6C42B8.7040902@molden.no> Message-ID: Robert Kern gmail.com> writes: > >> >> You can have each of those processes memory-map the whole file and > >> >> just operate on their own slices. Your operating system's virtual > >> >> memory manager should handle all of the details for you. > > > > Wow, I didn't know that. So as long as the ranges touched by each process do > > not overlap, I'll be safe? If I modify only a few discontiguous chunks in a > > range, will the virtual memory manager decide whether it is most efficient to > > write just the chunks or the entire range back to disk? > > It's up to the virtual memory manager, but usually, it will just load > those pages (chunks the size of mmap.PAGESIZE) that are touched by > your request and write them back. What if two processes touch adjacent chunks that are smaller than a page? Is there a risk that writing back an entire page will overwrite the efforts of another process? > > Use case: Generate "large" output for "many" parameter scenarios. > > 1. Preallocate "enormous" output file on disk. > > 2. Each process fills in part of the output. > > 3. Analyze, aggregate results, perhaps save to HDF or database, in a sliding- > > window fashion using a memory-mapped array. The aggregated results fit in > > memory, even though the raw output doesn't. [...] > Okay, in this case, I don't think that just adding an offset argument > to np.load() is very useful. You will want to read the dtype and shape > information from the header, *then* decide what offset and shape to > use for the memory-mapped segment. You will want to use the functions > read_magic() and read_array_header_1_0() from np.lib.format directly. Pardon me if I misunderstand, but isn't that what np.load does already, with or without my modifications? The existing np.load calls open_memmap if memory- mapping is requested. open_memmap does read the header first, using read_magic getting the shape and dtype from read_array_header_1_0(). It currently computes passes offset=fp.tell() to numpy.memmap. I just modify this offset based on the number of items to skip and the dtype's item size. > You can slightly modify the logic in open_memmap(): > > # Read the header of the file first. > fp = open(filename, 'rb') > try: > version = read_magic(fp) > if version != (1, 0): > msg = "only support version (1,0) of file format, not %r" > raise ValueError(msg % (version,)) > shape, fortran_order, dtype = read_array_header_1_0(fp) > if dtype.hasobject: > msg = "Array can't be memory-mapped: Python objects in dtype." > raise ValueError(msg) > offset = fp.tell() > finally: > fp.close() > > chunk_offset, chunk_shape = decide_offset_shape(dtype, shape, > fortran_order, offset) > > marray = np.memmap(filename, dtype=dtype, shape=chunk_shape, > order=('F' if fortran_order else 'C'), mode='r+', > offset=chunk_offset) To me this seems very equivalent to what my hack is doing, https://github.com/ jonovik/numpy/compare/master...offset_memmap. I guess decide_offset_shape() would encapsulate the gist of what I added. From robert.kern at gmail.com Tue Mar 1 17:04:54 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Mar 2011 16:04:54 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: <4D6C42B8.7040902@molden.no> Message-ID: On Tue, Mar 1, 2011 at 15:36, Jon Olav Vik wrote: > Robert Kern gmail.com> writes: >> >> >> You can have each of those processes memory-map the whole file and >> >> >> just operate on their own slices. Your operating system's virtual >> >> >> memory manager should handle all of the details for you. >> > >> > Wow, I didn't know that. So as long as the ranges touched by each process do >> > not overlap, I'll be safe? If I modify only a few discontiguous chunks in a >> > range, will the virtual memory manager decide whether it is most efficient > to >> > write just the chunks or the entire range back to disk? >> >> It's up to the virtual memory manager, but usually, it will just load >> those pages (chunks the size of mmap.PAGESIZE) that are touched by >> your request and write them back. > > What if two processes touch adjacent chunks that are smaller than a page? Is > there a risk that writing back an entire page will overwrite the efforts of > another process? I believe that there is only one page in main memory. Each process is simply pointed to the same page. As long as you don't write to the same specific byte, you'll be fine. >> > Use case: Generate "large" output for "many" parameter scenarios. >> > 1. Preallocate "enormous" output file on disk. >> > 2. Each process fills in part of the output. >> > 3. Analyze, aggregate results, perhaps save to HDF or database, in a > sliding- >> > window fashion using a memory-mapped array. The aggregated results fit in >> > memory, even though the raw output doesn't. > [...] >> Okay, in this case, I don't think that just adding an offset argument >> to np.load() is very useful. You will want to read the dtype and shape >> information from the header, *then* decide what offset and shape to >> use for the memory-mapped segment. You will want to use the functions >> read_magic() and read_array_header_1_0() from np.lib.format directly. > > Pardon me if I misunderstand, but isn't that what np.load does already, with or > without my modifications? With your modifications, the user does not get to see the header information before they pick the offset and shape. I contend that the user ought to read the shape information before deciding the shape to use. I don't think that changing the no.load() API is the best way to solve this problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From jonovik at gmail.com Tue Mar 1 18:06:51 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Tue, 1 Mar 2011 23:06:51 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: <4D6C42B8.7040902@molden.no> Message-ID: Robert Kern gmail.com> writes: > >> It's up to the virtual memory manager, but usually, it will just load > >> those pages (chunks the size of mmap.PAGESIZE) that are touched by > >> your request and write them back. > > > > What if two processes touch adjacent chunks that are smaller than a page? Is > > there a risk that writing back an entire page will overwrite the efforts of > > another process? > > I believe that there is only one page in main memory. Each process is > simply pointed to the same page. As long as you don't write to the > same specific byte, you'll be fine. Within a single machine, that sounds fine. What about processes running on different nodes, with different main memories? > > Pardon me if I misunderstand, but isn't that what np.load does already, with or > > without my modifications? > > With your modifications, the user does not get to see the header > information before they pick the offset and shape. I contend that the > user ought to read the shape information before deciding the shape to > use. Actually, that is what I've done for my own use (trivial parallelism, where I know that axis 0 is "long" and suitable for dividing the workload): Read the shape first, divide its first dimension into chunks with np.array_split(), then memmap the portion I need. I didn't submit that function for inclusion because it is rather specific to my own work. For process "ID" out of "NID", the code is roughly as follows: def memmap_chunk(filename, ID, NID, mode="r"): r = open_memmap(filename, "r") n = r.shape[0] i = np.array_split(range(n), NID)[ID] offset = i[0] shape = 1 + i[-1] - i[0] if len(i) > 0: return open_memmap(filename, mode=mode, offset=offset, shape=shape) else: return np.empty(0, r.dtype) > I don't think that changing the no.load() API is the best way to > solve this problem. I can agree with that. What I actually use is open_memmap() as shown above, but couldn't have done it without offset and shape arguments. In retrospect, changing np.load() was maybe a misstep in trying to generalize from my own hacks to something that might be useful to others. I kind of added offset and shape to np.load "for completeness", as it offers a mmap_mode argument but no way to memory-map just a portion of a file. So to attempt a summary: memory-mapping with np.load may be useful to conserve memory in a single process (with no need for offset and shape arguments), but splitting workload across multiple processes is best done with open_memmap. Then I humbly suggest that having offset and shape arguments to open_memmap is useful. From robert.kern at gmail.com Tue Mar 1 18:24:42 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Mar 2011 17:24:42 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: <4D6C42B8.7040902@molden.no> Message-ID: On Tue, Mar 1, 2011 at 17:06, Jon Olav Vik wrote: > Robert Kern gmail.com> writes: >> >> It's up to the virtual memory manager, but usually, it will just load >> >> those pages (chunks the size of mmap.PAGESIZE) that are touched by >> >> your request and write them back. >> > >> > What if two processes touch adjacent chunks that are smaller than a page? Is >> > there a risk that writing back an entire page will overwrite the efforts of >> > another process? >> >> I believe that there is only one page in main memory. Each process is >> simply pointed to the same page. As long as you don't write to the >> same specific byte, you'll be fine. > > Within a single machine, that sounds fine. What about processes running on > different nodes, with different main memories? You mean mmaping a file on a shared file system? Then it's up the file system. I'm honestly not sure what would happen for your particular file system. Try it and report back. In any case, using the offset won't help. The virtual memory manager always deals with whole pages of size mmap.ALLOCATIONBOUNDARY aligned with the start of the file. Under the covers, np.memmap() rounds the offset down to the nearest page boundary and then readjusts the pointer. For performance reasons, I don't recommend doing it anyway. The networked file system becomes the bottleneck, in my experience. >> > Pardon me if I misunderstand, but isn't that what np.load does already, > with or >> > without my modifications? >> >> With your modifications, the user does not get to see the header >> information before they pick the offset and shape. I contend that the >> user ought to read the shape information before deciding the shape to >> use. > > Actually, that is what I've done for my own use (trivial parallelism, where I > know that axis 0 is "long" and suitable for dividing the workload): Read the > shape first, divide its first dimension into chunks with np.array_split(), then > memmap the portion I need. I didn't submit that function for inclusion because > it is rather specific to my own work. For process "ID" out of "NID", the code > is roughly as follows: > > def memmap_chunk(filename, ID, NID, mode="r"): > ? ?r = open_memmap(filename, "r") > ? ?n = r.shape[0] > ? ?i = np.array_split(range(n), NID)[ID] > ? ?offset = i[0] > ? ?shape = 1 + i[-1] - i[0] > ? ?if len(i) > 0: > ? ? ? ?return open_memmap(filename, mode=mode, offset=offset, shape=shape) > ? ?else: > ? ? ? ?return np.empty(0, r.dtype) > >> I don't think that changing the no.load() API is the best way to >> solve this problem. > > I can agree with that. What I actually use is open_memmap() as shown above, but > couldn't have done it without offset and shape arguments. > > In retrospect, changing np.load() was maybe a misstep in trying to generalize > from my own hacks to something that might be useful to others. I kind of added > offset and shape to np.load "for completeness", as it offers a mmap_mode > argument but no way to memory-map just a portion of a file. > > So to attempt a summary: memory-mapping with np.load may be useful to conserve > memory in a single process (with no need for offset and shape arguments), but > splitting workload across multiple processes is best done with open_memmap. > Then I humbly suggest that having offset and shape arguments to open_memmap is > useful. I disagree. The important bit is to get the header information and the data offset out of the file without loading any data. Once you have that, np.memmap() suffices. You don't need to alter np.open_memmap() at all. In fact, if you do use np.open_memmap() to read the information, then you can't implement your "64-bit-large file on a 32-bit machine" use case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From jonovik at gmail.com Tue Mar 1 19:40:17 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Wed, 2 Mar 2011 00:40:17 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: <4D6C42B8.7040902@molden.no> Message-ID: Robert Kern gmail.com> writes: > > Within a single machine, that sounds fine. What about processes running on > > different nodes, with different main memories? > > You mean mmaping a file on a shared file system? Yes. GPFS, I believe, presumably this: http://en.wikipedia.org/wiki/GPFS Horrible latency on first access, but otherwise fast enough for my uses. I could have worked on local disk, copied them to my home directory, then consolidated the results, but the convenience of a single file appeals to my one-screenful attention span. > Then it's up the file > system. I'm honestly not sure what would happen for your particular > file system. Try it and report back. > > In any case, using the offset won't help. The virtual memory manager > always deals with whole pages of size mmap.ALLOCATIONBOUNDARY aligned > with the start of the file. Under the covers, np.memmap() rounds the > offset down to the nearest page boundary and then readjusts the > pointer. I have had 1440 processes writing timing information to a numpy file with about 60000 records of (starttime, finishtime) without problem. Likewise, I've written large amounts of output, which was sanity-checked during analysis. I ought to have noticed any errors. > For performance reasons, I don't recommend doing it anyway. The > networked file system becomes the bottleneck, in my experience. What would you suggest instead? Using separate files is an option, but requires a final pass to collect data, or lots of code to navigate the results. Having a master node collect data and write them to file is cumbersome on our queueing system (it's easier to schedule many small jobs that can run whenever there is free time, than require a master and workers to run at the same time). I don't recall the exact numbers, but I have had several hundred processors running simultaneously, writing to the same numpy file on disk. It has been my impression that this is much faster than doing it through a single process. I was hoping to get the speed of writing separate files with the self- documentation of a single structured np.array on disk (open_memmap also saves me a few lines of code in writing output back to disk). That was before I learned about how the virtual memory manager enters into memory-mapping, though -- maybe I was just imagining things 8-/ > > Then I humbly suggest that having offset and shape arguments to open_memmap is > > useful. > > I disagree. The important bit is to get the header information and the > data offset out of the file without loading any data. Once you have > that, np.memmap() suffices. You don't need to alter np.open_memmap() > at all. But if you're suggesting that the end user 1) use read_magic to check the version, 2) use read_array_header_1_0 to get shape, fortran_order, dtype, 3) call np.memmap with a suitable offset -- isn't that pretty much tantamount to duplicating np.lib.format.open_memmap? Actually, the function names read_array_header_1_0 and read_magic sound rather internal, not like something intended for an end-user. read_array_header_1_0 seems to be used only by open_memmap and read_array. Given the somewhat confusing array (pardon the pun) of ways to load files in Numpy, np.load() might in fact be a reasonable place to centralize all the options... > In fact, if you do use np.open_memmap() to read the > information, then you can't implement your "64-bit-large file on a > 32-bit machine" use case. Do you mean that it could be done with np.memmap? Pardon me for being slow, but what is the crucial difference between np.memmap and open_memmap in this respect? From sturla at molden.no Tue Mar 1 20:10:55 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 02 Mar 2011 02:10:55 +0100 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: <4D6C42B8.7040902@molden.no> Message-ID: <4D6D991F.9010905@molden.no> Den 01.03.2011 14:20, skrev Jon Olav Vik: > Use case: Generate "large" output for "many" parameter scenarios. > 1. Preallocate "enormous" output file on disk. > That's not a usecase, because this will in general require 64-bit, for which the offset parameter does not matter. > Maybe that is impossible with 32-bit Python: at least I cannot allocate that > big a file on my laptop. 32-bit Windows will give you 2 GB virtual memory available in user space. The reminding 2 GB is reserved for device drivers etc. I don't know about Linux, but it is approximately the same. Note that I am not talking about physical memory but virtual address space. I am not talking about RAM. When you memory map a file, you use up some of this virtual address space. That is the key. Because the 32-bit address space is so small by today's standard, we often cannot afford to memory map large portions of it. That is where the "offset" helps. Instead of memory mapping the whole file, we just work with a small window of it. But unlike a NumPy subarray view, this slice is in the kernel of the operating system. On 64-bit we have so much virtual memory that it does not matter. How much is system dependent. On recent AMD64 processors it is 256 TB, but I think Windows 64 "only" gives us 16 of those. Even so, this is still approximately 25 times the size of the hard disk on my computer. That is, with 64-bit Python I can memory map everything on my computer, and it would hardly be noticed in the virtual address space. That is why an offset is not needed. A typical usecase for "offset" is a 32-bit database server memory mapping a small vindow of a huge database. On 64-bit the offset could be ignored, and the whole database mapped to memory -- one of the reasons 64-bit database servers perform better. Sturla From robert.kern at gmail.com Tue Mar 1 20:13:31 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Mar 2011 19:13:31 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: <4D6C42B8.7040902@molden.no> Message-ID: On Tue, Mar 1, 2011 at 18:40, Jon Olav Vik wrote: > Robert Kern gmail.com> writes: >> > Within a single machine, that sounds fine. What about processes running on >> > different nodes, with different main memories? >> >> You mean mmaping a file on a shared file system? > > Yes. GPFS, I believe, presumably this: > http://en.wikipedia.org/wiki/GPFS > Horrible latency on first access, but otherwise fast enough for my uses. I > could have worked on local disk, copied them to my home directory, then > consolidated the results, but the convenience of a single file appeals to my > one-screenful attention span. > >> Then it's up the file >> system. I'm honestly not sure what would happen for your particular >> file system. Try it and report back. >> >> In any case, using the offset won't help. The virtual memory manager >> always deals with whole pages of size mmap.ALLOCATIONBOUNDARY aligned >> with the start of the file. Under the covers, np.memmap() rounds the >> offset down to the nearest page boundary and then readjusts the >> pointer. > > I have had 1440 processes writing timing information to a numpy file with about > 60000 records of (starttime, finishtime) without problem. Likewise, I've > written large amounts of output, which was sanity-checked during analysis. I > ought to have noticed any errors. > >> For performance reasons, I don't recommend doing it anyway. The >> networked file system becomes the bottleneck, in my experience. > > What would you suggest instead? Using separate files is an option, but requires > a final pass to collect data, or lots of code to navigate the results. Having a > master node collect data and write them to file is cumbersome on our queueing > system (it's easier to schedule many small jobs that can run whenever there is > free time, than require a master and workers to run at the same time). > > I don't recall the exact numbers, but I have had several hundred processors > running simultaneously, writing to the same numpy file on disk. It has been my > impression that this is much faster than doing it through a single process. I > was hoping to get the speed of writing separate files with the self- > documentation of a single structured np.array on disk (open_memmap also saves > me a few lines of code in writing output back to disk). > > That was before I learned about how the virtual memory manager enters into > memory-mapping, though -- maybe I was just imagining things 8-/ Well, if it's working for you, that's great! >> > Then I humbly suggest that having offset and shape arguments to open_memmap > is >> > useful. >> >> I disagree. The important bit is to get the header information and the >> data offset out of the file without loading any data. Once you have >> that, np.memmap() suffices. You don't need to alter np.open_memmap() >> at all. > > But if you're suggesting that the end user > 1) use read_magic to check the version, > 2) use read_array_header_1_0 to get shape, fortran_order, dtype, > 3) call np.memmap with a suitable offset > -- isn't that pretty much tantamount to duplicating np.lib.format.open_memmap? > > Actually, the function names read_array_header_1_0 and read_magic sound rather > internal, not like something intended for an end-user. read_array_header_1_0 > seems to be used only by open_memmap and read_array. That's not what I suggested. I suggested that 1+2 could be wrapped in a single function read_header() that provides the header information and the offset to the data. > Given the somewhat > confusing array (pardon the pun) of ways to load files in Numpy, np.load() > might in fact be a reasonable place to centralize all the options... np.load()'s primary purpose is to be the one *simple* way to access NPY files, not the one place to expose every way to access NPY files. >> In fact, if you do use np.open_memmap() to read the >> information, then you can't implement your "64-bit-large file on a >> 32-bit machine" use case. > > Do you mean that it could be done with np.memmap? Pardon me for being slow, but > what is the crucial difference between np.memmap and open_memmap in this > respect? You cannot use open_memmap() to read the header information on the 32-bit system since it will also try to map the data portion of the too-large file. If you can read the header information with something that does not try to read the data, then you can select a smaller shape that does fit into your 32-bit address space. It's not that np.memmap() works where open_memmap() doesn't; it's that my putative read_header() function would work where open_memmap() doesn't. My point about np.memmap() is that once you have the header information loaded, you don't need to use open_memmap() any more. np.memmap() does everything you need from that point on. There is no point in making open_memmap() or np.load() more flexible to support this use case. You just need the read_header() function. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ben.root at ou.edu Wed Mar 2 17:21:23 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 2 Mar 2011 16:21:23 -0600 Subject: [Numpy-discussion] ticket 1562 on loadtxt (was Re: Numpy 2.0 schedule) In-Reply-To: <4D6BDF36.3080903@gmail.com> References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> <4D6BB2D5.6010809@gmail.com> <4D6BBE80.608@gmail.com> <4D6BDF36.3080903@gmail.com> Message-ID: On Mon, Feb 28, 2011 at 11:45 AM, Bruce Southey wrote: > On 02/28/2011 09:47 AM, Benjamin Root wrote: > > On Mon, Feb 28, 2011 at 9:25 AM, Bruce Southey wrote: > >> On 02/28/2011 09:02 AM, Benjamin Root wrote: >> [snip] >> > >> > >> > So, is there still no hope in addressing this old bug report of mine? >> > >> > http://projects.scipy.org/numpy/ticket/1562 >> > >> > Ben Root >> > >> I think you need to add more details to this. So do you have an example >> of the problem that includes code and expected output? >> >> Perhaps genfromtxt is probably more appropriate than loadtxt for what >> you want: >> >> from StringIO import StringIO >> import numpy as np >> t = StringIO("1,1.3,abcde\n2,2.3,wxyz\n1\n3,3.3,mnop") >> data = np.genfromtxt(t, >> [('myint','i8'),('myfloat','f8'),('mystring','S5')], names = >> ['myint','myfloat','mystring'], delimiter=",", invalid_raise=False) >> print 'Bad data raise\n',data >> >> This gives the output that skips the incomplete 3rd line: >> >> /usr/lib64/python2.7/site-packages/numpy/lib/npyio.py:1507: >> ConversionWarning: Some errors were detected ! >> Line #3 (got 1 columns instead of 3) >> warnings.warn(errmsg, ConversionWarning) >> Bad data raise >> [(1, 1.3, 'abcde') (2, 2.3, 'wxyz') (3, 3.3, 'mnop')] >> >> >> Bruce >> >> > Bruce, > > I think you mis-understood the problem I was reporting. > > Probably - which is why I asked for more details. > > You can find the discussion thread here: > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg26235.html > > I have proposed that at the very least, an example of this problem is added > to the documentation of loadtxt so that users know to be aware of this > possibility. > > I did not connect the ticket to that email thread. Removing the structured > array part of your email, I think essentially the argument is which should > be the output of: > np.loadtxt(StringIO("89.23")) > np.arange(5)[1] > > These return an 0-d array and an rather old argument about that (which may > address the other part of the ticket). Really I see this behavior as > standard so you add an example to the documentation to reflect that. > > I agree that this behavior has become standard, and, by-and-large, desirable. It just comes with this sneaky pitfall when encountering single-line files. Therefore, I have a couple of suggestions that I would find suitable for resolution of this report. I will leave it up to the developers to decide which course to pursue. 1. Add a "mindims" parameter that would default to None (for current behavior). The caller can specify the minimum number of dimensions the resulting array should have and then call some sort of function like np.atleast_nd() (I know it doesn't exists, but such a function might be useful). The documentation for this keyword param would allude to the rational for its use. 2. Keep the current behavior, but possibly not for when a dtype is specified. Given that the squeeze() was meant for addressing the situation where the data structure is not known a priori, squeezing a known dtype seems to go against this rationale. 3. Keep the current behavior, but add some documentation for loadtxt() that illustrates the problem and shows the usage of a function like np.atleast_2d(). I would be willing to write up such an example. > > In addition, loadtxt fails on empty files even when provided with a > dtype. I believe genfromtxt also fails as well in this case. > > Ben Root > > Errors on empty files probably should be a new bug report as that was > not in the ticket. > > Done: http://projects.scipy.org/numpy/ticket/1752 Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ater1980 at gmail.com Wed Mar 2 17:25:14 2011 From: ater1980 at gmail.com (Alex Ter-Sarkissov) Date: Thu, 3 Mar 2011 11:25:14 +1300 Subject: [Numpy-discussion] (no subject) Message-ID: hi, the question is probably very silly, but can't get my head around it Say I have an NxM numerical array. What I want is to obtain the row and column number of the smallest value(kinda like find command in Matlab). I use something like where(min(array_name)), but keep getting the error message. I'd be glad if any1 could have any suggestions in this regard. cheers, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Wed Mar 2 17:28:30 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 2 Mar 2011 17:28:30 -0500 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: On Wed, Mar 2, 2011 at 5:25 PM, Alex Ter-Sarkissov wrote: > hi, the question is probably very silly, but can't get my head around it > > Say I have an NxM numerical array. What I want is to obtain the row and > column number of the smallest value(kinda like find command in Matlab). I > use something like where(min(array_name)), but keep getting the error > message. I'd be glad if any1 could have any suggestions in this regard. > In [1]: import numpy as np In [2]: A = np.random.randn(10,10) In [3]: np.where(A==np.min(A)) Out[3]: (array([5]), array([3])) Skipper From robert.kern at gmail.com Wed Mar 2 17:29:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 2 Mar 2011 16:29:27 -0600 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: On Wed, Mar 2, 2011 at 16:25, Alex Ter-Sarkissov wrote: > hi, the question is probably very silly, but can't get my head around it > > Say I have an NxM numerical array. What I want is to obtain the row and > column number of the smallest value(kinda like find command in Matlab). I > use something like where(min(array_name)), but keep getting the error > message. I'd be glad if any1 could have any suggestions in this regard. [~] |15> A = np.random.random([3,5]) [~] |16> i = A.min(axis=1).argmin() [~] |17> j = A.min(axis=0).argmin() [~] |18> A[i,j] 0.0065380564732848701 [~] |19> A.min() 0.0065380564732848701 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Wed Mar 2 22:53:14 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 2 Mar 2011 20:53:14 -0700 Subject: [Numpy-discussion] ticket #1619 Message-ID: I think that this ticket can benefit from some discussion on the list. The problem is as follows. In [2]: np.dtype('i1, i4, i1', align=True) Out[2]: dtype([('f0', '|i1'), ('', '|V3'), ('f1', ' in () ValueError: two fields with the same name Note that the second field in the dtype is inserted for alignment purposes and isn't named. However, the list in the dtype can not be used to define a dtype because the empty name is replaced by 'f1', which conflicts with the following field. The patch attached to the ticket has a rather complicated work around for this that I would rather avoid. I would prefer some sort of reserved name for alignment fields, maybe something like '*0', '*1', etc. Another question I have is if alignment is supposed to be preserved across architectures, that is, should it be possible to pickle/savez on one architecture and read things in on another and still have the elements aligned. I suspect this isn't worth doing, but one way or the other should be decided. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Mar 3 00:57:48 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 3 Mar 2011 13:57:48 +0800 Subject: [Numpy-discussion] polynomial: polypow et al not exported Message-ID: Hi Charles, Is it intentional that chebpow/legpow/polypow are not in __all__? That is breaking crosslinks in the docs and it looks to me like it's just an oversight. Cheers, Ralf From mail.till at gmx.de Thu Mar 3 08:37:46 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Thu, 3 Mar 2011 13:37:46 +0000 (UTC) Subject: [Numpy-discussion] (no subject) References: Message-ID: Alex Ter-Sarkissov gmail.com> writes: > > hi, the question is probably very silly, but can't get my head around itSay > I have an NxM numerical array. What I want is to obtain the row and > column number of the smallest value(kinda like find command in Matlab). > I use something like where(min(array_name)), but keep getting the error > message. I'd be glad if any1 could have any suggestions in this regard.cheers,Alex ? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > In [24]: a=np.random.rand(3,3) In [25]: a Out[25]: array([[ 0.77163699, 0.92324147, 0.20572183], [ 0.75230562, 0.06454533, 0.23930423], [ 0.9312196 , 0.62750468, 0.02125804]]) In [26]: np.unravel_index(a.argmin(),a.shape) Out[26]: (2, 2) From blattnem at gmail.com Thu Mar 3 08:54:28 2011 From: blattnem at gmail.com (Marcel Blattner) Date: Thu, 3 Mar 2011 14:54:28 +0100 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: ...for example: A = np.random.rand(5,5) In [366]: A Out[366]: array([[ 0.36380049, 0.26440478, 0.8515609 , 0.07893608, 0.48084575], [ 0.71133527, 0.90912083, 0.14812865, 0.23223621, 0.49983985], [ 0.51668793, 0.73303799, 0.18620246, 0.52968823, 0.51904697], [ 0.3988893 , 0.60350318, 0.63808109, 0.38470785, 0.48923821], [ 0.02427049, 0.80382225, 0.42494655, 0.8986108 , 0.39879547]]) In [367]: np.nonzero(A==np.min(A)) Out[367]: (array([4]), array([0])) And this may be of interest for you: http://www.scipy.org/NumPy_for_Matlab_Users cheers Marcel On Wed, Mar 2, 2011 at 11:25 PM, Alex Ter-Sarkissov wrote: > hi, the question is probably very silly, but can't get my head around it > > Say I have an NxM numerical array. What I want is to obtain the row and > column number of the smallest value(kinda like find command in Matlab). I > use something like where(min(array_name)), but keep getting the error > message. I'd be glad if any1 could have any suggestions in this regard. > > cheers, > > Alex > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 3 09:21:49 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 3 Mar 2011 07:21:49 -0700 Subject: [Numpy-discussion] polynomial: polypow et al not exported In-Reply-To: References: Message-ID: On Wed, Mar 2, 2011 at 10:57 PM, Ralf Gommers wrote: > Hi Charles, > > Is it intentional that chebpow/legpow/polypow are not in __all__? That > is breaking crosslinks in the docs and it looks to me like it's just > an oversight. > > It's an oversight. I'll fix it up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhyams at gmail.com Fri Mar 4 01:19:45 2011 From: dhyams at gmail.com (Daniel Hyams) Date: Fri, 4 Mar 2011 01:19:45 -0500 Subject: [Numpy-discussion] Taking a large number of dot products at once Message-ID: This is probably so easy, I'm embarrassed to ask it...but I've been casting around trying things to no avail for the last hour and a half, so here goes... I have a lot of dot products to take. The length-3 vectors that I want to dot are stacked in a 2D array like this: U = [u1 u2 u3....] and V = [v1 v2 v3....] So both of these arrays, are, say, 3x100 each. I just want to take the dot product of each of the corresponding vectors, so that the result is [u1.v1 u2.v2 u3.v3 ....] which would be a 1x100 array in this case. Which function do I need to use? I thought tensordot() was the one, but I couldn't make it work....pure user error I'm sure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Mar 4 01:54:55 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 4 Mar 2011 14:54:55 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers wrote: > On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey wrote: >> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>> Hi, >>> >>> On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant ?wrote: >>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would >>>> seem) have additional work and features that do not need to wait for the >>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>> If someone is willing to manage the release of NumPy 1.6, then it sounds >>>> like a great idea to me. >>> This thread ended without a conclusion a month ago. Now I think master >>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>> issues and segfault on OS X fixed, more testing of changes), and I >>> have a better idea of my free time for March/April. Basically, I have >>> a good amount of time for the next couple of weeks, and not so much at >>> the end of March / first half of April due to an inter-continental >>> move. But I think we can get out a beta by mid-March, and I can manage >>> the release. >>> >>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >>> #1748 (blocker: regression for astype('str')) >>> #1619 (issue with dtypes, with patch) >>> #1749 (distutils, py 3.2) >>> #1601 (distutils, py 3.2) >>> #1622 (Solaris segfault, with patch) >>> #1713 (Solaris segfault) >>> #1631 (Solaris segfault) The distutils tickets are resolved. >>> Proposed schedule: >>> March 15: beta 1 >>> March 28: rc 1 >>> April 17: rc 2 (if needed) >>> April 24: final release Any comments on the schedule or tickets? Before the first beta can be released I think #1748 should be fixed. Before the first RC the Solaris segfaults should be investigated, and documentation for the new iterator (Python docstrings and C API docs) and datetime should be written. Also, some testing on 64-bit Windows would be great, that usually turns up new issues so the sooner the better. Ralf From e.antero.tammi at gmail.com Fri Mar 4 02:29:37 2011 From: e.antero.tammi at gmail.com (eat) Date: Fri, 4 Mar 2011 09:29:37 +0200 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: References: Message-ID: Hi, On Fri, Mar 4, 2011 at 8:19 AM, Daniel Hyams wrote: > This is probably so easy, I'm embarrassed to ask it...but I've been casting > around trying things to no avail for the last hour and a half, so here > goes... > > I have a lot of dot products to take. The length-3 vectors that I want to > dot are stacked in a 2D array like this: > > U = [u1 u2 u3....] > > and > > V = [v1 v2 v3....] > > So both of these arrays, are, say, 3x100 each. I just want to take the dot > product of each of the corresponding vectors, so that the result is > > [u1.v1 u2.v2 u3.v3 ....] > > which would be a 1x100 array in this case. > > Which function do I need to use? I thought tensordot() was the one, but I > couldn't make it work....pure user error I'm sure. > No function needed for this case, just: In []: x= rand(3, 7) In []: y= rand(3, 7) In []: d= (x* y).sum(0) In [490]: d Out[490]: array([ 1.25404683, 0.19113117, 1.37267133, 0.74219888, 1.55296562, 0.15264303, 0.72039922]) In [493]: dot(x[:, 0].T, y[:, 0]) Out[493]: 1.2540468282421895 Regards, eat > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Fri Mar 4 04:00:09 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 04 Mar 2011 01:00:09 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: <4D70AA19.8000509@uci.edu> On 3/3/2011 10:54 PM, Ralf Gommers wrote: > On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers > wrote: >> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey wrote: >>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>>> Hi, >>>> >>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant wrote: >>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would >>>>> seem) have additional work and features that do not need to wait for the >>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>>> If someone is willing to manage the release of NumPy 1.6, then it sounds >>>>> like a great idea to me. >>>> This thread ended without a conclusion a month ago. Now I think master >>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>>> issues and segfault on OS X fixed, more testing of changes), and I >>>> have a better idea of my free time for March/April. Basically, I have >>>> a good amount of time for the next couple of weeks, and not so much at >>>> the end of March / first half of April due to an inter-continental >>>> move. But I think we can get out a beta by mid-March, and I can manage >>>> the release. >>>> >>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >>>> #1748 (blocker: regression for astype('str')) >>>> #1619 (issue with dtypes, with patch) >>>> #1749 (distutils, py 3.2) >>>> #1601 (distutils, py 3.2) >>>> #1622 (Solaris segfault, with patch) >>>> #1713 (Solaris segfault) >>>> #1631 (Solaris segfault) > > The distutils tickets are resolved. > >>>> Proposed schedule: >>>> March 15: beta 1 >>>> March 28: rc 1 >>>> April 17: rc 2 (if needed) >>>> April 24: final release > > Any comments on the schedule or tickets? > > Before the first beta can be released I think #1748 should be fixed. > Before the first RC the Solaris segfaults should be investigated, and > documentation for the new iterator (Python docstrings and C API docs) > and datetime should be written. > > Also, some testing on 64-bit Windows would be great, that usually > turns up new issues so the sooner the better. > > Ralf Hi Ralf, the numpy master branch on github can not be compiled with Visual Studio. A patch is attached. I'll test the builds tomorrow. Christoph -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: msvc.diff URL: From hector1618 at gmail.com Fri Mar 4 04:13:00 2011 From: hector1618 at gmail.com (Hector) Date: Fri, 4 Mar 2011 14:43:00 +0530 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: References: Message-ID: On Fri, Mar 4, 2011 at 12:59 PM, eat wrote: > Hi, > > On Fri, Mar 4, 2011 at 8:19 AM, Daniel Hyams wrote: > >> This is probably so easy, I'm embarrassed to ask it...but I've been >> casting around trying things to no avail for the last hour and a half, so >> here goes... >> >> I have a lot of dot products to take. The length-3 vectors that I want to >> dot are stacked in a 2D array like this: >> >> U = [u1 u2 u3....] >> >> and >> >> V = [v1 v2 v3....] >> >> So both of these arrays, are, say, 3x100 each. I just want to take the >> dot product of each of the corresponding vectors, so that the result is >> >> [u1.v1 u2.v2 u3.v3 ....] >> >> which would be a 1x100 array in this case. >> >> Which function do I need to use? I thought tensordot() was the one, but I >> couldn't make it work....pure user error I'm sure. >> > Hello Daniel Hyams and group, I guess the following code serves the purpose but I think something needs to be highlighted. I think Eat forgot to paste it here. No function needed for this case, just: > In [ ] : import numpy.random as rand > In []: x= rand(3, 7) > In []: y= rand(3, 7) > In []: d= (x* y).sum(0) > In [490]: d > Out[490]: > array([ 1.25404683, 0.19113117, 1.37267133, 0.74219888, 1.55296562, > 0.15264303, 0.72039922]) > In [493]: dot(x[:, 0].T, y[:, 0]) > Out[493]: 1.2540468282421895 > > Regards, > eat > >> >> >> I guess you can use 'rand' only this after import. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- -Regards Hector Whenever you think you can or you can't, in either way you are right. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Mar 4 06:18:27 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Mar 2011 06:18:27 -0500 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: References: Message-ID: On Fri, Mar 4, 2011 at 4:13 AM, Hector wrote: > > > On Fri, Mar 4, 2011 at 12:59 PM, eat wrote: >> >> Hi, >> >> On Fri, Mar 4, 2011 at 8:19 AM, Daniel Hyams wrote: >>> >>> This is probably so easy, I'm embarrassed to ask it...but I've been >>> casting around trying things to no avail for the last hour and a half, so >>> here goes... >>> I have a lot of dot products to take. ?The length-3 vectors that I want >>> to dot are stacked in a 2D array like this: >>> U = [u1 u2 u3....] >>> and >>> >>> V = [v1 v2 v3....] >>> So both of these arrays, are, say, 3x100 each. ?I just want to take the >>> dot product of each of the corresponding vectors, so that the result is >>> [u1.v1 u2.v2 ?u3.v3 ....] >>> which would be a 1x100 array in this case. >>> Which function do I need to use? ?I thought tensordot() was the one, but >>> I couldn't make it work....pure user error I'm sure. > > Hello Daniel Hyams and group, > > I guess the following code serves the purpose but I think something needs to > be highlighted. I think Eat forgot to paste it here. > > >> No function needed for this case, just: > > ? ? In [ ] : import numpy.random as rand >> >> In []: x= rand(3, 7) >> In []: y= rand(3, 7) >> In []: d= (x* y).sum(0) >> In [490]: d >> Out[490]: >> array([ 1.25404683, ?0.19113117, ?1.37267133, ?0.74219888, ?1.55296562, >> ?? ? ? ?0.15264303, ?0.72039922]) >> In [493]: dot(x[:, 0].T, y[:, 0]) >> Out[493]: 1.2540468282421895 >> Regards, >> eat >>> >>> > > I guess you can use 'rand' only this after? import. to clarify: random is the module, rand is a function in random >>> import numpy.random as rand >>> rand(3,7) Traceback (most recent call last): File "", line 1, in rand(3,7) TypeError: 'module' object is not callable >>> from numpy.random import rand >>> rand(3,7) array([[ 3.424 ........... Josef > >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > -Regards > Hector > Whenever you think you can or you can't, in either way you are right. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From sturla at molden.no Fri Mar 4 08:55:35 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 04 Mar 2011 14:55:35 +0100 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: References: Message-ID: <4D70EF57.5030402@molden.no> Den 04.03.2011 07:19, skrev Daniel Hyams: > This is probably so easy, I'm embarrassed to ask it...but I've been > casting around trying things to no avail for the last hour and a half, > so here goes... > > I have a lot of dot products to take. The length-3 vectors that I > want to dot are stacked in a 2D array like this: > > U = [u1 u2 u3....] > > and > V = [v1 v2 v3....] > > So both of these arrays, are, say, 3x100 each. That is the diagonal of matrix product U * V.T, that is np.diag(np.dot(U,V.T)) You can also write this as np.sum(U*V, axis=1) The latter is faster with 100 x 3 arrays: >>> timeit.Timer(stmt='np.diag(np.dot(a,a.T))', setup='import numpy as np; a=np.zeros((100,3))').timeit(number=1000) 0.0568983632238087 >>> timeit.Timer(stmt='np.sum(a*a,axis=1)', setup='import numpy as np; a=np.zeros((100,3))').timeit(number=1000) 0.03596760426239598 We can do this with Python lists of NumPy arrays as well, avoiding stacking the arrays in a 2D structure: [np.dot(u,v.T) for u,v in zip(U,V)] with U and V lists of 1D ndarrays instead of 2D ndarrays. >>> timeit.Timer(stmt='[np.dot(x.T,x) for x in a]', setup='import numpy as np; a=100*[np.zeros(3)]').timeit(number=1000) 0.21054868368582902 While this is slower, we don't need to create the 2D stacks, and it only took 200 ?s on average. It could be useful is vectors are so big (or plentyful) that memory is an issue, or just to write more Pythonic code. We don't need to write FORTRAN in Python, computers are fast. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Mar 4 09:01:49 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 04 Mar 2011 15:01:49 +0100 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: <4D70EF57.5030402@molden.no> References: <4D70EF57.5030402@molden.no> Message-ID: <4D70F0CD.3060104@molden.no> Den 04.03.2011 14:55, skrev Sturla Molden: > > We can do this with Python lists of NumPy arrays as well, avoiding > stacking the arrays in a 2D structure: > > [np.dot(u,v.T) for u,v in zip(U,V)] > Damn it, bad style. Inner product, not outer... [np.dot(u.T,v) for u,v in zip(U,V)] or just [np.dot(u,v) for u,v in zip(U,V)] While it does not matter for 1D vectors, it would for rank 1 x n matrices. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Mar 4 09:19:29 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 04 Mar 2011 09:19:29 -0500 Subject: [Numpy-discussion] generic slicing question Message-ID: This will subtract col 0 from all cols of a 2-d array: # subtract col 0 from all cols def ll2llr (m): return m - m[:,0][:,np.newaxis] In general, how to subtract the (n-1)d slice at index 0 from all elements of an n-d array? (I think a 3d version is: m - m[:,:,0][:,:,np.newaxis] ) From sturla at molden.no Fri Mar 4 09:19:46 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 04 Mar 2011 15:19:46 +0100 Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: <4D70EF57.5030402@molden.no> References: <4D70EF57.5030402@molden.no> Message-ID: <4D70F502.6040802@molden.no> Den 04.03.2011 14:55, skrev Sturla Molden: > or just to write more Pythonic code. We don't need to write FORTRAN in > Python, computers are fast. > Don't be afraid of Python containers and for-loops. They are not nearly as evil as Matlab's for loops used to be. Also operations like appending tend to be O(1) in Python, but often O(N) in Matlab. In Python we don't need to preallocate space for a list or stack, and manually hoist it out of the loop. Replacing Python lists, sets, dicts, tuples, etc. with NumPy ndarrays or record arrays, and pretending to use FORTRAN or Matlab, can be silly. Try it before you conclude it's slow and write twice the amount of code you need to. Those coming to NumPy from Matlab might not be aware of this. It's never wise to write Matlab in Python, we have Matlab for that. Nor is it wise to write Python in Matlab -- the Pythonic style is generally the one recommended against when using Matlab. Speed considerations are very different. Another realated sillyness I've seen is using NumPy record arrays to avoid Python dicts. A record array is nice if we need to memory map a file with an underlying C struct, and parse structured binary data. But using it to avoid Python dicts is dumb -- all Python objects have a __dict__ anyway. I think it's the Matlab way of solving problems that make some NumPy users think like this. We are programming Python, not Matlab. Sorry for the rant. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Mar 4 09:24:02 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 04 Mar 2011 15:24:02 +0100 Subject: [Numpy-discussion] generic slicing question In-Reply-To: References: Message-ID: <4D70F602.8020303@molden.no> Den 04.03.2011 15:19, skrev Neal Becker: > This will subtract col 0 from all cols of a 2-d array: > > # subtract col 0 from all cols > def ll2llr (m): > return m - m[:,0][:,np.newaxis] > > > In general, how to subtract the (n-1)d slice at index 0 from all elements of an > n-d array? > > (I think a 3d version is: > m - m[:,:,0][:,:,np.newaxis] > ) m - m[...,0][...,np.newaxis] Sturla From cgohlke at uci.edu Fri Mar 4 14:54:07 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 04 Mar 2011 11:54:07 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D70AA19.8000509@uci.edu> References: <4D70AA19.8000509@uci.edu> Message-ID: <4D71435F.2000403@uci.edu> On 3/4/2011 1:00 AM, Christoph Gohlke wrote: > > > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >> wrote: >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >>> wrote: >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>>>> Hi, >>>>> >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >>>>> Oliphant wrote: >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it >>>>>> would >>>>>> seem) have additional work and features that do not need to wait >>>>>> for the >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>>>> If someone is willing to manage the release of NumPy 1.6, then it >>>>>> sounds >>>>>> like a great idea to me. >>>>> This thread ended without a conclusion a month ago. Now I think master >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>>>> issues and segfault on OS X fixed, more testing of changes), and I >>>>> have a better idea of my free time for March/April. Basically, I have >>>>> a good amount of time for the next couple of weeks, and not so much at >>>>> the end of March / first half of April due to an inter-continental >>>>> move. But I think we can get out a beta by mid-March, and I can manage >>>>> the release. >>>>> >>>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >>>>> #1748 (blocker: regression for astype('str')) >>>>> #1619 (issue with dtypes, with patch) >>>>> #1749 (distutils, py 3.2) >>>>> #1601 (distutils, py 3.2) >>>>> #1622 (Solaris segfault, with patch) >>>>> #1713 (Solaris segfault) >>>>> #1631 (Solaris segfault) >> >> The distutils tickets are resolved. >> >>>>> Proposed schedule: >>>>> March 15: beta 1 >>>>> March 28: rc 1 >>>>> April 17: rc 2 (if needed) >>>>> April 24: final release >> >> Any comments on the schedule or tickets? >> >> Before the first beta can be released I think #1748 should be fixed. >> Before the first RC the Solaris segfaults should be investigated, and >> documentation for the new iterator (Python docstrings and C API docs) >> and datetime should be written. >> >> Also, some testing on 64-bit Windows would be great, that usually >> turns up new issues so the sooner the better. >> >> Ralf > > Hi Ralf, > > the numpy master branch on github can not be compiled with Visual > Studio. A patch is attached. I'll test the builds tomorrow. > > Christoph > I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. There are few test failures (listed below) that look familiar. I also ran tests and/or examples of a few 3rd party packages that were built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. Most packages don't have any apparent problems. Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test failure/error (also listed below). There is a problem with code generated by Cython 0.13: pytables-2.2.1 and pandas-0.3.0, which were built with Cython 0.13, report several failures and do crash during the tests. This can probably be fixed by "recythonizing" with Cython 0.14.1. Christoph numpy-1.6.1.dev-win-amd64-py2.7 ====================================================================== ERROR: Ticket #99 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_regression.py", line 147, in test_intp np.intp('0x' + 'f'*i_width,16) TypeError: function takes at most 1 argument (2 given) ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 248, in test_special_values assert_almost_equal(np.log(x), y) File "X:\Python27-x64\lib\site-packages\numpy\testing\utils.py", line 443, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: array([ nan+2.35619449j]) DESIRED: (inf+2.356194490192345j) numpy-1.6.1.dev-win-amd64-py3.2 ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 588, in assert_array_compare assert_array_equal(xnanid, ynanid) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 687, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 618, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 100.0%) x: array([ True], dtype=bool) y: array(False, dtype=bool) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 440, in assert_almost_equal assert_almost_equal(actualr, desiredr, decimal=decimal) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 447, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 775, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) AssertionError: Arrays are not almost equal to 7 decimals (x and y nan location mismatch [ True], False mismatch) x: array([ nan]) y: array(inf) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "X:\Python32\lib\site-packages\numpy\testing\decorators.py", line 147, in skipper_func return f(*args, **kwargs) File "X:\Python32\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 248, in test_special_values assert_almost_equal(np.log(x), y) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 443, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: array([ nan+2.35619449j]) DESIRED: (inf+2.356194490192345j) numpy-1.6.1.dev-win32-py2.7 numpy-1.6.1.dev-win32-py3.2 ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\numpy\core\tests\test_numeric.py", line 320, in test_floating_exceptions np.power, ftype(2), ftype(2**fi.nexp)) File "X:\Python27\lib\site-packages\numpy\core\tests\test_numeric.py", line 261, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "X:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error 'overflow'. ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "X:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 168, in test_special_values self.assertRaises(FloatingPointError, np.log, x) AssertionError: FloatingPointError not raised scipy-0.9.0-win-amd64-py3.2 ====================================================================== FAIL: minimum position 7 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python32\lib\site-packages\nose\case.py", line 188, in runTest self.test(*self.arg) File "X:\Python32\lib\site-packages\scipy\ndimage\tests\test_measurements.py", line 701, in test_minimum_position07 assert_equal(output[0], (0, 1)) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 251, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 300, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: nan DESIRED: 0 Bottleneck-0.3.0.win32-py2.7 Bottleneck-0.3.0.win-amd64-py2.7 ====================================================================== ERROR: Test group_nanmean. ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 187, in runTest self.test(*self.arg) File "D:\Bottleneck-0.3.0\bottleneck\tests\group_test.py", line 63, in unit_maker a1, lab1 = func(arr, label, axis=axis) File "group_nanmean.pyx", line 74, in group.group_nanmean (bottleneck/src/group/group.c:2873) File "group_nanmean.pyx", line 179, in group.group_nanmean_selector (bottleneck/src/group/group.c:3464) TypeError: Unsupported ndim/dtype (1/int32). From cgohlke at uci.edu Fri Mar 4 15:31:40 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 04 Mar 2011 12:31:40 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D71435F.2000403@uci.edu> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: <4D714C2C.4040005@uci.edu> On 3/4/2011 11:54 AM, Christoph Gohlke wrote: > > > On 3/4/2011 1:00 AM, Christoph Gohlke wrote: >> >> >> On 3/3/2011 10:54 PM, Ralf Gommers wrote: >>> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >>> wrote: >>>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >>>> wrote: >>>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>>>>> Hi, >>>>>> >>>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >>>>>> Oliphant wrote: >>>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it >>>>>>> would >>>>>>> seem) have additional work and features that do not need to wait >>>>>>> for the >>>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>>>>> If someone is willing to manage the release of NumPy 1.6, then it >>>>>>> sounds >>>>>>> like a great idea to me. >>>>>> This thread ended without a conclusion a month ago. Now I think master >>>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>>>>> issues and segfault on OS X fixed, more testing of changes), and I >>>>>> have a better idea of my free time for March/April. Basically, I have >>>>>> a good amount of time for the next couple of weeks, and not so much at >>>>>> the end of March / first half of April due to an inter-continental >>>>>> move. But I think we can get out a beta by mid-March, and I can manage >>>>>> the release. >>>>>> >>>>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >>>>>> #1748 (blocker: regression for astype('str')) >>>>>> #1619 (issue with dtypes, with patch) >>>>>> #1749 (distutils, py 3.2) >>>>>> #1601 (distutils, py 3.2) >>>>>> #1622 (Solaris segfault, with patch) >>>>>> #1713 (Solaris segfault) >>>>>> #1631 (Solaris segfault) >>> >>> The distutils tickets are resolved. >>> >>>>>> Proposed schedule: >>>>>> March 15: beta 1 >>>>>> March 28: rc 1 >>>>>> April 17: rc 2 (if needed) >>>>>> April 24: final release >>> >>> Any comments on the schedule or tickets? >>> >>> Before the first beta can be released I think #1748 should be fixed. >>> Before the first RC the Solaris segfaults should be investigated, and >>> documentation for the new iterator (Python docstrings and C API docs) >>> and datetime should be written. >>> >>> Also, some testing on 64-bit Windows would be great, that usually >>> turns up new issues so the sooner the better. >>> >>> Ralf >> >> Hi Ralf, >> >> the numpy master branch on github can not be compiled with Visual >> Studio. A patch is attached. I'll test the builds tomorrow. >> >> Christoph >> > > I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. > There are few test failures (listed below) that look familiar. > > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. > > Most packages don't have any apparent problems. > Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test > failure/error (also listed below). > > There is a problem with code generated by Cython 0.13: pytables-2.2.1 > and pandas-0.3.0, which were built with Cython 0.13, report several > failures and do crash during the tests. This can probably be fixed by > "recythonizing" with Cython 0.14.1. Unfortunately recythonizing with Cython 0.14.1 did not help. Both packages still crash during tests when run against numpy 1.6.dev. Christoph From gael.varoquaux at normalesup.org Fri Mar 4 15:49:11 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Mar 2011 21:49:11 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D71435F.2000403@uci.edu> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: <20110304204911.GA22453@phare.normalesup.org> On Fri, Mar 04, 2011 at 11:54:07AM -0800, Christoph Gohlke wrote: > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. Wow, this is community service. I am impressed. Thanks heaps. Gael From oliphant at enthought.com Fri Mar 4 18:40:14 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 4 Mar 2011 17:40:14 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: <9FD6C38D-979B-4B2C-B75C-A0E03FA7909E@enthought.com> Thanks (again) for taking this on Ralf. You are doing a superb job and deserve much thanks for keeping the NumPy project on track. I'm very encouraged by the NumPy 1.6 work. I was looking over the code some more on the plane yesterday that Mark Wiebe has checked in (I finally was able to create my own fork of NumPy and start hacking on it again). The new ufunc refactoring is very good and Mark should be congratulated. It puts NumPy in a good place for further work that can be done for NumPy 2.0. I do have one question, however. It looks like the code for ufunc casting behavior was changed almost completely. I have no intrinsic problem with that, but I am a bit concerned that we may not have tested all the corner cases very well. The code that was in NumPy was a straightforward alteration of the algorithm that has been in use since 1995 when Numeric was written. That is not to say that it is "correct," just that it had a certain behavior. Large scale code changes typically have alternative "accidental" behavior. I am just a little nervous that the code changes will mean that more code will break than we expect. To me, what this means is that we just emphasize the point in the release notes, and encourage people to test their code as soon as possible during the month-long beta-testing phase. All the best, -Travis On Mar 4, 2011, at 12:54 AM, Ralf Gommers wrote: > On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers > wrote: >> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey wrote: >>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>>> Hi, >>>> >>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant wrote: >>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would >>>>> seem) have additional work and features that do not need to wait for the >>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>>> If someone is willing to manage the release of NumPy 1.6, then it sounds >>>>> like a great idea to me. >>>> This thread ended without a conclusion a month ago. Now I think master >>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>>> issues and segfault on OS X fixed, more testing of changes), and I >>>> have a better idea of my free time for March/April. Basically, I have >>>> a good amount of time for the next couple of weeks, and not so much at >>>> the end of March / first half of April due to an inter-continental >>>> move. But I think we can get out a beta by mid-March, and I can manage >>>> the release. >>>> >>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >>>> #1748 (blocker: regression for astype('str')) >>>> #1619 (issue with dtypes, with patch) >>>> #1749 (distutils, py 3.2) >>>> #1601 (distutils, py 3.2) >>>> #1622 (Solaris segfault, with patch) >>>> #1713 (Solaris segfault) >>>> #1631 (Solaris segfault) > > The distutils tickets are resolved. > >>>> Proposed schedule: >>>> March 15: beta 1 >>>> March 28: rc 1 >>>> April 17: rc 2 (if needed) >>>> April 24: final release > > Any comments on the schedule or tickets? > > Before the first beta can be released I think #1748 should be fixed. > Before the first RC the Solaris segfaults should be investigated, and > documentation for the new iterator (Python docstrings and C API docs) > and datetime should be written. > > Also, some testing on 64-bit Windows would be great, that usually > turns up new issues so the sooner the better. > > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From rowen at uw.edu Fri Mar 4 19:09:02 2011 From: rowen at uw.edu (Russell E. Owen) Date: Fri, 04 Mar 2011 16:09:02 -0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page Message-ID: The page gives a good introduction to structured arrays. However, it says nothing about how to set a particular element (all fields at once) from a collection of data. For instance: stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) The question is how to set stArr[0]? >From experimentation it appears that you can provide a tuple, but not a list. Hence the following works just fine (and that the tuple can contain a list): strArr[0,0] = ([1.0, 1.1], 2.0) but the following fails: strArr[0,0] = [[1.0, 1.1], 2.0] with an error: TypeError: expected a readable buffer object This is useful information if one is trying to initialize a structured array from a collection of data, such as that returned from a database query. So this is a request to add a bit to the documentation. -- Russell From charlesr.harris at gmail.com Fri Mar 4 20:41:12 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Mar 2011 18:41:12 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D71435F.2000403@uci.edu> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: > > > On 3/4/2011 1:00 AM, Christoph Gohlke wrote: > > > > > > On 3/3/2011 10:54 PM, Ralf Gommers wrote: > >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers > >> wrote: > >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey > >>> wrote: > >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: > >>>>> Hi, > >>>>> > >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis > >>>>> Oliphant wrote: > >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it > >>>>>> would > >>>>>> seem) have additional work and features that do not need to wait > >>>>>> for the > >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. > >>>>>> If someone is willing to manage the release of NumPy 1.6, then it > >>>>>> sounds > >>>>>> like a great idea to me. > >>>>> This thread ended without a conclusion a month ago. Now I think > master > >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x > >>>>> issues and segfault on OS X fixed, more testing of changes), and I > >>>>> have a better idea of my free time for March/April. Basically, I have > >>>>> a good amount of time for the next couple of weeks, and not so much > at > >>>>> the end of March / first half of April due to an inter-continental > >>>>> move. But I think we can get out a beta by mid-March, and I can > manage > >>>>> the release. > >>>>> > >>>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: > >>>>> #1748 (blocker: regression for astype('str')) > >>>>> #1619 (issue with dtypes, with patch) > >>>>> #1749 (distutils, py 3.2) > >>>>> #1601 (distutils, py 3.2) > >>>>> #1622 (Solaris segfault, with patch) > >>>>> #1713 (Solaris segfault) > >>>>> #1631 (Solaris segfault) > >> > >> The distutils tickets are resolved. > >> > >>>>> Proposed schedule: > >>>>> March 15: beta 1 > >>>>> March 28: rc 1 > >>>>> April 17: rc 2 (if needed) > >>>>> April 24: final release > >> > >> Any comments on the schedule or tickets? > >> > >> Before the first beta can be released I think #1748 should be fixed. > >> Before the first RC the Solaris segfaults should be investigated, and > >> documentation for the new iterator (Python docstrings and C API docs) > >> and datetime should be written. > >> > >> Also, some testing on 64-bit Windows would be great, that usually > >> turns up new issues so the sooner the better. > >> > >> Ralf > > > > Hi Ralf, > > > > the numpy master branch on github can not be compiled with Visual > > Studio. A patch is attached. I'll test the builds tomorrow. > > > > Christoph > > > > I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. > There are few test failures (listed below) that look familiar. > > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. > > Most packages don't have any apparent problems. > Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test > failure/error (also listed below). > > There is a problem with code generated by Cython 0.13: pytables-2.2.1 > and pandas-0.3.0, which were built with Cython 0.13, report several > failures and do crash during the tests. This can probably be fixed by > "recythonizing" with Cython 0.14.1. > > I see pytables crash on Fedora 14 64 bit also, even with freshly built pytables/numexpr. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 4 21:49:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Mar 2011 19:49:10 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 6:41 PM, Charles R Harris wrote: > > > On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: > >> >> >> On 3/4/2011 1:00 AM, Christoph Gohlke wrote: >> > >> > >> > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >> >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >> >> wrote: >> >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >> >>> wrote: >> >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >> >>>>> Hi, >> >>>>> >> >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >> >>>>> Oliphant wrote: >> >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it >> >>>>>> would >> >>>>>> seem) have additional work and features that do not need to wait >> >>>>>> for the >> >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >> >>>>>> If someone is willing to manage the release of NumPy 1.6, then it >> >>>>>> sounds >> >>>>>> like a great idea to me. >> >>>>> This thread ended without a conclusion a month ago. Now I think >> master >> >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >> >>>>> issues and segfault on OS X fixed, more testing of changes), and I >> >>>>> have a better idea of my free time for March/April. Basically, I >> have >> >>>>> a good amount of time for the next couple of weeks, and not so much >> at >> >>>>> the end of March / first half of April due to an inter-continental >> >>>>> move. But I think we can get out a beta by mid-March, and I can >> manage >> >>>>> the release. >> >>>>> >> >>>>> I've had a look at the bug tracker, here's a list of tickets for >> 1.6: >> >>>>> #1748 (blocker: regression for astype('str')) >> >>>>> #1619 (issue with dtypes, with patch) >> >>>>> #1749 (distutils, py 3.2) >> >>>>> #1601 (distutils, py 3.2) >> >>>>> #1622 (Solaris segfault, with patch) >> >>>>> #1713 (Solaris segfault) >> >>>>> #1631 (Solaris segfault) >> >> >> >> The distutils tickets are resolved. >> >> >> >>>>> Proposed schedule: >> >>>>> March 15: beta 1 >> >>>>> March 28: rc 1 >> >>>>> April 17: rc 2 (if needed) >> >>>>> April 24: final release >> >> >> >> Any comments on the schedule or tickets? >> >> >> >> Before the first beta can be released I think #1748 should be fixed. >> >> Before the first RC the Solaris segfaults should be investigated, and >> >> documentation for the new iterator (Python docstrings and C API docs) >> >> and datetime should be written. >> >> >> >> Also, some testing on 64-bit Windows would be great, that usually >> >> turns up new issues so the sooner the better. >> >> >> >> Ralf >> > >> > Hi Ralf, >> > >> > the numpy master branch on github can not be compiled with Visual >> > Studio. A patch is attached. I'll test the builds tomorrow. >> > >> > Christoph >> > >> >> I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. >> There are few test failures (listed below) that look familiar. >> >> I also ran tests and/or examples of a few 3rd party packages that were >> built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, >> basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. >> >> Most packages don't have any apparent problems. >> Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test >> failure/error (also listed below). >> >> There is a problem with code generated by Cython 0.13: pytables-2.2.1 >> and pandas-0.3.0, which were built with Cython 0.13, report several >> failures and do crash during the tests. This can probably be fixed by >> "recythonizing" with Cython 0.14.1. >> >> > I see pytables crash on Fedora 14 64 bit also, even with freshly built > pytables/numexpr. > Backtrace #0 0x0000003e9647a52c in __libc_free (mem=0x16cd1f0) at malloc.c:3724 #1 0x00007fffefed0988 in array_dealloc (self=0x29617b0) at numpy/core/src/multiarray/arrayobject.c:266 #2 0x0000003ea84e8452 in PyEval_EvalFrameEx (f=, throwflag=) at /usr/src/debug/Python-2.7/Python/ceval.c:2344 #3 0x0000003ea84ea71d in fast_function (f=, throwflag=) at /usr/src/debug/Python-2.7/Python/ceval.c:4157 #4 call_function (f=, throwflag=) at /usr/src/debug/Python-2.7/Python/ceval.c:4092 Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 4 22:11:12 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Mar 2011 20:11:12 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 7:49 PM, Charles R Harris wrote: > > > On Fri, Mar 4, 2011 at 6:41 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: >> >>> >>> >>> On 3/4/2011 1:00 AM, Christoph Gohlke wrote: >>> > >>> > >>> > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >>> >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >>> >> wrote: >>> >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >>> >>> wrote: >>> >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>> >>>>> Hi, >>> >>>>> >>> >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >>> >>>>> Oliphant wrote: >>> >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it >>> >>>>>> would >>> >>>>>> seem) have additional work and features that do not need to wait >>> >>>>>> for the >>> >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>> >>>>>> If someone is willing to manage the release of NumPy 1.6, then it >>> >>>>>> sounds >>> >>>>>> like a great idea to me. >>> >>>>> This thread ended without a conclusion a month ago. Now I think >>> master >>> >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >>> >>>>> issues and segfault on OS X fixed, more testing of changes), and I >>> >>>>> have a better idea of my free time for March/April. Basically, I >>> have >>> >>>>> a good amount of time for the next couple of weeks, and not so much >>> at >>> >>>>> the end of March / first half of April due to an inter-continental >>> >>>>> move. But I think we can get out a beta by mid-March, and I can >>> manage >>> >>>>> the release. >>> >>>>> >>> >>>>> I've had a look at the bug tracker, here's a list of tickets for >>> 1.6: >>> >>>>> #1748 (blocker: regression for astype('str')) >>> >>>>> #1619 (issue with dtypes, with patch) >>> >>>>> #1749 (distutils, py 3.2) >>> >>>>> #1601 (distutils, py 3.2) >>> >>>>> #1622 (Solaris segfault, with patch) >>> >>>>> #1713 (Solaris segfault) >>> >>>>> #1631 (Solaris segfault) >>> >> >>> >> The distutils tickets are resolved. >>> >> >>> >>>>> Proposed schedule: >>> >>>>> March 15: beta 1 >>> >>>>> March 28: rc 1 >>> >>>>> April 17: rc 2 (if needed) >>> >>>>> April 24: final release >>> >> >>> >> Any comments on the schedule or tickets? >>> >> >>> >> Before the first beta can be released I think #1748 should be fixed. >>> >> Before the first RC the Solaris segfaults should be investigated, and >>> >> documentation for the new iterator (Python docstrings and C API docs) >>> >> and datetime should be written. >>> >> >>> >> Also, some testing on 64-bit Windows would be great, that usually >>> >> turns up new issues so the sooner the better. >>> >> >>> >> Ralf >>> > >>> > Hi Ralf, >>> > >>> > the numpy master branch on github can not be compiled with Visual >>> > Studio. A patch is attached. I'll test the builds tomorrow. >>> > >>> > Christoph >>> > >>> >>> I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. >>> There are few test failures (listed below) that look familiar. >>> >>> I also ran tests and/or examples of a few 3rd party packages that were >>> built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, >>> basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. >>> >>> Most packages don't have any apparent problems. >>> Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test >>> failure/error (also listed below). >>> >>> There is a problem with code generated by Cython 0.13: pytables-2.2.1 >>> and pandas-0.3.0, which were built with Cython 0.13, report several >>> failures and do crash during the tests. This can probably be fixed by >>> "recythonizing" with Cython 0.14.1. >>> >>> >> I see pytables crash on Fedora 14 64 bit also, even with freshly built >> pytables/numexpr. >> > > Backtrace > > #0 0x0000003e9647a52c in __libc_free (mem=0x16cd1f0) at malloc.c:3724 > #1 0x00007fffefed0988 in array_dealloc (self=0x29617b0) > at numpy/core/src/multiarray/arrayobject.c:266 > #2 0x0000003ea84e8452 in PyEval_EvalFrameEx (f=, > throwflag=) > at /usr/src/debug/Python-2.7/Python/ceval.c:2344 > #3 0x0000003ea84ea71d in fast_function (f=, > throwflag=) > at /usr/src/debug/Python-2.7/Python/ceval.c:4157 > #4 call_function (f=, throwflag= out>) > at /usr/src/debug/Python-2.7/Python/ceval.c:4092 > > > And a test failure before the crash: Running MDAtomReopen.test01a_assign ----------------------------------- Error in test:: ValueError: value parameter '[[[1 3] [4 5]]]' cannot be converted into an array object compliant with CArray: '/test (CArray(1,)) '' atom := Int32Atom(shape=(2, 2), dflt=array([[0, 0], [0, 0]], dtype=int32)) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := (4096,)' The error was: Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Mar 4 23:37:38 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 5 Mar 2011 12:37:38 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D70AA19.8000509@uci.edu> References: <4D70AA19.8000509@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 5:00 PM, Christoph Gohlke wrote: > > > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >> Before the first beta can be released I think #1748 should be fixed. >> Before the first RC the Solaris segfaults should be investigated, and >> documentation for the new iterator (Python docstrings and C API docs) >> and datetime should be written. >> >> Also, some testing on 64-bit Windows would be great, that usually >> turns up new issues so the sooner the better. >> >> Ralf > > Hi Ralf, > > the numpy master branch on github can not be compiled with Visual Studio. A > patch is attached. I'll test the builds tomorrow. That looks fine to me, but I'll leave it to someone more familiar with that code to commit it. I'm also wondering whether the printf debug code should not be removed completely. Either way, using variadic macros is not a good idea I think, since that's a C99 feature which seems to not be supported well by non-gcc compilers. Cheers, Ralf From ralf.gommers at googlemail.com Fri Mar 4 23:51:54 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 5 Mar 2011 12:51:54 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D71435F.2000403@uci.edu> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 3:54 AM, Christoph Gohlke wrote: > > > On 3/4/2011 1:00 AM, Christoph Gohlke wrote: > > I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. > There are few test failures (listed below) that look familiar. > > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. Thanks a lot for all the testing Christoph, that's very helpful! > Most packages don't have any apparent problems. > Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test > failure/error (also listed below). > > There is a problem with code generated by Cython 0.13: pytables-2.2.1 > and pandas-0.3.0, which were built with Cython 0.13, report several > failures and do crash during the tests. This can probably be fixed by > "recythonizing" with Cython 0.14.1. > > Christoph > > > > > numpy-1.6.1.dev-win-amd64-py2.7 > > ====================================================================== > ERROR: Ticket #99 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", > line 215, in knownfailer > ? ? return f(*args, **kwargs) > ? File > "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_regression.py", > line 147, in test_intp > ? ? np.intp('0x' + 'f'*i_width,16) > TypeError: function takes at most 1 argument (2 given) This was discussed before, http://thread.gmane.org/gmane.comp.python.numeric.general/40121 It can be marked as knownfail in the 1.6.x branch if no one gets around to looking at it. > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", > line 146, in skipper_func > ? ? return f(*args, **kwargs) > ? File > "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_umath_complex.py", > line 248, in test_special_values > ? ? assert_almost_equal(np.log(x), y) > ? File "X:\Python27-x64\lib\site-packages\numpy\testing\utils.py", line > 443, in assert_almost_equal > ? ? raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 7 decimals > ?ACTUAL: array([ nan+2.35619449j]) > ?DESIRED: (inf+2.356194490192345j) > > > > > numpy-1.6.1.dev-win-amd64-py3.2 > > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 588, in assert_array_compare > ? ? assert_array_equal(xnanid, ynanid) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 687, in assert_array_equal > ? ? verbose=verbose, header='Arrays are not equal') > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 618, in assert_array_compare > ? ? raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 100.0%) > ?x: array([ True], dtype=bool) > ?y: array(False, dtype=bool) > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 440, in assert_almost_equal > ? ? assert_almost_equal(actualr, desiredr, decimal=decimal) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 447, in assert_almost_equal > ? ? return assert_array_almost_equal(actual, desired, decimal, err_msg) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 775, in assert_array_almost_equal > ? ? header=('Arrays are not almost equal to %d decimals' % decimal)) > AssertionError: > Arrays are not almost equal to 7 decimals > > (x and y nan location mismatch [ True], False mismatch) > ?x: array([ nan]) > ?y: array(inf) > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ? File "X:\Python32\lib\site-packages\numpy\testing\decorators.py", > line 147, in skipper_func > ? ? return f(*args, **kwargs) > ? File > "X:\Python32\lib\site-packages\numpy\core\tests\test_umath_complex.py", > line 248, in test_special_values > ? ? assert_almost_equal(np.log(x), y) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 443, in assert_almost_equal > ? ? raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 7 decimals > ?ACTUAL: array([ nan+2.35619449j]) > ?DESIRED: (inf+2.356194490192345j) Those TestClog are marked as conditional knownfail on Windows depending on a check for a few corner cases, I'm thinking it should just be unconditional knownfail on Windows. > numpy-1.6.1.dev-win32-py2.7 > numpy-1.6.1.dev-win32-py3.2 > > ====================================================================== > FAIL: Test basic arithmetic function errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File > "X:\Python27\lib\site-packages\numpy\core\tests\test_numeric.py", line > 320, in test_floating_exceptions > ? ? np.power, ftype(2), ftype(2**fi.nexp)) > ? File > "X:\Python27\lib\site-packages\numpy\core\tests\test_numeric.py", line > 261, in assert_raises_fpe > ? ? "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) > ? File "X:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, > in assert_ > ? ? raise AssertionError(msg) > AssertionError: Type did not raise fpe error > 'overflow'. I had already opened #1755 for that. Would be good to have it fixed, but not very urgent I'd say. > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python27\lib\site-packages\numpy\testing\decorators.py", > line 146, in skipper_func > ? ? return f(*args, **kwargs) > ? File > "X:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", > line 168, in test_special_values > ? ? self.assertRaises(FloatingPointError, np.log, x) > AssertionError: FloatingPointError not raised > > > > > > scipy-0.9.0-win-amd64-py3.2 > > ====================================================================== > FAIL: minimum position 7 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python32\lib\site-packages\nose\case.py", line 188, in runTest > ? ? self.test(*self.arg) > ? File > "X:\Python32\lib\site-packages\scipy\ndimage\tests\test_measurements.py", > ?line 701, in test_minimum_position07 > ? ? assert_equal(output[0], (0, 1)) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 251, in assert_equal > ? ? assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), > verbose) > ? File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 300, in assert_equal > ? ? raise AssertionError(msg) > AssertionError: > Items are not equal: > item=0 > > ?ACTUAL: nan > ?DESIRED: 0 That one is new, but since we didn't test scipy 0.9.0 extensively on py-3.2 it's not that surprising to get one failure. Ralf > Bottleneck-0.3.0.win32-py2.7 > Bottleneck-0.3.0.win-amd64-py2.7 > > ====================================================================== > ERROR: Test group_nanmean. > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "X:\Python27-x64\lib\site-packages\nose\case.py", line 187, in > runTest > ? ? self.test(*self.arg) > ? File "D:\Bottleneck-0.3.0\bottleneck\tests\group_test.py", line 63, > in unit_maker > ? ? a1, lab1 = func(arr, label, axis=axis) > ? File "group_nanmean.pyx", line 74, in group.group_nanmean > (bottleneck/src/group/group.c:2873) > ? File "group_nanmean.pyx", line 179, in group.group_nanmean_selector > (bottleneck/src/group/group.c:3464) > TypeError: Unsupported ndim/dtype (1/int32). From ben.root at ou.edu Fri Mar 4 23:58:14 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Mar 2011 22:58:14 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 9:11 PM, Charles R Harris wrote: > > > On Fri, Mar 4, 2011 at 7:49 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Fri, Mar 4, 2011 at 6:41 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: >>> >>>> >>>> >>>> On 3/4/2011 1:00 AM, Christoph Gohlke wrote: >>>> > >>>> > >>>> > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >>>> >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >>>> >> wrote: >>>> >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >>>> >>> wrote: >>>> >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >>>> >>>>> Hi, >>>> >>>>> >>>> >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >>>> >>>>> Oliphant wrote: >>>> >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others >>>> it >>>> >>>>>> would >>>> >>>>>> seem) have additional work and features that do not need to wait >>>> >>>>>> for the >>>> >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >>>> >>>>>> If someone is willing to manage the release of NumPy 1.6, then it >>>> >>>>>> sounds >>>> >>>>>> like a great idea to me. >>>> >>>>> This thread ended without a conclusion a month ago. Now I think >>>> master >>>> >>>>> is in a better state than a month ago for a release (py >>>> 2.4/2.5/3.x >>>> >>>>> issues and segfault on OS X fixed, more testing of changes), and I >>>> >>>>> have a better idea of my free time for March/April. Basically, I >>>> have >>>> >>>>> a good amount of time for the next couple of weeks, and not so >>>> much at >>>> >>>>> the end of March / first half of April due to an inter-continental >>>> >>>>> move. But I think we can get out a beta by mid-March, and I can >>>> manage >>>> >>>>> the release. >>>> >>>>> >>>> >>>>> I've had a look at the bug tracker, here's a list of tickets for >>>> 1.6: >>>> >>>>> #1748 (blocker: regression for astype('str')) >>>> >>>>> #1619 (issue with dtypes, with patch) >>>> >>>>> #1749 (distutils, py 3.2) >>>> >>>>> #1601 (distutils, py 3.2) >>>> >>>>> #1622 (Solaris segfault, with patch) >>>> >>>>> #1713 (Solaris segfault) >>>> >>>>> #1631 (Solaris segfault) >>>> >> >>>> >> The distutils tickets are resolved. >>>> >> >>>> >>>>> Proposed schedule: >>>> >>>>> March 15: beta 1 >>>> >>>>> March 28: rc 1 >>>> >>>>> April 17: rc 2 (if needed) >>>> >>>>> April 24: final release >>>> >> >>>> >> Any comments on the schedule or tickets? >>>> >> >>>> >> Before the first beta can be released I think #1748 should be fixed. >>>> >> Before the first RC the Solaris segfaults should be investigated, and >>>> >> documentation for the new iterator (Python docstrings and C API docs) >>>> >> and datetime should be written. >>>> >> >>>> >> Also, some testing on 64-bit Windows would be great, that usually >>>> >> turns up new issues so the sooner the better. >>>> >> >>>> >> Ralf >>>> > >>>> > Hi Ralf, >>>> > >>>> > the numpy master branch on github can not be compiled with Visual >>>> > Studio. A patch is attached. I'll test the builds tomorrow. >>>> > >>>> > Christoph >>>> > >>>> >>>> I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. >>>> There are few test failures (listed below) that look familiar. >>>> >>>> I also ran tests and/or examples of a few 3rd party packages that were >>>> built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, >>>> basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. >>>> >>>> Most packages don't have any apparent problems. >>>> Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test >>>> failure/error (also listed below). >>>> >>>> There is a problem with code generated by Cython 0.13: pytables-2.2.1 >>>> and pandas-0.3.0, which were built with Cython 0.13, report several >>>> failures and do crash during the tests. This can probably be fixed by >>>> "recythonizing" with Cython 0.14.1. >>>> >>>> >>> I see pytables crash on Fedora 14 64 bit also, even with freshly built >>> pytables/numexpr. >>> >> >> Backtrace >> >> #0 0x0000003e9647a52c in __libc_free (mem=0x16cd1f0) at malloc.c:3724 >> #1 0x00007fffefed0988 in array_dealloc (self=0x29617b0) >> at numpy/core/src/multiarray/arrayobject.c:266 >> #2 0x0000003ea84e8452 in PyEval_EvalFrameEx (f=, >> throwflag=) >> at /usr/src/debug/Python-2.7/Python/ceval.c:2344 >> #3 0x0000003ea84ea71d in fast_function (f=, >> throwflag=) >> at /usr/src/debug/Python-2.7/Python/ceval.c:4157 >> #4 call_function (f=, throwflag=> out>) >> at /usr/src/debug/Python-2.7/Python/ceval.c:4092 >> >> >> > > And a test failure before the crash: > > Running MDAtomReopen.test01a_assign > ----------------------------------- > > Error in test:: > > ValueError: value parameter '[[[1 3] > [4 5]]]' cannot be converted into an array object > compliant with CArray: '/test (CArray(1,)) '' > atom := Int32Atom(shape=(2, 2), dflt=array([[0, 0], > [0, 0]], dtype=int32)) > maindim := 0 > flavor := 'numpy' > byteorder := 'little' > chunkshape := (4096,)' The error was: with shape (2,2) doesn't match the broadcast shape (2,2,1)> > > Chuck > >> >> > I recently had to fix an example in matplotlib where there was a 1xN array being assigned to a 1-D slice of a numpy array. It used to work, but it now doesn't. I don't know if this was intended or not, though. FWIW, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Mar 4 23:59:21 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 5 Mar 2011 12:59:21 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 11:11 AM, Charles R Harris wrote: > On Fri, Mar 4, 2011 at 7:49 PM, Charles R Harris > wrote: >> On Fri, Mar 4, 2011 at 6:41 PM, Charles R Harris >> wrote: >>> On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke >>> wrote: >>>> Most packages don't have any apparent problems. >>>> Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test >>>> failure/error (also listed below). >>>> >>>> There is a problem with code generated by Cython 0.13: pytables-2.2.1 >>>> and pandas-0.3.0, which were built with Cython 0.13, report several >>>> failures and do crash during the tests. This can probably be fixed by >>>> "recythonizing" with Cython 0.14.1. >>>> >>> >>> I see pytables crash on Fedora 14 64 bit also, even with freshly built >>> pytables/numexpr. That's actually good news, right? Looks like it's not due to a backwards incompatibility. On a related note, can we do something about this: $ python setup.py build_ext -i Running from numpy source directory.non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 5, with checksum 941534d1afbc085b996141e8027d1d1e, but recorded checksum for C API version 5 in codegen_dir/cversions.txt is 77e2e846db87f25d7cf99f9d812076f0. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.py. MismatchCAPIWarning) Ralf >> Backtrace >> >> #0? 0x0000003e9647a52c in __libc_free (mem=0x16cd1f0) at malloc.c:3724 >> #1? 0x00007fffefed0988 in array_dealloc (self=0x29617b0) >> ??? at numpy/core/src/multiarray/arrayobject.c:266 >> #2? 0x0000003ea84e8452 in PyEval_EvalFrameEx (f=, >> ??? throwflag=) >> ??? at /usr/src/debug/Python-2.7/Python/ceval.c:2344 >> #3? 0x0000003ea84ea71d in fast_function (f=, >> ??? throwflag=) >> ??? at /usr/src/debug/Python-2.7/Python/ceval.c:4157 >> #4? call_function (f=, throwflag=> out>) >> ??? at /usr/src/debug/Python-2.7/Python/ceval.c:4092 >> >> > > And a test failure before the crash: > > Running MDAtomReopen.test01a_assign > ----------------------------------- > > Error in test:: > > ? ValueError: value parameter '[[[1 3] > ? [4 5]]]' cannot be converted into an array object > compliant with CArray: '/test (CArray(1,)) '' > ? atom := Int32Atom(shape=(2, 2), dflt=array([[0, 0], > ?????? [0, 0]], dtype=int32)) > ? maindim := 0 > ? flavor := 'numpy' > ? byteorder := 'little' > ? chunkshape := (4096,)' The error was: with shape (2,2) doesn't match the broadcast shape (2,2,1)> > > Chuck From faltet at pytables.org Sat Mar 5 06:15:51 2011 From: faltet at pytables.org (Francesc Alted) Date: Sat, 5 Mar 2011 12:15:51 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D71435F.2000403@uci.edu> Message-ID: <201103051215.51967.faltet@pytables.org> A Saturday 05 March 2011 02:41:12 Charles R Harris escrigu?: > > There is a problem with code generated by Cython 0.13: > > pytables-2.2.1 and pandas-0.3.0, which were built with Cython > > 0.13, report several failures and do crash during the tests. This > > can probably be fixed by "recythonizing" with Cython 0.14.1. > > I see pytables crash on Fedora 14 64 bit also, even with freshly > built pytables/numexpr. I'm in a hurry now, but I isolated one problem in the master branch of NumPy that is creating segfaults: http://projects.scipy.org/numpy/ticket/1756 Next week, I'll look into other issues in PyTables test suite w/ NumPy 1.6. -- Francesc Alted From pav at iki.fi Sat Mar 5 08:44:05 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 5 Mar 2011 13:44:05 +0000 (UTC) Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: > I recently had to fix an example in matplotlib where there was a 1xN > array being assigned to a 1-D slice of a numpy array. It used to work, > but it now doesn't. I don't know if this was intended or not, though. Probably not -- please file a bug report. If you can also point to a Numpy version in which it worked, that would also be nice. From ralf.gommers at googlemail.com Sat Mar 5 09:28:02 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 5 Mar 2011 22:28:02 +0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: > The page > > gives a good introduction to structured arrays. However, it says nothing > about how to set a particular element (all fields at once) from a > collection of data. > > For instance: > > stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) > > The question is how to set stArr[0]? > > >From experimentation it appears that you can provide a tuple, but not a > list. Hence the following works just fine (and that the tuple can > contain a list): > strArr[0,0] = ([1.0, 1.1], 2.0) > > but the following fails: > strArr[0,0] = [[1.0, 1.1], 2.0] > with an error: > TypeError: expected a readable buffer object > > This is useful information if one is trying to initialize a structured > array from a collection of data, such as that returned from a database > query. > I'm wondering if that's not a bug? If it's intentional then it is certainly counterintuitive. Ralf From dhyams at gmail.com Sat Mar 5 11:49:04 2011 From: dhyams at gmail.com (dhyams) Date: Sat, 5 Mar 2011 08:49:04 -0800 (PST) Subject: [Numpy-discussion] Taking a large number of dot products at once In-Reply-To: Message-ID: <31702663.20.1299343744526.JavaMail.geo-discussion-forums@yqhp25> Thank you eat; that's exactly what I needed....couldn't see the forest for the trees. I was determined that there was going to be a single function that would do it :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Sat Mar 5 12:10:21 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 5 Mar 2011 12:10:21 -0500 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers wrote: > On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >> The page >> >> gives a good introduction to structured arrays. However, it says nothing >> about how to set a particular element (all fields at once) from a >> collection of data. >> >> For instance: >> >> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >> >> The question is how to set stArr[0]? >> >> >From experimentation it appears that you can provide a tuple, but not a >> list. Hence the following works just fine (and that the tuple can >> contain a list): >> strArr[0,0] = ([1.0, 1.1], 2.0) >> >> but the following fails: >> strArr[0,0] = [[1.0, 1.1], 2.0] >> with an error: >> TypeError: expected a readable buffer object >> >> This is useful information if one is trying to initialize a structured >> array from a collection of data, such as that returned from a database >> query. >> I added a bit at the end here, though it is mentioned briefly above. Feel free to expand. It's a wiki. You just need edit rights. http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ > I'm wondering if that's not a bug? If it's intentional then it is > certainly counterintuitive. > This comes up from time to time. http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 Perhaps an enhancement ticket could be filed? It doesn't sound trivial to implement. Skipper From charlesr.harris at gmail.com Sat Mar 5 17:04:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 15:04:10 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <201103051215.51967.faltet@pytables.org> References: <4D71435F.2000403@uci.edu> <201103051215.51967.faltet@pytables.org> Message-ID: On Sat, Mar 5, 2011 at 4:15 AM, Francesc Alted wrote: > A Saturday 05 March 2011 02:41:12 Charles R Harris escrigu?: > > > There is a problem with code generated by Cython 0.13: > > > pytables-2.2.1 and pandas-0.3.0, which were built with Cython > > > 0.13, report several failures and do crash during the tests. This > > > can probably be fixed by "recythonizing" with Cython 0.14.1. > > > > I see pytables crash on Fedora 14 64 bit also, even with freshly > > built pytables/numexpr. > > I'm in a hurry now, but I isolated one problem in the master branch of > NumPy that is creating segfaults: > > http://projects.scipy.org/numpy/ticket/1756 > > Next week, I'll look into other issues in PyTables test suite w/ NumPy > 1.6. > > The segfault should be fixed in 9eee670, but there remain other errors and failures. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Sat Mar 5 17:28:10 2011 From: ben.root at ou.edu (Benjamin Root) Date: Sat, 5 Mar 2011 16:28:10 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 7:44 AM, Pauli Virtanen wrote: > On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: > > I recently had to fix an example in matplotlib where there was a 1xN > > array being assigned to a 1-D slice of a numpy array. It used to work, > > but it now doesn't. I don't know if this was intended or not, though. > > Probably not -- please file a bug report. If you can also point to a > Numpy version in which it worked, that would also be nice. > > I decided to give git bisect a try. In testing this, I tried two things: a = np.empty((20,)) a[:] = np.random.random((1, 20)) and a[:] = np.random.random((20, 1)) These both currently fail with the same exception message. If I check out and build v1.5.0, the former works, but the latter does not. Going back to v1.4.0, and the latter still doesn't work. Maybe this really shouldn't be considered a bug, and rather a more consistent behavior? By the way, git bisect says that the winner is: d90f19abf18d59be959e04d73b3dbd7ae04b1e89 is the first bad commit commit d90f19abf18d59be959e04d73b3dbd7ae04b1e89 Author: Mark Wiebe Date: Mon Jan 17 18:26:12 2011 -0800 ENH: core: Change PyArray_MoveInto to use the new iterator as well :040000 040000 a23fbcff385fca9704a5313e81217a6d80e3512c 09b684bd8893e44405534fedad165ce85e751019 M numpy If we agree that this is still a bug and not a feature, I will file a report. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 17:43:00 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 15:43:00 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 3:28 PM, Benjamin Root wrote: > > > On Sat, Mar 5, 2011 at 7:44 AM, Pauli Virtanen wrote: > >> On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: >> > I recently had to fix an example in matplotlib where there was a 1xN >> > array being assigned to a 1-D slice of a numpy array. It used to work, >> > but it now doesn't. I don't know if this was intended or not, though. >> >> Probably not -- please file a bug report. If you can also point to a >> Numpy version in which it worked, that would also be nice. >> >> > I decided to give git bisect a try. In testing this, I tried two things: > > a = np.empty((20,)) > a[:] = np.random.random((1, 20)) > > and > > a[:] = np.random.random((20, 1)) > > These both currently fail with the same exception message. If I check out > and build v1.5.0, the former works, but the latter does not. Going back to > v1.4.0, and the latter still doesn't work. Maybe this really shouldn't be > considered a bug, and rather a more consistent behavior? > > By the way, git bisect says that the winner is: > > d90f19abf18d59be959e04d73b3dbd7ae04b1e89 is the first bad commit > commit d90f19abf18d59be959e04d73b3dbd7ae04b1e89 > Author: Mark Wiebe > Date: Mon Jan 17 18:26:12 2011 -0800 > > ENH: core: Change PyArray_MoveInto to use the new iterator as well > > :040000 040000 a23fbcff385fca9704a5313e81217a6d80e3512c > 09b684bd8893e44405534fedad165ce85e751019 M numpy > > If we agree that this is still a bug and not a feature, I will file a > report. > > I think it is more of a feature. The assignment should probably only work if the rhs can be broadcast to the lhs. Whatever is decided, we need to make a test to enforce it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Sat Mar 5 18:10:08 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 5 Mar 2011 15:10:08 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Thu, Mar 3, 2011 at 10:54 PM, Ralf Gommers wrote: > > > >>> I've had a look at the bug tracker, here's a list of tickets for 1.6: > >>> #1748 (blocker: regression for astype('str')) > >>> #1619 (issue with dtypes, with patch) > >>> #1749 (distutils, py 3.2) > >>> #1601 (distutils, py 3.2) > >>> #1622 (Solaris segfault, with patch) > >>> #1713 (Solaris segfault) > >>> #1631 (Solaris segfault) > > The distutils tickets are resolved. > > >>> Proposed schedule: > >>> March 15: beta 1 > >>> March 28: rc 1 > >>> April 17: rc 2 (if needed) > >>> April 24: final release > > Any comments on the schedule or tickets? > That all looks fine to me. There are a few things that I've changed in the core that could stand some discussion before being finalized in 1.6, mostly due to what was required to make things work without depending on the data type enumeration order. The combination of the numpy and scipy tests were pretty effective, but as Travis mentioned my changes are fairly invasive. * When copying array to array, structured types now copy based on field names instead of positions, effectively behaving like a 'dict' instead of a 'labeled tuple'. This behaviour is more intuitive to me, and several fixed bugs such as dtype comparison completely ignoring the structured type data suggest that this changes an area of numpy that has been used in a more limited fashion. It might be worthwhile to introduce a tuple-style flag in a future version which causes data to be copied by position instead of by name, as it is likely useful in some contexts. * Array memory layouts are preserved in many cases. This means that if a, b are Fortran ordered, a+b will be as well. It could be made more pervasive, for example ndarray.copy defaults to C-order, and that could be changed to 'K' to preserve the memory layout by default. Any comments about that? * The ufunc uses a more consistent algorithm for loop selection. The previous algorithm was ad hoc and lacked symmetry, while the new algorithm is based on a simple minimization definition. This change exposed a bug in scipy's ndimage, which did not handle all of the numpy data type enums properly, so its possible there is more code out there which will be affected similarly. In general, I've used the implementation strategy of substituting my code into the core critical paths of numpy to maximize the amount of exercise it gets. While this creates more short-term hiccups as we are seeing now, it also means the new functionality conforms to the current system better and is much more stable since it is getting well tested. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Sat Mar 5 18:15:27 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 5 Mar 2011 15:15:27 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 8:37 PM, Ralf Gommers wrote: > On Fri, Mar 4, 2011 at 5:00 PM, Christoph Gohlke wrote: > > > > > > On 3/3/2011 10:54 PM, Ralf Gommers wrote: > >> Before the first beta can be released I think #1748 should be fixed. > >> Before the first RC the Solaris segfaults should be investigated, and > >> documentation for the new iterator (Python docstrings and C API docs) > >> and datetime should be written. > >> > >> Also, some testing on 64-bit Windows would be great, that usually > >> turns up new issues so the sooner the better. > >> > >> Ralf > > > > Hi Ralf, > > > > the numpy master branch on github can not be compiled with Visual Studio. > A > > patch is attached. I'll test the builds tomorrow. > > That looks fine to me, but I'll leave it to someone more familiar with > that code to commit it. > > I'm also wondering whether the printf debug code should not be removed > completely. Either way, using variadic macros is not a good idea I > think, since that's a C99 feature which seems to not be supported well > by non-gcc compilers. > If the approach to fixing it in the patch works, I'd like to keep the debug prints in. They don't affect run time, and can be useful to figure out where something is first deviating from the expected behavior when dealing with subtle bugs. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Sat Mar 5 18:20:59 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 5 Mar 2011 15:20:59 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 2:43 PM, Charles R Harris wrote: > On Sat, Mar 5, 2011 at 3:28 PM, Benjamin Root wrote: > >> >> >> On Sat, Mar 5, 2011 at 7:44 AM, Pauli Virtanen wrote: >> >>> On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: >>> > I recently had to fix an example in matplotlib where there was a 1xN >>> > array being assigned to a 1-D slice of a numpy array. It used to work, >>> > but it now doesn't. I don't know if this was intended or not, though. >>> >>> Probably not -- please file a bug report. If you can also point to a >>> Numpy version in which it worked, that would also be nice. >>> >>> >> I decided to give git bisect a try. In testing this, I tried two things: >> >> a = np.empty((20,)) >> a[:] = np.random.random((1, 20)) >> >> and >> >> a[:] = np.random.random((20, 1)) >> >> These both currently fail with the same exception message. If I check out >> and build v1.5.0, the former works, but the latter does not. Going back to >> v1.4.0, and the latter still doesn't work. Maybe this really shouldn't be >> considered a bug, and rather a more consistent behavior? >> >> By the way, git bisect says that the winner is: >> >> d90f19abf18d59be959e04d73b3dbd7ae04b1e89 is the first bad commit >> commit d90f19abf18d59be959e04d73b3dbd7ae04b1e89 >> Author: Mark Wiebe >> Date: Mon Jan 17 18:26:12 2011 -0800 >> >> ENH: core: Change PyArray_MoveInto to use the new iterator as well >> >> :040000 040000 a23fbcff385fca9704a5313e81217a6d80e3512c >> 09b684bd8893e44405534fedad165ce85e751019 M numpy >> >> If we agree that this is still a bug and not a feature, I will file a >> report. >> >> > I think it is more of a feature. The assignment should probably only work > if the rhs can be broadcast to the lhs. Whatever is decided, we need to make > a test to enforce it. > +1 for feature. I like stricter checking in most cases. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.gnata at gmail.com Sat Mar 5 18:27:31 2011 From: xavier.gnata at gmail.com (Xavier Gnata) Date: Sun, 06 Mar 2011 00:27:31 +0100 Subject: [Numpy-discussion] np.linalg.norm terrible behavior Message-ID: <4D72C6E3.6050002@gmail.com> Hi, I got this problem in a real life code and it took me some time to figure out that np.linalg.norm has a terrible numerical behavior. The problem is nicely described here http://fseoane.net/blog/2011/computing-the-vector-norm/ numpy/linalg/linalg.py claims to be a "high-level Python interface to the LAPACK library". That's great but the low level norm function is coded in pure python : x = asarray(x) if ord is None: # check the default case first and handle it immediately return sqrt(add.reduce((x.conj() * x).ravel().real)) Is there a way to use *by default* the blas function to compute the norm? ok....looks like sum as the same terrible numerical behavior x=(np.array([1e4] + [1]*10000, dtype=np.float32)) np.sum(x*x) returns 1e+08 instead of 1.0001e+08 Moreover, np.linalg.norm is slow compare to blas. Is there a way/plan to fix it? Xavier From kk1674 at nyu.edu Sat Mar 5 18:43:05 2011 From: kk1674 at nyu.edu (Kibeom Kim) Date: Sat, 5 Mar 2011 18:43:05 -0500 Subject: [Numpy-discussion] np.linalg.norm terrible behavior In-Reply-To: <4D72C6E3.6050002@gmail.com> References: <4D72C6E3.6050002@gmail.com> Message-ID: for the sum function part, It's not a good way to fix but... if you want more accuracy x=(np.array([1]*10000 + [1e4], dtype=np.float32)) np.sum(x*x) 1.0001e+08 You can sort x from small numbers to bigger numbers before you call sum. -Kibeom Kim On Sat, Mar 5, 2011 at 6:27 PM, Xavier Gnata wrote: > Hi, > > I got this problem in a real life code and it took me some time to > figure out that np.linalg.norm has a terrible numerical behavior. > The problem is nicely described here > http://fseoane.net/blog/2011/computing-the-vector-norm/ > > numpy/linalg/linalg.py claims to be a "high-level Python interface to > the LAPACK library". > That's great but the low level norm function is coded in pure python : > x = asarray(x) > if ord is None: # check the default case first and handle it > immediately > return sqrt(add.reduce((x.conj() * x).ravel().real)) > > Is there a way to use *by default* the blas function to compute the norm? > > ok....looks like sum as the same terrible numerical behavior > x=(np.array([1e4] + [1]*10000, dtype=np.float32)) > np.sum(x*x) > returns 1e+08 instead of 1.0001e+08 > > Moreover, np.linalg.norm is slow compare to blas. > > Is there a way/plan to fix it? > > Xavier > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From irving at naml.us Sat Mar 5 19:02:35 2011 From: irving at naml.us (Geoffrey Irving) Date: Sat, 5 Mar 2011 16:02:35 -0800 Subject: [Numpy-discussion] crash in C++ extension module on process shutdown Message-ID: Hello, I have an extension module which holds on to a reference to a numpy array in a static variable. When the process shuts down, the C++ destructor triggers array_dealloc via Py_DECREF, and I get Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000048 0x00000001000c7c0e in PyErr_Occurred () This is after Py_Finalize, so presumably what I'm doing is technically wrong, but I'm wondering if there's any easy way to make it not die. I'm using Python 2.6 with Numpy 1.5.1; the code works fine with Python 2.5. The stack trace is #0 0x00000001000c7c0e in PyErr_Occurred () #1 0x000000010252cdd9 in array_dealloc () #2 0x000000010296f06d in Try::ARRAY::~ARRAY (this=0x102990250) at ARRAY.h:138 During symbol reading, inner block (0x102962c90-0x102962cd4 '_ZN3Try5ARRAYIKiLi1EEC1Ev') not inside outer block (0x102962d84-0x10297ef30). During symbol reading, inner block (0x102962c42-0x102962c4c '_ZN3Try10ARRAY_BASEIKiNS_5ARRAYIS1_Li1EEEEC2Ev') not inside outer block (0x102962c90-0x10297ef30). During symbol reading, inner block (0x102962c38-0x102962c42 '_ZN3Try10ARRAY_BASEIiNS_5ARRAYIiLi1EEEEC2Ev') not inside outer block (0x102962c42-0x10297ef30). During symbol reading, inner block (0x102962c38-0x10297ef30) not inside outer block (0x102962d84-0x10297ef30). #3 0x000000010297eefa in __tcf_1 () at Arrays/Module.cpp:9 #4 0x00007fff856678d4 in __cxa_finalize () #5 0x00007fff856677ec in exit () #6 0x00000001000ddcfd in handle_system_exit () #7 0x00000001000ddf45 in PyErr_PrintEx () #8 0x00000001000dec5a in PyRun_SimpleFileExFlags () #9 0x00000001000ee40c in Py_Main () #10 0x0000000100000f14 in ?? () Thanks in advance for any suggestions! Geoffrey From charlesr.harris at gmail.com Sat Mar 5 19:10:57 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 17:10:57 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <4D71435F.2000403@uci.edu> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: > > > On 3/4/2011 1:00 AM, Christoph Gohlke wrote: > > > > > > On 3/3/2011 10:54 PM, Ralf Gommers wrote: > >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers > >> wrote: > >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey > >>> wrote: > >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: > >>>>> Hi, > >>>>> > >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis > >>>>> Oliphant wrote: > >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it > >>>>>> would > >>>>>> seem) have additional work and features that do not need to wait > >>>>>> for the > >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. > >>>>>> If someone is willing to manage the release of NumPy 1.6, then it > >>>>>> sounds > >>>>>> like a great idea to me. > >>>>> This thread ended without a conclusion a month ago. Now I think > master > >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x > >>>>> issues and segfault on OS X fixed, more testing of changes), and I > >>>>> have a better idea of my free time for March/April. Basically, I have > >>>>> a good amount of time for the next couple of weeks, and not so much > at > >>>>> the end of March / first half of April due to an inter-continental > >>>>> move. But I think we can get out a beta by mid-March, and I can > manage > >>>>> the release. > >>>>> > >>>>> I've had a look at the bug tracker, here's a list of tickets for 1.6: > >>>>> #1748 (blocker: regression for astype('str')) > >>>>> #1619 (issue with dtypes, with patch) > >>>>> #1749 (distutils, py 3.2) > >>>>> #1601 (distutils, py 3.2) > >>>>> #1622 (Solaris segfault, with patch) > >>>>> #1713 (Solaris segfault) > >>>>> #1631 (Solaris segfault) > >> > >> The distutils tickets are resolved. > >> > >>>>> Proposed schedule: > >>>>> March 15: beta 1 > >>>>> March 28: rc 1 > >>>>> April 17: rc 2 (if needed) > >>>>> April 24: final release > >> > >> Any comments on the schedule or tickets? > >> > >> Before the first beta can be released I think #1748 should be fixed. > >> Before the first RC the Solaris segfaults should be investigated, and > >> documentation for the new iterator (Python docstrings and C API docs) > >> and datetime should be written. > >> > >> Also, some testing on 64-bit Windows would be great, that usually > >> turns up new issues so the sooner the better. > >> > >> Ralf > > > > Hi Ralf, > > > > the numpy master branch on github can not be compiled with Visual > > Studio. A patch is attached. I'll test the builds tomorrow. > > > > Christoph > > > > I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. > There are few test failures (listed below) that look familiar. > > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. > > Most packages don't have any apparent problems. > Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test > failure/error (also listed below). > > There is a problem with code generated by Cython 0.13: pytables-2.2.1 > and pandas-0.3.0, which were built with Cython 0.13, report several > failures and do crash during the tests. This can probably be fixed by > "recythonizing" with Cython 0.14.1. > > The tables segfault is fixed, although other errors remain. Pandas still segfaults and I'm guessing that the problem is somewhere in the creation of object arrays/subtypes. The short code to reproduce the problem is >>> import pandas >>> import pandas.util.testing as common >>> df = common.makeTimeDataFrame() >>> objs = [df, df] >>> s = Series(objs, index=[0, 1]) ISTR that the behaviour of numpy with respect to the creation of object arrays might have changed a bit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 19:22:39 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 17:22:39 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 5:10 PM, Charles R Harris wrote: > > > On Fri, Mar 4, 2011 at 12:54 PM, Christoph Gohlke wrote: > >> >> >> On 3/4/2011 1:00 AM, Christoph Gohlke wrote: >> > >> > >> > On 3/3/2011 10:54 PM, Ralf Gommers wrote: >> >> On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers >> >> wrote: >> >>> On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey >> >>> wrote: >> >>>> On 02/28/2011 02:00 AM, Ralf Gommers wrote: >> >>>>> Hi, >> >>>>> >> >>>>> On Fri, Jan 28, 2011 at 7:15 AM, Travis >> >>>>> Oliphant wrote: >> >>>>>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it >> >>>>>> would >> >>>>>> seem) have additional work and features that do not need to wait >> >>>>>> for the >> >>>>>> NumPy 2.0 ABI design to finalize in order to get out there. >> >>>>>> If someone is willing to manage the release of NumPy 1.6, then it >> >>>>>> sounds >> >>>>>> like a great idea to me. >> >>>>> This thread ended without a conclusion a month ago. Now I think >> master >> >>>>> is in a better state than a month ago for a release (py 2.4/2.5/3.x >> >>>>> issues and segfault on OS X fixed, more testing of changes), and I >> >>>>> have a better idea of my free time for March/April. Basically, I >> have >> >>>>> a good amount of time for the next couple of weeks, and not so much >> at >> >>>>> the end of March / first half of April due to an inter-continental >> >>>>> move. But I think we can get out a beta by mid-March, and I can >> manage >> >>>>> the release. >> >>>>> >> >>>>> I've had a look at the bug tracker, here's a list of tickets for >> 1.6: >> >>>>> #1748 (blocker: regression for astype('str')) >> >>>>> #1619 (issue with dtypes, with patch) >> >>>>> #1749 (distutils, py 3.2) >> >>>>> #1601 (distutils, py 3.2) >> >>>>> #1622 (Solaris segfault, with patch) >> >>>>> #1713 (Solaris segfault) >> >>>>> #1631 (Solaris segfault) >> >> >> >> The distutils tickets are resolved. >> >> >> >>>>> Proposed schedule: >> >>>>> March 15: beta 1 >> >>>>> March 28: rc 1 >> >>>>> April 17: rc 2 (if needed) >> >>>>> April 24: final release >> >> >> >> Any comments on the schedule or tickets? >> >> >> >> Before the first beta can be released I think #1748 should be fixed. >> >> Before the first RC the Solaris segfaults should be investigated, and >> >> documentation for the new iterator (Python docstrings and C API docs) >> >> and datetime should be written. >> >> >> >> Also, some testing on 64-bit Windows would be great, that usually >> >> turns up new issues so the sooner the better. >> >> >> >> Ralf >> > >> > Hi Ralf, >> > >> > the numpy master branch on github can not be compiled with Visual >> > Studio. A patch is attached. I'll test the builds tomorrow. >> > >> > Christoph >> > >> >> I tested the 32 and 64 bit msvc9/MKL builds for Python 2.7 and 3.2. >> There are few test failures (listed below) that look familiar. >> >> I also ran tests and/or examples of a few 3rd party packages that were >> built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, >> basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. >> >> Most packages don't have any apparent problems. >> Scipy-0.9.0-win-amd64-py3.2 and Bottleneck-0.3.0 each have one test >> failure/error (also listed below). >> >> There is a problem with code generated by Cython 0.13: pytables-2.2.1 >> and pandas-0.3.0, which were built with Cython 0.13, report several >> failures and do crash during the tests. This can probably be fixed by >> "recythonizing" with Cython 0.14.1. >> >> > The tables segfault is fixed, although other errors remain. Pandas still > segfaults and I'm guessing that the problem is somewhere in the creation of > object arrays/subtypes. The short code to reproduce the problem is > > >>> import pandas > >>> import pandas.util.testing as common > >>> df = common.makeTimeDataFrame() > >>> objs = [df, df] > >>> s = Series(objs, index=[0, 1]) > > > Make that >>> import pandas >>> import pandas.util.testing as common >>> df = common.makeTimeDataFrame() >>> objs = [df, df] >>> s = pandas.Series(objs, index=[0, 1]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 19:53:58 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 17:53:58 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: On Sat, Mar 5, 2011 at 4:20 PM, Mark Wiebe wrote: > On Sat, Mar 5, 2011 at 2:43 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Sat, Mar 5, 2011 at 3:28 PM, Benjamin Root wrote: >> >>> >>> >>> On Sat, Mar 5, 2011 at 7:44 AM, Pauli Virtanen wrote: >>> >>>> On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: >>>> > I recently had to fix an example in matplotlib where there was a 1xN >>>> > array being assigned to a 1-D slice of a numpy array. It used to >>>> work, >>>> > but it now doesn't. I don't know if this was intended or not, though. >>>> >>>> Probably not -- please file a bug report. If you can also point to a >>>> Numpy version in which it worked, that would also be nice. >>>> >>>> >>> I decided to give git bisect a try. In testing this, I tried two things: >>> >>> a = np.empty((20,)) >>> a[:] = np.random.random((1, 20)) >>> >>> and >>> >>> a[:] = np.random.random((20, 1)) >>> >>> These both currently fail with the same exception message. If I check >>> out and build v1.5.0, the former works, but the latter does not. Going back >>> to v1.4.0, and the latter still doesn't work. Maybe this really shouldn't >>> be considered a bug, and rather a more consistent behavior? >>> >>> By the way, git bisect says that the winner is: >>> >>> d90f19abf18d59be959e04d73b3dbd7ae04b1e89 is the first bad commit >>> commit d90f19abf18d59be959e04d73b3dbd7ae04b1e89 >>> Author: Mark Wiebe >>> Date: Mon Jan 17 18:26:12 2011 -0800 >>> >>> ENH: core: Change PyArray_MoveInto to use the new iterator as well >>> >>> :040000 040000 a23fbcff385fca9704a5313e81217a6d80e3512c >>> 09b684bd8893e44405534fedad165ce85e751019 M numpy >>> >>> If we agree that this is still a bug and not a feature, I will file a >>> report. >>> >>> >> I think it is more of a feature. The assignment should probably only work >> if the rhs can be broadcast to the lhs. Whatever is decided, we need to make >> a test to enforce it. >> > > +1 for feature. I like stricter checking in most cases. > > Although I think this accounts for some of the failures in tables. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Mar 5 21:23:16 2011 From: oliphant at enthought.com (Enthought) Date: Sat, 5 Mar 2011 20:23:16 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> Message-ID: <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> This should be changed back so that the former works and the later does not. It was intentional that the former worked --- it was consistent with broadcasting rules. A (1,20) array can be interpreted as a (20,) array. Travis (mobile phone of) Travis Oliphant Enthought, Inc. www.enthought.com On Mar 5, 2011, at 6:53 PM, Charles R Harris wrote: > > > On Sat, Mar 5, 2011 at 4:20 PM, Mark Wiebe wrote: > On Sat, Mar 5, 2011 at 2:43 PM, Charles R Harris wrote: > On Sat, Mar 5, 2011 at 3:28 PM, Benjamin Root wrote: > > > On Sat, Mar 5, 2011 at 7:44 AM, Pauli Virtanen wrote: > On Fri, 04 Mar 2011 22:58:14 -0600, Benjamin Root wrote: > > I recently had to fix an example in matplotlib where there was a 1xN > > array being assigned to a 1-D slice of a numpy array. It used to work, > > but it now doesn't. I don't know if this was intended or not, though. > > Probably not -- please file a bug report. If you can also point to a > Numpy version in which it worked, that would also be nice. > > > I decided to give git bisect a try. In testing this, I tried two things: > > a = np.empty((20,)) > a[:] = np.random.random((1, 20)) > > and > > a[:] = np.random.random((20, 1)) > > These both currently fail with the same exception message. If I check out and build v1.5.0, the former works, but the latter does not. Going back to v1.4.0, and the latter still doesn't work. Maybe this really shouldn't be considered a bug, and rather a more consistent behavior? > > By the way, git bisect says that the winner is: > > d90f19abf18d59be959e04d73b3dbd7ae04b1e89 is the first bad commit > commit d90f19abf18d59be959e04d73b3dbd7ae04b1e89 > Author: Mark Wiebe > Date: Mon Jan 17 18:26:12 2011 -0800 > > ENH: core: Change PyArray_MoveInto to use the new iterator as well > > :040000 040000 a23fbcff385fca9704a5313e81217a6d80e3512c 09b684bd8893e44405534fedad165ce85e751019 M numpy > > If we agree that this is still a bug and not a feature, I will file a report. > > > I think it is more of a feature. The assignment should probably only work if the rhs can be broadcast to the lhs. Whatever is decided, we need to make a test to enforce it. > > +1 for feature. I like stricter checking in most cases. > > > Although I think this accounts for some of the failures in tables. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 21:41:12 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 19:41:12 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> Message-ID: On Sat, Mar 5, 2011 at 7:23 PM, Enthought wrote: > This should be changed back so that the former works and the later does > not. It was intentional that the former worked --- it was consistent with > broadcasting rules. > > A (1,20) array can be interpreted as a (20,) array. > > So should a (1,1,1,1,20) array also be interpreted as a (20,) array? From what you say, an arbitrary number of 1's can not only be appended on the left, the usual broadcasting, but an arbitrary number of 1's can also be removed. Is that so? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 21:51:13 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 19:51:13 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> Message-ID: On Sat, Mar 5, 2011 at 7:41 PM, Charles R Harris wrote: > > > On Sat, Mar 5, 2011 at 7:23 PM, Enthought wrote: > >> This should be changed back so that the former works and the later does >> not. It was intentional that the former worked --- it was consistent with >> broadcasting rules. >> >> A (1,20) array can be interpreted as a (20,) array. >> >> > So should a (1,1,1,1,20) array also be interpreted as a (20,) array? From > what you say, an arbitrary number of 1's can not only be appended on the > left, the usual broadcasting, but an arbitrary number of 1's can also be > removed. Is that so? > > The point being, that assignment is now a special case that does not follow the same rules as, say, addition. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Mar 5 23:00:08 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Sat, 5 Mar 2011 22:00:08 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> Message-ID: My point is also that we need to make sure the broadcasting rules are consistent for both addition and array copy. Addition would not create an error if a (1,20) array was added to a (20,) array. Therefore, a (1,20) array can also copied into a (20,) array --- as can a (1,1,1,1,20) array. Conversely, a (20,1) or a (1,1,20,1,1) array can not be copied into a (20,) array. For the purposes of assignment, NumPy was designed so that if the rhs and lhs could be broadcast to the same shape, then the assignment could proceed: otherwise, it would fail. This behavior should not change in 1.6. -Travis On Mar 5, 2011, at 8:51 PM, Charles R Harris wrote: > > > On Sat, Mar 5, 2011 at 7:41 PM, Charles R Harris wrote: > > > On Sat, Mar 5, 2011 at 7:23 PM, Enthought wrote: > This should be changed back so that the former works and the later does not. It was intentional that the former worked --- it was consistent with broadcasting rules. > > A (1,20) array can be interpreted as a (20,) array. > > > So should a (1,1,1,1,20) array also be interpreted as a (20,) array? From what you say, an arbitrary number of 1's can not only be appended on the left, the usual broadcasting, but an arbitrary number of 1's can also be removed. Is that so? > > > The point being, that assignment is now a special case that does not follow the same rules as, say, addition. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Mar 5 23:13:12 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Sat, 5 Mar 2011 22:13:12 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Mar 5, 2011, at 5:10 PM, Mark Wiebe wrote: > On Thu, Mar 3, 2011 at 10:54 PM, Ralf Gommers wrote: > > > >>> I've had a look at the bug tracker, here's a list of tickets for 1.6: > >>> #1748 (blocker: regression for astype('str')) > >>> #1619 (issue with dtypes, with patch) > >>> #1749 (distutils, py 3.2) > >>> #1601 (distutils, py 3.2) > >>> #1622 (Solaris segfault, with patch) > >>> #1713 (Solaris segfault) > >>> #1631 (Solaris segfault) > > The distutils tickets are resolved. > > >>> Proposed schedule: > >>> March 15: beta 1 > >>> March 28: rc 1 > >>> April 17: rc 2 (if needed) > >>> April 24: final release > > Any comments on the schedule or tickets? > > That all looks fine to me. There are a few things that I've changed in the core that could stand some discussion before being finalized in 1.6, mostly due to what was required to make things work without depending on the data type enumeration order. The combination of the numpy and scipy tests were pretty effective, but as Travis mentioned my changes are fairly invasive. > > * When copying array to array, structured types now copy based on field names instead of positions, effectively behaving like a 'dict' instead of a 'labeled tuple'. This behaviour is more intuitive to me, and several fixed bugs such as dtype comparison completely ignoring the structured type data suggest that this changes an area of numpy that has been used in a more limited fashion. It might be worthwhile to introduce a tuple-style flag in a future version which causes data to be copied by position instead of by name, as it is likely useful in some contexts. This is a semantic change that does make me a tiny bit nervous. Structured arrays are actually used quite a bit in the wild, and so this could raise some errors. What I don't know is how often sub-parts of a structured arrays get copied into other structured arrays with a different order to the fields. From what I gather, Mark's changes would allow this case and do an arguably useful thing. Previously, a copy was only allowed if the structured array contained the same fields in the same order. It seems like this is a relaxation of a rule and should not raise any errors (unless extant code was relying on the previous errors for some reason). > > * Array memory layouts are preserved in many cases. This means that if a, b are Fortran ordered, a+b will be as well. It could be made more pervasive, for example ndarray.copy defaults to C-order, and that could be changed to 'K' to preserve the memory layout by default. Any comments about that? I like this change quite a bit, but it has similar potential "expectation" issues. I think the default should be changed to 'K' in NumPy 2.0, but perhaps we should preserve C-order for now to avoid the subtle breakages that might occur based on changed expectations. What are others thoughts? > > * The ufunc uses a more consistent algorithm for loop selection. The previous algorithm was ad hoc and lacked symmetry, while the new algorithm is based on a simple minimization definition. This change exposed a bug in scipy's ndimage, which did not handle all of the numpy data type enums properly, so its possible there is more code out there which will be affected similarly. This change has me the most nervous. I'm looking forward to the more consistent algorithm. As I said, the algorithm presently used as been there since Numeric in 1995 (I modified it only a little bit to handle scalar-array casting rules a bit differently). This kind of change will have different corner cases and this should be understood before a release. I'm also wondering what happened to the optional arguments to ufuncs (are they still there)? One of these allowed you to choose the loop yourself and bypass the selection algorithm. > > In general, I've used the implementation strategy of substituting my code into the core critical paths of numpy to maximize the amount of exercise it gets. While this creates more short-term hiccups as we are seeing now, it also means the new functionality conforms to the current system better and is much more stable since it is getting well tested. Thanks again for all the good core-algorithm work, Mark. You have being doing a great job. -Travis > > -Mark > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 5 23:13:44 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Mar 2011 21:13:44 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> Message-ID: On Sat, Mar 5, 2011 at 9:00 PM, Travis Oliphant wrote: > My point is also that we need to make sure the broadcasting rules are > consistent for both addition and array copy. > > Addition would not create an error if a (1,20) array was added to a (20,) > array. Therefore, a (1,20) array can also copied into a (20,) array --- as > can a (1,1,1,1,20) array. Conversely, a (20,1) or a (1,1,20,1,1) array can > not be copied into a (20,) array. > Well, adding a (1,20) array to a (20,) array gives a (1,20) array. Changing this for assignment is like english spelling ( 'I' before 'e' except after 'c', or sounded 'a' as in neighbor and weigh -- with various exceptions) or having to learn to conjugate irregular verbs. Most illogical. That said, I suppose we have to support it for backward compatibility. It doesn't seem to be used that much, however, probably because it isn't obvious. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Sun Mar 6 00:47:34 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 5 Mar 2011 21:47:34 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <4D70AA19.8000509@uci.edu> <4D71435F.2000403@uci.edu> <1E728FE9-4404-4B32-906E-60C8AF954776@enthought.com> Message-ID: On Sat, Mar 5, 2011 at 8:13 PM, Charles R Harris wrote: > > > On Sat, Mar 5, 2011 at 9:00 PM, Travis Oliphant wrote: > >> My point is also that we need to make sure the broadcasting rules are >> consistent for both addition and array copy. >> >> Addition would not create an error if a (1,20) array was added to a (20,) >> array. Therefore, a (1,20) array can also copied into a (20,) array --- as >> can a (1,1,1,1,20) array. Conversely, a (20,1) or a (1,1,20,1,1) array can >> not be copied into a (20,) array. >> > > Well, adding a (1,20) array to a (20,) array gives a (1,20) array. Changing > this for assignment is like english spelling ( 'I' before 'e' except after > 'c', or sounded 'a' as in neighbor and weigh -- with various exceptions) or > having to learn to conjugate irregular verbs. Most illogical. That said, I > suppose we have to support it for backward compatibility. It doesn't seem to > be used that much, however, probably because it isn't obvious. > I think it's ok to revert this behavior for backwards compatibility, but believe it's an inconsistent and unintuitive choice. In broadcasting, there are two operations, growing a dimension 1 -> n, and appending a new 1 dimension to the left. The behaviour under discussion in assignment is different from normal broadcasting in that only the second one is permitted. It is broadcasting the output to the input, rather than broadcasting the input to the output. Suppose a has shape (20,), b has shape (1,20), and c has shape (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c has shape (20,20). If we do "b[...] = a", a will be broadcast to match b by adding a 1 dimension to the left. This is reasonable and consistent with addition. If we do "a[...]=b", under 1.5 rules, a will once again be broadcast to match b by adding a 1 dimension to the left. If we do "a[...]=c", we could broadcast both a and c together to the shape (20,20). This results in multiple assignments to each element of a, which is inconsistent. This is not analogous to a+c, but rather to np.add(c, c, out=a). The distinction is subtle, but the inconsistent behavior is harmless enough for assignment that keeping backwards compatibility seems reasonable. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Sun Mar 6 01:11:25 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 5 Mar 2011 22:11:25 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Sat, Mar 5, 2011 at 8:13 PM, Travis Oliphant wrote: > > On Mar 5, 2011, at 5:10 PM, Mark Wiebe wrote: > > On Thu, Mar 3, 2011 at 10:54 PM, Ralf Gommers > wrote: > >> >> > > >> >>> I've had a look at the bug tracker, here's a list of tickets for 1.6: >> >>> #1748 (blocker: regression for astype('str')) >> >>> #1619 (issue with dtypes, with patch) >> >>> #1749 (distutils, py 3.2) >> >>> #1601 (distutils, py 3.2) >> >>> #1622 (Solaris segfault, with patch) >> >>> #1713 (Solaris segfault) >> >>> #1631 (Solaris segfault) >> >> The distutils tickets are resolved. >> >> >>> Proposed schedule: >> >>> March 15: beta 1 >> >>> March 28: rc 1 >> >>> April 17: rc 2 (if needed) >> >>> April 24: final release >> >> Any comments on the schedule or tickets? >> > > That all looks fine to me. There are a few things that I've changed in the > core that could stand some discussion before being finalized in 1.6, mostly > due to what was required to make things work without depending on the data > type enumeration order. The combination of the numpy and scipy tests were > pretty effective, but as Travis mentioned my changes are fairly invasive. > > * When copying array to array, structured types now copy based on field > names instead of positions, effectively behaving like a 'dict' instead of a > 'labeled tuple'. This behaviour is more intuitive to me, and several fixed > bugs such as dtype comparison completely ignoring the structured type data > suggest that this changes an area of numpy that has been used in a more > limited fashion. It might be worthwhile to introduce a tuple-style flag in a > future version which causes data to be copied by position instead of by > name, as it is likely useful in some contexts. > > > This is a semantic change that does make me a tiny bit nervous. > Structured arrays are actually used quite a bit in the wild, and so this > could raise some errors. What I don't know is how often sub-parts of a > structured arrays get copied into other structured arrays with a different > order to the fields. From what I gather, Mark's changes would allow this > case and do an arguably useful thing. Previously, a copy was only allowed > if the structured array contained the same fields in the same order. It > seems like this is a relaxation of a rule and should not raise any errors > (unless extant code was relying on the previous errors for some reason). > Another important factor is that previously the performance was poor, because each copy involved converting the array element to a Python tuple, then copying the tuple into the destination array. The new code directly copies the elements with no Python overhead. I haven't directly benchmarked this, but if someone wants to confirm this with some numbers that would be great. > * Array memory layouts are preserved in many cases. This means that if a, b > are Fortran ordered, a+b will be as well. It could be made more pervasive, > for example ndarray.copy defaults to C-order, and that could be changed to > 'K' to preserve the memory layout by default. Any comments about that? > > > I like this change quite a bit, but it has similar potential "expectation" > issues. I think the default should be changed to 'K' in NumPy 2.0, but > perhaps we should preserve C-order for now to avoid the subtle breakages > that might occur based on changed expectations. What are others thoughts? > I suspect defaulting to 'C' might be desirable, but I initially set it to 'K' to see how it would work out. Defaulting it to 'C' unfortunately kills most of the performance benefits of the new code, so it might be nice to leave it as 'K' if no issues arise that are traced back to here. * The ufunc uses a more consistent algorithm for loop selection. The > previous algorithm was ad hoc and lacked symmetry, while the new algorithm > is based on a simple minimization definition. This change exposed a bug in > scipy's ndimage, which did not handle all of the numpy data type enums > properly, so its possible there is more code out there which will be > affected similarly. > > > This change has me the most nervous. I'm looking forward to the more > consistent algorithm. As I said, the algorithm presently used as been there > since Numeric in 1995 (I modified it only a little bit to handle > scalar-array casting rules a bit differently). This kind of change will > have different corner cases and this should be understood before a release. > I don't think there is much reason to worry here. The only substantive difference is that the new algorithm doesn't skip ahead based on the type of the first operand. This code is also extremely thoroughly exercised, messing with the ufuncs can even cause numpy startup to fail. The trickiest ad hoc part of loop selection was how accumulation of logical operations settles on the boolean loop when the inputs are not boolean, and this still works the same, but now is coded explicitly instead of appearing to be an accident. Of course, testing this thoroughly is necessary, and the work being done by Ralf, Christoph, and others is fantastic. I'm also wondering what happened to the optional arguments to ufuncs (are > they still there)? One of these allowed you to choose the loop > yourself and bypass the selection algorithm. > All the existing parameters are still there, and I added tests for the 'sig' parameter. It might be worthwhile for someone who has used these features to write examples for the documentation, as they're a bit tricky to understand from the descriptions. I also added an 'out' keyword parameter to make ufuncs more consistent with other functions' output parameters. > In general, I've used the implementation strategy of substituting my code > into the core critical paths of numpy to maximize the amount of exercise it > gets. While this creates more short-term hiccups as we are seeing now, it > also means the new functionality conforms to the current system better and > is much more stable since it is getting well tested. > > > Thanks again for all the good core-algorithm work, Mark. You have being > doing a great job. > Thanks! Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 6 08:54:39 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Mar 2011 06:54:39 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Sat, Mar 5, 2011 at 11:11 PM, Mark Wiebe wrote: > On Sat, Mar 5, 2011 at 8:13 PM, Travis Oliphant wrote: > >> >> On Mar 5, 2011, at 5:10 PM, Mark Wiebe wrote: >> >> On Thu, Mar 3, 2011 at 10:54 PM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> >>> >> >> >>> >>> I've had a look at the bug tracker, here's a list of tickets for >>> 1.6: >>> >>> #1748 (blocker: regression for astype('str')) >>> >>> #1619 (issue with dtypes, with patch) >>> >>> #1749 (distutils, py 3.2) >>> >>> #1601 (distutils, py 3.2) >>> >>> #1622 (Solaris segfault, with patch) >>> >>> #1713 (Solaris segfault) >>> >>> #1631 (Solaris segfault) >>> >>> The distutils tickets are resolved. >>> >>> >>> Proposed schedule: >>> >>> March 15: beta 1 >>> >>> March 28: rc 1 >>> >>> April 17: rc 2 (if needed) >>> >>> April 24: final release >>> >>> Any comments on the schedule or tickets? >>> >> >> That all looks fine to me. There are a few things that I've changed in the >> core that could stand some discussion before being finalized in 1.6, mostly >> due to what was required to make things work without depending on the data >> type enumeration order. The combination of the numpy and scipy tests were >> pretty effective, but as Travis mentioned my changes are fairly invasive. >> >> * When copying array to array, structured types now copy based on field >> names instead of positions, effectively behaving like a 'dict' instead of a >> 'labeled tuple'. This behaviour is more intuitive to me, and several fixed >> bugs such as dtype comparison completely ignoring the structured type data >> suggest that this changes an area of numpy that has been used in a more >> limited fashion. It might be worthwhile to introduce a tuple-style flag in a >> future version which causes data to be copied by position instead of by >> name, as it is likely useful in some contexts. >> >> >> This is a semantic change that does make me a tiny bit nervous. >> Structured arrays are actually used quite a bit in the wild, and so this >> could raise some errors. What I don't know is how often sub-parts of a >> structured arrays get copied into other structured arrays with a different >> order to the fields. From what I gather, Mark's changes would allow this >> case and do an arguably useful thing. Previously, a copy was only allowed >> if the structured array contained the same fields in the same order. It >> seems like this is a relaxation of a rule and should not raise any errors >> (unless extant code was relying on the previous errors for some reason). >> > > Another important factor is that previously the performance was poor, > because each copy involved converting the array element to a Python tuple, > then copying the tuple into the destination array. The new code directly > copies the elements with no Python overhead. I haven't directly benchmarked > this, but if someone wants to confirm this with some numbers that would be > great. > >> * Array memory layouts are preserved in many cases. This means that if a, >> b are Fortran ordered, a+b will be as well. It could be made more pervasive, >> for example ndarray.copy defaults to C-order, and that could be changed to >> 'K' to preserve the memory layout by default. Any comments about that? >> >> >> I like this change quite a bit, but it has similar potential "expectation" >> issues. I think the default should be changed to 'K' in NumPy 2.0, but >> perhaps we should preserve C-order for now to avoid the subtle breakages >> that might occur based on changed expectations. What are others thoughts? >> > > I suspect defaulting to 'C' might be desirable, but I initially set it to > 'K' to see how it would work out. Defaulting it to 'C' unfortunately kills > most of the performance benefits of the new code, so it might be nice to > leave it as 'K' if no issues arise that are traced back to here. > > I suppose this might cause a problem with lazy/quick c extensions that expected elements in a certain order, so some breakage could occur. The strict rule for backward compatibility would be no breakage, and if there was no performance gain I would opt for that. But in this case there is a real gain in breaking compatibility in a small way that is unlikely to be noticed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.gnata at gmail.com Sun Mar 6 09:41:20 2011 From: xavier.gnata at gmail.com (Xavier Gnata) Date: Sun, 06 Mar 2011 15:41:20 +0100 Subject: [Numpy-discussion] np.linalg.norm terrible behavior In-Reply-To: References: <4D72C6E3.6050002@gmail.com> Message-ID: <4D739D10.3090403@gmail.com> Kahan summation algorithm? pairwise summation? Looks like BLAS does not use these algorithms. I'm curious to know which algo are coded in matlab/scilab. Another issue is that sum is slow. Xavier > for the sum function part, > It's not a good way to fix but... if you want more accuracy > > x=(np.array([1]*10000 + [1e4], dtype=np.float32)) > np.sum(x*x) > 1.0001e+08 > > You can sort x from small numbers to bigger numbers before you call sum. > > -Kibeom Kim > > On Sat, Mar 5, 2011 at 6:27 PM, Xavier Gnata > wrote: > > Hi, > > I got this problem in a real life code and it took me some time to > figure out that np.linalg.norm has a terrible numerical behavior. > The problem is nicely described here > http://fseoane.net/blog/2011/computing-the-vector-norm/ > > numpy/linalg/linalg.py claims to be a "high-level Python interface to > the LAPACK library". > That's great but the low level norm function is coded in pure python : > x = asarray(x) > if ord is None: # check the default case first and handle it > immediately > return sqrt(add.reduce((x.conj() * x).ravel().real)) > > Is there a way to use *by default* the blas function to compute > the norm? > > ok....looks like sum as the same terrible numerical behavior > x=(np.array([1e4] + [1]*10000, dtype=np.float32)) > np.sum(x*x) > returns 1e+08 instead of 1.0001e+08 > > Moreover, np.linalg.norm is slow compare to blas. > > Is there a way/plan to fix it? > > Xavier > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From mail.till at gmx.de Sun Mar 6 12:13:23 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Sun, 6 Mar 2011 17:13:23 +0000 (UTC) Subject: [Numpy-discussion] np.linalg.norm terrible behavior References: <4D72C6E3.6050002@gmail.com> Message-ID: > > Moreover, np.linalg.norm is slow compare to blas. > > Is there a way/plan to fix it? > > Xavier > Have a look at: http://fseoane.net/blog/2011/computing-the-vector-norm/ greetings Till From charlesr.harris at gmail.com Sun Mar 6 14:00:30 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Mar 2011 12:00:30 -0700 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses In-Reply-To: References: <28A39E36-034E-4A62-AB24-3D545769C7F6@gmail.com> Message-ID: On Tue, Mar 1, 2011 at 8:51 AM, Wes McKinney wrote: > On Mon, Feb 28, 2011 at 10:52 PM, Wes McKinney > wrote: > > On Mon, Feb 28, 2011 at 7:24 PM, Pierre GM wrote: > >> > >> On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: > >> > >>> On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney > wrote: > >>>> I'm having some trouble with the zeros_like function via np.fix: > >>>> > >>>> def zeros_like(a): > >>>> if isinstance(a, ndarray): > >>>> res = ndarray.__new__(type(a), a.shape, a.dtype, > order=a.flags.fnc) > >>>> res.fill(0) > >>>> return res > >>>> try: > >>>> wrap = a.__array_wrap__ > >>>> except AttributeError: > >>>> wrap = None > >>>> a = asarray(a) > >>>> res = zeros(a.shape, a.dtype) > >>>> if wrap: > >>>> res = wrap(res) > >>>> return res > >>>> > >>>> As you can see this is going to discard any metadata stored in a > >>>> subtype. I'm not sure whether this is a bug or a feature but wanted to > >>>> bring it up. > >>>> > >>>> Thanks, > >>>> Wes > >>>> _______________________________________________ > >>>> NumPy-Discussion mailing list > >>>> NumPy-Discussion at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>> > >>> > >>> I guess this is ticket 929. > >>> http://projects.scipy.org/numpy/ticket/929 > >>> > >>> I was looking at it today but was not sure what is really desired > >>> here. I considered that this just meant shape and dtype but not sure > >>> about masked or record arrays behavior. So: > >>> What is the value of having the metadata? > >>> What is the meaning of 'like' here? > >> > >> Well, that depends on what you wanna do, of course. To handle metadata, > I use some kind of dictionary updated in the __array_finalize__. Check > numpy.ma.MaskedArray and its subclasses (like scikits.timeseries.TimeSeries) > for the details. > >> Now that you could store some extra data in the dtype (if I remmbr and > understand correctly), it might be worth considering a proper way to deal > with that. > >> > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > The ticket is exactly related to the problem at hand-- having > > __array_finalize__ defined won't help you as it never gets called. > > > > Looks like this commit fixed the problem, so the ticket can be closed > > > https://github.com/numpy/numpy/commit/c9d1849332ae5bf73299ea1268f6a55f78624688#numpy/core/numeric.py > ______ > I've closed the ticket. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.c.cartwright at comcast.net Sun Mar 6 18:18:43 2011 From: john.c.cartwright at comcast.net (John Cartwright) Date: Sun, 6 Mar 2011 16:18:43 -0700 Subject: [Numpy-discussion] problems w/ fortran compiler when compiling 1.5.1 on RHEL5 Message-ID: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> Hello All, I'm trying to compile 1.5.1 on a RHEL5 system and I'm specifying: python setup.py config_fc --fcompiler=gnu95 build when doing the build. I see the message "Found executable /usr/bin/gfortran" - so far so good. However when I go to install using: python setup.py install I see the warnings "gnu: no Fortran 90 compiler found". How can I avoid this? install doesn't recognize the fcompiler option. I also notice that "python setup.py clean" keeps reporting "language = f77". It appears that both f77 and gfortran are installed although the gfortran appears to have been used to compile liblapack.so (libgfortran.so.1 is listed as a dependency). Can someone please help me? Thanks! --john From ralf.gommers at googlemail.com Sun Mar 6 22:03:17 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 7 Mar 2011 11:03:17 +0800 Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal Message-ID: Hi, I went through the current deprecations, and removed a number of things that have been deprecated for a long time, see https://github.com/rgommers/numpy/commits/deprecations-1.6. I'll commit that unless anyone objects. If anyone has new deprecations they want to put in for 1.6, discussing them now would be good. I found one item in Trac, #1543. The proposal in the ticket is to deprecate assert_almost_equal because it is quite badly behaved. This function is quite widely used however (also in scipy and presumably in third party code), so it will be a real pain if it starts spewing out warnings. I propose instead to just add a note at the top of the assert_almost_equal docstring that assert_array_max_ulp or assert_tol_equal (to be added from scipy.special in 1.6) should be used instead for new code. Cheers, Ralf From ralf.gommers at googlemail.com Sun Mar 6 23:12:55 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 7 Mar 2011 12:12:55 +0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Sun, Mar 6, 2011 at 1:10 AM, Skipper Seabold wrote: > On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers > wrote: >> On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >>> The page >>> >>> gives a good introduction to structured arrays. However, it says nothing >>> about how to set a particular element (all fields at once) from a >>> collection of data. >>> >>> For instance: >>> >>> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >>> >>> The question is how to set stArr[0]? >>> >>> >From experimentation it appears that you can provide a tuple, but not a >>> list. Hence the following works just fine (and that the tuple can >>> contain a list): >>> strArr[0,0] = ([1.0, 1.1], 2.0) >>> >>> but the following fails: >>> strArr[0,0] = [[1.0, 1.1], 2.0] >>> with an error: >>> TypeError: expected a readable buffer object >>> >>> This is useful information if one is trying to initialize a structured >>> array from a collection of data, such as that returned from a database >>> query. >>> > > I added a bit at the end here, though it is mentioned briefly above. > Feel free to expand. It's a wiki. You just need edit rights. > > http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ Thanks, I'll make sure that goes in for 1.6.0. >> I'm wondering if that's not a bug? If it's intentional then it is >> certainly counterintuitive. >> > > This comes up from time to time. > > http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 > > Perhaps an enhancement ticket could be filed? It doesn't sound trivial > to implement. I filed #1758. You can also assign with an array which fails silently, certainly a bug: >>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>> arr['var1'] = np.arange(5) >>> arr[0] = (10,20) >>> arr[0] (10.0, 20.0) >>> arr[0] = np.array([10,20]) # no exception, but garbage out >>> arr[0] (4.2439915824246103e-313, 0.0) Ralf From ralf.gommers at googlemail.com Mon Mar 7 02:56:01 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 7 Mar 2011 15:56:01 +0800 Subject: [Numpy-discussion] problems w/ fortran compiler when compiling 1.5.1 on RHEL5 In-Reply-To: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> References: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> Message-ID: On Mon, Mar 7, 2011 at 7:18 AM, John Cartwright wrote: > Hello All, > > I'm trying to compile 1.5.1 on a RHEL5 system and I'm specifying: > > ?python setup.py config_fc --fcompiler=gnu95 build > > when doing the build. I see the message "Found executable /usr/bin/gfortran" - so far so good. ?However when I go to install using: > > python setup.py install > > I see the warnings "gnu: no Fortran 90 compiler found". ?How can I avoid this? install doesn't recognize the fcompiler option. Are you actually getting an error or just a warning? Are you trying to install numpy or scipy (Fortran is not needed for scipy)? If no error, what do you get when running ">>> numpy.test()" ? > > It appears that both f77 and gfortran are installed although the gfortran appears to have been used to compile liblapack.so (libgfortran.so.1 is listed as a dependency). You can move g77 off your PATH, then there's no chance of mixing up compilers. Ralf From gokhansever at gmail.com Mon Mar 7 03:53:45 2011 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 7 Mar 2011 01:53:45 -0700 Subject: [Numpy-discussion] Anybody going to PyCon? Message-ID: Hello, I am going to the PyCon this week. I am presenting a poster about an atmospheric sciences related project -- the most active development from my coding site over at http://code.google.com/p/ccnworks/ Is there anybody in the community participating there as well? Any plans for sprinting or similar activities? See you at PyCon. -- G?khan From pav at iki.fi Mon Mar 7 04:10:20 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 09:10:20 +0000 (UTC) Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal References: Message-ID: Mon, 07 Mar 2011 11:03:17 +0800, Ralf Gommers wrote: [clip] > If anyone has new deprecations they want to put in for 1.6, discussing > them now would be good. I found one item in Trac, #1543. The proposal in > the ticket is to deprecate assert_almost_equal because it is quite badly > behaved. This function is quite widely used however (also in scipy and > presumably in third party code), so it will be a real pain if it starts > spewing out warnings. > > I propose instead to just add a note at the top of the > assert_almost_equal docstring that assert_array_max_ulp or > assert_tol_equal (to be added from scipy.special in 1.6) should be used > instead for new code. Numpy 1.5 has `assert_allclose` which is functionally equivalent to the proposed `assert_tol_equal`, so no new functions need to be added. I'm OK with documentation-only recommendation. Pauli From josef.pktd at gmail.com Mon Mar 7 06:39:20 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 7 Mar 2011 06:39:20 -0500 Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 4:10 AM, Pauli Virtanen wrote: > Mon, 07 Mar 2011 11:03:17 +0800, Ralf Gommers wrote: > [clip] >> If anyone has new deprecations they want to put in for 1.6, discussing >> them now would be good. I found one item in Trac, #1543. The proposal in >> the ticket is to deprecate assert_almost_equal because it is quite badly >> behaved. This function is quite widely used however (also in scipy and >> presumably in third party code), so it will be a real pain if it starts >> spewing out warnings. >> >> I propose instead to just add a note at the top of the >> assert_almost_equal docstring that assert_array_max_ulp or >> assert_tol_equal (to be added from scipy.special in 1.6) should be used >> instead for new code. > > Numpy 1.5 has `assert_allclose` which is functionally equivalent to the > proposed `assert_tol_equal`, so no new functions need to be added. I'm OK > with documentation-only recommendation. I also think assert_almost_equal should not be depreciated. Besides being very heavily used, it is also the most appropriate test for many statistical tests. Test statistics like t-tests and p-values have an absolute scale, and only a few decimals make statistical sense. The tolerance for calculations (minimization, numerical differentiation and integration) is often only targeted for this. Changing the signature just to replicate it with assert_allclose is a pain without any gain. And going over all the tests to see what the appropriate relative error is, is a lot of work. But we have some tickets in scipy stats to improve numerical precision, beyond some decimals, and relative tolerance would be useful in those cases. I didn't know about assert_allclose and more advertising will help to use it more often for new tests. Why does assert_allclose have atol=0, while np.allclose has rtol=1.e-5, atol=1.e-8 ? What's the status on np.testing.assert_approx_equal, I would have liked to use it more often, except it doesn't work on arrays. Josef > > ? ? ? ?Pauli > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From miguel.deval at gmail.com Mon Mar 7 06:47:47 2011 From: miguel.deval at gmail.com (Miguel de Val-Borro) Date: Mon, 7 Mar 2011 12:47:47 +0100 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: <20110307114746.GE2296@poincare> Hi G?khan, I will present a poster about astronomical data reduction and visualization on Sunday. I will arrive to Atlanta on Wednesday morning to attend a couple of tutorials before the conference and look forward to participating in sprints or other activities. Regards, Miguel On Mon, Mar 07, 2011 at 01:53:45AM -0700, G?khan Sever wrote: > Hello, > > I am going to the PyCon this week. I am presenting a poster about an > atmospheric sciences related project -- the most active development > from my coding site over at http://code.google.com/p/ccnworks/ > > Is there anybody in the community participating there as well? Any > plans for sprinting or similar activities? > > See you at PyCon. > > -- > G?khan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Mon Mar 7 06:53:02 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 11:53:02 +0000 (UTC) Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal References: Message-ID: Mon, 07 Mar 2011 06:39:20 -0500, josef.pktd wrote: [clip] > Why does assert_allclose have atol=0, while np.allclose has rtol=1.e-5, > atol=1.e-8 ? Probably no reason, it should be fixed. > What's the status on np.testing.assert_approx_equal, I would have liked > to use it more often, except it doesn't work on arrays. It's probably going to stay as well. The test routines have a common framework for working with arrays, but maybe that routine dates from way back, and wasn't ported to use it. Both the assert_approx_equal and assert_almost_equal also have the built-in ugliness of specifying precision via decimals= which is somewhat clumsy for numerical work. -- Pauli Virtanen From faltet at pytables.org Mon Mar 7 07:17:55 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 13:17:55 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: <201103071317.55110.faltet@pytables.org> A Monday 28 February 2011 16:31:59 Ralf Gommers escrigu?: > >> Proposed schedule: > >> March 15: beta 1 > >> March 28: rc 1 > >> April 17: rc 2 (if needed) > >> April 24: final release > >> > >> Let me know what you think. Bonus points for volunteering to fix > >> some of those tickets:) While doing tests on the new NumPy 1.6 code, I'm seeing that the compiled code with NumPy 1.5 no longer works with 1.6. Here it is the problem: $ PYTHONPATH=. python tables/tests/test_all.py Traceback (most recent call last): File "tables/tests/test_all.py", line 13, in import tables File "/home/faltet/PyTables/pytables/trunk/tables/__init__.py", line 63, in from tables.utilsExtension import getPyTablesVersion, getHDF5Version File "definitions.pxd", line 138, in init tables.utilsExtension (tables/utilsExtension.c:9238) ValueError: numpy.dtype has the wrong size, try recompiling I've seen the same issue when NumPy jumped from 1.3.x to 1.4.1, forcing all the packages depending on NumPy to be recompiled to work with 1.4.1 and higher. Do we really want to repeat the experience with 1.5.x --> 1.6.x? -- Francesc Alted From mailshuwei at gmail.com Mon Mar 7 07:16:15 2011 From: mailshuwei at gmail.com (shu wei) Date: Mon, 7 Mar 2011 06:16:15 -0600 Subject: [Numpy-discussion] How to sum weighted matrices Message-ID: Hello all, I am new to python and numpy. My question is how to sum up N weighted matrices. For example w=[1,2] (N=2 case) m1=[1 2 3, 3 4 5] m2=[3 4 5, 4 5 6] I want to get a matrix Y=w[1]*m1+w[2]*m2 by using a loop. My original problem is like this X=[1 2 3, 3 4 5, 4 5 6] a1=[1 2 3] 1st row of X m1=a1'*a1 a matirx a2=[3 4 5] 2nd row of X m2=a2'*a2 a3=[ 4 5 6] 3rd row of X m3=a3'*a3 I want to get Y1=w[1]*m1+w[2]*m2 Y2=w[1]*m2+w[2]*m3 So basically it is rolling and to sum up the weighted matries I have a big X, the rolling window is relatively small. I tried to use sq=np.array([x[i].reshape(-1,1)*x[i] for i in np.arange(0,len(x)]) # s=len(x) m=np.array([sq[i:i+t] for i in np.arange(0,s-t+1)]) # t is the len(w) then I was stuck, I tried to use a loop somethig like Y=np.array([np.sum(w[i]*m[j,i],axis=0) for i in np.arange(0,t)] ) Any suggestion is welcome. sue -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Mon Mar 7 07:30:30 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 13:30:30 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: <201103071330.30522.faltet@pytables.org> A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: > I think it's ok to revert this behavior for backwards compatibility, > but believe it's an inconsistent and unintuitive choice. In > broadcasting, there are two operations, growing a dimension 1 -> n, > and appending a new 1 dimension to the left. The behaviour under > discussion in assignment is different from normal broadcasting in > that only the second one is permitted. It is broadcasting the output > to the input, rather than broadcasting the input to the output. > > Suppose a has shape (20,), b has shape (1,20), and c has shape > (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c > has shape (20,20). > > If we do "b[...] = a", a will be broadcast to match b by adding a 1 > dimension to the left. This is reasonable and consistent with > addition. > > If we do "a[...]=b", under 1.5 rules, a will once again be broadcast > to match b by adding a 1 dimension to the left. > > If we do "a[...]=c", we could broadcast both a and c together to the > shape (20,20). This results in multiple assignments to each element > of a, which is inconsistent. This is not analogous to a+c, but > rather to np.add(c, c, out=a). > > The distinction is subtle, but the inconsistent behavior is harmless > enough for assignment that keeping backwards compatibility seems > reasonable. For what is worth, I also like the behaviour that Mark proposes, and have updated tables test suite to adapt to this. But I'm fine if it is decided to revert to the previous behaviour. -- Francesc Alted From josef.pktd at gmail.com Mon Mar 7 08:30:11 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 7 Mar 2011 08:30:11 -0500 Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 6:53 AM, Pauli Virtanen wrote: > Mon, 07 Mar 2011 06:39:20 -0500, josef.pktd wrote: > [clip] >> Why does assert_allclose have atol=0, while np.allclose has rtol=1.e-5, >> atol=1.e-8 ? > > Probably no reason, it should be fixed. > >> What's the status on np.testing.assert_approx_equal, I would have liked >> to use it more often, except it doesn't work on arrays. > > It's probably going to stay as well. The test routines have a common > framework for working with arrays, but maybe that routine dates from way > back, and wasn't ported to use it. > > Both the assert_approx_equal and assert_almost_equal also have the > built-in ugliness of specifying precision via decimals= which is somewhat > clumsy for numerical work. assert_approx_equal checks for signigicant digits in decimal system, which looks like it's easy to interpret. I don't have much idea what a nulp is, and whether it's machine dependent. >>> np.testing.assert_array_max_ulp(x, np.ones(10)) Traceback (most recent call last): File "", line 1, in np.testing.assert_array_max_ulp(x, np.ones(10)) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 1231, in assert_array_max_ulp ret = nulp_diff(a, b, dtype) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 1271, in nulp_diff raise NotImplementedError("_nulp not implemented for complex array") NotImplementedError: _nulp not implemented for complex array Josef > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Mon Mar 7 08:37:24 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 13:37:24 +0000 (UTC) Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal References: Message-ID: Mon, 07 Mar 2011 08:30:11 -0500, josef.pktd wrote: [clip] > assert_approx_equal checks for signigicant digits in decimal system, > which looks like it's easy to interpret. Ditto for tolerance=1e-7, which has the advantage that it's what "print abs(desired-actual)" prints. > I don't have much idea what a nulp is, and whether it's machine > dependent. It's the number of floating point values between the desired and the actual results --- this depends only on the fp type. It is *the* correct measure when you want "uniform" accuracy across the range of all floating point values. For instance, this is quite useful for special functions, whose range of values typically vary widely, but for which one still wants to have accuracy which is as good as possible. -- Pauli Virtanen From charlesr.harris at gmail.com Mon Mar 7 08:40:51 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Mar 2011 06:40:51 -0700 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: <201103071317.55110.faltet@pytables.org> References: <201103071317.55110.faltet@pytables.org> Message-ID: On Mon, Mar 7, 2011 at 5:17 AM, Francesc Alted wrote: > A Monday 28 February 2011 16:31:59 Ralf Gommers escrigu?: > > >> Proposed schedule: > > >> March 15: beta 1 > > >> March 28: rc 1 > > >> April 17: rc 2 (if needed) > > >> April 24: final release > > >> > > >> Let me know what you think. Bonus points for volunteering to fix > > >> some of those tickets:) > > While doing tests on the new NumPy 1.6 code, I'm seeing that the > compiled code with NumPy 1.5 no longer works with 1.6. Here it is the > problem: > > $ PYTHONPATH=. python tables/tests/test_all.py > Traceback (most recent call last): > File "tables/tests/test_all.py", line 13, in > import tables > File "/home/faltet/PyTables/pytables/trunk/tables/__init__.py", line > 63, in > from tables.utilsExtension import getPyTablesVersion, getHDF5Version > File "definitions.pxd", line 138, in init tables.utilsExtension > (tables/utilsExtension.c:9238) > ValueError: numpy.dtype has the wrong size, try recompiling > > I've seen the same issue when NumPy jumped from 1.3.x to 1.4.1, forcing > all the packages depending on NumPy to be recompiled to work with 1.4.1 > and higher. Do we really want to repeat the experience with 1.5.x --> > 1.6.x? > > IIRC, that was a cython problem. I thought they fixed that to issue a warning instead. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Mar 7 08:49:26 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 13:49:26 +0000 (UTC) Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) References: <201103071317.55110.faltet@pytables.org> Message-ID: Mon, 07 Mar 2011 13:17:55 +0100, Francesc Alted wrote: [clip] > from tables.utilsExtension import getPyTablesVersion, getHDF5Version > File "definitions.pxd", line 138, in init tables.utilsExtension > (tables/utilsExtension.c:9238) > ValueError: numpy.dtype has the wrong size, try recompiling > > I've seen the same issue when NumPy jumped from 1.3.x to 1.4.1, forcing > all the packages depending on NumPy to be recompiled to work with 1.4.1 > and higher. Do we really want to repeat the experience with 1.5.x --> > 1.6.x? I got the impression that this unnecessary check was removed in newer Cython versions? If so, some of the existing code should not have this problem. However, the size of PyArray_Descr does not seem to have changed between 1.5.1 and the Git master. So I'm not sure why you see this error... -- Pauli Virtanen From faltet at pytables.org Mon Mar 7 08:57:39 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 14:57:39 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <201103071317.55110.faltet@pytables.org> Message-ID: <201103071457.39202.faltet@pytables.org> A Monday 07 March 2011 14:49:26 Pauli Virtanen escrigu?: > Mon, 07 Mar 2011 13:17:55 +0100, Francesc Alted wrote: > [clip] > > > from tables.utilsExtension import getPyTablesVersion, > > getHDF5Version > > > > File "definitions.pxd", line 138, in init tables.utilsExtension > > > > (tables/utilsExtension.c:9238) > > ValueError: numpy.dtype has the wrong size, try recompiling > > > > I've seen the same issue when NumPy jumped from 1.3.x to 1.4.1, > > forcing all the packages depending on NumPy to be recompiled to > > work with 1.4.1 and higher. Do we really want to repeat the > > experience with 1.5.x --> 1.6.x? > > I got the impression that this unnecessary check was removed in newer > Cython versions? If so, some of the existing code should not have > this problem. Well, I'm using the latest Cython: $ cython -V Cython version 0.14.1 > However, the size of PyArray_Descr does not seem to have changed > between 1.5.1 and the Git master. So I'm not sure why you see this > error... Maybe a Cython problem? -- Francesc Alted From cournape at gmail.com Mon Mar 7 09:11:51 2011 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Mar 2011 23:11:51 +0900 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 5:53 PM, G?khan Sever wrote: > Hello, > > I am going to the PyCon this week. I am presenting a poster about an > atmospheric sciences related project -- the most active development > from my coding site over at http://code.google.com/p/ccnworks/ > > Is there anybody in the community participating there as well? Any > plans for sprinting or similar activities? Will be there as well, but only for the WE + monday, cheers, David From pav at iki.fi Mon Mar 7 09:17:17 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 14:17:17 +0000 (UTC) Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) References: <201103071317.55110.faltet@pytables.org> <201103071457.39202.faltet@pytables.org> Message-ID: Mon, 07 Mar 2011 14:57:39 +0100, Francesc Alted wrote: [clip] > > However, the size of PyArray_Descr does not seem to have changed > > between 1.5.1 and the Git master. So I'm not sure why you see this > > error... > > Maybe a Cython problem? That would be seriously weird. Maybe the binaries you have were compiled against Numpy < 1.5? If so, the same errors should come up with Numpy 1.5.x. In any case, it seems to me that since there have been no changes in the dtype struct contents since 1.5, the Git master should behave exactly the same as 1.5 in this respect. -- Pauli Virtanen From faltet at pytables.org Mon Mar 7 09:23:10 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 15:23:10 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <201103071457.39202.faltet@pytables.org> Message-ID: <201103071523.10717.faltet@pytables.org> A Monday 07 March 2011 15:17:17 Pauli Virtanen escrigu?: > Mon, 07 Mar 2011 14:57:39 +0100, Francesc Alted wrote: > [clip] > > > > However, the size of PyArray_Descr does not seem to have changed > > > between 1.5.1 and the Git master. So I'm not sure why you see > > > this error... > > > > Maybe a Cython problem? > > That would be seriously weird. Maybe the binaries you have were > compiled against Numpy < 1.5? If so, the same errors should come up > with Numpy 1.5.x. > > In any case, it seems to me that since there have been no changes in > the dtype struct contents since 1.5, the Git master should behave > exactly the same as 1.5 in this respect. I've just tried again, and I'm reproducing the problem. Using NumPy 1.5.0: $ PYTHONPATH=.:~/python/numpy/trunk:~/python/numexpr/branches/1.4 python tables/tests/test_all.py -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-= PyTables version: 2.3b1.dev HDF5 version: 1.8.5 NumPy version: 1.5.0.dev Numexpr version: 1.4.3.dev (not using Intel's VML/MKL) .... [OK] With the same PyTables binaries, but using NumPy 1.6: $ PYTHONPATH=.:~/python/numpy/git/numpy:~/python/numexpr/branches/1.4/ python tables/tests/test_all.py Traceback (most recent call last): File "tables/tests/test_all.py", line 13, in import tables File "/home/faltet/PyTables/pytables/trunk/tables/__init__.py", line 63, in from tables.utilsExtension import getPyTablesVersion, getHDF5Version File "definitions.pxd", line 138, in init tables.utilsExtension (tables/utilsExtension.c:9238) ValueError: numpy.dtype has the wrong size, try recompiling I don't think I'm wrong here, but I'd appreciate if somebody else can reproduce this (either with tables or with another Cython-dependent package). -- Francesc Alted From pav at iki.fi Mon Mar 7 09:39:38 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 14:39:38 +0000 (UTC) Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) References: <201103071457.39202.faltet@pytables.org> <201103071523.10717.faltet@pytables.org> Message-ID: Mon, 07 Mar 2011 15:23:10 +0100, Francesc Alted wrote: [clip] > ValueError: numpy.dtype has the wrong size, try recompiling > > I don't think I'm wrong here, but I'd appreciate if somebody else can > reproduce this (either with tables or with another Cython-dependent > package). Ok, seems this needs looking into. I don't immediately see how it can happen, and agree that it shouldn't (AFAIK, there are no changes in 1.6 that require additions to the structs). -- Pauli Virtanen From john.c.cartwright at comcast.net Mon Mar 7 09:55:45 2011 From: john.c.cartwright at comcast.net (John Cartwright) Date: Mon, 7 Mar 2011 07:55:45 -0700 Subject: [Numpy-discussion] problems w/ fortran compiler when compiling 1.5.1 on RHEL5 In-Reply-To: References: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> Message-ID: <790167FC-D6B1-495D-AAAF-BC7A3EEA5F33@comcast.net> Thanks for your reply Ralf. I'm not sure if it's a warning or error, output looks like: ... building library "npymath" sources customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config ... Not sure about the best way to move g77 off the path since it shares the same directory as gfortran and would require sysadmin intervention. Is there no way through site.cfg or some other means to force the setup.py to ignore g77? --john On Mar 7, 2011, at 12:56 AM, Ralf Gommers wrote: > On Mon, Mar 7, 2011 at 7:18 AM, John Cartwright > wrote: >> Hello All, >> >> I'm trying to compile 1.5.1 on a RHEL5 system and I'm specifying: >> >> python setup.py config_fc --fcompiler=gnu95 build >> >> when doing the build. I see the message "Found executable /usr/bin/gfortran" - so far so good. However when I go to install using: >> >> python setup.py install >> >> I see the warnings "gnu: no Fortran 90 compiler found". How can I avoid this? install doesn't recognize the fcompiler option. > > Are you actually getting an error or just a warning? Are you trying to > install numpy or scipy (Fortran is not needed for scipy)? If no error, > what do you get when running ">>> numpy.test()" ? >> >> It appears that both f77 and gfortran are installed although the gfortran appears to have been used to compile liblapack.so (libgfortran.so.1 is listed as a dependency). > > You can move g77 off your PATH, then there's no chance of mixing up compilers. > > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ndbecker2 at gmail.com Mon Mar 7 10:01:21 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 07 Mar 2011 10:01:21 -0500 Subject: [Numpy-discussion] view 1d array as overlapping segments? Message-ID: reshape can view a 1d array as non-overlapping segments. Is there a convenient way to view a 1d array as a 2d array of overlapping segments? nonoverlapping: l: segment length k: overlap u is the 1d array v is a 2d array v[i] = u[l*i:(l+1)*i] overlapping: v[i] = u[l*i:(l+1)*i+k] From josef.pktd at gmail.com Mon Mar 7 10:05:50 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 7 Mar 2011 10:05:50 -0500 Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 8:37 AM, Pauli Virtanen wrote: > Mon, 07 Mar 2011 08:30:11 -0500, josef.pktd wrote: > [clip] >> assert_approx_equal ?checks for signigicant digits in decimal system, >> which looks like it's easy to interpret. > > Ditto for tolerance=1e-7, which has the advantage that it's what > "print abs(desired-actual)" prints. Ok, it took me a while to figure out what the issue is, atol is more intuitive then decimal, especially since decimal has the half point/rounding behavior that I'm never quite sure about. I initially thought the issue is between atol versus rtol versus nulp. Is there a reason that assert_array_almost_equal uses around(z, decimal) <= 10.0**(-decimal) while the last part of assert _almost_equal is if round(abs(desired - actual),decimal) != 0 > >> I don't have much idea what a nulp is, and whether it's machine >> dependent. > > It's the number of floating point values between the desired and the > actual results --- this depends only on the fp type. It is *the* correct > measure when you want "uniform" accuracy across the range of all floating > point values. For instance, this is quite useful for special functions, > whose range of values typically vary widely, but for which one still > wants to have accuracy which is as good as possible. Is nulp_diff exposed somewhere? It looks like a useful functions to write nulp tests, and I only saw it in the source. Josef > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Mon Mar 7 10:13:30 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 15:13:30 +0000 (UTC) Subject: [Numpy-discussion] view 1d array as overlapping segments? References: Message-ID: Mon, 07 Mar 2011 10:01:21 -0500, Neal Becker wrote: > reshape can view a 1d array as non-overlapping segments. > > Is there a convenient way to view a 1d array as a 2d array of > overlapping segments? > > nonoverlapping: > l: segment length > k: overlap > u is the 1d array > v is a 2d array > > v[i] = u[l*i:(l+1)*i] > > overlapping: > v[i] = u[l*i:(l+1)*i+k] Yes, specify from numpy.lib.stride_tricks import as_strided strides = (u.dtype.itemsize*l, u.dtype.itemsize) shape = (num_slices, l+k) v = as_strided(u, strides=strides, shape=shape) num_slices needs to be chosen so that you don't access out-of-bounds memory. `as_strided` doesn't check for this. From davide.lasagna at polito.it Mon Mar 7 10:14:44 2011 From: davide.lasagna at polito.it (Davide) Date: Mon, 07 Mar 2011 16:14:44 +0100 Subject: [Numpy-discussion] view 1d array as overlapping segments? In-Reply-To: References: Message-ID: <4D74F664.8010401@polito.it> Here is some *working* code i wrote once. It uses strides, look at the docs for what it is. from numpy.lib import stride_tricks def overlap_array( y, len_blocks, overlap=0 ): """ Make use of strides to return a two dimensional whose rows come from a one dimensional array. Strides are used to return rows that partially overlap. Parameters ---------- y : a one dimensional array len_blocks : the row length. The length of chunks from y. overlap : number of elements that overlap. From 0 (no overlap) to len_blocks-1 (almost full overlap). Returns ------- x : a strided array """ overlap = int(overlap) len_blocks = int(len_blocks) if not type(y) == np.ndarray: raise ValueError( 'y must be a numpy.ndarray' ) if overlap >= len_blocks: raise ValueError( 'overlap must be less than n_points' ) # compute shape and strides of the strided vector strides = ( (len_blocks - overlap)*y.itemsize, y.itemsize ) shape = ( 1 + (y.nbytes - len_blocks*y.itemsize)/strides[0], len_blocks) # create a strided array return stride_tricks.as_strided( y, shape=shape, strides=strides ) On 03/07/2011 04:01 PM, Neal Becker wrote: > reshape can view a 1d array as non-overlapping segments. > > Is there a convenient way to view a 1d array as a 2d array of overlapping > segments? > > nonoverlapping: > l: segment length > k: overlap > u is the 1d array > v is a 2d array > > v[i] = u[l*i:(l+1)*i] > > overlapping: > v[i] = u[l*i:(l+1)*i+k] > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From d.s.seljebotn at astro.uio.no Mon Mar 7 10:15:06 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 07 Mar 2011 16:15:06 +0100 Subject: [Numpy-discussion] problems w/ fortran compiler when compiling 1.5.1 on RHEL5 In-Reply-To: <790167FC-D6B1-495D-AAAF-BC7A3EEA5F33@comcast.net> References: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> <790167FC-D6B1-495D-AAAF-BC7A3EEA5F33@comcast.net> Message-ID: <4D74F67A.6080203@student.matnat.uio.no> On 03/07/2011 03:55 PM, John Cartwright wrote: > Thanks for your reply Ralf. I'm not sure if it's a warning or error, output looks like: > > ... > building library "npymath" sources > customize GnuFCompiler > Found executable /usr/bin/g77 > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize GnuFCompiler > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize GnuFCompiler using config > ... > > Not sure about the best way to move g77 off the path since it shares the same directory as gfortran and would require sysadmin intervention. > > Is there no way through site.cfg or some other means to force the setup.py to ignore g77? > > --john > > On Mar 7, 2011, at 12:56 AM, Ralf Gommers wrote: > >> On Mon, Mar 7, 2011 at 7:18 AM, John Cartwright >> wrote: >>> Hello All, >>> >>> I'm trying to compile 1.5.1 on a RHEL5 system and I'm specifying: >>> >>> python setup.py config_fc --fcompiler=gnu95 build >>> >>> when doing the build. I see the message "Found executable /usr/bin/gfortran" - so far so good. However when I go to install using: >>> >>> python setup.py install >>> >>> I see the warnings "gnu: no Fortran 90 compiler found". How can I avoid this? install doesn't recognize the fcompiler option. Does either of these work? python setup.py config_fc --fcompiler=gnu95 build install python setup.py config_fc --fcompiler=gnu95 build --fcompiler=gnu95 install The sooner numpy gets rid of distutils the better... Dag Sverre From d.s.seljebotn at astro.uio.no Mon Mar 7 10:18:21 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 07 Mar 2011 16:18:21 +0100 Subject: [Numpy-discussion] problems w/ fortran compiler when compiling 1.5.1 on RHEL5 In-Reply-To: <4D74F67A.6080203@student.matnat.uio.no> References: <409F94B7-0FB6-4A6E-A5C4-D4FB092293D7@comcast.net> <790167FC-D6B1-495D-AAAF-BC7A3EEA5F33@comcast.net> <4D74F67A.6080203@student.matnat.uio.no> Message-ID: <4D74F73D.8060102@student.matnat.uio.no> On 03/07/2011 04:15 PM, Dag Sverre Seljebotn wrote: > On 03/07/2011 03:55 PM, John Cartwright wrote: >> Thanks for your reply Ralf. I'm not sure if it's a warning or error, output looks like: >> >> ... >> building library "npymath" sources >> customize GnuFCompiler >> Found executable /usr/bin/g77 >> gnu: no Fortran 90 compiler found >> gnu: no Fortran 90 compiler found >> customize GnuFCompiler >> gnu: no Fortran 90 compiler found >> gnu: no Fortran 90 compiler found >> customize GnuFCompiler using config >> ... >> >> Not sure about the best way to move g77 off the path since it shares the same directory as gfortran and would require sysadmin intervention. >> >> Is there no way through site.cfg or some other means to force the setup.py to ignore g77? >> >> --john >> >> On Mar 7, 2011, at 12:56 AM, Ralf Gommers wrote: >> >>> On Mon, Mar 7, 2011 at 7:18 AM, John Cartwright >>> wrote: >>>> Hello All, >>>> >>>> I'm trying to compile 1.5.1 on a RHEL5 system and I'm specifying: >>>> >>>> python setup.py config_fc --fcompiler=gnu95 build >>>> >>>> when doing the build. I see the message "Found executable /usr/bin/gfortran" - so far so good. However when I go to install using: >>>> >>>> python setup.py install >>>> >>>> I see the warnings "gnu: no Fortran 90 compiler found". How can I avoid this? install doesn't recognize the fcompiler option. > Does either of these work? > > python setup.py config_fc --fcompiler=gnu95 build install > python setup.py config_fc --fcompiler=gnu95 build --fcompiler=gnu95 install > > The sooner numpy gets rid of distutils the better... And on that subject, check out David's post from February ("Experimental waf support for NumPy"). You may have better luck building NumPy that way. Dag Sverre From e.antero.tammi at gmail.com Mon Mar 7 10:31:37 2011 From: e.antero.tammi at gmail.com (eat) Date: Mon, 7 Mar 2011 17:31:37 +0200 Subject: [Numpy-discussion] view 1d array as overlapping segments? In-Reply-To: References: Message-ID: Hi, On Mon, Mar 7, 2011 at 5:01 PM, Neal Becker wrote: > reshape can view a 1d array as non-overlapping segments. > > Is there a convenient way to view a 1d array as a 2d array of overlapping > segments? > > nonoverlapping: > l: segment length > k: overlap > u is the 1d array > v is a 2d array > > v[i] = u[l*i:(l+1)*i] > > overlapping: > v[i] = u[l*i:(l+1)*i+k] > In case actually v[i]= u[i* l: (i+ 1)* l+ k], then this may be useful from numpy.lib.stride_tricks import as_strided as ast def os(a, l, k= 0): shape, strides= (a.shape[0]- l+ 1, l+ k), a.strides* 2 return ast(a, shape= shape, strides= strides) if __name__ == '__main__': import numpy as np a= np.arange(7, dtype= np.int8) print os(a, 3) # [[0 1 2] # [1 2 3] # [2 3 4] # [3 4 5] # [4 5 6]] print os(a, 3, 2) # [[ 0 1 2 3 4] # [ 1 2 3 4 5] # [ 2 3 4 5 6] # [ 3 4 5 6 0] # last item garbage # [ 4 5 6 0 34]] # 2 last items garbage My two cents, eat > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Mar 7 12:08:24 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 09:08:24 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule In-Reply-To: References: Message-ID: <4D751108.5040104@noaa.gov> On 3/6/11 5:54 AM, Charles R Harris wrote: > I suppose this might cause a problem with lazy/quick c extensions that > expected elements in a certain order, so some breakage could occur. absolutely! (I've gotten a bit confused about this thread, but if this is about the question of whether structured dtypes are dtypes are assumed to always keep order, read on -- otherwise, ignore).. I have always assumed a structured dtype was much like a C struct: i.e. arrangement in memory is fixed, and the named fields are just for convenience. I was very surprised to learn that they could ever be used in more a a dict-like manner. Also, I noted a recent post to this list suggesting that folks sometime miss-use stuctured arrays (particularly folks coming from MATLAB, etc), by using them when what they really want is a dict. In that post, the poster suggested that the primary use of structured arrays was for interacting with data files and C code that has arrays of structures, and indeed, that is how I have primarily used them. So I think for performance and consistency with C code, keeping dtype order is the best way to go. If that breaks backward compatibility too much, then it shouldn't be changed, but I think it better reflects what dtypes are all about. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Mar 7 12:23:33 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Mar 2011 11:23:33 -0600 Subject: [Numpy-discussion] Numpy 1.6 schedule In-Reply-To: <4D751108.5040104@noaa.gov> References: <4D751108.5040104@noaa.gov> Message-ID: On Mon, Mar 7, 2011 at 11:08, Christopher Barker wrote: > On 3/6/11 5:54 AM, Charles R Harris wrote: >> I suppose this might cause a problem with lazy/quick c extensions that >> expected elements in a certain order, so some breakage could occur. > > absolutely! > > (I've gotten a bit confused about this thread, but if this is about the > question of whether structured dtypes are dtypes are assumed to always > keep order, read on -- otherwise, ignore).. Can someone explain exactly what changed? Or point to the changeset that made it? It's not clear to me what operations are different under Mark's changes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From Chris.Barker at noaa.gov Mon Mar 7 12:28:11 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 09:28:11 -0800 Subject: [Numpy-discussion] ragged array implimentation Message-ID: <4D7515AB.6000006@noaa.gov> Hi folks, I'm setting out to write some code to access and work with ragged arrays stored in netcdf files. It dawned on me that ragged arrays are not all that uncommon, so I'm wondering if any of you have any code you've developed that I could learn-from borrow from, etc. note that when I say a "ragged array", I mean a set of data where the each row could be a different arbitrary length: 1, 2, 3, 4 5, 6 7, 8, 9, 10, 11, 12 13, 14, 15 ... In my case, these will only be 2-d, though I suppose one could have a n-d version where the last dimension was ragged (or any dimension, I suppose, though I'm having trouble wrapping my brain around what that would look like... I'm not getting more specific about what I think the API should look like -- that is part of what I'm looking for suggestions, previous implementations, etc for. Is there any "standard" way to work with such data? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at pytables.org Mon Mar 7 12:30:17 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 18:30:17 +0100 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: <201103071523.10717.faltet@pytables.org> Message-ID: <201103071830.17195.faltet@pytables.org> A Monday 07 March 2011 15:39:38 Pauli Virtanen escrigu?: > Mon, 07 Mar 2011 15:23:10 +0100, Francesc Alted wrote: > [clip] > > > ValueError: numpy.dtype has the wrong size, try recompiling > > > > I don't think I'm wrong here, but I'd appreciate if somebody else > > can reproduce this (either with tables or with another > > Cython-dependent package). > > Ok, seems this needs looking into. I don't immediately see how it can > happen, and agree that it shouldn't (AFAIK, there are no changes in > 1.6 that require additions to the structs). Ups, forget about this. It turns out that I was using a somewhat outdated version of 1.5.x (probably pre-1.5.0). I have updated to 1.5.1, and now the 1.6 binaries are completely backward compatibles: tables suite with 1.5.1: Ran 2981 tests in 37.183s Using 1.6.0.dev-c081ad7 without recompiling tables: Ran 2981 tests in 36.716s Sorry for the false alarm! -- Francesc Alted From faltet at pytables.org Mon Mar 7 12:33:00 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 18:33:00 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7515AB.6000006@noaa.gov> References: <4D7515AB.6000006@noaa.gov> Message-ID: <201103071833.00291.faltet@pytables.org> A Monday 07 March 2011 18:28:11 Christopher Barker escrigu?: > Hi folks, > > I'm setting out to write some code to access and work with ragged > arrays stored in netcdf files. It dawned on me that ragged arrays > are not all that uncommon, so I'm wondering if any of you have any > code you've developed that I could learn-from borrow from, etc. > > note that when I say a "ragged array", I mean a set of data where the > each row could be a different arbitrary length: > > 1, 2, 3, 4 > 5, 6 > 7, 8, 9, 10, 11, 12 > 13, 14, 15 > ... > > In my case, these will only be 2-d, though I suppose one could have a > n-d version where the last dimension was ragged (or any dimension, I > suppose, though I'm having trouble wrapping my brain around what that > would look like... > > I'm not getting more specific about what I think the API should look > like -- that is part of what I'm looking for suggestions, previous > implementations, etc for. > > Is there any "standard" way to work with such data? A list of numpy arrays would not be enough? Or you want something more specific? -- Francesc Alted From pav at iki.fi Mon Mar 7 12:35:14 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 7 Mar 2011 17:35:14 +0000 (UTC) Subject: [Numpy-discussion] Numpy 1.6 schedule References: <4D751108.5040104@noaa.gov> Message-ID: Mon, 07 Mar 2011 11:23:33 -0600, Robert Kern wrote: [clip] > Can someone explain exactly what changed? Or point to the changeset that > made it? It's not clear to me what operations are different under Mark's > changes. Mark mentioned three points here: http://permalink.gmane.org/gmane.comp.python.numeric.general/42753 Below, first two. 1) Structured array copying copies by field name. Commit 22d96096bf7d5fb199ca80f2fcd04e8d27815476 Before: >>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>> y = np.array([(2, 3)], dtype=[('a', int), ('b', int)]) >>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>> x[:] = y >>> x array([(2, 3)], dtype=[('a', '>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>> x[:] = y >>> x array([(3, 2)], dtype=[('a', '>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides (200, 40, 8) After: >>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides (40, 8, 200) This gets +1 from me; the drawbacks to this are mostly in code that makes too much assumptions and makes no checks. -- Pauli Virtanen From jswhit at fastmail.fm Mon Mar 7 12:37:03 2011 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Mon, 07 Mar 2011 10:37:03 -0700 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7515AB.6000006@noaa.gov> References: <4D7515AB.6000006@noaa.gov> Message-ID: <4D7517BF.7030409@fastmail.fm> On 3/7/11 10:28 AM, Christopher Barker wrote: > Hi folks, > > I'm setting out to write some code to access and work with ragged arrays > stored in netcdf files. It dawned on me that ragged arrays are not all > that uncommon, so I'm wondering if any of you have any code you've > developed that I could learn-from borrow from, etc. > > note that when I say a "ragged array", I mean a set of data where the > each row could be a different arbitrary length: > > 1, 2, 3, 4 > 5, 6 > 7, 8, 9, 10, 11, 12 > 13, 14, 15 > ... > > In my case, these will only be 2-d, though I suppose one could have a > n-d version where the last dimension was ragged (or any dimension, I > suppose, though I'm having trouble wrapping my brain around what that > would look like... > > I'm not getting more specific about what I think the API should look > like -- that is part of what I'm looking for suggestions, previous > implementations, etc for. > > Is there any "standard" way to work with such data? > > -Chris > Chris: The netcdf4-python modules reads netcdf vlen arrays and returns numpy object arrays, where the elements of the object arrays are themselves 1d numpy arrays. I don't think there is any other way to do it. In your example, the 'ragged' array would be a 1d numpy array with dtype='O', and the individual elements would be 1d numpy arrays with dtype=int. Of course, these arrays are very awkward to deal with and operations will be slow. -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-113 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From jsalvati at u.washington.edu Mon Mar 7 12:41:47 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 7 Mar 2011 09:41:47 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7517BF.7030409@fastmail.fm> References: <4D7515AB.6000006@noaa.gov> <4D7517BF.7030409@fastmail.fm> Message-ID: @Jeff I need to work with ragged arrays too. Are object arrays of 1d numpy arrays slower than lists of 1d numpy arrays? @ Christopher I'd be interested in hearing if you come up with any better solutions. On Mon, Mar 7, 2011 at 9:37 AM, Jeff Whitaker wrote: > On 3/7/11 10:28 AM, Christopher Barker wrote: > > Hi folks, > > > > I'm setting out to write some code to access and work with ragged arrays > > stored in netcdf files. It dawned on me that ragged arrays are not all > > that uncommon, so I'm wondering if any of you have any code you've > > developed that I could learn-from borrow from, etc. > > > > note that when I say a "ragged array", I mean a set of data where the > > each row could be a different arbitrary length: > > > > 1, 2, 3, 4 > > 5, 6 > > 7, 8, 9, 10, 11, 12 > > 13, 14, 15 > > ... > > > > In my case, these will only be 2-d, though I suppose one could have a > > n-d version where the last dimension was ragged (or any dimension, I > > suppose, though I'm having trouble wrapping my brain around what that > > would look like... > > > > I'm not getting more specific about what I think the API should look > > like -- that is part of what I'm looking for suggestions, previous > > implementations, etc for. > > > > Is there any "standard" way to work with such data? > > > > -Chris > > > > Chris: The netcdf4-python modules reads netcdf vlen arrays and returns > numpy object arrays, where the elements of the object arrays are > themselves 1d numpy arrays. I don't think there is any other way to do > it. In your example, the 'ragged' array would be a 1d numpy array with > dtype='O', and the individual elements would be 1d numpy arrays with > dtype=int. Of course, these arrays are very awkward to deal with and > operations will be slow. > > -Jeff > > -- > Jeffrey S. Whitaker Phone : (303)497-6313 > Meteorologist FAX : (303)497-6449 > NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov > 325 Broadway Office : Skaggs Research Cntr 1D-113 > Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Mon Mar 7 12:50:43 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Mon, 7 Mar 2011 09:50:43 -0800 Subject: [Numpy-discussion] Numpy 1.6 schedule In-Reply-To: References: <4D751108.5040104@noaa.gov> Message-ID: On Mon, Mar 7, 2011 at 9:23 AM, Robert Kern wrote: > On Mon, Mar 7, 2011 at 11:08, Christopher Barker > wrote: > > On 3/6/11 5:54 AM, Charles R Harris wrote: > >> I suppose this might cause a problem with lazy/quick c extensions that > >> expected elements in a certain order, so some breakage could occur. > > > > absolutely! > > > > (I've gotten a bit confused about this thread, but if this is about the > > question of whether structured dtypes are dtypes are assumed to always > > keep order, read on -- otherwise, ignore).. > > Can someone explain exactly what changed? Or point to the changeset > that made it? It's not clear to me what operations are different under > Mark's changes. Here is the structured data type difference, in an example where the 1.5 behavior is problematic. In 1.5, because values are copied 'tuple-like', the previous X coordinate which is thrown away for saving but needed during simulation, gets copied into the velocity. >>> import numpy as np >>> np.__version__ '1.5.1' >>> sim_particles = np.zeros((2,), dtype=[('X', 'f4', 3), ('X_prev', 'f4', 3), ('V', 'f4', 3)]).view(np.recarray) >>> save_particles = np.zeros((2,), dtype=[('X', 'f4', 3), ('V', 'f4', 3), ('Color', 'u8', 3)]).view(np.recarray) >>> # Some particle data ... sim_particles.X_prev = 0.5 >>> sim_particles.V = 1.0 >>> sim_particles.X = sim_particles.X_prev + sim_particles.V >>> # Save the particles with a solid color ... save_particles[...] = sim_particles >>> save_particles.Color = [0,255,128] >>> save_particles rec.array([([1.5, 1.5, 1.5], [0.5, 0.5, 0.5], [0L, 255L, 128L]), ([1.5, 1.5, 1.5], [0.5, 0.5, 0.5], [0L, 255L, 128L])], dtype=[('X', '>> import numpy as np >>> np.__version__ '1.6.0.dev-766cac0' >>> sim_particles = np.zeros((2,), dtype=[('X', 'f4', 3), ('X_prev', 'f4', 3), ('V', 'f4', 3)]).view(np.recarray) >>> save_particles = np.zeros((2,), dtype=[('X', 'f4', 3), ('V', 'f4', 3), ('Color', 'u8', 3)]).view(np.recarray) >>> # Some particle data ... sim_particles.X_prev = 0.5 >>> sim_particles.V = 1.0 >>> sim_particles.X = sim_particles.X_prev + sim_particles.V >>> # Save the particles with a solid color ... save_particles[...] = sim_particles >>> save_particles.Color = [0,255,128] >>> save_particles rec.array([([1.5, 1.5, 1.5], [1.0, 1.0, 1.0], [0L, 255L, 128L]), ([1.5, 1.5, 1.5], [1.0, 1.0, 1.0], [0L, 255L, 128L])], dtype=[('X', ' From qubax at gmx.at Mon Mar 7 12:53:11 2011 From: qubax at gmx.at (qubax at gmx.at) Date: Mon, 7 Mar 2011 18:53:11 +0100 Subject: [Numpy-discussion] How to sum weighted matrices In-Reply-To: References: Message-ID: <20110307175311.GA8231@tux.hotze.com> for your problem, you can do: ---------------------------- import numpy as np weights = np.array([1,2]) matrix1 = np.ones((2,3)) matrix2 = 2*np.ones((2,3)) matrices = np.array([matrix1,matrix2]) weighted_sum = np.tensordot(weights, matrices, (0,0)) -------------------------- On Mon, Mar 07, 2011 at 06:16:15AM -0600, shu wei wrote: > Hello all, > > I am new to python and numpy. > My question is how to sum up N weighted matrices. > For example w=[1,2] (N=2 case) > m1=[1 2 3, > 3 4 5] > > m2=[3 4 5, > 4 5 6] > I want to get a matrix Y=w[1]*m1+w[2]*m2 by using a loop. > > My original problem is like this > X=[1 2 3, > 3 4 5, > 4 5 6] > > a1=[1 2 3] 1st row of X > m1=a1'*a1 a matirx > a2=[3 4 5] 2nd row of X > m2=a2'*a2 > a3=[ 4 5 6] 3rd row of X > m3=a3'*a3 > > I want to get Y1=w[1]*m1+w[2]*m2 > Y2=w[1]*m2+w[2]*m3 > So basically it is rolling and to sum up the weighted matries > I have a big X, the rolling window is relatively small. > > I tried to use > > sq=np.array([x[i].reshape(-1,1)*x[i] for i in np.arange(0,len(x)]) # > s=len(x) > m=np.array([sq[i:i+t] for i in np.arange(0,s-t+1)]) # t is the len(w) > > then I was stuck, I tried to use a loop somethig like > Y=np.array([np.sum(w[i]*m[j,i],axis=0) for i in np.arange(0,t)] ) > Any suggestion is welcome. > > sue > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- There are two things children should get from their parents: roots and wings. The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity. From frank at qfin.net Mon Mar 7 13:19:09 2011 From: frank at qfin.net (Frank Conradie) Date: Mon, 07 Mar 2011 10:19:09 -0800 Subject: [Numpy-discussion] Numeric 24.2 incompatible with Windows7 SP1 Message-ID: <4D75219D.10309@qfin.net> Hi I know Numeric 24.2 is really old and probably unsupported by now, but I thought it might be of interest that Service Pack 1 for Windows 7 has "broken" Numeric 24.2. We have an old software product that uses Python 2.3 and Numeric 24.2, "frozen" using py2exe, and we started receiving reports from users recently that the software was failing upon launch. After investigation it became clear that the single commonality among the users was that they all recently installed SP1 for Windows 7, so we tried that ourselves, and sure enough, importing Numeric now causes this error: Python 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric Traceback (most recent call last): File "", line 1, in ? File "C:\python23\lib\site-packages\Numeric\Numeric.py", line 91, in ? import multiarray ImportError: DLL load failed: The parameter is incorrect. I will try and recompile Numeric 24.2 to see if this solves the problem, but I was wondering if anyone out there has any experience and/or comments on this? Thanks! Frank Conradie From Chris.Barker at noaa.gov Mon Mar 7 13:42:00 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 10:42:00 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <201103071833.00291.faltet@pytables.org> References: <4D7515AB.6000006@noaa.gov> <201103071833.00291.faltet@pytables.org> Message-ID: <4D7526F8.1040409@noaa.gov> On 3/7/11 9:33 AM, Francesc Alted wrote: > A Monday 07 March 2011 18:28:11 Christopher Barker escrigu?: >> I'm setting out to write some code to access and work with ragged >> arrays stored in netcdf files. It dawned on me that ragged arrays >> are not all that uncommon, so I'm wondering if any of you have any >> code you've developed that I could learn-from borrow from, etc. > A list of numpy arrays would not be enough? Or you want something more > specific? Maybe that would, but in mapping to the netcdf data model, I'm thinking more like a big 1-d numpy array, with a way to index into it. Also, that would allow you to do some math with the arrays, if the broad casting made sense, anyway. But now that you've entered the conversation, does HDF and/or pytables have a standard way of dealing with this? On 3/7/11 9:37 AM, Jeff Whitaker wrote: > Chris: The netcdf4-python modules reads netcdf vlen arrays arethose a netcdf4 feature? So far, I'm still workign with netcdf3 -- though this could be a compelling reason to move on! We've talked about this some on the CF list, and I don't think anyone brought that up. > and returns > numpy object arrays, where the elements of the object arrays are > themselves 1d numpy arrays. I don't think there is any other way to do > it. In your example, the 'ragged' array would be a 1d numpy array with > dtype='O', and the individual elements would be 1d numpy arrays with > dtype=int. Of course, these arrays are very awkward to deal with and > operations will be slow. Depends some of the operations, but yes. That still may be the best option, but I'm exploring others. is a "vlen array" stored contiguously in netcdf? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jswhit at fastmail.fm Mon Mar 7 14:05:06 2011 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Mon, 07 Mar 2011 12:05:06 -0700 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7526F8.1040409@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <201103071833.00291.faltet@pytables.org> <4D7526F8.1040409@noaa.gov> Message-ID: <4D752C62.90703@fastmail.fm> On 3/7/11 11:42 AM, Christopher Barker wrote: > On 3/7/11 9:33 AM, Francesc Alted wrote: >> A Monday 07 March 2011 18:28:11 Christopher Barker escrigu?: >>> I'm setting out to write some code to access and work with ragged >>> arrays stored in netcdf files. It dawned on me that ragged arrays >>> are not all that uncommon, so I'm wondering if any of you have any >>> code you've developed that I could learn-from borrow from, etc. >> A list of numpy arrays would not be enough? Or you want something more >> specific? > Maybe that would, but in mapping to the netcdf data model, I'm thinking > more like a big 1-d numpy array, with a way to index into it. Also, that > would allow you to do some math with the arrays, if the broad casting > made sense, anyway. > > But now that you've entered the conversation, does HDF and/or pytables > have a standard way of dealing with this? > > On 3/7/11 9:37 AM, Jeff Whitaker wrote: >> Chris: The netcdf4-python modules reads netcdf vlen arrays > arethose a netcdf4 feature? Chris: Yes, although I don't think many people are using it. See section 10 in http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html. > So far, I'm still workign with netcdf3 -- > though this could be a compelling reason to move on! > > We've talked about this some on the CF list, and I don't think anyone > brought that up. > >> and returns >> numpy object arrays, where the elements of the object arrays are >> themselves 1d numpy arrays. I don't think there is any other way to do >> it. In your example, the 'ragged' array would be a 1d numpy array with >> dtype='O', and the individual elements would be 1d numpy arrays with >> dtype=int. Of course, these arrays are very awkward to deal with and >> operations will be slow. > Depends some of the operations, but yes. > > That still may be the best option, but I'm exploring others. > > is a "vlen array" stored contiguously in netcdf? Probably, although I don't know for sure what the underlying HDF5 layer is doing. -Jeff > -Chris > > > > -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-113 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From faltet at pytables.org Mon Mar 7 14:18:00 2011 From: faltet at pytables.org (Francesc Alted) Date: Mon, 7 Mar 2011 20:18:00 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7526F8.1040409@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <201103071833.00291.faltet@pytables.org> <4D7526F8.1040409@noaa.gov> Message-ID: <201103072018.00142.faltet@pytables.org> A Monday 07 March 2011 19:42:00 Christopher Barker escrigu?: > But now that you've entered the conversation, does HDF and/or > pytables have a standard way of dealing with this? Well, I don't think there is such a 'standard' way for dealing with ragged arrays, but yes, pytables has support for them. Creating them is easy: # Create a VLArray: fileh = tables.openFile('vlarray1.h5', mode='w') vlarray = fileh.createVLArray(fileh.root, 'vlarray1', tables.Int32Atom(shape=()), "ragged array of ints", filters=tables.Filters(1)) # Append some (variable length) rows: vlarray.append(array([5, 6])) vlarray.append(array([5, 6, 7])) vlarray.append([5, 6, 9, 8]) Then, you can access the rows in a variety of ways, like iterators: print '-->', vlarray.title for x in vlarray: print '%s[%d]--> %s' % (vlarray.name, vlarray.nrow, x) --> ragged array of ints vlarray1[0]--> [5 6] vlarray1[1]--> [5 6 7] vlarray1[2]--> [5 6 9 8] or via __getitem__, using general fancy indexing: a_row = vlarray[2] a_list = vlarray[::2] a_list2 = vlarray[[0,2]] # get list of coords a_list3 = vlarray[[0,-2]] # negative values accepted a_list4 = vlarray[numpy.array([True,...,False])] # array of bools but, instead of returning a numpy array of 'object' elements, plain python lists are returned instead. More info on VLArray object in: http://www.pytables.org/docs/manual/ch04.html#VLArrayClassDescr > is a "vlen array" stored contiguously in netcdf? I don't really know, but one limitation of variable length arrays in HDF5 (and hence NetCDF4) is that they cannot be compressed (but that should be addressed in the future). -- Francesc Alted From Chris.Barker at noaa.gov Mon Mar 7 15:29:35 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 12:29:35 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <201103072018.00142.faltet@pytables.org> References: <4D7515AB.6000006@noaa.gov> <201103071833.00291.faltet@pytables.org> <4D7526F8.1040409@noaa.gov> <201103072018.00142.faltet@pytables.org> Message-ID: <4D75402F.4010900@noaa.gov> On 3/7/11 11:18 AM, Francesc Alted wrote: > Well, I don't think there is such a 'standard' way for dealing with > ragged arrays, but yes, pytables has support for them. Creating them is > easy: > > # Create a VLArray: > fileh = tables.openFile('vlarray1.h5', mode='w') > vlarray = fileh.createVLArray(fileh.root, 'vlarray1', > tables.Int32Atom(shape=()), > "ragged array of ints", > filters=tables.Filters(1)) > # Append some (variable length) rows: > vlarray.append(array([5, 6])) > vlarray.append(array([5, 6, 7])) > vlarray.append([5, 6, 9, 8]) > > Then, you can access the rows in a variety of ways, like iterators: > > print '-->', vlarray.title > for x in vlarray: > print '%s[%d]--> %s' % (vlarray.name, vlarray.nrow, x) > > --> ragged array of ints > vlarray1[0]--> [5 6] > vlarray1[1]--> [5 6 7] > vlarray1[2]--> [5 6 9 8] > > or via __getitem__, using general fancy indexing: > > a_row = vlarray[2] > a_list = vlarray[::2] > a_list2 = vlarray[[0,2]] # get list of coords > a_list3 = vlarray[[0,-2]] # negative values accepted > a_list4 = vlarray[numpy.array([True,...,False])] # array of bools > but, instead of returning a numpy array of 'object' elements, plain > python lists are returned instead. which gives you the append option -- I can see how that would be usefull. Though I'd kind of like to have numpy ufunc/broadcasting performance. i.e.: vlarray * some_constant Be fast, and work out of the box! > More info on VLArray object in: > > http://www.pytables.org/docs/manual/ch04.html#VLArrayClassDescr > >> is a "vlen array" stored contiguously in netcdf? great, thanks! that gives me an example of one API I might want to use. > I don't really know, but one limitation of variable length arrays in > HDF5 (and hence NetCDF4) is that they cannot be compressed (but that > should be addressed in the future). good to know. Thanks to both you and Jeff, This has given me some things to ponder. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aarchiba at physics.mcgill.ca Mon Mar 7 17:25:19 2011 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Mon, 7 Mar 2011 17:25:19 -0500 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D75402F.4010900@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <201103071833.00291.faltet@pytables.org> <4D7526F8.1040409@noaa.gov> <201103072018.00142.faltet@pytables.org> <4D75402F.4010900@noaa.gov> Message-ID: On 7 March 2011 15:29, Christopher Barker wrote: > On 3/7/11 11:18 AM, Francesc Alted wrote: > >> but, instead of returning a numpy array of 'object' elements, plain >> python lists are returned instead. > > which gives you the append option -- I can see how that would be > usefull. Though I'd kind of like to have numpy ufunc/broadcasting > performance. i.e.: > > vlarray * some_constant > > Be fast, and work out of the box! How about this? Represent them as object arrays (or lists) of variable-length 1d arrays. But create the 1d arrays by building one large 1d array with them all concatenated, then taking slices. Keep the single array around, perhaps in an attribute of the object array; after all, most of the numpy operations are elementwise and don't care about array structure. You'd want to be able to convert between a flat array plus length vector and object array in some reasonably efficient manner, and I think this allows that. Anne From halbert at halwitz.org Mon Mar 7 18:36:02 2011 From: halbert at halwitz.org (Dan Halbert) Date: Mon, 7 Mar 2011 18:36:02 -0500 (EST) Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar Message-ID: <1299540962.65641901@beta.apps.rackspace.com> We currently have some straightforward NumPy code that indirectly implements a C API defined by a third party. We built a Cython layer that directly provides the API in a .a library, and then calls Python. The layering looks like this: C main program -> API in Cython -> Python -> NumPy This is difficult to package for distribution, because of the Python and NumPy dependencies. We may need to reimplement our library so it factors out the Python dependency, and I would like to explore the alternatives. (Performance may also be a reason to do this, but that is not the main issue right now.) Do you all have some recommendations about tools, libraries, or languages that you have used to rewrite NumPy code easily into something that's more self-contained and callable from C? For instance, are there some nice C++ linear algebra libraries that map closely to NumPy? Or is there some higher-level compiled array language that looks something like NumPy code? I apologize if the answers are obvious: I am not very familiar with the tools in this space. Thanks, Dan (I saw the NumPy Refactoring project discussion from earlier. When that is finished, the resulting Python-independent library might be a nice way to handle this, but I am thinking shorter-term.) From Chris.Barker at noaa.gov Mon Mar 7 18:48:30 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 15:48:30 -0800 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <1299540962.65641901@beta.apps.rackspace.com> References: <1299540962.65641901@beta.apps.rackspace.com> Message-ID: <4D756ECE.3040101@noaa.gov> On 3/7/11 3:36 PM, Dan Halbert wrote: > We currently have some straightforward NumPy code that indirectly implements a C API defined by a third party. We built a Cython layer that directly provides the API in a .a library, and then calls Python. The layering looks like this: > > C main program -> API in Cython -> Python -> NumPy > > This is difficult to package for distribution, because of the Python and NumPy dependencies. I'd say learn py2exe, py2app and friends, and be done with it. Otherwise, I wonder if you could re-factor your Cython code enough to remove all python dependencies -- or at least have something you could compile and run without a python interpreter running. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From halbert at halwitz.org Mon Mar 7 19:22:25 2011 From: halbert at halwitz.org (Dan Halbert) Date: Mon, 07 Mar 2011 19:22:25 -0500 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D756ECE.3040101@noaa.gov> References: <1299540962.65641901@beta.apps.rackspace.com> <4D756ECE.3040101@noaa.gov> Message-ID: <4D7576C1.3020208@halwitz.org> On 3/7/2011 6:48 PM, Christopher Barker wrote: > On 3/7/11 3:36 PM, Dan Halbert wrote: >> We currently have some straightforward NumPy code that indirectly implements a C API defined by a third party. We built a Cython layer that directly provides the API in a .a library, and then calls Python. The layering looks like this: >> >> C main program -> API in Cython -> Python -> NumPy >> >> This is difficult to package for distribution, because of the Python and NumPy dependencies. > > I'd say learn py2exe, py2app and friends, and be done with it. > > Otherwise, I wonder if you could re-factor your Cython code enough to > remove all python dependencies -- or at least have something you could > compile and run without a python interpreter running. What we ship is the library and associated files. The library gets linked with someone else's main program. So py2exe and pyinstaller, etc. can't help much here, since we don't control the executable. I have thought about doing more in Cython, but I still need a libpython2.x.so, etc., I believe. Dan From sturla at molden.no Mon Mar 7 20:51:55 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 02:51:55 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7515AB.6000006@noaa.gov> References: <4D7515AB.6000006@noaa.gov> Message-ID: <4D758BBB.3070406@molden.no> Den 07.03.2011 18:28, skrev Christopher Barker: > 1, 2, 3, 4 > 5, 6 > 7, 8, 9, 10, 11, 12 > 13, 14, 15 > ... > > In my case, these will only be 2-d, though I suppose one could have a > n-d version where the last dimension was ragged (or any dimension, I > suppose, though I'm having trouble wrapping my brain around what that > would look like.. A ragged array, as implemented in C++, Java or C# is just an array of arrays (or 'a pointer to an array of pointers'). Basically, that is an ndarray of ndarrays (or a list of ndarrays, whatever you prefer). >>> ra = np.zeros(4, dtype=np.ndarray) >>> ra[0] = np.array([1,2,3,4]) >>> ra[1] = np.array([5,6]) >>> ra[2] = np.array([7,8,9,10,11,12]) >>> ra[3] = np.array([13,14,15]) >>> ra array([[1 2 3 4], [5 6], [ 7 8 9 10 11 12], [13 14 15]], dtype=object) >>> ra[1][1] 6 >>> ra[2][:] array([ 7, 8, 9, 10, 11, 12]) Slicing in two dimensions does not work as some might expect: >>> ra[:2][:2] array([[1 2 3 4], [5 6]], dtype=object) This "strange" result would actually not be different from the ragged array semantics of C++, C# or Java if those languages had supported slicing. Indexing [::][::] instead of [::,::] corresponds to what you must do in C#, C++ and Java as well. (We are doing exactly the same, so why should it not be?) Sturla From mailshuwei at gmail.com Mon Mar 7 21:03:53 2011 From: mailshuwei at gmail.com (shu wei) Date: Mon, 7 Mar 2011 20:03:53 -0600 Subject: [Numpy-discussion] How to sum weighted matrices In-Reply-To: <20110307175311.GA8231@tux.hotze.com> References: <20110307175311.GA8231@tux.hotze.com> Message-ID: Thanks very much. It works. On Mon, Mar 7, 2011 at 11:53 AM, wrote: > for your problem, you can do: > > ---------------------------- > > import numpy as np > > weights = np.array([1,2]) > > matrix1 = np.ones((2,3)) > matrix2 = 2*np.ones((2,3)) > > matrices = np.array([matrix1,matrix2]) > > weighted_sum = np.tensordot(weights, matrices, (0,0)) > > -------------------------- > > On Mon, Mar 07, 2011 at 06:16:15AM -0600, shu wei wrote: > > Hello all, > > > > I am new to python and numpy. > > My question is how to sum up N weighted matrices. > > For example w=[1,2] (N=2 case) > > m1=[1 2 3, > > 3 4 5] > > > > m2=[3 4 5, > > 4 5 6] > > I want to get a matrix Y=w[1]*m1+w[2]*m2 by using a loop. > > > > My original problem is like this > > X=[1 2 3, > > 3 4 5, > > 4 5 6] > > > > a1=[1 2 3] 1st row of X > > m1=a1'*a1 a matirx > > a2=[3 4 5] 2nd row of X > > m2=a2'*a2 > > a3=[ 4 5 6] 3rd row of X > > m3=a3'*a3 > > > > I want to get Y1=w[1]*m1+w[2]*m2 > > Y2=w[1]*m2+w[2]*m3 > > So basically it is rolling and to sum up the weighted matries > > I have a big X, the rolling window is relatively small. > > > > I tried to use > > > > sq=np.array([x[i].reshape(-1,1)*x[i] for i in np.arange(0,len(x)]) # > > s=len(x) > > m=np.array([sq[i:i+t] for i in np.arange(0,s-t+1)]) # t is the len(w) > > > > then I was stuck, I tried to use a loop somethig like > > Y=np.array([np.sum(w[i]*m[j,i],axis=0) for i in np.arange(0,t)] ) > > Any suggestion is welcome. > > > > sue > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > -- > There are two things children should get > from their parents: roots and wings. > > The king who needs to remind his people of his rank, is no king. > > A beggar's mistake harms no one but the beggar. A king's mistake, > however, harms everyone but the king. Too often, the measure of > power lies not in the number who obey your will, but in the number > who suffer your stupidity. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Mar 7 21:13:40 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 03:13:40 +0100 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <1299540962.65641901@beta.apps.rackspace.com> References: <1299540962.65641901@beta.apps.rackspace.com> Message-ID: <4D7590D4.1060402@molden.no> Den 08.03.2011 00:36, skrev Dan Halbert: > Do you all have some recommendations about tools, libraries, or languages that you have used to rewrite NumPy code easily into something that's more self-contained and callable from C? Fortran 95 It has the power of C, without the unsafe pointer arithmetics, almost the readability of Python, and all the array magic of NumPy. Almost every serious numerical library have bindings for Fortran. Calling Fortran from C: - Name mangling: usually lower-case with appended underscore. - Assumed-size or explicit shape arrays: just pass a pointer (cf. f2py). Avoid assumed-shape or deferred-size arrays in the interface to C. - Fortran 2003 ISO C bindings (cf. fwrap). > For instance, are there some nice C++ linear algebra libraries that map closely to NumPy? Or is there some higher-level compiled array language that looks something like NumPy code? The 'standard' linear algebra packages are BLAS and LAPACK, e.g. as implemented in ACML or Intel MKL. (Callable from C although implemented in Fortran.) Two comprehensive scientific C and Fortran libraries are IMSL and NAG. Sturla From njs at pobox.com Mon Mar 7 21:25:15 2011 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 7 Mar 2011 18:25:15 -0800 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <1299540962.65641901@beta.apps.rackspace.com> References: <1299540962.65641901@beta.apps.rackspace.com> Message-ID: On Mon, Mar 7, 2011 at 3:36 PM, Dan Halbert wrote: > Or is there some higher-level compiled array language that looks something like NumPy code? You might want to try Eigen: http://eigen.tuxfamily.org/ -- Nathaniel From halbert at halwitz.org Mon Mar 7 23:05:18 2011 From: halbert at halwitz.org (Dan Halbert) Date: Mon, 07 Mar 2011 23:05:18 -0500 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D7590D4.1060402@molden.no> References: <1299540962.65641901@beta.apps.rackspace.com> <4D7590D4.1060402@molden.no> Message-ID: <4D75AAFE.9030703@halwitz.org> On 3/7/2011 9:13 PM, Sturla Molden wrote: > Den 08.03.2011 00:36, skrev Dan Halbert: >> Do you all have some recommendations about tools, libraries, or languages that you have used to rewrite NumPy code easily into something that's more self-contained and callable from C? > > Fortran 95 > > It has the power of C, without the unsafe pointer arithmetics, almost > the readability of Python, and all the array magic of NumPy. > > Almost every serious numerical library have bindings for Fortran. > > Calling Fortran from C: > - Name mangling: usually lower-case with appended underscore. > - Assumed-size or explicit shape arrays: just pass a pointer (cf. f2py). > Avoid assumed-shape or deferred-size arrays in the interface to C. > - Fortran 2003 ISO C bindings (cf. fwrap). Thanks, that's a good suggestion. I have not written Fortran since 1971, but it's come a long way. I was a little worried about the row-major vs column-major issue, but perhaps that can be handled just by remembering to reverse the subscript order between C and Fortran. Dan From halbert at halwitz.org Mon Mar 7 23:07:32 2011 From: halbert at halwitz.org (Dan Halbert) Date: Mon, 07 Mar 2011 23:07:32 -0500 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: References: <1299540962.65641901@beta.apps.rackspace.com> Message-ID: <4D75AB84.6020206@halwitz.org> On 3/7/2011 9:25 PM, Nathaniel Smith wrote: > On Mon, Mar 7, 2011 at 3:36 PM, Dan Halbert wrote: >> Or is there some higher-level compiled array language that looks something like NumPy code? > > You might want to try Eigen: > http://eigen.tuxfamily.org/ Thanks - didn't know about this. I'm also looking at ROOT, from CERN. --Dan From robertwb at math.washington.edu Tue Mar 8 00:54:08 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Mar 2011 21:54:08 -0800 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D7576C1.3020208@halwitz.org> References: <1299540962.65641901@beta.apps.rackspace.com> <4D756ECE.3040101@noaa.gov> <4D7576C1.3020208@halwitz.org> Message-ID: On Mon, Mar 7, 2011 at 4:22 PM, Dan Halbert wrote: > On 3/7/2011 6:48 PM, Christopher Barker wrote: >> On 3/7/11 3:36 PM, Dan Halbert wrote: >>> We currently have some straightforward NumPy code that indirectly implements a C API defined by a third party. We built a Cython layer that directly provides the API in a .a library, and then calls Python. The layering looks like this: >>> >>> ? ? C main program -> ? API in Cython -> ? Python -> ? NumPy >>> >>> This is difficult to package for distribution, because of the Python and NumPy dependencies. >> >> I'd say learn py2exe, py2app and friends, and be done with it. >> >> Otherwise, I wonder if you could re-factor your Cython code enough to >> remove all python dependencies -- or at least have something you could >> compile and run without a python interpreter running. > > What we ship is the library and associated files. The library gets > linked with someone else's main program. So py2exe and pyinstaller, etc. > can't help much here, since we don't control the executable. I have > thought about doing more in Cython, but I still need a libpython2.x.so, > etc., I believe. Can't you link libpython and all of the other dependancies into a single .so file? This is the "embedding Python" idea--you can embed it in another library, you don't have to be the top-level executable. - Robert From robertwb at math.washington.edu Tue Mar 8 03:12:35 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Mar 2011 00:12:35 -0800 Subject: [Numpy-discussion] [cython-users] errors with numpy 1.5.1 In-Reply-To: References: Message-ID: On Fri, Mar 4, 2011 at 4:19 PM, Choy wrote: > Hello -- > > I recently upgraded from python 2.6 -> 2.7, cython 0.13 -> 14.1, and > numpy 1.4.1 -> 1.5.1. ?Unfortunately, one of my cython modules no > longer works. > > I tracked my bug and used the web to see that there is already a > ticket for exact same bug with the new buffer interface in numpy > 1.5.1. > > http://trac.cython.org/cython_trac/ticket/630 > > This is precisely the bug I am observing. > > Let me know if there is an easy patch, or if a release is planned that > addresses this in the near future. Unfortunately, I don't think anyone's actively working on it right now. It's probably a superficial fix for someone who knows NumPy and Cython decently well, but I have no idea myself (not having looked that deeply into it.) Does anyone know what the backwards incompatible NumPy 1.4 -> 1.5 changes were? - Robert From pav at iki.fi Tue Mar 8 04:11:55 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 8 Mar 2011 09:11:55 +0000 (UTC) Subject: [Numpy-discussion] [cython-users] errors with numpy 1.5.1 References: Message-ID: Tue, 08 Mar 2011 00:12:35 -0800, Robert Bradshaw wrote: [clip] > Unfortunately, I don't think anyone's actively working on it right now. > It's probably a superficial fix for someone who knows NumPy and Cython > decently well, but I have no idea myself (not having looked that deeply > into it.) Does anyone know what the backwards incompatible NumPy 1.4 -> > 1.5 changes were? No backward incompatible changes, but 1.5 has the new buffer interface implemented. The old buffer interface etc. are still there, but maybe Cython only looks at the new buffer interface, and if it does not understand what it sees (AFAIK, it only implements a subset of PEP 3118) it fails instead of trying the other options? The errors in the post you linked to fall into two categories: 1) The exact wording of an exception is different "ndarray is not C contiguous" vs. "ndarray is not C-contiguous" or "Non-native byte order not supported" vs. "Big-endian buffer not supported on little-endian compiler", and others which the doctests catch. 2) Cython apparently does/did not recognize the ':' character in structured array dtype strings. Pauli From binet at cern.ch Tue Mar 8 05:20:37 2011 From: binet at cern.ch (Sebastien Binet) Date: Tue, 8 Mar 2011 11:20:37 +0100 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D75AB84.6020206@halwitz.org> References: <1299540962.65641901@beta.apps.rackspace.com> <4D75AB84.6020206@halwitz.org> Message-ID: <87pqq1vrpm.fsf@cern.ch> On Mon, 7 Mar 2011 23:07:32 -0500, Dan Halbert wrote: > On 3/7/2011 9:25 PM, Nathaniel Smith wrote: > > On Mon, Mar 7, 2011 at 3:36 PM, Dan Halbert wrote: > >> Or is there some higher-level compiled array language that looks something like NumPy code? > > > > You might want to try Eigen: > > http://eigen.tuxfamily.org/ > > Thanks - didn't know about this. I'm also looking at ROOT, from CERN. that's a whole other can of worms, if you ask me... depending on what you really need, you'd probably be better off with just eigen or the C rewrite of numpy... -s -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From Sam.Tygier at hep.manchester.ac.uk Tue Mar 8 08:59:32 2011 From: Sam.Tygier at hep.manchester.ac.uk (Sam Tygier) Date: Tue, 08 Mar 2011 13:59:32 +0000 Subject: [Numpy-discussion] Boolean indexing Message-ID: <1299592772.16213.19.camel@hydrogen> Hi I am having an issue with boolean slicing. it seems to work fine for reading a value, but I can use it to set a value: import numpy b = numpy.array([[1,2],[3,4],[5,6],[7,8],[9,10]]) m = numpy.array([0,1,0,0,0], dtype=bool) print b[m] print b[m][0,0] b[m][0,0] = -1 print b[m][0,0] I think the boolean slicing is making a copy instead of a view. is there a way around this? My full problem is to modify a column of a rec array, if another column has a condition. ie, a neat way of doing for n in xrange(len(a)): if a[n]['name'] == "foo": a[n]['x'] += 1 i have been trying things like: a[a[n]['name'] == "foo"]['x'] += 1 but does not work. I am using Python version 2.7 (r27:82500, Sep 16 2010, 18:02:00) numpy 1:1.4.1-6.fc14 (fedora package) Thanks Sam From cournape at gmail.com Tue Mar 8 09:07:31 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 8 Mar 2011 23:07:31 +0900 Subject: [Numpy-discussion] [ANN] Bento 0.0.5, a packaging solution for python software Message-ID: Hi, I am pleased to announce a new release of bento, a packaging solution for python which aims at reproducibility, extensibility and simplicity. You can take a look at its main features on bento's main documentation page (http://cournape.github.com/Bento). The main features of this 0.0.5 release are: - All python versions from 2.4 up to 3.1 now pass the test suite (3.2 will follow once a distribute issue with 3.2 is fixed) - If run under a virtual environment (virtualenv), bento will install the package inside the virtualenvironment by default - Internal changes to enable easier change of build tool (an example for simple extensions is available for waf 1.6.x) - Added experimental distutils compatibility layer so that one can write a setup.py which will pick up all information from bento.info. This enables projects using bento to be able to use tools such as pip. Bento now has its own mailing list: bento at librelist.com. If you are curious about bento, I will present it at pycon 2011 as a poster. cheers, David From sturla at molden.no Tue Mar 8 09:25:12 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 15:25:12 +0100 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D75AAFE.9030703@halwitz.org> References: <1299540962.65641901@beta.apps.rackspace.com> <4D7590D4.1060402@molden.no> <4D75AAFE.9030703@halwitz.org> Message-ID: <4D763C48.9020201@molden.no> Den 08.03.2011 05:05, skrev Dan Halbert: > Thanks, that's a good suggestion. I have not written Fortran since 1971, > but it's come a long way. I was a little worried about the row-major vs > column-major issue, but perhaps that can be handled just by remembering > to reverse the subscript order between C and Fortran. In practice this is not a problem. Most numerical libraries for C assume Fortran-ordering, even OpenGL assumes Fortran-ordering. People program MEX files for Matlab in C all the time. Fortran-ordering is assumed in MEX files too. In ANSI C, array bounds must be known at compile time, so a Fortran routine with the interface subroutine foobar( lda, A ) integer lda double precision A(lda,*) end subroutine will usually be written like void foobar( int lda, double A[]); in C, ignoring different calling convention for lda. Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is that too difficult to remember? In ANSI C the issue actually only arises with small "array of arrays" having static shape, or convoluted contructs like "pointer to an array of pointers to arrays". Just avoid those and stay with 1D arrays in C -- do the 1D to 2D mapping in you mind. In C99 arrays are allowed to have dynamic size, which mean we can do void foobar( int lda, double *pA ) { typedef double row_t [lda]; vector_t *A = (vector_t*)((void*)&pA[0]); Here we must index A[k][i] to match A(i,k) in Fortran. I still have not seen anyone use C99 like this, so I think it is merely theoretical. Chances are if you know how to do this with C99, you also know how to get the subscripts right. If you are afraid to forget "to reverse the subscript order between C and Fortran", it just tells me you don't really know what you are doing when using C, and should probably use something else. Why not Cython? It has "native support" for NumPy arrays. Personally I prefer Fortran 95, but that is merely a matter of taste. Sturla From sturla at molden.no Tue Mar 8 09:29:07 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 15:29:07 +0100 Subject: [Numpy-discussion] Boolean indexing In-Reply-To: <1299592772.16213.19.camel@hydrogen> References: <1299592772.16213.19.camel@hydrogen> Message-ID: <4D763D33.6050706@molden.no> Den 08.03.2011 14:59, skrev Sam Tygier: > I think the boolean slicing is making a copy instead of a view. Yes. > is there > a way around this? A "boolean slice" cannot be indexed with the dot product of dimensions and strides, hence the copy. You probably want to use masked arrays instead. Sturla From Sam.Tygier at hep.manchester.ac.uk Tue Mar 8 09:53:29 2011 From: Sam.Tygier at hep.manchester.ac.uk (Sam Tygier) Date: Tue, 08 Mar 2011 14:53:29 +0000 Subject: [Numpy-discussion] Boolean indexing In-Reply-To: <1299592772.16213.19.camel@hydrogen> References: <1299592772.16213.19.camel@hydrogen> Message-ID: <1299596009.16213.26.camel@hydrogen> 2011-03-08 14:29:07 GMT, Sturla Molden: >A "boolean slice" cannot be indexed with the dot product of dimensions and strides, hence the copy. >You probably want to use masked arrays instead. Masked array does not seem to help. when i do: am = numpy.ma.array(a, mask=a[n]['name']=="foo") am['x'] += 1 then it modifies all the elements, not just the unmasked ones. sam From robert.kern at gmail.com Tue Mar 8 12:01:46 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 8 Mar 2011 11:01:46 -0600 Subject: [Numpy-discussion] Boolean indexing In-Reply-To: <1299592772.16213.19.camel@hydrogen> References: <1299592772.16213.19.camel@hydrogen> Message-ID: On Tue, Mar 8, 2011 at 07:59, Sam Tygier wrote: > Hi > > I am having an ?issue with boolean slicing. it seems to work fine for > reading a value, but I can use it to set a value: > > import numpy > b = ?numpy.array([[1,2],[3,4],[5,6],[7,8],[9,10]]) > m = numpy.array([0,1,0,0,0], dtype=bool) > print b[m] > print b[m][0,0] > b[m][0,0] = -1 > print b[m][0,0] > > I think the boolean slicing is making a copy instead of a view. is there > a way around this? bm = b[m] bm[0,0] = -1 b[m] = bm -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From scheffer.nicolas at gmail.com Tue Mar 8 12:29:00 2011 From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER) Date: Tue, 8 Mar 2011 09:29:00 -0800 Subject: [Numpy-discussion] How to sum weighted matrices In-Reply-To: References: <20110307175311.GA8231@tux.hotze.com> Message-ID: Or just with a dot: === In [17]: np.tensordot(weights, matrices, (0,0)) Out[17]: array([[ 5., 5., 5.], [ 5., 5., 5.]]) In [18]: np.dot(matrices.T,weights).T Out[18]: array([[ 5., 5., 5.], [ 5., 5., 5.]]) == make matrices.T C_CONTIGUOUS for maximum speed. -n On Mon, Mar 7, 2011 at 6:03 PM, shu wei wrote: > Thanks very much. It works. > > On Mon, Mar 7, 2011 at 11:53 AM, wrote: >> >> for your problem, you can do: >> >> ---------------------------- >> >> import numpy as np >> >> weights = np.array([1,2]) >> >> matrix1 = np.ones((2,3)) >> matrix2 = 2*np.ones((2,3)) >> >> matrices = np.array([matrix1,matrix2]) >> >> weighted_sum = np.tensordot(weights, matrices, (0,0)) >> >> -------------------------- >> >> On Mon, Mar 07, 2011 at 06:16:15AM -0600, shu wei wrote: >> > ? ?Hello all, >> > >> > ? ?I am new to python and numpy. >> > ? ?My question is how to sum up N weighted matrices. >> > ? ?For example w=[1,2] (N=2 case) >> > ? ?m1=[1 2 3, >> > ? ? ? ? ? 3 4 5] >> > >> > ? ?m2=[3 4 5, >> > ? ? ? ? ? 4 5 6] >> > ? ?I want to get a matrix Y=w[1]*m1+w[2]*m2 by using a loop. >> > >> > ? ?My original problem is like this >> > ? ?X=[1 2 3, >> > ? ? ? ? 3 4 5, >> > ? ? ? ? 4 5 6] >> > >> > ? ?a1=[1 2 3] ?1st row of X >> > ? ?m1=a1'*a1 a matirx >> > ? ?a2=[3 4 5] 2nd row of X >> > ? ?m2=a2'*a2 >> > ? ?a3=[ 4 5 6] 3rd row of X >> > ? ?m3=a3'*a3 >> > >> > ? ?I want to get Y1=w[1]*m1+w[2]*m2 >> > ? ? ? ? ? ? ? ? ? ? ? ? ?Y2=w[1]*m2+w[2]*m3 >> > ? ?So basically it is rolling and to sum up the weighted matries >> > ? ?I have a big X, the rolling window is relatively small. >> > >> > ? ?I tried to use >> > >> > ? ?sq=np.array([x[i].reshape(-1,1)*x[i] for i in np.arange(0,len(x)]) # >> > ? ?s=len(x) >> > ? ?m=np.array([sq[i:i+t] for i in np.arange(0,s-t+1)]) # t is the len(w) >> > >> > ? ?then I was stuck, I tried to use a loop somethig like >> > ? ?Y=np.array([np.sum(w[i]*m[j,i],axis=0) for i in np.arange(0,t)] ) >> > ? ?Any suggestion is welcome. >> > >> > ? ?sue >> >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> -- >> There are two things children should get >> from their parents: roots and wings. >> >> The king who needs to remind his people of his rank, is no king. >> >> A beggar's mistake harms no one but the beggar. A king's mistake, >> however, harms everyone but the king. Too often, the measure of >> power lies not in the number who obey your will, but in the number >> who suffer your stupidity. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From jmhykes at ncsu.edu Tue Mar 8 14:56:46 2011 From: jmhykes at ncsu.edu (Josh Hykes) Date: Tue, 8 Mar 2011 14:56:46 -0500 Subject: [Numpy-discussion] Is there a memory-efficient alternative to choose? Message-ID: Hello, I am writing a small PDE code in Python using a Cartesian mesh. Each mesh cell is one of a few materials, with associated properties. I store these properties in a dictionary and have a "mesh map" that tells me which material is in each cell. At some point in my code, I need to do a cell-wise multiplication of the properties with a state variable. The ideal method would (1) be fast (no Python loops) and (2) not waste memory constructing an entire property map. My best attempt using choose does (1) but not (2). To make things concrete, a simple 2D example is: >>> import numpy as np >>> properties = {0: 0.5, 1: 2.} # 2 materials >>> mesh_map = np.array([[0,0,0,0], [0,1,1,0], [0,1,1,0], [0,0,0,0]]) # 4x4 mesh >>> properties_map = np.choose(mesh_map, (properties[0], properties[1])) >>> state_variables = np.arange(mesh_map.size).reshape(mesh_map.shape) # fake state variables >>> answer = properties_map * state_variables Can I get answer without storing properties_map? This seems like a common problem, but I haven't found a pure Numpy solution. I can't seem to find the correct keywords in Google to find what I'm looking for. Two further notes: First, I'll be computing answer many times, but properties_map shouldn't change. Second, in the real problem, the properties are themselves vectors or matrices, thus my desire to avoid storing them repeatedly. Thanks for any insights or pointers. -Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at utoronto.ca Tue Mar 8 15:03:28 2011 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Tue, 8 Mar 2011 15:03:28 -0500 Subject: [Numpy-discussion] Is this the optimal way to take index along a single axis? Message-ID: I am wanting to use an array b to index into an array x with dimension bigger by 1 where the element of b indicates what value to extract along a certain direction. For example, b = x.argmin(axis=1). Perhaps I want to use b to create x.min(axis=1) but also to index perhaps another array of the same size. I had a difficult time finding a way to do this with np.take easily and even with fancy indexing the resulting line is very complicated: In [322]: x.shape Out[322]: (2, 3, 4) In [323]: x.min(axis=1) Out[323]: array([[ 2, 1, 7, 4], [ 8, 0, 15, 12]]) In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis], idx[:,np.newaxis,:], np.arange(x.shape[2])] Out[324]: array([[[ 2, 1, 7, 4]], [[ 8, 0, 15, 12]]]) In any case I wrote myself my own function for doing this (below) and am wondering if this is the best way to do this or if there is something else in numpy that I should be using? -- I figure that this is a relatively common usecase. Thanks, Jon. def mytake(A, b, axis): assert len(A.shape) == len(b.shape)+1 idx = [] for i in range(len(A.shape)): if i == axis: temp = b.copy() shapey = list(temp.shape) shapey.insert(i,1) else: temp = np.arange(A.shape[i]) shapey = [1]*len(b.shape) shapey.insert(i,A.shape[i]) shapey = tuple(shapey) temp = temp.reshape(shapey) idx += [temp] return A[tuple(idx)].squeeze() In [319]: util.mytake(x,x.argmin(axis=1), 1) Out[319]: array([[ 2, 1, 7, 4], [ 8, 0, 15, 12]]) In [320]: x.min(axis=1) Out[320]: array([[ 2, 1, 7, 4], [ 8, 0, 15, 12]]) From josef.pktd at gmail.com Tue Mar 8 15:39:18 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 8 Mar 2011 15:39:18 -0500 Subject: [Numpy-discussion] Is this the optimal way to take index along a single axis? In-Reply-To: References: Message-ID: On Tue, Mar 8, 2011 at 3:03 PM, Jonathan Taylor wrote: > I am wanting to use an array b to index into an array x with dimension > bigger by 1 where the element of b indicates what value to extract > along a certain direction. ?For example, b = x.argmin(axis=1). > Perhaps I want to use b to create x.min(axis=1) but also to index > perhaps another array of the same size. > > I had a difficult time finding a way to do this with np.take easily > and even with fancy indexing the resulting line is very complicated: > > In [322]: x.shape > Out[322]: (2, 3, 4) > > In [323]: x.min(axis=1) > Out[323]: > array([[ 2, ?1, ?7, ?4], > ? ? ? [ 8, ?0, 15, 12]]) > > In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis], > idx[:,np.newaxis,:], np.arange(x.shape[2])] > Out[324]: > array([[[ 2, ?1, ?7, ?4]], > > ? ? ? [[ 8, ?0, 15, 12]]]) > > In any case I wrote myself my own function for doing this (below) and > am wondering if this is the best way to do this or if there is > something else in numpy that I should be using? -- I figure that this > is a relatively common usecase. > > Thanks, > Jon. > > def mytake(A, b, axis): > ? ?assert len(A.shape) == len(b.shape)+1 > > ? ?idx = [] > ? ?for i in range(len(A.shape)): > ? ? ? ?if i == axis: > ? ? ? ? ? ?temp = b.copy() > ? ? ? ? ? ?shapey = list(temp.shape) > ? ? ? ? ? ?shapey.insert(i,1) > ? ? ? ?else: > ? ? ? ? ? ?temp = np.arange(A.shape[i]) > ? ? ? ? ? ?shapey = [1]*len(b.shape) > ? ? ? ? ? ?shapey.insert(i,A.shape[i]) > ? ? ? ?shapey = tuple(shapey) > ? ? ? ?temp = temp.reshape(shapey) > ? ? ? ?idx += [temp] > > ? ?return A[tuple(idx)].squeeze() > > > In [319]: util.mytake(x,x.argmin(axis=1), 1) > Out[319]: > array([[ 2, ?1, ?7, ?4], > ? ? ? [ 8, ?0, 15, 12]]) > > In [320]: x.min(axis=1) > Out[320]: > array([[ 2, ?1, ?7, ?4], > ? ? ? [ 8, ?0, 15, 12]]) fewer lines but essentially the same thing and no shortcuts, I think >>> x= np.random.randint(5, size=(2, 3, 4)) >>> x array([[[3, 1, 0, 1], [4, 2, 2, 1], [2, 3, 2, 2]], [[2, 1, 1, 1], [0, 2, 0, 3], [2, 3, 3, 1]]]) >>> idx = [np.arange(i) for i in x.shape] >>> idx = list(np.ix_(*idx)) >>> idx[axis]=np.expand_dims(x.argmin(axis),axis) >>> x[idx] array([[[2, 1, 0, 1]], [[0, 1, 0, 1]]]) >>> np.squeeze(x[idx]) array([[2, 1, 0, 1], [0, 1, 0, 1]]) >>> mytake(x,x.argmin(axis=1), 1) array([[2, 1, 0, 1], [0, 1, 0, 1]]) Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lutz.maibaum at gmail.com Tue Mar 8 16:53:52 2011 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Tue, 8 Mar 2011 13:53:52 -0800 Subject: [Numpy-discussion] Is there a memory-efficient alternative to choose? In-Reply-To: References: Message-ID: On Mar 8, 2011, at 11:56 AM, Josh Hykes wrote: > At some point in my code, I need to do a cell-wise multiplication of the properties with a state variable. The ideal method would (1) be fast (no Python loops) and (2) not waste memory constructing an entire property map. My best attempt using choose does (1) but not (2). Would the "out" keyword of np.choose be a possible solution? Using that, you should be able to create the properies_map in place, and perhaps you can then do the multiplication with the state variables in place also: import numpy as np properties = {0: 0.5, 1: 2.} # 2 materials mesh_map = np.array([[0,0,0,0], [0,1,1,0], [0,1,1,0], [0,0,0,0]]) # 4x4 mesh state_variables = np.arange(mesh_map.size).reshape(mesh_map.shape) # fake state variables answer = np.empty (mesh_map.shape, np.float) # do this only once with dtype of properties np.choose(mesh_map, (properties[0], properties[1]), out=answer) answer[:] = answer * state_variables Hope this helps, Lutz From hao at biostat.ucsf.edu Tue Mar 8 20:01:50 2011 From: hao at biostat.ucsf.edu (Hao Xiong) Date: Tue, 08 Mar 2011 17:01:50 -0800 Subject: [Numpy-discussion] Inaccuracy in documentation of np.linalg.pinv Message-ID: <4D76D17E.3070605@biostat.ucsf.edu> I think the documentation for np.linalg.pinv contains some inaccuracies. Most importantly, Moore-Penrose is not defined by the solution to the least-square problem. It was defined by the unique solution to 4 equations. Since SVD can be easily shown to satisfy the same 4 equations, it is the Moore-Penrose inverse. Another way that definition is wrong is that the least-square problem does not have to have unique solution. In fact, it can have infinitely many. In this case, Moore-Penrose is the minimum norm solution. All of this is discussed in the Matlab documentation of pinv, which is very good. Hao From jsseabold at gmail.com Tue Mar 8 20:08:57 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 8 Mar 2011 20:08:57 -0500 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Sun, Mar 6, 2011 at 11:12 PM, Ralf Gommers wrote: > On Sun, Mar 6, 2011 at 1:10 AM, Skipper Seabold wrote: >> On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers >> wrote: >>> On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >>>> The page >>>> >>>> gives a good introduction to structured arrays. However, it says nothing >>>> about how to set a particular element (all fields at once) from a >>>> collection of data. >>>> >>>> For instance: >>>> >>>> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >>>> >>>> The question is how to set stArr[0]? >>>> >>>> >From experimentation it appears that you can provide a tuple, but not a >>>> list. Hence the following works just fine (and that the tuple can >>>> contain a list): >>>> strArr[0,0] = ([1.0, 1.1], 2.0) >>>> >>>> but the following fails: >>>> strArr[0,0] = [[1.0, 1.1], 2.0] >>>> with an error: >>>> TypeError: expected a readable buffer object >>>> >>>> This is useful information if one is trying to initialize a structured >>>> array from a collection of data, such as that returned from a database >>>> query. >>>> >> >> I added a bit at the end here, though it is mentioned briefly above. >> Feel free to expand. It's a wiki. You just need edit rights. >> >> http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ > > Thanks, I'll make sure that goes in for 1.6.0. > >>> I'm wondering if that's not a bug? If it's intentional then it is >>> certainly counterintuitive. >>> >> >> This comes up from time to time. >> >> http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 >> >> Perhaps an enhancement ticket could be filed? It doesn't sound trivial >> to implement. > > I filed #1758. > > You can also assign with an array which fails silently, certainly a bug: > >>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>> arr['var1'] = np.arange(5) >>>> arr[0] = (10,20) >>>> arr[0] > (10.0, 20.0) > >>>> arr[0] = np.array([10,20]) ?# no exception, but garbage out >>>> arr[0] > (4.2439915824246103e-313, 0.0) > This is a casting issue. Your array is an integer array. You can assign with an array. arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) arr[0] = np.array([10.0,20]) arr[0] (10.0, 20.0) Skipper From jsseabold at gmail.com Tue Mar 8 20:19:25 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 8 Mar 2011 20:19:25 -0500 Subject: [Numpy-discussion] append_fields behavior with a (sort of) empty array Message-ID: First consider this example of column_stack import numpy as np x = np.random.random((5,1)) y = np.random.random((5,3)) arr = np.column_stack((x[[]], y)) # this fails which is expected arr = np.column_stack((x[:,[]], y)) # this happily works I guess because x[:,[]] # array([], shape=(5, 0), dtype=float64) Now if I want to something analogous with structured array I don't see off the bat how. Is this desired behavior? At the very least it's a partial bug I think. import numpy.lib.recfunctions as nprf xx = np.zeros(5, dtype=[('variable',float)]) xx['variable'] = np.random.random(5) yy = np.random.random((5,2)) arr2 = nprf.append_fields((xx[[]][:], ['var1','var2'], yy.T, usemask=False) arr2 array([(1e+20, 0.24277386045950911, 0.3801949115638894), (1e+20, 0.81574702386474807, 0.95094000248766541), (1e+20, 0.7901945469951196, 0.49315509384277167), (1e+20, 0.84140435880491093, 0.37125966704723368), (1e+20, 0.66595836283749366, 0.18154580574750145)], dtype=[('variable', ' Hi folks, I was compiling some extensions for 32-bit python 2.5 on windows using a 64 bit gcc compiler (4.5.1). After a bit of head scratching it seems that I needed to patch numpy/distutils/mingw32ccompiler.py to add the following flags around where it says "set_executables": -m32 -static-libgcc The first flag means I want 32 bit code, the second one is to link lib-gcc as a static library. Seems the static library part might prevent C++ exceptions being passed across dll boundaries, but I don't really know about that, and I have an irrational fear of missing dlls. Anyway, the -m32 option could be useful for other folks when it is appropriate. The static versus dynamic libgcc story would depend on what the actual answer is for exception passing. My vote would be static; since this didn't work in the past, what will be broken? Do these seem to be worth fixing? Python2.5 is quite old now... Cheers, Jon --- C:\Users\wright\Programming\fable\ImageD11\test\demo>gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=c:/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/4.5.1/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../../src/gcc-4.5.1/configure --prefix=/crossdev/prefix-tdm64 - -build=x86_64-w64-mingw32 --enable-targets=all --enable-libgomp --enable-lto --e nable-libstdcxx-debug --enable-languages=c,c++,fortran,objc,obj-c++ --disable-we rror --disable-nls --disable-win32-registry --enable-version-specific-runtime-li bs --enable-fully-dynamic-string --with-pkgversion=tdm64-1 --with-bugurl=http:// tdm-gcc.tdragon.net/bugs Thread model: win32 gcc version 4.5.1 (tdm64-1) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 9 06:21:26 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 9 Mar 2011 19:21:26 +0800 Subject: [Numpy-discussion] deprecations for 1.6; assert_almost_equal In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 7:53 PM, Pauli Virtanen wrote: > Mon, 07 Mar 2011 06:39:20 -0500, josef.pktd wrote: Doc update committed in f527b49a. >> Why does assert_allclose have atol=0, while np.allclose has rtol=1.e-5, >> atol=1.e-8 ? > > Probably no reason, it should be fixed. I'm guessing that whoever wrote assert_allclose wanted only relative tolerance by default, which makes sense to me - it's a little easier to use than "atol + rtol * desired". But either way is fine with me. Cheers, Ralf From ralf.gommers at googlemail.com Wed Mar 9 06:34:36 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 9 Mar 2011 19:34:36 +0800 Subject: [Numpy-discussion] [ANN] Bento 0.0.5, a packaging solution for python software In-Reply-To: References: Message-ID: Hi David, On Tue, Mar 8, 2011 at 10:07 PM, David Cournapeau wrote: > Hi, > > ? ?I am pleased to announce a new release of bento, a packaging > solution for > python which aims at reproducibility, extensibility and simplicity. Looks great! Out of curiosity, do you have a timeline in mind for adding support for bento to numpy master (probably with a "tech preview" label or something like that at first)? Cheers, Ralf From ralf.gommers at googlemail.com Wed Mar 9 07:16:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 9 Mar 2011 20:16:29 +0800 Subject: [Numpy-discussion] Inaccuracy in documentation of np.linalg.pinv In-Reply-To: <4D76D17E.3070605@biostat.ucsf.edu> References: <4D76D17E.3070605@biostat.ucsf.edu> Message-ID: Hi Hao, On Wed, Mar 9, 2011 at 9:01 AM, Hao Xiong wrote: > I think the documentation for np.linalg.pinv contains some inaccuracies. Thanks for your comments. Are you interested to make some improvements yourself? It's quite easy to do on the numpy doc wiki: http://docs.scipy.org/numpy/docs/numpy.linalg.linalg.pinv/ Just ask for edit rights on this list after registering a username. Cheers, Ralf > Most importantly, Moore-Penrose is not defined by the solution to the > least-square > problem. It was defined by the unique solution to 4 equations. Since SVD > can be easily shown to satisfy the same 4 equations, it is the > Moore-Penrose inverse. > > Another way that definition is wrong is that the least-square problem > does not have to have > unique solution. In fact, it can have infinitely many. In this case, > Moore-Penrose is the minimum > norm solution. > > All of this is discussed in the Matlab documentation of pinv, which is > very good. > > > Hao > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Wed Mar 9 09:06:02 2011 From: cournape at gmail.com (David Cournapeau) Date: Wed, 9 Mar 2011 23:06:02 +0900 Subject: [Numpy-discussion] [ANN] Bento 0.0.5, a packaging solution for python software In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 8:34 PM, Ralf Gommers wrote: > Hi David, > > On Tue, Mar 8, 2011 at 10:07 PM, David Cournapeau wrote: >> Hi, >> >> ? ?I am pleased to announce a new release of bento, a packaging >> solution for >> python which aims at reproducibility, extensibility and simplicity. > > Looks great! thanks. > > Out of curiosity, do you have a timeline in mind for adding support > for bento to numpy master (probably with a "tech preview" label or > something like that at first)? Feature-wise, it misses a few things so that waf and bento can be seemingly integrated - that will be one focus for 0.1 release (the next one). At that point, I think it would make sense to include bento in master as preview, and replace the numscons stuff with it. cheers, David From ndbecker2 at gmail.com Wed Mar 9 09:45:36 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Mar 2011 09:45:36 -0500 Subject: [Numpy-discussion] Any easy way to do this? Message-ID: given: w[i,j,k], y[l, k] find: d[l,i,j] = norm(w[i,j] - y[l]) for each triple (l,i,j), w[i,j]-y[l] is a vector, of which I want to find the norm, and store into d[l,i,j] From amcmorl at gmail.com Wed Mar 9 09:56:05 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 9 Mar 2011 09:56:05 -0500 Subject: [Numpy-discussion] Any easy way to do this? In-Reply-To: References: Message-ID: On 9 March 2011 09:45, Neal Becker wrote: > given: w[i,j,k], y[l, k] > > find: > d[l,i,j] = norm(w[i,j] - y[l]) > > for each triple (l,i,j), w[i,j]-y[l] is a vector, of which I want to find the > norm, and store into d[l,i,j] Is something like this what you want to do? w = np.random.randint(100, size=(4,5,6)) y = np.random.randint(100, size=(7,6)) norm = lambda x, axis: np.sqrt(np.sum(x**2, axis=axis)) d = norm(w[:,:,None] - y[None,None], -1) Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From jsseabold at gmail.com Wed Mar 9 10:24:28 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 9 Mar 2011 10:24:28 -0500 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Tue, Mar 8, 2011 at 8:08 PM, Skipper Seabold wrote: > On Sun, Mar 6, 2011 at 11:12 PM, Ralf Gommers > wrote: >> On Sun, Mar 6, 2011 at 1:10 AM, Skipper Seabold wrote: >>> On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers >>> wrote: >>>> On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >>>>> The page >>>>> >>>>> gives a good introduction to structured arrays. However, it says nothing >>>>> about how to set a particular element (all fields at once) from a >>>>> collection of data. >>>>> >>>>> For instance: >>>>> >>>>> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >>>>> >>>>> The question is how to set stArr[0]? >>>>> >>>>> >From experimentation it appears that you can provide a tuple, but not a >>>>> list. Hence the following works just fine (and that the tuple can >>>>> contain a list): >>>>> strArr[0,0] = ([1.0, 1.1], 2.0) >>>>> >>>>> but the following fails: >>>>> strArr[0,0] = [[1.0, 1.1], 2.0] >>>>> with an error: >>>>> TypeError: expected a readable buffer object >>>>> >>>>> This is useful information if one is trying to initialize a structured >>>>> array from a collection of data, such as that returned from a database >>>>> query. >>>>> >>> >>> I added a bit at the end here, though it is mentioned briefly above. >>> Feel free to expand. It's a wiki. You just need edit rights. >>> >>> http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ >> >> Thanks, I'll make sure that goes in for 1.6.0. >> >>>> I'm wondering if that's not a bug? If it's intentional then it is >>>> certainly counterintuitive. >>>> >>> >>> This comes up from time to time. >>> >>> http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 >>> >>> Perhaps an enhancement ticket could be filed? It doesn't sound trivial >>> to implement. >> >> I filed #1758. >> >> You can also assign with an array which fails silently, certainly a bug: >> >>>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>>> arr['var1'] = np.arange(5) >>>>> arr[0] = (10,20) >>>>> arr[0] >> (10.0, 20.0) >> >>>>> arr[0] = np.array([10,20]) ?# no exception, but garbage out >>>>> arr[0] >> (4.2439915824246103e-313, 0.0) >> > > This is a casting issue. Your array is an integer array. You can > assign with an array. > > arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) > arr[0] = np.array([10.0,20]) > arr[0] > (10.0, 20.0) > FYI, I fixed the docs to reflect this. I know numpy is already pretty verbose by default, but should the integer case throw up a warning similar to casting from complex to real? >>> x = np.zeros(2) >>> x[:] = np.array([1+1j,1+1j]) ComplexWarning: Casting complex values to real discards the imaginary part Skipper From ndbecker2 at gmail.com Wed Mar 9 10:37:23 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Mar 2011 10:37:23 -0500 Subject: [Numpy-discussion] Any easy way to do this? References: Message-ID: Angus McMorland wrote: > On 9 March 2011 09:45, Neal Becker wrote: >> given: w[i,j,k], y[l, k] >> >> find: >> d[l,i,j] = norm(w[i,j] - y[l]) >> >> for each triple (l,i,j), w[i,j]-y[l] is a vector, of which I want to find the >> norm, and store into d[l,i,j] > > Is something like this what you want to do? > > w = np.random.randint(100, size=(4,5,6)) > y = np.random.randint(100, size=(7,6)) > norm = lambda x, axis: np.sqrt(np.sum(x**2, axis=axis)) > d = norm(w[:,:,None] - y[None,None], -1) > > Angus. Thanks! Now if I could understand why w[:,:,None] - y[None,None] is what I needed... From robert.kern at gmail.com Wed Mar 9 10:42:19 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 9 Mar 2011 09:42:19 -0600 Subject: [Numpy-discussion] Any easy way to do this? In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 09:37, Neal Becker wrote: > Angus McMorland wrote: > >> On 9 March 2011 09:45, Neal Becker wrote: >>> given: w[i,j,k], y[l, k] >>> >>> find: >>> d[l,i,j] = norm(w[i,j] - y[l]) >>> >>> for each triple (l,i,j), w[i,j]-y[l] is a vector, of which I want to find the >>> norm, and store into d[l,i,j] >> >> Is something like this what you want to do? >> >> w = np.random.randint(100, size=(4,5,6)) >> y = np.random.randint(100, size=(7,6)) >> norm = lambda x, axis: np.sqrt(np.sum(x**2, axis=axis)) >> d = norm(w[:,:,None] - y[None,None], -1) >> >> Angus. > > Thanks! ?Now if I could understand why > w[:,:,None] - y[None,None] is what I needed... np.newaxis is None. Idiomatically, that should be written d = norm(w[:,:,np.newaxis] - y[np.newaxis,np.newaxis], -1) http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#basic-slicing -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From amcmorl at gmail.com Wed Mar 9 10:48:18 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 9 Mar 2011 10:48:18 -0500 Subject: [Numpy-discussion] Any easy way to do this? In-Reply-To: References: Message-ID: On 9 March 2011 10:37, Neal Becker wrote: > Angus McMorland wrote: > >> On 9 March 2011 09:45, Neal Becker wrote: >>> given: w[i,j,k], y[l, k] >>> >>> find: >>> d[l,i,j] = norm(w[i,j] - y[l]) >>> >>> for each triple (l,i,j), w[i,j]-y[l] is a vector, of which I want to find the >>> norm, and store into d[l,i,j] >> >> Is something like this what you want to do? >> >> w = np.random.randint(100, size=(4,5,6)) >> y = np.random.randint(100, size=(7,6)) >> norm = lambda x, axis: np.sqrt(np.sum(x**2, axis=axis)) >> d = norm(w[:,:,None] - y[None,None], -1) >> >> Angus. > > Thanks! ?Now if I could understand why > w[:,:,None] - y[None,None] is what I needed... This makes the corresponding dimensions of the two arrays match up in the same position and order so that broadcasting can occur correctly. ":" means include the existing axis, in the order they appear in the original array "None" is the same as "np.newaxis" (but takes fewer characters to type), and means insert a dimension of length 1. at the end of each list of dimension list, enough ":" to correspond to any unmentioned dimensions are implied, i.e. y[None, None] is the same as y[None, None, :, :] Clear as mud? Angus -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From jmhykes at ncsu.edu Wed Mar 9 10:58:06 2011 From: jmhykes at ncsu.edu (Josh Hykes) Date: Wed, 9 Mar 2011 10:58:06 -0500 Subject: [Numpy-discussion] Is there a memory-efficient alternative to choose? In-Reply-To: References: Message-ID: > > Would the "out" keyword of np.choose be a possible solution? Using that, > you should be able to create the properies_map in place, and perhaps you can > then do the multiplication with the state variables in place also: Thanks, Lutz, for this suggestion. I hadn't thought of using the "out" keyword. This should do the trick for a number of instances in my code. Unfortunately, I'm not sure that it will work for the more complicated expressions. The problem is that the properties_map array can be larger than the answer array. For example, each property could be a matrix that gets multiplied by a vector of state variables in each cell. For example: import numpy as np p0 = np.array([[1., 0.], [1., 1.]]) # matrix properties p1 = np.array([[1., 1.], [0., 1.]]) properties = {0: p0, 1: p1} mesh_map = np.array([[0,0,0,0], [0,1,1,0], [0,1,1,0], [0,0,0,0]]) state_variables = np.arange(mesh_map.size*2).reshape(mesh_map.shape+(2,)) answer = np.zeros_like(state_variables) for i in range(mesh_map.shape[0]): for j in range(mesh_map.shape[1]): m = mesh_map[i,j] answer[i,j,:] = np.dot(properties[m], state_variables[i,j,:]) I was envisioning a sort of generator function defined through Numpy that would operate at C speed, maybe something like: properties_map = np.choose_generator(mesh_map, [properties[0], properties[1]]) answer = np.tensordot(properties_map, state_variables, axes=([3,],[2])) This may not be possible without using f2py or Cython. I just figured I'd ask the experts before going that route. Thanks again. -Josh On Tue, Mar 8, 2011 at 4:53 PM, Lutz Maibaum wrote: > On Mar 8, 2011, at 11:56 AM, Josh Hykes wrote: > > At some point in my code, I need to do a cell-wise multiplication of the > properties with a state variable. The ideal method would (1) be fast (no > Python loops) and (2) not waste memory constructing an entire property map. > My best attempt using choose does (1) but not (2). > > Would the "out" keyword of np.choose be a possible solution? Using that, > you should be able to create the properies_map in place, and perhaps you can > then do the multiplication with the state variables in place also: > > import numpy as np > properties = {0: 0.5, 1: 2.} # 2 materials > mesh_map = np.array([[0,0,0,0], [0,1,1,0], [0,1,1,0], [0,0,0,0]]) # 4x4 > mesh > state_variables = np.arange(mesh_map.size).reshape(mesh_map.shape) # fake > state variables > answer = np.empty (mesh_map.shape, np.float) # do this only once with > dtype of properties > np.choose(mesh_map, (properties[0], properties[1]), out=answer) > answer[:] = answer * state_variables > > Hope this helps, > > Lutz > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailshuwei at gmail.com Wed Mar 9 19:59:30 2011 From: mailshuwei at gmail.com (shu wei) Date: Wed, 9 Mar 2011 18:59:30 -0600 Subject: [Numpy-discussion] : How to sum weighted matrices Message-ID: ---------- Forwarded message ---------- From: shu wei Date: Wed, Mar 9, 2011 at 6:34 PM Subject: Re: [Numpy-discussion] How to sum weighted matrices To: Nicolas SCHEFFER Thanks very much. I just copied the code your guys gave me. Both of them work great I used to program in Matlab. not familiar with the array multiplication in python numpy. I have looked around for some basic explanation on this. However,I am still very confused with the array axes. Could someone explain a little bit about these? Many thanks in advance On Tue, Mar 8, 2011 at 11:29 AM, Nicolas SCHEFFER < scheffer.nicolas at gmail.com> wrote: > Or just with a dot: > > === > In [17]: np.tensordot(weights, matrices, (0,0)) > > Out[17]: > array([[ 5., 5., 5.], > [ 5., 5., 5.]]) > > In [18]: np.dot(matrices.T,weights).T > > Out[18]: > array([[ 5., 5., 5.], > [ 5., 5., 5.]]) > == > make matrices.T C_CONTIGUOUS for maximum speed. > > -n > > > On Mon, Mar 7, 2011 at 6:03 PM, shu wei wrote: > > Thanks very much. It works. > > > > On Mon, Mar 7, 2011 at 11:53 AM, wrote: > >> > >> for your problem, you can do: > >> > >> ---------------------------- > >> > >> import numpy as np > >> > >> weights = np.array([1,2]) > >> > >> matrix1 = np.ones((2,3)) > >> matrix2 = 2*np.ones((2,3)) > >> > >> matrices = np.array([matrix1,matrix2]) > >> > >> weighted_sum = np.tensordot(weights, matrices, (0,0)) > >> > >> -------------------------- > >> > >> On Mon, Mar 07, 2011 at 06:16:15AM -0600, shu wei wrote: > >> > Hello all, > >> > > >> > I am new to python and numpy. > >> > My question is how to sum up N weighted matrices. > >> > For example w=[1,2] (N=2 case) > >> > m1=[1 2 3, > >> > 3 4 5] > >> > > >> > m2=[3 4 5, > >> > 4 5 6] > >> > I want to get a matrix Y=w[1]*m1+w[2]*m2 by using a loop. > >> > > >> > My original problem is like this > >> > X=[1 2 3, > >> > 3 4 5, > >> > 4 5 6] > >> > > >> > a1=[1 2 3] 1st row of X > >> > m1=a1'*a1 a matirx > >> > a2=[3 4 5] 2nd row of X > >> > m2=a2'*a2 > >> > a3=[ 4 5 6] 3rd row of X > >> > m3=a3'*a3 > >> > > >> > I want to get Y1=w[1]*m1+w[2]*m2 > >> > Y2=w[1]*m2+w[2]*m3 > >> > So basically it is rolling and to sum up the weighted matries > >> > I have a big X, the rolling window is relatively small. > >> > > >> > I tried to use > >> > > >> > sq=np.array([x[i].reshape(-1,1)*x[i] for i in np.arange(0,len(x)]) > # > >> > s=len(x) > >> > m=np.array([sq[i:i+t] for i in np.arange(0,s-t+1)]) # t is the > len(w) > >> > > >> > then I was stuck, I tried to use a loop somethig like > >> > Y=np.array([np.sum(w[i]*m[j,i],axis=0) for i in np.arange(0,t)] ) > >> > Any suggestion is welcome. > >> > > >> > sue > >> > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > >> -- > >> There are two things children should get > >> from their parents: roots and wings. > >> > >> The king who needs to remind his people of his rank, is no king. > >> > >> A beggar's mistake harms no one but the beggar. A king's mistake, > >> however, harms everyone but the king. Too often, the measure of > >> power lies not in the number who obey your will, but in the number > >> who suffer your stupidity. > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 9 20:13:54 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 10 Mar 2011 09:13:54 +0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 11:24 PM, Skipper Seabold wrote: > On Tue, Mar 8, 2011 at 8:08 PM, Skipper Seabold wrote: >> On Sun, Mar 6, 2011 at 11:12 PM, Ralf Gommers >> wrote: >>> On Sun, Mar 6, 2011 at 1:10 AM, Skipper Seabold wrote: >>>> On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers >>>> wrote: >>>>> On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >>>>>> The page >>>>>> >>>>>> gives a good introduction to structured arrays. However, it says nothing >>>>>> about how to set a particular element (all fields at once) from a >>>>>> collection of data. >>>>>> >>>>>> For instance: >>>>>> >>>>>> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >>>>>> >>>>>> The question is how to set stArr[0]? >>>>>> >>>>>> >From experimentation it appears that you can provide a tuple, but not a >>>>>> list. Hence the following works just fine (and that the tuple can >>>>>> contain a list): >>>>>> strArr[0,0] = ([1.0, 1.1], 2.0) >>>>>> >>>>>> but the following fails: >>>>>> strArr[0,0] = [[1.0, 1.1], 2.0] >>>>>> with an error: >>>>>> TypeError: expected a readable buffer object >>>>>> >>>>>> This is useful information if one is trying to initialize a structured >>>>>> array from a collection of data, such as that returned from a database >>>>>> query. >>>>>> >>>> >>>> I added a bit at the end here, though it is mentioned briefly above. >>>> Feel free to expand. It's a wiki. You just need edit rights. >>>> >>>> http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ >>> >>> Thanks, I'll make sure that goes in for 1.6.0. >>> >>>>> I'm wondering if that's not a bug? If it's intentional then it is >>>>> certainly counterintuitive. >>>>> >>>> >>>> This comes up from time to time. >>>> >>>> http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 >>>> >>>> Perhaps an enhancement ticket could be filed? It doesn't sound trivial >>>> to implement. >>> >>> I filed #1758. >>> >>> You can also assign with an array which fails silently, certainly a bug: >>> >>>>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>>>> arr['var1'] = np.arange(5) >>>>>> arr[0] = (10,20) >>>>>> arr[0] >>> (10.0, 20.0) >>> >>>>>> arr[0] = np.array([10,20]) ?# no exception, but garbage out >>>>>> arr[0] >>> (4.2439915824246103e-313, 0.0) >>> >> >> This is a casting issue. Your array is an integer array. You can >> assign with an array. >> >> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >> arr[0] = np.array([10.0,20]) >> arr[0] >> (10.0, 20.0) >> > > FYI, I fixed the docs to reflect this. Thanks, the doc is accurate now. Although I'm not sure we want to document a bug (which I'm still sure it is) like that without making clear it it is a bug. > I know numpy is already pretty verbose by default, but should the > integer case throw up a warning similar to casting from complex to > real? Please no, that will be very annoying. Plus it's much better defined then complex -> real. Cheers, Ralf >>>> x = np.zeros(2) >>>> x[:] = np.array([1+1j,1+1j]) > ComplexWarning: Casting complex values to real discards the imaginary part > > Skipper From jsseabold at gmail.com Wed Mar 9 20:37:48 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 9 Mar 2011 20:37:48 -0500 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 8:13 PM, Ralf Gommers wrote: > On Wed, Mar 9, 2011 at 11:24 PM, Skipper Seabold wrote: >> On Tue, Mar 8, 2011 at 8:08 PM, Skipper Seabold wrote: >>> On Sun, Mar 6, 2011 at 11:12 PM, Ralf Gommers >>> wrote: >>>> On Sun, Mar 6, 2011 at 1:10 AM, Skipper Seabold wrote: >>>>> On Sat, Mar 5, 2011 at 9:28 AM, Ralf Gommers >>>>> wrote: >>>>>> On Sat, Mar 5, 2011 at 8:09 AM, Russell E. Owen wrote: >>>>>>> The page >>>>>>> >>>>>>> gives a good introduction to structured arrays. However, it says nothing >>>>>>> about how to set a particular element (all fields at once) from a >>>>>>> collection of data. >>>>>>> >>>>>>> For instance: >>>>>>> >>>>>>> stArr = numpy.zeros([4,5], dtype=[("pos", float, (2,)), ("rot", float)]) >>>>>>> >>>>>>> The question is how to set stArr[0]? >>>>>>> >>>>>>> >From experimentation it appears that you can provide a tuple, but not a >>>>>>> list. Hence the following works just fine (and that the tuple can >>>>>>> contain a list): >>>>>>> strArr[0,0] = ([1.0, 1.1], 2.0) >>>>>>> >>>>>>> but the following fails: >>>>>>> strArr[0,0] = [[1.0, 1.1], 2.0] >>>>>>> with an error: >>>>>>> TypeError: expected a readable buffer object >>>>>>> >>>>>>> This is useful information if one is trying to initialize a structured >>>>>>> array from a collection of data, such as that returned from a database >>>>>>> query. >>>>>>> >>>>> >>>>> I added a bit at the end here, though it is mentioned briefly above. >>>>> Feel free to expand. It's a wiki. You just need edit rights. >>>>> >>>>> http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ >>>> >>>> Thanks, I'll make sure that goes in for 1.6.0. >>>> >>>>>> I'm wondering if that's not a bug? If it's intentional then it is >>>>>> certainly counterintuitive. >>>>>> >>>>> >>>>> This comes up from time to time. >>>>> >>>>> http://thread.gmane.org/gmane.comp.python.numeric.general/30793/focus=30793 >>>>> >>>>> Perhaps an enhancement ticket could be filed? It doesn't sound trivial >>>>> to implement. >>>> >>>> I filed #1758. >>>> >>>> You can also assign with an array which fails silently, certainly a bug: >>>> >>>>>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>>>>> arr['var1'] = np.arange(5) >>>>>>> arr[0] = (10,20) >>>>>>> arr[0] >>>> (10.0, 20.0) >>>> >>>>>>> arr[0] = np.array([10,20]) ?# no exception, but garbage out >>>>>>> arr[0] >>>> (4.2439915824246103e-313, 0.0) >>>> >>> >>> This is a casting issue. Your array is an integer array. You can >>> assign with an array. >>> >>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>> arr[0] = np.array([10.0,20]) >>> arr[0] >>> (10.0, 20.0) >>> >> >> FYI, I fixed the docs to reflect this. > > Thanks, the doc is accurate now. Although I'm not sure we want to > document a bug (which I'm still sure it is) like that without making > clear it it is a bug. > >> I know numpy is already pretty verbose by default, but should the >> integer case throw up a warning similar to casting from complex to >> real? > > Please no, that will be very annoying. Plus it's much better defined > then complex -> real. > I assume this is the same thing x = np.array([1.5]) x.view(int) array([4609434218613702656]) x = np.array([1]) x.view(float) array([ 4.94065646e-324]) I've been bit by this before, so I know not to do it, but I think it would be nice if it threw up a "don't try do that!". Skipper From ben.root at ou.edu Wed Mar 9 20:41:09 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 9 Mar 2011 19:41:09 -0600 Subject: [Numpy-discussion] : How to sum weighted matrices In-Reply-To: References: Message-ID: This is a very good webpage that helped me when I was transitioning from matlab to numpy. I hope it is helpful for you as well. http://www.scipy.org/NumPy_for_Matlab_Users Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From yyc at solvcon.net Wed Mar 9 22:10:58 2011 From: yyc at solvcon.net (Yung-Yu Chen) Date: Wed, 9 Mar 2011 22:10:58 -0500 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 09:11, David Cournapeau wrote: > On Mon, Mar 7, 2011 at 5:53 PM, G?khan Sever > wrote: > > Hello, > > > > I am going to the PyCon this week. I am presenting a poster about an > > atmospheric sciences related project -- the most active development > > from my coding site over at http://code.google.com/p/ccnworks/ > > > > Is there anybody in the community participating there as well? Any > > plans for sprinting or similar activities? > > Will be there as well, but only for the WE + monday, > > I will be there tomorrow, and giving a talk about solving PDEs on Saturday morning (plus a poster). Looking forward to meet you guys and to learn more about contribution to the community. with regards, Yung-Yu Chen > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Yung-Yu Chen PhD candidate of Mechanical Engineering The Ohio State University, Columbus, Ohio +1 (614) 859 2436 http://solvcon.net/yyc/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Wed Mar 9 22:17:46 2011 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Wed, 9 Mar 2011 22:17:46 -0500 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: Hi all, I will be giving a tutorial on machine learning with scikit-learn tomorrow morning and a talk on text classification on Friday. Then I will stay until Monday evening. Regards, -- Olivier From mailshuwei at gmail.com Wed Mar 9 22:25:57 2011 From: mailshuwei at gmail.com (shu wei) Date: Wed, 9 Mar 2011 21:25:57 -0600 Subject: [Numpy-discussion] : How to sum weighted matrices In-Reply-To: References: Message-ID: Thanks, it is very useful. On Wed, Mar 9, 2011 at 7:41 PM, Benjamin Root wrote: > This is a very good webpage that helped me when I was transitioning from > matlab to numpy. I hope it is helpful for you as well. > > http://www.scipy.org/NumPy_for_Matlab_Users > > Ben Root > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kk1674 at nyu.edu Wed Mar 9 23:25:39 2011 From: kk1674 at nyu.edu (Kibeom Kim) Date: Wed, 9 Mar 2011 23:25:39 -0500 Subject: [Numpy-discussion] np.linalg.norm terrible behavior In-Reply-To: References: <4D72C6E3.6050002@gmail.com> Message-ID: I wonder why numpy.norm doesn't call blas norm function by default. is there any good reason? or it's just not coded? -Kibeom Kim On Sun, Mar 6, 2011 at 12:13 PM, Till Stensitzki wrote: > > > > > Moreover, np.linalg.norm is slow compare to blas. > > > > Is there a way/plan to fix it? > > > > Xavier > > > Have a look at: > > http://fseoane.net/blog/2011/computing-the-vector-norm/ > > greetings > Till > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Thu Mar 10 01:20:07 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 09 Mar 2011 22:20:07 -0800 Subject: [Numpy-discussion] Crash with nested recarray on Python 3.x Message-ID: <4D786D97.1090207@uci.edu> Hello, the following code crashes in the last line when using numpy 1.5.1 on Python 3.1 and 3.2, 32 and 64 bit, for Windows. It works with Python 2.x. Can anyone confirm the crash on other platforms? import numpy RECORD1 = [('i', 'i4')] RECORD2 = [('j', RECORD1, 2)] a = numpy.recarray((1,), RECORD2) a.data Christoph From charlesr.harris at gmail.com Thu Mar 10 01:29:11 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Mar 2011 23:29:11 -0700 Subject: [Numpy-discussion] Crash with nested recarray on Python 3.x In-Reply-To: <4D786D97.1090207@uci.edu> References: <4D786D97.1090207@uci.edu> Message-ID: On Wed, Mar 9, 2011 at 11:20 PM, Christoph Gohlke wrote: > Hello, > > the following code crashes in the last line when using numpy 1.5.1 on > Python 3.1 and 3.2, 32 and 64 bit, for Windows. It works with Python > 2.x. Can anyone confirm the crash on other platforms? > > import numpy > RECORD1 = [('i', 'i4')] > RECORD2 = [('j', RECORD1, 2)] > a = numpy.recarray((1,), RECORD2) > a.data > > Don't see it here. Python 3.1.2 (r312:79147, Sep 8 2010, 23:02:57) [GCC 4.5.1 20100812 (Red Hat 4.5.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> RECORD1 = [('i', 'i4')] >>> RECORD2 = [('j', RECORD1, 2)] >>> a = numpy.recarray((1,), RECORD2) >>> a.data >>> numpy.__version__ '1.6.0.dev-3f0f12f' OTOH, the values look suspicious >>> a rec.array([(array([(-1770425816,), (62,)], dtype=[('i', ' From cgohlke at uci.edu Thu Mar 10 01:37:17 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 09 Mar 2011 22:37:17 -0800 Subject: [Numpy-discussion] Crash with nested recarray on Python 3.x In-Reply-To: References: <4D786D97.1090207@uci.edu> Message-ID: <4D78719D.1020704@uci.edu> On 3/9/2011 10:29 PM, Charles R Harris wrote: > > > On Wed, Mar 9, 2011 at 11:20 PM, Christoph Gohlke > wrote: > > Hello, > > the following code crashes in the last line when using numpy 1.5.1 on > Python 3.1 and 3.2, 32 and 64 bit, for Windows. It works with Python > 2.x. Can anyone confirm the crash on other platforms? > > import numpy > RECORD1 = [('i', 'i4')] > RECORD2 = [('j', RECORD1, 2)] > a = numpy.recarray((1,), RECORD2) > a.data > > > Don't see it here. > > Python 3.1.2 (r312:79147, Sep 8 2010, 23:02:57) > [GCC 4.5.1 20100812 (Red Hat 4.5.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >> >> import numpy >> >> RECORD1 = [('i', 'i4')] >> >> RECORD2 = [('j', RECORD1, 2)] >> >> a = numpy.recarray((1,), RECORD2) >> >> a.data > >> >> numpy.__version__ > '1.6.0.dev-3f0f12f' > > OTOH, the values look suspicious > >> >> a > rec.array([(array([(-1770425816,), (62,)], > dtype=[('i', ' dtype=[('j', [('i', ' > Hmm... > > Chuck > Thank you. The values are uninitialized and are supposed to be read from file with "fd.readinto(a.data)", except that in my case the 'data' attribute lookup crashes. Christoph From ralf.gommers at googlemail.com Thu Mar 10 02:15:40 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 10 Mar 2011 15:15:40 +0800 Subject: [Numpy-discussion] C API freeze and review Message-ID: Hi all, In preparation for making a 1.6.x branch, I just updated the C API version. Please do not add any more functions before the branch is created. Over 70 new functions and types were added, we should also review if all those are necessary and if they are documented. Below is a list of all new objects (generated with attached script), and an indication of those items being documented. Most function are from the new iterator, are they all necessary? And can the authors of the items with missing docs please write some? Cheers, Ralf NpyIter_Type PyDatetimeArrType_Type PyHalfArrType_Type PyTimeIntegerArrType_Type PyTimedeltaArrType_Type NpyIter_Copy YES NpyIter_CreateCompatibleStrides NpyIter_Deallocate YES NpyIter_DebugPrint NpyIter_GetAxisStrideArray YES NpyIter_GetBufferSize YES NpyIter_GetDataPtrArray YES NpyIter_GetDescrArray YES NpyIter_GetGetCoords YES NpyIter_GetIndexPtr YES NpyIter_GetInitialDataPtrArray NpyIter_GetInnerFixedStrideArray YES NpyIter_GetInnerLoopSizePtr YES NpyIter_GetInnerStrideArray YES NpyIter_GetIterIndex YES NpyIter_GetIterIndexRange YES NpyIter_GetIterNext YES NpyIter_GetIterSize YES NpyIter_GetIterView YES NpyIter_GetNDim YES NpyIter_GetNIter YES NpyIter_GetOperandArray YES NpyIter_GetReadFlags YES NpyIter_GetShape YES NpyIter_GetWriteFlags YES NpyIter_GotoCoords YES NpyIter_GotoIndex YES NpyIter_GotoIterIndex YES NpyIter_HasCoords YES NpyIter_HasDelayedBufAlloc NpyIter_HasIndex YES NpyIter_HasInnerLoop YES NpyIter_IsBuffered YES NpyIter_IsGrowInner YES NpyIter_IterationNeedsAPI YES NpyIter_MultiNew YES NpyIter_New YES NpyIter_RemoveAxis YES NpyIter_RemoveCoords YES NpyIter_RemoveInnerLoop YES NpyIter_RequiresBuffering NpyIter_Reset YES NpyIter_ResetBasePointers YES NpyIter_ResetToIterIndexRange YES PyArray_CanCastArrayTo YES PyArray_CanCastTypeTo YES PyArray_CastingConverter YES PyArray_ConvertClipmodeSequence PyArray_CountNonzero YES PyArray_DatetimeStructToDatetime PyArray_DatetimeToDatetimeStruct PyArray_EinsteinSum PyArray_FillWithZero PyArray_GetArrayParamsFromObject PyArray_MatrixProduct2 YES PyArray_MinScalarType YES PyArray_NewLikeArray YES PyArray_PromoteTypes YES PyArray_ResultType YES PyArray_SetDatetimeParseFunction PyArray_TimedeltaStructToTimedelta PyArray_TimedeltaToTimedeltaStruct PyUFunc_e_e YES PyUFunc_e_e_As_d_d YES PyUFunc_e_e_As_f_f YES PyUFunc_ee_e YES PyUFunc_ee_e_As_dd_d YES PyUFunc_ee_e_As_ff_f YES -------------- next part -------------- A non-text attachment was scrubbed... Name: scan_c_api_doc.py Type: application/octet-stream Size: 1293 bytes Desc: not available URL: From ralf.gommers at googlemail.com Thu Mar 10 02:23:59 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 10 Mar 2011 15:23:59 +0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 9:37 AM, Skipper Seabold wrote: > On Wed, Mar 9, 2011 at 8:13 PM, Ralf Gommers > wrote: >> On Wed, Mar 9, 2011 at 11:24 PM, Skipper Seabold wrote: >>>>> >>>>> I filed #1758. >>>>> >>>>> You can also assign with an array which fails silently, certainly a bug: >>>>> >>>>>>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>>>>>> arr['var1'] = np.arange(5) >>>>>>>> arr[0] = (10,20) >>>>>>>> arr[0] >>>>> (10.0, 20.0) >>>>> >>>>>>>> arr[0] = np.array([10,20]) ?# no exception, but garbage out >>>>>>>> arr[0] >>>>> (4.2439915824246103e-313, 0.0) >>>>> >>>> >>>> This is a casting issue. Your array is an integer array. You can >>>> assign with an array. >>>> >>>> arr = np.zeros((5,), dtype=[('var1','f8'),('var2','f8')]) >>>> arr[0] = np.array([10.0,20]) >>>> arr[0] >>>> (10.0, 20.0) >>>> >>> >>> FYI, I fixed the docs to reflect this. >> >> Thanks, the doc is accurate now. Although I'm not sure we want to >> document a bug (which I'm still sure it is) like that without making >> clear it it is a bug. >> >>> I know numpy is already pretty verbose by default, but should the >>> integer case throw up a warning similar to casting from complex to >>> real? >> >> Please no, that will be very annoying. Plus it's much better defined >> then complex -> real. >> > > I assume this is the same thing Taking a view is explicitly documented as "reinterpreting bytes in memory". Casting on assignment to a structured array should have nothing to do with that (says my common sense, not sure what's actually happening). Ralf > x = np.array([1.5]) > x.view(int) > array([4609434218613702656]) > > x = np.array([1]) > x.view(float) > array([ ?4.94065646e-324]) > > I've been bit by this before, so I know not to do it, but I think it > would be nice if it threw up a "don't try do that!". > > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Mar 10 02:25:31 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Mar 2011 00:25:31 -0700 Subject: [Numpy-discussion] C API freeze and review In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 12:15 AM, Ralf Gommers wrote: > Hi all, > > In preparation for making a 1.6.x branch, I just updated the C API > version. Please do not add any more functions before the branch is > created. > > Over 70 new functions and types were added, we should also review if > all those are necessary and if they are documented. Below is a list of > all new objects (generated with attached script), and an indication of > those items being documented. > > Most function are from the new iterator, are they all necessary? > > And can the authors of the items with missing docs please write some? > > Cheers, > Ralf > > > NpyIter_Type > PyDatetimeArrType_Type > PyHalfArrType_Type > PyTimeIntegerArrType_Type > PyTimedeltaArrType_Type > > NpyIter_Copy YES > NpyIter_CreateCompatibleStrides > NpyIter_Deallocate YES > NpyIter_DebugPrint > NpyIter_GetAxisStrideArray YES > NpyIter_GetBufferSize YES > NpyIter_GetDataPtrArray YES > NpyIter_GetDescrArray YES > NpyIter_GetGetCoords YES > NpyIter_GetIndexPtr YES > NpyIter_GetInitialDataPtrArray > NpyIter_GetInnerFixedStrideArray YES > NpyIter_GetInnerLoopSizePtr YES > NpyIter_GetInnerStrideArray YES > NpyIter_GetIterIndex YES > NpyIter_GetIterIndexRange YES > NpyIter_GetIterNext YES > NpyIter_GetIterSize YES > NpyIter_GetIterView YES > NpyIter_GetNDim YES > NpyIter_GetNIter YES > NpyIter_GetOperandArray YES > NpyIter_GetReadFlags YES > NpyIter_GetShape YES > NpyIter_GetWriteFlags YES > NpyIter_GotoCoords YES > I'd like to rename this something like NpyIter_GotoMultiIndex, NpyIter_GotoPolyIndex, or some such. We need a name for extended indices. Suggestions? NpyIter_GotoIndex YES > NpyIter_GotoIterIndex YES > NpyIter_HasCoords YES > NpyIter_HasDelayedBufAlloc > NpyIter_HasIndex YES > NpyIter_HasInnerLoop YES > NpyIter_IsBuffered YES > NpyIter_IsGrowInner YES > NpyIter_IterationNeedsAPI YES > NpyIter_MultiNew YES > NpyIter_New YES > NpyIter_RemoveAxis YES > NpyIter_RemoveCoords YES > NpyIter_RemoveInnerLoop YES > NpyIter_RequiresBuffering > NpyIter_Reset YES > NpyIter_ResetBasePointers YES > NpyIter_ResetToIterIndexRange YES > PyArray_CanCastArrayTo YES > PyArray_CanCastTypeTo YES > PyArray_CastingConverter YES > PyArray_ConvertClipmodeSequence > PyArray_CountNonzero YES > PyArray_DatetimeStructToDatetime > PyArray_DatetimeToDatetimeStruct > PyArray_EinsteinSum > PyArray_FillWithZero > PyArray_GetArrayParamsFromObject > PyArray_MatrixProduct2 YES > PyArray_MinScalarType YES > PyArray_NewLikeArray YES > PyArray_PromoteTypes YES > PyArray_ResultType YES > PyArray_SetDatetimeParseFunction > PyArray_TimedeltaStructToTimedelta > PyArray_TimedeltaToTimedeltaStruct > > PyUFunc_e_e YES > PyUFunc_e_e_As_d_d YES > PyUFunc_e_e_As_f_f YES > PyUFunc_ee_e YES > PyUFunc_ee_e_As_dd_d YES > PyUFunc_ee_e_As_ff_f YES > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Mar 10 04:21:04 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Mar 2011 09:21:04 +0000 (UTC) Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page References: Message-ID: Thu, 10 Mar 2011 15:23:59 +0800, Ralf Gommers wrote: [clip] >> x = np.array([1.5]) >> x.view(int) >> array([4609434218613702656]) Yes, `view` is meant to do exactly that. Use `astype` if you want a type cast. Pauli From michael.cognacc at gmail.com Thu Mar 10 05:04:54 2011 From: michael.cognacc at gmail.com (Mic J) Date: Thu, 10 Mar 2011 11:04:54 +0100 Subject: [Numpy-discussion] problem with array/ndarray/list parameter to function Message-ID: Hi i have a problem with passing a parameter to a function xtickvals = np.arange(0, total_project_days+5, 5) its for setting ticks on an axis, the x-axis Ex1: This works xticks = pylab.setp(ax2, xticklabels=['0','10','20','30','40','50','60','70','80','90','100']) Ex2: This doesnt work xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45', '50', '55', '60', '65'] xticks = pylab.setp(ax2, xticklabels) I want to calculate how many tick is needed on that axis and then pass it to function. So that why i need something like the second example (or another way?) ------ I couldnt se code tags? Code to calculate ticks and steps for output -- #xticksval = np.arange(0, total_project_days+stpsz=5, stepsz) xtickvals = np.arange(0, total_project_days+5, 5) # why do we need to define it here, instead of it being created in the for loop xticklabels = [] for i in xtickvals: xticklabels.append(str(i)) xticks = pylab.setp(ax2, xticklabels) ----- The above code gives the same problem as example2 regards mic From charlesr.harris at gmail.com Thu Mar 10 09:10:06 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Mar 2011 07:10:06 -0700 Subject: [Numpy-discussion] C API freeze and review In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 12:25 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Thu, Mar 10, 2011 at 12:15 AM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> Hi all, >> >> In preparation for making a 1.6.x branch, I just updated the C API >> version. Please do not add any more functions before the branch is >> created. >> >> Over 70 new functions and types were added, we should also review if >> all those are necessary and if they are documented. Below is a list of >> all new objects (generated with attached script), and an indication of >> those items being documented. >> >> Most function are from the new iterator, are they all necessary? >> >> And can the authors of the items with missing docs please write some? >> >> Cheers, >> Ralf >> >> >> NpyIter_Type >> PyDatetimeArrType_Type >> PyHalfArrType_Type >> PyTimeIntegerArrType_Type >> PyTimedeltaArrType_Type >> >> NpyIter_Copy YES >> NpyIter_CreateCompatibleStrides >> NpyIter_Deallocate YES >> NpyIter_DebugPrint >> NpyIter_GetAxisStrideArray YES >> NpyIter_GetBufferSize YES >> NpyIter_GetDataPtrArray YES >> NpyIter_GetDescrArray YES >> NpyIter_GetGetCoords YES >> NpyIter_GetIndexPtr YES >> NpyIter_GetInitialDataPtrArray >> NpyIter_GetInnerFixedStrideArray YES >> NpyIter_GetInnerLoopSizePtr YES >> NpyIter_GetInnerStrideArray YES >> NpyIter_GetIterIndex YES >> NpyIter_GetIterIndexRange YES >> NpyIter_GetIterNext YES >> NpyIter_GetIterSize YES >> NpyIter_GetIterView YES >> NpyIter_GetNDim YES >> NpyIter_GetNIter YES >> NpyIter_GetOperandArray YES >> NpyIter_GetReadFlags YES >> NpyIter_GetShape YES >> NpyIter_GetWriteFlags YES >> NpyIter_GotoCoords YES >> > > I'd like to rename this something like NpyIter_GotoMultiIndex, > NpyIter_GotoPolyIndex, or some such. We need a name for extended indices. > Suggestions? > > Maybe the following terms? Index -- single index FlatIndex -- single index into raveled array FullIndex -- [1,2,3] FancyIndex -- fancy index Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Mar 10 11:08:28 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 10 Mar 2011 10:08:28 -0600 Subject: [Numpy-discussion] problem with array/ndarray/list parameter to function In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 4:04 AM, Mic J wrote: > Hi i have a problem with passing a parameter to a function > > > xtickvals = np.arange(0, total_project_days+5, 5) > > its for setting ticks on an axis, the x-axis > > Ex1: This works > xticks = pylab.setp(ax2, > xticklabels=['0','10','20','30','40','50','60','70','80','90','100']) > > Ex2: This doesnt work > xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45', > '50', > '55', '60', '65'] > xticks = pylab.setp(ax2, xticklabels) > > I want to calculate how many tick is needed on that axis and then pass it > to > function. > So that why i need something like the second example (or another way?) > Mic, The problem is that there is a difference between "positional" arguments and "keyword" arguments. In the first example, the second argument is passed as a keyword argument (due to the equal sign). This allows python to know which argument (by name) you want that value to be associated with. However, in the second example, the second argument is given as a positional argument (due to the lack of an equal sign). Python knows which argument (by position) you want that value associated with. So, to make your second example work, just do this: xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45', '50', '55', '60', '65'] xticks = pylab.setp(ax2, xticklabels=xticklabels) That's it! I hope this helps! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at utoronto.ca Thu Mar 10 12:00:05 2011 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Thu, 10 Mar 2011 12:00:05 -0500 Subject: [Numpy-discussion] Is this the optimal way to take index along a single axis? In-Reply-To: References: Message-ID: I see. Should functionality like this be included in numpy? Jon. On Tue, Mar 8, 2011 at 3:39 PM, wrote: > On Tue, Mar 8, 2011 at 3:03 PM, Jonathan Taylor > wrote: >> I am wanting to use an array b to index into an array x with dimension >> bigger by 1 where the element of b indicates what value to extract >> along a certain direction. ?For example, b = x.argmin(axis=1). >> Perhaps I want to use b to create x.min(axis=1) but also to index >> perhaps another array of the same size. >> >> I had a difficult time finding a way to do this with np.take easily >> and even with fancy indexing the resulting line is very complicated: >> >> In [322]: x.shape >> Out[322]: (2, 3, 4) >> >> In [323]: x.min(axis=1) >> Out[323]: >> array([[ 2, ?1, ?7, ?4], >> ? ? ? [ 8, ?0, 15, 12]]) >> >> In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis], >> idx[:,np.newaxis,:], np.arange(x.shape[2])] >> Out[324]: >> array([[[ 2, ?1, ?7, ?4]], >> >> ? ? ? [[ 8, ?0, 15, 12]]]) >> >> In any case I wrote myself my own function for doing this (below) and >> am wondering if this is the best way to do this or if there is >> something else in numpy that I should be using? -- I figure that this >> is a relatively common usecase. >> >> Thanks, >> Jon. >> >> def mytake(A, b, axis): >> ? ?assert len(A.shape) == len(b.shape)+1 >> >> ? ?idx = [] >> ? ?for i in range(len(A.shape)): >> ? ? ? ?if i == axis: >> ? ? ? ? ? ?temp = b.copy() >> ? ? ? ? ? ?shapey = list(temp.shape) >> ? ? ? ? ? ?shapey.insert(i,1) >> ? ? ? ?else: >> ? ? ? ? ? ?temp = np.arange(A.shape[i]) >> ? ? ? ? ? ?shapey = [1]*len(b.shape) >> ? ? ? ? ? ?shapey.insert(i,A.shape[i]) >> ? ? ? ?shapey = tuple(shapey) >> ? ? ? ?temp = temp.reshape(shapey) >> ? ? ? ?idx += [temp] >> >> ? ?return A[tuple(idx)].squeeze() >> >> >> In [319]: util.mytake(x,x.argmin(axis=1), 1) >> Out[319]: >> array([[ 2, ?1, ?7, ?4], >> ? ? ? [ 8, ?0, 15, 12]]) >> >> In [320]: x.min(axis=1) >> Out[320]: >> array([[ 2, ?1, ?7, ?4], >> ? ? ? [ 8, ?0, 15, 12]]) > > fewer lines but essentially the same thing and no shortcuts, I think > >>>> x= np.random.randint(5, size=(2, 3, 4)) >>>> x > array([[[3, 1, 0, 1], > ? ? ? ?[4, 2, 2, 1], > ? ? ? ?[2, 3, 2, 2]], > > ? ? ? [[2, 1, 1, 1], > ? ? ? ?[0, 2, 0, 3], > ? ? ? ?[2, 3, 3, 1]]]) > >>>> idx = [np.arange(i) for i in x.shape] >>>> idx = list(np.ix_(*idx)) >>>> idx[axis]=np.expand_dims(x.argmin(axis),axis) >>>> x[idx] > array([[[2, 1, 0, 1]], > > ? ? ? [[0, 1, 0, 1]]]) > >>>> np.squeeze(x[idx]) > array([[2, 1, 0, 1], > ? ? ? [0, 1, 0, 1]]) > >>>> mytake(x,x.argmin(axis=1), 1) > array([[2, 1, 0, 1], > ? ? ? [0, 1, 0, 1]]) > > Josef > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Mar 10 12:05:11 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Mar 2011 09:05:11 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D758BBB.3070406@molden.no> References: <4D7515AB.6000006@noaa.gov> <4D758BBB.3070406@molden.no> Message-ID: <4D7904C7.7050700@noaa.gov> On 3/7/11 5:51 PM, Sturla Molden wrote: > Den 07.03.2011 18:28, skrev Christopher Barker: >> 1, 2, 3, 4 >> 5, 6 >> 7, 8, 9, 10, 11, 12 >> 13, 14, 15 >> ... >> > A ragged array, as implemented in C++, Java or C# is just an array of > arrays (or 'a pointer to an array of pointers'). Sure, but as a rule I don't find direct translation of C++ or Java code to Pyton the best approach ;-) > Basically, that is an > ndarray of ndarrays (or a list of ndarrays, whatever you prefer). > > >>> ra = np.zeros(4, dtype=np.ndarray) > >>> ra[0] = np.array([1,2,3,4]) > >>> ra[1] = np.array([5,6]) > >>> ra[2] = np.array([7,8,9,10,11,12]) > >>> ra[3] = np.array([13,14,15]) > >>> ra > array([[1 2 3 4], [5 6], [ 7 8 9 10 11 12], [13 14 15]], dtype=object) > >>> ra[1][1] > 6 > >>> ra[2][:] > array([ 7, 8, 9, 10, 11, 12]) yup -- or I could use a list to store the rows, which would add the ability to append rows. > Slicing in two dimensions does not work as some might expect: > > >>> ra[:2][:2] > array([[1 2 3 4], [5 6]], dtype=object) yup -- might want to overload indexing to do something smarter about that, though in myuse-case, slicing "vertically" isn't really useful anyway -- the nth element in one row doesn't neccessarily have anything to do with the nth element in another row. However, asside from the slicing syntax issue, what I lose with the approach is the ability to get reasonable performance on operations on the entire array: ra *= 3.3 I"d like that to be numpy-efficient. What I need to grapple with is: 1) Is there a point to trying to build a general purpose ragged array? Or should I jsut build something that satisfies my use-case at hand? 2) What's the balance I need between performance and flexibility? putting the rows in a list give a lot more flexibility, putting it all in one 1-d numpy array could give better performance. NOTE: this looks like it could use a "growable" numpy array, much like one I've written before -- maybe it's time to revive that project and use it here, fixing some performance issues while I'm at it. Thanks for all your ideas, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at pytables.org Thu Mar 10 12:51:50 2011 From: faltet at pytables.org (Francesc Alted) Date: Thu, 10 Mar 2011 18:51:50 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7904C7.7050700@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <4D758BBB.3070406@molden.no> <4D7904C7.7050700@noaa.gov> Message-ID: <201103101851.50649.faltet@pytables.org> A Thursday 10 March 2011 18:05:11 Christopher Barker escrigu?: > NOTE: this looks like it could use a "growable" numpy array, much > like one I've written before -- maybe it's time to revive that > project and use it here, fixing some performance issues while I'm at > it. A growable array would be indeed very useful in many situations. My carray project [1] is another attempt to create such an object. It grows (or shrink) arrays by adding (removing) chunks, so they are fast beasts for performing these operations. I suppose this is more or less like your approach. I also thought about implementing ragged arrays in carray, like those you are suggesting, but not sure when they will be ready for public consumption. Perhaps it is a good time to join efforts and create a nice 'growable' array package? [1] https://github.com/FrancescAlted/carray -- Francesc Alted From cgohlke at uci.edu Thu Mar 10 13:55:44 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 10 Mar 2011 10:55:44 -0800 Subject: [Numpy-discussion] Crash with nested recarray on Python 3.x In-Reply-To: References: <4D786D97.1090207@uci.edu> Message-ID: <4D791EB0.1040000@uci.edu> On 3/9/2011 10:29 PM, Charles R Harris wrote: > > > On Wed, Mar 9, 2011 at 11:20 PM, Christoph Gohlke > wrote: > > Hello, > > the following code crashes in the last line when using numpy 1.5.1 on > Python 3.1 and 3.2, 32 and 64 bit, for Windows. It works with Python > 2.x. Can anyone confirm the crash on other platforms? > > import numpy > RECORD1 = [('i', 'i4')] > RECORD2 = [('j', RECORD1, 2)] > a = numpy.recarray((1,), RECORD2) > a.data > > > Don't see it here. > > Python 3.1.2 (r312:79147, Sep 8 2010, 23:02:57) > [GCC 4.5.1 20100812 (Red Hat 4.5.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >> >> import numpy >> >> RECORD1 = [('i', 'i4')] >> >> RECORD2 = [('j', RECORD1, 2)] >> >> a = numpy.recarray((1,), RECORD2) >> >> a.data > >> >> numpy.__version__ > '1.6.0.dev-3f0f12f' > > OTOH, the values look suspicious > >> >> a > rec.array([(array([(-1770425816,), (62,)], > dtype=[('i', ' dtype=[('j', [('i', ' > Hmm... > > Chuck > This is apparently fixed in numpy 1.6.0.dev. I opened a ticket for numpy 1.5.x at . Christoph From Chris.Barker at noaa.gov Thu Mar 10 14:29:00 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Mar 2011 11:29:00 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <201103101851.50649.faltet@pytables.org> References: <4D7515AB.6000006@noaa.gov> <4D758BBB.3070406@molden.no> <4D7904C7.7050700@noaa.gov> <201103101851.50649.faltet@pytables.org> Message-ID: <4D79267C.9070107@noaa.gov> On 3/10/11 9:51 AM, Francesc Alted wrote: > A Thursday 10 March 2011 18:05:11 Christopher Barker escrigu?: >> NOTE: this looks like it could use a "growable" numpy array, much >> like one I've written before -- maybe it's time to revive that >> project and use it here, fixing some performance issues while I'm at >> it. > > A growable array would be indeed very useful in many situations. My > carray project [1] is another attempt to create such an object. It > grows (or shrink) arrays by adding (removing) chunks, so they are fast > beasts for performing these operations. I suppose this is more or less > like your approach. I don't think so -- my approach is a lot simpler that I think carray is: it acts much like a python list: 1) it pre-allocates extra space when an array is created. 2) when you append, it uses that extra space 3) when the extra space is used up, it re-allocates the entire array, with some more extra room And to keep it really simple, I'm using a numpy array internally, and np.ndarray.resize to grow it. It can only be grown on the first axis (C-order). IIUC, carray is doing a lot more smarts about keeping the array in chunks that can be individually manipulated, which is very cool.NOt to mention the compression! > I also thought about implementing ragged arrays in carray, like those > you are suggesting, but not sure when they will be ready for public > consumption. That would be very cool, and I think you are right, the facilities in carray could make a really nice ragged array. > Perhaps it is a good time to join efforts and create a nice 'growable' > array package? Sure, but I don't know that I have much to offer, except maybe some test and timing code. By the way, it would be great to have a growable array that could be efficiently used in Cython code -- so you could accumulate a bunch of native C datatype data easily. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Mar 10 14:31:33 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Mar 2011 11:31:33 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D79267C.9070107@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <4D758BBB.3070406@molden.no> <4D7904C7.7050700@noaa.gov> <201103101851.50649.faltet@pytables.org> <4D79267C.9070107@noaa.gov> Message-ID: <4D792715.8050502@noaa.gov> On 3/10/11 11:29 AM, Christopher Barker wrote: > By the way, it would be great to have a growable array that could be > efficiently used in Cython code -- so you could accumulate a bunch of > native C datatype data easily. I just noticed that carray is written in Cython, so that part should be easy! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at pytables.org Thu Mar 10 15:01:32 2011 From: faltet at pytables.org (Francesc Alted) Date: Thu, 10 Mar 2011 21:01:32 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D79267C.9070107@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <201103101851.50649.faltet@pytables.org> <4D79267C.9070107@noaa.gov> Message-ID: <201103102101.32341.faltet@pytables.org> A Thursday 10 March 2011 20:29:00 Christopher Barker escrigu?: > I don't think so -- my approach is a lot simpler that I think carray > is: > > it acts much like a python list: > > 1) it pre-allocates extra space when an array is created. same for carray > 2) when you append, it uses that extra space ditto for carray > 3) when the extra space is used up, it re-allocates the entire array, > with some more extra room again, carray works exactly the same: the extra room is just a new chunk > And to keep it really simple, I'm using a numpy array internally, and > np.ndarray.resize to grow it. It can only be grown on the first axis > (C-order). uh, very similar, except that due to the chunked nature of carray, it does not use `ndarray.resize`, but simply allocates a new chunk. > > Perhaps it is a good time to join efforts and create a nice > > 'growable' array package? > > Sure, but I don't know that I have much to offer, except maybe some > test and timing code. Well, that sounds like it could be a start. Your experience with your package would be a nice thing to have. BTW, can you remember me where it is your code? Maybe I can get some ideas by looking at it. > By the way, it would be great to have a growable array that could be > efficiently used in Cython code -- so you could accumulate a bunch of > native C datatype data easily. As you noticed, chunks and carrays are both native Cython extensions, so it should be pretty easy to call them from Cython directly, avoiding the Python overhead. -- Francesc Alted From gokhansever at gmail.com Thu Mar 10 15:46:15 2011 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 10 Mar 2011 13:46:15 -0700 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: Yung-Yu, We are advertised on this blog http://pycon.blogspot.com/2011/03/pycon-2011-outside-talks-poster-session.html I will be in the conference venue by tomorrow morning. There are many interesting talks and posters that I look forward seeing plus meeting those presenters. See you in Atlanta. On 3/9/11, Yung-Yu Chen wrote: > I will be there tomorrow, and giving a talk about solving PDEs on Saturday > morning (plus a poster). Looking forward to meet you guys and to learn more > about contribution to the community. > > with regards, > Yung-Yu Chen > > -- > Yung-Yu Chen > PhD candidate of Mechanical Engineering > The Ohio State University, Columbus, Ohio > +1 (614) 859 2436 > http://solvcon.net/yyc/ > -- G?khan From Chris.Barker at noaa.gov Thu Mar 10 16:25:27 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Mar 2011 13:25:27 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <201103102101.32341.faltet@pytables.org> References: <4D7515AB.6000006@noaa.gov> <201103101851.50649.faltet@pytables.org> <4D79267C.9070107@noaa.gov> <201103102101.32341.faltet@pytables.org> Message-ID: <4D7941C7.9080903@noaa.gov> On 3/10/11 12:01 PM, Francesc Alted wrote: >> 3) when the extra space is used up, it re-allocates the entire array, >> with some more extra room > > again, carray works exactly the same: the extra room is just a new chunk does it re-allocate the entire thing? >> And to keep it really simple, I'm using a numpy array internally, and >> np.ndarray.resize to grow it. It can only be grown on the first axis >> (C-order). > > uh, very similar, except that due to the chunked nature of carray, it > does not use `ndarray.resize`, but simply allocates a new chunk. right, is each chunk is a ndarray? Or just a c pointer? Anyway, the advantage to your approach (IIUC) is that you are not continually re-allocating and copying data as the array grows. My method reduces how often that happens, but it still does happen, and I imagine (haven't tested) that that has performance issues if the array gets really large, as it needs a contiguous chunk of memory, and can't free the old one until the data has been copied over. > Well, that sounds like it could be a start. Your experience with your > package would be a nice thing to have. BTW, can you remember me where > it is your code? Maybe I can get some ideas by looking at it. I never posted it anywhere public. It's not much code (really!), so I've enclosed it here. > As you noticed, chunks and carrays are both native Cython extensions, so > it should be pretty easy to call them from Cython directly, avoiding the > Python overhead. cool. I hope I'll get a chance to poke into this soon. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: Accumulator.zip Type: application/zip Size: 4703 bytes Desc: not available URL: From pwang at enthought.com Thu Mar 10 16:28:19 2011 From: pwang at enthought.com (Peter Wang) Date: Thu, 10 Mar 2011 16:28:19 -0500 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: I will be there tonight through Sunday night. I posted on Convore about a Visualization BoF, which may be of interest to folks in the numpy/scipy crowd: https://convore.com/pycon-2011/python-visualization-bof/ See you all there! -Peter On Mon, Mar 7, 2011 at 3:53 AM, G?khan Sever wrote: > Hello, > > I am going to the PyCon this week. I am presenting a poster about an > atmospheric sciences related project -- the most active development > from my coding site over at http://code.google.com/p/ccnworks/ > > Is there anybody in the community participating there as well? Any > plans for sprinting or similar activities? > > See you at PyCon. > > -- > G?khan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From rowen at uw.edu Thu Mar 10 20:18:59 2011 From: rowen at uw.edu (Russell E. Owen) Date: Thu, 10 Mar 2011 17:18:59 -0800 Subject: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page References: Message-ID: In article , Skipper Seabold wrote: >> The page >> >> gives a good introduction to structured arrays. However, it says nothing >> about how to set a particular element (all fields at once) from a >> collection of data. > ... > I added a bit at the end here, though it is mentioned briefly above. > Feel free to expand. It's a wiki. You just need edit rights. > > http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/ Thank you very much. I had already tried to update the page but it's auto-generated from docs and I had no idea where to find those docs. -- Russell From ralf.gommers at googlemail.com Thu Mar 10 20:20:06 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 11 Mar 2011 09:20:06 +0800 Subject: [Numpy-discussion] Intel C Compiler users: please test Message-ID: Hi, In commit dede2691 I've committed a change to the 32-bit Intel C compiler (adding -fPIC) and added a 64-bit "intelem" compiler. Please test, if there's any issue the 32-bit flag change can be reverted. Thanks, Ralf From ondrej at certik.cz Thu Mar 10 20:58:36 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 10 Mar 2011 17:58:36 -0800 Subject: [Numpy-discussion] how to compile Fortran using setup.py Message-ID: Hi, I spent about an hour googling and didn't figure this out. Here is my setup.py: setup( name = "libqsnake", cmdclass = {'build_ext': build_ext}, version = "0.1", packages = [ 'qsnake', 'qsnake.calculators', 'qsnake.calculators.tests', 'qsnake.data', 'qsnake.mesh2d', 'qsnake.tests', ], package_data = { 'qsnake.tests': ['phaml_data/domain.*'], }, include_dirs=[numpy.get_include()], ext_modules = [Extension("qsnake.cmesh", [ "qsnake/cmesh.pyx", "qsnake/fmesh.f90", ])], description = "Qsnake standard library", license = "BSD", ) The qsnake.cmesh extension needs to compile .pyx into .c and later to .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then link both things. That's it. In other words, this is what I want distutils to do: $ cython cmesh.pyx $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include $ gfortran -fPIC -o fmesh.o -c fmesh.f90 $ gcc -shared -o cmesh.so cmesh.o fmesh.o If I import the relevant commands above by: from numpy.distutils.core import setup from numpy.distutils.extension import Extension from Cython.Distutils import build_ext import numpy then it gives me: running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building extension "qsnake.cmesh" sources f2py options: [] f2py:> build/src.linux-x86_64-2.6/qsnake/cmeshmodule.c creating build creating build/src.linux-x86_64-2.6 creating build/src.linux-x86_64-2.6/qsnake Reading fortran codes... Reading file 'qsnake/fmesh.f90' (format:free) Traceback (most recent call last): File "setup.py", line 29, in license = "BSD", File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/certik1/repos/qsnake/local/lib/python/distutils/core.py", line 152, in setup dist.run_commands() File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/install.py", line 55, in run r = old_install.run(self) File "/home/certik1/repos/qsnake/local/lib/python/distutils/command/install.py", line 577, in run self.run_command('build') File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/home/certik1/repos/qsnake/local/lib/python/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 334, in build_extension_sources sources = self.f2py_sources(sources, ext) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 593, in f2py_sources ['-m',ext_name]+f_sources) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py", line 342, in run_main postlist=callcrackfortran(files,options) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py", line 276, in callcrackfortran postlist=crackfortran.crackfortran(files) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py", line 2683, in crackfortran readfortrancode(files,crackline) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py", line 420, in readfortrancode dowithline(finalline) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py", line 624, in crackline analyzeline(m,pat[1],line) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py", line 755, in analyzeline if args: args=rmbadname([x.strip() for x in markoutercomma(args).split('@,@')]) File "/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py", line 665, in markoutercomma assert not f,`f,line,l,cc` AssertionError: (1, 'r_min, r_max, a, n, mesh@)@ bind@(@c', 'r_min@,@ r_max@,@ a@,@ n@,@ mesh@)@ bind@(@c', ')') And it seems it's trying to use f2py. I don't want to use f2py, just use regular gfortran to compile .f90 to .o, as shown above. Does anybody know how to do it? Thanks, Ondrej From robert.kern at gmail.com Thu Mar 10 23:25:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Mar 2011 22:25:14 -0600 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote: > Hi, > > I spent about an hour googling and didn't figure this out. Here is my setup.py: > > setup( > ? ?name = "libqsnake", > ? ?cmdclass = {'build_ext': build_ext}, > ? ?version = "0.1", > ? ?packages = [ > ? ? ? ?'qsnake', > ? ? ? ?'qsnake.calculators', > ? ? ? ?'qsnake.calculators.tests', > ? ? ? ?'qsnake.data', > ? ? ? ?'qsnake.mesh2d', > ? ? ? ?'qsnake.tests', > ? ? ? ?], > ? ?package_data = { > ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], > ? ? ? ?}, > ? ?include_dirs=[numpy.get_include()], > ? ?ext_modules = [Extension("qsnake.cmesh", [ > ? ? ? ?"qsnake/cmesh.pyx", > ? ? ? ?"qsnake/fmesh.f90", > ? ? ? ?])], > ? ?description = "Qsnake standard library", > ? ?license = "BSD", > ) > > The qsnake.cmesh extension needs to compile .pyx into .c and later to > .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then > link both things. That's it. In other words, this is what I want > distutils to do: > > $ cython cmesh.pyx > $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 > -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include > $ gfortran -fPIC -o fmesh.o -c fmesh.f90 > $ gcc -shared -o cmesh.so cmesh.o fmesh.o Difficult if sticking with distutils of any flavor. You would have probably to replace the build_ext command to handle both the Fortran and the Cython. You may want to give David Cournapeau's Bento a try: http://pypi.python.org/pypi/bento/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Thu Mar 10 23:55:33 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 11 Mar 2011 12:55:33 +0800 Subject: [Numpy-discussion] C API freeze and review In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 10:10 PM, Charles R Harris wrote: > > > On Thu, Mar 10, 2011 at 12:25 AM, Charles R Harris > wrote: >> >> >> On Thu, Mar 10, 2011 at 12:15 AM, Ralf Gommers >> wrote: >>> >>> Hi all, >>> >>> In preparation for making a 1.6.x branch, I just updated the C API >>> version. Please do not add any more functions before the branch is >>> created. >>> >>> Over 70 new functions and types were added, we should also review if >>> all those are necessary and if they are documented. Below is a list of >>> all new objects (generated with attached script), and an indication of >>> those items being documented. >>> >>> Most function are from the new iterator, are they all necessary? >>> >>> And can the authors of the items with missing docs please write some? >>> >>> Cheers, >>> Ralf >>> >>> >>> NpyIter_Type >>> PyDatetimeArrType_Type >>> PyHalfArrType_Type >>> PyTimeIntegerArrType_Type >>> PyTimedeltaArrType_Type >>> >>> NpyIter_Copy ? ? ? ? ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_CreateCompatibleStrides >>> NpyIter_Deallocate ? ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_DebugPrint >>> NpyIter_GetAxisStrideArray ? ? ? ? ? ? ?YES >>> NpyIter_GetBufferSize ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetDataPtrArray ? ? ? ? ? ? ? ? YES >>> NpyIter_GetDescrArray ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetGetCoords ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_GetIndexPtr ? ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetInitialDataPtrArray >>> NpyIter_GetInnerFixedStrideArray ? ? ? ?YES >>> NpyIter_GetInnerLoopSizePtr ? ? ? ? ? ? YES >>> NpyIter_GetInnerStrideArray ? ? ? ? ? ? YES >>> NpyIter_GetIterIndex ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_GetIterIndexRange ? ? ? ? ? ? ? YES >>> NpyIter_GetIterNext ? ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetIterSize ? ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetIterView ? ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetNDim ? ? ? ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GetNIter ? ? ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_GetOperandArray ? ? ? ? ? ? ? ? YES >>> NpyIter_GetReadFlags ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_GetShape ? ? ? ? ? ? ? ? ? ? ? ?YES >>> NpyIter_GetWriteFlags ? ? ? ? ? ? ? ? ? YES >>> NpyIter_GotoCoords ? ? ? ? ? ? ? ? ? ? ?YES >> >> I'd like to rename this something like? NpyIter_GotoMultiIndex, >> NpyIter_GotoPolyIndex, or some such. We need a name for extended indices. >> Suggestions? >> > > Maybe the following terms? > > Index -- single index > FlatIndex -- single index into raveled array > FullIndex -- [1,2,3] > FancyIndex -- fancy index Thanks to Mark for quickly adding docs and renaming some of the above items. The undocumented items left are: NpyIter_Type NpyIter_CreateCompatibleStrides NpyIter_DebugPrint # does this need to be exposed? PyDatetimeArrType_Type PyTimeIntegerArrType_Type PyTimedeltaArrType_Type PyArray_DatetimeStructToDatetime PyArray_DatetimeToDatetimeStruct PyArray_SetDatetimeParseFunction PyArray_TimedeltaStructToTimedelta PyArray_TimedeltaToTimedeltaStruct PyHalfArrType_Type PyArray_ConvertClipmodeSequence I know the datetime authors are quite busy, but it would really be helpful if they would make a small update to the NEPs to indicate what is implemented, still to come, or will not be done, and copy some of the applicable documentation from the NEP to the C API docs and relevant docstrings. Thanks, Ralf From ondrej at certik.cz Fri Mar 11 01:57:54 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 10 Mar 2011 22:57:54 -0800 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote: > On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote: >> Hi, >> >> I spent about an hour googling and didn't figure this out. Here is my setup.py: >> >> setup( >> ? ?name = "libqsnake", >> ? ?cmdclass = {'build_ext': build_ext}, >> ? ?version = "0.1", >> ? ?packages = [ >> ? ? ? ?'qsnake', >> ? ? ? ?'qsnake.calculators', >> ? ? ? ?'qsnake.calculators.tests', >> ? ? ? ?'qsnake.data', >> ? ? ? ?'qsnake.mesh2d', >> ? ? ? ?'qsnake.tests', >> ? ? ? ?], >> ? ?package_data = { >> ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], >> ? ? ? ?}, >> ? ?include_dirs=[numpy.get_include()], >> ? ?ext_modules = [Extension("qsnake.cmesh", [ >> ? ? ? ?"qsnake/cmesh.pyx", >> ? ? ? ?"qsnake/fmesh.f90", >> ? ? ? ?])], >> ? ?description = "Qsnake standard library", >> ? ?license = "BSD", >> ) >> >> The qsnake.cmesh extension needs to compile .pyx into .c and later to >> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then >> link both things. That's it. In other words, this is what I want >> distutils to do: >> >> $ cython cmesh.pyx >> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 >> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include >> $ gfortran -fPIC -o fmesh.o -c fmesh.f90 >> $ gcc -shared -o cmesh.so cmesh.o fmesh.o > > Difficult if sticking with distutils of any flavor. You would have > probably to replace the build_ext command to handle both the Fortran > and the Cython. You may want to give David Cournapeau's Bento a try: > > ?http://pypi.python.org/pypi/bento/ Thanks Robert. I burned most of my today's evening on this, trying to replace the command, but so far I didn't figure it out. It is indeed difficult, but I wasn't sure, whether it is because I am not so familiar with distutils. I looked at bento, but I'll simply stick to cmake. I thought that for a Python package (that uses cython + C or fortran, which I thought is a pretty standard configuration for scientific computing), I would use distutils, just like other Python packages. With cmake, I have already figured out how to mix fotran, c, c++, Cython and Python and everything works nicely together, very robustly. So if I have to hack distutils anyway, I would write a simple distutils -> cmake bridge. Ondrej From d.s.seljebotn at astro.uio.no Fri Mar 11 03:13:41 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 09:13:41 +0100 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: <4D79D9B5.1080405@student.matnat.uio.no> On 03/11/2011 07:57 AM, Ondrej Certik wrote: > On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote: >> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote: >>> Hi, >>> >>> I spent about an hour googling and didn't figure this out. Here is my setup.py: >>> >>> setup( >>> name = "libqsnake", >>> cmdclass = {'build_ext': build_ext}, >>> version = "0.1", >>> packages = [ >>> 'qsnake', >>> 'qsnake.calculators', >>> 'qsnake.calculators.tests', >>> 'qsnake.data', >>> 'qsnake.mesh2d', >>> 'qsnake.tests', >>> ], >>> package_data = { >>> 'qsnake.tests': ['phaml_data/domain.*'], >>> }, >>> include_dirs=[numpy.get_include()], >>> ext_modules = [Extension("qsnake.cmesh", [ >>> "qsnake/cmesh.pyx", >>> "qsnake/fmesh.f90", >>> ])], >>> description = "Qsnake standard library", >>> license = "BSD", >>> ) >>> >>> The qsnake.cmesh extension needs to compile .pyx into .c and later to >>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then >>> link both things. That's it. In other words, this is what I want >>> distutils to do: >>> >>> $ cython cmesh.pyx >>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 >>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include >>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90 >>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o >> Difficult if sticking with distutils of any flavor. You would have >> probably to replace the build_ext command to handle both the Fortran >> and the Cython. You may want to give David Cournapeau's Bento a try: >> >> http://pypi.python.org/pypi/bento/ > Thanks Robert. I burned most of my today's evening on this, trying to > replace the command, but so far I didn't figure it out. It is indeed > difficult, but I wasn't sure, whether it is because I am not so > familiar with distutils. Kurt Smith spent much more than an evening on Fortran+distutils for the first release of Fwrap, I believe you could find something in the Fwrap 0.1 release... BUT, for Fwrap trunk, Kurt ditched all of that in favour of waf. And it works much, much better (I can simply export "FC=ifort" and it finds the right flags for Intel Fortran). I guess in your case, that means sticking with cmake. > I looked at bento, but I'll simply stick to cmake. I thought that for > a Python package (that uses cython + C or fortran, which I thought is > a pretty standard configuration for scientific computing), I would use > distutils, just like other Python packages. The point of Bento as I understand it is exactly to seperate packaging from building, so that it becomes possible to ditch distutils but still have things behave like a Python package more or less. In theory at least you should be able to plug cmake into Bento (perhaps by having Bento generate files included into your cmake scripts, and then launching a cmake process). David is currently working on using Bento and waf together for building NumPy. > With cmake, I have already figured out how to mix fotran, c, c++, > Cython and Python and everything works nicely together, very robustly. > So if I have to hack distutils anyway, I would write a simple > distutils -> cmake bridge. Please, write a bento+cmake bridge instead :-) Too much time that could have been spent better has gone into distutils already, it's time to just stop using it. Dag Sverre From d.s.seljebotn at astro.uio.no Fri Mar 11 03:24:26 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 09:24:26 +0100 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: <4D79D9B5.1080405@student.matnat.uio.no> References: <4D79D9B5.1080405@student.matnat.uio.no> Message-ID: <4D79DC3A.6030306@student.matnat.uio.no> On 03/11/2011 09:13 AM, Dag Sverre Seljebotn wrote: > On 03/11/2011 07:57 AM, Ondrej Certik wrote: >> On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote: >>> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote: >>>> Hi, >>>> >>>> I spent about an hour googling and didn't figure this out. Here is my setup.py: >>>> >>>> setup( >>>> name = "libqsnake", >>>> cmdclass = {'build_ext': build_ext}, >>>> version = "0.1", >>>> packages = [ >>>> 'qsnake', >>>> 'qsnake.calculators', >>>> 'qsnake.calculators.tests', >>>> 'qsnake.data', >>>> 'qsnake.mesh2d', >>>> 'qsnake.tests', >>>> ], >>>> package_data = { >>>> 'qsnake.tests': ['phaml_data/domain.*'], >>>> }, >>>> include_dirs=[numpy.get_include()], >>>> ext_modules = [Extension("qsnake.cmesh", [ >>>> "qsnake/cmesh.pyx", >>>> "qsnake/fmesh.f90", >>>> ])], >>>> description = "Qsnake standard library", >>>> license = "BSD", >>>> ) >>>> >>>> The qsnake.cmesh extension needs to compile .pyx into .c and later to >>>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then >>>> link both things. That's it. In other words, this is what I want >>>> distutils to do: >>>> >>>> $ cython cmesh.pyx >>>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 >>>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include >>>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90 >>>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o >>> Difficult if sticking with distutils of any flavor. You would have >>> probably to replace the build_ext command to handle both the Fortran >>> and the Cython. You may want to give David Cournapeau's Bento a try: >>> >>> http://pypi.python.org/pypi/bento/ >> Thanks Robert. I burned most of my today's evening on this, trying to >> replace the command, but so far I didn't figure it out. It is indeed >> difficult, but I wasn't sure, whether it is because I am not so >> familiar with distutils. > Kurt Smith spent much more than an evening on Fortran+distutils for the > first release of Fwrap, I believe you could find something in the Fwrap > 0.1 release... > > BUT, for Fwrap trunk, Kurt ditched all of that in favour of waf. And it > works much, much better (I can simply export "FC=ifort" and it finds the > right flags for Intel Fortran). I guess in your case, that means > sticking with cmake. > >> I looked at bento, but I'll simply stick to cmake. I thought that for >> a Python package (that uses cython + C or fortran, which I thought is >> a pretty standard configuration for scientific computing), I would use >> distutils, just like other Python packages. > The point of Bento as I understand it is exactly to seperate packaging > from building, so that it becomes possible to ditch distutils but still > have things behave like a Python package more or less. > > In theory at least you should be able to plug cmake into Bento (perhaps > by having Bento generate files included into your cmake scripts, and > then launching a cmake process). > > David is currently working on using Bento and waf together for building > NumPy. > >> With cmake, I have already figured out how to mix fotran, c, c++, >> Cython and Python and everything works nicely together, very robustly. >> So if I have to hack distutils anyway, I would write a simple >> distutils -> cmake bridge. > Please, write a bento+cmake bridge instead :-) Too much time that could > have been spent better has gone into distutils already, it's time to > just stop using it. Also, bento comes with a distutils bridge now. So with bento->cmake, one should be able to go distutils->bento->cmake. DS From ralf.gommers at googlemail.com Fri Mar 11 04:01:53 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 11 Mar 2011 17:01:53 +0800 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) Message-ID: I'm just going through the very long 1.6 schedule thread to see what is still on the TODO list before a 1.6.x branch can be made. So I'll send a few separate mails, one for each topic. On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted wrote: > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: >> I think it's ok to revert this behavior for backwards compatibility, >> but believe it's an inconsistent and unintuitive choice. In >> broadcasting, there are two operations, growing a dimension 1 -> n, >> and appending a new 1 dimension to the left. The behaviour under >> discussion in assignment is different from normal broadcasting in >> that only the second one is permitted. It is broadcasting the output >> to the input, rather than broadcasting the input to the output. >> >> Suppose a has shape (20,), b has shape (1,20), and c has shape >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c >> has shape (20,20). >> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 >> dimension to the left. This is reasonable and consistent with >> addition. >> >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast >> to match b by adding a 1 dimension to the left. >> >> If we do "a[...]=c", we could broadcast both a and c together to the >> shape (20,20). This results in multiple assignments to each element >> of a, which is inconsistent. This is not analogous to a+c, but >> rather to np.add(c, c, out=a). >> >> The distinction is subtle, but the inconsistent behavior is harmless >> enough for assignment that keeping backwards compatibility seems >> reasonable. > > For what is worth, I also like the behaviour that Mark proposes, and > have updated tables test suite to adapt to this. ?But I'm fine if it is > decided to revert to the previous behaviour. The conclusion on this topic, as I read the discussion, is that we need to keep backwards compatible behavior (even though the proposed change is more intuitive). Has backwards compatibility been fixed already? Ralf From ralf.gommers at googlemail.com Fri Mar 11 04:07:37 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 11 Mar 2011 17:07:37 +0800 Subject: [Numpy-discussion] Structured array copying by field name (was: Numpy 1.6 schedule) Message-ID: On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: > > Structured array copying copies by field name. > > Commit 22d96096bf7d5fb199ca80f2fcd04e8d27815476 > > Before: > >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>>> y = np.array([(2, 3)], dtype=[('a', int), ('b', int)]) >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>>> x[:] = y >>>> x > array([(2, 3)], > ? ? ?dtype=[('a', ' > After: > >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>>> x[:] = y >>>> x > array([(3, 2)], > ? ? ?dtype=[('a', ' > This seems like a pretty hazardous change. Granted, it's in > a bit of a grey area, but people may rely on this. This is still backwards incompatible in current master. Should it be changed back for 1.6? Ralf From ralf.gommers at googlemail.com Fri Mar 11 04:13:05 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 11 Mar 2011 17:13:05 +0800 Subject: [Numpy-discussion] memory layout, K vs C order (was: Numpy 1.6 schedule) Message-ID: On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: > > Ufuncs don't necessarily return arrays in C-order, but instead > try to keep the memory layout of the original if it seems > advantageous vs. memory access. > > Before: > >>>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides > (200, 40, 8) > > After: > >>>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides > (40, 8, 200) > > This gets +1 from me; the drawbacks to this are mostly in code that > makes too much assumptions and makes no checks. My conclusion of this discussion was that it can be left as-is. If an issue turns up during testing of the 1.6.0 beta it can still be changed. Correct? Ralf From charlesr.harris at gmail.com Fri Mar 11 09:42:32 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 07:42:32 -0700 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers wrote: > I'm just going through the very long 1.6 schedule thread to see what > is still on the TODO list before a 1.6.x branch can be made. So I'll > send a few separate mails, one for each topic. > > On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted > wrote: > > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: > >> I think it's ok to revert this behavior for backwards compatibility, > >> but believe it's an inconsistent and unintuitive choice. In > >> broadcasting, there are two operations, growing a dimension 1 -> n, > >> and appending a new 1 dimension to the left. The behaviour under > >> discussion in assignment is different from normal broadcasting in > >> that only the second one is permitted. It is broadcasting the output > >> to the input, rather than broadcasting the input to the output. > >> > >> Suppose a has shape (20,), b has shape (1,20), and c has shape > >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c > >> has shape (20,20). > >> > >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 > >> dimension to the left. This is reasonable and consistent with > >> addition. > >> > >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast > >> to match b by adding a 1 dimension to the left. > >> > >> If we do "a[...]=c", we could broadcast both a and c together to the > >> shape (20,20). This results in multiple assignments to each element > >> of a, which is inconsistent. This is not analogous to a+c, but > >> rather to np.add(c, c, out=a). > >> > >> The distinction is subtle, but the inconsistent behavior is harmless > >> enough for assignment that keeping backwards compatibility seems > >> reasonable. > > > > For what is worth, I also like the behaviour that Mark proposes, and > > have updated tables test suite to adapt to this. But I'm fine if it is > > decided to revert to the previous behaviour. > > The conclusion on this topic, as I read the discussion, is that we > need to keep backwards compatible behavior (even though the proposed > change is more intuitive). Has backwards compatibility been fixed > already? > > I don't think an official conclusion was reached, at least in so far as numpy has an official anything ;) But this change does show up as an error in one of the pandas tests, so it is likely to affect other folks as well. Probably the route of least compatibility hassle is to revert to the old behavior and maybe switch to the new behavior, which I prefer, for 2.0. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 11 09:57:19 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 07:57:19 -0700 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris wrote: > > > On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers > wrote: > >> I'm just going through the very long 1.6 schedule thread to see what >> is still on the TODO list before a 1.6.x branch can be made. So I'll >> send a few separate mails, one for each topic. >> >> On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted >> wrote: >> > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: >> >> I think it's ok to revert this behavior for backwards compatibility, >> >> but believe it's an inconsistent and unintuitive choice. In >> >> broadcasting, there are two operations, growing a dimension 1 -> n, >> >> and appending a new 1 dimension to the left. The behaviour under >> >> discussion in assignment is different from normal broadcasting in >> >> that only the second one is permitted. It is broadcasting the output >> >> to the input, rather than broadcasting the input to the output. >> >> >> >> Suppose a has shape (20,), b has shape (1,20), and c has shape >> >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c >> >> has shape (20,20). >> >> >> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 >> >> dimension to the left. This is reasonable and consistent with >> >> addition. >> >> >> >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast >> >> to match b by adding a 1 dimension to the left. >> >> >> >> If we do "a[...]=c", we could broadcast both a and c together to the >> >> shape (20,20). This results in multiple assignments to each element >> >> of a, which is inconsistent. This is not analogous to a+c, but >> >> rather to np.add(c, c, out=a). >> >> >> >> The distinction is subtle, but the inconsistent behavior is harmless >> >> enough for assignment that keeping backwards compatibility seems >> >> reasonable. >> > >> > For what is worth, I also like the behaviour that Mark proposes, and >> > have updated tables test suite to adapt to this. But I'm fine if it is >> > decided to revert to the previous behaviour. >> >> The conclusion on this topic, as I read the discussion, is that we >> need to keep backwards compatible behavior (even though the proposed >> change is more intuitive). Has backwards compatibility been fixed >> already? >> >> > I don't think an official conclusion was reached, at least in so far as > numpy has an official anything ;) But this change does show up as an error > in one of the pandas tests, so it is likely to affect other folks as well. > Probably the route of least compatibility hassle is to revert to the old > behavior and maybe switch to the new behavior, which I prefer, for 2.0. > > That said, apart from pandas and pytables, and the latter has been fixed, the new behavior doesn't seem to have much fallout. I think it actually exposes unoticed assumptions in code that slipped by because there was no consequence. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Fri Mar 11 10:06:27 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Fri, 11 Mar 2011 10:06:27 -0500 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 9:57 AM, Charles R Harris wrote: > > > On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris > wrote: >> >> >> On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers >> wrote: >>> >>> I'm just going through the very long 1.6 schedule thread to see what >>> is still on the TODO list before a 1.6.x branch can be made. So I'll >>> send a few separate mails, one for each topic. >>> >>> On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted >>> wrote: >>> > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: >>> >> I think it's ok to revert this behavior for backwards compatibility, >>> >> but believe it's an inconsistent and unintuitive choice. In >>> >> broadcasting, there are two operations, growing a dimension 1 -> n, >>> >> and appending a new 1 dimension to the left. The behaviour under >>> >> discussion in assignment is different from normal broadcasting in >>> >> that only the second one is permitted. It is broadcasting the output >>> >> to the input, rather than broadcasting the input to the output. >>> >> >>> >> Suppose a has shape (20,), b has shape (1,20), and c has shape >>> >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c >>> >> has shape (20,20). >>> >> >>> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 >>> >> dimension to the left. This is reasonable and consistent with >>> >> addition. >>> >> >>> >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast >>> >> to match b by adding a 1 dimension to the left. >>> >> >>> >> If we do "a[...]=c", we could broadcast both a and c together to the >>> >> shape (20,20). This results in multiple assignments to each element >>> >> of a, which is inconsistent. This is not analogous to a+c, but >>> >> rather to np.add(c, c, out=a). >>> >> >>> >> The distinction is subtle, but the inconsistent behavior is harmless >>> >> enough for assignment that keeping backwards compatibility seems >>> >> reasonable. >>> > >>> > For what is worth, I also like the behaviour that Mark proposes, and >>> > have updated tables test suite to adapt to this. ?But I'm fine if it is >>> > decided to revert to the previous behaviour. >>> >>> The conclusion on this topic, as I read the discussion, is that we >>> need to keep backwards compatible behavior (even though the proposed >>> change is more intuitive). Has backwards compatibility been fixed >>> already? >>> >> >> I don't think an official conclusion was reached, at least in so far as >> numpy has an official anything ;) But this change does show up as an error >> in one of the pandas tests, so it is likely to affect other folks as well. >> Probably the route of least compatibility hassle is to revert to the old >> behavior and maybe switch to the new behavior, which I prefer, for 2.0. >> > > That said, apart from pandas and pytables, and the latter has been fixed, > the new behavior doesn't seem to have much fallout. I think it actually > exposes unoticed assumptions in code that slipped by because there was no > consequence. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I've fixed the pandas issue-- I'll put out a bugfix release whenever NumPy 1.6 final is out. I don't suspect it will cause very many problems (and those problems will--hopefully--be easy to fix). From charlesr.harris at gmail.com Fri Mar 11 10:51:08 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 08:51:08 -0700 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 8:06 AM, Wes McKinney wrote: > On Fri, Mar 11, 2011 at 9:57 AM, Charles R Harris > wrote: > > > > > > On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers > >> wrote: > >>> > >>> I'm just going through the very long 1.6 schedule thread to see what > >>> is still on the TODO list before a 1.6.x branch can be made. So I'll > >>> send a few separate mails, one for each topic. > >>> > >>> On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted > >>> wrote: > >>> > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: > >>> >> I think it's ok to revert this behavior for backwards compatibility, > >>> >> but believe it's an inconsistent and unintuitive choice. In > >>> >> broadcasting, there are two operations, growing a dimension 1 -> n, > >>> >> and appending a new 1 dimension to the left. The behaviour under > >>> >> discussion in assignment is different from normal broadcasting in > >>> >> that only the second one is permitted. It is broadcasting the output > >>> >> to the input, rather than broadcasting the input to the output. > >>> >> > >>> >> Suppose a has shape (20,), b has shape (1,20), and c has shape > >>> >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c > >>> >> has shape (20,20). > >>> >> > >>> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 > >>> >> dimension to the left. This is reasonable and consistent with > >>> >> addition. > >>> >> > >>> >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast > >>> >> to match b by adding a 1 dimension to the left. > >>> >> > >>> >> If we do "a[...]=c", we could broadcast both a and c together to the > >>> >> shape (20,20). This results in multiple assignments to each element > >>> >> of a, which is inconsistent. This is not analogous to a+c, but > >>> >> rather to np.add(c, c, out=a). > >>> >> > >>> >> The distinction is subtle, but the inconsistent behavior is harmless > >>> >> enough for assignment that keeping backwards compatibility seems > >>> >> reasonable. > >>> > > >>> > For what is worth, I also like the behaviour that Mark proposes, and > >>> > have updated tables test suite to adapt to this. But I'm fine if it > is > >>> > decided to revert to the previous behaviour. > >>> > >>> The conclusion on this topic, as I read the discussion, is that we > >>> need to keep backwards compatible behavior (even though the proposed > >>> change is more intuitive). Has backwards compatibility been fixed > >>> already? > >>> > >> > >> I don't think an official conclusion was reached, at least in so far as > >> numpy has an official anything ;) But this change does show up as an > error > >> in one of the pandas tests, so it is likely to affect other folks as > well. > >> Probably the route of least compatibility hassle is to revert to the old > >> behavior and maybe switch to the new behavior, which I prefer, for 2.0. > >> > > > > That said, apart from pandas and pytables, and the latter has been fixed, > > the new behavior doesn't seem to have much fallout. I think it actually > > exposes unoticed assumptions in code that slipped by because there was no > > consequence. > > > > Chuck > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > I've fixed the pandas issue-- I'll put out a bugfix release whenever > NumPy 1.6 final is out. I don't suspect it will cause very many > problems (and those problems will--hopefully--be easy to fix). > __ > Now I'm really vacillating. I do prefer the new behavior and the fallout does seem minimal. Put me +1 for the change unless a strong objection surfaces. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Mar 11 11:10:09 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 11 Mar 2011 08:10:09 -0800 Subject: [Numpy-discussion] assert_almost_equal bug? Message-ID: assert_almost_equal() and assert_array_almost_equal() raise a ValueError instead of an AssertionError when the array contains np.inf: >> a = np.array([[1., 2.], [3., 4.]]) >> b = a.copy() >> np.testing.assert_almost_equal(a, b) >> b[0,0] = np.inf >> np.testing.assert_almost_equal(a, b) ValueError: Arrays are not almost equal x: array([[ 1., 2.], [ 3., 4.]]) y: array([[ inf, 2.], [ 3., 4.]]) >> np.testing.assert_array_almost_equal(a, b) ValueError: Arrays are not almost equal x: array([[ 1., 2.], [ 3., 4.]]) y: array([[ inf, 2.], [ 3., 4.]]) ticket: http://projects.scipy.org/numpy/ticket/1769 From oliphant at enthought.com Fri Mar 11 11:19:33 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 11 Mar 2011 11:19:33 -0500 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: This discussion is interesting and useful for NumPy 2.0, but the subtle change is not acceptable for NumPy 1.6. The rules were consistent, even if seen as unintuitive by some. The fact that two libraries we know of already had tests break should be a big red warning flag. There are a lot of other libraries that may have tests break. Having to fix tests would be expected for NumPy 2.0, but not 1.6. I am a big -10 for rolling out a change like this in NumPy 1.6 For NumPy 2.0, the discussion is interesting, but I am still a -0 Travis -- (mobile phone of) Travis Oliphant Enthought, Inc. http://www.enthought.com On Mar 11, 2011, at 10:51 AM, Charles R Harris wrote: > > > On Fri, Mar 11, 2011 at 8:06 AM, Wes McKinney wrote: > On Fri, Mar 11, 2011 at 9:57 AM, Charles R Harris > wrote: > > > > > > On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers > >> wrote: > >>> > >>> I'm just going through the very long 1.6 schedule thread to see what > >>> is still on the TODO list before a 1.6.x branch can be made. So I'll > >>> send a few separate mails, one for each topic. > >>> > >>> On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted > >>> wrote: > >>> > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: > >>> >> I think it's ok to revert this behavior for backwards compatibility, > >>> >> but believe it's an inconsistent and unintuitive choice. In > >>> >> broadcasting, there are two operations, growing a dimension 1 -> n, > >>> >> and appending a new 1 dimension to the left. The behaviour under > >>> >> discussion in assignment is different from normal broadcasting in > >>> >> that only the second one is permitted. It is broadcasting the output > >>> >> to the input, rather than broadcasting the input to the output. > >>> >> > >>> >> Suppose a has shape (20,), b has shape (1,20), and c has shape > >>> >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c > >>> >> has shape (20,20). > >>> >> > >>> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 > >>> >> dimension to the left. This is reasonable and consistent with > >>> >> addition. > >>> >> > >>> >> If we do "a[...]=b", under 1.5 rules, a will once again be broadcast > >>> >> to match b by adding a 1 dimension to the left. > >>> >> > >>> >> If we do "a[...]=c", we could broadcast both a and c together to the > >>> >> shape (20,20). This results in multiple assignments to each element > >>> >> of a, which is inconsistent. This is not analogous to a+c, but > >>> >> rather to np.add(c, c, out=a). > >>> >> > >>> >> The distinction is subtle, but the inconsistent behavior is harmless > >>> >> enough for assignment that keeping backwards compatibility seems > >>> >> reasonable. > >>> > > >>> > For what is worth, I also like the behaviour that Mark proposes, and > >>> > have updated tables test suite to adapt to this. But I'm fine if it is > >>> > decided to revert to the previous behaviour. > >>> > >>> The conclusion on this topic, as I read the discussion, is that we > >>> need to keep backwards compatible behavior (even though the proposed > >>> change is more intuitive). Has backwards compatibility been fixed > >>> already? > >>> > >> > >> I don't think an official conclusion was reached, at least in so far as > >> numpy has an official anything ;) But this change does show up as an error > >> in one of the pandas tests, so it is likely to affect other folks as well. > >> Probably the route of least compatibility hassle is to revert to the old > >> behavior and maybe switch to the new behavior, which I prefer, for 2.0. > >> > > > > That said, apart from pandas and pytables, and the latter has been fixed, > > the new behavior doesn't seem to have much fallout. I think it actually > > exposes unoticed assumptions in code that slipped by because there was no > > consequence. > > > > Chuck > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > I've fixed the pandas issue-- I'll put out a bugfix release whenever > NumPy 1.6 final is out. I don't suspect it will cause very many > problems (and those problems will--hopefully--be easy to fix). > __ > > Now I'm really vacillating. I do prefer the new behavior and the fallout does seem minimal. Put me +1 for the change unless a strong objection surfaces. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 11 11:34:41 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 09:34:41 -0700 Subject: [Numpy-discussion] Anybody going to PyCon? In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 7:11 AM, David Cournapeau wrote: > On Mon, Mar 7, 2011 at 5:53 PM, G?khan Sever > wrote: > > Hello, > > > > I am going to the PyCon this week. I am presenting a poster about an > > atmospheric sciences related project -- the most active development > > from my coding site over at http://code.google.com/p/ccnworks/ > > > > Is there anybody in the community participating there as well? Any > > plans for sprinting or similar activities? > > Will be there as well, but only for the WE + monday, > > I hope you are well after the earthquake. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 11 11:40:32 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 09:40:32 -0700 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 9:19 AM, Travis Oliphant wrote: > This discussion is interesting and useful for NumPy 2.0, but the subtle > change is not acceptable for NumPy 1.6. > > The rules were consistent, even if seen as unintuitive by some. > > The fact that two libraries we know of already had tests break should be a > big red warning flag. There are a lot of other libraries that may have > tests break. > > Having to fix tests would be expected for NumPy 2.0, but not 1.6. > > I am a big -10 for rolling out a change like this in NumPy 1.6 > > For NumPy 2.0, the discussion is interesting, but I am still a -0 > > Out of curiosity, what do you see as the essential use case for this feature? Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Fri Mar 11 12:48:58 2011 From: faltet at pytables.org (Francesc Alted) Date: Fri, 11 Mar 2011 18:48:58 +0100 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <4D7941C7.9080903@noaa.gov> References: <4D7515AB.6000006@noaa.gov> <201103102101.32341.faltet@pytables.org> <4D7941C7.9080903@noaa.gov> Message-ID: <201103111848.58689.faltet@pytables.org> A Thursday 10 March 2011 22:25:27 Christopher Barker escrigu?: > On 3/10/11 12:01 PM, Francesc Alted wrote: > >> 3) when the extra space is used up, it re-allocates the entire > >> array, with some more extra room > > > > again, carray works exactly the same: the extra room is just a new > > chunk > > does it re-allocate the entire thing? Ups. Of course not, just a new chunk. > right, is each chunk is a ndarray? Or just a c pointer? Just a C pointer to a malloc'ed area (it cannot be an ndarray, as the data area might be compressed). > Anyway, the advantage to your approach (IIUC) is that you are not > continually re-allocating and copying data as the array grows. My > method reduces how often that happens, but it still does happen, and > I imagine (haven't tested) that that has performance issues if the > array gets really large, as it needs a contiguous chunk of memory, > and can't free the old one until the data has been copied over. Exactly, carray can use memory by small chunks (typically < 1 MB), actually allowing to use fragmented areas across the main memory. This allows for an optimal memory usage (even if compression is not used). > I never posted it anywhere public. It's not much code (really!), so > I've enclosed it here. Ok. Thanks. The code is really simple, and that's great. However, carray is quite more sophisticated, as it supports not only enlarging arrays, but also shrinking, and since 0.4 on, it also supports n- dimensional arrays (although you can only resize along the first dimension). -- Francesc Alted From ondrej at certik.cz Fri Mar 11 13:11:27 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 11 Mar 2011 10:11:27 -0800 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: <4D79DC3A.6030306@student.matnat.uio.no> References: <4D79D9B5.1080405@student.matnat.uio.no> <4D79DC3A.6030306@student.matnat.uio.no> Message-ID: On Fri, Mar 11, 2011 at 12:24 AM, Dag Sverre Seljebotn wrote: > On 03/11/2011 09:13 AM, Dag Sverre Seljebotn wrote: >> On 03/11/2011 07:57 AM, Ondrej Certik wrote: >>> On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern ? wrote: >>>> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik ? wrote: >>>>> Hi, >>>>> >>>>> I spent about an hour googling and didn't figure this out. Here is my setup.py: >>>>> >>>>> setup( >>>>> ? ? ?name = "libqsnake", >>>>> ? ? ?cmdclass = {'build_ext': build_ext}, >>>>> ? ? ?version = "0.1", >>>>> ? ? ?packages = [ >>>>> ? ? ? ? ?'qsnake', >>>>> ? ? ? ? ?'qsnake.calculators', >>>>> ? ? ? ? ?'qsnake.calculators.tests', >>>>> ? ? ? ? ?'qsnake.data', >>>>> ? ? ? ? ?'qsnake.mesh2d', >>>>> ? ? ? ? ?'qsnake.tests', >>>>> ? ? ? ? ?], >>>>> ? ? ?package_data = { >>>>> ? ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], >>>>> ? ? ? ? ?}, >>>>> ? ? ?include_dirs=[numpy.get_include()], >>>>> ? ? ?ext_modules = [Extension("qsnake.cmesh", [ >>>>> ? ? ? ? ?"qsnake/cmesh.pyx", >>>>> ? ? ? ? ?"qsnake/fmesh.f90", >>>>> ? ? ? ? ?])], >>>>> ? ? ?description = "Qsnake standard library", >>>>> ? ? ?license = "BSD", >>>>> ) >>>>> >>>>> The qsnake.cmesh extension needs to compile .pyx into .c and later to >>>>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then >>>>> link both things. That's it. In other words, this is what I want >>>>> distutils to do: >>>>> >>>>> $ cython cmesh.pyx >>>>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 >>>>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include >>>>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90 >>>>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o >>>> Difficult if sticking with distutils of any flavor. You would have >>>> probably to replace the build_ext command to handle both the Fortran >>>> and the Cython. You may want to give David Cournapeau's Bento a try: >>>> >>>> ? ?http://pypi.python.org/pypi/bento/ >>> Thanks Robert. I burned most of my today's evening on this, trying to >>> replace the command, but so far I didn't figure it out. It is indeed >>> difficult, but I wasn't sure, whether it is because I am not so >>> familiar with distutils. >> Kurt Smith spent much more than an evening on Fortran+distutils for the >> first release of Fwrap, I believe you could find something in the Fwrap >> 0.1 release... >> >> BUT, for Fwrap trunk, Kurt ditched all of that in favour of waf. And it >> works much, much better (I can simply export "FC=ifort" and it finds the >> right flags for Intel Fortran). I guess in your case, that means >> sticking with cmake. >> >>> I looked at bento, but I'll simply stick to cmake. I thought that for >>> a Python package (that uses cython + C or fortran, which I thought is >>> a pretty standard configuration for scientific computing), I would use >>> distutils, just like other Python packages. >> The point of Bento as I understand it is exactly to seperate packaging >> from building, so that it becomes possible to ditch distutils but still >> have things behave like a Python package more or less. >> >> In theory at least you should be able to plug cmake into Bento (perhaps >> by having Bento generate files included into your cmake scripts, and >> then launching a cmake process). >> >> David is currently working on using Bento and waf together for building >> NumPy. >> >>> With cmake, I have already figured out how to mix fotran, c, c++, >>> Cython and Python and everything works nicely together, very robustly. >>> So if I have to hack distutils anyway, I would write a simple >>> distutils -> ? cmake bridge. >> Please, write a bento+cmake bridge instead :-) Too much time that could >> have been spent better has gone into distutils already, it's time to >> just stop using it. > > Also, bento comes with a distutils bridge now. So with bento->cmake, one > should be able to go distutils->bento->cmake. Thanks Dag for the explanation about bento. Indeed, you are right, what I want is a solid build system, that can do the job (let's say cmake in my case), and also create Pythonic packaging, so that it does what people expect from a Python package. So the right approach in my case is to stick to distutils to support setup.py, use bento to hook things in, and write a simple bento->cmake bridge if I have time. Ondrej From tkg at lanl.gov Fri Mar 11 13:13:38 2011 From: tkg at lanl.gov (Thomas K Gamble) Date: Fri, 11 Mar 2011 11:13:38 -0700 Subject: [Numpy-discussion] loop vectorization Message-ID: <201103111113.38295.tkg@lanl.gov> I have the followin loop in my code: for i in range(0, nFrames): data += dot(transpose(bipData[:,i,:]), bipData[:,i,:]) bipData is a 1024x258x256 double precision float array. The loop takes all of 15 seconds to run on my computer and, with several hundred files to process... Is there a way to do something like: data = sum(dot(transpose(bipData), bipData)) with dot done on the desired axis of bipData? This might give a fair speed increase. Or perhaps a different approach I'm not seeing? -- Thomas K. Gamble Research Technologist, System/Network Administrator Chemical Diagnostics and Engineering (C-CDE) Los Alamos National Laboratory MS-E543,p:505-665-4323 f:505-665-4267 "There cannot be a crisis next week. My schedule is already full." Henry Kissinger -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshuaamayer at gmail.com Fri Mar 11 13:25:01 2011 From: joshuaamayer at gmail.com (Josh Mayer) Date: Fri, 11 Mar 2011 13:25:01 -0500 Subject: [Numpy-discussion] Using ndpointer with ctypes argtypes Message-ID: I am trying to call a C function that takes a double array as one of its parameters. I am using ndpointer to define the argtypes so that I can pass a numpy array directly. The issue I'm having is that this parameter need not always be set, and when it is not set, the C functions expects a null pointer to a double in it's place. Is there any way I can set up a numpy array such that when ndpointer calls from_param() it returns a null pointer? Or is there some other way to go about it? Thanks From d.s.seljebotn at astro.uio.no Fri Mar 11 13:41:39 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 19:41:39 +0100 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) Message-ID: <4D7A6CE3.2010902@student.matnat.uio.no> There's a few libraries out there that needs to know whether or not an array changed since the last time it was used: joblib and pymc comes to mind. I believe joblib computes a SHA1 or md5 hash of array contents, while pymc simply assume you never change an array and uses the id(). The pymc approach is fragile, while in my case the joblib approach is too expensive since I'll call the function again many times in a row with the same large array (yes, I can code around it, but the code gets less streamlined). So, would it be possible to very quickly detect whether a NumPy array is guaranteed to not have changed? Here's a revision counter approach: 1) Introduce a new 64-bit int field "modification_count" in the array object struct. 2) modification_count is incremented any time it is possible that an array changes. In particular, PyArray_DATA would increment the counter. 3) A new PyArray_READONLYDATA is introduced that does not increment the counter, which can be used in strategic spots. However, the point is simply to rule out *most* sources of having to recompute a checksum for the array -- a non-matching modification_count is not a guarantee the array has changed, but an unmatched modification_count is a guarantee of an unchanged array 4) The counter can be ignored for readonly (base) arrays. 5a) A method is introduced Python-side, arr.checksum(algorithm="md5"|"sha1"), that uses this machinery to cache checksum computation and that can be plugged into joblib. 5b) Alternatively, the modification count is exposed directly to Python-side, and it is up to users to store the modification count (e.g. in a WeakKeyDictionary indexed by the array's base array). Another solution to the problem would be to allow registering event handlers. Main reason I'm not proposing that is because I don't want to spend the time to implement it (sounds a lot more difficult), it appears to be considerably less backwards-compatible, and so on. Why not a simple dirty flag? Because you'd need one for every possible application of this (e.g, md5 and sha1 would need seperate dirty flags, and other uses than hashing would need yet more flags, and so on). Dag Sverre From charlesr.harris at gmail.com Fri Mar 11 13:47:58 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 11:47:58 -0700 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) In-Reply-To: <4D7A6CE3.2010902@student.matnat.uio.no> References: <4D7A6CE3.2010902@student.matnat.uio.no> Message-ID: On Fri, Mar 11, 2011 at 11:41 AM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > There's a few libraries out there that needs to know whether or not an > array changed since the last time it was used: joblib and pymc comes to > mind. I believe joblib computes a SHA1 or md5 hash of array contents, > while pymc simply assume you never change an array and uses the id(). > > The pymc approach is fragile, while in my case the joblib approach is > too expensive since I'll call the function again many times in a row > with the same large array (yes, I can code around it, but the code gets > less streamlined). > > So, would it be possible to very quickly detect whether a NumPy array is > guaranteed to not have changed? Here's a revision counter approach: > > 1) Introduce a new 64-bit int field "modification_count" in the array > object struct. > > 2) modification_count is incremented any time it is possible that an > array changes. In particular, PyArray_DATA would increment the counter. > > 3) A new PyArray_READONLYDATA is introduced that does not increment > the counter, which can be used in strategic spots. However, the point is > simply to rule out *most* sources of having to recompute a checksum for > the array -- a non-matching modification_count is not a guarantee the > array has changed, but an unmatched modification_count is a guarantee of > an unchanged array > > 4) The counter can be ignored for readonly (base) arrays. > > 5a) A method is introduced Python-side, > arr.checksum(algorithm="md5"|"sha1"), that uses this machinery to cache > checksum computation and that can be plugged into joblib. > > 5b) Alternatively, the modification count is exposed directly to > Python-side, and it is up to users to store the modification count (e.g. > in a WeakKeyDictionary indexed by the array's base array). > > Another solution to the problem would be to allow registering event > handlers. Main reason I'm not proposing that is because I don't want to > spend the time to implement it (sounds a lot more difficult), it appears > to be considerably less backwards-compatible, and so on. > > Why not a simple dirty flag? Because you'd need one for every possible > application of this (e.g, md5 and sha1 would need seperate dirty flags, > and other uses than hashing would need yet more flags, and so on). > > What about views? Wouldn't it be easier to write another object wrapping an ndarray? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmhykes at ncsu.edu Fri Mar 11 14:00:09 2011 From: jmhykes at ncsu.edu (Josh Hykes) Date: Fri, 11 Mar 2011 14:00:09 -0500 Subject: [Numpy-discussion] loop vectorization In-Reply-To: <201103111113.38295.tkg@lanl.gov> References: <201103111113.38295.tkg@lanl.gov> Message-ID: I think you can use tensordot here. Maybe something like the following: from numpy.random import random import numpy as np ni, nj, nk = 4, 5, 6 bipData = random((ni,nj,nk)) data1 = np.zeros((nk,nk)) # loop for i in range(nj): data1 += np.dot(np.transpose(bipData[:,i,:]), bipData[:,i,:]) # tensordot axes_list = [(1,2), (1,0)] data2 = np.tensordot(bipData.T, bipData, axes=axes_list) diff = np.max((data1 - data2)/data1) print diff I hope this helps. -Josh On Fri, Mar 11, 2011 at 1:13 PM, Thomas K Gamble wrote: > I have the followin loop in my code: > > for i in range(0, nFrames): > > data += dot(transpose(bipData[:,i,:]), bipData[:,i,:]) > > bipData is a 1024x258x256 double precision float array. > > The loop takes all of 15 seconds to run on my computer and, with several > hundred files to process... > > Is there a way to do something like: > > data = sum(dot(transpose(bipData), bipData)) > > with dot done on the desired axis of bipData? > > This might give a fair speed increase. Or perhaps a different approach I'm > not seeing? > > -- > > Thomas K. Gamble > > Research Technologist, System/Network Administrator > > Chemical Diagnostics and Engineering (C-CDE) > > Los Alamos National Laboratory > > MS-E543,p:505-665-4323 f:505-665-4267 > > "There cannot be a crisis next week. My schedule is already full." > > Henry Kissinger > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Josh Hykes NCSU grad student (717) 742-0264 jmhykes at ncsu.edu or jhykes at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From qubax at gmx.at Fri Mar 11 14:07:49 2011 From: qubax at gmx.at (qubax at gmx.at) Date: Fri, 11 Mar 2011 20:07:49 +0100 Subject: [Numpy-discussion] loop vectorization In-Reply-To: References: <201103111113.38295.tkg@lanl.gov> Message-ID: <20110311190749.GB10536@tux.hotze.com> we have had that discussion about ... two days ago. please look up 'How to sum weighted matrices' with at least two efficient solutions. On Fri, Mar 11, 2011 at 02:00:09PM -0500, Josh Hykes wrote: > I think you can use tensordot here. Maybe something like the > following: > > from numpy.random import random > import numpy as np > ni, nj, nk = 4, 5, 6 > bipData = random((ni,nj,nk)) > data1 = np.zeros((nk,nk)) > # loop > for i in range(nj): > data1 += np.dot(np.transpose(bipData[:,i,:]), bipData[:,i,:]) > # tensordot > axes_list = [(1,2), (1,0)] > data2 = np.tensordot(bipData.T, bipData, axes=axes_list) > diff = np.max((data1 - data2)/data1) > print diff > I hope this helps. > -Josh > On Fri, Mar 11, 2011 at 1:13 PM, Thomas K Gamble <[1]tkg at lanl.gov> > wrote: > > I have the followin loop in my code: > > for i in range(0, nFrames): > > data += dot(transpose(bipData[:,i,:]), bipData[:,i,:]) > > bipData is a 1024x258x256 double precision float array. > > The loop takes all of 15 seconds to run on my computer and, with > several hundred files to process... > > Is there a way to do something like: > > data = sum(dot(transpose(bipData), bipData)) > > with dot done on the desired axis of bipData? > > This might give a fair speed increase. Or perhaps a different approach > I'm not seeing? > > -- > > Thomas K. Gamble > > Research Technologist, System/Network Administrator > > Chemical Diagnostics and Engineering (C-CDE) > > Los Alamos National Laboratory > > MS-E543,p:[2]505-665-4323 f:[3]505-665-4267 > > "There cannot be a crisis next week. My schedule is already full." > > Henry Kissinger > > _______________________________________________ > NumPy-Discussion mailing list > [4]NumPy-Discussion at scipy.org > [5]http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- > Josh Hykes > NCSU grad student > (717) 742-0264 > [6]jmhykes at ncsu.edu or [7]jhykes at gmail.com > > References > > 1. mailto:tkg at lanl.gov > 2. tel:505-665-4323 > 3. tel:505-665-4267 > 4. mailto:NumPy-Discussion at scipy.org > 5. http://mail.scipy.org/mailman/listinfo/numpy-discussion > 6. mailto:jmhykes at ncsu.edu > 7. mailto:jhykes at gmail.com > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- There are two things children should get from their parents: roots and wings. The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity. From pav at iki.fi Fri Mar 11 14:37:42 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Mar 2011 19:37:42 +0000 (UTC) Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) References: <4D7A6CE3.2010902@student.matnat.uio.no> Message-ID: On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: [clip] > What about views? Wouldn't it be easier to write another object wrapping > an ndarray? I think the buffer interfaces and all other various ways Numpy provides exports for arrays make keeping tabs on modification impossible to do completely reliably. Pauli From d.s.seljebotn at astro.uio.no Fri Mar 11 15:06:27 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 21:06:27 +0100 Subject: [Numpy-discussion] =?utf-8?q?RFC=3A_Detecting_array_changes_=28Nu?= =?utf-8?b?bVB5IDIuMD8p?= In-Reply-To: References: <4D7A6CE3.2010902@student.matnat.uio.no> Message-ID: <28c33ee927fe22b57a7a57b78b20c371@ulrik.uio.no> On Fri, 11 Mar 2011 19:37:42 +0000 (UTC), Pauli Virtanen wrote: > On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: > [clip] >> What about views? Wouldn't it be easier to write another object >> wrapping >> an ndarray? > > I think the buffer interfaces and all other various ways Numpy > provides > exports for arrays make keeping tabs on modification impossible to do > completely reliably. Not to mention all the pain of making sure the arrays are wrapped and stay wrapped in the first place. In particular in combination with other array wrappers. I wasn't saying this is absolutely needed, just that it'd be a really convenient feature helpful for caching. Sometimes, introducing fast caching this way can remove a lot of logic from the code. Introducing a Python-space visible wrapper object kind of defeats the purpose for me. Dag Sverre From mwwiebe at gmail.com Fri Mar 11 15:10:18 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 11 Mar 2011 12:10:18 -0800 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 8:19 AM, Travis Oliphant wrote: > This discussion is interesting and useful for NumPy 2.0, but the subtle > change is not acceptable for NumPy 1.6. > > The rules were consistent, even if seen as unintuitive by some. > > The fact that two libraries we know of already had tests break should be a > big red warning flag. There are a lot of other libraries that may have > tests break. > > Having to fix tests would be expected for NumPy 2.0, but not 1.6. > > I am a big -10 for rolling out a change like this in NumPy 1.6 > I've pushed a patch to change this, with tests. -Mark > > For NumPy 2.0, the discussion is interesting, but I am still a -0 > > Travis > > > -- > (mobile phone of) > Travis Oliphant > Enthought, Inc. > http://www.enthought.com > > On Mar 11, 2011, at 10:51 AM, Charles R Harris > wrote: > > > > On Fri, Mar 11, 2011 at 8:06 AM, Wes McKinney < > wesmckinn at gmail.com> wrote: > >> On Fri, Mar 11, 2011 at 9:57 AM, Charles R Harris >> < charlesr.harris at gmail.com> wrote: >> > >> > >> > On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris >> > < charlesr.harris at gmail.com> wrote: >> >> >> >> >> >> On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers >> >> < ralf.gommers at googlemail.com> wrote: >> >>> >> >>> I'm just going through the very long 1.6 schedule thread to see what >> >>> is still on the TODO list before a 1.6.x branch can be made. So I'll >> >>> send a few separate mails, one for each topic. >> >>> >> >>> On Mon, Mar 7, 2011 at 8:30 PM, Francesc Alted < >> faltet at pytables.org> >> >>> wrote: >> >>> > A Sunday 06 March 2011 06:47:34 Mark Wiebe escrigu?: >> >>> >> I think it's ok to revert this behavior for backwards >> compatibility, >> >>> >> but believe it's an inconsistent and unintuitive choice. In >> >>> >> broadcasting, there are two operations, growing a dimension 1 -> n, >> >>> >> and appending a new 1 dimension to the left. The behaviour under >> >>> >> discussion in assignment is different from normal broadcasting in >> >>> >> that only the second one is permitted. It is broadcasting the >> output >> >>> >> to the input, rather than broadcasting the input to the output. >> >>> >> >> >>> >> Suppose a has shape (20,), b has shape (1,20), and c has shape >> >>> >> (20,1). Then a+b has shape (1,20), a+c has shape (20,20), and b+c >> >>> >> has shape (20,20). >> >>> >> >> >>> >> If we do "b[...] = a", a will be broadcast to match b by adding a 1 >> >>> >> dimension to the left. This is reasonable and consistent with >> >>> >> addition. >> >>> >> >> >>> >> If we do "a[...]=b", under 1.5 rules, a will once again be >> broadcast >> >>> >> to match b by adding a 1 dimension to the left. >> >>> >> >> >>> >> If we do "a[...]=c", we could broadcast both a and c together to >> the >> >>> >> shape (20,20). This results in multiple assignments to each element >> >>> >> of a, which is inconsistent. This is not analogous to a+c, but >> >>> >> rather to np.add(c, c, out=a). >> >>> >> >> >>> >> The distinction is subtle, but the inconsistent behavior is >> harmless >> >>> >> enough for assignment that keeping backwards compatibility seems >> >>> >> reasonable. >> >>> > >> >>> > For what is worth, I also like the behaviour that Mark proposes, and >> >>> > have updated tables test suite to adapt to this. But I'm fine if it >> is >> >>> > decided to revert to the previous behaviour. >> >>> >> >>> The conclusion on this topic, as I read the discussion, is that we >> >>> need to keep backwards compatible behavior (even though the proposed >> >>> change is more intuitive). Has backwards compatibility been fixed >> >>> already? >> >>> >> >> >> >> I don't think an official conclusion was reached, at least in so far as >> >> numpy has an official anything ;) But this change does show up as an >> error >> >> in one of the pandas tests, so it is likely to affect other folks as >> well. >> >> Probably the route of least compatibility hassle is to revert to the >> old >> >> behavior and maybe switch to the new behavior, which I prefer, for 2.0. >> >> >> > >> > That said, apart from pandas and pytables, and the latter has been >> fixed, >> > the new behavior doesn't seem to have much fallout. I think it actually >> > exposes unoticed assumptions in code that slipped by because there was >> no >> > consequence. >> > >> > Chuck >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> >> I've fixed the pandas issue-- I'll put out a bugfix release whenever >> NumPy 1.6 final is out. I don't suspect it will cause very many >> problems (and those problems will--hopefully--be easy to fix). >> __ >> > > Now I'm really vacillating. I do prefer the new behavior and the fallout > does seem minimal. Put me +1 for the change unless a strong objection > surfaces. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Mar 11 15:20:01 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 11 Mar 2011 12:20:01 -0800 Subject: [Numpy-discussion] Structured array copying by field name (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 1:07 AM, Ralf Gommers wrote: > On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: > > > > Structured array copying copies by field name. > > > > Commit 22d96096bf7d5fb199ca80f2fcd04e8d27815476 > > > > Before: > > > >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) > >>>> y = np.array([(2, 3)], dtype=[('a', int), ('b', int)]) > >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) > >>>> x[:] = y > >>>> x > > array([(2, 3)], > > dtype=[('a', ' > > > After: > > > >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) > >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) > >>>> x[:] = y > >>>> x > > array([(3, 2)], > > dtype=[('a', ' > > > This seems like a pretty hazardous change. Granted, it's in > > a bit of a grey area, but people may rely on this. > > This is still backwards incompatible in current master. Should it be > changed back for 1.6? > I strongly dislike the behavior in 1.5 and earlier, for reasons such as the example given here: http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055214.html No problems so far have been traced back to this change, indicating that this type of assignment was previously utilized very little, so I'm strongly in favor of keeping it in. Based on my knowledge of the code, I'm pretty sure it's a significant performance improvement as well. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Mar 11 15:22:48 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 11 Mar 2011 12:22:48 -0800 Subject: [Numpy-discussion] memory layout, K vs C order (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 1:13 AM, Ralf Gommers wrote: > On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: > > > > Ufuncs don't necessarily return arrays in C-order, but instead > > try to keep the memory layout of the original if it seems > > advantageous vs. memory access. > > > > Before: > > > >>>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides > > (200, 40, 8) > > > > After: > > > >>>> np.sin(np.zeros((5,5,5)).transpose(1,2,0)).strides > > (40, 8, 200) > > > > This gets +1 from me; the drawbacks to this are mostly in code that > > makes too much assumptions and makes no checks. > > My conclusion of this discussion was that it can be left as-is. If an > issue turns up during testing of the 1.6.0 beta it can still be > changed. Correct? > Yes, this is very easy to revert if necessary. When using the ufuncs directly, the old behavior is still accessible as np.add(a, b, order='C'). -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 11 15:34:00 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 13:34:00 -0700 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) In-Reply-To: <28c33ee927fe22b57a7a57b78b20c371@ulrik.uio.no> References: <4D7A6CE3.2010902@student.matnat.uio.no> <28c33ee927fe22b57a7a57b78b20c371@ulrik.uio.no> Message-ID: On Fri, Mar 11, 2011 at 1:06 PM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > On Fri, 11 Mar 2011 19:37:42 +0000 (UTC), Pauli Virtanen > wrote: > > On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: > > [clip] > >> What about views? Wouldn't it be easier to write another object > >> wrapping > >> an ndarray? > > > > I think the buffer interfaces and all other various ways Numpy > > provides > > exports for arrays make keeping tabs on modification impossible to do > > completely reliably. > > Not to mention all the pain of making sure the arrays are wrapped and > stay wrapped in the first place. In particular in combination with other > array wrappers. > > I wasn't saying this is absolutely needed, just that it'd be a really > convenient feature helpful for caching. Sometimes, introducing fast > caching this way can remove a lot of logic from the code. Introducing a > Python-space visible wrapper object kind of defeats the purpose for me. > > Well, starting with a wrapped object would allow you to experiment and discover what it is you really need. A smallish specialized object is probably a better starting point for development than a big solution. Operating systems do this sort of thing with the VM, but they have hardware assistance down at the lowest level and rather extensive structures to track status. Furthermore, the memory is organized into blocks and that makes it a lot easier to monitor than strided memory. In fact, I think you might want to set up your own memory subsystem and have the arrays sit on top of that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aarchiba at physics.mcgill.ca Fri Mar 11 16:04:30 2011 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Fri, 11 Mar 2011 16:04:30 -0500 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) In-Reply-To: References: <4D7A6CE3.2010902@student.matnat.uio.no> <28c33ee927fe22b57a7a57b78b20c371@ulrik.uio.no> Message-ID: On 11 March 2011 15:34, Charles R Harris wrote: > > On Fri, Mar 11, 2011 at 1:06 PM, Dag Sverre Seljebotn > wrote: >> >> On Fri, 11 Mar 2011 19:37:42 +0000 (UTC), Pauli Virtanen >> wrote: >> > On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: >> > [clip] >> >> What about views? Wouldn't it be easier to write another object >> >> wrapping >> >> an ndarray? >> > >> > I think the buffer interfaces and all other various ways Numpy >> > provides >> > exports for arrays make keeping tabs on modification impossible to do >> > completely reliably. >> >> Not to mention all the pain of making sure the arrays are wrapped and >> stay wrapped in the first place. In particular in combination with other >> array wrappers. >> >> I wasn't saying this is absolutely needed, just that it'd be a really >> convenient feature helpful for caching. Sometimes, introducing fast >> caching this way can remove a lot of logic from the code. Introducing a >> Python-space visible wrapper object kind of defeats the purpose for me. >> > > Well, starting with a wrapped object would allow you to experiment and > discover what it is you really need. A smallish specialized object is > probably a better starting point for development than a big solution. > Operating systems do this sort of thing with the VM, but they have hardware > assistance down at the lowest level and rather extensive structures to track > status. Furthermore, the memory is organized into blocks and that makes it a > lot easier to monitor than strided memory. In fact, I think you might want > to set up your own memory subsystem and have the arrays sit on top of that. In fact, on many systems, using malloc on large contiguous blocks of memory returns a freshly-mmaped region. It's possible that with a little deviousness (and, sadly, some system-specific code) one could arrange to allocate some arrays in a way that would trigger modification-count updating by the VM system. If you're serious about detecting modifications, this sort of thing may be the only way to go - a modification-detection system that misses some modifications might be worse than none at all. An internal numpy setup is going to be a nightmare even if all you have to worry about is views and you're willing to allow non-overlapping views to count as modifying each other - you'd have to add a modification count to the ultimate base array (the one whose deletion triggers disposal of the memory arena), and then every modification to a view would have to walk the linked list of views all the way up to the top to increment the modification counter. You'll also be triggering increments of the modification counter on all sorts of non-modifications that occur in C code. Doable, but a huge job for dubious benefit. Anne From charlesr.harris at gmail.com Fri Mar 11 17:33:57 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 15:33:57 -0700 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 1:10 PM, Mark Wiebe wrote: > On Fri, Mar 11, 2011 at 8:19 AM, Travis Oliphant wrote: > >> This discussion is interesting and useful for NumPy 2.0, but the subtle >> change is not acceptable for NumPy 1.6. >> >> The rules were consistent, even if seen as unintuitive by some. >> >> The fact that two libraries we know of already had tests break should be a >> big red warning flag. There are a lot of other libraries that may have >> tests break. >> >> Having to fix tests would be expected for NumPy 2.0, but not 1.6. >> >> I am a big -10 for rolling out a change like this in NumPy 1.6 >> > > I've pushed a patch to change this, with tests. > > I'm impressed by how little you needed to change. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Fri Mar 11 17:36:29 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 23:36:29 +0100 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) In-Reply-To: References: <4D7A6CE3.2010902@student.matnat.uio.no> <28c33ee927fe22b57a7a57b78b20c371@ulrik.uio.no> Message-ID: <4D7AA3ED.40007@student.matnat.uio.no> On 03/11/2011 10:04 PM, Anne Archibald wrote: > On 11 March 2011 15:34, Charles R Harris wrote: >> On Fri, Mar 11, 2011 at 1:06 PM, Dag Sverre Seljebotn >> wrote: >>> On Fri, 11 Mar 2011 19:37:42 +0000 (UTC), Pauli Virtanen >>> wrote: >>>> On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: >>>> [clip] >>>>> What about views? Wouldn't it be easier to write another object >>>>> wrapping >>>>> an ndarray? >>>> I think the buffer interfaces and all other various ways Numpy >>>> provides >>>> exports for arrays make keeping tabs on modification impossible to do >>>> completely reliably. >>> Not to mention all the pain of making sure the arrays are wrapped and >>> stay wrapped in the first place. In particular in combination with other >>> array wrappers. >>> >>> I wasn't saying this is absolutely needed, just that it'd be a really >>> convenient feature helpful for caching. Sometimes, introducing fast >>> caching this way can remove a lot of logic from the code. Introducing a >>> Python-space visible wrapper object kind of defeats the purpose for me. >>> >> Well, starting with a wrapped object would allow you to experiment and >> discover what it is you really need. A smallish specialized object is >> probably a better starting point for development than a big solution. >> Operating systems do this sort of thing with the VM, but they have hardware >> assistance down at the lowest level and rather extensive structures to track >> status. Furthermore, the memory is organized into blocks and that makes it a >> lot easier to monitor than strided memory. In fact, I think you might want >> to set up your own memory subsystem and have the arrays sit on top of that. > In fact, on many systems, using malloc on large contiguous blocks of > memory returns a freshly-mmaped region. It's possible that with a > little deviousness (and, sadly, some system-specific code) one could > arrange to allocate some arrays in a way that would trigger > modification-count updating by the VM system. If you're serious about > detecting modifications, this sort of thing may be the only way to go > - a modification-detection system that misses some modifications might > be worse than none at all. > > An internal numpy setup is going to be a nightmare even if all you > have to worry about is views and you're willing to allow > non-overlapping views to count as modifying each other - you'd have to > add a modification count to the ultimate base array (the one whose > deletion triggers disposal of the memory arena), and then every > modification to a view would have to walk the linked list of views all > the way up to the top to increment the modification counter. You'll > also be triggering increments of the modification counter on all sorts > of non-modifications that occur in C code. Doable, but a huge job for > dubious benefit. Yes, you are right. For instance PEP-3118 makes it rather natural to hold only the data pointer and object refcount for some time and only modify the data later, and things like that can't be coded around in NumPy no matter the effort. Thanks for your sobering comments. I'll just keep using explicit mechanisms in my program. (I didn't know about the VM modification counting, but wasn't able to find much on Google either. At any rate that is definitely overkill here.) Dag Sverre From ben.root at ou.edu Fri Mar 11 17:42:51 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 11 Mar 2011 16:42:51 -0600 Subject: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 4:33 PM, Charles R Harris wrote: > > > On Fri, Mar 11, 2011 at 1:10 PM, Mark Wiebe wrote: > >> On Fri, Mar 11, 2011 at 8:19 AM, Travis Oliphant wrote: >> >>> This discussion is interesting and useful for NumPy 2.0, but the subtle >>> change is not acceptable for NumPy 1.6. >>> >>> The rules were consistent, even if seen as unintuitive by some. >>> >>> The fact that two libraries we know of already had tests break should be >>> a big red warning flag. There are a lot of other libraries that may have >>> tests break. >>> >>> Having to fix tests would be expected for NumPy 2.0, but not 1.6. >>> >>> I am a big -10 for rolling out a change like this in NumPy 1.6 >>> >> >> I've pushed a patch to change this, with tests. >> >> > I'm impressed by how little you needed to change. > > > > Chuck > > Should tests be added to detect future changes? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Mar 11 17:51:12 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 11 Mar 2011 16:51:12 -0600 Subject: [Numpy-discussion] Structured array copying by field name (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 2:20 PM, Mark Wiebe wrote: > On Fri, Mar 11, 2011 at 1:07 AM, Ralf Gommers > wrote: > >> On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: >> > >> > Structured array copying copies by field name. >> > >> > Commit 22d96096bf7d5fb199ca80f2fcd04e8d27815476 >> > >> > Before: >> > >> >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >> >>>> y = np.array([(2, 3)], dtype=[('a', int), ('b', int)]) >> >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >> >>>> x[:] = y >> >>>> x >> > array([(2, 3)], >> > dtype=[('a', '> > >> > After: >> > >> >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >> >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >> >>>> x[:] = y >> >>>> x >> > array([(3, 2)], >> > dtype=[('a', '> > >> > This seems like a pretty hazardous change. Granted, it's in >> > a bit of a grey area, but people may rely on this. >> >> This is still backwards incompatible in current master. Should it be >> changed back for 1.6? >> > > I strongly dislike the behavior in 1.5 and earlier, for reasons such as the > example given here: > > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055214.html > > No > problems so far have been traced back to this change, indicating that this > type of assignment was previously utilized very little, so I'm strongly in > favor of keeping it in. Based on my knowledge of the code, I'm pretty sure > it's a significant performance improvement as well. > > -Mark > > Ditto on this. However, it isn't like this is fully featured anyway. Consider the following: x = np.array([(0, 1.1)], dtype=[('a', int), ('b', float)]) y = np.array([(2.1, 4)], dtype=[('b', float), ('a', int)]) print x + y Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' So, how much has people been relying on either behaviors, given that basic math operations weren't permissible in a similar manner? I doubt either approaches would be that noticeable. Note, I would love to be able to do the above eventually. I see a lot of uses for labeled arrays, especially for in the matplotlib library. I personally use the larry package from time to time. Just my two cents. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 11 18:03:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 11 Mar 2011 16:03:10 -0700 Subject: [Numpy-discussion] RFC: Detecting array changes (NumPy 2.0?) In-Reply-To: References: <4D7A6CE3.2010902@student.matnat.uio.no> Message-ID: On Fri, Mar 11, 2011 at 12:37 PM, Pauli Virtanen wrote: > On Fri, 11 Mar 2011 11:47:58 -0700, Charles R Harris wrote: > [clip] > > What about views? Wouldn't it be easier to write another object wrapping > > an ndarray? > > I think the buffer interfaces and all other various ways Numpy provides > exports for arrays make keeping tabs on modification impossible to do > completely reliably. > > I didn't mean wrap, as in subclass, I meant wrap in the sense of having ndarray as a private component in a class. One could maybe make a class factory that made a wholly new class for every freshly instanced monitor object and access info could be kept in class variables. Every instance of the generated class would essentially be a view of a single ndarray instance. I don't think all possible accesses could be caught, but for the purposes of mc something less perfect would probably serve. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hao at biostat.ucsf.edu Fri Mar 11 21:50:55 2011 From: hao at biostat.ucsf.edu (Hao Xiong) Date: Fri, 11 Mar 2011 18:50:55 -0800 Subject: [Numpy-discussion] Inaccuracy in documentation of np.linalg.pinv In-Reply-To: References: <4D76D17E.3070605@biostat.ucsf.edu> Message-ID: <4D7ADF8F.6040403@biostat.ucsf.edu> Hi Ralf, I am happy to edit it, although I will have to do it later as I won't have much free time recently. I have registered as haoxiong on docs.scipy.org and would like to request edit rights to the pinv page. Thanks, Hao On 03/09/11 04:16, Ralf Gommers wrote: > Hi Hao, > > On Wed, Mar 9, 2011 at 9:01 AM, Hao Xiong wrote: >> I think the documentation for np.linalg.pinv contains some inaccuracies. > Thanks for your comments. Are you interested to make some improvements > yourself? It's quite easy to do on the numpy doc wiki: > http://docs.scipy.org/numpy/docs/numpy.linalg.linalg.pinv/ > Just ask for edit rights on this list after registering a username. > > Cheers, > Ralf > > >> Most importantly, Moore-Penrose is not defined by the solution to the >> least-square >> problem. It was defined by the unique solution to 4 equations. Since SVD >> can be easily shown to satisfy the same 4 equations, it is the >> Moore-Penrose inverse. >> >> Another way that definition is wrong is that the least-square problem >> does not have to have >> unique solution. In fact, it can have infinitely many. In this case, >> Moore-Penrose is the minimum >> norm solution. >> >> All of this is discussed in the Matlab documentation of pinv, which is >> very good. >> >> >> Hao >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Sat Mar 12 03:30:35 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 12 Mar 2011 00:30:35 -0800 Subject: [Numpy-discussion] ragged array implimentation In-Reply-To: <201103111848.58689.faltet@pytables.org> References: <4D7515AB.6000006@noaa.gov> <201103102101.32341.faltet@pytables.org> <4D7941C7.9080903@noaa.gov> <201103111848.58689.faltet@pytables.org> Message-ID: <4D7B2F2B.6090906@noaa.gov> Francesc Alted wrote: > Just a C pointer to a malloc'ed area (it cannot be an ndarray, as the > data area might be compressed). got it. > Ok. Thanks. The code is really simple, and that's great. However, > carray is quite more sophisticated, as it supports not only enlarging > arrays, but also shrinking, that's handy. > and since 0.4 on, it also supports n- > dimensional arrays (although you can only resize along the first > dimension). I've been meaning to add that, too - not much to it really, but haven't had a need yet, so haven't gotten around to it. I hope it was a bit helpful, and I'm looking forward to seeing what's next for carray! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nadavh at visionsense.com Sat Mar 12 04:08:35 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 12 Mar 2011 01:08:35 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> Having numpy, scipy, and matplotlib working reasonably with python3, a major piece of code I miss for a major python3 migration is an image IO. I found that pylab's imread works fine for png image, but I need to read all the other image format as well as png and jpeg output. Any hints (including advices how easyly construct my own module) are appreciated. Nadav. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sat Mar 12 04:40:32 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 12 Mar 2011 01:40:32 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> Message-ID: <4D7B3F90.8080702@uci.edu> On 3/12/2011 1:08 AM, Nadav Horesh wrote: > Having numpy, scipy, and matplotlib working reasonably with python3, a > major piece of code I miss for a major python3 migration is an image IO. > I found that pylab's imread works fine for png image, but I need to read > all the other image format as well as png and jpeg output. > Any hints (including advices how easyly construct my own module) are > appreciated. > Nadav. > On Windows, PIL (private port at ), PythonMagick , and pygame 1.9.2pre are working reasonably well for image IO. Also the FreeImage library is easy to use with ctypes . Christoph From pearu.peterson at gmail.com Sat Mar 12 05:30:08 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Sat, 12 Mar 2011 12:30:08 +0200 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 3:58 AM, Ondrej Certik wrote: > Hi, > > I spent about an hour googling and didn't figure this out. Here is my > setup.py: > > setup( > name = "libqsnake", > cmdclass = {'build_ext': build_ext}, > version = "0.1", > packages = [ > 'qsnake', > 'qsnake.calculators', > 'qsnake.calculators.tests', > 'qsnake.data', > 'qsnake.mesh2d', > 'qsnake.tests', > ], > package_data = { > 'qsnake.tests': ['phaml_data/domain.*'], > }, > include_dirs=[numpy.get_include()], > ext_modules = [Extension("qsnake.cmesh", [ > "qsnake/cmesh.pyx", > "qsnake/fmesh.f90", > ])], > description = "Qsnake standard library", > license = "BSD", > ) > > You can specify Fortran code, that you don't want to process with f2py, in the libraries list and then use the corresponding library in the extension, for example: setup(... libraries = [('foo', dict(sources=['qsnake/fmesh.f90']))], ext_modules = [Extension("qsnake.cmesh", sources = ["qsnake/cmesh.pyx"], libraries = ['foo'] )], ... ) See also scipy/integrate/setup.py that resolves the same issue but just using the configuration function approach. HTH, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Mar 12 07:00:29 2011 From: cournape at gmail.com (David Cournapeau) Date: Sat, 12 Mar 2011 07:00:29 -0500 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: On Fri, Mar 11, 2011 at 1:57 AM, Ondrej Certik wrote: > On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote: >> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote: >>> Hi, >>> >>> I spent about an hour googling and didn't figure this out. Here is my setup.py: >>> >>> setup( >>> ? ?name = "libqsnake", >>> ? ?cmdclass = {'build_ext': build_ext}, >>> ? ?version = "0.1", >>> ? ?packages = [ >>> ? ? ? ?'qsnake', >>> ? ? ? ?'qsnake.calculators', >>> ? ? ? ?'qsnake.calculators.tests', >>> ? ? ? ?'qsnake.data', >>> ? ? ? ?'qsnake.mesh2d', >>> ? ? ? ?'qsnake.tests', >>> ? ? ? ?], >>> ? ?package_data = { >>> ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], >>> ? ? ? ?}, >>> ? ?include_dirs=[numpy.get_include()], >>> ? ?ext_modules = [Extension("qsnake.cmesh", [ >>> ? ? ? ?"qsnake/cmesh.pyx", >>> ? ? ? ?"qsnake/fmesh.f90", >>> ? ? ? ?])], >>> ? ?description = "Qsnake standard library", >>> ? ?license = "BSD", >>> ) >>> >>> The qsnake.cmesh extension needs to compile .pyx into .c and later to >>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then >>> link both things. That's it. In other words, this is what I want >>> distutils to do: >>> >>> $ cython cmesh.pyx >>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6 >>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include >>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90 >>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o >> >> Difficult if sticking with distutils of any flavor. You would have >> probably to replace the build_ext command to handle both the Fortran >> and the Cython. You may want to give David Cournapeau's Bento a try: >> >> ?http://pypi.python.org/pypi/bento/ > > Thanks Robert. I burned most of my today's evening on this, trying to > replace the command, but so far I didn't figure it out. It is indeed > difficult, but I wasn't sure, whether it is because I am not so > familiar with distutils. > > I looked at bento, but I'll simply stick to cmake. I thought that for > a Python package (that uses cython + C or fortran, which I thought is > a pretty standard configuration for scientific computing), I would use > distutils, just like other Python packages. The initial reason why I started bento is because this is practically impossible: distutils is very adverse to changes, and you cannot easily change this (I know because that's exactly what numscons was). For example, if you generate a cmake file from distutils, you will quickly realize that getting the necessary data is not easy - if you create a build_cmake, you still need install options, but those are not fully available before install is run (you can do some very horrible hacks to get most of them by copying running commands). Instead bento is based around the usual configure/build/install ala autoconf, where all the options are resolved after configure. I am not pretending that bento can solve all your issues, but you can *today* build non trivial python packages with it, including metadata, eggs and windows installers. And there is already a proof of concept to use either distutils compiler classes, waf and a custom build tool without the need to customize anything outside the build phase (i.e. install is exactly the same and does not even need to know about build). As for supporting setup.py, I have started something in that direction: http://cournape.wordpress.com/2011/03/06/adding-a-distutils-compatibility-layer-to-bento/ cheers, David From ralf.gommers at googlemail.com Sat Mar 12 07:16:38 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 12 Mar 2011 20:16:38 +0800 Subject: [Numpy-discussion] assert_almost_equal bug? In-Reply-To: References: Message-ID: On Sat, Mar 12, 2011 at 12:10 AM, Keith Goodman wrote: > assert_almost_equal() and assert_array_almost_equal() raise a > ValueError instead of an AssertionError when the array contains > np.inf: That's a bug, is fixed in 45269ee1. assert_array_compare was checking for nans but not for infs. Cheers, Ralf > >>> a = np.array([[1., 2.], [3., 4.]]) >>> b = a.copy() >>> np.testing.assert_almost_equal(a, b) >>> b[0,0] = np.inf >>> np.testing.assert_almost_equal(a, b) > > ValueError: > Arrays are not almost equal > ?x: array([[ 1., ?2.], > ? ? ? [ 3., ?4.]]) > ?y: array([[ inf, ? 2.], > ? ? ? [ ?3., ? 4.]]) >>> np.testing.assert_array_almost_equal(a, b) > > ValueError: > Arrays are not almost equal > ?x: array([[ 1., ?2.], > ? ? ? [ 3., ?4.]]) > ?y: array([[ inf, ? 2.], > ? ? ? [ ?3., ? 4.]]) > > ticket: http://projects.scipy.org/numpy/ticket/1769 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Sat Mar 12 07:34:13 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 12 Mar 2011 20:34:13 +0800 Subject: [Numpy-discussion] Structured array copying by field name (was: Numpy 1.6 schedule) In-Reply-To: References: Message-ID: On Sat, Mar 12, 2011 at 6:51 AM, Benjamin Root wrote: > > > On Fri, Mar 11, 2011 at 2:20 PM, Mark Wiebe wrote: >> >> On Fri, Mar 11, 2011 at 1:07 AM, Ralf Gommers >> wrote: >>> >>> On Tue, Mar 8, 2011 at 1:35 AM, Pauli Virtanen wrote: >>> > >>> > Structured array copying copies by field name. >>> > >>> > Commit 22d96096bf7d5fb199ca80f2fcd04e8d27815476 >>> > >>> > Before: >>> > >>> >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>> >>>> y = np.array([(2, 3)], dtype=[('a', int), ('b', int)]) >>> >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>> >>>> x[:] = y >>> >>>> x >>> > array([(2, 3)], >>> > ? ? ?dtype=[('a', '>> > >>> > After: >>> > >>> >>>> x = np.array([(0, 1)], dtype=[('a', int), ('b', int)]) >>> >>>> y = np.array([(2, 3)], dtype=[('b', int), ('a', int)]) >>> >>>> x[:] = y >>> >>>> x >>> > array([(3, 2)], >>> > ? ? ?dtype=[('a', '>> > >>> > This seems like a pretty hazardous change. Granted, it's in >>> > a bit of a grey area, but people may rely on this. >>> >>> This is still backwards incompatible in current master. Should it be >>> changed back for 1.6? >> >> I strongly dislike the behavior in 1.5 and earlier, for reasons such as >> the example given here: >> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055214.html >> No problems so far have been traced back to this change, indicating that >> this type of assignment was previously utilized very little, so I'm strongly >> in favor of keeping it in. Based on my knowledge of the code, I'm pretty >> sure it's a significant performance improvement as well. >> -Mark > > Ditto on this. If no problems turn up and no one objects, I am also okay with keeping it in. Either way, I don't think this needs to be resolved before the release of the first beta. Cheers, Ralf > However, it isn't like this is fully featured anyway. > Consider the following: > > x = np.array([(0, 1.1)], dtype=[('a', int), ('b', float)]) > y = np.array([(2.1, 4)], dtype=[('b', float), ('a', int)]) > > print x + y > > Traceback (most recent call last): > ? File "", line 1, in > TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and > 'numpy.ndarray' > > So, how much has people been relying on either behaviors, given that basic > math operations weren't permissible in a similar manner?? I doubt either > approaches would be that noticeable. > > Note, I would love to be able to do the above eventually.? I see a lot of > uses for labeled arrays, especially for in the matplotlib library.? I > personally use the larry package from time to time. > > Just my two cents. > Ben Root From zachary.pincus at yale.edu Sat Mar 12 07:35:03 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 12 Mar 2011 07:35:03 -0500 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <4D7B3F90.8080702@uci.edu> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> Message-ID: Here's a ctypes interface to FreeImage that I wrote a while back and was since cleaned up (and maintained) by the scikits.image folk: https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py If it doesn't work out of the box on python 3, then it should be pretty simple to fix. Zach On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: > > > On 3/12/2011 1:08 AM, Nadav Horesh wrote: >> Having numpy, scipy, and matplotlib working reasonably with >> python3, a >> major piece of code I miss for a major python3 migration is an >> image IO. >> I found that pylab's imread works fine for png image, but I need to >> read >> all the other image format as well as png and jpeg output. >> Any hints (including advices how easyly construct my own module) are >> appreciated. >> Nadav. >> > > On Windows, PIL (private port at > ), PythonMagick > , and pygame 1.9.2pre > are working reasonably well for image IO. Also > the FreeImage library is easy to > use > with ctypes . > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From joshuaamayer at gmail.com Sat Mar 12 09:53:42 2011 From: joshuaamayer at gmail.com (Josh Mayer) Date: Sat, 12 Mar 2011 09:53:42 -0500 Subject: [Numpy-discussion] Using ndpointer with ctypes argtypes In-Reply-To: References: Message-ID: I found a solution that works. What I did was I wrote a custom class to define the argtypes. The from_param(obj) method calls ndpointer when it is a numpy array or if obj is none it returns a null pointer. On Fri, Mar 11, 2011 at 1:25 PM, Josh Mayer wrote: > I am trying to call a C function that takes a double array as one of > its parameters. > > I am using ndpointer to define the argtypes so that I can pass a numpy > array directly. ?The issue I'm having is that this parameter need not > always be set, and when it is not set, the C functions expects a null > pointer to a double in it's place. Is there any way I can set up a > numpy array such that when ndpointer calls from_param() it returns a > null pointer? Or is there some other way to go about it? > > Thanks > From kwgoodman at gmail.com Sat Mar 12 10:00:45 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 12 Mar 2011 07:00:45 -0800 Subject: [Numpy-discussion] assert_almost_equal bug? In-Reply-To: References: Message-ID: On Sat, Mar 12, 2011 at 4:16 AM, Ralf Gommers wrote: > On Sat, Mar 12, 2011 at 12:10 AM, Keith Goodman wrote: >> assert_almost_equal() and assert_array_almost_equal() raise a >> ValueError instead of an AssertionError when the array contains >> np.inf: > > That's a bug, is fixed in 45269ee1. assert_array_compare was checking > for nans but not for infs. Thanks, Ralf! From nadavh at visionsense.com Sat Mar 12 11:45:27 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 12 Mar 2011 08:45:27 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: 1. PythonMagick: Needs boost which I do not have it avaiable on python3 2. Pygame: I have the stable version(1.9.1) should it work? 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' warnings.warn(msg) running install running build running build_py running build_ext -------------------------------------------------------------------- PIL 1.1.7 SETUP SUMMARY -------------------------------------------------------------------- version 1.1.7 platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) [GCC 4.4.4] -------------------------------------------------------------------- --- TKINTER support available --- JPEG support available --- ZLIB (PNG/ZIP) support available --- FREETYPE2 support available --- LITTLECMS support available . . . byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc Traceback (most recent call last): File "setup.py", line 520, in setup(*(), **configuration) # old school :-) File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands self.run_command(cmd) File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run self.run_command(cmd_name) File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command self.distribution.run_command(command) File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run self.byte_compile(outfiles) File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile dry_run=self.dry_run) File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile compile(file, cfile, dfile) File "/usr/lib64/python3.1/py_compile.py", line 137, in compile codestring = f.read() File "/usr/lib64/python3.1/codecs.py", line 300, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte Any idea on how to correct it? Any elegant way to avoid byte compiling? Nadav ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] Sent: 12 March 2011 14:35 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? Here's a ctypes interface to FreeImage that I wrote a while back and was since cleaned up (and maintained) by the scikits.image folk: https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py If it doesn't work out of the box on python 3, then it should be pretty simple to fix. Zach On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: > > > On 3/12/2011 1:08 AM, Nadav Horesh wrote: >> Having numpy, scipy, and matplotlib working reasonably with >> python3, a >> major piece of code I miss for a major python3 migration is an >> image IO. >> I found that pylab's imread works fine for png image, but I need to >> read >> all the other image format as well as png and jpeg output. >> Any hints (including advices how easyly construct my own module) are >> appreciated. >> Nadav. >> > > On Windows, PIL (private port at > ), PythonMagick > , and pygame 1.9.2pre > are working reasonably well for image IO. Also > the FreeImage library is easy to > use > with ctypes . > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From tmp50 at ukr.net Sat Mar 12 12:43:34 2011 From: tmp50 at ukr.net (Dmitrey) Date: Sat, 12 Mar 2011 19:43:34 +0200 Subject: [Numpy-discussion] Inplace remove some array rows Message-ID: hi all, currently I use a = array(m,n) ... a = delete(a, indices, 0) # delete some rows Can I somehow perform the operation in-place, without creating auxiliary array? If I'll use numpy.compress(condition, a, axis=0, out=a), or numpy.take(a, indices, axis=0, out=a) will the operation be inplace? Thank you in advance, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wardefar at iro.umontreal.ca Sat Mar 12 14:43:11 2011 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Sat, 12 Mar 2011 14:43:11 -0500 Subject: [Numpy-discussion] Inplace remove some array rows In-Reply-To: References: Message-ID: <4FDF73A4-9EA7-48E4-BC16-B3C4B15DF21B@iro.umontreal.ca> On 2011-03-12, at 12:43 PM, Dmitrey wrote: > hi all, > currently I use > a = array(m,n) > ... > a = delete(a, indices, 0) # delete some rows > > Can I somehow perform the operation in-place, without creating auxiliary array? > If I'll use > > numpy.compress(condition, a, axis=0, out=a), > or > numpy.take(a, indices, axis=0, out=a) > > will the operation be inplace? a will be the wrong shape to hold the output of either of those operations. You could use a[:len(indices)], and that will be "in-place", though it looks like numpy makes a temporary copy internally to avoid conflicts when the input array and output array share memory (at least in the case of take()). I was expecting In [15]: a = arange(50).reshape(10, 5) In [16]: numpy.take(a,[2,0,1],axis=0, out=a[:3]) to place 3 copies of original row 2 in rows 0, 1 and 2. The fact that it doesn't seems to suggest NumPy is being more clever (but also more memory-hungry). David From cgohlke at uci.edu Sat Mar 12 14:49:35 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 12 Mar 2011 11:49:35 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local> Message-ID: <4D7BCE4F.7090404@uci.edu> On 3/12/2011 8:45 AM, Nadav Horesh wrote: > I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: > > 1. PythonMagick: Needs boost which I do not have it avaiable on python3 Boost works on Python 3.1. You might need to compile it. > 2. Pygame: I have the stable version(1.9.1) should it work? You need the developer version from svn. > 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. FreeImagePy is unmaintained, does not work on Python 3, and has problems on 64 bit platforms. Just wrap the functions you need in ctypes. > 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py Christoph > > nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install > /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' > warnings.warn(msg) > running install > running build > running build_py > running build_ext > -------------------------------------------------------------------- > PIL 1.1.7 SETUP SUMMARY > -------------------------------------------------------------------- > version 1.1.7 > platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) > [GCC 4.4.4] > -------------------------------------------------------------------- > --- TKINTER support available > --- JPEG support available > --- ZLIB (PNG/ZIP) support available > --- FREETYPE2 support available > --- LITTLECMS support available > > . > . > . > > byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc > Traceback (most recent call last): > File "setup.py", line 520, in > setup(*(), **configuration) # old school :-) > File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup > dist.run_commands() > File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands > self.run_command(cmd) > File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command > cmd_obj.run() > File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run > self.run_command(cmd_name) > File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command > cmd_obj.run() > File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run > self.byte_compile(outfiles) > File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile > dry_run=self.dry_run) > File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile > compile(file, cfile, dfile) > File "/usr/lib64/python3.1/py_compile.py", line 137, in compile > codestring = f.read() > File "/usr/lib64/python3.1/codecs.py", line 300, in decode > (result, consumed) = self._buffer_decode(data, self.errors, final) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte > > > > Any idea on how to correct it? Any elegant way to avoid byte compiling? > > Nadav > > > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] > Sent: 12 March 2011 14:35 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > Here's a ctypes interface to FreeImage that I wrote a while back and > was since cleaned up (and maintained) by the scikits.image folk: > > https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py > > If it doesn't work out of the box on python 3, then it should be > pretty simple to fix. > > Zach > > > > On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: > >> >> >> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>> Having numpy, scipy, and matplotlib working reasonably with >>> python3, a >>> major piece of code I miss for a major python3 migration is an >>> image IO. >>> I found that pylab's imread works fine for png image, but I need to >>> read >>> all the other image format as well as png and jpeg output. >>> Any hints (including advices how easyly construct my own module) are >>> appreciated. >>> Nadav. >>> >> >> On Windows, PIL (private port at >> ), PythonMagick >> , and pygame 1.9.2pre >> are working reasonably well for image IO. Also >> the FreeImage library is easy to >> use >> with ctypes. >> >> Christoph >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From nadavh at visionsense.com Sat Mar 12 15:15:54 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 12 Mar 2011 12:15:54 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5233@VA3DIAXVS361.RED001.local> It started to work after processing it with 2to3 and omitting the conversion of file names with the str function (I supply the file names as "bytes"). Issues: 1. It refuses to save in jpeg format 2. There is a worning of possible segfult on 64 bit machine (which is the target platform). I'll keep on test it. Thank you Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] Sent: 12 March 2011 14:35 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? Here's a ctypes interface to FreeImage that I wrote a while back and was since cleaned up (and maintained) by the scikits.image folk: https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py If it doesn't work out of the box on python 3, then it should be pretty simple to fix. Zach On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: > > > On 3/12/2011 1:08 AM, Nadav Horesh wrote: >> Having numpy, scipy, and matplotlib working reasonably with >> python3, a >> major piece of code I miss for a major python3 migration is an >> image IO. >> I found that pylab's imread works fine for png image, but I need to >> read >> all the other image format as well as png and jpeg output. >> Any hints (including advices how easyly construct my own module) are >> appreciated. >> Nadav. >> > > On Windows, PIL (private port at > ), PythonMagick > , and pygame 1.9.2pre > are working reasonably well for image IO. Also > the FreeImage library is easy to > use > with ctypes . > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From nadavh at visionsense.com Sat Mar 12 15:47:39 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 12 Mar 2011 12:47:39 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <4D7BCE4F.7090404@uci.edu> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local> After the replacement of ? with o, the installation went without errors, but: nadav at nadav_home ~ $ python3 Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) [GCC 4.4.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import _imaging Traceback (most recent call last): File "", line 1, in ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod Thank you, Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] Sent: 12 March 2011 21:49 To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? On 3/12/2011 8:45 AM, Nadav Horesh wrote: > I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: > > 1. PythonMagick: Needs boost which I do not have it avaiable on python3 Boost works on Python 3.1. You might need to compile it. > 2. Pygame: I have the stable version(1.9.1) should it work? You need the developer version from svn. > 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. FreeImagePy is unmaintained, does not work on Python 3, and has problems on 64 bit platforms. Just wrap the functions you need in ctypes. > 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py Christoph > > nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install > /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' > warnings.warn(msg) > running install > running build > running build_py > running build_ext > -------------------------------------------------------------------- > PIL 1.1.7 SETUP SUMMARY > -------------------------------------------------------------------- > version 1.1.7 > platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) > [GCC 4.4.4] > -------------------------------------------------------------------- > --- TKINTER support available > --- JPEG support available > --- ZLIB (PNG/ZIP) support available > --- FREETYPE2 support available > --- LITTLECMS support available > > . > . > . > > byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc > Traceback (most recent call last): > File "setup.py", line 520, in > setup(*(), **configuration) # old school :-) > File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup > dist.run_commands() > File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands > self.run_command(cmd) > File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command > cmd_obj.run() > File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run > self.run_command(cmd_name) > File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command > cmd_obj.run() > File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run > self.byte_compile(outfiles) > File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile > dry_run=self.dry_run) > File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile > compile(file, cfile, dfile) > File "/usr/lib64/python3.1/py_compile.py", line 137, in compile > codestring = f.read() > File "/usr/lib64/python3.1/codecs.py", line 300, in decode > (result, consumed) = self._buffer_decode(data, self.errors, final) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte > > > > Any idea on how to correct it? Any elegant way to avoid byte compiling? > > Nadav > > > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] > Sent: 12 March 2011 14:35 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > Here's a ctypes interface to FreeImage that I wrote a while back and > was since cleaned up (and maintained) by the scikits.image folk: > > https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py > > If it doesn't work out of the box on python 3, then it should be > pretty simple to fix. > > Zach > > > > On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: > >> >> >> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>> Having numpy, scipy, and matplotlib working reasonably with >>> python3, a >>> major piece of code I miss for a major python3 migration is an >>> image IO. >>> I found that pylab's imread works fine for png image, but I need to >>> read >>> all the other image format as well as png and jpeg output. >>> Any hints (including advices how easyly construct my own module) are >>> appreciated. >>> Nadav. >>> >> >> On Windows, PIL (private port at >> ), PythonMagick >> , and pygame 1.9.2pre >> are working reasonably well for image IO. Also >> the FreeImage library is easy to >> use >> with ctypes. >> >> Christoph >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From josh.holbrook at gmail.com Sat Mar 12 17:08:12 2011 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Sat, 12 Mar 2011 13:08:12 -0900 Subject: [Numpy-discussion] import error after compiling from HEAD Message-ID: Sup y'all, I just installed numpy from github, but when I tried to import it I got some odd errors! What's going on, and how can it be fixed? josh at pidgey:~$ python -c 'import numpy; numpy.test()' Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py", line 137, in File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in # File "/usr/local/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in File "/usr/local/lib/python2.6/dist-packages/numpy/core/__init__.py", line 10, in # ImportError: No module named _sort Thanks, --Josh From cgohlke at uci.edu Sat Mar 12 17:37:03 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 12 Mar 2011 14:37:03 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local> Message-ID: <4D7BF58F.4010005@uci.edu> On 3/12/2011 12:47 PM, Nadav Horesh wrote: > After the replacement of ? with o, the installation went without errors, but: > > nadav at nadav_home ~ $ python3 > Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) > [GCC 4.4.4] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import _imaging > Traceback (most recent call last): > File "", line 1, in > ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod Py_FindMethod should be excluded by `#ifndef PY3` or similar preprocessor statements. There is a typo in map.c line 65: change `#ifdef PY3` to `#ifndef PY3` and clean your build directory before rebuilding. Christoph > > Thank you, > > Nadav. > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] > Sent: 12 March 2011 21:49 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > On 3/12/2011 8:45 AM, Nadav Horesh wrote: >> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: >> >> 1. PythonMagick: Needs boost which I do not have it avaiable on python3 > Boost works on Python 3.1. You might need to compile it. >> 2. Pygame: I have the stable version(1.9.1) should it work? > You need the developer version from svn. >> 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. > FreeImagePy is unmaintained, does not work on Python 3, and has problems > on 64 bit platforms. Just wrap the functions you need in ctypes. >> 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: > Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py > > Christoph > >> >> nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install >> /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' >> warnings.warn(msg) >> running install >> running build >> running build_py >> running build_ext >> -------------------------------------------------------------------- >> PIL 1.1.7 SETUP SUMMARY >> -------------------------------------------------------------------- >> version 1.1.7 >> platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >> [GCC 4.4.4] >> -------------------------------------------------------------------- >> --- TKINTER support available >> --- JPEG support available >> --- ZLIB (PNG/ZIP) support available >> --- FREETYPE2 support available >> --- LITTLECMS support available >> >> . >> . >> . >> >> byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc >> Traceback (most recent call last): >> File "setup.py", line 520, in >> setup(*(), **configuration) # old school :-) >> File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup >> dist.run_commands() >> File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands >> self.run_command(cmd) >> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >> cmd_obj.run() >> File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run >> self.run_command(cmd_name) >> File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command >> self.distribution.run_command(command) >> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >> cmd_obj.run() >> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run >> self.byte_compile(outfiles) >> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile >> dry_run=self.dry_run) >> File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile >> compile(file, cfile, dfile) >> File "/usr/lib64/python3.1/py_compile.py", line 137, in compile >> codestring = f.read() >> File "/usr/lib64/python3.1/codecs.py", line 300, in decode >> (result, consumed) = self._buffer_decode(data, self.errors, final) >> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte >> >> >> >> Any idea on how to correct it? Any elegant way to avoid byte compiling? >> >> Nadav >> >> >> ________________________________________ >> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] >> Sent: 12 March 2011 14:35 >> To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >> >> Here's a ctypes interface to FreeImage that I wrote a while back and >> was since cleaned up (and maintained) by the scikits.image folk: >> >> https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >> >> If it doesn't work out of the box on python 3, then it should be >> pretty simple to fix. >> >> Zach >> >> >> >> On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: >> >>> >>> >>> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>>> Having numpy, scipy, and matplotlib working reasonably with >>>> python3, a >>>> major piece of code I miss for a major python3 migration is an >>>> image IO. >>>> I found that pylab's imread works fine for png image, but I need to >>>> read >>>> all the other image format as well as png and jpeg output. >>>> Any hints (including advices how easyly construct my own module) are >>>> appreciated. >>>> Nadav. >>>> >>> >>> On Windows, PIL (private port at >>> ), PythonMagick >>> , and pygame 1.9.2pre >>> are working reasonably well for image IO. Also >>> the FreeImage library is easy to >>> use >>> with ctypes. >>> >>> Christoph >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Sat Mar 12 21:32:06 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 12 Mar 2011 19:32:06 -0700 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default Message-ID: Hi All, I'd like to change the polynomial package to only import the Classes, leaving the large number of implementation functions to be imported directly from the different modules if needed. I always regarded those functions as implementation helpers and kept them separate from the class so that others could use them to build their own classes if they desired. For most purposes I think the classes are more useful. So I think it was a mistake to import the functions by; default and I'm looking for a graceful and acceptable way out. Any suggestions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Sun Mar 13 00:13:12 2011 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 12 Mar 2011 21:13:12 -0800 Subject: [Numpy-discussion] Call for GSoC 2011 NumPy mentors Message-ID: Hi, It is time to start preparing for the 2011 Google Summer of Code (SoC). As in the past, we will participate in SoC with the Python Software Foundation (PSF) as our mentoring organization. The PSF has requested that every project, which wishes to participate in the SoC, provide a list of at least *three* potential mentors. If you are interested and willing to potentially mentor someone this summer to work on NumPy, please send me the following information by Monday evening: Name, Email, Phone, and Link_ID. You can find additional information on the 2011 SoC homepage: http://socghop.appspot.com/ Here is the PSF SoC page: http://wiki.python.org/moin/SummerOfCode Please start thinking about potential projects and add them to the SoC ideas page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Thanks, Jarrod From wardefar at iro.umontreal.ca Sun Mar 13 00:57:02 2011 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Sun, 13 Mar 2011 00:57:02 -0500 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default In-Reply-To: References: Message-ID: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> On 2011-03-12, at 9:32 PM, Charles R Harris wrote: > I'd like to change the polynomial package to only import the Classes, leaving the large number of implementation functions to be imported directly from the different modules if needed. I always regarded those functions as implementation helpers and kept them separate from the class so that others could use them to build their own classes if they desired. For most purposes I think the classes are more useful. So I think it was a mistake to import the functions by; default and I'm looking for a graceful and acceptable way out. Any suggestions. I hope this wouldn't include polyfit, polyval, roots and vander at least (I'd also be -1 on removing poly{add,sub,mul,div,der,int}, but more weakly so). Those 4 seem useful and "basic" enough to leave in the default namespace. David From nadavh at visionsense.com Sun Mar 13 00:56:38 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 12 Mar 2011 21:56:38 -0800 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <4D7BF58F.4010005@uci.edu> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local>, <4D7BF58F.4010005@uci.edu> Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> This lead to another error probably due to line 68 in map.h. As much as I could trace it, ob_type is a member of PyObject, not of PyTypeObject. I have no clue how to resolve this. Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] Sent: 13 March 2011 00:37 To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? On 3/12/2011 12:47 PM, Nadav Horesh wrote: > After the replacement of ? with o, the installation went without errors, but: > > nadav at nadav_home ~ $ python3 > Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) > [GCC 4.4.4] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import _imaging > Traceback (most recent call last): > File "", line 1, in > ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod Py_FindMethod should be excluded by `#ifndef PY3` or similar preprocessor statements. There is a typo in map.c line 65: change `#ifdef PY3` to `#ifndef PY3` and clean your build directory before rebuilding. Christoph > > Thank you, > > Nadav. > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] > Sent: 12 March 2011 21:49 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > On 3/12/2011 8:45 AM, Nadav Horesh wrote: >> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: >> >> 1. PythonMagick: Needs boost which I do not have it avaiable on python3 > Boost works on Python 3.1. You might need to compile it. >> 2. Pygame: I have the stable version(1.9.1) should it work? > You need the developer version from svn. >> 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. > FreeImagePy is unmaintained, does not work on Python 3, and has problems > on 64 bit platforms. Just wrap the functions you need in ctypes. >> 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: > Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py > > Christoph > >> >> nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install >> /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' >> warnings.warn(msg) >> running install >> running build >> running build_py >> running build_ext >> -------------------------------------------------------------------- >> PIL 1.1.7 SETUP SUMMARY >> -------------------------------------------------------------------- >> version 1.1.7 >> platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >> [GCC 4.4.4] >> -------------------------------------------------------------------- >> --- TKINTER support available >> --- JPEG support available >> --- ZLIB (PNG/ZIP) support available >> --- FREETYPE2 support available >> --- LITTLECMS support available >> >> . >> . >> . >> >> byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc >> Traceback (most recent call last): >> File "setup.py", line 520, in >> setup(*(), **configuration) # old school :-) >> File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup >> dist.run_commands() >> File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands >> self.run_command(cmd) >> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >> cmd_obj.run() >> File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run >> self.run_command(cmd_name) >> File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command >> self.distribution.run_command(command) >> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >> cmd_obj.run() >> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run >> self.byte_compile(outfiles) >> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile >> dry_run=self.dry_run) >> File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile >> compile(file, cfile, dfile) >> File "/usr/lib64/python3.1/py_compile.py", line 137, in compile >> codestring = f.read() >> File "/usr/lib64/python3.1/codecs.py", line 300, in decode >> (result, consumed) = self._buffer_decode(data, self.errors, final) >> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte >> >> >> >> Any idea on how to correct it? Any elegant way to avoid byte compiling? >> >> Nadav >> >> >> ________________________________________ >> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] >> Sent: 12 March 2011 14:35 >> To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >> >> Here's a ctypes interface to FreeImage that I wrote a while back and >> was since cleaned up (and maintained) by the scikits.image folk: >> >> https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >> >> If it doesn't work out of the box on python 3, then it should be >> pretty simple to fix. >> >> Zach >> >> >> >> On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: >> >>> >>> >>> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>>> Having numpy, scipy, and matplotlib working reasonably with >>>> python3, a >>>> major piece of code I miss for a major python3 migration is an >>>> image IO. >>>> I found that pylab's imread works fine for png image, but I need to >>>> read >>>> all the other image format as well as png and jpeg output. >>>> Any hints (including advices how easyly construct my own module) are >>>> appreciated. >>>> Nadav. >>>> >>> >>> On Windows, PIL (private port at >>> ), PythonMagick >>> , and pygame 1.9.2pre >>> are working reasonably well for image IO. Also >>> the FreeImage library is easy to >>> use >>> with ctypes. >>> >>> Christoph >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at googlemail.com Sun Mar 13 05:22:48 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 13 Mar 2011 17:22:48 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes Message-ID: Hi all, On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the first beta. So please get your last commits for 1.6 in by Monday evening. Also, please review and add to the 1.6.0 release notes. I put in headers for several items that need a few lines in the notes, I hope this can be filled in by the authors of those features (Charles: Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of stuff). Cheers, Ralf From matthew.brett at gmail.com Sun Mar 13 05:57:24 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Mar 2011 01:57:24 -0800 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field Message-ID: Hi, I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy head (30ee1d352): $ python3.2 Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>> a['f1'] = 1 >>> a['f2'] = 1 Segmentation fault All tests pass with np.test() Expected behavior with same code on python2.6: >>> a['f2'] = 1 Traceback (most recent call last): File "", line 1, in ValueError: field named f2 not found. Cheers, Matthew From charlesr.harris at gmail.com Sun Mar 13 06:49:02 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Mar 2011 04:49:02 -0600 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default In-Reply-To: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> References: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> Message-ID: On Sat, Mar 12, 2011 at 10:57 PM, David Warde-Farley < wardefar at iro.umontreal.ca> wrote: > > On 2011-03-12, at 9:32 PM, Charles R Harris wrote: > > > I'd like to change the polynomial package to only import the Classes, > leaving the large number of implementation functions to be imported directly > from the different modules if needed. I always regarded those functions as > implementation helpers and kept them separate from the class so that others > could use them to build their own classes if they desired. For most purposes > I think the classes are more useful. So I think it was a mistake to import > the functions by; default and I'm looking for a graceful and acceptable way > out. Any suggestions. > > > I hope this wouldn't include polyfit, polyval, roots and vander at least > (I'd also be -1 on removing poly{add,sub,mul,div,der,int}, but more weakly > so). Those 4 seem useful and "basic" enough to leave in the default > namespace. > > I use *vander a lot myself, so I agree with you there. The rest can all be reached within the Polynomial class. Maybe I should add the *vander stuff to the classes also. But I guess I'm looking for feedback from people who might be using these functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Mar 13 07:01:28 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 13 Mar 2011 19:01:28 +0800 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default In-Reply-To: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> References: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> Message-ID: On Sun, Mar 13, 2011 at 1:57 PM, David Warde-Farley wrote: > > On 2011-03-12, at 9:32 PM, Charles R Harris wrote: > >> I'd like to change the polynomial package to only import the Classes, leaving the large number of implementation functions to be imported directly from the different modules if needed. I always regarded those functions as implementation helpers and kept them separate from the class so that others could use them to build their own classes if they desired. For most purposes I think the classes are more useful. So I think it was a mistake to import the functions by; default and I'm looking for a graceful and acceptable way out. Any suggestions. How about just deprecating in 1.6 and removing from the polynomial namespace in 2.0? Then you do need to add to the docs explicitly that the submodules contain these functions, that they are part of the public API and are not going anywhere. Those functions are unlikely to be heavily used already, so the impact of a deprecation will be small. > > > I hope this wouldn't include polyfit, polyval, roots and vander at least (I'd also be -1 on removing poly{add,sub,mul,div,der,int}, but more weakly so). Those 4 seem useful and "basic" enough to leave in the default namespace. Are you talking about these functions in the numpy or numpy.polynomial namespace? I'm guessing the former (since np.polynomial.roots does not exist), while Charles means the latter. Cheers, Ralf From ralf.gommers at googlemail.com Sun Mar 13 07:14:13 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 13 Mar 2011 19:14:13 +0800 Subject: [Numpy-discussion] import error after compiling from HEAD In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 6:08 AM, Joshua Holbrook wrote: > Sup y'all, > > I just installed numpy from github, but when I tried to import it I > got some odd errors! What's going on, and how can it be fixed? Are you in the numpy source tree when trying that command? If not, try to clean ("git clean -xdf") and rebuild. If then you still get the same issue, give us the build command you used, compiler versions, OS, etc. Cheers, Ralf > > josh at pidgey:~$ python -c 'import numpy; numpy.test()' > Traceback (most recent call last): > ?File "", line 1, in > ?File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py", > line 137, in > > ?File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py", > line 9, in > ? ?from numpy.lib import add_newdoc > ?File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py", > line 4, in > ? ?# > ?File "/usr/local/lib/python2.6/dist-packages/numpy/lib/type_check.py", > line 8, in > ?File "/usr/local/lib/python2.6/dist-packages/numpy/core/__init__.py", > line 10, in > ? ?# > ImportError: No module named _sort > > > Thanks, > > --Josh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pearu.peterson at gmail.com Sun Mar 13 07:44:22 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Sun, 13 Mar 2011 13:44:22 +0200 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 11:22 AM, Ralf Gommers wrote: > Hi all, > > On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the > first beta. So please get your last commits for 1.6 in by Monday > evening. > > Also, please review and add to the 1.6.0 release notes. I put in > headers for several items that need a few lines in the notes, I hope > this can be filled in by the authors of those features (Charles: > Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of > stuff). > Done for assumed shape arrays and size function supports. Best regards, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From magawake at gmail.com Sun Mar 13 09:03:22 2011 From: magawake at gmail.com (Mag Gam) Date: Sun, 13 Mar 2011 09:03:22 -0400 Subject: [Numpy-discussion] Benefit of Intel C compiler Message-ID: Planning to compile Numpy with Intel C compiler (http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974). I was wondering if there was a benefit. From stefan at sun.ac.za Sun Mar 13 11:19:43 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 13 Mar 2011 17:19:43 +0200 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> Message-ID: On Sat, Mar 12, 2011 at 2:35 PM, Zachary Pincus wrote: > Here's a ctypes interface to FreeImage that I wrote a while back and > was since cleaned up (and maintained) by the scikits.image folk: > https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py > > If it doesn't work out of the box on python 3, then it should be > pretty simple to fix. I now fixed the scikits.image build process under Python 3, so this should be easy to try out. Regards St?fan From cgohlke at uci.edu Sun Mar 13 12:28:30 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 13 Mar 2011 09:28:30 -0700 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local>, <4D7BF58F.4010005@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> Message-ID: <4D7CF0AE.2000007@uci.edu> On 3/12/2011 9:56 PM, Nadav Horesh wrote: > This lead to another error probably due to line 68 in map.h. As much as I could trace it, ob_type is a member of PyObject, not of PyTypeObject. I have no clue how to resolve this. > > Nadav. Line 68 in map.c is excluded by a preprocessor statement. There is likely another use of ob_type in a code that is not used on Windows. I have not ported _sane.c for example. I'll try to get access to an Ubuntu system and update the source. Christoph > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] > Sent: 13 March 2011 00:37 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > On 3/12/2011 12:47 PM, Nadav Horesh wrote: >> After the replacement of ? with o, the installation went without errors, but: >> >> nadav at nadav_home ~ $ python3 >> Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >> [GCC 4.4.4] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import _imaging >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod > > Py_FindMethod should be excluded by `#ifndef PY3` or similar > preprocessor statements. There is a typo in map.c line 65: change > `#ifdef PY3` to `#ifndef PY3` and clean your build directory before > rebuilding. > > Christoph > >> >> Thank you, >> >> Nadav. >> ________________________________________ >> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] >> Sent: 12 March 2011 21:49 >> To: numpy-discussion at scipy.org >> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >> >> On 3/12/2011 8:45 AM, Nadav Horesh wrote: >>> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: >>> >>> 1. PythonMagick: Needs boost which I do not have it avaiable on python3 >> Boost works on Python 3.1. You might need to compile it. >>> 2. Pygame: I have the stable version(1.9.1) should it work? >> You need the developer version from svn. >>> 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. >> FreeImagePy is unmaintained, does not work on Python 3, and has problems >> on 64 bit platforms. Just wrap the functions you need in ctypes. >>> 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: >> Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py >> >> Christoph >> >>> >>> nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install >>> /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' >>> warnings.warn(msg) >>> running install >>> running build >>> running build_py >>> running build_ext >>> -------------------------------------------------------------------- >>> PIL 1.1.7 SETUP SUMMARY >>> -------------------------------------------------------------------- >>> version 1.1.7 >>> platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >>> [GCC 4.4.4] >>> -------------------------------------------------------------------- >>> --- TKINTER support available >>> --- JPEG support available >>> --- ZLIB (PNG/ZIP) support available >>> --- FREETYPE2 support available >>> --- LITTLECMS support available >>> >>> . >>> . >>> . >>> >>> byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc >>> Traceback (most recent call last): >>> File "setup.py", line 520, in >>> setup(*(), **configuration) # old school :-) >>> File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup >>> dist.run_commands() >>> File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands >>> self.run_command(cmd) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run >>> self.run_command(cmd_name) >>> File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command >>> self.distribution.run_command(command) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run >>> self.byte_compile(outfiles) >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile >>> dry_run=self.dry_run) >>> File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile >>> compile(file, cfile, dfile) >>> File "/usr/lib64/python3.1/py_compile.py", line 137, in compile >>> codestring = f.read() >>> File "/usr/lib64/python3.1/codecs.py", line 300, in decode >>> (result, consumed) = self._buffer_decode(data, self.errors, final) >>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte >>> >>> >>> >>> Any idea on how to correct it? Any elegant way to avoid byte compiling? >>> >>> Nadav >>> >>> >>> ________________________________________ >>> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] >>> Sent: 12 March 2011 14:35 >>> To: Discussion of Numerical Python >>> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >>> >>> Here's a ctypes interface to FreeImage that I wrote a while back and >>> was since cleaned up (and maintained) by the scikits.image folk: >>> >>> https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >>> >>> If it doesn't work out of the box on python 3, then it should be >>> pretty simple to fix. >>> >>> Zach >>> >>> >>> >>> On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: >>> >>>> >>>> >>>> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>>>> Having numpy, scipy, and matplotlib working reasonably with >>>>> python3, a >>>>> major piece of code I miss for a major python3 migration is an >>>>> image IO. >>>>> I found that pylab's imread works fine for png image, but I need to >>>>> read >>>>> all the other image format as well as png and jpeg output. >>>>> Any hints (including advices how easyly construct my own module) are >>>>> appreciated. >>>>> Nadav. >>>>> >>>> >>>> On Windows, PIL (private port at >>>> ), PythonMagick >>>> , and pygame 1.9.2pre >>>> are working reasonably well for image IO. Also >>>> the FreeImage library is easy to >>>> use >>>> with ctypes. >>>> >>>> Christoph >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From cgohlke at uci.edu Sun Mar 13 12:54:10 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 13 Mar 2011 09:54:10 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: Message-ID: <4D7CF6B2.7070202@uci.edu> On 3/13/2011 1:57 AM, Matthew Brett wrote: > Hi, > > I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy > head (30ee1d352): > > $ python3.2 > Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) > [GCC 4.2.1 (Apple Inc. build 5664)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy as np >>>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>>> a['f1'] = 1 >>>> a['f2'] = 1 > Segmentation fault > > All tests pass with np.test() > > Expected behavior with same code on python2.6: > >>>> a['f2'] = 1 > Traceback (most recent call last): > File "", line 1, in > ValueError: field named f2 not found. > > Cheers, > > Matthew Confirmed on Windows. The crash is in line 816 of mapping.c: PyErr_Format(PyExc_ValueError, "field named %s not found.", PyString_AsString(index)); This works with Python 3.x: PyErr_Format(PyExc_ValueError, "field named %S not found.", index); Christoph From nadavh at visionsense.com Sun Mar 13 14:20:59 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 13 Mar 2011 11:20:59 -0700 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> , Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D523B@VA3DIAXVS361.RED001.local> Jest tested the installation (after git clone ...). I had to correct the following lines in _build.py to pass installation: lines 72, and 75 should be: f0 = open(f0,'br') f1 = open(f1,'br') Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of St?fan van der Walt [stefan at sun.ac.za] Sent: 13 March 2011 17:19 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? On Sat, Mar 12, 2011 at 2:35 PM, Zachary Pincus wrote: > Here's a ctypes interface to FreeImage that I wrote a while back and > was since cleaned up (and maintained) by the scikits.image folk: > https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py > > If it doesn't work out of the box on python 3, then it should be > pretty simple to fix. I now fixed the scikits.image build process under Python 3, so this should be easy to try out. Regards St?fan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Sun Mar 13 14:29:29 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Mar 2011 11:29:29 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: <4D7CF6B2.7070202@uci.edu> References: <4D7CF6B2.7070202@uci.edu> Message-ID: Hi On Sun, Mar 13, 2011 at 9:54 AM, Christoph Gohlke wrote: > On 3/13/2011 1:57 AM, Matthew Brett wrote: >> Hi, >> I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy >> head (30ee1d352): >> >> $ python3.2 >> Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) >> [GCC 4.2.1 (Apple Inc. build 5664)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy as np >>>>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>>>> a['f1'] = 1 >>>>> a['f2'] = 1 >> Segmentation fault >> >> All tests pass with np.test() >> >> Expected behavior with same code on python2.6: >> >>>>> a['f2'] = 1 >> Traceback (most recent call last): >> ? ?File "", line 1, in >> ValueError: field named f2 not found. >> >> Cheers, >> >> Matthew > > Confirmed on Windows. The crash is in line 816 of mapping.c: > > ? ? ? ? PyErr_Format(PyExc_ValueError, > ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", > ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(index)); > > This works with Python 3.x: > > ? ? ? ? PyErr_Format(PyExc_ValueError, > ? ? ? ? ? ? ? ? ? ? ?"field named %S not found.", > ? ? ? ? ? ? ? ? ? ? ?index); Sure enough, in python3.2: >>> a[b'f2'] = 1 Traceback (most recent call last): File "", line 1, in ValueError: field named f2 not found. That is - it is specifically passing a python 3 string that causes the segmentation fault. Is this a release blocker? I'm afraid I don't know the code well enough to be confident of the right fix. Is there something I can do to help? Cheers, Matthew From cgohlke at uci.edu Sun Mar 13 14:51:07 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 13 Mar 2011 11:51:07 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> Message-ID: <4D7D121B.9060902@uci.edu> On 3/13/2011 11:29 AM, Matthew Brett wrote: > Hi > > On Sun, Mar 13, 2011 at 9:54 AM, Christoph Gohlke wrote: >> On 3/13/2011 1:57 AM, Matthew Brett wrote: >>> Hi, >>> I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy >>> head (30ee1d352): >>> >>> $ python3.2 >>> Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) >>> [GCC 4.2.1 (Apple Inc. build 5664)] on darwin >>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> import numpy as np >>>>>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>>>>> a['f1'] = 1 >>>>>> a['f2'] = 1 >>> Segmentation fault >>> >>> All tests pass with np.test() >>> >>> Expected behavior with same code on python2.6: >>> >>>>>> a['f2'] = 1 >>> Traceback (most recent call last): >>> File "", line 1, in >>> ValueError: field named f2 not found. >>> >>> Cheers, >>> >>> Matthew >> >> Confirmed on Windows. The crash is in line 816 of mapping.c: >> >> PyErr_Format(PyExc_ValueError, >> "field named %s not found.", >> PyString_AsString(index)); >> >> This works with Python 3.x: >> >> PyErr_Format(PyExc_ValueError, >> "field named %S not found.", >> index); > > Sure enough, in python3.2: > >>>> a[b'f2'] = 1 > Traceback (most recent call last): > File "", line 1, in > ValueError: field named f2 not found. > > That is - it is specifically passing a python 3 string that causes the > segmentation fault. > > Is this a release blocker? I'm afraid I don't know the code well > enough to be confident of the right fix. Is there something I can do > to help? Please open a ticket at and refer to this discussion and proposed fix. diff --git a/numpy/core/src/multiarray/mapping.c b/numpy/core/src/multiarray/mapping.c index 8db85bf..3a72811 100644 --- a/numpy/core/src/multiarray/mapping.c +++ b/numpy/core/src/multiarray/mapping.c @@ -812,10 +812,16 @@ array_ass_sub(PyArrayObject *self, PyObject *index, PyObject *op) } } } - +#if defined(NPY_PY3K) + PyErr_Format(PyExc_ValueError, + "field named %S not found.", + index); +#else PyErr_Format(PyExc_ValueError, "field named %s not found.", PyString_AsString(index)); +#endif + return -1; } Christoph > > Cheers, > > Matthew > > From matthew.brett at gmail.com Sun Mar 13 15:07:56 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Mar 2011 12:07:56 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Hi, On Sun, Mar 13, 2011 at 11:51 AM, Christoph Gohlke wrote: > > > On 3/13/2011 11:29 AM, Matthew Brett wrote: >> >> Hi >> >> On Sun, Mar 13, 2011 at 9:54 AM, Christoph Gohlke ?wrote: >>> >>> On 3/13/2011 1:57 AM, Matthew Brett wrote: >>>> >>>> Hi, >>>> I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy >>>> head (30ee1d352): >>>> >>>> $ python3.2 >>>> Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) >>>> [GCC 4.2.1 (Apple Inc. build 5664)] on darwin >>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>>> >>>>>>> import numpy as np >>>>>>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>>>>>> a['f1'] = 1 >>>>>>> a['f2'] = 1 >>>> >>>> Segmentation fault >>>> >>>> All tests pass with np.test() >>>> >>>> Expected behavior with same code on python2.6: >>>> >>>>>>> a['f2'] = 1 >>>> >>>> Traceback (most recent call last): >>>> ? ?File "", line 1, in >>>> ValueError: field named f2 not found. >>>> >>>> Cheers, >>>> >>>> Matthew >>> >>> Confirmed on Windows. The crash is in line 816 of mapping.c: >>> >>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>> ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", >>> ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(index)); >>> >>> This works with Python 3.x: >>> >>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>> ? ? ? ? ? ? ? ? ? ? ?"field named %S not found.", >>> ? ? ? ? ? ? ? ? ? ? ?index); >> >> Sure enough, in python3.2: >> >>>>> a[b'f2'] = 1 >> >> Traceback (most recent call last): >> ? File "", line 1, in >> ValueError: field named f2 not found. >> >> That is - it is specifically passing a python 3 string that causes the >> segmentation fault. >> >> Is this a release blocker? ?I'm afraid I don't know the code well >> enough to be confident of the right fix. ?Is there something I can do >> to help? > > Please open a ticket at and refer to this > discussion and proposed fix. > > diff --git a/numpy/core/src/multiarray/mapping.c > b/numpy/core/src/multiarray/mapping.c > index 8db85bf..3a72811 100644 > --- a/numpy/core/src/multiarray/mapping.c > +++ b/numpy/core/src/multiarray/mapping.c > @@ -812,10 +812,16 @@ array_ass_sub(PyArrayObject *self, PyObject *index, > PyObject *op) > ? ? ? ? ? ? ? ? } > ? ? ? ? ? ? } > ? ? ? ? } > - > +#if defined(NPY_PY3K) > + ? ? ? ?PyErr_Format(PyExc_ValueError, > + ? ? ? ? ? ? ? ? ? ? "field named %S not found.", > + ? ? ? ? ? ? ? ? ? ? index); > +#else > ? ? ? ? PyErr_Format(PyExc_ValueError, > ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", > ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(index)); > +#endif > + > ? ? ? ? return -1; > ? ? } http://projects.scipy.org/numpy/ticket/1770 Thanks, Matthew From charlesr.harris at gmail.com Sun Mar 13 15:11:38 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Mar 2011 13:11:38 -0600 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: <4D7D121B.9060902@uci.edu> References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: On Sun, Mar 13, 2011 at 12:51 PM, Christoph Gohlke wrote: > > > On 3/13/2011 11:29 AM, Matthew Brett wrote: > > Hi > > > > On Sun, Mar 13, 2011 at 9:54 AM, Christoph Gohlke > wrote: > >> On 3/13/2011 1:57 AM, Matthew Brett wrote: > >>> Hi, > >>> I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy > >>> head (30ee1d352): > >>> > >>> $ python3.2 > >>> Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) > >>> [GCC 4.2.1 (Apple Inc. build 5664)] on darwin > >>> Type "help", "copyright", "credits" or "license" for more information. > >>>>>> import numpy as np > >>>>>> a = np.zeros((1,), dtype=[('f1', 'f')]) > >>>>>> a['f1'] = 1 > >>>>>> a['f2'] = 1 > >>> Segmentation fault > >>> > >>> All tests pass with np.test() > >>> > >>> Expected behavior with same code on python2.6: > >>> > >>>>>> a['f2'] = 1 > >>> Traceback (most recent call last): > >>> File "", line 1, in > >>> ValueError: field named f2 not found. > >>> > >>> Cheers, > >>> > >>> Matthew > >> > >> Confirmed on Windows. The crash is in line 816 of mapping.c: > >> > >> PyErr_Format(PyExc_ValueError, > >> "field named %s not found.", > >> PyString_AsString(index)); > >> > >> This works with Python 3.x: > >> > >> PyErr_Format(PyExc_ValueError, > >> "field named %S not found.", > >> index); > > > > Sure enough, in python3.2: > > > >>>> a[b'f2'] = 1 > > Traceback (most recent call last): > > File "", line 1, in > > ValueError: field named f2 not found. > > > > That is - it is specifically passing a python 3 string that causes the > > segmentation fault. > > > > Is this a release blocker? I'm afraid I don't know the code well > > enough to be confident of the right fix. Is there something I can do > > to help? > > Please open a ticket at and refer to > this discussion and proposed fix. > > diff --git a/numpy/core/src/multiarray/mapping.c > b/numpy/core/src/multiarray/mapping.c > index 8db85bf..3a72811 100644 > --- a/numpy/core/src/multiarray/mapping.c > +++ b/numpy/core/src/multiarray/mapping.c > @@ -812,10 +812,16 @@ array_ass_sub(PyArrayObject *self, PyObject > *index, PyObject *op) > } > } > } > - > +#if defined(NPY_PY3K) > + PyErr_Format(PyExc_ValueError, > + "field named %S not found.", > + index); > +#else > PyErr_Format(PyExc_ValueError, > "field named %s not found.", > PyString_AsString(index)); > +#endif > + > return -1; > } > > > Christoph > > There are two more occurrences of this construction in descriptor.c. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Sun Mar 13 17:24:01 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Sun, 13 Mar 2011 14:24:01 -0700 Subject: [Numpy-discussion] ndarray special methods should raise NotImplemented instead of TypeError Message-ID: Apologies for cross posting -- I posted this to the scipy developer list a few days ago, since it's really a development question, but after getting no response I reread my question and realized it was a bit unclear and also was more appropriate for numpy. So here goes. Currently, the numerical special methods of the ndarray class, like __add__, __gt__, __mul__, etc., raise a TypeError when they encounter a class or type they don't understand. I do not think this is the correct behavior. Instead, they should raise a NotImplemented error. Here's the logic behind this. Classes can also define right operators that correspond to each of the above methods -- e.g. __radd__, __le__, __rmul__. However, these are only called under three conditions (see http://docs.python.org/reference/datamodel.html#object.__radd__): 1. When the left object doesn't implement the left operator. 2. When the left object raises a NotImplemented exception. 3. When the right object is a subclass of the left object. ?In this case, the left operator method is never called. In my use case, I've been working on implementing a matrix-like class of symbolic objects that have to interact seamlessly with numpy arrays. ?It's a wrapper for CPlex's Concert library, which effectively presents variables in an optimization problem as symbolic C++ objects that can be used to build the model. ?I'm trying to wrap this to work with numpy (I'd be happy to share my code when it's done) and the current behavior is causing a bit of a headache. Consider working with A + X, where A is an ndarray and X is an instance of my class that represents my symbolic objects, and let's suppose my matrix class implements __radd__ sensibly. ?Obviously, 1. doesn't apply here. ?If I go with 2., a TypeError is raised, which blows things apart. ?So the only solution I have is 3. However, it isn't possible for me to use the underlying array machinery provided by ndarray, as the python class simply wraps other C++ containers. Thus, if I subclass ndarray, the underlying array is meaningless -- while I do implement slicing and various other array-like methods, the only reason I'm subclassing ndarray is (3). The big problem with this is that many of the attributes and methods of ndarray don't make sense for my class, so I have to cover them up with dummy methods that simply raise exceptions. ?Doable, but far from ideal. For now, I'm doing (3) to get it to work, but it would be really nice if ndarray would raise a NotImplemented error instead of a TypeError. Is there any reason / use case justifying the current behavior? Would it be possible/easy to change it? Thanks! -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From charlesr.harris at gmail.com Sun Mar 13 17:45:01 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Mar 2011 15:45:01 -0600 Subject: [Numpy-discussion] ndarray special methods should raise NotImplemented instead of TypeError In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 3:24 PM, Hoyt Koepke wrote: > Apologies for cross posting -- I posted this to the scipy developer > list a few days ago, since it's really a development question, but > after getting no response I reread my question and realized it was a > bit unclear and also was more appropriate for numpy. So here goes. > > Currently, the numerical special methods of the ndarray class, like > __add__, __gt__, __mul__, etc., raise a TypeError when they encounter > a class or type they don't understand. I do not think this is the > correct behavior. Instead, they should raise a NotImplemented error. > > Here's the logic behind this. Classes can also define right operators > that correspond to each of the above methods -- e.g. __radd__, __le__, > __rmul__. However, these are only called under three conditions (see > http://docs.python.org/reference/datamodel.html#object.__radd__): > > 1. When the left object doesn't implement the left operator. > 2. When the left object raises a NotImplemented exception. > 3. When the right object is a subclass of the left object. In this > case, the left operator method is never called. > > In my use case, I've been working on implementing a matrix-like class > of symbolic objects that have to interact seamlessly with numpy > arrays. It's a wrapper for CPlex's Concert library, which effectively > presents variables in an optimization problem as symbolic C++ objects > that can be used to build the model. I'm trying to wrap this to work > with numpy (I'd be happy to share my code when it's done) and the > current behavior is causing a bit of a headache. > > Consider working with A + X, where A is an ndarray and X is an > instance of my class that represents my symbolic objects, and let's > suppose my matrix class implements __radd__ sensibly. Obviously, 1. > doesn't apply here. If I go with 2., a TypeError is raised, which > blows things apart. So the only solution I have is 3. > > However, it isn't possible for me to use the underlying array > machinery provided by ndarray, as the python class simply wraps other > C++ containers. Thus, if I subclass ndarray, the underlying array is > meaningless -- while I do implement slicing and various other > array-like methods, the only reason I'm subclassing ndarray is (3). > The big problem with this is that many of the attributes and methods > of ndarray don't make sense for my class, so I have to cover them up > with dummy methods that simply raise exceptions. Doable, but far from > ideal. > > For now, I'm doing (3) to get it to work, but it would be really nice > if ndarray would raise a NotImplemented error instead of a TypeError. > Is there any reason / use case justifying the current behavior? Would > it be possible/easy to change it? > > You can also define __array_priority__ in your class. If your class isn't a subclass, then numpy will always call your class methods whether or not your class is on the right or left. Sounds like you don't want to sublass ndarray here so that should work. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Sun Mar 13 17:53:55 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Sun, 13 Mar 2011 14:53:55 -0700 Subject: [Numpy-discussion] ndarray special methods should raise NotImplemented instead of TypeError In-Reply-To: References: Message-ID: > You can also define __array_priority__? in your class. If your class isn't a > subclass, then numpy will always call your class methods whether or not your > class is on the right or left. Sounds like you don't want to sublass ndarray > here so that should work. Excellent! This sounds like exactly what I was hoping for. I assume in my case, I just need __array_priority__ > 10 to gain priority over a matrix (from reading http://docs.scipy.org/doc/numpy/reference/arrays.classes.html). Is that correct? Thanks! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan at sun.ac.za Sun Mar 13 18:16:12 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 14 Mar 2011 00:16:12 +0200 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D523B@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D523B@VA3DIAXVS361.RED001.local> Message-ID: Hi Nadav On Sun, Mar 13, 2011 at 8:20 PM, Nadav Horesh wrote: > Jest tested the installation (after git clone ...). I had to correct the following lines in _build.py to pass installation: > lines 72, and 75 should be: > ? ?f0 = open(f0,'br') > ? ?f1 = open(f1,'br') Thanks so much for testing and for the patch; I've pushed your changes: https://github.com/stefanv/scikits.image/commit/b47ae98ffb92e2de33d9e530201e402e04d865d3 Are you able to load images now? Cheers St?fan From cgohlke at uci.edu Sun Mar 13 18:16:28 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 13 Mar 2011 15:16:28 -0700 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> Message-ID: <4D7D423C.7040904@uci.edu> Hi St?fan, I am sending you this off list. I just tried scikits.image from git on Windows. The attached patch fixes some minor build and test issues. Scikits.image.io occasionally seems to shadow Python's io module such that `import gzip` fails. There are 5 test failures like this: ====================================================================== ERROR: test_selem.TestSElem.test_selem_disk ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 187, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scikits\image\morphology\tests\test_selem.py", line 39, in test_selem_disk self.strel_worker("disk-matlab-output.npy", selem.disk) File "X:\Python27\lib\site-packages\scikits\image\morphology\tests\test_selem.py", line 29, in strel_worker matlab_masks = np.load(os.path.join(data_dir, fn)) File "X:\Python27\lib\site-packages\numpy\lib\npyio.py", line 316, in load import gzip File "X:\Python27\lib\gzip.py", line 36, in class GzipFile(io.BufferedIOBase): AttributeError: 'module' object has no attribute 'BufferedIOBase' Christoph On 3/13/2011 8:19 AM, St?fan van der Walt wrote: > On Sat, Mar 12, 2011 at 2:35 PM, Zachary Pincus wrote: >> Here's a ctypes interface to FreeImage that I wrote a while back and >> was since cleaned up (and maintained) by the scikits.image folk: >> > https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >> >> If it doesn't work out of the box on python 3, then it should be >> pretty simple to fix. > > I now fixed the scikits.image build process under Python 3, so this > should be easy to try out. > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ski.diff URL: From charlesr.harris at gmail.com Sun Mar 13 18:17:23 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Mar 2011 16:17:23 -0600 Subject: [Numpy-discussion] ndarray special methods should raise NotImplemented instead of TypeError In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 3:53 PM, Hoyt Koepke wrote: > > You can also define __array_priority__ in your class. If your class > isn't a > > subclass, then numpy will always call your class methods whether or not > your > > class is on the right or left. Sounds like you don't want to sublass > ndarray > > here so that should work. > > Excellent! This sounds like exactly what I was hoping for. I assume > in my case, I just need __array_priority__ > 10 to gain priority over > a matrix (from reading > http://docs.scipy.org/doc/numpy/reference/arrays.classes.html). Is > that correct? > > If you aren't a subclass you just need to define it. I don't know if that is documented anywhere outside the C source though. I just set it to zero in the polynomial template in numpy.polynomial.polytemplate.py Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Mar 13 18:28:00 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 14 Mar 2011 00:28:00 +0200 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <4D7D423C.7040904@uci.edu> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> <4D7D423C.7040904@uci.edu> Message-ID: Hi Christoph On Mon, Mar 14, 2011 at 12:16 AM, Christoph Gohlke wrote: > I just tried scikits.image from git on Windows. The attached patch fixes > some minor build and test issues. Great, thanks so much! Pushed into the repo. > Scikits.image.io occasionally seems to shadow Python's io module such that > `import gzip` fails. There are 5 test failures like this: Hrm, that's not good; any idea why that happens? Cheers St?fan From mwwiebe at gmail.com Sun Mar 13 22:23:20 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 13 Mar 2011 19:23:20 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers wrote: > Hi all, > > On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the > first beta. So please get your last commits for 1.6 in by Monday > evening. > > Also, please review and add to the 1.6.0 release notes. I put in > headers for several items that need a few lines in the notes, I hope > this can be filled in by the authors of those features (Charles: > Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of > stuff). > I've added a few more things, and made a small change to the iterator construction API that I've discovered is useful, but would be more difficult to do later. The Python exposure of the iterator is renamed from 'newiter' to 'nditer', is that a reasonable name or does anyone have a better suggestion? -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Mar 13 22:40:31 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 13 Mar 2011 21:40:31 -0500 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 21:23, Mark Wiebe wrote: > On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers > wrote: >> >> Hi all, >> >> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >> first beta. So please get your last commits for 1.6 in by Monday >> evening. >> >> Also, please review and add to the 1.6.0 release notes. I put in >> headers for several items that need a few lines in the notes, I hope >> this can be filled in by the authors of those features (Charles: >> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >> stuff). > > I've added a few more things, and made a small change to the iterator > construction API that I've discovered is useful, but would be more difficult > to do later. The Python exposure of the iterator is renamed from 'newiter' > to 'nditer', is that a reasonable name or does anyone have a better > suggestion? +1. As a general rule, never name anything "new". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Sun Mar 13 22:47:16 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Mar 2011 20:47:16 -0600 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: > On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers > wrote: > >> Hi all, >> >> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >> first beta. So please get your last commits for 1.6 in by Monday >> evening. >> >> Also, please review and add to the 1.6.0 release notes. I put in >> headers for several items that need a few lines in the notes, I hope >> this can be filled in by the authors of those features (Charles: >> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >> stuff). >> > > I've added a few more things, and made a small change to the iterator > construction API that I've discovered is useful, but would be more difficult > to do later. The Python exposure of the iterator is renamed from 'newiter' > to 'nditer', is that a reasonable name or does anyone have a better > suggestion? > > I think nditer is fine, certainly better than newiter. I don't see where nditer appears in the changes though, the test still uses newiter. On the arr_ravel_coords name and the python exposure of same, I've been thinking ravel_fancyindex might be more suggestive than ravel_coords. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Mon Mar 14 02:22:12 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 13 Mar 2011 23:22:12 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris wrote: > On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: > >> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> Hi all, >>> >>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >>> first beta. So please get your last commits for 1.6 in by Monday >>> evening. >>> >>> Also, please review and add to the 1.6.0 release notes. I put in >>> headers for several items that need a few lines in the notes, I hope >>> this can be filled in by the authors of those features (Charles: >>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >>> stuff). >>> >> >> I've added a few more things, and made a small change to the iterator >> construction API that I've discovered is useful, but would be more difficult >> to do later. The Python exposure of the iterator is renamed from 'newiter' >> to 'nditer', is that a reasonable name or does anyone have a better >> suggestion? >> >> > I think nditer is fine, certainly better than newiter. I don't see where > nditer appears in the changes though, the test still uses newiter. > I didn't rename the files, I can do that too. > > On the arr_ravel_coords name and the python exposure of same, I've been > thinking ravel_fancyindex might be more suggestive than ravel_coords. > Hmm, that doesn't seem quite right to me, it implies something fancier about it than I think it actually is. Maybe the ideal would be to have it be ravel_index/unravel_flatindex, but the unravel_index function already existed as precedent. On the other hand, in lots of contexts just saying "index" sounds like it should be one number, not a tuple, which is why in the iterator API the "index" usage refers to a C or Fortran-order flat index. Of the options we've considered so far, probably ravel_multiindex is my favorite, though nothing has really stood out. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Mar 14 02:59:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 14 Mar 2011 14:59:39 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris > wrote: >> >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: >>> >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers >>> wrote: >>>> >>>> Hi all, >>>> >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >>>> first beta. So please get your last commits for 1.6 in by Monday >>>> evening. >>>> >>>> Also, please review and add to the 1.6.0 release notes. I put in >>>> headers for several items that need a few lines in the notes, I hope >>>> this can be filled in by the authors of those features (Charles: >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >>>> stuff). >>> >>> I've added a few more things, and made a small change to the iterator >>> construction API that I've discovered is useful, but would be more difficult >>> to do later. The Python exposure of the iterator is renamed from 'newiter' >>> to 'nditer', is that a reasonable name or does anyone have a better >>> suggestion? >> >> I think nditer is fine, certainly better than newiter. I don't see where >> nditer appears in the changes though, the test still uses newiter. > > I didn't rename the files, I can do that too. Hi Mark, I see you just did this, but is there anything else you want/need to do? If it's necessary I can postpone the first beta by a couple of days. Better that than rush things too much and end up with an API you have reservations about. Cheers, Ralf >> On the arr_ravel_coords name and the python exposure of same, I've been >> thinking ravel_fancyindex might be more suggestive than ravel_coords. > > Hmm, that doesn't seem quite right to me, it implies something fancier about > it than I think it actually is. Maybe the ideal would be to have it be > ravel_index/unravel_flatindex, but the unravel_index function already > existed as precedent. On the other hand, in lots of contexts just saying > "index" sounds like it should be one number, not a tuple, which is why in > the iterator API the "index" usage refers to a C or Fortran-order flat > index. Of the options we've considered so far, probably ravel_multiindex is > my favorite, though nothing has really stood out. > -Mark > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From nadavh at visionsense.com Mon Mar 14 03:55:57 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 14 Mar 2011 00:55:57 -0700 Subject: [Numpy-discussion] =?utf-8?b?16rXqdeV15HXlDogIFtPVF0gYW55IGltYWdl?= =?utf-8?q?_io_module_that_works_with=09python3=3F?= In-Reply-To: Message-ID: <26FC23E7C398A64083C980D16001012D1CE08CB352@VA3DIAXVS361.RED001.local> The instillation is OK. The problem is that on my wok PC I do not have PIL installed. So: In [6]: import scikits.image.io as io --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/nadav/ in () ----> 1 import scikits.image.io as io /usr/lib64/python3.1/site-packages/scikits.image-0.3dev-py3.1-linux-x86_64.egg/scikits/image/io/__init__.py in () 11 # Add this plugin so that we can read images by default 12 use_plugin('null') ---> 13 use_plugin('pil') 14 15 from .sift import * /usr/lib64/python3.1/site-packages/scikits.image-0.3dev-py3.1-linux-x86_64.egg/scikits/image/io/_plugins/plugin.py in use(name, kind) 122 123 if not name in available(loaded=True): --> 124 _load(name) 125 126 for k in kind: /usr/lib64/python3.1/site-packages/scikits.image-0.3dev-py3.1-linux-x86_64.egg/scikits/image/io/_plugins/plugin.py in _load(plugin) 178 modname = plugin_module_name[plugin] 179 plugin_module = __import__('scikits.image.io._plugins.' + modname, --> 180 fromlist=[modname]) 181 182 provides = plugin_provides[plugin] /usr/lib64/python3.1/site-packages/scikits.image-0.3dev-py3.1-linux-x86_64.egg/scikits/image/io/_plugins/pil_plugin.py in () 6 from PIL import Image 7 except ImportError: ----> 8 raise ImportError("The Python Image Library could not be found. " 9 "Please refer to http://pypi.python.org/pypi/PIL/ " 10 "for further instructions.") ImportError: The Python Image Library could not be found. Please refer to http://pypi.python.org/pypi/PIL/ for further instructions. Shouldn't it skip quietly on missing plugins? (It is easy to bypass by a patch, but I am sure you has some design considerations here. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] ??? St?fan van der Walt ????: Monday, March 14, 2011 00:16 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] [OT] any image io module that works with python3? Hi Nadav On Sun, Mar 13, 2011 at 8:20 PM, Nadav Horesh wrote: > Jest tested the installation (after git clone ...). I had to correct the following lines in _build.py to pass installation: > lines 72, and 75 should be: > ? ?f0 = open(f0,'br') > ? ?f1 = open(f1,'br') Thanks so much for testing and for the patch; I've pushed your changes: https://github.com/stefanv/scikits.image/commit/b47ae98ffb92e2de33d9e530201e402e04d865d3 Are you able to load images now? Cheers St?fan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From mwwiebe at gmail.com Mon Mar 14 04:14:26 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Mon, 14 Mar 2011 01:14:26 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers wrote: > On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: > > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris > > wrote: > >> > >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: > >>> > >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers > >>> wrote: > >>>> > >>>> Hi all, > >>>> > >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the > >>>> first beta. So please get your last commits for 1.6 in by Monday > >>>> evening. > >>>> > >>>> Also, please review and add to the 1.6.0 release notes. I put in > >>>> headers for several items that need a few lines in the notes, I hope > >>>> this can be filled in by the authors of those features (Charles: > >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of > >>>> stuff). > >>> > >>> I've added a few more things, and made a small change to the iterator > >>> construction API that I've discovered is useful, but would be more > difficult > >>> to do later. The Python exposure of the iterator is renamed from > 'newiter' > >>> to 'nditer', is that a reasonable name or does anyone have a better > >>> suggestion? > >> > >> I think nditer is fine, certainly better than newiter. I don't see where > >> nditer appears in the changes though, the test still uses newiter. > > > > I didn't rename the files, I can do that too. > > Hi Mark, I see you just did this, but is there anything else you > want/need to do? If it's necessary I can postpone the first beta by a > couple of days. Better that than rush things too much and end up with > an API you have reservations about. > I've committed one other change I wanted, renaming NPY_ITER_NO_INNER_ITERATION to something hopefully a bit more intuitive. Nothing else was nagging at me, but it would be great if some people went through the iterator documentation to see if all the names fit their intuition. We should also come to a consensus on what to call index tuples, and rename ravel_coords, NPY_ITER_COORDS, NpyIter_GotoCoords, and NpyIter_GetGetCoords based on the chosen name. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Mar 14 04:28:41 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 14 Mar 2011 16:28:41 +0800 Subject: [Numpy-discussion] doc wiki edit rights (was: Inaccuracy in documentation of np.linalg.pinv) Message-ID: On Sat, Mar 12, 2011 at 10:50 AM, Hao Xiong wrote: > Hi Ralf, > > I am happy to edit it, although I will have to do it later as I won't > have much free time recently. > > I have registered as haoxiong on docs.scipy.org and would like > to request edit rights to the pinv page. Hi, can someone with permissions on the doc server give Hao Xiong edit rights? Thanks, Ralf > > On 03/09/11 04:16, Ralf Gommers wrote: >> Hi Hao, >> >> On Wed, Mar 9, 2011 at 9:01 AM, Hao Xiong wrote: >>> I think the documentation for np.linalg.pinv contains some inaccuracies. >> Thanks for your comments. Are you interested to make some improvements >> yourself? It's quite easy to do on the numpy doc wiki: >> http://docs.scipy.org/numpy/docs/numpy.linalg.linalg.pinv/ >> Just ask for edit rights on this list after registering a username. >> >> Cheers, >> Ralf >> >> >>> Most importantly, Moore-Penrose is not defined by the solution to the >>> least-square >>> problem. It was defined by the unique solution to 4 equations. Since SVD >>> can be easily shown to satisfy the same 4 equations, it is the >>> Moore-Penrose inverse. >>> >>> Another way that definition is wrong is that the least-square problem >>> does not have to have >>> unique solution. In fact, it can have infinitely many. In this case, >>> Moore-Penrose is the minimum >>> norm solution. >>> >>> All of this is discussed in the Matlab documentation of pinv, which is >>> very good. >>> >>> >>> Hao >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jsalvati at u.washington.edu Mon Mar 14 09:11:56 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 14 Mar 2011 06:11:56 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: If they return a tuple of indexes I think 'mulitiindex' sounds quite good. That is exactly what a multiindex is ( http://en.wikipedia.org/wiki/Multi-index_notation). On Mon, Mar 14, 2011 at 1:14 AM, Mark Wiebe wrote: > On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: >> > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris >> > wrote: >> >> >> >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: >> >>> >> >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers >> >>> wrote: >> >>>> >> >>>> Hi all, >> >>>> >> >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >> >>>> first beta. So please get your last commits for 1.6 in by Monday >> >>>> evening. >> >>>> >> >>>> Also, please review and add to the 1.6.0 release notes. I put in >> >>>> headers for several items that need a few lines in the notes, I hope >> >>>> this can be filled in by the authors of those features (Charles: >> >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >> >>>> stuff). >> >>> >> >>> I've added a few more things, and made a small change to the iterator >> >>> construction API that I've discovered is useful, but would be more >> difficult >> >>> to do later. The Python exposure of the iterator is renamed from >> 'newiter' >> >>> to 'nditer', is that a reasonable name or does anyone have a better >> >>> suggestion? >> >> >> >> I think nditer is fine, certainly better than newiter. I don't see >> where >> >> nditer appears in the changes though, the test still uses newiter. >> > >> > I didn't rename the files, I can do that too. >> >> Hi Mark, I see you just did this, but is there anything else you >> want/need to do? If it's necessary I can postpone the first beta by a >> couple of days. Better that than rush things too much and end up with >> an API you have reservations about. >> > > I've committed one other change I wanted, renaming > NPY_ITER_NO_INNER_ITERATION to something hopefully a bit more intuitive. > Nothing else was nagging at me, but it would be great if some people went > through the iterator documentation to see if all the names fit their > intuition. We should also come to a consensus on what to call index tuples, > and rename ravel_coords, NPY_ITER_COORDS, NpyIter_GotoCoords, and > NpyIter_GetGetCoords based on the chosen name. > > -Mark > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dronne at protocase.com Mon Mar 14 10:45:35 2011 From: dronne at protocase.com (Douglas Ronne) Date: Mon, 14 Mar 2011 14:45:35 +0000 (UTC) Subject: [Numpy-discussion] Numeric 24.2 incompatible with Windows7 SP1 References: <4D75219D.10309@qfin.net> Message-ID: Frank Conradie qfin.net> writes: > > Hi > > I know Numeric 24.2 is really old and probably unsupported by now, but I > thought it might be of interest that Service Pack 1 for Windows 7 has > "broken" Numeric 24.2. [pruned] > > Thanks! > Frank Conradie > Yes, I have had the exact same error, python 2.4, numeric 24.2. The error is in multiarray. Just discovered it this morning. If you have any luck, please let me know! --Douglas From stefan at sun.ac.za Mon Mar 14 11:09:02 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 14 Mar 2011 17:09:02 +0200 Subject: [Numpy-discussion] =?utf-8?b?16rXqdeV15HXlDogW09UXSBhbnkgaW1hZ2Ug?= =?utf-8?q?io_module_that_works_with_python3=3F?= In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE08CB352@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE08CB352@VA3DIAXVS361.RED001.local> Message-ID: On Mon, Mar 14, 2011 at 9:55 AM, Nadav Horesh wrote: > The instillation is OK. The problem is that on my wok PC I do not have PIL installed. So: Thanks, you are right of course: no plugin should be required upon import. I now put the "use_plugin" statement inside a try: except: block. Cheers St?fan From frank at qfin.net Mon Mar 14 12:29:32 2011 From: frank at qfin.net (Frank Conradie) Date: Mon, 14 Mar 2011 09:29:32 -0700 Subject: [Numpy-discussion] Numeric 24.2 incompatible with Windows7 SP1 In-Reply-To: References: <4D75219D.10309@qfin.net> Message-ID: <4D7E426C.8050206@qfin.net> Hi Douglas, simply recompiling the affected modules "fixes" this issue for me, but unfortunately for me I am having trouble recompiling some old 3rd party compiled libraries. What are the chances all module owners/maintainers will go back and recompile their Python 2.3/2.4 modules - not very high I would guess. - Frank On 14/03/2011 7:45 AM, Douglas Ronne wrote: > Frank Conradie qfin.net> writes: > >> Hi >> >> I know Numeric 24.2 is really old and probably unsupported by now, but I >> thought it might be of interest that Service Pack 1 for Windows 7 has >> "broken" Numeric 24.2. > [pruned] >> Thanks! >> Frank Conradie >> > Yes, I have had the exact same error, python 2.4, numeric 24.2. The error is in > multiarray. Just discovered it this morning. If you have any luck, please let > me know! > > --Douglas > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jmccampbell at enthought.com Mon Mar 14 13:09:05 2011 From: jmccampbell at enthought.com (Jason McCampbell) Date: Mon, 14 Mar 2011 12:09:05 -0500 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <1299540962.65641901@beta.apps.rackspace.com> References: <1299540962.65641901@beta.apps.rackspace.com> Message-ID: Hi Dan, I am just catching up with the mailing list after falling behind getting a release. I am the PM for Enthought's part of refactoring NumPy. The first phase of the refactoring project is done except for some clean up and the new version of NumPy is quite stable. (25 regression failures against the core, largely corner cases). If you want to take a look at it, the code is in the Numpy github repository: https://github.com/numpy/numpy-refactor Under the root of the tree, look in the 'libndarray' directory. This is the Python-independent core and might be helpful for what you are trying to do. It has not been released as a part of an official numpy release yet (under consideration as the core of 2.0) but has been released as the first beta version of NumPy and SciPy for .NET. Regards, Jason On Mon, Mar 7, 2011 at 5:36 PM, Dan Halbert wrote: > We currently have some straightforward NumPy code that indirectly > implements a C API defined by a third party. We built a Cython layer that > directly provides the API in a .a library, and then calls Python. The > layering looks like this: > > C main program -> API in Cython -> Python -> NumPy > > This is difficult to package for distribution, because of the Python and > NumPy dependencies. We may need to reimplement our library so it factors out > the Python dependency, and I would like to explore the alternatives. > (Performance may also be a reason to do this, but that is not the main issue > right now.) > > Do you all have some recommendations about tools, libraries, or languages > that you have used to rewrite NumPy code easily into something that's more > self-contained and callable from C? For instance, are there some nice C++ > linear algebra libraries that map closely to NumPy? Or is there some > higher-level compiled array language that looks something like NumPy code? I > apologize if the answers are obvious: I am not very familiar with the tools > in this space. > > Thanks, > Dan > > (I saw the NumPy Refactoring project discussion from earlier. When that is > finished, the resulting Python-independent library might be a nice way to > handle this, but I am thinking shorter-term.) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Jason McCampbell* Enthought, Inc. 512.850.6069 jmccampbell at enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Mon Mar 14 14:04:17 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 14 Mar 2011 11:04:17 -0700 Subject: [Numpy-discussion] =?utf-8?b?16rXqdeV15HXlDogW09UXSBhbnkgaW1hZ2Ug?= =?utf-8?q?io_module_that_works_with_python3=3F?= In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE08CB352@VA3DIAXVS361.RED001.local>, Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D523F@VA3DIAXVS361.RED001.local> After the following corrections, it works on both 2.7 and 3.1 python versions. My limited test included: python2.7: imread and imshow usng pil, freeimage and qt python3.1 imread via freeimage and imshow via qt Thank you very much, Nadav (The file _build.patch is a patch for _build.py) ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of St?fan van der Walt [stefan at sun.ac.za] Sent: 14 March 2011 17:09 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] ?????: [OT] any image io module that works with python3? On Mon, Mar 14, 2011 at 9:55 AM, Nadav Horesh wrote: > The instillation is OK. The problem is that on my wok PC I do not have PIL installed. So: Thanks, you are right of course: no plugin should be required upon import. I now put the "use_plugin" statement inside a try: except: block. Cheers St?fan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: freeimage_plugin.py Type: application/python Size: 15043 bytes Desc: freeimage_plugin.py URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: _build.patch Type: application/octet-stream Size: 124 bytes Desc: _build.patch URL: From cgohlke at uci.edu Mon Mar 14 15:55:14 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 14 Mar 2011 12:55:14 -0700 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local>, <4D7BF58F.4010005@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> Message-ID: <4D7E72A2.6010509@uci.edu> On 3/12/2011 9:56 PM, Nadav Horesh wrote: > This lead to another error probably due to line 68 in map.h. As much as I could trace it, ob_type is a member of PyObject, not of PyTypeObject. I have no clue how to resolve this. I just tried PIL-1.1.7-py3 on an Ubuntu 64 bit system: after one change to map.c it builds OK (without sane support) and passes all tests but one numpy int64 bit related test. Please download again from to make sure you have the same sources and try build again. Christoph > > Nadav. > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] > Sent: 13 March 2011 00:37 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > On 3/12/2011 12:47 PM, Nadav Horesh wrote: >> After the replacement of ? with o, the installation went without errors, but: >> >> nadav at nadav_home ~ $ python3 >> Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >> [GCC 4.4.4] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import _imaging >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod > > Py_FindMethod should be excluded by `#ifndef PY3` or similar > preprocessor statements. There is a typo in map.c line 65: change > `#ifdef PY3` to `#ifndef PY3` and clean your build directory before > rebuilding. > > Christoph > >> >> Thank you, >> >> Nadav. >> ________________________________________ >> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] >> Sent: 12 March 2011 21:49 >> To: numpy-discussion at scipy.org >> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >> >> On 3/12/2011 8:45 AM, Nadav Horesh wrote: >>> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: >>> >>> 1. PythonMagick: Needs boost which I do not have it avaiable on python3 >> Boost works on Python 3.1. You might need to compile it. >>> 2. Pygame: I have the stable version(1.9.1) should it work? >> You need the developer version from svn. >>> 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. >> FreeImagePy is unmaintained, does not work on Python 3, and has problems >> on 64 bit platforms. Just wrap the functions you need in ctypes. >>> 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: >> Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py >> >> Christoph >> >>> >>> nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install >>> /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' >>> warnings.warn(msg) >>> running install >>> running build >>> running build_py >>> running build_ext >>> -------------------------------------------------------------------- >>> PIL 1.1.7 SETUP SUMMARY >>> -------------------------------------------------------------------- >>> version 1.1.7 >>> platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >>> [GCC 4.4.4] >>> -------------------------------------------------------------------- >>> --- TKINTER support available >>> --- JPEG support available >>> --- ZLIB (PNG/ZIP) support available >>> --- FREETYPE2 support available >>> --- LITTLECMS support available >>> >>> . >>> . >>> . >>> >>> byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc >>> Traceback (most recent call last): >>> File "setup.py", line 520, in >>> setup(*(), **configuration) # old school :-) >>> File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup >>> dist.run_commands() >>> File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands >>> self.run_command(cmd) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run >>> self.run_command(cmd_name) >>> File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command >>> self.distribution.run_command(command) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run >>> self.byte_compile(outfiles) >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile >>> dry_run=self.dry_run) >>> File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile >>> compile(file, cfile, dfile) >>> File "/usr/lib64/python3.1/py_compile.py", line 137, in compile >>> codestring = f.read() >>> File "/usr/lib64/python3.1/codecs.py", line 300, in decode >>> (result, consumed) = self._buffer_decode(data, self.errors, final) >>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte >>> >>> >>> >>> Any idea on how to correct it? Any elegant way to avoid byte compiling? >>> >>> Nadav >>> >>> >>> ________________________________________ >>> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] >>> Sent: 12 March 2011 14:35 >>> To: Discussion of Numerical Python >>> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >>> >>> Here's a ctypes interface to FreeImage that I wrote a while back and >>> was since cleaned up (and maintained) by the scikits.image folk: >>> >>> https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >>> >>> If it doesn't work out of the box on python 3, then it should be >>> pretty simple to fix. >>> >>> Zach >>> >>> >>> >>> On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: >>> >>>> >>>> >>>> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>>>> Having numpy, scipy, and matplotlib working reasonably with >>>>> python3, a >>>>> major piece of code I miss for a major python3 migration is an >>>>> image IO. >>>>> I found that pylab's imread works fine for png image, but I need to >>>>> read >>>>> all the other image format as well as png and jpeg output. >>>>> Any hints (including advices how easyly construct my own module) are >>>>> appreciated. >>>>> Nadav. >>>>> >>>> >>>> On Windows, PIL (private port at >>>> ), PythonMagick >>>> , and pygame 1.9.2pre >>>> are working reasonably well for image IO. Also >>>> the FreeImage library is easy to >>>> use >>>> with ctypes. >>>> >>>> Christoph >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ondrej at certik.cz Mon Mar 14 16:24:35 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 14 Mar 2011 13:24:35 -0700 Subject: [Numpy-discussion] rewriting NumPy code in C or C++ or similar In-Reply-To: <4D763C48.9020201@molden.no> References: <1299540962.65641901@beta.apps.rackspace.com> <4D7590D4.1060402@molden.no> <4D75AAFE.9030703@halwitz.org> <4D763C48.9020201@molden.no> Message-ID: Hi Sturla, On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden wrote: > Den 08.03.2011 05:05, skrev Dan Halbert: >> Thanks, that's a good suggestion. I have not written Fortran since 1971, >> but it's come a long way. I was a little worried about the row-major vs >> column-major issue, but perhaps that can be handled just by remembering >> to reverse the subscript order between C and Fortran. > > In practice this is not a problem. Most numerical libraries for C assume > Fortran-ordering, even OpenGL assumes Fortran-ordering. People program > MEX files for Matlab in C all the time. Fortran-ordering is assumed in > MEX files too. > > In ANSI C, array bounds must be known at compile time, so a Fortran > routine with the interface > > ? ? subroutine foobar( lda, A ) > ? ? ? ? integer lda > ? ? ? ? double precision A(lda,*) > ? ? end subroutine > > will usually be written like > > ? ? void foobar( int lda, double A[]); > > in C, ignoring different calling convention for lda. > > Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is > that too difficult to remember? > > In ANSI C the issue actually only arises with small "array of arrays" > having static shape, or convoluted contructs like "pointer to an array > of pointers to arrays". Just avoid those and stay with 1D arrays in C -- > do the 1D to 2D mapping in you mind. > > In C99 arrays are allowed to have dynamic size, which mean we can do > > ? ?void foobar( int lda, double *pA ) > ? ?{ > ? ? ? typedef double row_t [lda]; > ? ? ? vector_t *A = (vector_t*)((void*)&pA[0]); > > Here we must index A[k][i] to match A(i,k) in Fortran. I still have not > seen anyone use C99 like this, so I think it is merely theoretical. > > Chances are if you know how to do this with C99, you also know how to > get the subscripts right. If you are afraid to forget "to reverse the > subscript order between C and Fortran", it just tells me you don't > really know what you are doing when using C, and should probably use > something else. > > Why not Cython? It has "native support" for NumPy arrays. > > Personally I prefer Fortran 95, but that is merely a matter of taste. +1 to all that you wrote about Fortran. I am pretty much switching to it from C/C++ for all my numerical work, and then I use Cython to call it from Python, as well as cmake for the build system. Ondrej From ondrej at certik.cz Mon Mar 14 16:44:27 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 14 Mar 2011 13:44:27 -0700 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: Hi Pearu! On Sat, Mar 12, 2011 at 2:30 AM, Pearu Peterson wrote: > > > On Fri, Mar 11, 2011 at 3:58 AM, Ondrej Certik wrote: >> >> Hi, >> >> I spent about an hour googling and didn't figure this out. Here is my >> setup.py: >> >> setup( >> ? ?name = "libqsnake", >> ? ?cmdclass = {'build_ext': build_ext}, >> ? ?version = "0.1", >> ? ?packages = [ >> ? ? ? ?'qsnake', >> ? ? ? ?'qsnake.calculators', >> ? ? ? ?'qsnake.calculators.tests', >> ? ? ? ?'qsnake.data', >> ? ? ? ?'qsnake.mesh2d', >> ? ? ? ?'qsnake.tests', >> ? ? ? ?], >> ? ?package_data = { >> ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], >> ? ? ? ?}, >> ? ?include_dirs=[numpy.get_include()], >> ? ?ext_modules = [Extension("qsnake.cmesh", [ >> ? ? ? ?"qsnake/cmesh.pyx", >> ? ? ? ?"qsnake/fmesh.f90", >> ? ? ? ?])], >> ? ?description = "Qsnake standard library", >> ? ?license = "BSD", >> ) >> > > You can specify Fortran code, that you don't want to process with f2py, in > the libraries list > and then use the corresponding library in the extension, for example: > > setup(... > ?? libraries = [('foo', dict(sources=['qsnake/fmesh.f90']))], > ?? ext_modules = [Extension("qsnake.cmesh", > ????????????????????????????????????????????? sources = > ["qsnake/cmesh.pyx"], > ????????????????????????????????????????????? libraries = ['foo'] > ??????????????????????????? )], > ? ... > ) > > See also scipy/integrate/setup.py that resolves the same issue but just > using the configuration function approach. Indeed, I just tried it and it works! Thanks. I am now able to compile fortran code into the extension module. The only problem is that I am not able to convince Cython to also parse the .pyx files, it clashes with the numpy's implementation of build_ext. Which usually is not a problem, as long as one doesn't need to compile Fortran and .pyx at the same time. So cmake seems like the only robust option anyway. Ondrej From robertwb at math.washington.edu Mon Mar 14 17:00:12 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 14 Mar 2011 14:00:12 -0700 Subject: [Numpy-discussion] how to compile Fortran using setup.py In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 1:44 PM, Ondrej Certik wrote: > Hi Pearu! > > On Sat, Mar 12, 2011 at 2:30 AM, Pearu Peterson > wrote: >> >> >> On Fri, Mar 11, 2011 at 3:58 AM, Ondrej Certik wrote: >>> >>> Hi, >>> >>> I spent about an hour googling and didn't figure this out. Here is my >>> setup.py: >>> >>> setup( >>> ? ?name = "libqsnake", >>> ? ?cmdclass = {'build_ext': build_ext}, >>> ? ?version = "0.1", >>> ? ?packages = [ >>> ? ? ? ?'qsnake', >>> ? ? ? ?'qsnake.calculators', >>> ? ? ? ?'qsnake.calculators.tests', >>> ? ? ? ?'qsnake.data', >>> ? ? ? ?'qsnake.mesh2d', >>> ? ? ? ?'qsnake.tests', >>> ? ? ? ?], >>> ? ?package_data = { >>> ? ? ? ?'qsnake.tests': ['phaml_data/domain.*'], >>> ? ? ? ?}, >>> ? ?include_dirs=[numpy.get_include()], >>> ? ?ext_modules = [Extension("qsnake.cmesh", [ >>> ? ? ? ?"qsnake/cmesh.pyx", >>> ? ? ? ?"qsnake/fmesh.f90", >>> ? ? ? ?])], >>> ? ?description = "Qsnake standard library", >>> ? ?license = "BSD", >>> ) >>> >> >> You can specify Fortran code, that you don't want to process with f2py, in >> the libraries list >> and then use the corresponding library in the extension, for example: >> >> setup(... >> ?? libraries = [('foo', dict(sources=['qsnake/fmesh.f90']))], >> ?? ext_modules = [Extension("qsnake.cmesh", >> ????????????????????????????????????????????? sources = >> ["qsnake/cmesh.pyx"], >> ????????????????????????????????????????????? libraries = ['foo'] >> ??????????????????????????? )], >> ? ... >> ) >> >> See also scipy/integrate/setup.py that resolves the same issue but just >> using the configuration function approach. > > Indeed, I just tried it and it works! Thanks. I am now able to compile > fortran code into the extension module. > > The only problem is that I am not able to convince Cython to also > parse the .pyx files, it clashes with the numpy's implementation of > build_ext. Which usually is not a problem, as long as one doesn't need > to compile Fortran and .pyx at the same time. In the current Cython tip you can do ext_modules = cythonize([Extension("qsnake.cmesh", [ "qsnake/cmesh.pyx", "qsnake/fmesh.f90", ])]), which will translate the .pyx file to a .c file before hitting distutils. > So cmake seems like the only robust option anyway. > > Ondrej > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Mar 14 17:41:50 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 14 Mar 2011 15:41:50 -0600 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default In-Reply-To: References: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> Message-ID: On Sun, Mar 13, 2011 at 5:01 AM, Ralf Gommers wrote: > On Sun, Mar 13, 2011 at 1:57 PM, David Warde-Farley > wrote: > > > > On 2011-03-12, at 9:32 PM, Charles R Harris wrote: > > > >> I'd like to change the polynomial package to only import the Classes, > leaving the large number of implementation functions to be imported directly > from the different modules if needed. I always regarded those functions as > implementation helpers and kept them separate from the class so that others > could use them to build their own classes if they desired. For most purposes > I think the classes are more useful. So I think it was a mistake to import > the functions by; default and I'm looking for a graceful and acceptable way > out. Any suggestions. > > How about just deprecating in 1.6 and removing from the polynomial > namespace in 2.0? Then you do need to add to the docs explicitly that > the submodules contain these functions, that they are part of the > public API and are not going anywhere. > > Is there a clever way to deprecate the functions only in that namespace? I wrote a script that writes a lot of dummy functions that call the original with a deprecation message, but I'd sure welcome a cleaner way to do things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Mon Mar 14 18:02:32 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 14 Mar 2011 23:02:32 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] Message-ID: On Mon, Mar 14, 2011 at 9:24 PM, Ondrej Certik wrote: > Hi Sturla, > > On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden wrote: >> Den 08.03.2011 05:05, skrev Dan Halbert: >>> Thanks, that's a good suggestion. I have not written Fortran since 1971, >>> but it's come a long way. I was a little worried about the row-major vs >>> column-major issue, but perhaps that can be handled just by remembering >>> to reverse the subscript order between C and Fortran. >> >> In practice this is not a problem. Most numerical libraries for C assume >> Fortran-ordering, even OpenGL assumes Fortran-ordering. People program >> MEX files for Matlab in C all the time. Fortran-ordering is assumed in >> MEX files too. >> >> In ANSI C, array bounds must be known at compile time, so a Fortran >> routine with the interface >> >> ? ? subroutine foobar( lda, A ) >> ? ? ? ? integer lda >> ? ? ? ? double precision A(lda,*) >> ? ? end subroutine >> >> will usually be written like >> >> ? ? void foobar( int lda, double A[]); >> >> in C, ignoring different calling convention for lda. >> >> Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is >> that too difficult to remember? >> >> In ANSI C the issue actually only arises with small "array of arrays" >> having static shape, or convoluted contructs like "pointer to an array >> of pointers to arrays". Just avoid those and stay with 1D arrays in C -- >> do the 1D to 2D mapping in you mind. >> >> In C99 arrays are allowed to have dynamic size, which mean we can do >> >> ? ?void foobar( int lda, double *pA ) >> ? ?{ >> ? ? ? typedef double row_t [lda]; >> ? ? ? vector_t *A = (vector_t*)((void*)&pA[0]); >> >> Here we must index A[k][i] to match A(i,k) in Fortran. I still have not >> seen anyone use C99 like this, so I think it is merely theoretical. >> >> Chances are if you know how to do this with C99, you also know how to >> get the subscripts right. If you are afraid to forget "to reverse the >> subscript order between C and Fortran", it just tells me you don't >> really know what you are doing when using C, and should probably use >> something else. >> >> Why not Cython? It has "native support" for NumPy arrays. >> >> Personally I prefer Fortran 95, but that is merely a matter of taste. > > +1 to all that you wrote about Fortran. I am pretty much switching to > it from C/C++ for all my numerical work, and then I use Cython to call > it from Python, as well as cmake for the build system. > > Ondrej Hi, this is quite amazing... Sturla has been writing so much about Fortran recently, and Ondrej now says he has done the move from C/C++ to Fortran -- I thought Fortran was dead ... !? ;-) What am I missing here ? Apparently (from what I was able to read-up so far) there is a BIG difference between FORTRAN 77 and F95. But isn't gcc or gfortran still only supporting F77 ? How about IntelCC's Fortran ? Is that superior? Do you guys have any info / blogs / docs where one could get an up-to-date picture? Like: 1. How about debugging - does gdb work or is there somthing better ? 2. How is the move of the F77 community to F95 in general ? How many people / projects are switching. 3. Or is the move rather going like Fortran 77 -> C -> Python -> Fortran 95 !? ;-) Thanks, Sebastian Haase From matthieu.brucher at gmail.com Mon Mar 14 18:10:13 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 14 Mar 2011 23:10:13 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: Hi, Intel Fortran is an excellent Fortran compiler. Why is Fortran still better than C and C++? - some rules are different, like arrays passed to functions are ALWAYS supposed to be independent in Fortran, whereas in C, you have to add a restrict keyword - due to the last fact, Fortran is a language where its compiler could do more work (vectorization, autoparallelization...) - Fortran 95 has an excellent array support, which is not currently available in C/C++ (perhaps with ArBB?) Nevertheless, when you know C++ correctly, when you want to do something really efficient, you don't use Fortran. You can be as efficient as in C++, and you can do fancy stuff (I/O, network...). Class and templates are also better supported in C++. Matthieu 2011/3/14 Sebastian Haase > On Mon, Mar 14, 2011 at 9:24 PM, Ondrej Certik wrote: > > Hi Sturla, > > > > On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden wrote: > >> Den 08.03.2011 05:05, skrev Dan Halbert: > >>> Thanks, that's a good suggestion. I have not written Fortran since > 1971, > >>> but it's come a long way. I was a little worried about the row-major vs > >>> column-major issue, but perhaps that can be handled just by remembering > >>> to reverse the subscript order between C and Fortran. > >> > >> In practice this is not a problem. Most numerical libraries for C assume > >> Fortran-ordering, even OpenGL assumes Fortran-ordering. People program > >> MEX files for Matlab in C all the time. Fortran-ordering is assumed in > >> MEX files too. > >> > >> In ANSI C, array bounds must be known at compile time, so a Fortran > >> routine with the interface > >> > >> subroutine foobar( lda, A ) > >> integer lda > >> double precision A(lda,*) > >> end subroutine > >> > >> will usually be written like > >> > >> void foobar( int lda, double A[]); > >> > >> in C, ignoring different calling convention for lda. > >> > >> Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is > >> that too difficult to remember? > >> > >> In ANSI C the issue actually only arises with small "array of arrays" > >> having static shape, or convoluted contructs like "pointer to an array > >> of pointers to arrays". Just avoid those and stay with 1D arrays in C -- > >> do the 1D to 2D mapping in you mind. > >> > >> In C99 arrays are allowed to have dynamic size, which mean we can do > >> > >> void foobar( int lda, double *pA ) > >> { > >> typedef double row_t [lda]; > >> vector_t *A = (vector_t*)((void*)&pA[0]); > >> > >> Here we must index A[k][i] to match A(i,k) in Fortran. I still have not > >> seen anyone use C99 like this, so I think it is merely theoretical. > >> > >> Chances are if you know how to do this with C99, you also know how to > >> get the subscripts right. If you are afraid to forget "to reverse the > >> subscript order between C and Fortran", it just tells me you don't > >> really know what you are doing when using C, and should probably use > >> something else. > >> > >> Why not Cython? It has "native support" for NumPy arrays. > >> > >> Personally I prefer Fortran 95, but that is merely a matter of taste. > > > > +1 to all that you wrote about Fortran. I am pretty much switching to > > it from C/C++ for all my numerical work, and then I use Cython to call > > it from Python, as well as cmake for the build system. > > > > Ondrej > > > Hi, > this is quite amazing... > Sturla has been writing so much about Fortran recently, and Ondrej now > says he has done the move from C/C++ to Fortran -- I thought Fortran > was dead ... !? ;-) > What am I missing here ? > Apparently (from what I was able to read-up so far) there is a BIG > difference between FORTRAN 77 and F95. > But isn't gcc or gfortran still only supporting F77 ? > How about IntelCC's Fortran ? Is that superior? > Do you guys have any info / blogs / docs where one could get an > up-to-date picture? > Like: > 1. How about debugging - does gdb work or is there somthing better ? > 2. How is the move of the F77 community to F95 in general ? How many > people / projects are switching. > 3. Or is the move rather going like Fortran 77 -> C -> Python -> > Fortran 95 !? ;-) > > Thanks, > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Mar 14 19:02:54 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 15 Mar 2011 01:02:54 +0200 Subject: [Numpy-discussion] =?utf-8?b?16rXqdeV15HXlDogW09UXSBhbnkgaW1hZ2Ug?= =?utf-8?q?io_module_that_works_with_python3=3F?= In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D523F@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE08CB352@VA3DIAXVS361.RED001.local> <26FC23E7C398A64083C980D16001012D1CE03D523F@VA3DIAXVS361.RED001.local> Message-ID: On Mon, Mar 14, 2011 at 8:04 PM, Nadav Horesh wrote: > After the following corrections, it works on both 2.7 and 3.1 python versions. My limited test included: > python2.7: imread and imshow usng pil, freeimage and qt > python3.1 imread via freeimage and imshow via qt Thanks, I tried to fix the filename problem using numpy.compat.asbytes. I'm now building scipy under Py3 before I can test the changes, but I hope they're ok. Regards St?fan From stefan at sun.ac.za Mon Mar 14 19:11:33 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 15 Mar 2011 01:11:33 +0200 Subject: [Numpy-discussion] doc wiki edit rights (was: Inaccuracy in documentation of np.linalg.pinv) In-Reply-To: References: Message-ID: Hi Ralf On Mon, Mar 14, 2011 at 10:28 AM, Ralf Gommers wrote: > Hi, can someone with permissions on the doc server give Hao Xiong edit rights? You now have admin status. I've also set Hao up as an editor. Regards St?fan From magawake at gmail.com Mon Mar 14 20:12:14 2011 From: magawake at gmail.com (Mag Gam) Date: Mon, 14 Mar 2011 20:12:14 -0400 Subject: [Numpy-discussion] finding atlas Message-ID: Trying to compile Numpy with Intel's MKL. I have exported the proper paths for BLAS and LAPACK and I think the build script found it. However, I am having a lot of trouble with ATLAS. What library file should I use for it? tia From dsm054 at gmail.com Mon Mar 14 20:17:06 2011 From: dsm054 at gmail.com (D. S. McNeil) Date: Tue, 15 Mar 2011 08:17:06 +0800 Subject: [Numpy-discussion] sage: isscalar can't hear quacks Message-ID: (Sending upstream from http://trac.sagemath.org/sage_trac/ticket/10928) sage: import numpy sage: m = numpy.matrix(numpy.arange(6)).reshape(3,2) sage: m[:,int(0)] matrix([[0], [2], [4]]) sage: m[:,0] matrix([[0, 2, 4]]) That is, Python ints produce a column but Sage Integers produce a row! This happens because sage: numpy.isscalar(int(0)) True sage: numpy.isscalar(Integer(0)) False where isscalar simply tries if isinstance(num, generic): return True else: return type(num) in ScalarType Since Sage Integers (or Python Fractions, or any other unknown scalar) aren't instances of generic or listed in ScalarType, isscalar returns False, and there's an unintended fallthrough in matrix.__getitem__. ISTM either __getitem__ needs to change or isscalar needs to become more accepting, or both.. but the former might leave some other unexpected isscalar-related issues elsewhere and I don't know enough about numpy internals to know if broadening isscalar would cause problems of its own. Recommendations? Doug -- Department of Earth Sciences University of Hong Kong From jonathan.taylor at utoronto.ca Mon Mar 14 21:30:44 2011 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Mon, 14 Mar 2011 21:30:44 -0400 Subject: [Numpy-discussion] Add New Functionality for Indexing Along an Axis to Numpy? Message-ID: Please excuse the double post as I suspect people who may have thoughts on the inclusion of such functionality in numpy were not following the discussion due to the old subject. I am perfectly happy keeping this functionality locally but some of my colleagues have also indicated that they have resorted to loops in the past to solve this not uncommon use case so perhaps it would be helpful to more people if it (or something similar?) was included in numpy? Jonathan. On Thu, Mar 10, 2011 at 12:00 PM, Jonathan Taylor wrote: > I see. > > Should functionality like this be included in numpy? > > Jon. > > > On Tue, Mar 8, 2011 at 3:39 PM, ? wrote: >> On Tue, Mar 8, 2011 at 3:03 PM, Jonathan Taylor >> wrote: >>> I am wanting to use an array b to index into an array x with dimension >>> bigger by 1 where the element of b indicates what value to extract >>> along a certain direction. ?For example, b = x.argmin(axis=1). >>> Perhaps I want to use b to create x.min(axis=1) but also to index >>> perhaps another array of the same size. >>> >>> I had a difficult time finding a way to do this with np.take easily >>> and even with fancy indexing the resulting line is very complicated: >>> >>> In [322]: x.shape >>> Out[322]: (2, 3, 4) >>> >>> In [323]: x.min(axis=1) >>> Out[323]: >>> array([[ 2, ?1, ?7, ?4], >>> ? ? ? [ 8, ?0, 15, 12]]) >>> >>> In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis], >>> idx[:,np.newaxis,:], np.arange(x.shape[2])] >>> Out[324]: >>> array([[[ 2, ?1, ?7, ?4]], >>> >>> ? ? ? [[ 8, ?0, 15, 12]]]) >>> >>> In any case I wrote myself my own function for doing this (below) and >>> am wondering if this is the best way to do this or if there is >>> something else in numpy that I should be using? -- I figure that this >>> is a relatively common usecase. >>> >>> Thanks, >>> Jon. >>> >>> def mytake(A, b, axis): >>> ? ?assert len(A.shape) == len(b.shape)+1 >>> >>> ? ?idx = [] >>> ? ?for i in range(len(A.shape)): >>> ? ? ? ?if i == axis: >>> ? ? ? ? ? ?temp = b.copy() >>> ? ? ? ? ? ?shapey = list(temp.shape) >>> ? ? ? ? ? ?shapey.insert(i,1) >>> ? ? ? ?else: >>> ? ? ? ? ? ?temp = np.arange(A.shape[i]) >>> ? ? ? ? ? ?shapey = [1]*len(b.shape) >>> ? ? ? ? ? ?shapey.insert(i,A.shape[i]) >>> ? ? ? ?shapey = tuple(shapey) >>> ? ? ? ?temp = temp.reshape(shapey) >>> ? ? ? ?idx += [temp] >>> >>> ? ?return A[tuple(idx)].squeeze() >>> >>> >>> In [319]: util.mytake(x,x.argmin(axis=1), 1) >>> Out[319]: >>> array([[ 2, ?1, ?7, ?4], >>> ? ? ? [ 8, ?0, 15, 12]]) >>> >>> In [320]: x.min(axis=1) >>> Out[320]: >>> array([[ 2, ?1, ?7, ?4], >>> ? ? ? [ 8, ?0, 15, 12]]) >> >> fewer lines but essentially the same thing and no shortcuts, I think >> >>>>> x= np.random.randint(5, size=(2, 3, 4)) >>>>> x >> array([[[3, 1, 0, 1], >> ? ? ? ?[4, 2, 2, 1], >> ? ? ? ?[2, 3, 2, 2]], >> >> ? ? ? [[2, 1, 1, 1], >> ? ? ? ?[0, 2, 0, 3], >> ? ? ? ?[2, 3, 3, 1]]]) >> >>>>> idx = [np.arange(i) for i in x.shape] >>>>> idx = list(np.ix_(*idx)) >>>>> idx[axis]=np.expand_dims(x.argmin(axis),axis) >>>>> x[idx] >> array([[[2, 1, 0, 1]], >> >> ? ? ? [[0, 1, 0, 1]]]) >> >>>>> np.squeeze(x[idx]) >> array([[2, 1, 0, 1], >> ? ? ? [0, 1, 0, 1]]) >> >>>>> mytake(x,x.argmin(axis=1), 1) >> array([[2, 1, 0, 1], >> ? ? ? [0, 1, 0, 1]]) >> >> Josef >> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From jsalvati at u.washington.edu Mon Mar 14 22:38:20 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 14 Mar 2011 19:38:20 -0700 Subject: [Numpy-discussion] Add New Functionality for Indexing Along an Axis to Numpy? In-Reply-To: References: Message-ID: The new iteration functionality will be providing this in the near future (along with many other things). See https://github.com/numpy/numpy/blob/master/doc/neps/new-iterator-ufunc.rst On Mon, Mar 14, 2011 at 6:30 PM, Jonathan Taylor < jonathan.taylor at utoronto.ca> wrote: > Please excuse the double post as I suspect people who may have > thoughts on the inclusion of such functionality in numpy were not > following the discussion due to the old subject. I am perfectly happy > keeping this functionality locally but some of my colleagues have also > indicated that they have resorted to loops in the past to solve this > not uncommon use case so perhaps it would be helpful to more people if > it (or something similar?) was included in numpy? > > Jonathan. > > On Thu, Mar 10, 2011 at 12:00 PM, Jonathan Taylor > wrote: > > I see. > > > > Should functionality like this be included in numpy? > > > > Jon. > > > > > > On Tue, Mar 8, 2011 at 3:39 PM, wrote: > >> On Tue, Mar 8, 2011 at 3:03 PM, Jonathan Taylor > >> wrote: > >>> I am wanting to use an array b to index into an array x with dimension > >>> bigger by 1 where the element of b indicates what value to extract > >>> along a certain direction. For example, b = x.argmin(axis=1). > >>> Perhaps I want to use b to create x.min(axis=1) but also to index > >>> perhaps another array of the same size. > >>> > >>> I had a difficult time finding a way to do this with np.take easily > >>> and even with fancy indexing the resulting line is very complicated: > >>> > >>> In [322]: x.shape > >>> Out[322]: (2, 3, 4) > >>> > >>> In [323]: x.min(axis=1) > >>> Out[323]: > >>> array([[ 2, 1, 7, 4], > >>> [ 8, 0, 15, 12]]) > >>> > >>> In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis], > >>> idx[:,np.newaxis,:], np.arange(x.shape[2])] > >>> Out[324]: > >>> array([[[ 2, 1, 7, 4]], > >>> > >>> [[ 8, 0, 15, 12]]]) > >>> > >>> In any case I wrote myself my own function for doing this (below) and > >>> am wondering if this is the best way to do this or if there is > >>> something else in numpy that I should be using? -- I figure that this > >>> is a relatively common usecase. > >>> > >>> Thanks, > >>> Jon. > >>> > >>> def mytake(A, b, axis): > >>> assert len(A.shape) == len(b.shape)+1 > >>> > >>> idx = [] > >>> for i in range(len(A.shape)): > >>> if i == axis: > >>> temp = b.copy() > >>> shapey = list(temp.shape) > >>> shapey.insert(i,1) > >>> else: > >>> temp = np.arange(A.shape[i]) > >>> shapey = [1]*len(b.shape) > >>> shapey.insert(i,A.shape[i]) > >>> shapey = tuple(shapey) > >>> temp = temp.reshape(shapey) > >>> idx += [temp] > >>> > >>> return A[tuple(idx)].squeeze() > >>> > >>> > >>> In [319]: util.mytake(x,x.argmin(axis=1), 1) > >>> Out[319]: > >>> array([[ 2, 1, 7, 4], > >>> [ 8, 0, 15, 12]]) > >>> > >>> In [320]: x.min(axis=1) > >>> Out[320]: > >>> array([[ 2, 1, 7, 4], > >>> [ 8, 0, 15, 12]]) > >> > >> fewer lines but essentially the same thing and no shortcuts, I think > >> > >>>>> x= np.random.randint(5, size=(2, 3, 4)) > >>>>> x > >> array([[[3, 1, 0, 1], > >> [4, 2, 2, 1], > >> [2, 3, 2, 2]], > >> > >> [[2, 1, 1, 1], > >> [0, 2, 0, 3], > >> [2, 3, 3, 1]]]) > >> > >>>>> idx = [np.arange(i) for i in x.shape] > >>>>> idx = list(np.ix_(*idx)) > >>>>> idx[axis]=np.expand_dims(x.argmin(axis),axis) > >>>>> x[idx] > >> array([[[2, 1, 0, 1]], > >> > >> [[0, 1, 0, 1]]]) > >> > >>>>> np.squeeze(x[idx]) > >> array([[2, 1, 0, 1], > >> [0, 1, 0, 1]]) > >> > >>>>> mytake(x,x.argmin(axis=1), 1) > >> array([[2, 1, 0, 1], > >> [0, 1, 0, 1]]) > >> > >> Josef > >> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Mar 14 22:57:18 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 15 Mar 2011 10:57:18 +0800 Subject: [Numpy-discussion] Not importing polynomial implementation functions by default In-Reply-To: References: <40B1309B-CD45-4073-8787-E0FFF7022737@iro.umontreal.ca> Message-ID: On Tue, Mar 15, 2011 at 5:41 AM, Charles R Harris wrote: > > > On Sun, Mar 13, 2011 at 5:01 AM, Ralf Gommers > wrote: >> >> On Sun, Mar 13, 2011 at 1:57 PM, David Warde-Farley >> wrote: >> > >> > On 2011-03-12, at 9:32 PM, Charles R Harris wrote: >> > >> >> I'd like to change the polynomial package to only import the Classes, >> >> leaving the large number of implementation functions to be imported directly >> >> from the different modules if needed. I always regarded those functions as >> >> implementation helpers and kept them separate from the class so that others >> >> could use them to build their own classes if they desired. For most purposes >> >> I think the classes are more useful. So I think it was a mistake to import >> >> the functions by; default and I'm looking for a graceful and acceptable way >> >> out. Any suggestions. >> >> How about just deprecating in 1.6 and removing from the polynomial >> namespace in 2.0? Then you do need to add to the docs explicitly that >> the submodules contain these functions, that they are part of the >> public API and are not going anywhere. >> > > Is there a clever way to deprecate the functions only in that namespace? I > wrote a script that writes a lot of dummy functions that call the original > with a deprecation message, but I'd sure welcome a cleaner way to do things. Seems clean to me if you're using numpy.deprecate for it. Can you commit that today? Ralf From ralf.gommers at googlemail.com Mon Mar 14 23:24:37 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 15 Mar 2011 11:24:37 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 9:11 PM, John Salvatier wrote: > If they return a tuple of indexes I think 'mulitiindex' sounds quite good. > That is exactly what a multiindex is > (http://en.wikipedia.org/wiki/Multi-index_notation). > > On Mon, Mar 14, 2011 at 1:14 AM, Mark Wiebe wrote: >> >> On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers >> wrote: >>> >>> On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: >>> > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris >>> > wrote: >>> >> >>> >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: >>> >>> >>> >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers >>> >>> wrote: >>> >>>> >>> >>>> Hi all, >>> >>>> >>> >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >>> >>>> first beta. So please get your last commits for 1.6 in by Monday >>> >>>> evening. >>> >>>> >>> >>>> Also, please review and add to the 1.6.0 release notes. I put in >>> >>>> headers for several items that need a few lines in the notes, I hope >>> >>>> this can be filled in by the authors of those features (Charles: >>> >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >>> >>>> stuff). >>> >>> >>> >>> I've added a few more things, and made a small change to the iterator >>> >>> construction API that I've discovered is useful, but would be more >>> >>> difficult >>> >>> to do later. The Python exposure of the iterator is renamed from >>> >>> 'newiter' >>> >>> to 'nditer', is that a reasonable name or does anyone have a better >>> >>> suggestion? >>> >> >>> >> I think nditer is fine, certainly better than newiter. I don't see >>> >> where >>> >> nditer appears in the changes though, the test still uses newiter. >>> > >>> > I didn't rename the files, I can do that too. >>> >>> Hi Mark, I see you just did this, but is there anything else you >>> want/need to do? If it's necessary I can postpone the first beta by a >>> couple of days. Better that than rush things too much and end up with >>> an API you have reservations about. >> >> I've committed one other change I wanted, renaming >> NPY_ITER_NO_INNER_ITERATION to something hopefully a bit more intuitive. >> Nothing else was nagging at me, but it would be great if some people went >> through the iterator documentation to see if all the names fit their >> intuition. We should also come to a consensus on what to call index tuples, >> and rename ravel_coords, NPY_ITER_COORDS, NpyIter_GotoCoords, and >> NpyIter_GetGetCoords based on the chosen name. MultiIndex / multi_index works for me too. Can you rename that today if no one comes up with something better? Ralf From ralf.gommers at googlemail.com Mon Mar 14 23:58:44 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 15 Mar 2011 11:58:44 +0800 Subject: [Numpy-discussion] finding atlas In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 8:12 AM, Mag Gam wrote: > Trying to compile Numpy with Intel's MKL. I have exported the proper > paths for BLAS and LAPACK and I think the build script found it. > However, I am having a lot of trouble with ATLAS. What library file > should I use for it? That's not very specific. Can you send us your site.cfg, build log and details like platform, python version, etc? Ralf From mwwiebe at gmail.com Tue Mar 15 01:35:36 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Mon, 14 Mar 2011 22:35:36 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 8:24 PM, Ralf Gommers wrote: > On Mon, Mar 14, 2011 at 9:11 PM, John Salvatier > wrote: > > If they return a tuple of indexes I think 'mulitiindex' sounds quite > good. > > That is exactly what a multiindex is > > (http://en.wikipedia.org/wiki/Multi-index_notation). > > > > On Mon, Mar 14, 2011 at 1:14 AM, Mark Wiebe wrote: > >> > >> On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers > >> wrote: > >>> > >>> On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: > >>> > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris > >>> > wrote: > >>> >> > >>> >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe > wrote: > >>> >>> > >>> >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers > >>> >>> wrote: > >>> >>>> > >>> >>>> Hi all, > >>> >>>> > >>> >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag > the > >>> >>>> first beta. So please get your last commits for 1.6 in by Monday > >>> >>>> evening. > >>> >>>> > >>> >>>> Also, please review and add to the 1.6.0 release notes. I put in > >>> >>>> headers for several items that need a few lines in the notes, I > hope > >>> >>>> this can be filled in by the authors of those features (Charles: > >>> >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch > of > >>> >>>> stuff). > >>> >>> > >>> >>> I've added a few more things, and made a small change to the > iterator > >>> >>> construction API that I've discovered is useful, but would be more > >>> >>> difficult > >>> >>> to do later. The Python exposure of the iterator is renamed from > >>> >>> 'newiter' > >>> >>> to 'nditer', is that a reasonable name or does anyone have a better > >>> >>> suggestion? > >>> >> > >>> >> I think nditer is fine, certainly better than newiter. I don't see > >>> >> where > >>> >> nditer appears in the changes though, the test still uses newiter. > >>> > > >>> > I didn't rename the files, I can do that too. > >>> > >>> Hi Mark, I see you just did this, but is there anything else you > >>> want/need to do? If it's necessary I can postpone the first beta by a > >>> couple of days. Better that than rush things too much and end up with > >>> an API you have reservations about. > >> > >> I've committed one other change I wanted, renaming > >> NPY_ITER_NO_INNER_ITERATION to something hopefully a bit more intuitive. > >> Nothing else was nagging at me, but it would be great if some people > went > >> through the iterator documentation to see if all the names fit their > >> intuition. We should also come to a consensus on what to call index > tuples, > >> and rename ravel_coords, NPY_ITER_COORDS, NpyIter_GotoCoords, and > >> NpyIter_GetGetCoords based on the chosen name. > > MultiIndex / multi_index works for me too. Can you rename that today > if no one comes up with something better? I've pushed a patch with the rename. Can someone review it to check that it's all good? -Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From binet at cern.ch Tue Mar 15 05:27:46 2011 From: binet at cern.ch (Sebastien Binet) Date: Tue, 15 Mar 2011 10:27:46 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: <871v28hgx9.fsf@cern.ch> On Mon, 14 Mar 2011 23:10:13 +0100, Matthieu Brucher wrote: > Hi, > > Intel Fortran is an excellent Fortran compiler. Why is Fortran still better > than C and C++? > - some rules are different, like arrays passed to functions are ALWAYS > supposed to be independent in Fortran, whereas in C, you have to add a > restrict keyword > - due to the last fact, Fortran is a language where its compiler could do > more work (vectorization, autoparallelization...) > - Fortran 95 has an excellent array support, which is not currently > available in C/C++ (perhaps with ArBB?) > > Nevertheless, when you know C++ correctly, when you want to do something > really efficient, you don't use Fortran. You can be as efficient as in C++, > and you can do fancy stuff (I/O, network...). Class and templates are also > better supported in C++. bah... usually, a scientist doesn't want to do fancy stuff. fancy stuff gets in the way at the end of the day. what a scientist wants is to do number crunching as efficiently as possible w/o having to learn about all the traps that a language can lay for the unwarry. gfortran has received less attention than its fellow C and C++ friends lately (or so I hear) but it supports almost all of Fortran-2003 and bits and pieces of Fortran-2008: http://gcc.gnu.org/wiki/Fortran2003 http://gcc.gnu.org/wiki/Fortran2003Status http://gcc.gnu.org/wiki/Fortran2008Status not having to rely on a computing farm to build large scale software is really something nice when you come back from C++ and templates. my only problems with Fortran are the indexing (not 0-based, even if you can declare arrays to start at index 0, it isn't common) and the non C-order of arrays. both of these worries are just stemming from habit rather than standing on technical grounds, though. -s > > Matthieu > > 2011/3/14 Sebastian Haase > > > On Mon, Mar 14, 2011 at 9:24 PM, Ondrej Certik wrote: > > > Hi Sturla, > > > > > > On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden wrote: > > >> Den 08.03.2011 05:05, skrev Dan Halbert: > > >>> Thanks, that's a good suggestion. I have not written Fortran since > > 1971, > > >>> but it's come a long way. I was a little worried about the row-major vs > > >>> column-major issue, but perhaps that can be handled just by remembering > > >>> to reverse the subscript order between C and Fortran. > > >> > > >> In practice this is not a problem. Most numerical libraries for C assume > > >> Fortran-ordering, even OpenGL assumes Fortran-ordering. People program > > >> MEX files for Matlab in C all the time. Fortran-ordering is assumed in > > >> MEX files too. > > >> > > >> In ANSI C, array bounds must be known at compile time, so a Fortran > > >> routine with the interface > > >> > > >> subroutine foobar( lda, A ) > > >> integer lda > > >> double precision A(lda,*) > > >> end subroutine > > >> > > >> will usually be written like > > >> > > >> void foobar( int lda, double A[]); > > >> > > >> in C, ignoring different calling convention for lda. > > >> > > >> Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is > > >> that too difficult to remember? > > >> > > >> In ANSI C the issue actually only arises with small "array of arrays" > > >> having static shape, or convoluted contructs like "pointer to an array > > >> of pointers to arrays". Just avoid those and stay with 1D arrays in C -- > > >> do the 1D to 2D mapping in you mind. > > >> > > >> In C99 arrays are allowed to have dynamic size, which mean we can do > > >> > > >> void foobar( int lda, double *pA ) > > >> { > > >> typedef double row_t [lda]; > > >> vector_t *A = (vector_t*)((void*)&pA[0]); > > >> > > >> Here we must index A[k][i] to match A(i,k) in Fortran. I still have not > > >> seen anyone use C99 like this, so I think it is merely theoretical. > > >> > > >> Chances are if you know how to do this with C99, you also know how to > > >> get the subscripts right. If you are afraid to forget "to reverse the > > >> subscript order between C and Fortran", it just tells me you don't > > >> really know what you are doing when using C, and should probably use > > >> something else. > > >> > > >> Why not Cython? It has "native support" for NumPy arrays. > > >> > > >> Personally I prefer Fortran 95, but that is merely a matter of taste. > > > > > > +1 to all that you wrote about Fortran. I am pretty much switching to > > > it from C/C++ for all my numerical work, and then I use Cython to call > > > it from Python, as well as cmake for the build system. > > > > > > Ondrej > > > > > > Hi, > > this is quite amazing... > > Sturla has been writing so much about Fortran recently, and Ondrej now > > says he has done the move from C/C++ to Fortran -- I thought Fortran > > was dead ... !? ;-) > > What am I missing here ? > > Apparently (from what I was able to read-up so far) there is a BIG > > difference between FORTRAN 77 and F95. > > But isn't gcc or gfortran still only supporting F77 ? > > How about IntelCC's Fortran ? Is that superior? > > Do you guys have any info / blogs / docs where one could get an > > up-to-date picture? > > Like: > > 1. How about debugging - does gdb work or is there somthing better ? > > 2. How is the move of the F77 community to F95 in general ? How many > > people / projects are switching. > > 3. Or is the move rather going like Fortran 77 -> C -> Python -> > > Fortran 95 !? ;-) > > > > Thanks, > > Sebastian Haase > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher Non-text part: text/html > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From magawake at gmail.com Tue Mar 15 07:14:43 2011 From: magawake at gmail.com (Mag Gam) Date: Tue, 15 Mar 2011 07:14:43 -0400 Subject: [Numpy-discussion] finding atlas In-Reply-To: References: Message-ID: Ralf, Actually, ATLAS isn't needed when using MKL. http://software.intel.com/en-us/forums/showthread.php?t=81451&p=1#144534 I am going to try this link line out $(MKLROOT)/lib/em64t/libmkl_solver_ilp64.a -Wl,--start-group $(MKLROOT)/lib/em64t/libmkl_intel_ilp64.a $(MKLROOT)/lib/em64t/libmkl_intel_thread.a $(MKLROOT)/lib/em64t/libmkl_core.a -Wl,--end-group -openmp -lpthread On Mon, Mar 14, 2011 at 11:58 PM, Ralf Gommers wrote: > On Tue, Mar 15, 2011 at 8:12 AM, Mag Gam wrote: >> Trying to compile Numpy with Intel's MKL. I have exported the proper >> paths for BLAS and LAPACK and I think the build script found it. >> However, I am having a lot of trouble with ATLAS. What library file >> should I use for it? > > That's not very specific. Can you send us your site.cfg, build log and > details like platform, python version, etc? > > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Tue Mar 15 07:26:12 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 15 Mar 2011 07:26:12 -0400 Subject: [Numpy-discussion] your morning zen Message-ID: I think that I shall never see a rotate right divide by three From sturla at molden.no Tue Mar 15 09:32:41 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 14:32:41 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: <4D7F6A79.3070400@molden.no> Den 14.03.2011 23:02, skrev Sebastian Haase: > Sturla has been writing so much about Fortran recently, and Ondrej now > says he has done the move from C/C++ to Fortran -- I thought Fortran > was dead ... !? ;-) > What am I missing here No, it is just that Fortran receives less hype. If Fortran was dead, why does OpenML define matrices in column major der? Why does ACML and Intel MKL define arrays as column major order? > Apparently (from what I was able to read-up so far) there is a BIG > difference between FORTRAN 77 and F95. > But isn't gcc or gfortran still only supporting F77 ? gfortran supports Fortran 77, Fortran 90, Fortran 95, most of Fortran 2003 and some of Fortran 2008. > How about IntelCC's Fortran ? Is that superior? Intel, Absoft and Portland are among the best Fortran compilers on the market, but gfortran is closing the gap fast. Currently what maily makes Intel ifort and Absoft f95 produce faster binaries than gfortran is actually a faster math library. > 1. How about debugging - does gdb work or is there somthing better ? I use Absoft's Fx debugger. gdb is ok, but it does not know anything about Fortran arrays. > 2. How is the move of the F77 community to F95 in general ? How many > people / projects are switching. F77 is hardly used for new projects. But F77 is merely a subset of F95. > 3. Or is the move rather going like Fortran 77 -> C -> Python -> > Fortran 95 !? ;-) Fortran is good for only one thing, which is number crunching with data stored in arrays. Fortran is much simpler than C, but still just as efficient. There are less room for shooting your own foot off. First, Fortran 90 (and later) has a convinient array syntax, basically the same as NumPy and Matlab. The difference is that as Fortran compilers know these arrays, they can generate optimized code. Think of it as a form of compiled NumPy. In C++ one can make smart array classes that "think like a Fortran compiler" (cf. Blitz++) to avoid many temporary intermediates, but no array class for C++ is as smart as an autovectorizing Fortran compiler (Blitz++ is lagging far behind, and mainly abandonware). Second, Fortran compilers can generate efficient code for parallel processing. This e.g. includes: - autovectorization (SIMD, OpenMP) of array expressions. - broadcasting arrays - fortran pointers. - aliasing disallowed. - a large set of built-in, vectorized math functions. - manual use of OpenMP. - "where" and "forall" contructs. - elemental and pure functions. - internal functions. - vectorized I/O with arrays. For example: Parallel I/O in C or Python can take a lot of boiler-plate code and tunig with threads, asynchronous i/o, etc. In Fortran this is all done by the compiler. If a write or read statement takes an array as argument, the compiler Fortran knows what to do. Sturla From sturla at molden.no Tue Mar 15 09:45:26 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 14:45:26 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: <4D7F6D76.20204@molden.no> Den 14.03.2011 23:10, skrev Matthieu Brucher: > > Intel Fortran is an excellent Fortran compiler. Why is Fortran still > better than C and C++? > - some rules are different, like arrays passed to functions are ALWAYS > supposed to be independent in Fortran, whereas in C, you have to add a > restrict keyword > - due to the last fact, Fortran is a language where its compiler could > do more work (vectorization, autoparallelization...) Fortran 90 (and later) has a big room for "autovectorization" but it not mainly due to aliasing rules. Where, forall, array i/o, pure and elemental functions, intrisic funtiouns etc. are more important. Aliasing rules can also be controlled in ANSI C or C++: g++ will recognize __restrict__, MSVC will recognise __restrict, and MSVC has compiler pragmans, etc., > Nevertheless, when you know C++ correctly, when you want to do > something really efficient, you don't use Fortran. You can be as > efficient as in C++, and you can do fancy stuff (I/O, network...). > Class and templates are also better supported in C++. You can get supernaturally fast i/o on Windows if you use Windows' i/o ports and tune the C-kode. But small Fortran statements (open, close, read, wite) will almost always do the same job. What is fast, and what is fast enough? C++ templates maks binaries almost impossible to debug. Sturla From matthieu.brucher at gmail.com Tue Mar 15 11:05:00 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 15 Mar 2011 16:05:00 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D7F6D76.20204@molden.no> References: <4D7F6D76.20204@molden.no> Message-ID: > > C++ templates maks binaries almost impossible to debug. > Never had an issue with this and all my number crunching code is done through metaprogramming (with vectorization, cache blocking...) So I have a lot of complex template structures, and debugging them is easy. Then, if someone doesn't want to do fancy stuff and uses templates a la Fortran, it's _very_ easy to debug. BTW, instead of Blitzz++, you have vigra and Eigen that are the new equivalent libraries, and you may want to keep an eye on Intel's ArBB. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Mar 15 11:12:10 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 16:12:10 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: <4D7F81CA.30808@molden.no> Den 14.03.2011 23:10, skrev Matthieu Brucher: > - Fortran 95 has an excellent array support, which is not currently > available in C/C++ (perhaps with ArBB?) In C++ you can actually make array libraries that behave almost like a Fortran compiler (cf. Blitz++, Intel Array Building Blocks), but they must be maintained to keep pace with Fortran compilers. > > Class and templates are also better supported in C++. Fortran has generic programming, but not like C++ templates. C++ templates were actually the model of what not to do in Java and .NET generics. Classes: Fortran 2003 has classes. Fortran 95 only have modules, user defined types and operator overloading. Most of my scientific programs don't require fancy OOP. Even with Python, origanizing functions in modules covers 90% of what I need to do. Matlab programmers also have the habit of just collecting functions in .m-files. So I think the "subroutine in a module" abstraction of Fortran is sufficient. If I need OOP, I have Python or Cython :-) Python's i/o is easier to use, if one is used to think of I/O as a data stream and familiar with C standard i/o. Fortran's i/o is vectorized (cf. arrays) and record based (cf. data bases). This allows parallel data import or export. This is because the argument is implicitely parallel (an n-dimensional array), and record based i/o means the order or read and write does not matter. That is unlike streams, where data[n+1] always follows data [n]. It is easy to jump to the conclusion that Fortran's i/o is inferior to C, but there is a reason i/o look that different. With C one could e.g. use the OS to memory map a file, and OpenMP to read the file in parallel (if hardware allows this). In Fortran, all such details are implicit in the i/o statements (e.g. open, read, write, close), the syntax is standardized, and the implementation details for each platform is deferred to the compiler. If we care more about scientific programming than systems programming (incidentally the typical difference between users of Fortran and C), it might not be that interesting to control these low-level implementation datails about i/o. Sturla From charlesr.harris at gmail.com Tue Mar 15 11:33:05 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Mar 2011 09:33:05 -0600 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D7F81CA.30808@molden.no> References: <4D7F81CA.30808@molden.no> Message-ID: On Tue, Mar 15, 2011 at 9:12 AM, Sturla Molden wrote: > Den 14.03.2011 23:10, skrev Matthieu Brucher: > > - Fortran 95 has an excellent array support, which is not currently > > available in C/C++ (perhaps with ArBB?) > > In C++ you can actually make array libraries that behave almost like a > Fortran compiler (cf. Blitz++, Intel Array Building Blocks), but they > must be maintained to keep pace with Fortran compilers. > > There really isn't a satisfactory array library for C++. The fact that every couple of years there is another project to produce one testifies to that fact. And don't overlook that Fortran knows about complex numbers as well as arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 15 12:10:12 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 15 Mar 2011 09:10:12 -0700 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> Message-ID: <4D7F8F64.2020302@noaa.gov> On 3/15/11 8:33 AM, Charles R Harris wrote: > There really isn't a satisfactory array library for C++. The fact that > every couple of years there is another project to produce one testifies > to that fact. And I think not just the fact that there is not one, but that perhaps C++ the language, or maybe the culture, simply doesn't support that way of thinking. I've been slowly arriving to the conclusion that that is no place for C++ in programming. If you really need to twiddle bits, use C. If you need high performance numerics, use Fortran. If you need high level complex data structures, use Python. And the fact that using all these in the same program is pretty easy makes it all possible. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From sienkiew at stsci.edu Tue Mar 15 12:20:52 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 15 Mar 2011 12:20:52 -0400 Subject: [Numpy-discussion] Is this a bug in repr ? Message-ID: <4D7F91E4.1040105@stsci.edu> The usual expectation is that (when possible) repr() returns a value that you can eval() to get the original data back. But, >>> from numpy import * >>> a = array( [ 16.50698631688883822 ] ) >>> b = eval(repr(a)) >>> a-b array([ -3.11116111e-09]) >>> import numpy.testing >>> numpy.testing.assert_array_almost_equal(a,b,decimal=15) Traceback (most recent call last): File "", line 1, in File "/usr/stsci/pyssgdev/2.7/numpy/testing/utils.py", line 775, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/usr/stsci/pyssgdev/2.7/numpy/testing/utils.py", line 618, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 15 decimals (mismatch 100.0%) x: array([ 16.50698632]) y: array([ 16.50698632]) >>> I noticed this because a bunch of tests failed exactly this way. Of course, the problem is that assert_array_almost_equal does not print with the precision that it compared, which in turn happens because it just uses repr() to convert the array. I would expect that repr would print the values at least to the resolution that they are stored, so I think this is a bug. This happens with the current trunk of numpy in python 2.7 on Red Hat Enterprise linux in 32 and 64 bits, and on Macintosh Leopard in 32 bits. I did not try any other configuration. Mark From sturla at molden.no Tue Mar 15 12:21:51 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 17:21:51 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> Message-ID: <4D7F921F.6010902@molden.no> Den 15.03.2011 16:33, skrev Charles R Harris: > > There really isn't a satisfactory array library for C++. The fact that > every couple of years there is another project to produce one > testifies to that fact. In order to be competitive against Fortran 95, an array library for C++ must do all the work of a Fortran compiler. There is no (universally accepted) industry standard array library for C++, which discourage vendors from providing one for their particular platform. > And don't overlook that Fortran knows about complex numbers as well as > arrays. Any decent C++ compiler knows what to do with std::complex template types. Sturla From sturla at molden.no Tue Mar 15 12:34:17 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 17:34:17 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D7F8F64.2020302@noaa.gov> References: <4D7F81CA.30808@molden.no> <4D7F8F64.2020302@noaa.gov> Message-ID: <4D7F9509.4080709@molden.no> Den 15.03.2011 17:10, skrev Christopher Barker: > I've been slowly arriving to the conclusion that that is no place for > C++ in programming. If you really need to twiddle bits, use C. If you > need high performance numerics, use Fortran. If you need high level > complex data structures, use Python It is sad that C++ takes 5 years to master, and this insight comes after using it for another 5 years or more. I regret ever learning C++, but how could I have known that in advance? Sturla From sturla at molden.no Tue Mar 15 12:38:36 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 17:38:36 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F6D76.20204@molden.no> Message-ID: <4D7F960C.1090200@molden.no> Den 15.03.2011 16:05, skrev Matthieu Brucher: > BTW, instead of Blitzz++, you have vigra and Eigen that are the new > equivalent libraries, and you may want to keep an eye on Intel's ArBB. > Intel's ArBB is interesting. But in order for this to work, there must be an idustry standard that other vendors can implement for their platform. You can get an optimized Lapack for any platform. You can only get Intel ArBB for Intel processors. Sturla From yyc at solvcon.net Tue Mar 15 13:01:19 2011 From: yyc at solvcon.net (Yung-Yu Chen) Date: Tue, 15 Mar 2011 13:01:19 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D7F8F64.2020302@noaa.gov> References: <4D7F81CA.30808@molden.no> <4D7F8F64.2020302@noaa.gov> Message-ID: On Tue, Mar 15, 2011 at 12:10, Christopher Barker wrote: > On 3/15/11 8:33 AM, Charles R Harris wrote: > > There really isn't a satisfactory array library for C++. The fact that > > every couple of years there is another project to produce one testifies > > to that fact. > > And I think not just the fact that there is not one, but that perhaps > C++ the language, or maybe the culture, simply doesn't support that way > of thinking. > > I've been slowly arriving to the conclusion that that is no place for > C++ in programming. If you really need to twiddle bits, use C. If you > need high performance numerics, use Fortran. If you need high level > complex data structures, use Python. > > I can't agree more. I really love the capabilities Fortran provides for quick array operations, especially floating-points. What I think Fortran is still lacking is better support of C pointers and structures. I once wrote all number-crunching code in Fortran and then interfaced it from Python, but the long argument list is too annoying. I then translate everything to C, which makes life much easier. I also get the bonus of straightforward porting to CUDA (and OpenCL when it's ready). Column-majoring of Fortran arrays is also annoying, but manageable. C++ has its strengths. But as the mixing-language approach is becoming more and more mature, I think C++ will gradually lose ground. yyc > And the fact that using all these in the same program is pretty easy > makes it all possible. > > -Chris > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Yung-Yu Chen PhD candidate of Mechanical Engineering The Ohio State University, Columbus, Ohio +1 (614) 859 2436 http://solvcon.net/yyc/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Mar 15 13:06:09 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 10:06:09 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Hi, On Sun, Mar 13, 2011 at 12:07 PM, Matthew Brett wrote: > Hi, > > On Sun, Mar 13, 2011 at 11:51 AM, Christoph Gohlke wrote: >> >> >> On 3/13/2011 11:29 AM, Matthew Brett wrote: >>> >>> Hi >>> >>> On Sun, Mar 13, 2011 at 9:54 AM, Christoph Gohlke ?wrote: >>>> >>>> On 3/13/2011 1:57 AM, Matthew Brett wrote: >>>>> >>>>> Hi, >>>>> I have this on my OSX 10.6 system and numpy 1.5.1 and current numpy >>>>> head (30ee1d352): >>>>> >>>>> $ python3.2 >>>>> Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) >>>>> [GCC 4.2.1 (Apple Inc. build 5664)] on darwin >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>>>> >>>>>>>> import numpy as np >>>>>>>> a = np.zeros((1,), dtype=[('f1', 'f')]) >>>>>>>> a['f1'] = 1 >>>>>>>> a['f2'] = 1 >>>>> >>>>> Segmentation fault >>>>> >>>>> All tests pass with np.test() >>>>> >>>>> Expected behavior with same code on python2.6: >>>>> >>>>>>>> a['f2'] = 1 >>>>> >>>>> Traceback (most recent call last): >>>>> ? ?File "", line 1, in >>>>> ValueError: field named f2 not found. >>>>> >>>>> Cheers, >>>>> >>>>> Matthew >>>> >>>> Confirmed on Windows. The crash is in line 816 of mapping.c: >>>> >>>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>>> ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", >>>> ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(index)); >>>> >>>> This works with Python 3.x: >>>> >>>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>>> ? ? ? ? ? ? ? ? ? ? ?"field named %S not found.", >>>> ? ? ? ? ? ? ? ? ? ? ?index); >>> >>> Sure enough, in python3.2: >>> >>>>>> a[b'f2'] = 1 >>> >>> Traceback (most recent call last): >>> ? File "", line 1, in >>> ValueError: field named f2 not found. >>> >>> That is - it is specifically passing a python 3 string that causes the >>> segmentation fault. >>> >>> Is this a release blocker? ?I'm afraid I don't know the code well >>> enough to be confident of the right fix. ?Is there something I can do >>> to help? >> >> Please open a ticket at and refer to this >> discussion and proposed fix. >> >> diff --git a/numpy/core/src/multiarray/mapping.c >> b/numpy/core/src/multiarray/mapping.c >> index 8db85bf..3a72811 100644 >> --- a/numpy/core/src/multiarray/mapping.c >> +++ b/numpy/core/src/multiarray/mapping.c >> @@ -812,10 +812,16 @@ array_ass_sub(PyArrayObject *self, PyObject *index, >> PyObject *op) >> ? ? ? ? ? ? ? ? } >> ? ? ? ? ? ? } >> ? ? ? ? } >> - >> +#if defined(NPY_PY3K) >> + ? ? ? ?PyErr_Format(PyExc_ValueError, >> + ? ? ? ? ? ? ? ? ? ? "field named %S not found.", >> + ? ? ? ? ? ? ? ? ? ? index); >> +#else >> ? ? ? ? PyErr_Format(PyExc_ValueError, >> ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", >> ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(index)); >> +#endif >> + >> ? ? ? ? return -1; >> ? ? } > > http://projects.scipy.org/numpy/ticket/1770 Sorry to ask, and I ask partly because I'm in the middle of a py3k port, but is this the right fix to this problem? I was confused by the presence of the old PyString_AsString function. Best, Matthew From pav at iki.fi Tue Mar 15 13:12:50 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 15 Mar 2011 17:12:50 +0000 (UTC) Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: > Sorry to ask, and I ask partly because I'm in the middle of a py3k port, > but is this the right fix to this problem? I was confused by the > presence of the old PyString_AsString function. It's not a correct fix. The original code seems also wrong ("index" can either be Unicode or Bytes/String object), and will probably bomb when indexing with Unicode strings on Python 2. The right thing to do is to make it show the repr of the "index" object. The PyString_AsString is present, as it's mapped on Py3 to PyBytes_AsString by "npy_3kcompat.h". Pauli From matthew.brett at gmail.com Tue Mar 15 13:23:35 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 10:23:35 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Hi, On Tue, Mar 15, 2011 at 10:12 AM, Pauli Virtanen wrote: > Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >> but is this the right fix to this problem? ?I was confused by the >> presence of the old PyString_AsString function. > > It's not a correct fix. The original code seems also wrong ("index" can > either be Unicode or Bytes/String object), and will probably bomb when > indexing with Unicode strings on Python 2. The right thing to do is to > make it show the repr of the "index" object. OK - I realize I'm being very lazy here but, do you mean: PyErr_Format(PyExc_ValueError, >> "field named %s not found.", >> PyString_AsString(PyObject_Repr(index))); > The PyString_AsString is present, as it's mapped on Py3 to > PyBytes_AsString by "npy_3kcompat.h". Oh - dear - I think I felt a blood vessel pop somewhere in my brain :) See you, Matthew From charlesr.harris at gmail.com Tue Mar 15 13:25:22 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Mar 2011 11:25:22 -0600 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: On Tue, Mar 15, 2011 at 11:12 AM, Pauli Virtanen wrote: > Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: > > Sorry to ask, and I ask partly because I'm in the middle of a py3k port, > > but is this the right fix to this problem? I was confused by the > > presence of the old PyString_AsString function. > > It's not a correct fix. The original code seems also wrong ("index" can > either be Unicode or Bytes/String object), and will probably bomb when > indexing with Unicode strings on Python 2. The right thing to do is to > make it show the repr of the "index" object. > > The PyString_AsString is present, as it's mapped on Py3 to > PyBytes_AsString by "npy_3kcompat.h". > > Well, I applied the fix, so that needs to be fixed ;) What does repr return in python 3k, utf-8? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Mar 15 13:39:58 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Mar 2011 11:39:58 -0600 Subject: [Numpy-discussion] Is this a bug in repr ? In-Reply-To: <4D7F91E4.1040105@stsci.edu> References: <4D7F91E4.1040105@stsci.edu> Message-ID: On Tue, Mar 15, 2011 at 10:20 AM, Mark Sienkiewicz wrote: > The usual expectation is that (when possible) repr() returns a value > that you can eval() to get the original data back. But, > > >>> from numpy import * > >>> a = array( [ 16.50698631688883822 ] ) > >>> b = eval(repr(a)) > >>> a-b > array([ -3.11116111e-09]) > >>> import numpy.testing > >>> numpy.testing.assert_array_almost_equal(a,b,decimal=15) > Traceback (most recent call last): > File "", line 1, in > File "/usr/stsci/pyssgdev/2.7/numpy/testing/utils.py", line 775, in > assert_array_almost_equal > header=('Arrays are not almost equal to %d decimals' % decimal)) > File "/usr/stsci/pyssgdev/2.7/numpy/testing/utils.py", line 618, in > assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 15 decimals > > (mismatch 100.0%) > x: array([ 16.50698632]) > y: array([ 16.50698632]) > >>> > > I noticed this because a bunch of tests failed exactly this way. Of > course, the problem is that assert_array_almost_equal does not print > with the precision that it compared, which in turn happens because it > just uses repr() to convert the array. > > I would expect that repr would print the values at least to the > resolution that they are stored, so I think this is a bug. > > This happens with the current trunk of numpy in python 2.7 on Red Hat > Enterprise linux in 32 and 64 bits, and on Macintosh Leopard in 32 > bits. I did not try any other configuration. > > Yes, I think it is a bug. IIRC, it also shows up for object arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Mar 15 13:55:09 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 15 Mar 2011 18:55:09 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> <4D7F8F64.2020302@noaa.gov> Message-ID: <4D7FA7FD.5070106@molden.no> Den 15.03.2011 18:01, skrev Yung-Yu Chen: > I really love the capabilities Fortran provides for quick array > operations, especially floating-points. What I think Fortran is still > lacking is better support of C pointers and structures. Fortran 90 has user defined types, but they are not ABI compatible with C structs. Fortran 2003 has ISO C bindings for this purpose. There are usually proprietary extensions to Fortran for interfacing with C, it can be ways to declare C structs and pointers (e.g. Cray pointer). Relying on this is not protable though. Another annoying part about Fortran is that it has four different ways of representing array dummy arguments: "explicit shape", "assumed-size", "assumed shape", and "deferred size". Usually one can assume that explicit shape and assumed-size arrays are passed as pointers to the first element, but the standard does not require this. For assumed shape and deferred size arrays, all bets are off, and we have to call this via a Fortran driver/proxy routine. One would think that only one method would suffice, instead of four... Not to mention that it e.g. is illegal to pass an array declared assumed-size to subroutines requesting an assumed-shape array. So they are not even compatible within the Fortran language. Here they are, and observe that they cause different semantics and ABI: Explicit shape: integer :: m,n real :: X(m,n) Assumed size: integer :: m real :: X(m,*) Assumed shape: real :: X(:,:) Deferred shape: real, pointer :: X(:,:) real, allocatable :: X(:,:) Sturla From robert.kern at gmail.com Tue Mar 15 13:55:13 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Mar 2011 12:55:13 -0500 Subject: [Numpy-discussion] Is this a bug in repr ? In-Reply-To: References: <4D7F91E4.1040105@stsci.edu> Message-ID: On Tue, Mar 15, 2011 at 12:39, Charles R Harris wrote: > Yes, I think it is a bug. IIRC, it also shows up for object arrays. It's extremely long-standing, documented, intentional behavior dating back to Numeric. [~] |1> import Numeric [~] |2> a = Numeric.array( [ 16.50698631688883822 ] ) [~] |3> print repr(a) array([ 16.50698632]) You can disagree with the feature, but it's not a bug. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Tue Mar 15 14:07:00 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 15 Mar 2011 18:07:00 +0000 (UTC) Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Tue, 15 Mar 2011 10:23:35 -0700, Matthew Brett wrote: [clip] > OK - I realize I'm being very lazy here but, do you mean: > > PyErr_Format(PyExc_ValueError, >>> "field named %s not found.", >>> PyString_AsString(PyObject_Repr(index))); > >> The PyString_AsString is present, as it's mapped on Py3 to >> PyBytes_AsString by "npy_3kcompat.h". > > Oh - dear - I think I felt a blood vessel pop somewhere in my brain :) This was an "answer" to your question as I understood it: "PyString_AsString is no longer a part of the API on Python 3.x. So how come this code can work on Python 3 if it appears here?" Pauli From sienkiew at stsci.edu Tue Mar 15 14:10:41 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 15 Mar 2011 14:10:41 -0400 Subject: [Numpy-discussion] Is this a bug in repr ? In-Reply-To: References: <4D7F91E4.1040105@stsci.edu> Message-ID: <4D7FABA1.7040804@stsci.edu> Robert Kern wrote: > On Tue, Mar 15, 2011 at 12:39, Charles R Harris > wrote: > > >> Yes, I think it is a bug. IIRC, it also shows up for object arrays. >> > > It's extremely long-standing, documented, intentional behavior dating > back to Numeric. > > [~] > |1> import Numeric > > [~] > |2> a = Numeric.array( [ 16.50698631688883822 ] ) > > [~] > |3> print repr(a) > array([ 16.50698632]) > > > You can disagree with the feature, but it's not a bug. > So it is needed to maintain backward compatibility? (Still?) In that case, would you agree that it is a bug for assert_array_almost_equal to use repr() to display the arrays, since it is printing identical values and saying they are different? Or is there also a reason to do that? From matthew.brett at gmail.com Tue Mar 15 14:16:49 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 11:16:49 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Hi, On Tue, Mar 15, 2011 at 11:07 AM, Pauli Virtanen wrote: > Tue, 15 Mar 2011 10:23:35 -0700, Matthew Brett wrote: > [clip] >> OK - I realize I'm being very lazy here but, do you mean: >> >> ? ? ? ? PyErr_Format(PyExc_ValueError, >>>> ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", >>>> ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(PyObject_Repr(index))); >> >>> The PyString_AsString is present, as it's mapped on Py3 to >>> PyBytes_AsString by "npy_3kcompat.h". >> >> Oh - dear - I think I felt a blood vessel pop somewhere in my brain :) > > This was an "answer" to your question as I understood it: > "PyString_AsString is no longer a part of the API on Python 3.x. So how > come this code can work on Python 3 if it appears here?" Oh - dear - again. Yes it was a helpful answer and direct to my question - implication otherwise entirely accidental. The blood vessel was only because my brain was already twisted from trying to unlearn the correspondence between string and byte in python 3... I fear that my suggestion as to what you meant by repr doesn't make sense though. See you, Matthew From mwwiebe at gmail.com Tue Mar 15 14:26:33 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 15 Mar 2011 11:26:33 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers wrote: > On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: > > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris > > wrote: > >> > >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: > >>> > >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers > >>> wrote: > >>>> > >>>> Hi all, > >>>> > >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the > >>>> first beta. So please get your last commits for 1.6 in by Monday > >>>> evening. > >>>> > >>>> Also, please review and add to the 1.6.0 release notes. I put in > >>>> headers for several items that need a few lines in the notes, I hope > >>>> this can be filled in by the authors of those features (Charles: > >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of > >>>> stuff). > >>> > >>> I've added a few more things, and made a small change to the iterator > >>> construction API that I've discovered is useful, but would be more > difficult > >>> to do later. The Python exposure of the iterator is renamed from > 'newiter' > >>> to 'nditer', is that a reasonable name or does anyone have a better > >>> suggestion? > >> > >> I think nditer is fine, certainly better than newiter. I don't see where > >> nditer appears in the changes though, the test still uses newiter. > > > > I didn't rename the files, I can do that too. > > Hi Mark, I see you just did this, but is there anything else you > want/need to do? If it's necessary I can postpone the first beta by a > couple of days. Better that than rush things too much and end up with > an API you have reservations about. > I pushed one more small API change to PyArray_NewLikeArray, adding a 'subok' parameter which lets you disable sub-types. The things missing still are documentation (maybe others can help?). The Python nditer exposure is undocumented, as well as the new parameters to ufuncs (full list: 'casting', 'out', 'order', 'subok', 'dtype'). -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Mar 15 14:26:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Mar 2011 13:26:14 -0500 Subject: [Numpy-discussion] Is this a bug in repr ? In-Reply-To: <4D7FABA1.7040804@stsci.edu> References: <4D7F91E4.1040105@stsci.edu> <4D7FABA1.7040804@stsci.edu> Message-ID: On Tue, Mar 15, 2011 at 13:10, Mark Sienkiewicz wrote: > Robert Kern wrote: >> On Tue, Mar 15, 2011 at 12:39, Charles R Harris >> wrote: >> >> >>> Yes, I think it is a bug. IIRC, it also shows up for object arrays. >>> >> >> It's extremely long-standing, documented, intentional behavior dating >> back to Numeric. >> >> [~] >> |1> import Numeric >> >> [~] >> |2> a = Numeric.array( [ ?16.50698631688883822 ] ) >> >> [~] >> |3> print repr(a) >> array([ 16.50698632]) >> >> >> You can disagree with the feature, but it's not a bug. >> > > So it is needed to maintain backward compatibility? ?(Still?) No, it's an intentional feature, and the reasons for it then still pertain now. It's a pragmatic compromise to display useful amounts of information at the interactive prompt without overwhelming the user with what is usually unimportant detail that would take up excessive space. We also elide elements using "..." when there are too many elements to display in a reasonable amount of time for the same reason. You can control these settings using np.set_printoptions(). > In that case, would you agree that it is a bug for > assert_array_almost_equal to use repr() to display the arrays, since it > is printing identical values and saying they are different? ?Or is there > also a reason to do that? It should probably use np.array_repr(x, precision=16) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From nadavh at visionsense.com Tue Mar 15 14:31:39 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 15 Mar 2011 11:31:39 -0700 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <4D7E72A2.6010509@uci.edu> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu>, <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local>, <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local>, <4D7BF58F.4010005@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local>, <4D7E72A2.6010509@uci.edu> Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5241@VA3DIAXVS361.RED001.local> Just downloaded and got the same problems. I'll try to debug and provide a decent analysis, but it would take some days as I am busy with other things. Thank you, Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] Sent: 14 March 2011 21:55 To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? On 3/12/2011 9:56 PM, Nadav Horesh wrote: > This lead to another error probably due to line 68 in map.h. As much as I could trace it, ob_type is a member of PyObject, not of PyTypeObject. I have no clue how to resolve this. I just tried PIL-1.1.7-py3 on an Ubuntu 64 bit system: after one change to map.c it builds OK (without sane support) and passes all tests but one numpy int64 bit related test. Please download again from to make sure you have the same sources and try build again. Christoph > > Nadav. > ________________________________________ > From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] > Sent: 13 March 2011 00:37 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? > > On 3/12/2011 12:47 PM, Nadav Horesh wrote: >> After the replacement of ? with o, the installation went without errors, but: >> >> nadav at nadav_home ~ $ python3 >> Python 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >> [GCC 4.4.4] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import _imaging >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: /usr/lib64/python3.1/site-packages/PIL/_imaging.so: undefined symbol: Py_FindMethod > > Py_FindMethod should be excluded by `#ifndef PY3` or similar > preprocessor statements. There is a typo in map.c line 65: change > `#ifdef PY3` to `#ifndef PY3` and clean your build directory before > rebuilding. > > Christoph > >> >> Thank you, >> >> Nadav. >> ________________________________________ >> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Christoph Gohlke [cgohlke at uci.edu] >> Sent: 12 March 2011 21:49 >> To: numpy-discussion at scipy.org >> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >> >> On 3/12/2011 8:45 AM, Nadav Horesh wrote: >>> I forgot to mention that I work on linux (gentoo x86-64).Here are my achievements till now: >>> >>> 1. PythonMagick: Needs boost which I do not have it avaiable on python3 >> Boost works on Python 3.1. You might need to compile it. >>> 2. Pygame: I have the stable version(1.9.1) should it work? >> You need the developer version from svn. >>> 3. FreeImage: I installed FreeImagePy on python3, but it doesn't work yet. >> FreeImagePy is unmaintained, does not work on Python 3, and has problems >> on 64 bit platforms. Just wrap the functions you need in ctypes. >>> 4. PIL: I patched setup.py and map.c so "python3 setup.py build" is working, but: >> Try replace "Hans H?ggstr?m" with "Hans Haggstrom" in PIL/WalImageFile.py >> >> Christoph >> >>> >>> nadav at nadav_home /dev/shm/PIL-1.1.7-py3 $ sudo python3.1 setup.py install >>> /usr/lib64/python3.1/distutils/dist.py:259: UserWarning: Unknown distribution option: 'ext_comp_args' >>> warnings.warn(msg) >>> running install >>> running build >>> running build_py >>> running build_ext >>> -------------------------------------------------------------------- >>> PIL 1.1.7 SETUP SUMMARY >>> -------------------------------------------------------------------- >>> version 1.1.7 >>> platform linux2 3.1.3 (r313:86834, Feb 25 2011, 11:08:33) >>> [GCC 4.4.4] >>> -------------------------------------------------------------------- >>> --- TKINTER support available >>> --- JPEG support available >>> --- ZLIB (PNG/ZIP) support available >>> --- FREETYPE2 support available >>> --- LITTLECMS support available >>> >>> . >>> . >>> . >>> >>> byte-compiling /usr/lib64/python3.1/site-packages/PIL/WalImageFile.py to WalImageFile.pyc >>> Traceback (most recent call last): >>> File "setup.py", line 520, in >>> setup(*(), **configuration) # old school :-) >>> File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup >>> dist.run_commands() >>> File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands >>> self.run_command(cmd) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install.py", line 592, in run >>> self.run_command(cmd_name) >>> File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command >>> self.distribution.run_command(command) >>> File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command >>> cmd_obj.run() >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 98, in run >>> self.byte_compile(outfiles) >>> File "/usr/lib64/python3.1/distutils/command/install_lib.py", line 135, in byte_compile >>> dry_run=self.dry_run) >>> File "/usr/lib64/python3.1/distutils/util.py", line 560, in byte_compile >>> compile(file, cfile, dfile) >>> File "/usr/lib64/python3.1/py_compile.py", line 137, in compile >>> codestring = f.read() >>> File "/usr/lib64/python3.1/codecs.py", line 300, in decode >>> (result, consumed) = self._buffer_decode(data, self.errors, final) >>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1909: invalid continuation byte >>> >>> >>> >>> Any idea on how to correct it? Any elegant way to avoid byte compiling? >>> >>> Nadav >>> >>> >>> ________________________________________ >>> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus [zachary.pincus at yale.edu] >>> Sent: 12 March 2011 14:35 >>> To: Discussion of Numerical Python >>> Subject: Re: [Numpy-discussion] [OT] any image io module that works with python3? >>> >>> Here's a ctypes interface to FreeImage that I wrote a while back and >>> was since cleaned up (and maintained) by the scikits.image folk: >>> >>> https://github.com/stefanv/scikits.image/blob/master/scikits/image/io/_plugins/freeimage_plugin.py >>> >>> If it doesn't work out of the box on python 3, then it should be >>> pretty simple to fix. >>> >>> Zach >>> >>> >>> >>> On Mar 12, 2011, at 4:40 AM, Christoph Gohlke wrote: >>> >>>> >>>> >>>> On 3/12/2011 1:08 AM, Nadav Horesh wrote: >>>>> Having numpy, scipy, and matplotlib working reasonably with >>>>> python3, a >>>>> major piece of code I miss for a major python3 migration is an >>>>> image IO. >>>>> I found that pylab's imread works fine for png image, but I need to >>>>> read >>>>> all the other image format as well as png and jpeg output. >>>>> Any hints (including advices how easyly construct my own module) are >>>>> appreciated. >>>>> Nadav. >>>>> >>>> >>>> On Windows, PIL (private port at >>>> ), PythonMagick >>>> , and pygame 1.9.2pre >>>> are working reasonably well for image IO. Also >>>> the FreeImage library is easy to >>>> use >>>> with ctypes. >>>> >>>> Christoph >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Tue Mar 15 14:31:43 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Mar 2011 12:31:43 -0600 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: > On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> On Mon, Mar 14, 2011 at 2:22 PM, Mark Wiebe wrote: >> > On Sun, Mar 13, 2011 at 7:47 PM, Charles R Harris >> > wrote: >> >> >> >> On Sun, Mar 13, 2011 at 8:23 PM, Mark Wiebe wrote: >> >>> >> >>> On Sun, Mar 13, 2011 at 1:22 AM, Ralf Gommers >> >>> wrote: >> >>>> >> >>>> Hi all, >> >>>> >> >>>> On Tuesday (~2am GMT) I plan to create the 1.6.x branch and tag the >> >>>> first beta. So please get your last commits for 1.6 in by Monday >> >>>> evening. >> >>>> >> >>>> Also, please review and add to the 1.6.0 release notes. I put in >> >>>> headers for several items that need a few lines in the notes, I hope >> >>>> this can be filled in by the authors of those features (Charles: >> >>>> Legendre polynomials, Pearu: assumed shape arrays, Mark: a bunch of >> >>>> stuff). >> >>> >> >>> I've added a few more things, and made a small change to the iterator >> >>> construction API that I've discovered is useful, but would be more >> difficult >> >>> to do later. The Python exposure of the iterator is renamed from >> 'newiter' >> >>> to 'nditer', is that a reasonable name or does anyone have a better >> >>> suggestion? >> >> >> >> I think nditer is fine, certainly better than newiter. I don't see >> where >> >> nditer appears in the changes though, the test still uses newiter. >> > >> > I didn't rename the files, I can do that too. >> >> Hi Mark, I see you just did this, but is there anything else you >> want/need to do? If it's necessary I can postpone the first beta by a >> couple of days. Better that than rush things too much and end up with >> an API you have reservations about. >> > > I pushed one more small API change to PyArray_NewLikeArray, adding a > 'subok' parameter which lets you disable sub-types. The things missing still > are documentation (maybe others can help?). The Python nditer exposure is > undocumented, as well as the new parameters to ufuncs (full list: 'casting', > 'out', 'order', 'subok', 'dtype'). > > We should probably postpone the beta by a few days, there are some other loose ends floating about. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Mar 15 14:34:06 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 15 Mar 2011 11:34:06 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: <4D7FB11E.2010204@uci.edu> On 3/15/2011 10:12 AM, Pauli Virtanen wrote: > Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >> but is this the right fix to this problem? I was confused by the >> presence of the old PyString_AsString function. > > It's not a correct fix. The original code seems also wrong ("index" can > either be Unicode or Bytes/String object), and will probably bomb when > indexing with Unicode strings on Python 2. The right thing to do is to > make it show the repr of the "index" object. > > The PyString_AsString is present, as it's mapped on Py3 to > PyBytes_AsString by "npy_3kcompat.h". > > Pauli I think the proposed patch does correctly fix ticket #1770 and the OP's problem under Python 3. On Python 3.x str() and repr() are the same for byte strings: >>> str(b's') "b's'" >>> repr(b's') "b's'" I agree that the code is broken for Python 2.x when indexing with a byte object. Christoph From cgohlke at uci.edu Tue Mar 15 14:50:45 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 15 Mar 2011 11:50:45 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: <4D7FB11E.2010204@uci.edu> References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> <4D7FB11E.2010204@uci.edu> Message-ID: <4D7FB505.4020504@uci.edu> On 3/15/2011 11:34 AM, Christoph Gohlke wrote: > > > On 3/15/2011 10:12 AM, Pauli Virtanen wrote: >> Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >>> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >>> but is this the right fix to this problem? I was confused by the >>> presence of the old PyString_AsString function. >> >> It's not a correct fix. The original code seems also wrong ("index" can >> either be Unicode or Bytes/String object), and will probably bomb when >> indexing with Unicode strings on Python 2. The right thing to do is to >> make it show the repr of the "index" object. >> >> The PyString_AsString is present, as it's mapped on Py3 to >> PyBytes_AsString by "npy_3kcompat.h". >> >> Pauli > > I think the proposed patch does correctly fix ticket #1770 and the OP's > problem under Python 3. > > On Python 3.x str() and repr() are the same for byte strings: > >>>> str(b's') > "b's'" >>>> repr(b's') > "b's'" > > I agree that the code is broken for Python 2.x when indexing with a byte > object. sorry, this should read: I agree that the original code is broken for Python 2.x when indexing with a unicode object. Christoph > > Christoph > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From seb.haase at gmail.com Tue Mar 15 15:25:19 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 15 Mar 2011 20:25:19 +0100 Subject: [Numpy-discussion] [Numpy-svn] aada93: ENH: Add 'subok' parameter to PyArray_NewLikeArray... In-Reply-To: <20110315182237.0428D421F8@smtp1.rs.github.com> References: <20110315182237.0428D421F8@smtp1.rs.github.com> Message-ID: On Tue, Mar 15, 2011 at 7:22 PM, wrote: > Branch: refs/heads/master > Home: ? https://github.com/numpy/numpy > > Commit: aada93306acfb4e2eb816faf32652edf8825cf45 > ? ?https://github.com/numpy/numpy/commit/aada93306acfb4e2eb816faf32652edf8825cf45 > Author: Mark Wiebe > Date: ? 2011-03-15 (Tue, 15 Mar 2011) > > Changed paths: > ?M doc/source/reference/c-api.array.rst > ?M numpy/add_newdocs.py > ?M numpy/core/numeric.py > ?M numpy/core/src/multiarray/convert.c > ?M numpy/core/src/multiarray/ctors.c > ?M numpy/core/src/multiarray/multiarraymodule.c > ?M numpy/core/src/umath/ufunc_object.c > ?M numpy/core/tests/test_numeric.py > > Log Message: > ----------- > ENH: Add 'subok' parameter to PyArray_NewLikeArray, np.empty_like, np.zeros_like, and np.ones_like > > This way, the sub-type can be avoided if necessary. This helps mitigate, > but doesn't fix, ticket #1753, by allowing "b = np.empty_like(a, subok=False)". > I'm really not in a position to comment on the depths of the numpy API, but my understanding of np._like( ... ) was that it would create always "normal" ndarrays just taking shape and dtype from the given array. So what should the interpretation of "subok" be ? Can you elaborate ... ? Thanks, Sebastian Haase From wesmckinn at gmail.com Tue Mar 15 15:32:05 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 15 Mar 2011 15:32:05 -0400 Subject: [Numpy-discussion] [Numpy-svn] aada93: ENH: Add 'subok' parameter to PyArray_NewLikeArray... In-Reply-To: References: <20110315182237.0428D421F8@smtp1.rs.github.com> Message-ID: On Tue, Mar 15, 2011 at 3:25 PM, Sebastian Haase wrote: > On Tue, Mar 15, 2011 at 7:22 PM, ? wrote: >> Branch: refs/heads/master >> Home: ? https://github.com/numpy/numpy >> >> Commit: aada93306acfb4e2eb816faf32652edf8825cf45 >> ? ?https://github.com/numpy/numpy/commit/aada93306acfb4e2eb816faf32652edf8825cf45 >> Author: Mark Wiebe >> Date: ? 2011-03-15 (Tue, 15 Mar 2011) >> >> Changed paths: >> ?M doc/source/reference/c-api.array.rst >> ?M numpy/add_newdocs.py >> ?M numpy/core/numeric.py >> ?M numpy/core/src/multiarray/convert.c >> ?M numpy/core/src/multiarray/ctors.c >> ?M numpy/core/src/multiarray/multiarraymodule.c >> ?M numpy/core/src/umath/ufunc_object.c >> ?M numpy/core/tests/test_numeric.py >> >> Log Message: >> ----------- >> ENH: Add 'subok' parameter to PyArray_NewLikeArray, np.empty_like, np.zeros_like, and np.ones_like >> >> This way, the sub-type can be avoided if necessary. This helps mitigate, >> but doesn't fix, ticket #1753, by allowing "b = np.empty_like(a, subok=False)". >> > I'm really not in a position to comment on the depths of the numpy API, > but my understanding of np._like( ... ) ?was that it would create always > "normal" ndarrays just taking shape and dtype from the given array. > So what should the interpretation of "subok" be ? > Can you elaborate ... ? > > Thanks, > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I know that things like numpy.ma and pandas are directly impacted by this change-- in NumPy < 1.6 many API functions cannot be used on subclasses because they discard any additional information you wish to be passed on (like the mask in a masked array, for example). From mwwiebe at gmail.com Tue Mar 15 15:33:04 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 15 Mar 2011 12:33:04 -0700 Subject: [Numpy-discussion] [Numpy-svn] aada93: ENH: Add 'subok' parameter to PyArray_NewLikeArray... In-Reply-To: References: <20110315182237.0428D421F8@smtp1.rs.github.com> Message-ID: On Tue, Mar 15, 2011 at 12:25 PM, Sebastian Haase wrote: > > > > > Log Message: > > ----------- > > ENH: Add 'subok' parameter to PyArray_NewLikeArray, np.empty_like, > np.zeros_like, and np.ones_like > > > > This way, the sub-type can be avoided if necessary. This helps mitigate, > > but doesn't fix, ticket #1753, by allowing "b = np.empty_like(a, > subok=False)". > > > I'm really not in a position to comment on the depths of the numpy API, > but my understanding of np._like( ... ) was that it would create > always > "normal" ndarrays just taking shape and dtype from the given array. > So what should the interpretation of "subok" be ? > Can you elaborate ... ? > You're understanding is half-right. There was previously an inconsistency, where zeros_like and empty_like did what you expect, while ones_like kept the subtype, see this ticket: http://projects.scipy.org/numpy/ticket/929. Because ones_like is a ufunc, switching it to behave like the others did is much harder than going the other way, and intuitively I would expect "*_like" to produce a matrix when given a matrix, so I fixed it to preserve the subtype. This patch extends the functions to allow for a choice between the previous two behaviors. The parameter "subok" is exactly like "subok" in the array constructor, the name was chosen for consistency. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Mar 15 18:38:07 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Mar 2011 00:38:07 +0200 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5241@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> <4D7B3F90.8080702@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5231@VA3DIAXVS361.RED001.local> <4D7BCE4F.7090404@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5234@VA3DIAXVS361.RED001.local> <4D7BF58F.4010005@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5235@VA3DIAXVS361.RED001.local> <4D7E72A2.6010509@uci.edu> <26FC23E7C398A64083C980D16001012D1CE03D5241@VA3DIAXVS361.RED001.local> Message-ID: On Tue, Mar 15, 2011 at 8:31 PM, Nadav Horesh wrote: > Just downloaded and got the same problems. I'll try to debug and provide a decent analysis, but it would take some days as I am busy with other things. Thanks, that'd be appreciated; I am able to load images using freeimage without a problem in Python 3 now. Cheers St?fan From ndbecker2 at gmail.com Tue Mar 15 19:01:35 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 15 Mar 2011 19:01:35 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] References: <4D7F81CA.30808@molden.no> Message-ID: Charles R Harris wrote: > On Tue, Mar 15, 2011 at 9:12 AM, Sturla Molden wrote: > >> Den 14.03.2011 23:10, skrev Matthieu Brucher: >> > - Fortran 95 has an excellent array support, which is not currently >> > available in C/C++ (perhaps with ArBB?) >> >> In C++ you can actually make array libraries that behave almost like a >> Fortran compiler (cf. Blitz++, Intel Array Building Blocks), but they >> must be maintained to keep pace with Fortran compilers. >> >> > There really isn't a satisfactory array library for C++. The fact that every > couple of years there is another project to produce one testifies to that > fact. And don't overlook that Fortran knows about complex numbers as well as > arrays. > > Chuck This is partly (largely?) because different users want different things from the array abstraction. I don't know if I can even start to list the differences, but let's try: * 1-d, 2-d only or N-d?? * support for slice views? What exactly kind of support? * semantics like numpy, that make many operations avoid copy? * what support for arithmetic operations? Do views support arithmetic operations? * expression templates? * How's the performance with indexing? Multi-D indexing? How about iteration? * What are the semantics of iterators? I don't think I've seen 2 libs that treat multi-d iterators the same way (and I've tried a lot of them). From matthew.brett at gmail.com Tue Mar 15 20:13:37 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 17:13:37 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: Hi, On Tue, Mar 15, 2011 at 10:23 AM, Matthew Brett wrote: > Hi, > > On Tue, Mar 15, 2011 at 10:12 AM, Pauli Virtanen wrote: >> Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >>> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >>> but is this the right fix to this problem? ?I was confused by the >>> presence of the old PyString_AsString function. >> >> It's not a correct fix. The original code seems also wrong ("index" can >> either be Unicode or Bytes/String object), and will probably bomb when >> indexing with Unicode strings on Python 2. The right thing to do is to >> make it show the repr of the "index" object. > > OK - I realize I'm being very lazy here but, do you mean: > > ? ? ? ?PyErr_Format(PyExc_ValueError, >>> ? ? ? ? ? ? ? ? ? ? ?"field named %s not found.", >>> ? ? ? ? ? ? ? ? ? ? ?PyString_AsString(PyObject_Repr(index))); Being less lazy, and having read the cpython source, and read Christoph's mail more carefully, I believe Christoph's patch is correct... Unicode indexing of structured array fields doesn't raise an error on python 2.x; I assume because PyString_AsString is returning a char* using the Unicode default encoding, as per the docs. Thanks, Matthew From matthew.brett at gmail.com Tue Mar 15 20:24:37 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 17:24:37 -0700 Subject: [Numpy-discussion] structured array indexing oddity Message-ID: Hi, I just wrote a short test for indexing into structured arrays with strings and found this: In [4]: a = np.zeros((1,), dtype=[('f1', 'i4')]) In [5]: a['f1'] Out[5]: array([0]) In [6]: a['f2'] # not present -> error --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/mb312/ in () ValueError: field named f2 not found. In [7]: a[0]['f1'] # OK Out[7]: 0 In [8]: a[0]['f2'] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /Users/mb312/ in () IndexError: invalid index It seems odd to raise an 'IndexError' without a message about the field name in the second case, and a ValueError with a good message in the first. Do y'all agree? Matthew From cgohlke at uci.edu Tue Mar 15 20:30:10 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 15 Mar 2011 17:30:10 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> Message-ID: <4D800492.4060909@uci.edu> On 3/15/2011 5:13 PM, Matthew Brett wrote: > Hi, > > On Tue, Mar 15, 2011 at 10:23 AM, Matthew Brett wrote: >> Hi, >> >> On Tue, Mar 15, 2011 at 10:12 AM, Pauli Virtanen wrote: >>> Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >>>> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >>>> but is this the right fix to this problem? I was confused by the >>>> presence of the old PyString_AsString function. >>> >>> It's not a correct fix. The original code seems also wrong ("index" can >>> either be Unicode or Bytes/String object), and will probably bomb when >>> indexing with Unicode strings on Python 2. The right thing to do is to >>> make it show the repr of the "index" object. >> >> OK - I realize I'm being very lazy here but, do you mean: >> >> PyErr_Format(PyExc_ValueError, >>>> "field named %s not found.", >>>> PyString_AsString(PyObject_Repr(index))); > > Being less lazy, and having read the cpython source, and read > Christoph's mail more carefully, I believe Christoph's patch is > correct... > > Unicode indexing of structured array fields doesn't raise an error on > python 2.x; I assume because PyString_AsString is returning a char* > using the Unicode default encoding, as per the docs. > I think the patch is correct for Python 3 but, as Pauli pointed out, the original code can crash also under Python 2.x when indexing with an unicode string that contains non-ascii7 characters, which seems much less likely and apparently has been undetected for quite a while. For example, this crashes for me on Python 2.7: import numpy as np a = np.zeros((1,), dtype=[('f1', 'f')]) a[u's'] = 1 # works a[u'?'] = 1 # crash So, the proposed patch works for Python 3, but there could be a better patch fixing also the corner case on Python 2. Christoph > Thanks, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthew.brett at gmail.com Tue Mar 15 20:55:32 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 17:55:32 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: <4D800492.4060909@uci.edu> References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> <4D800492.4060909@uci.edu> Message-ID: Hi, On Tue, Mar 15, 2011 at 5:30 PM, Christoph Gohlke wrote: > > > On 3/15/2011 5:13 PM, Matthew Brett wrote: >> Hi, >> >> On Tue, Mar 15, 2011 at 10:23 AM, Matthew Brett ?wrote: >>> Hi, >>> >>> On Tue, Mar 15, 2011 at 10:12 AM, Pauli Virtanen ?wrote: >>>> Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >>>>> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >>>>> but is this the right fix to this problem? ?I was confused by the >>>>> presence of the old PyString_AsString function. >>>> >>>> It's not a correct fix. The original code seems also wrong ("index" can >>>> either be Unicode or Bytes/String object), and will probably bomb when >>>> indexing with Unicode strings on Python 2. The right thing to do is to >>>> make it show the repr of the "index" object. >>> >>> OK - I realize I'm being very lazy here but, do you mean: >>> >>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>>>> ? ? ? ? ? ? ? ? ? ? ? "field named %s not found.", >>>>> ? ? ? ? ? ? ? ? ? ? ? PyString_AsString(PyObject_Repr(index))); >> >> Being less lazy, and having read the cpython source, and read >> Christoph's mail more carefully, I believe Christoph's patch is >> correct... >> >> Unicode indexing of structured array fields doesn't raise an error on >> python 2.x; I assume because PyString_AsString is returning a char* >> using the Unicode default encoding, as per the docs. >> > > I think the patch is correct for Python 3 but, as Pauli pointed out, the > original code can crash also under Python 2.x when indexing with an > unicode string that contains non-ascii7 characters, which seems much > less likely and apparently has been undetected for quite a while. For > example, this crashes for me on Python 2.7: > > import numpy as np > a = np.zeros((1,), dtype=[('f1', 'f')]) > a[u's'] = 1 ?# works > a[u'?'] = 1 ?# crash > > So, the proposed patch works for Python 3, but there could be a better > patch fixing also the corner case on Python 2. Thanks. How about something like the check further up the file: if (PyUnicode_Check(op)) { temp = PyUnicode_AsUnicodeEscapeString(op); } PyErr_Format(PyExc_ValueError, "field named %s not found.", PyBytes_AsString(temp)); ? I'm happy to submit a pull request with tests if that kind of thing looks sensible to y'all, Matthew From scopatz at gmail.com Tue Mar 15 21:21:01 2011 From: scopatz at gmail.com (Anthony Scopatz) Date: Tue, 15 Mar 2011 20:21:01 -0500 Subject: [Numpy-discussion] structured array indexing oddity In-Reply-To: References: Message-ID: Hi Matt, Actually, based on what is happening under the covers I think this behavior makes sense. a['f2'] would in theory grab the whole column, and so 'f2' not existing is a field error. a[0] on the other hand grabs the first row from the full structured array and treats this as (key, value) pair, much like a dictionary. Thus when 'f2' doesn't exist in a[0]['f2'], it means it cannot find the key and is therefore an IndexError. I hope this helps Be Well Anthony On Tue, Mar 15, 2011 at 7:24 PM, Matthew Brett wrote: > Hi, > > I just wrote a short test for indexing into structured arrays with > strings and found this: > > In [4]: a = np.zeros((1,), dtype=[('f1', 'i4')]) > In [5]: a['f1'] > Out[5]: array([0]) > In [6]: a['f2'] # not present -> error > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > > /Users/mb312/ in () > > ValueError: field named f2 not found. > In [7]: a[0]['f1'] # OK > Out[7]: 0 > In [8]: a[0]['f2'] > --------------------------------------------------------------------------- > IndexError Traceback (most recent call last) > > /Users/mb312/ in () > > IndexError: invalid index > > It seems odd to raise an 'IndexError' without a message about the > field name in the second case, and a ValueError with a good message in > the first. > > Do y'all agree? > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Mar 15 23:29:09 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 16 Mar 2011 11:29:09 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris wrote: > > > On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: > >> On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> >>> Hi Mark, I see you just did this, but is there anything else you >>> want/need to do? If it's necessary I can postpone the first beta by a >>> couple of days. Better that than rush things too much and end up with >>> an API you have reservations about. >>> >> >> I pushed one more small API change to PyArray_NewLikeArray, adding a >> 'subok' parameter which lets you disable sub-types. The things missing still >> are documentation (maybe others can help?). The Python nditer exposure is >> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >> 'out', 'order', 'subok', 'dtype'). >> >> Okay, I'll write some docs for the nditer object. > We should probably postpone the beta by a few days, there are some other > loose ends floating about. > > Besides documentation the only thing I can think of is the structured array non-existing filed segfault thing. And perhaps #1619 would be good to have fixed, but not essential for a beta IMHO. Anything else? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Mar 16 00:02:59 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Mar 2011 22:02:59 -0600 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 9:29 PM, Ralf Gommers wrote: > > > On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >> >>> On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers < >>> ralf.gommers at googlemail.com> wrote: >>> >>>> >>>> Hi Mark, I see you just did this, but is there anything else you >>>> want/need to do? If it's necessary I can postpone the first beta by a >>>> couple of days. Better that than rush things too much and end up with >>>> an API you have reservations about. >>>> >>> >>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>> 'subok' parameter which lets you disable sub-types. The things missing still >>> are documentation (maybe others can help?). The Python nditer exposure is >>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>> 'out', 'order', 'subok', 'dtype'). >>> >>> > Okay, I'll write some docs for the nditer object. > > >> We should probably postpone the beta by a few days, there are some other >> loose ends floating about. >> >> Besides documentation the only thing I can think of is the structured > array non-existing filed segfault thing. And perhaps #1619 would be good to > have fixed, but not essential for a beta IMHO. Anything else? > > Just that things that go in at the last moment should be let sit a bit to settle. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Mar 16 00:48:19 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 21:48:19 -0700 Subject: [Numpy-discussion] Segfault with python 3.2 structured array non-existent field In-Reply-To: References: <4D7CF6B2.7070202@uci.edu> <4D7D121B.9060902@uci.edu> <4D800492.4060909@uci.edu> Message-ID: Hi, On Tue, Mar 15, 2011 at 5:55 PM, Matthew Brett wrote: > Hi, > > On Tue, Mar 15, 2011 at 5:30 PM, Christoph Gohlke wrote: >> >> >> On 3/15/2011 5:13 PM, Matthew Brett wrote: >>> Hi, >>> >>> On Tue, Mar 15, 2011 at 10:23 AM, Matthew Brett ?wrote: >>>> Hi, >>>> >>>> On Tue, Mar 15, 2011 at 10:12 AM, Pauli Virtanen ?wrote: >>>>> Tue, 15 Mar 2011 10:06:09 -0700, Matthew Brett wrote: >>>>>> Sorry to ask, and I ask partly because I'm in the middle of a py3k port, >>>>>> but is this the right fix to this problem? ?I was confused by the >>>>>> presence of the old PyString_AsString function. >>>>> >>>>> It's not a correct fix. The original code seems also wrong ("index" can >>>>> either be Unicode or Bytes/String object), and will probably bomb when >>>>> indexing with Unicode strings on Python 2. The right thing to do is to >>>>> make it show the repr of the "index" object. >>>> >>>> OK - I realize I'm being very lazy here but, do you mean: >>>> >>>> ? ? ? ? PyErr_Format(PyExc_ValueError, >>>>>> ? ? ? ? ? ? ? ? ? ? ? "field named %s not found.", >>>>>> ? ? ? ? ? ? ? ? ? ? ? PyString_AsString(PyObject_Repr(index))); >>> >>> Being less lazy, and having read the cpython source, and read >>> Christoph's mail more carefully, I believe Christoph's patch is >>> correct... >>> >>> Unicode indexing of structured array fields doesn't raise an error on >>> python 2.x; I assume because PyString_AsString is returning a char* >>> using the Unicode default encoding, as per the docs. >>> >> >> I think the patch is correct for Python 3 but, as Pauli pointed out, the >> original code can crash also under Python 2.x when indexing with an >> unicode string that contains non-ascii7 characters, which seems much >> less likely and apparently has been undetected for quite a while. For >> example, this crashes for me on Python 2.7: >> >> import numpy as np >> a = np.zeros((1,), dtype=[('f1', 'f')]) >> a[u's'] = 1 ?# works >> a[u'?'] = 1 ?# crash >> >> So, the proposed patch works for Python 3, but there could be a better >> patch fixing also the corner case on Python 2. > > Thanks. ?How about something like the check further up the file: > > ? ? ? if (PyUnicode_Check(op)) { > ? ? ? ? ? ?temp = PyUnicode_AsUnicodeEscapeString(op); > ? ? ? ?} > ? ? ? ?PyErr_Format(PyExc_ValueError, > ? ? ? ? ? ? ? ? ? ? "field named %s not found.", > ? ? ? ? ? ? ? ? ? ? PyBytes_AsString(temp)); > > ? I'm happy to submit a pull request with tests if that kind of thing > looks sensible to y'all, That fix works for 2.6 but crashes 3.2 when it is rejecting (by design I think) field indexing with byte strings. I've put a test function in this branch which exercises the routes I could think of. It will crash current 2.x because of the unicode error that Pauli pointed out, but I think it corresponds to the expected behavior: https://github.com/matthew-brett/numpy/compare/numpy:master...matthew-brett:struct-arr-fields Do these tests look correct? I'd be happy to hear why the code above does not work, it wasn't obvious to me. See you, Matthew From kk1674 at nyu.edu Wed Mar 16 01:05:48 2011 From: kk1674 at nyu.edu (Kibeom Kim) Date: Wed, 16 Mar 2011 01:05:48 -0400 Subject: [Numpy-discussion] a newbie parallelization question Message-ID: Example code: from pylab import * cos(random([1000,1000])) Q1. Is there any reason that numpy doesn't automatically support parallelization of element-wise operations like the example, even with Atlas or MKL? Q2. What is the most advisable way to parallelize(multi-thread) the example? numexpr? Thanks. -Kibeom Kim -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 16 01:42:58 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 16 Mar 2011 13:42:58 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 11:29 AM, Ralf Gommers wrote: > > > On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >> >> I pushed one more small API change to PyArray_NewLikeArray, adding a >>> 'subok' parameter which lets you disable sub-types. The things missing still >>> are documentation (maybe others can help?). The Python nditer exposure is >>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>> 'out', 'order', 'subok', 'dtype'). >>> >>> > Okay, I'll write some docs for the nditer object. > Hi Mark, could you review and fill in a few blanks: https://github.com/rgommers/numpy/tree/nditer-docs Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 16 01:45:09 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 16 Mar 2011 13:45:09 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 12:02 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Tue, Mar 15, 2011 at 9:29 PM, Ralf Gommers > wrote: > >> >> >> On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >>> >>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>>> 'subok' parameter which lets you disable sub-types. The things missing still >>>> are documentation (maybe others can help?). The Python nditer exposure is >>>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>>> 'out', 'order', 'subok', 'dtype'). >>>> >>> >> >>> We should probably postpone the beta by a few days, there are some other >>> loose ends floating about. >>> >>> Besides documentation the only thing I can think of is the structured >> array non-existing filed segfault thing. And perhaps #1619 would be good to >> have fixed, but not essential for a beta IMHO. Anything else? >> >> > Just that things that go in at the last moment should be let sit a bit to > settle. > > True. I will create the branch already though, otherwise there will continue to be things that just went in. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Mar 16 02:18:59 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2011 23:18:59 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme Message-ID: Hi, Running the test suite for one of our libraries, there seems to have been a recent breakage of the behavior of dtype hashing. This script: import numpy as np data0 = np.arange(10) data1 = data0 - 10 dt0 = data0.dtype dt1 = data1.dtype assert dt0 == dt1 # always passes assert hash(dt0) == hash(dt1) # fails on latest fails on the current latest-ish - aada93306 and passes on a stock 1.5.0. Is this expected? See you, Matthew From dileepkunjaai at gmail.com Wed Mar 16 02:53:20 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Wed, 16 Mar 2011 12:23:20 +0530 Subject: [Numpy-discussion] Multiple Linear Regression Message-ID: Dear sir, Can we do multiple linear regression(MLR) in python.... is there any inbuilt function for MLR -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Wed Mar 16 03:03:03 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 16 Mar 2011 00:03:03 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 8:29 PM, Ralf Gommers wrote: > > > On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >> >>> On Sun, Mar 13, 2011 at 11:59 PM, Ralf Gommers < >>> ralf.gommers at googlemail.com> wrote: >>> >>>> >>>> Hi Mark, I see you just did this, but is there anything else you >>>> want/need to do? If it's necessary I can postpone the first beta by a >>>> couple of days. Better that than rush things too much and end up with >>>> an API you have reservations about. >>>> >>> >>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>> 'subok' parameter which lets you disable sub-types. The things missing still >>> are documentation (maybe others can help?). The Python nditer exposure is >>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>> 'out', 'order', 'subok', 'dtype'). >>> >>> > Okay, I'll write some docs for the nditer object. > > >> We should probably postpone the beta by a few days, there are some other >> loose ends floating about. >> >> Besides documentation the only thing I can think of is the structured > array non-existing filed segfault thing. And perhaps #1619 would be good to > have fixed, but not essential for a beta IMHO. Anything else? > I took a look at #1619, and added a comment suggesting an approach to fix it. For the goal of having structured types work reasonably, this one definitely needs a fix. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Wed Mar 16 03:29:14 2011 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 16 Mar 2011 09:29:14 +0200 Subject: [Numpy-discussion] OpenOpt Suite release 0.33 Message-ID: Hi all, I'm glad to inform you about new release 0.33 of our completely free (license: BSD) cross-platform software: OpenOpt: > * cplex has been connected > * New global solver interalg with guarantied precision, competitor to LGO, BARON, MATLAB's intsolver and Direct (also can work in inexact mode) > * New solver amsg2p for unconstrained medium-scaled NLP and NSP > FuncDesigner: > * Essential speedup for automatic differentiation when vector-variables are involved, for both dense and sparse cases > * Solving MINLP became available > * Add uncertainty analysis > * Add interval analysis > * Now you can solve systems of equations with automatic determination is the system linear or nonlinear (subjected to given set of free or fixed variables) > * FD Funcs min and max can work on lists of oofuns > * Bugfix for sparse SLE (system of linear equations), that slowed down computation time and demanded more memory > * New oofuns angle, cross > * Using OpenOpt result(oovars) is available, also, start points with oovars() now can be assigned easier > SpaceFuncs (2D, 3D, N-dimensional geometric package with abilities for parametrized calculations, solving systems of geometric equations and numerical optimization with automatic differentiation): > * Some bugfixes > DerApproximator: > * Adjusted with some changes in FuncDesigner > For more details visit our site http://openopt.org. > Regards, Dmitrey. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Wed Mar 16 04:16:57 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 16 Mar 2011 01:16:57 -0700 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 10:42 PM, Ralf Gommers wrote: > > > On Wed, Mar 16, 2011 at 11:29 AM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> >> >> On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >>> >>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>>> 'subok' parameter which lets you disable sub-types. The things missing still >>>> are documentation (maybe others can help?). The Python nditer exposure is >>>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>>> 'out', 'order', 'subok', 'dtype'). >>>> >>>> >> Okay, I'll write some docs for the nditer object. >> > > Hi Mark, could you review and fill in a few blanks: > https://github.com/rgommers/numpy/tree/nditer-docs > I've changed some and filled in more details. Sphinx appears to get this totally wrong though, it completely ignores the Attributes section (maybe those need to be separated out?), and links to numpy.copy instead of nditer.copy in the Methods section. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 16 04:33:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 16 Mar 2011 16:33:39 +0800 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 4:16 PM, Mark Wiebe wrote: > > > On Tue, Mar 15, 2011 at 10:42 PM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> >> >> On Wed, Mar 16, 2011 at 11:29 AM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> >>> >>> On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> >>>> >>>> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >>>> >>>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>>>> 'subok' parameter which lets you disable sub-types. The things missing still >>>>> are documentation (maybe others can help?). The Python nditer exposure is >>>>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>>>> 'out', 'order', 'subok', 'dtype'). >>>>> >>>>> >>> Okay, I'll write some docs for the nditer object. >>> >> >> Hi Mark, could you review and fill in a few blanks: >> https://github.com/rgommers/numpy/tree/nditer-docs >> > > I've changed some and filled in more details. Sphinx appears to get this > totally wrong though, it completely ignores the Attributes section (maybe > those need to be separated out?), and links to numpy.copy instead of > nditer.copy in the Methods section. > > Yes, that's a bug in the Sphinx autosummary extension. And also in the class template in the numpy reference guide source. I already filed #1772 for that. Attributes section should be fine as it is. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From washakie at gmail.com Wed Mar 16 04:36:42 2011 From: washakie at gmail.com (John) Date: Wed, 16 Mar 2011 09:36:42 +0100 Subject: [Numpy-discussion] convert dictionary of arrays into single array Message-ID: Hello, I have a dictionary with structured arrays, keyed by integers 0...n. There are no other keys in the dictionary. What is the most efficient way to convert the dictionary of arrays to a single array? All the arrays have the same 'headings' and width, but different lengths. Is there something I can do that would be more efficient than looping through and using np.concatenate (vstack)? --john From amcmorl at gmail.com Wed Mar 16 07:47:44 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 16 Mar 2011 07:47:44 -0400 Subject: [Numpy-discussion] Multiple Linear Regression In-Reply-To: References: Message-ID: On 16 March 2011 02:53, dileep kunjaai wrote: > Dear sir, > ?Can we do multiple linear regression(MLR)? in python.... is there any > inbuilt function for MLR Yes, you can use np.linalg.lstsq [1] for this. Here's a quick example: import numpy as np # model is y = b0.x0 + b1.x1 + b2.x2 b = np.array([3.,2.,1.]) noise = np.random.standard_normal(size=(10,3)) * 0.1 bn = b[None] + noise x = np.random.standard_normal(size=(10,3)) y = np.sum(bn * x, axis=1) be = np.linalg.lstsq(x,y) and be[0] should be close to the original b (3,2,1.). [1] http://docs.scipy.org/doc/numpy-1.4.x/reference/generated/numpy.linalg.lstsq.html > -- > DILEEPKUMAR. R > J R F, IIT DELHI -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From sturla at molden.no Wed Mar 16 08:25:44 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 16 Mar 2011 13:25:44 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> Message-ID: <4D80AC48.3020809@molden.no> Den 16.03.2011 00:01, skrev Neal Becker: Here is how Fortran compares: > * 1-d, 2-d only or N-d?? Any of those. > * support for slice views? What exactly kind of support? Fortran 90 pointers create a view. real*8, target :: array(n,m) real*8, pointer :: view view => array(::2, ::2) Slicing alone creates a view or new array depending on context. > * semantics like numpy, that make many operations avoid copy? Whenever the compiler can infer that it's safe, i.e. if the same data cannot be referenced on lhs and rhs. Otherwise it will make a temporary copy. > * what support for arithmetic operations? Any. > Do views support arithmetic > operations? Yes. > * expression templates? No. > * How's the performance with indexing? Multi-D indexing? How about iteration? Same as C for release builds. Boundschecking is a compiler dependent option e.g. for debugging. > * What are the semantics of iterators? I don't think I've seen 2 libs that > treat multi-d iterators the same way (and I've tried a lot of them). do k = 1,size(x) x(k) = blabblabla......... end do Sturla From sturla at molden.no Wed Mar 16 08:32:06 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 16 Mar 2011 13:32:06 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D80AC48.3020809@molden.no> References: <4D7F81CA.30808@molden.no> <4D80AC48.3020809@molden.no> Message-ID: <4D80ADC6.30507@molden.no> Den 16.03.2011 13:25, skrev Sturla Molden: > > Fortran 90 pointers create a view. > > real*8, target :: array(n,m) > real*8, pointer :: view > > view => array(::2, ::2) Pardon, the second line should be real*8, pointer :: view(:,:) Sturla From dsdale24 at gmail.com Wed Mar 16 08:52:05 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 16 Mar 2011 08:52:05 -0400 Subject: [Numpy-discussion] When was the ddof kwarg added to std()? Message-ID: Does anyone know when the ddof kwarg was added to std()? Has it always been there? Thanks, Darren From dileepkunjaai at gmail.com Wed Mar 16 09:07:46 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Wed, 16 Mar 2011 18:37:46 +0530 Subject: [Numpy-discussion] Multiple Linear Regression In-Reply-To: References: Message-ID: Thank you for your time and consideration................. On Wed, Mar 16, 2011 at 5:17 PM, Angus McMorland wrote: > On 16 March 2011 02:53, dileep kunjaai wrote: > > Dear sir, > > Can we do multiple linear regression(MLR) in python.... is there any > > inbuilt function for MLR > > Yes, you can use np.linalg.lstsq [1] for this. > Here's a quick example: > > import numpy as np > # model is y = b0.x0 + b1.x1 + b2.x2 > b = np.array([3.,2.,1.]) > noise = np.random.standard_normal(size=(10,3)) * 0.1 > bn = b[None] + noise > x = np.random.standard_normal(size=(10,3)) > y = np.sum(bn * x, axis=1) > be = np.linalg.lstsq(x,y) > > and be[0] should be close to the original b (3,2,1.). > > [1] > http://docs.scipy.org/doc/numpy-1.4.x/reference/generated/numpy.linalg.lstsq.html > > > -- > > DILEEPKUMAR. R > > J R F, IIT DELHI > -- > AJC McMorland > Post-doctoral research fellow > Neurobiology, University of Pittsburgh > -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Wed Mar 16 09:10:47 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Wed, 16 Mar 2011 15:10:47 +0200 Subject: [Numpy-discussion] When was the ddof kwarg added to std()? In-Reply-To: References: Message-ID: On 16 March 2011 14:52, Darren Dale wrote: > Does anyone know when the ddof kwarg was added to std()? Has it always > been there? Does 'git log --grep=ddof' help? Cheers, Scott From paul.anton.letnes at gmail.com Wed Mar 16 09:24:49 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 16 Mar 2011 14:24:49 +0100 Subject: [Numpy-discussion] *= operator not intuitive Message-ID: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> Hi! This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! Paul. ++++++++++++++++++ >>> from numpy import * >>> a = arange(10) >>> b = linspace(0,1,10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> b array([ 0. , 0.11111111, 0.22222222, 0.33333333, 0.44444444, 0.55555556, 0.66666667, 0.77777778, 0.88888889, 1. ]) >>> a * b array([ 0. , 0.11111111, 0.44444444, 1. , 1.77777778, 2.77777778, 4. , 5.44444444, 7.11111111, 9. ]) >>> a *= b >>> a array([0, 0, 0, 1, 1, 2, 4, 5, 7, 9]) From amcmorl at gmail.com Wed Mar 16 09:29:05 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 16 Mar 2011 09:29:05 -0400 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> Message-ID: On 16 March 2011 09:24, Paul Anton Letnes wrote: > Hi! > > This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! This is intentional: a *= b works inplace, i.e. it's the equivalent, not of a = a * b, but of a[:] = a * b Angus. > Paul. > > ++++++++++++++++++ >>>> from numpy import * >>>> a = arange(10) >>>> b = linspace(0,1,10) >>>> a > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>> b > array([ 0. ? ? ? ?, ?0.11111111, ?0.22222222, ?0.33333333, ?0.44444444, > ? ? ? 0.55555556, ?0.66666667, ?0.77777778, ?0.88888889, ?1. ? ? ? ?]) >>>> a * b > array([ 0. ? ? ? ?, ?0.11111111, ?0.44444444, ?1. ? ? ? ?, ?1.77777778, > ? ? ? 2.77777778, ?4. ? ? ? ?, ?5.44444444, ?7.11111111, ?9. ? ? ? ?]) >>>> a *= b >>>> a > array([0, 0, 0, 1, 1, 2, 4, 5, 7, 9]) > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From d.s.seljebotn at astro.uio.no Wed Mar 16 09:30:58 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 14:30:58 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> Message-ID: <4D80BB92.5000506@student.matnat.uio.no> On 03/16/2011 02:24 PM, Paul Anton Letnes wrote: > Hi! > > This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! In [1]: a = np.ones(5) In [2]: b = a In [3]: c = a * 2 In [4]: b Out[4]: array([ 1., 1., 1., 1., 1.]) In [5]: a *= 2 In [6]: b Out[6]: array([ 2., 2., 2., 2., 2.]) -- Dag From jsseabold at gmail.com Wed Mar 16 09:30:48 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 16 Mar 2011 09:30:48 -0400 Subject: [Numpy-discussion] Multiple Linear Regression In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 2:53 AM, dileep kunjaai wrote: > Dear sir, > ?Can we do multiple linear regression(MLR)? in python.... is there any > inbuilt function for MLR > You might be interested in statsmodels http://statsmodels.sourceforge.net/ Skipper From paul.anton.letnes at gmail.com Wed Mar 16 09:35:03 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 16 Mar 2011 14:35:03 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <4D80BB92.5000506@student.matnat.uio.no> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80BB92.5000506@student.matnat.uio.no> Message-ID: Heisann! On 16. mars 2011, at 14.30, Dag Sverre Seljebotn wrote: > On 03/16/2011 02:24 PM, Paul Anton Letnes wrote: >> Hi! >> >> This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! > > > In [1]: a = np.ones(5) Here, a is numpy.float64: >>> numpy.ones(5).dtype dtype('float64') > In [2]: b = a > > In [3]: c = a * 2 > > In [4]: b > Out[4]: array([ 1., 1., 1., 1., 1.]) > > In [5]: a *= 2 So since a is already float, and b is the same object as a, the resulting a and b are of course floats. > > In [6]: b > Out[6]: array([ 2., 2., 2., 2., 2.]) > This is not the case I am describing, as in my case, a was of dtype integer. Or did I miss something? Paul. From charlesr.harris at gmail.com Wed Mar 16 09:34:16 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Mar 2011 07:34:16 -0600 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> Message-ID: On Wed, Mar 16, 2011 at 7:24 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > Hi! > > This little snippet of code tricked me (in a more convoluted form). The *= > operator does not change the datatype of the left hand side array. Is this > intentional? It did fool me and throw my results quite a bit off. I always > assumed that 'a *= b' means exactly the same as 'a = a * b' but this is > clearly not the case! > > Yes, it is intentional. Numpy is more C than Python in this case, it actually does the multiplication in-place so that the result must have the same type as the left hand side. In this case Python just creates a new object. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Mar 16 09:39:34 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Mar 2011 07:39:34 -0600 Subject: [Numpy-discussion] When was the ddof kwarg added to std()? In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 6:52 AM, Darren Dale wrote: > Does anyone know when the ddof kwarg was added to std()? Has it always > been there? > > IIRC, a few years back there was a long thread on the list about unbiased vs biased estimates of std and ddof may have been added then... or it may have been there from the beginning. I don't know. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Wed Mar 16 09:41:41 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 16 Mar 2011 09:41:41 -0400 Subject: [Numpy-discussion] When was the ddof kwarg added to std()? In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 9:10 AM, Scott Sinclair wrote: > On 16 March 2011 14:52, Darren Dale wrote: >> Does anyone know when the ddof kwarg was added to std()? Has it always >> been there? > > Does 'git log --grep=ddof' help? Yes: March 7, 2008 Thanks From ndbecker2 at gmail.com Wed Mar 16 09:46:42 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 16 Mar 2011 09:46:42 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] References: <4D7F81CA.30808@molden.no> <4D80AC48.3020809@molden.no> <4D80ADC6.30507@molden.no> Message-ID: Sturla Molden wrote: > Den 16.03.2011 13:25, skrev Sturla Molden: >> >> Fortran 90 pointers create a view. >> >> real*8, target :: array(n,m) >> real*8, pointer :: view >> >> view => array(::2, ::2) > > Pardon, the second line should be > > real*8, pointer :: view(:,:) > > > Sturla Also: * can it adopt external memory? * can it interwork with numpy? (kinda required for this audience) From jsalvati at u.washington.edu Wed Mar 16 09:56:18 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 16 Mar 2011 06:56:18 -0700 Subject: [Numpy-discussion] convert dictionary of arrays into single array In-Reply-To: References: Message-ID: Loop through to build a list of arrays, then use vstack on the list. On Wed, Mar 16, 2011 at 1:36 AM, John wrote: > Hello, > > I have a dictionary with structured arrays, keyed by integers 0...n. > There are no other keys in the dictionary. > > What is the most efficient way to convert the dictionary of arrays to > a single array? > > All the arrays have the same 'headings' and width, but different lengths. > > Is there something I can do that would be more efficient than looping > through and using np.concatenate (vstack)? > --john > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yyc at solvcon.net Wed Mar 16 10:00:09 2011 From: yyc at solvcon.net (Yung-Yu Chen) Date: Wed, 16 Mar 2011 10:00:09 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> <4D80AC48.3020809@molden.no> <4D80ADC6.30507@molden.no> Message-ID: On Wed, Mar 16, 2011 at 09:46, Neal Becker wrote: > Sturla Molden wrote: > > > Den 16.03.2011 13:25, skrev Sturla Molden: > >> > >> Fortran 90 pointers create a view. > >> > >> real*8, target :: array(n,m) > >> real*8, pointer :: view > >> > >> view => array(::2, ::2) > > > > Pardon, the second line should be > > > > real*8, pointer :: view(:,:) > > > > > > Sturla > > Also: > * can it adopt external memory? > What is external memory? > * can it interwork with numpy? (kinda required for this audience) > > You can use f2py, swig, ctypes, or just Python extension to invoke Fortran code and pass numpy arrays to it and get them back. Not sure whether or not Cython has Fortran support, but Cython can be used as a simplified approach of Python extension. Usually numpy arrays are allocated on heap. In this way we can minimize allocate()/deallocate() inside Fortran subroutines and let Python to do the memory management. Fortran pointers are more like a view rather than C pointers which point to the beginning of a memory block. yyc _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Yung-Yu Chen PhD candidate of Mechanical Engineering The Ohio State University, Columbus, Ohio +1 (614) 859 2436 http://solvcon.net/yyc/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed Mar 16 10:30:29 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 16 Mar 2011 09:30:29 -0500 Subject: [Numpy-discussion] convert dictionary of arrays into single array In-Reply-To: References: Message-ID: <4D80C985.6030205@gmail.com> On 03/16/2011 08:56 AM, John Salvatier wrote: > Loop through to build a list of arrays, then use vstack on the list. > > On Wed, Mar 16, 2011 at 1:36 AM, John > wrote: > > Hello, > > I have a dictionary with structured arrays, keyed by integers 0...n. > There are no other keys in the dictionary. > > What is the most efficient way to convert the dictionary of arrays to > a single array? > > All the arrays have the same 'headings' and width, but different > lengths. > > Is there something I can do that would be more efficient than looping > through and using np.concatenate (vstack)? > --john > _______________________________________________ > Numpy does not permit a 'single' array of different shapes - ie a 'ragged array'. Sure you could convert this into a structured array (you know n so you can create an appropriate empty structured array and assign the array by looping across the dict) but that is still not a 'single' array. You can use a masked array where you masked the missing elements across your arrays. Francesc Alted pointed out in the 'ragged array implimentation' thread (http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055219.html) that pytables does support this. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Wed Mar 16 10:38:27 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 16 Mar 2011 07:38:27 -0700 Subject: [Numpy-discussion] convert dictionary of arrays into single array In-Reply-To: <4D80C985.6030205@gmail.com> References: <4D80C985.6030205@gmail.com> Message-ID: I think he wants to stack them (same widths) so stacking them should be fine. On Wed, Mar 16, 2011 at 7:30 AM, Bruce Southey wrote: > On 03/16/2011 08:56 AM, John Salvatier wrote: > > Loop through to build a list of arrays, then use vstack on the list. > > On Wed, Mar 16, 2011 at 1:36 AM, John wrote: > >> Hello, >> >> I have a dictionary with structured arrays, keyed by integers 0...n. >> There are no other keys in the dictionary. >> >> What is the most efficient way to convert the dictionary of arrays to >> a single array? >> >> All the arrays have the same 'headings' and width, but different lengths. >> >> Is there something I can do that would be more efficient than looping >> through and using np.concatenate (vstack)? >> --john >> _______________________________________________ >> > > Numpy does not permit a 'single' array of different shapes - ie a 'ragged > array'. > Sure you could convert this into a structured array (you know n so you can > create an appropriate empty structured array and assign the array by looping > across the dict) but that is still not a 'single' array. You can use a > masked array where you masked the missing elements across your arrays. > > Francesc Alted pointed out in the 'ragged array implimentation' thread ( > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055219.html) > that pytables does support this. > > Bruce > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 16 10:46:13 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 09:46:13 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 01:18, Matthew Brett wrote: > Hi, > > Running the test suite for one of our libraries, there seems to have > been a recent breakage of the behavior of dtype hashing. > > This script: > > import numpy as np > > data0 = np.arange(10) > data1 = data0 - 10 > > dt0 = data0.dtype > dt1 = data1.dtype > > assert dt0 == dt1 # always passes > assert hash(dt0) == hash(dt1) # fails on latest > > fails on the current latest-ish - aada93306 ?and passes on a stock 1.5.0. > > Is this expected? According to "git log hashdescr.c", nothing has changed in the implementation of the hash function since Oct 31, before numpy 1.5.1 which also passes the second test. I'm not sure what would be causing the difference in HEAD. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From Chris.Barker at noaa.gov Wed Mar 16 10:49:52 2011 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed, 16 Mar 2011 07:49:52 -0700 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> Message-ID: <4D80CE10.6000000@noaa.gov> On 3/16/11 6:34 AM, Charles R Harris wrote: > On Wed, Mar 16, 2011 at 7:24 AM, Paul Anton Letnes > Yes, it is intentional. Numpy is more C than Python in this case, I don't know that C has anything to do with it -- the *= operators were added specifically to be "in-place" operators -- otherwise they would be nothing but syntactic sugar. And IIRC, numpy was one of the motivators. IMHO, the mistake was even allowing += and friends for immutables, as that inherently means something different. Of course, using += with integers is probably the most common case. -Chris From charlesr.harris at gmail.com Wed Mar 16 10:56:29 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Mar 2011 08:56:29 -0600 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 8:46 AM, Robert Kern wrote: > On Wed, Mar 16, 2011 at 01:18, Matthew Brett > wrote: > > Hi, > > > > Running the test suite for one of our libraries, there seems to have > > been a recent breakage of the behavior of dtype hashing. > > > > This script: > > > > import numpy as np > > > > data0 = np.arange(10) > > data1 = data0 - 10 > > > > dt0 = data0.dtype > > dt1 = data1.dtype > > > > assert dt0 == dt1 # always passes > > assert hash(dt0) == hash(dt1) # fails on latest > > > > fails on the current latest-ish - aada93306 and passes on a stock 1.5.0. > > > > Is this expected? > > According to "git log hashdescr.c", nothing has changed in the > implementation of the hash function since Oct 31, before numpy 1.5.1 > which also passes the second test. I'm not sure what would be causing > the difference in HEAD. > > The 1.5.1 branch was based on 1.5.x, not master. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Wed Mar 16 10:57:45 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 15:57:45 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80BB92.5000506@student.matnat.uio.no> Message-ID: <4D80CFE9.9000406@student.matnat.uio.no> On 03/16/2011 02:35 PM, Paul Anton Letnes wrote: > Heisann! Hei der, > On 16. mars 2011, at 14.30, Dag Sverre Seljebotn wrote: > >> On 03/16/2011 02:24 PM, Paul Anton Letnes wrote: >>> Hi! >>> >>> This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! >> >> In [1]: a = np.ones(5) > Here, a is numpy.float64: >>>> numpy.ones(5).dtype > dtype('float64') > >> In [2]: b = a >> >> In [3]: c = a * 2 >> >> In [4]: b >> Out[4]: array([ 1., 1., 1., 1., 1.]) >> >> In [5]: a *= 2 > So since a is already float, and b is the same object as a, the resulting a and b are of course floats. >> In [6]: b >> Out[6]: array([ 2., 2., 2., 2., 2.]) >> > This is not the case I am describing, as in my case, a was of dtype integer. > Or did I miss something? I was just trying to demonstrate that it is NOT the case that "a = a * 2 is exactly the same as a *= 2". If you assume that the two statements are the same, then it does not make sense that b = [1, 1, ...] the first time around, but b = [2, 2, 2...] the second time around. And in trying to figure out why that happened, perhaps you'd see how it all fits together... OK, it perhaps wasn't a very good explanation... Dag Sverre From paul.anton.letnes at gmail.com Wed Mar 16 10:58:48 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 16 Mar 2011 15:58:48 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <4D80CE10.6000000@noaa.gov> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80CE10.6000000@noaa.gov> Message-ID: On 16. mars 2011, at 15.49, Chris Barker wrote: > On 3/16/11 6:34 AM, Charles R Harris wrote: >> On Wed, Mar 16, 2011 at 7:24 AM, Paul Anton Letnes > >> Yes, it is intentional. Numpy is more C than Python in this case, > > I don't know that C has anything to do with it -- the *= operators were > added specifically to be "in-place" operators -- otherwise they would be > nothing but syntactic sugar. And IIRC, numpy was one of the motivators. > > IMHO, the mistake was even allowing += and friends for immutables, as > that inherently means something different. > > Of course, using += with integers is probably the most common case. > > -Chris I see. In that case, I have the following either/or christmas wish: Either: implement a warning along the following lines: >>> from numpy import * >>> a = zeros(10, dtype=complex) >>> a.astype(float) /Users/paulanto/Library/Python/2.7/bin/bpython:2: ComplexWarning: Casting complex values to real discards the imaginary part # EASY-INSTALL-ENTRY-SCRIPT: 'bpython==0.9.7.1','console_scripts','bpython' array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) Or: give me a hint how and where to change the numpy code, and I could try to write a patch. Paul. From paul.anton.letnes at gmail.com Wed Mar 16 10:59:45 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 16 Mar 2011 15:59:45 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <4D80CFE9.9000406@student.matnat.uio.no> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80BB92.5000506@student.matnat.uio.no> <4D80CFE9.9000406@student.matnat.uio.no> Message-ID: <9151BB14-6038-4CE7-ACF5-3C287F79F4D8@gmail.com> On 16. mars 2011, at 15.57, Dag Sverre Seljebotn wrote: > On 03/16/2011 02:35 PM, Paul Anton Letnes wrote: >> Heisann! > > Hei der, > >> On 16. mars 2011, at 14.30, Dag Sverre Seljebotn wrote: >> >>> On 03/16/2011 02:24 PM, Paul Anton Letnes wrote: >>>> Hi! >>>> >>>> This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! >>> >>> In [1]: a = np.ones(5) >> Here, a is numpy.float64: >>>>> numpy.ones(5).dtype >> dtype('float64') >> >>> In [2]: b = a >>> >>> In [3]: c = a * 2 >>> >>> In [4]: b >>> Out[4]: array([ 1., 1., 1., 1., 1.]) >>> >>> In [5]: a *= 2 >> So since a is already float, and b is the same object as a, the resulting a and b are of course floats. >>> In [6]: b >>> Out[6]: array([ 2., 2., 2., 2., 2.]) >>> >> This is not the case I am describing, as in my case, a was of dtype integer. >> Or did I miss something? > > I was just trying to demonstrate that it is NOT the case that "a = a * 2 > is exactly the same as a *= 2". If you assume that the two statements > are the same, then it does not make sense that b = [1, 1, ...] the first > time around, but b = [2, 2, 2...] the second time around. And in trying > to figure out why that happened, perhaps you'd see how it all fits > together... > > OK, it perhaps wasn't a very good explanation... > > Dag Sverre I see, I misunderstood your point. That's another interesting aspect of this, though. Paul. From renesd at gmail.com Wed Mar 16 11:10:54 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 16 Mar 2011 15:10:54 +0000 Subject: [Numpy-discussion] [OT] any image io module that works with python3? In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D522C@VA3DIAXVS361.RED001.local> Message-ID: pygame On Sat, Mar 12, 2011 at 9:08 AM, Nadav Horesh wrote: > Having numpy, scipy, and matplotlib working reasonably with python3, a > major piece of code I miss for a major python3 migration is an image IO. I > found that pylab's imread works fine for png image, but I need to read all > the other image format as well as png and jpeg output. > > Any hints (including advices how easyly construct my own module) are > appreciated. > > Nadav. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Mar 16 11:27:49 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Mar 2011 09:27:49 -0600 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 8:56 AM, Charles R Harris wrote: > > > On Wed, Mar 16, 2011 at 8:46 AM, Robert Kern wrote: > >> On Wed, Mar 16, 2011 at 01:18, Matthew Brett >> wrote: >> > Hi, >> > >> > Running the test suite for one of our libraries, there seems to have >> > been a recent breakage of the behavior of dtype hashing. >> > >> > This script: >> > >> > import numpy as np >> > >> > data0 = np.arange(10) >> > data1 = data0 - 10 >> > >> > dt0 = data0.dtype >> > dt1 = data1.dtype >> > >> > assert dt0 == dt1 # always passes >> > assert hash(dt0) == hash(dt1) # fails on latest >> > >> > fails on the current latest-ish - aada93306 and passes on a stock >> 1.5.0. >> > >> > Is this expected? >> >> According to "git log hashdescr.c", nothing has changed in the >> implementation of the hash function since Oct 31, before numpy 1.5.1 >> which also passes the second test. I'm not sure what would be causing >> the difference in HEAD. >> >> > The 1.5.1 branch was based on 1.5.x, not master. > > David's change isn't in 1.5.x, so apparently it wasn't backported. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Mar 16 11:34:26 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Mar 2011 09:34:26 -0600 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80CE10.6000000@noaa.gov> Message-ID: On Wed, Mar 16, 2011 at 8:58 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 16. mars 2011, at 15.49, Chris Barker wrote: > > > On 3/16/11 6:34 AM, Charles R Harris wrote: > >> On Wed, Mar 16, 2011 at 7:24 AM, Paul Anton Letnes > > > >> Yes, it is intentional. Numpy is more C than Python in this case, > > > > I don't know that C has anything to do with it -- the *= operators were > > added specifically to be "in-place" operators -- otherwise they would be > > nothing but syntactic sugar. And IIRC, numpy was one of the motivators. > > > > IMHO, the mistake was even allowing += and friends for immutables, as > > that inherently means something different. > > > > Of course, using += with integers is probably the most common case. > > > > -Chris > > I see. In that case, I have the following either/or christmas wish: > > Either: implement a warning along the following lines: > >>> from numpy import * > >>> a = zeros(10, dtype=complex) > >>> a.astype(float) > /Users/paulanto/Library/Python/2.7/bin/bpython:2: ComplexWarning: Casting > complex values to real discards the imaginary part > # EASY-INSTALL-ENTRY-SCRIPT: > 'bpython==0.9.7.1','console_scripts','bpython' > array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) > > This comes up for discussion on a fairly regular basis. I tend towards the more warnings side myself, but you aren't going to get the current behavior changed unless you can convince a large bunch of people that it is the right thing to do, which won't be easy. For one thing, a lot of current code in the wild would begin to raise warnings that weren't there before. Or: give me a hint how and where to change the numpy code, and I could try > to write a patch. > > You have to go down to the C level to deal with this. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sienkiew at stsci.edu Wed Mar 16 11:55:46 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Wed, 16 Mar 2011 11:55:46 -0400 Subject: [Numpy-discussion] Is this a bug in repr ? In-Reply-To: References: <4D7F91E4.1040105@stsci.edu> <4D7FABA1.7040804@stsci.edu> Message-ID: <4D80DD82.6010401@stsci.edu> >> In that case, would you agree that it is a bug for >> assert_array_almost_equal to use repr() to display the arrays, since it >> is printing identical values and saying they are different? Or is there >> also a reason to do that? >> > > It should probably use np.array_repr(x, precision=16) > Ok, thanks - I see the issue. I'll enter a ticket for an enhancement request for assert_array_almost_equal From robert.kern at gmail.com Wed Mar 16 12:21:53 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 11:21:53 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 10:27, Charles R Harris wrote: > > On Wed, Mar 16, 2011 at 8:56 AM, Charles R Harris > wrote: >> >> >> On Wed, Mar 16, 2011 at 8:46 AM, Robert Kern >> wrote: >>> >>> On Wed, Mar 16, 2011 at 01:18, Matthew Brett >>> wrote: >>> > Hi, >>> > >>> > Running the test suite for one of our libraries, there seems to have >>> > been a recent breakage of the behavior of dtype hashing. >>> > >>> > This script: >>> > >>> > import numpy as np >>> > >>> > data0 = np.arange(10) >>> > data1 = data0 - 10 >>> > >>> > dt0 = data0.dtype >>> > dt1 = data1.dtype >>> > >>> > assert dt0 == dt1 # always passes >>> > assert hash(dt0) == hash(dt1) # fails on latest >>> > >>> > fails on the current latest-ish - aada93306 ?and passes on a stock >>> > 1.5.0. >>> > >>> > Is this expected? >>> >>> According to "git log hashdescr.c", nothing has changed in the >>> implementation of the hash function since Oct 31, before numpy 1.5.1 >>> which also passes the second test. I'm not sure what would be causing >>> the difference in HEAD. >>> >> >> The 1.5.1 branch was based on 1.5.x, not master. >> > > David's change isn't in 1.5.x, so apparently it wasn't backported. Hmm. It works just before and just after that change, so the problem is somewhere else. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From paul.anton.letnes at gmail.com Wed Mar 16 12:22:10 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 16 Mar 2011 17:22:10 +0100 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80CE10.6000000@noaa.gov> Message-ID: <691DAE0D-5C1D-4EE4-8DA6-9FD39272E85F@gmail.com> > > This comes up for discussion on a fairly regular basis. I tend towards the more warnings side myself, but you aren't going to get the current behavior changed unless you can convince a large bunch of people that it is the right thing to do, which won't be easy. For one thing, a lot of current code in the wild would begin to raise warnings that weren't there before. That could also be a good thing for locating bugs, right? > Or: give me a hint how and where to change the numpy code, and I could try to write a patch. > > > You have to go down to the C level to deal with this. I guess code containing such warnings must exist in other parts of the numpy library? Paul. From m.c.dixon at leeds.ac.uk Wed Mar 16 12:26:58 2011 From: m.c.dixon at leeds.ac.uk (Mark Dixon) Date: Wed, 16 Mar 2011 16:26:58 +0000 (GMT) Subject: [Numpy-discussion] Numpy 1.5.1 test failures Message-ID: Hi, Sorry if this is a noob question, but I've been trying to install Numpy for a while now and I keep having problems getting it to pass the test suite. I'm on a RHEL5 system, building against Python 2.6.5 (self-built with GCC 4.1.2), gfortran 4.1.2 and MKL 10.3 Update 2 (shipped with Intel compiler 2011.2.137). I'm failing the following tests (see below for full output): FAIL: test_complex (test_numeric.TestCorrelate) FAIL: test_complex (test_numeric.TestCorrelateNew) The imaginary components for the results of test_numeric.TestCorrelateNew have the correct magnitude but the wrong sign, and test_numeric.TestCorrelate is just wrong wrong wrong. Is this a known issue? searching for test_complex in the mail archive didn't find anything. Thanks, Mark -- ----------------------------------------------------------------- Mark Dixon Email : m.c.dixon at leeds.ac.uk HPC/Grid Systems Support Tel (int): 35429 Information Systems Services Tel (ext): +44(0)113 343 5429 University of Leeds, LS2 9JT, UK ----------------------------------------------------------------- ====================================================================== FAIL: test_complex (test_numeric.TestCorrelate) ---------------------------------------------------------------------- Traceback (most recent call last): File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/testing/decorators.py", lin e 257, in _deprecated_imp f(*args, **kwargs) File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/core/tests/test_numeric.py" , line 942, in test_complex assert_array_almost_equal(z, r_z) File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/testing/utils.py", line 774 , in assert_array_almost_equal header='Arrays are not almost equal') File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/testing/utils.py", line 618 , in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 1.14014444e-312 +2.83449967e-316j, 2.80975212e-316 +4.94065646e-324j, 2.81612043e-316 +2.81612082e-316j,... y: array([ 3.+1.j, 6.+0.j, 8.-1.j, 9.+1.j, -1.-8.j, -4.-1.j]) ====================================================================== FAIL: test_complex (test_numeric.TestCorrelateNew) ---------------------------------------------------------------------- Traceback (most recent call last): File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/core/tests/test_numeric.py" , line 963, in test_complex assert_array_almost_equal(z, r_z) File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/testing/utils.py", line 774 , in assert_array_almost_equal header='Arrays are not almost equal') File "/nobackup/issmcd/pybuild2.6.5/lib/python2.6/site-packages/numpy/testing/utils.py", line 618 , in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 83.3333333333%) x: array([ -4.00000000e+000 -1.00000000e+000j, -5.00000000e+000 +8.00000000e+000j, 1.10000000e+001 +5.00000000e+000j,... y: array([ -4.+1.j, -5.-8.j, 11.-5.j, 8.-1.j, 6.-0.j, 3.+1.j]) ---------------------------------------------------------------------- Ran 3006 tests in 63.290s FAILED (KNOWNFAIL=4, failures=2) From matthew.brett at gmail.com Wed Mar 16 12:43:09 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 16 Mar 2011 09:43:09 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: Hi, On Wed, Mar 16, 2011 at 9:21 AM, Robert Kern wrote: > On Wed, Mar 16, 2011 at 10:27, Charles R Harris > wrote: >> >> On Wed, Mar 16, 2011 at 8:56 AM, Charles R Harris >> wrote: >>> >>> >>> On Wed, Mar 16, 2011 at 8:46 AM, Robert Kern >>> wrote: >>>> >>>> On Wed, Mar 16, 2011 at 01:18, Matthew Brett >>>> wrote: >>>> > Hi, >>>> > >>>> > Running the test suite for one of our libraries, there seems to have >>>> > been a recent breakage of the behavior of dtype hashing. >>>> > >>>> > This script: >>>> > >>>> > import numpy as np >>>> > >>>> > data0 = np.arange(10) >>>> > data1 = data0 - 10 >>>> > >>>> > dt0 = data0.dtype >>>> > dt1 = data1.dtype >>>> > >>>> > assert dt0 == dt1 # always passes >>>> > assert hash(dt0) == hash(dt1) # fails on latest >>>> > >>>> > fails on the current latest-ish - aada93306 ?and passes on a stock >>>> > 1.5.0. >>>> > >>>> > Is this expected? >>>> >>>> According to "git log hashdescr.c", nothing has changed in the >>>> implementation of the hash function since Oct 31, before numpy 1.5.1 >>>> which also passes the second test. I'm not sure what would be causing >>>> the difference in HEAD. >>>> >>> >>> The 1.5.1 branch was based on 1.5.x, not master. >>> >> >> David's change isn't in 1.5.x, so apparently it wasn't backported. > > Hmm. It works just before and just after that change, so the problem > is somewhere else. I can git-bisect it later in the day, will do so unless it's become clear in the meantime. Thanks, Matthew From robert.kern at gmail.com Wed Mar 16 12:55:13 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 11:55:13 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 11:43, Matthew Brett wrote: > I can git-bisect it later in the day, will do so unless it's become > clear in the meantime. I'm almost done bisecting. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Wed Mar 16 13:00:09 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 12:00:09 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 11:55, Robert Kern wrote: > On Wed, Mar 16, 2011 at 11:43, Matthew Brett wrote: > >> I can git-bisect it later in the day, will do so unless it's become >> clear in the meantime. > > I'm almost done bisecting. 6c6dc487ca15818d1f4cc764debb15d73a61c03b is the first bad commit commit 6c6dc487ca15818d1f4cc764debb15d73a61c03b Author: Mark Wiebe Date: Thu Jan 20 20:41:03 2011 -0800 ENH: ufunc: Made the iterator ufunc default :040000 040000 15033eb0c0e295161cd29a31677e7b88ac431143 ae077a44ccce0014e017537b31f53261495f870e M numpy -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mwwiebe at gmail.com Wed Mar 16 13:15:36 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 16 Mar 2011 10:15:36 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 10:00 AM, Robert Kern wrote: > On Wed, Mar 16, 2011 at 11:55, Robert Kern wrote: > > On Wed, Mar 16, 2011 at 11:43, Matthew Brett > wrote: > > > >> I can git-bisect it later in the day, will do so unless it's become > >> clear in the meantime. > > > > I'm almost done bisecting. > > 6c6dc487ca15818d1f4cc764debb15d73a61c03b is the first bad commit > commit 6c6dc487ca15818d1f4cc764debb15d73a61c03b > Author: Mark Wiebe > Date: Thu Jan 20 20:41:03 2011 -0800 > > ENH: ufunc: Made the iterator ufunc default > > :040000 040000 15033eb0c0e295161cd29a31677e7b88ac431143 > ae077a44ccce0014e017537b31f53261495f870e M numpy I'm guessing this is another case where the type numbers being ambiguous is the problem. On my 64-bit system: >>> np.dtype(np.int) == np.dtype(np.long) True >>> hash(np.dtype(np.int)) == hash(np.dtype(np.long)) False >>> np.dtype(np.int).num 7 >>> np.dtype(np.long).num 9 On a 32-bit system, types 5 and 7 are similarly aliased. By modifying the example slightly, possibly just switching the "data0 - 10" to "10 + data0", 1.5 probably will fail this test as well. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 16 13:53:21 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 12:53:21 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 12:15, Mark Wiebe wrote: > On Wed, Mar 16, 2011 at 10:00 AM, Robert Kern wrote: >> >> On Wed, Mar 16, 2011 at 11:55, Robert Kern wrote: >> > On Wed, Mar 16, 2011 at 11:43, Matthew Brett >> > wrote: >> > >> >> I can git-bisect it later in the day, will do so unless it's become >> >> clear in the meantime. >> > >> > I'm almost done bisecting. >> >> 6c6dc487ca15818d1f4cc764debb15d73a61c03b is the first bad commit >> commit 6c6dc487ca15818d1f4cc764debb15d73a61c03b >> Author: Mark Wiebe >> Date: ? Thu Jan 20 20:41:03 2011 -0800 >> >> ? ?ENH: ufunc: Made the iterator ufunc default >> >> :040000 040000 15033eb0c0e295161cd29a31677e7b88ac431143 >> ae077a44ccce0014e017537b31f53261495f870e M ? ? ?numpy > > I'm guessing this is another case where the type numbers being ambiguous is > the problem. On my 64-bit system: > ?>>> np.dtype(np.int) == np.dtype(np.long) > True >>>> hash(np.dtype(np.int)) == hash(np.dtype(np.long)) > False >>>> np.dtype(np.int).num > 7 >>>> np.dtype(np.long).num > 9 > On a 32-bit system, types 5 and 7 are similarly aliased. By modifying the > example slightly, possibly just switching the "data0 - 10" to "10 + data0", > 1.5 probably will fail this test as well. Correct. Should we even include the type_num in the key to be hashed for the builtin dtypes? They are not used in the tp_richcompare comparison, only the kind and el_size. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From lists_ravi at lavabit.com Wed Mar 16 15:10:25 2011 From: lists_ravi at lavabit.com (Ravi) Date: Wed, 16 Mar 2011 12:10:25 -0700 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: Message-ID: <201103161210.25058.lists_ravi@lavabit.com> On Monday 14 March 2011 15:02:32 Sebastian Haase wrote: > Sturla has been writing so much about Fortran recently, and Ondrej now > says he has done the move from C/C++ to Fortran -- I thought Fortran > was dead ... !? ;-) > What am I missing here ? Comparing Fortran with C++ is like comparing Matlab with Python. Fortran is very good at what it does but it does not serve the same audience as C++. The typical comparisons work like this: 1. C++ does not have a standard array type but Fortran does. Python does not have a standard numerical array type either; numpy is nice, but, it is an external package. The point is that you pick the external array type that works well for you in Python/C++ but you use the built-in array type in Matlab/Fortran. Whether the flexibility is a problem or a god-send is entirely up to you. For modeling high-performance algorithms in silicon, usually in fixed-point, Fortran/Matlab are worthless, but C++ (and to some extent, python) works very well. 2. C++ (resp. python+numpy) does not perform as well as Fortran (resp. Matlab). The issue with C++ is that aliasing is allowed, but virtually all compilers will allow you to use "restrict" to get almost the same benefits as Fortran. The analogous python+numpy issue is that it is very hard to create a good JIT (unladen-swallow was, at best, a partial success) for python while Matlab has a very nice JIT (and an even better integration with the java virtual machine). 3. "Template metaprogramming makes my head hurt." (Equivalently, "python generators and decorators make my head hurt".) Neither template metaprogramming nor python generators/decorators are *required* for most scientific programming tasks, especially when you use libraries designed to shield you from such details. However, knowledge and use of those techniques makes one much more productive in those corner cases where they are useful. 4. "I do not need all the extra stuff that C++ (resp. python) provides compared to Fortran (resp. Matlab)". C++/python are industrial strength programming languages which serve a large audience, where each interested niche group can create efficient libraries unhindered by the orientation of the language itself. Fortran/Matlab are numerical languages with extra general purpose stuff tacked on. Building large scale Fortran/Matlab programs are possible (see the Joint Strike Fighter models in Matlab or any large-scale Fortran application), but the lack of a real programming language is a pain whose intensity increases exponentially with the size of the codebase. Another way it is usually phrased: "I will use Fortran's (resp. Matlab's) integration with Python (resp. Java) when I need a real programming language." 5. "OOP/generic programming are useless for most scientific programming." This is usually just elitism. (While I personally do not believe that OO is worth one percent of the hype, I do believe that generic programming, as practiced in Python & C++, are pretty much our primary hope for reusable software.) When not elitism, it is a measure of the immaturity of computational science (some would say scientists), where large repositories of reusable code are few and far between. OO, generic programming, and functional programming are the only techniques of which I am aware for building large scale programs with manageable complexity. I would take any Fortran hype with large grains of salt. Regards, Ravi From d.s.seljebotn at astro.uio.no Wed Mar 16 15:12:21 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 20:12:21 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <201103161210.25058.lists_ravi@lavabit.com> References: <201103161210.25058.lists_ravi@lavabit.com> Message-ID: <4D810B95.4000509@student.matnat.uio.no> On 03/16/2011 08:10 PM, Ravi wrote: > On Monday 14 March 2011 15:02:32 Sebastian Haase wrote: >> Sturla has been writing so much about Fortran recently, and Ondrej now >> says he has done the move from C/C++ to Fortran -- I thought Fortran >> was dead ... !? ;-) >> What am I missing here ? > Comparing Fortran with C++ is like comparing Matlab with Python. Fortran is > very good at what it does but it does not serve the same audience as C++. The > typical comparisons work like this: I think the main point being made by most here though is that *in combination with Python*, Fortran can be quite helpful. If one is using Python anyway for the high-level stuff, the relative strengths of C++ w.r.t. Fortran that you list become much less important. Same for code-reuse: When only used from a Python wrapper, the Fortran code can become so simplistic that it also becomes reusable. Dag Sverre From baker.alexander at gmail.com Wed Mar 16 15:33:01 2011 From: baker.alexander at gmail.com (baker.alexander at gmail.com) Date: Wed, 16 Mar 2011 19:33:01 +0000 Subject: [Numpy-discussion] Fortran was dead ... [was Re:rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D810B95.4000509@student.matnat.uio.no> References: <201103161210.25058.lists_ravi@lavabit.com><4D810B95.4000509@student.matnat.uio.no> Message-ID: <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> My two pence worth, my experience is across python, C++ and fortran (and a few other languages) and the posts here are interesting and relevant. I think that the true value of any of these languages is knowing any of them well, if you happen to work with other folks who share the same skills more the better. No more than that. As a user of a very old large very fortran codebase as well as engineer of more structured approaches, I would take the OO toolset everytime, for reasons already covered. The real challenge I see every day in scientific community is the lack of software craftmanship skills, code archiving, unit testing. End of two pence. Alex Sent from my BlackBerry? wireless device -----Original Message----- From: Dag Sverre Seljebotn Sender: numpy-discussion-bounces at scipy.org Date: Wed, 16 Mar 2011 20:12:21 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] On 03/16/2011 08:10 PM, Ravi wrote: > On Monday 14 March 2011 15:02:32 Sebastian Haase wrote: >> Sturla has been writing so much about Fortran recently, and Ondrej now >> says he has done the move from C/C++ to Fortran -- I thought Fortran >> was dead ... !? ;-) >> What am I missing here ? > Comparing Fortran with C++ is like comparing Matlab with Python. Fortran is > very good at what it does but it does not serve the same audience as C++. The > typical comparisons work like this: I think the main point being made by most here though is that *in combination with Python*, Fortran can be quite helpful. If one is using Python anyway for the high-level stuff, the relative strengths of C++ w.r.t. Fortran that you list become much less important. Same for code-reuse: When only used from a Python wrapper, the Fortran code can become so simplistic that it also becomes reusable. Dag Sverre _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From lists_ravi at lavabit.com Wed Mar 16 16:18:03 2011 From: lists_ravi at lavabit.com (lists_ravi at lavabit.com) Date: Wed, 16 Mar 2011 16:18:03 -0400 (EDT) Subject: [Numpy-discussion] Accessing elements of an object array Message-ID: <47107.216.31.211.11.1300306683.squirrel@lavabit.com> Hi, How do I access elements of an object array? The object array was created by scipy.io.loadmat from a MAT file. Here's an example: In [10]: x Out[10]: array(array((7.399500875785845e-10, 7.721153414752673e-10, -0.984375), dtype=[('cl', '|O8'), ('tl', '|O8'), ('dagc', '|O8')]), dtype=object) In [11]: x.shape, x.size Out[11]: ((), 1) In [12]: x.flat[0]['cl'] Out[12]: array(array(7.399500875785845e-10), dtype=object) In [13]: x[0] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /src/ in () IndexError: 0-d arrays can't be indexed I am using numpy 1.4.1 on Linux x86_64, if that matters. Regards, Ravi From robert.kern at gmail.com Wed Mar 16 16:20:28 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Mar 2011 15:20:28 -0500 Subject: [Numpy-discussion] Accessing elements of an object array In-Reply-To: <47107.216.31.211.11.1300306683.squirrel@lavabit.com> References: <47107.216.31.211.11.1300306683.squirrel@lavabit.com> Message-ID: On Wed, Mar 16, 2011 at 15:18, wrote: > Hi, > ?How do I access elements of an object array? The object array was > created by scipy.io.loadmat from a MAT file. Here's an example: > > In [10]: x > Out[10]: > array(array((7.399500875785845e-10, 7.721153414752673e-10, -0.984375), > ? ? ?dtype=[('cl', '|O8'), ('tl', '|O8'), ('dagc', '|O8')]), dtype=object) > > In [11]: x.shape, x.size > Out[11]: ((), 1) > > In [12]: x.flat[0]['cl'] > Out[12]: array(array(7.399500875785845e-10), dtype=object) > > In [13]: x[0] > --------------------------------------------------------------------------- > IndexError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last) > > /src/ in () > > IndexError: 0-d arrays can't be indexed It's not that it's an object array. It's that it is a ()-shape array. You index it with an empty tuple: x[()] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mwwiebe at gmail.com Wed Mar 16 16:21:50 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 16 Mar 2011 13:21:50 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 10:53 AM, Robert Kern wrote: > On Wed, Mar 16, 2011 at 12:15, Mark Wiebe wrote: > > On Wed, Mar 16, 2011 at 10:00 AM, Robert Kern > wrote: > >> > >> On Wed, Mar 16, 2011 at 11:55, Robert Kern > wrote: > >> > On Wed, Mar 16, 2011 at 11:43, Matthew Brett > > >> > wrote: > >> > > >> >> I can git-bisect it later in the day, will do so unless it's become > >> >> clear in the meantime. > >> > > >> > I'm almost done bisecting. > >> > >> 6c6dc487ca15818d1f4cc764debb15d73a61c03b is the first bad commit > >> commit 6c6dc487ca15818d1f4cc764debb15d73a61c03b > >> Author: Mark Wiebe > >> Date: Thu Jan 20 20:41:03 2011 -0800 > >> > >> ENH: ufunc: Made the iterator ufunc default > >> > >> :040000 040000 15033eb0c0e295161cd29a31677e7b88ac431143 > >> ae077a44ccce0014e017537b31f53261495f870e M numpy > > > > I'm guessing this is another case where the type numbers being ambiguous > is > > the problem. On my 64-bit system: > > >>> np.dtype(np.int) == np.dtype(np.long) > > True > >>>> hash(np.dtype(np.int)) == hash(np.dtype(np.long)) > > False > >>>> np.dtype(np.int).num > > 7 > >>>> np.dtype(np.long).num > > 9 > > On a 32-bit system, types 5 and 7 are similarly aliased. By modifying the > > example slightly, possibly just switching the "data0 - 10" to "10 + > data0", > > 1.5 probably will fail this test as well. > > Correct. > > Should we even include the type_num in the key to be hashed for the > builtin dtypes? They are not used in the tp_richcompare comparison, > only the kind and el_size. That sounds like a good fix to me. Whenever objects compare equal, they should hash to the same value. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists_ravi at lavabit.com Wed Mar 16 16:56:16 2011 From: lists_ravi at lavabit.com (lists_ravi at lavabit.com) Date: Wed, 16 Mar 2011 16:56:16 -0400 (EDT) Subject: [Numpy-discussion] loadmat output (was Re: Accessing elements of an object array) In-Reply-To: References: <47107.216.31.211.11.1300306683.squirrel@lavabit.com> Message-ID: <44973.216.31.211.11.1300308976.squirrel@lavabit.com> > On Wed, Mar 16, 2011 at 15:18, wrote: >> In [10]: x >> Out[10]: >> array(array((7.399500875785845e-10, 7.721153414752673e-10, -0.984375), >> ? ? ? dtype=[('cl', '|O8'), ('tl', '|O8'), ('dagc', '|O8')]), >> dtype=object) >> >> In [11]: x.shape, x.size >> Out[11]: ((), 1) > > It's not that it's an object array. It's that it is a ()-shape array. > You index it with an empty tuple: > > x[()] Why does loadmat return such arrays? Is there a way to make it produce arrays that are not object arrays? Regards, Ravi From william.ratcliff at gmail.com Wed Mar 16 17:14:39 2011 From: william.ratcliff at gmail.com (william ratcliff) Date: Wed, 16 Mar 2011 17:14:39 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re:rewriting NumPy code in C or C++ or similar] In-Reply-To: <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> References: <201103161210.25058.lists_ravi@lavabit.com> <4D810B95.4000509@student.matnat.uio.no> <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> Message-ID: Related to this, what is the status of fwrap? Can it be used with fortran 95/2003 language features? There is a rather large code crystallographic codebase (fullprof) that is written in fortran 77 that the author has been porting to fortran 95/2003 and actually using modules for. I'd like to write python bindings for it to make it more scriptable... William On Wed, Mar 16, 2011 at 3:33 PM, wrote: > > My two pence worth, my experience is across python, C++ and fortran (and a > few other languages) and the posts here are interesting and relevant. I > think that the true value of any of these languages is knowing any of them > well, if you happen to work with other folks who share the same skills more > the better. No more than that. > > As a user of a very old large very fortran codebase as well as engineer of > more structured approaches, I would take the OO toolset everytime, for > reasons already covered. > > The real challenge I see every day in scientific community is the lack of > software craftmanship skills, code archiving, unit testing. End of two > pence. > > Alex > Sent from my BlackBerry? wireless device > > -----Original Message----- > From: Dag Sverre Seljebotn > Sender: numpy-discussion-bounces at scipy.org > Date: Wed, 16 Mar 2011 20:12:21 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Fortran was dead ... [was Re: > rewriting NumPy code in C or C++ or similar] > > On 03/16/2011 08:10 PM, Ravi wrote: > > On Monday 14 March 2011 15:02:32 Sebastian Haase wrote: > >> Sturla has been writing so much about Fortran recently, and Ondrej now > >> says he has done the move from C/C++ to Fortran -- I thought Fortran > >> was dead ... !? ;-) > >> What am I missing here ? > > Comparing Fortran with C++ is like comparing Matlab with Python. Fortran > is > > very good at what it does but it does not serve the same audience as C++. > The > > typical comparisons work like this: > > > > I think the main point being made by most here though is that *in > combination with Python*, Fortran can be quite helpful. If one is using > Python anyway for the high-level stuff, the relative strengths of C++ > w.r.t. Fortran that you list become much less important. Same for > code-reuse: When only used from a Python wrapper, the Fortran code can > become so simplistic that it also becomes reusable. > > Dag Sverre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From washakie at gmail.com Wed Mar 16 18:02:46 2011 From: washakie at gmail.com (John) Date: Wed, 16 Mar 2011 23:02:46 +0100 Subject: [Numpy-discussion] convert dictionary of arrays into single array In-Reply-To: References: <4D80C985.6030205@gmail.com> Message-ID: Yes, stacking is fine, and looping per John's suggestion is what I've done, I was just wondering if there was possibly a more 'pythonic' or more importantly efficient way than the loop. Thanks, john On Wed, Mar 16, 2011 at 3:38 PM, John Salvatier wrote: > I think he wants to stack them (same widths) so stacking them should be > fine. > > On Wed, Mar 16, 2011 at 7:30 AM, Bruce Southey wrote: >> >> On 03/16/2011 08:56 AM, John Salvatier wrote: >> >> Loop through to build a list of arrays, then use vstack on the list. >> >> On Wed, Mar 16, 2011 at 1:36 AM, John wrote: >>> >>> Hello, >>> >>> I have a dictionary with structured arrays, keyed by integers 0...n. >>> There are no other keys in the dictionary. >>> >>> What is the most efficient way to convert the dictionary of arrays to >>> a single array? >>> >>> All the arrays have the same 'headings' and width, but different lengths. >>> >>> Is there something I can do that would be more efficient than looping >>> through and using np.concatenate (vstack)? >>> --john >>> _______________________________________________ >> >> Numpy does not permit a 'single' array of different shapes - ie a 'ragged >> array'. >> Sure you could convert this into a structured array (you know n so you can >> create an appropriate empty structured array and assign the array by looping >> across the dict) but that is still not a 'single' array. You can use a >> masked array where you masked the missing elements across your arrays. >> >> Francesc Alted pointed out in the 'ragged array implimentation' thread >> (http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055219.html) >> that pytables does support this. >> >> Bruce >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Configuration `````````````````````````` Plone 2.5.3-final, CMF-1.6.4, Zope (Zope 2.9.7-final, python 2.4.4, linux2), Python 2.6 PIL 1.1.6 Mailman 2.1.9 Postfix 2.4.5 Procmail v3.22 2001/09/10 Basemap: 1.0 Matplotlib: 1.0.0 From matthew.brett at gmail.com Wed Mar 16 18:05:00 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 16 Mar 2011 15:05:00 -0700 Subject: [Numpy-discussion] loadmat output (was Re: Accessing elements of an object array) In-Reply-To: <44973.216.31.211.11.1300308976.squirrel@lavabit.com> References: <47107.216.31.211.11.1300306683.squirrel@lavabit.com> <44973.216.31.211.11.1300308976.squirrel@lavabit.com> Message-ID: Hi, On Wed, Mar 16, 2011 at 1:56 PM, wrote: >> On Wed, Mar 16, 2011 at 15:18, ? wrote: >>> In [10]: x >>> Out[10]: >>> array(array((7.399500875785845e-10, 7.721153414752673e-10, -0.984375), >>> ? ?? ?? dtype=[('cl', '|O8'), ('tl', '|O8'), ('dagc', '|O8')]), >>> dtype=object) >>> >>> In [11]: x.shape, x.size >>> Out[11]: ((), 1) >> >> It's not that it's an object array. It's that it is a ()-shape array. >> You index it with an empty tuple: >> >> ? x[()] > > Why does loadmat return such arrays? Is there a way to make it produce > arrays that are not object arrays? Did you find the struct_as_record option to loadmat? http://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html Best, Matthew From Chris.Barker at noaa.gov Wed Mar 16 18:43:27 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 16 Mar 2011 15:43:27 -0700 Subject: [Numpy-discussion] *= operator not intuitive In-Reply-To: <691DAE0D-5C1D-4EE4-8DA6-9FD39272E85F@gmail.com> References: <0570E46B-30E5-440C-B50B-9AF50CD5331E@gmail.com> <4D80CE10.6000000@noaa.gov> <691DAE0D-5C1D-4EE4-8DA6-9FD39272E85F@gmail.com> Message-ID: <4D813D0F.80901@noaa.gov> On 3/16/11 9:22 AM, Paul Anton Letnes wrote: >> This comes up for discussion on a fairly regular basis. I tend towards the more warnings side myself, but you aren't going to get the current behavior changed unless you can convince a large bunch of people that it is the right thing to do, which won't be easy. Indeed, I don't think I'd want a warning for this -- though honestly, I'm not sure what you think should invoke a warning: a = np.ones((3,), dtype=np.float32) a + = 2.0 Should that be a warning? you are adding a 64 bit float to a 32bit float array. a = np.ones((3,), dtype=np.float32) a + = 2 now you are adding an integer -- should that be a warning? a = np.ones((3,), dtype=np.int32) a + = 2.1 now a float to an integer array -- should that be a warning? As you can see -- there are a lot of options. As there are defined as in-place operators, I don't expect any casting to occur. I think this is the kind of thing that would be great to have a warning the first time you do it, but once you understand, the warnings would be really, really annoying! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From d.s.seljebotn at astro.uio.no Wed Mar 16 17:46:03 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 22:46:03 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re:rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <201103161210.25058.lists_ravi@lavabit.com> <4D810B95.4000509@student.matnat.uio.no> <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> Message-ID: <4D812F9B.1010309@student.matnat.uio.no> On 03/16/2011 10:14 PM, william ratcliff wrote: > Related to this, what is the status of fwrap? Can it be used with > fortran 95/2003 language features? There is a rather large code > crystallographic codebase (fullprof) that is written in fortran 77 > that the author has been porting to fortran 95/2003 and actually using > modules for. I'd like to write python bindings for it to make it more > scriptable... Fwrap 0.1.1 is out; it supports a subset of Fortran 95/2003, biggest limitation being modules not being present. Since then there's been quite a few unreleased improvements (like a much better and more flexible build based on waf instead of distutils). I'm currently working on module support. Or, was ... I've put in a week this month, but then some simulation results grabbed my attention and I got derailed. Finishing up module support and making an Fwrap 0.2 release is #2 on my "stack" of things to work on, and once I get around to it I expect it to take about a week. So I think it will happen :-) If you (or anyone else) want to get involved and put in a day or two to help with polishing (command line interface, writing tests, documentation...) just email me. Dag Sverre From borreguero at gmail.com Thu Mar 17 00:23:13 2011 From: borreguero at gmail.com (Jose Borreguero) Date: Thu, 17 Mar 2011 00:23:13 -0400 Subject: [Numpy-discussion] Build ERROR ConfigParser.MissingSectionHeaderError: File contains no section headers Message-ID: Dear Numpy/SciPy users, I have a build error with Numpy: $ /usr/local/bin/python2.7 setup.py build .... File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 'library_dirs = /usr/local/lib\n' The relevant lines in my site.cfg file: library_dirs = /usr/local/lib include_dirs = /usr/local/include #[blas_opt] libraries = f77blas, cblas, atlas #[lapack_opt] libraries = lapack, f77blas, cblas, atlas I have installed BLAS+LAPACK+ATLAS libraries under /usr/local/lib/atlas I also installed UMFPACK+AMD+UFConfig+CHOLMOD I would appreciate any comments. I'm stuck here :( Best regards, Jose M. Borreguero Below is the full error traceback: $ /usr/local/bin/python2.7 setup.py build Running from numpy source directory.F2PY Version 1 Traceback (most recent call last): File "setup.py", line 211, in setup_package() File "setup.py", line 204, in setup_package configuration=configuration ) File "/projects/tmp/numpy-1.5.1/numpy/distutils/core.py", line 152, in setup config = configuration() File "setup.py", line 151, in configuration config.add_subpackage('numpy') File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/setup.py", line 9, in configuration config.add_subpackage('core') File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/core/setup.py", line 807, in configuration blas_info = get_info('blas_opt',0) File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 310, in get_info return cl().get_info(notfound_action) File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 409, in __init__ self.parse_config_files() File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 416, in parse_config_files self.cp.read(self.files) File "/usr/local/lib/python2.7/ConfigParser.py", line 297, in read self._read(fp, filename) File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 'library_dirs = /usr/local/lib\n' -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Mar 17 00:40:28 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 17 Mar 2011 00:40:28 -0400 Subject: [Numpy-discussion] Build ERROR ConfigParser.MissingSectionHeaderError: File contains no section headers In-Reply-To: References: Message-ID: On Thu, Mar 17, 2011 at 12:23 AM, Jose Borreguero wrote: > Dear Numpy/SciPy users, > > I have a build error with Numpy: > > $? /usr/local/bin/python2.7 setup.py build > .... > ? File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read > ??? raise MissingSectionHeaderError(fpname, lineno, line) > ConfigParser.MissingSectionHeaderError: File contains no section headers. > file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 > 'library_dirs = /usr/local/lib\n' > > The relevant lines in my site.cfg file: > I think you just need to uncomment all the section headers that you use, that`s what the exception says > library_dirs = /usr/local/lib > include_dirs = /usr/local/include [blas_opt] > libraries = f77blas, cblas, atlas [lapack_opt] > libraries = lapack, f77blas, cblas, atlas Josef > > > I have installed BLAS+LAPACK+ATLAS libraries under /usr/local/lib/atlas > > I also installed UMFPACK+AMD+UFConfig+CHOLMOD > > I would appreciate any comments. I'm stuck here :( > > Best regards, > Jose M. Borreguero > > > > Below is the full error traceback: > > $? /usr/local/bin/python2.7 setup.py build > > Running from numpy source directory.F2PY Version 1 > Traceback (most recent call last): > ? File "setup.py", line 211, in > ??? setup_package() > ? File "setup.py", line 204, in setup_package > ??? configuration=configuration ) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/core.py", line 152, in > setup > ??? config = configuration() > ? File "setup.py", line 151, in configuration > ??? config.add_subpackage('numpy') > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 972, > in add_subpackage > ??? caller_level = 2) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 941, > in get_subpackage > ??? caller_level = caller_level + 1) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 878, > in _get_configuration_from_setup_py > ??? config = setup_module.configuration(*args) > ? File "numpy/setup.py", line 9, in configuration > ??? config.add_subpackage('core') > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 972, > in add_subpackage > ??? caller_level = 2) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 941, > in get_subpackage > ??? caller_level = caller_level + 1) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line 878, > in _get_configuration_from_setup_py > ??? config = setup_module.configuration(*args) > ? File "numpy/core/setup.py", line 807, in configuration > ??? blas_info = get_info('blas_opt',0) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 310, > in get_info > ??? return cl().get_info(notfound_action) > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 409, > in __init__ > ??? self.parse_config_files() > ? File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line 416, > in parse_config_files > ??? self.cp.read(self.files) > ? File "/usr/local/lib/python2.7/ConfigParser.py", line 297, in read > ??? self._read(fp, filename) > ? File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read > ??? raise MissingSectionHeaderError(fpname, lineno, line) > ConfigParser.MissingSectionHeaderError: File contains no section headers. > file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 > 'library_dirs = /usr/local/lib\n' > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From dileepkunjaai at gmail.com Thu Mar 17 03:36:10 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Thu, 17 Mar 2011 13:06:10 +0530 Subject: [Numpy-discussion] avoid a line... Message-ID: Dear sir, I am try to read a file of the following format, I want to avoid the first line and read the remaining as 'float' . Please help me... Rain Wind Temp Pr Sal 0.1 1.1 0.02 0.2 0.2 0.5 0. 0. 0.4 0.8 0.5 5.5 1.5 0.5 1.5 3.5 0.5 1.5 5.0 2.6 5.1 4.1 3.2 2.3 1.5 4.4 0.9 1.5 2. 2.3 -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Thu Mar 17 03:46:31 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 17 Mar 2011 09:46:31 +0200 Subject: [Numpy-discussion] avoid a line... In-Reply-To: References: Message-ID: Hi, On Thu, Mar 17, 2011 at 9:36 AM, dileep kunjaai wrote: > Dear sir, > I am try to read a file of the following format, I want to avoid the first > line and read the remaining as 'float' . > Please help me... > > > Rain Wind Temp Pr Sal > 0.1 1.1 0.02 0.2 0.2 > 0.5 0. 0. 0.4 0.8 > 0.5 5.5 1.5 0.5 1.5 > 3.5 0.5 1.5 5.0 2.6 > 5.1 4.1 3.2 2.3 1.5 > 4.4 0.9 1.5 2. 2.3 > You may use loadtxt(), with skiprows argument. (See more on http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html). Regards, eat > > -- > DILEEPKUMAR. R > J R F, IIT DELHI > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From inconnu at list.ru Thu Mar 17 04:44:33 2011 From: inconnu at list.ru (Andrey N. Sobolev) Date: Thu, 17 Mar 2011 13:44:33 +0500 Subject: [Numpy-discussion] Norm of array of vectors Message-ID: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> Dear all, Sorry if that's a noob question, but anyway. I have several thousands of vectors stacked in 2d array. I'd like to get new array containing Euclidean norms of these vectors and get the vector with minimal norm. Is there more efficient way to do this than argmin(array([sqrt(dot(x,x)) for x in vec_array]))? Thanks in advance. Andrey. From e.antero.tammi at gmail.com Thu Mar 17 04:51:19 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 17 Mar 2011 10:51:19 +0200 Subject: [Numpy-discussion] Norm of array of vectors In-Reply-To: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> References: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> Message-ID: Hi, On Thu, Mar 17, 2011 at 10:44 AM, Andrey N. Sobolev wrote: > Dear all, > > Sorry if that's a noob question, but anyway. I have several thousands of > vectors stacked in 2d array. I'd like to get new array containing > Euclidean norms of these vectors and get the vector with minimal norm. > > Is there more efficient way to do this than > argmin(array([sqrt(dot(x,x)) for x in vec_array]))? > Try argmin(sum(vec_array** 2, 0)** 0.5) Regards, eat > > Thanks in advance. > Andrey. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gruben at bigpond.net.au Thu Mar 17 05:58:18 2011 From: gruben at bigpond.net.au (gary ruben) Date: Thu, 17 Mar 2011 20:58:18 +1100 Subject: [Numpy-discussion] Norm of array of vectors In-Reply-To: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> References: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> Message-ID: How about argmin(add.reduce((a*a),axis=1)) In [5]: a Out[5]: array([[ 0.24202827, 0.01269182, 0.95162307], [ 0.02979253, 0.4333354 , 0.49650111], [ 0.52626565, 0.08363861, 0.56444878], [ 0.89639659, 0.54259354, 0.29245881], [ 0.75301013, 0.6248646 , 0.24565827], [ 0.67501358, 0.58920861, 0.37420961], [ 0.8776001 , 0.58055258, 0.16623637], [ 0.26271551, 0.24441225, 0.47543652], [ 0.12793549, 0.88453877, 0.8479841 ], [ 0.49148293, 0.45352964, 0.65575962]]) In [6]: argmin(add.reduce((a*a),axis=1)) Out[6]: 7 In [7]: timeit argmin(array([sqrt(dot(x,x)) for x in a])) 10000 loops, best of 3: 67.2 us per loop In [8]: timeit argmin(array([dot(x,x) for x in a])) 10000 loops, best of 3: 36 us per loop In [9]: timeit argmin(add.reduce((a*a),axis=1)) 100000 loops, best of 3: 13.6 us per loop Gary R On Thu, Mar 17, 2011 at 7:44 PM, Andrey N. Sobolev wrote: > Dear all, > > Sorry if that's a noob question, but anyway. I have several thousands of > vectors stacked in 2d array. I'd like to get new array containing > Euclidean norms of these vectors and get the vector with minimal norm. > > Is there more efficient way to do this than > argmin(array([sqrt(dot(x,x)) for x in vec_array]))? > > Thanks in advance. > Andrey. From dileepkunjaai at gmail.com Thu Mar 17 06:09:26 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Thu, 17 Mar 2011 15:39:26 +0530 Subject: [Numpy-discussion] avoid a line... In-Reply-To: References: Message-ID: Thanks sir,,,,,, thanks a lot ...... On Thu, Mar 17, 2011 at 1:16 PM, eat wrote: > Hi, > > On Thu, Mar 17, 2011 at 9:36 AM, dileep kunjaai wrote: > >> Dear sir, >> I am try to read a file of the following format, I want to avoid the >> first line and read the remaining as 'float' . >> Please help me... >> >> >> Rain Wind Temp Pr Sal >> 0.1 1.1 0.02 0.2 0.2 >> 0.5 0. 0. 0.4 0.8 >> 0.5 5.5 1.5 0.5 1.5 >> 3.5 0.5 1.5 5.0 2.6 >> 5.1 4.1 3.2 2.3 1.5 >> 4.4 0.9 1.5 2. 2.3 >> > You may use loadtxt(), with skiprows argument. (See more on > http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html). > > Regards, > eat > >> >> -- >> DILEEPKUMAR. R >> J R F, IIT DELHI >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From borreguero at gmail.com Thu Mar 17 08:34:17 2011 From: borreguero at gmail.com (Jose Borreguero) Date: Thu, 17 Mar 2011 08:34:17 -0400 Subject: [Numpy-discussion] Build ERROR ConfigParser.MissingSectionHeaderError: File contains no section headers In-Reply-To: References: Message-ID: Thanks Josef, that worked! I was confused because I was thinking of site.cfg as some sort of bash script :) Jose On Thu, Mar 17, 2011 at 12:40 AM, wrote: > On Thu, Mar 17, 2011 at 12:23 AM, Jose Borreguero > wrote: > > Dear Numpy/SciPy users, > > > > I have a build error with Numpy: > > > > $ /usr/local/bin/python2.7 setup.py build > > .... > > File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read > > raise MissingSectionHeaderError(fpname, lineno, line) > > ConfigParser.MissingSectionHeaderError: File contains no section headers. > > file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 > > 'library_dirs = /usr/local/lib\n' > > > > The relevant lines in my site.cfg file: > > > > I think you just need to uncomment all the section headers that you > use, that`s what the exception says > > > > library_dirs = /usr/local/lib > > include_dirs = /usr/local/include > [blas_opt] > > libraries = f77blas, cblas, atlas > [lapack_opt] > > libraries = lapack, f77blas, cblas, atlas > > Josef > > > > > > > I have installed BLAS+LAPACK+ATLAS libraries under /usr/local/lib/atlas > > > > I also installed UMFPACK+AMD+UFConfig+CHOLMOD > > > > I would appreciate any comments. I'm stuck here :( > > > > Best regards, > > Jose M. Borreguero > > > > > > > > Below is the full error traceback: > > > > $ /usr/local/bin/python2.7 setup.py build > > > > Running from numpy source directory.F2PY Version 1 > > Traceback (most recent call last): > > File "setup.py", line 211, in > > setup_package() > > File "setup.py", line 204, in setup_package > > configuration=configuration ) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/core.py", line 152, in > > setup > > config = configuration() > > File "setup.py", line 151, in configuration > > config.add_subpackage('numpy') > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 972, > > in add_subpackage > > caller_level = 2) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 941, > > in get_subpackage > > caller_level = caller_level + 1) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 878, > > in _get_configuration_from_setup_py > > config = setup_module.configuration(*args) > > File "numpy/setup.py", line 9, in configuration > > config.add_subpackage('core') > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 972, > > in add_subpackage > > caller_level = 2) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 941, > > in get_subpackage > > caller_level = caller_level + 1) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/misc_util.py", line > 878, > > in _get_configuration_from_setup_py > > config = setup_module.configuration(*args) > > File "numpy/core/setup.py", line 807, in configuration > > blas_info = get_info('blas_opt',0) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line > 310, > > in get_info > > return cl().get_info(notfound_action) > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line > 409, > > in __init__ > > self.parse_config_files() > > File "/projects/tmp/numpy-1.5.1/numpy/distutils/system_info.py", line > 416, > > in parse_config_files > > self.cp.read(self.files) > > File "/usr/local/lib/python2.7/ConfigParser.py", line 297, in read > > self._read(fp, filename) > > File "/usr/local/lib/python2.7/ConfigParser.py", line 504, in _read > > raise MissingSectionHeaderError(fpname, lineno, line) > > ConfigParser.MissingSectionHeaderError: File contains no section headers. > > file: /projects/tmp/numpy-1.5.1/site.cfg, line: 60 > > 'library_dirs = /usr/local/lib\n' > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yyc at solvcon.net Thu Mar 17 10:23:23 2011 From: yyc at solvcon.net (Yung-Yu Chen) Date: Thu, 17 Mar 2011 10:23:23 -0400 Subject: [Numpy-discussion] Fortran was dead ... [was Re:rewriting NumPy code in C or C++ or similar] In-Reply-To: <4D812F9B.1010309@student.matnat.uio.no> References: <201103161210.25058.lists_ravi@lavabit.com> <4D810B95.4000509@student.matnat.uio.no> <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> <4D812F9B.1010309@student.matnat.uio.no> Message-ID: On Wed, Mar 16, 2011 at 17:46, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > On 03/16/2011 10:14 PM, william ratcliff wrote: > > Related to this, what is the status of fwrap? Can it be used with > > fortran 95/2003 language features? There is a rather large code > > crystallographic codebase (fullprof) that is written in fortran 77 > > that the author has been porting to fortran 95/2003 and actually using > > modules for. I'd like to write python bindings for it to make it more > > scriptable... > > Fwrap 0.1.1 is out; it supports a subset of Fortran 95/2003, biggest > limitation being modules not being present. > > Since then there's been quite a few unreleased improvements (like a much > better and more flexible build based on waf instead of distutils). > > Does Fwrap support SCons? I use SCons everywhere :) Building binaries with distutil is annoying to me, and that's one of the reasons for me to stop using f2py. I am very interested in interfacing binaries with Python. I also started a thread for this topic at Convore: https://convore.com/python-scientific-computing/mixing-languages-with-python/ . For those who might not know, as a result of discussion at the recent PyCon scientific computing BOF, some people started an initiative for an online scientific community focusing on broader application of Python, not specifically to any Python package such as NumPy or SciPy. The Convore group is an early attempt. You can find some context in the following threads: https://convore.com/python-scientific-computing/my-domainspecialty-of-scientific-computing-doesnt-have-a-community/ and https://convore.com/python-scientific-computing/a-scientific-community-using-python/ . yyc > I'm currently working on module support. Or, was ... I've put in a week > this month, but then some simulation results grabbed my attention and I > got derailed. Finishing up module support and making an Fwrap 0.2 > release is #2 on my "stack" of things to work on, and once I get around > to it I expect it to take about a week. So I think it will happen :-) > > If you (or anyone else) want to get involved and put in a day or two to > help with polishing (command line interface, writing tests, > documentation...) just email me. > > Dag Sverre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Yung-Yu Chen PhD candidate of Mechanical Engineering The Ohio State University, Columbus, Ohio +1 (614) 859 2436 http://solvcon.net/yyc/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Thu Mar 17 10:49:48 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 17 Mar 2011 15:49:48 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re:rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <201103161210.25058.lists_ravi@lavabit.com> <4D810B95.4000509@student.matnat.uio.no> <1116300276-1300303983-cardhu_decombobulator_blackberry.rim.net-324447382-@b26.c9.bise7.blackberry> <4D812F9B.1010309@student.matnat.uio.no> Message-ID: <4D821F8C.90009@student.matnat.uio.no> On 03/17/2011 03:23 PM, Yung-Yu Chen wrote: > On Wed, Mar 16, 2011 at 17:46, Dag Sverre Seljebotn > > wrote: > > On 03/16/2011 10:14 PM, william ratcliff wrote: > > Related to this, what is the status of fwrap? Can it be used with > > fortran 95/2003 language features? There is a rather large code > > crystallographic codebase (fullprof) that is written in fortran 77 > > that the author has been porting to fortran 95/2003 and actually > using > > modules for. I'd like to write python bindings for it to make > it more > > scriptable... > > Fwrap 0.1.1 is out; it supports a subset of Fortran 95/2003, biggest > limitation being modules not being present. > > Since then there's been quite a few unreleased improvements (like > a much > better and more flexible build based on waf instead of distutils). > > > Does Fwrap support SCons? I use SCons everywhere :) The current focus is on waf only. But "support" can mean a lot. Fwrap really simply generate .h, .f90 and .pyx files (the latter being Cython sources, which Cython use to generate .c sources). Oh, and there's also a small probe script that tries to probe whether a Fortran "real*8" really is a iso_c_binding c_double, and generates some headers. So when I say we "support" waf, what is meant is we ship a default build system which makes it convenient to build the results of using Fwrap and/or invoke Fwrap on the fly. So supporting scons is certainly a possibility. Last time I checked the Fortran tools in waf were significantly better though (as a result of Kurt's Fwrap work, I believe). Myself I'll be migrating a project from scons to waf any day now. Since the "numscons" effort died/failed it just seems more promising for Python packaging (see Bento). Dag Sverre -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Mar 17 11:32:37 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 17 Mar 2011 08:32:37 -0700 Subject: [Numpy-discussion] avoid a line... In-Reply-To: References: Message-ID: <4D822995.3060904@noaa.gov> On 3/17/11 12:46 AM, eat wrote: > I am try to read a file of the following format, I want to avoid > the first line and read the remaining as 'float' . > Please help me... > > > Rain Wind Temp Pr Sal > 0.1 1.1 0.02 0.2 0.2 > 0.5 0. 0. 0.4 0.8 > 0.5 5.5 1.5 0.5 1.5 It's not as robust, but if performance matters, fromfile() should be faster: f.readline() # to skip the first line arr = np.fromfile(f, sep=' ', dtype=np.float64).reshape((-1, 5)) (untested) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From borreguero at gmail.com Thu Mar 17 13:23:04 2011 From: borreguero at gmail.com (Jose Borreguero) Date: Thu, 17 Mar 2011 13:23:04 -0400 Subject: [Numpy-discussion] ImportError: libatlas.so: cannot open shared object file: No such file or directory Message-ID: Dear Numpy/Scipy users, I just installed numpy but I have an error when importing >>> import numpy .... from numpy.linalg import lapack_lite ImportError: libatlas.so: cannot open shared object file: No such file or directory I have ATLAS libraries under /usr/local/atlas/lib libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a libptf77blas.a libatlas.so libcblas.so libf77blas.so liblapack.so libptcblas.so libptf77blas.so Aso the header files in /usr/local/atlas/include This was my site.cfg file: [DEFAULT] library_dirs = /usr/local/atlas/lib include_dirs = /usr/local/atlas/include [blas_opt] libraries = ptf77blas, ptcblas, atlas [lapack_opt] libraries = lapack, ptf77blas, ptcblas, atlas This is the full Traceback: >>> import numpy Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line 136, in import add_newdocs File "/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 13, in from polynomial import * File "/usr/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 17, in from numpy.linalg import eigvals, lstsq File "/usr/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", line 48, in from linalg import * File "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 23, in from numpy.linalg import lapack_lite ImportError: libatlas.so: cannot open shared object file: No such file or directory I don't understand why libatlas.so cannot be found. Please, any comments are welcomed! Jose M. Borreguero -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischnell at enthought.com Thu Mar 17 13:35:53 2011 From: ischnell at enthought.com (Ilan Schnell) Date: Thu, 17 Mar 2011 12:35:53 -0500 Subject: [Numpy-discussion] ImportError: libatlas.so: cannot open shared object file: No such file or directory In-Reply-To: References: Message-ID: It looks like atlas wasn't linked right. If you /usr/local/atlas/lib to your LIBRARY_PATH environment variable it should work. - Ilan On Thu, Mar 17, 2011 at 12:23 PM, Jose Borreguero wrote: > Dear Numpy/Scipy users, > > I just installed numpy but I have an error when importing >>>> import numpy > .... > ??? from numpy.linalg import lapack_lite > ImportError: libatlas.so: cannot open shared object file: No such file or > directory > > I have ATLAS libraries under /usr/local/atlas/lib > libatlas.a?? libcblas.a?? libf77blas.a?? liblapack.a?? libptcblas.a > libptf77blas.a > libatlas.so? libcblas.so? libf77blas.so? liblapack.so? libptcblas.so > libptf77blas.so > Aso the header files in? /usr/local/atlas/include > > This was my site.cfg file: > [DEFAULT] > library_dirs = /usr/local/atlas/lib > include_dirs = /usr/local/atlas/include > [blas_opt] > libraries = ptf77blas, ptcblas, atlas > [lapack_opt] > libraries = lapack, ptf77blas, ptcblas, atlas > > This is the full Traceback: >>>> import numpy > Traceback (most recent call last): > ? File "", line 1, in > ? File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line 136, > in > ??? import add_newdocs > ? File "/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line > 9, in > ??? from numpy.lib import add_newdoc > ? File "/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line > 13, in > ??? from polynomial import * > ? File "/usr/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", > line 17, in > ??? from numpy.linalg import eigvals, lstsq > ? File "/usr/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", > line 48, in > ??? from linalg import * > ? File "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line > 23, in > ??? from numpy.linalg import lapack_lite > ImportError: libatlas.so: cannot open shared object file: No such file or > directory > > I don't understand why libatlas.so cannot be found. Please, any comments are > welcomed! > > Jose M. Borreguero > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From borreguero at gmail.com Thu Mar 17 14:15:20 2011 From: borreguero at gmail.com (Jose Borreguero) Date: Thu, 17 Mar 2011 14:15:20 -0400 Subject: [Numpy-discussion] ImportError: libatlas.so: cannot open shared object file: No such file or directory In-Reply-To: References: Message-ID: Unfortunately it didn't work. I did: > export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/atlas/lib > python setup.py build > python setup.py install Then in another terminal > export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/atlas/lib > python >>> import numpy >>>... >>>from numpy.linalg import lapack_lite >>> ImportError: libatlas.so: cannot open shared object file: No such file or directory However, I used a different environment variable: > export LD_LIBRARY_PATH=/usr/local/atlas/lib > python >>> import numpy >>>... >>>from numpy.linalg import lapack_lite >>>ImportError: libifport.so.5: cannot open shared object file: No such file or directory I located libifport.so.5 and added the directory to LD_LIBRARY_PATH: >export LD_LIBRARY_PATH=/usr/local/atlas/lib:/opt/intel/Compiler/11.1/069/lib/ia32 > python >>> import numpy >>>... >>>from numpy.linalg import lapack_lite >>>from numpy.linalg import lapack_lite ImportError: /usr/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_concat_string I googled this error, I found some comments on mixing fortran compilers. I haven't advance much because go from error to error. Maybe I'm mixing things up by adding directories to LD_LIBRARY_PATH :( Jose On Thu, Mar 17, 2011 at 1:35 PM, Ilan Schnell wrote: > It looks like atlas wasn't linked right. If you /usr/local/atlas/lib > to your LIBRARY_PATH environment variable it should work. > > - Ilan > > On Thu, Mar 17, 2011 at 12:23 PM, Jose Borreguero > wrote: > > Dear Numpy/Scipy users, > > > > I just installed numpy but I have an error when importing > >>>> import numpy > > .... > > from numpy.linalg import lapack_lite > > ImportError: libatlas.so: cannot open shared object file: No such file or > > directory > > > > I have ATLAS libraries under /usr/local/atlas/lib > > libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a > > libptf77blas.a > > libatlas.so libcblas.so libf77blas.so liblapack.so libptcblas.so > > libptf77blas.so > > Aso the header files in /usr/local/atlas/include > > > > This was my site.cfg file: > > [DEFAULT] > > library_dirs = /usr/local/atlas/lib > > include_dirs = /usr/local/atlas/include > > [blas_opt] > > libraries = ptf77blas, ptcblas, atlas > > [lapack_opt] > > libraries = lapack, ptf77blas, ptcblas, atlas > > > > This is the full Traceback: > >>>> import numpy > > Traceback (most recent call last): > > File "", line 1, in > > File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line > 136, > > in > > import add_newdocs > > File "/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", > line > > 9, in > > from numpy.lib import add_newdoc > > File "/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", > line > > 13, in > > from polynomial import * > > File "/usr/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", > > line 17, in > > from numpy.linalg import eigvals, lstsq > > File "/usr/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", > > line 48, in > > from linalg import * > > File "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", > line > > 23, in > > from numpy.linalg import lapack_lite > > ImportError: libatlas.so: cannot open shared object file: No such file or > > directory > > > > I don't understand why libatlas.so cannot be found. Please, any comments > are > > welcomed! > > > > Jose M. Borreguero > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 17 17:35:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Mar 2011 16:35:27 -0500 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 15:21, Mark Wiebe wrote: > That sounds like a good fix to me. Whenever objects compare equal, they > should hash to the same value. There is a limit to how far we can actually satisfy this requirement. For the implementation of np.dtype.__eq__(), we coerce the other argument to a dtype object. [~] |1> np.dtype(int) == int True [~] |2> np.dtype(int) == 'i4' True [~] |3> hash(int) != hash('i4') True As long as we define np.dtype.__eq__() in that way, we cannot satisfy the requirement. The best we can do is make sure that for all true dtype objects, if they compare equal then they hash equal. So you could make a well-behaved dict with true dtypes as keys, but not dtypes and "dtype-coercable" objects. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mwwiebe at gmail.com Thu Mar 17 17:57:30 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 17 Mar 2011 14:57:30 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: On Thu, Mar 17, 2011 at 2:35 PM, Robert Kern wrote: > On Wed, Mar 16, 2011 at 15:21, Mark Wiebe wrote: > > > That sounds like a good fix to me. Whenever objects compare equal, they > > should hash to the same value. > > There is a limit to how far we can actually satisfy this requirement. > For the implementation of np.dtype.__eq__(), we coerce the other > argument to a dtype object. > > [~] > |1> np.dtype(int) == int > True > > [~] > |2> np.dtype(int) == 'i4' > True > > [~] > |3> hash(int) != hash('i4') > True > > > As long as we define np.dtype.__eq__() in that way, we cannot satisfy > the requirement. The best we can do is make sure that for all true > dtype objects, if they compare equal then they hash equal. So you > could make a well-behaved dict with true dtypes as keys, but not > dtypes and "dtype-coercable" objects. > Dtypes being mutable looks like a serious bug to me, it's violating the definition of 'hashable' given here: http://docs.python.org/glossary.html#term-hashable. This can be used to make an impossible situation: >>> t = np.dtype('i4,i4') >>> d = {t : 3} >>> t.names = ('a','b') >>> d[t] = 4 >>> d {dtype([('a', ' From mesanthu at gmail.com Thu Mar 17 18:17:16 2011 From: mesanthu at gmail.com (santhu kumar) Date: Thu, 17 Mar 2011 17:17:16 -0500 Subject: [Numpy-discussion] Nonzero behaving strangely? Message-ID: Hello all, I am new to Numpy. I used to program before in matlab and am getting used to Numpy. I have a array like: res array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. ], [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. ], [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. ], [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. ], [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. ], [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. ], [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. ], [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. ], [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. ], [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. ], [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. ], [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. ], [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. ], [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. ], [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. ], [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. ], [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. ], [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. ], [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. ]]) The array is 19X5. When I do: nid = (res[:,4]==2).nonzero() nid tuple turns out to be empty. But the very first row satisfies the criteria. nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. Am i doing something wrong? I basically want to find the rows whose fifth coloumn(4th in numpy matrix format) is 2. Any suggestions? Thanks Santhosh -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Mar 17 18:19:42 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Mar 2011 23:19:42 +0100 Subject: [Numpy-discussion] Nonzero behaving strangely? In-Reply-To: References: Message-ID: Hi, Did you try np.where(res[:,4]==2) ? Matthieu 2011/3/17 santhu kumar > Hello all, > > I am new to Numpy. I used to program before in matlab and am getting used > to Numpy. > > I have a array like: > res > array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. > ], > [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. > ], > [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. > ], > [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. > ], > [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. > ], > [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. > ], > [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. > ], > [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. > ], > [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. > ], > [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. > ], > [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. > ], > [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. > ], > [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. > ], > [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. > ], > [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. > ], > [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. > ], > [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. > ], > [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. > ], > [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. > ]]) > > The array is 19X5. > When I do: > nid = (res[:,4]==2).nonzero() > nid tuple turns out to be empty. But the very first row satisfies the > criteria. > > nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. > > Am i doing something wrong? > I basically want to find the rows whose fifth coloumn(4th in numpy matrix > format) is 2. > > Any suggestions? > Thanks > Santhosh > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Mar 17 18:23:19 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 17 Mar 2011 15:23:19 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: Message-ID: <4D8289D7.5030301@noaa.gov> On 3/17/11 2:57 PM, Mark Wiebe wrote: > Dtypes being mutable looks like a serious bug to me, it's violating the > definition of 'hashable' given here: I can imagine other problems is would cause, as well -- is there any reason that dtypes should be mutable? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From warren.weckesser at enthought.com Thu Mar 17 18:27:19 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 17 Mar 2011 17:27:19 -0500 Subject: [Numpy-discussion] Nonzero behaving strangely? In-Reply-To: References: Message-ID: On Thu, Mar 17, 2011 at 5:17 PM, santhu kumar wrote: > Hello all, > > I am new to Numpy. I used to program before in matlab and am getting used > to Numpy. > > I have a array like: > res > array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. > ], > [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. > ], > [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. > ], > [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. > ], > [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. > ], > [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. > ], > [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. > ], > [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. > ], > [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. > ], > [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. > ], > [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. > ], > [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. > ], > [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. > ], > [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. > ], > [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. > ], > [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. > ], > [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. > ], > [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. > ], > [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. > ]]) > > The array is 19X5. > When I do: > nid = (res[:,4]==2).nonzero() > nid tuple turns out to be empty. But the very first row satisfies the > criteria. > > nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. > > Am i doing something wrong? > I basically want to find the rows whose fifth coloumn(4th in numpy matrix > format) is 2. > > Are you sure the value of res[0,4] is *exactly* 2.0, and not something like, say, 2.0000000000000009? Warren > Any suggestions? > Thanks > Santhosh > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Mar 17 18:30:43 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 17 Mar 2011 22:30:43 +0000 (UTC) Subject: [Numpy-discussion] hashing dtypes, new variation, old theme References: <4D8289D7.5030301@noaa.gov> Message-ID: Thu, 17 Mar 2011 15:23:19 -0700, Christopher Barker wrote: > On 3/17/11 2:57 PM, Mark Wiebe wrote: >> Dtypes being mutable looks like a serious bug to me, it's violating the >> definition of 'hashable' given here: > > I can imagine other problems is would cause, as well -- is there any > reason that dtypes should be mutable? Changing field names via the `names` attribute. AFAIK, the other attributes in dtypes are not mutable, and mutability was added long ago as a workaround for `.view()` not handling name changes properly. There's possibly some code out there that touches the `names` attribute, so it's not clear if this can be fixed in the 1.x series. -- Pauli Virtanen From Chris.Barker at noaa.gov Thu Mar 17 18:31:18 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 17 Mar 2011 15:31:18 -0700 Subject: [Numpy-discussion] Nonzero behaving strangely? In-Reply-To: References: Message-ID: <4D828BB6.1080906@noaa.gov> On 3/17/11 3:17 PM, santhu kumar wrote: > nid = (res[:,4]==2).nonzero() > nid tuple turns out to be empty. But the very first row satisfies the > criteria. it works for me: In [45]: arr Out[45]: array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. ], [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. ], [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. ], [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. ], [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. ], [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. ], [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. ], [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. ], [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. ], [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. ], [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. ], [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. ], [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. ], [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. ], [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. ], [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. ], [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. ], [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. ], [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. ]]) In [46]: arr.shape Out[46]: (19, 5) In [47]: nid = (arr[:,4]==2).nonzero() In [48]: nid Out[48]: (array([0]),) Maybe you are having a floating point comparison problem -- i.e.e that "2.0" is really "1.9999999999" or something. Checking equality with floating point numbers is fraught with problems. Also, y9ou amy not need teh nonzero: In [56]: arr[:,4]==2 Out[56]: array([ True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], dtype=bool) that's a boolean array that can be used for indexing, operating with "where", etc. HTH, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mwwiebe at gmail.com Thu Mar 17 18:48:48 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 17 Mar 2011 15:48:48 -0700 Subject: [Numpy-discussion] hashing dtypes, new variation, old theme In-Reply-To: References: <4D8289D7.5030301@noaa.gov> Message-ID: On Thu, Mar 17, 2011 at 3:30 PM, Pauli Virtanen wrote: > Thu, 17 Mar 2011 15:23:19 -0700, Christopher Barker wrote: > > On 3/17/11 2:57 PM, Mark Wiebe wrote: > >> Dtypes being mutable looks like a serious bug to me, it's violating the > >> definition of 'hashable' given here: > > > > I can imagine other problems is would cause, as well -- is there any > > reason that dtypes should be mutable? > > Changing field names via the `names` attribute. AFAIK, the other > attributes in dtypes are not mutable, and mutability was added long ago > as a workaround for `.view()` not handling name changes properly. > > There's possibly some code out there that touches the `names` attribute, > so it's not clear if this can be fixed in the 1.x series. > Fixing ticket #1619 looks like it needs changes to dtype construction and __repr__, probably all these things should be fixed together at once. I agree they likely have too many ripple effects to change for 1.6. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mesanthu at gmail.com Thu Mar 17 22:10:21 2011 From: mesanthu at gmail.com (santhu kumar) Date: Thu, 17 Mar 2011 21:10:21 -0500 Subject: [Numpy-discussion] Nonzero behaving strangely? Message-ID: Thanks a lot for the replies. I have misunderstood the output. When it says (array[0]), i thought it did not find anything as opposed to the zeroth row. (Still getting used to matlab indexing i guess !! ) .. Sorry !! Thanks a lot for the sugestions though, Would keep it in mind .. From: santhu kumar > Subject: [Numpy-discussion] Nonzero behaving strangely? > To: numpy-discussion at scipy.org > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Hello all, > > I am new to Numpy. I used to program before in matlab and am getting used > to > Numpy. > > I have a array like: > res > array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. > ], > [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. > ], > [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. > ], > [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. > ], > [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. > ], > [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. > ], > [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. > ], > [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. > ], > [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. > ], > [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. > ], > [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. > ], > [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. > ], > [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. > ], > [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. > ], > [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. > ], > [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. > ], > [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. > ], > [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. > ], > [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. > ]]) > > The array is 19X5. > When I do: > nid = (res[:,4]==2).nonzero() > nid tuple turns out to be empty. But the very first row satisfies the > criteria. > > nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. > > Am i doing something wrong? > I basically want to find the rows whose fifth coloumn(4th in numpy matrix > format) is 2. > > Any suggestions? > Thanks > Santhosh > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110317/2f46f2c1/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Thu, 17 Mar 2011 23:19:42 +0100 > From: Matthieu Brucher > Subject: Re: [Numpy-discussion] Nonzero behaving strangely? > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > Did you try np.where(res[:,4]==2) ? > > Matthieu > > 2011/3/17 santhu kumar > > > Hello all, > > > > I am new to Numpy. I used to program before in matlab and am getting used > > to Numpy. > > > > I have a array like: > > res > > array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. > > ], > > [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. > > ], > > [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. > > ], > > [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. > > ], > > [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. > > ], > > [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. > > ], > > [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. > > ], > > [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. > > ], > > [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. > > ], > > [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. > > ], > > [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. > > ], > > [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. > > ], > > [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. > > ], > > [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. > > ], > > [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. > > ], > > [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. > > ], > > [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. > > ], > > [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. > > ], > > [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. > > ]]) > > > > The array is 19X5. > > When I do: > > nid = (res[:,4]==2).nonzero() > > nid tuple turns out to be empty. But the very first row satisfies the > > criteria. > > > > nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. > > > > Am i doing something wrong? > > I basically want to find the rows whose fifth coloumn(4th in numpy matrix > > format) is 2. > > > > Any suggestions? > > Thanks > > Santhosh > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110317/3b48f06c/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Thu, 17 Mar 2011 17:27:19 -0500 > From: Warren Weckesser > Subject: Re: [Numpy-discussion] Nonzero behaving strangely? > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Thu, Mar 17, 2011 at 5:17 PM, santhu kumar wrote: > > > Hello all, > > > > I am new to Numpy. I used to program before in matlab and am getting used > > to Numpy. > > > > I have a array like: > > res > > array([[ 33.35053669, 49.4615004 , 44.27631299, 1. , 2. > > ], > > [ 32.84263059, 50.24752036, 43.92291659, 1. , 0. > > ], > > [ 33.68999668, 48.90554673, 43.51746687, 1. , 0. > > ], > > [ 34.11564931, 49.77487763, 44.83843076, 1. , 0. > > ], > > [ 32.4641859 , 48.65469145, 45.09300791, 1. , 3. > > ], > > [ 32.15428526, 49.26922262, 45.92959026, 1. , 0. > > ], > > [ 31.23860825, 48.21824628, 44.30816331, 1. , 0. > > ], > > [ 30.71171138, 47.45600573, 44.9282456 , 1. , 0. > > ], > > [ 30.53843426, 49.07713258, 44.20899822, 1. , 0. > > ], > > [ 31.54722284, 47.61953925, 42.95235178, 1. , 0. > > ], > > [ 32.44334635, 48.10500653, 42.51103537, 1. , 0. > > ], > > [ 31.77269609, 46.53603145, 43.06468455, 1. , 0. > > ], > > [ 30.1820843 , 47.80819604, 41.77667819, 1. , 0. > > ], > > [ 30.78652668, 46.82907769, 40.38586451, 1. , 0. > > ], > > [ 30.05963091, 46.84268609, 39.54583693, 1. , 0. > > ], > > [ 31.75239177, 47.22768463, 40.00717713, 1. , 0. > > ], > > [ 30.94617127, 45.76986265, 40.68226643, 1. , 0. > > ], > > [ 33.20069679, 47.42127403, 45.66738249, 1. , 0. > > ], > > [ 34.39608116, 47.25481126, 45.4438599 , 1. , 0. > > ]]) > > > > The array is 19X5. > > When I do: > > nid = (res[:,4]==2).nonzero() > > nid tuple turns out to be empty. But the very first row satisfies the > > criteria. > > > > nid = (res[:,4]==3).nonzero(), works out and finds the 5th row. > > > > Am i doing something wrong? > > I basically want to find the rows whose fifth coloumn(4th in numpy matrix > > format) is 2. > > > > > > Are you sure the value of res[0,4] is *exactly* 2.0, and not something > like, > say, 2.0000000000000009? > > Warren > > > > Any suggestions? > > Thanks > > Santhosh > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 54, Issue 75 > ************************************************ > -- bye and take care santhu -------------- next part -------------- An HTML attachment was scrubbed... URL: From inconnu at list.ru Fri Mar 18 00:21:01 2011 From: inconnu at list.ru (Andrey N. Sobolev) Date: Fri, 18 Mar 2011 09:21:01 +0500 Subject: [Numpy-discussion] Norm of array of vectors In-Reply-To: References: <20110317134433.50500a30@alsgaard.r249.physics.susu.ac.ru> Message-ID: <20110318092101.79fd2014@aao.localdomain> Hi eat and Gary, Thanks a lot for your suggestions, I've tried them in my code and achieved similar speedup of ~270% for both of them. So I guess I'll stick to one of those. Regards, Andrey. > Hi, > > On Thu, Mar 17, 2011 at 10:44 AM, Andrey N. Sobolev > wrote: > > > Dear all, > > > > Sorry if that's a noob question, but anyway. I have several > > thousands of vectors stacked in 2d array. I'd like to get new array > > containing Euclidean norms of these vectors and get the vector with > > minimal norm. > > > > Is there more efficient way to do this than > > argmin(array([sqrt(dot(x,x)) for x in vec_array]))? > > > Try > argmin(sum(vec_array** 2, 0)** 0.5) > > Regards, > eat > > > > > Thanks in advance. > > Andrey. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ben.root at ou.edu Fri Mar 18 00:47:57 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 17 Mar 2011 23:47:57 -0500 Subject: [Numpy-discussion] split() for slices? Message-ID: Hello, I really like the split() family of functions, but I have the need to split multiple arrays in a similar manner, and it would seem logical to me to have a split() function that would return a list of slice tuples that I could use on multiple arrays. Is there such a function? Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohaq at dearinternet.com Fri Mar 18 16:27:17 2011 From: rohaq at dearinternet.com (Rohaq) Date: Fri, 18 Mar 2011 20:27:17 +0000 Subject: [Numpy-discussion] Extracting data from genfromtxt? Message-ID: I've got a CSV file with the following layout: Location,Result1,Result2 1,0,0 2,0,0 3,1,0 4,1,0 5,1,1 6,0,1 7,0,1 8,0,0 ...etc., and I've loaded it using the following: import numpy as np data = np.genfromtxt('file.csv',delimiter=',',dtype=None,names=True) Resulting in the following: array([(1, 0, 0), (2, 0, 0), (3, 0, 0), ..., (16382, 0, 0), (16383, 0, 0), (16384, 0, 0)], dtype=[('Location', ' References: Message-ID: On Fri, Mar 18, 2011 at 4:27 PM, Rohaq wrote: > I've got a CSV file with the following layout: > > Location,Result1,Result2 > 1,0,0 > 2,0,0 > 3,1,0 > 4,1,0 > 5,1,1 > 6,0,1 > 7,0,1 > 8,0,0 > > ...etc., and I've loaded it using the following: > > import numpy as np > data = np.genfromtxt('file.csv',delimiter=',',dtype=None,names=True) > > Resulting in the following: > > array([(1, 0, 0), (2, 0, 0), (3, 0, 0), ..., (16382, 0, 0), (16383, 0, 0), > ? ? ? (16384, 0, 0)], > ? ? ?dtype=[('Location', ' > I want to plot the data in matplotlib.pyplot, but I'm having trouble > reshaping the data into something it'll recognise. I can pull the > Location easily enough with data['Location'], but I need to get the > results from every other column present. > > There could be a lot more than 2 other columns, Result3,Result4, etc., > so I need to code it to be scalable; if this were acting like a normal > dict, I could write a loop that iterates over the keys, and loads the > results data into a new object, but I can't seem to find a function > that will return the keys so that I can do this. > > Can anybody offer any tips for this? I'm fairly new to Python/Numpy, > so any help is appreciated. > You've got a few options here. If you just want a plain array from genfromtxt in the first place, then you can pass skip_headers=1, to genfromtxt and it should will give you a plain array, if all the data is of a homogeneous type. If you want use your structured array, the most generic way to get an ndarray (I think) is data[list(data.dtype.names)].view((float,len(data.dtype.names)) You can replace float with int, if that's what you want. Skipper From bgamari.foss at gmail.com Fri Mar 18 21:52:38 2011 From: bgamari.foss at gmail.com (Ben Gamari) Date: Fri, 18 Mar 2011 21:52:38 -0400 Subject: [Numpy-discussion] Extracting data from genfromtxt? In-Reply-To: References: Message-ID: <87ei63uba1.fsf@gmail.com> On Fri, 18 Mar 2011 20:27:17 +0000, Rohaq wrote: > There could be a lot more than 2 other columns, Result3,Result4, etc., > so I need to code it to be scalable; if this were acting like a normal > dict, I could write a loop that iterates over the keys, and loads the > results data into a new object, but I can't seem to find a function > that will return the keys so that I can do this. > You could also iterate over data.dtype.fields, which will behave much like dict.keys(). - Ben From rohaq at dearinternet.com Sat Mar 19 10:58:41 2011 From: rohaq at dearinternet.com (Rohaq) Date: Sat, 19 Mar 2011 14:58:41 +0000 Subject: [Numpy-discussion] Extracting data from genfromtxt? In-Reply-To: <87ei63uba1.fsf@gmail.com> References: <87ei63uba1.fsf@gmail.com> Message-ID: On Sat, Mar 19, 2011 at 01:52, Ben Gamari wrote: > On Fri, 18 Mar 2011 20:27:17 +0000, Rohaq wrote: > > There could be a lot more than 2 other columns, Result3,Result4, etc., > > so I need to code it to be scalable; if this were acting like a normal > > dict, I could write a loop that iterates over the keys, and loads the > > results data into a new object, but I can't seem to find a function > > that will return the keys so that I can do this. > > > You could also iterate over data.dtype.fields, which will behave much > like dict.keys(). > > - Ben > Actually I just found last night I could iterate over data.dtype.names, and unlike fields, it seems to print them in the order of the columns from the CSV file :) Thanks for your help though! -- Rohaq -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Sat Mar 19 15:35:01 2011 From: tmp50 at ukr.net (Dmitrey) Date: Sat, 19 Mar 2011 21:35:01 +0200 Subject: [Numpy-discussion] BUG: ndarray subclass calls __mul__ when ** (pow) is involved Message-ID: I have ndarray subclass, its instance x and use r = x**2 I expected it will call for each array element elem.__pow__(2) but it calls elem.__mul__(elem) instead. It essentially (tens or even more times) decreases my calculations speed for lots of cases. >>> numpy.__version__ '2.0.0.dev-1fe8136' (taken some days ago from git) doesn't work with 1.3.0 as well.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 19 18:01:22 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Mar 2011 17:01:22 -0500 Subject: [Numpy-discussion] BUG: ndarray subclass calls __mul__ when ** (pow) is involved In-Reply-To: References: Message-ID: 2011/3/19 Dmitrey : > I have ndarray subclass, its instance x and use > r = x**2 > > I expected it will call for each array element > elem.__pow__(2) > but it calls > elem.__mul__(elem) > instead. > > It essentially (tens or even more times) decreases my calculations speed for > lots of cases. x.__pow__(2) is indeed strength-reduced down to multiplication by default. This occurs in the C implementation of ndarray.__pow__(). Feel free to override __pow__() in your class to directly call np.power() which will just do the power calculation directly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From me at mydis.org Sat Mar 19 22:35:52 2011 From: me at mydis.org (Michael Mersky) Date: Sat, 19 Mar 2011 22:35:52 -0400 Subject: [Numpy-discussion] Cellular Automata Neighborhoods & Numpy Message-ID: Hello, I stumbled upon this group tonight ( http://mail.scipy.org/pipermail/numpy-discussion/2010-October/053420.html) while searching Google for examples of Cellular Automata(CA) using Numpy. The "Game of Life Strides" example looks great, but I don't fully comprehend how this example is working: http://scipy.org/Cookbook/GameOfLifeStrides If I have: >>> world=arange(9).reshape(3,3) >>> world array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) how could I as_strided(world,shape?,strides) such that my 9 cell neighborhood at [0][0] would be something like this (i.e. wrapped): array([[8,6,7], [2,0,1], [5,3,4]]) Greatly appreciate your help! -m -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Sun Mar 20 04:12:11 2011 From: cimrman3 at ntc.zcu.cz (=?utf-8?B?Um9iZXJ0IENpbXJtYW4=?=) Date: Sun, 20 Mar 2011 09:12:11 +0100 Subject: [Numpy-discussion] =?utf-8?q?Cellular_Automata_Neighborhoods_=26_?= =?utf-8?q?Numpy?= Message-ID: <20110320081213.3DB7D6EFD9@scipy.org> Hi Michael, You can find the full game of life script at [1]. There is also Belousov-Zhabotinsky cellular automaton. Both have a strided version. r. [1] http://docs.sfepy.org/scientific-python-tutorial/examples ----- Reply message ----- From: "Michael Mersky" To: Subject: [Numpy-discussion] Cellular Automata Neighborhoods & Numpy Date: Sun, Mar 20, 2011 03:35 Hello, I stumbled upon this group tonight (http://mail.scipy.org/pipermail/numpy-discussion/2010-October/053420.html) while searching Google for examples of Cellular Automata(CA) using Numpy. The "Game of Life Strides" example looks great, but I don't fully comprehend how this example is working: http://scipy.org/Cookbook/GameOfLifeStrides If I have: >>> world=arange(9).reshape(3,3) >>> world array([[0, 1, 2], ????? ?? [3, 4, 5], ????? ?? [6, 7, 8]]) how could I as_strided(world,shape?,strides) such that my 9 cell neighborhood at [0][0] would be something like this (i.e. wrapped): array([[8,6,7], ???????? [2,0,1], ???????? [5,3,4]]) Greatly appreciate your help! -m -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at wbpsystems.com Sun Mar 20 11:08:30 2011 From: ben at wbpsystems.com (Ben Smith) Date: Sun, 20 Mar 2011 08:08:30 -0700 Subject: [Numpy-discussion] What Requires C and what is just python Message-ID: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? Thanks! Ben -- Ben Smith Founder / CSA WBP SYSTEMS http://www.wbpsystems.com From josef.pktd at gmail.com Sun Mar 20 11:44:08 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 20 Mar 2011 11:44:08 -0400 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: On Sun, Mar 20, 2011 at 11:08 AM, Ben Smith wrote: > > So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. > > So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? I think you can package also numpy with py2exe. Overall I think restricting to pure python is a very bad idea if you want to compete with Gauss. Even for a minimal translation of Gauss programs I need at least numpy and scipy, and statsmodels for the econometrics specific parts. linear algebra, optimization and special functions for distributions look like a minimum to me, and some scipy.signal for time series analysis, and more random numbers than in python`s standard library. Pure python will be slow for this and I doubt you will get anyone to switch from Gauss to pure python. Also, I haven`t seen yet a pure python matrix inverse, or linalg solver. If they want to write their own python programs for analysis and use python later on, then they are much better of getting a full python distribution, EPD, pythonxy or similar. Binary distributions are available and just one click or one command installs. And, for example, using Spyder would be a lot nicer and easier for writing scripts, that are equivalent to Gauss scripts, than using commandline tools. I fully agree with the objective of getting python/numpy/scipy tools to get economists, econometricians to switch from gauss or matlab, but to make it competitive we need enough supporting functions and we need the speed that some Monte Carlo simulations don`t take days instead of hours. I hope you are successful with getting economists or econ students to use python. Josef > Thanks! > > Ben > > -- > Ben Smith > Founder / CSA > WBP SYSTEMS > http://www.wbpsystems.com > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From halbert at halwitz.org Sun Mar 20 11:47:53 2011 From: halbert at halwitz.org (Dan Halbert) Date: Sun, 20 Mar 2011 11:47:53 -0400 (EDT) Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: <1300636073.257419466@beta.apps.rackspace.com> On Sunday, March 20, 2011 11:08am, "Ben Smith" said: > ... What I'd like to do is > limit myself to just the functions that are implemented in python, package it with > py2exe and hand that to anyone that needs it. So, my question, if anyone knows, > what's implemented in python and what depends on the c libraries? Is this even > possible? The low-level code is all in non-Python libraries. However, I have used pyinstaller (similar to py2exe) to build self-contained packages that include numpy, and it includes the necessary non-Python libraries in the package. My impression is that py2exe will do the same thing. Also, you don't need to build numpy on everyone's machine. There is a numpy installer for Windows. You could just give people directions for installing Python and Numpy. IPython might be nice too. Other alternatives: The Enthought distribution is free for academic use. Sage is free. There's also Python(x,y), though I don't know much about it. They all include a ton of stuff. Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Mar 20 11:49:47 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 20 Mar 2011 11:49:47 -0400 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: On Sun, Mar 20, 2011 at 11:44 AM, wrote: > On Sun, Mar 20, 2011 at 11:08 AM, Ben Smith wrote: >> >> So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. >> >> So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? > > I think you can package also numpy with py2exe. I should have explained this first: all basic numpy array calculations are in C, extra packages in scipy are often in fortran. numpy.linalg uses C, but scipy.linalg uses the fortran libraries that are the same (LAPACK,..) or similar versions as in GAUSS. numpy.random is in C, scipy.special for distribution functions is in C and fortran. Josef > > Overall I think restricting to pure python is a very bad idea if you > want to compete with Gauss. > Even for a minimal translation of Gauss programs I need at least numpy > and scipy, and statsmodels for the econometrics specific parts. linear > algebra, optimization and special functions for distributions look > like a minimum to me, and some scipy.signal for time series analysis, > and more random numbers than in python`s standard library. > > Pure python will be slow for this and I doubt you will get anyone to > switch from Gauss to pure python. > Also, I haven`t seen yet a pure python matrix inverse, or linalg solver. > > If they want to write their own python programs for analysis and use > python later on, then they are much better of getting a full python > distribution, EPD, pythonxy or similar. Binary distributions are > available and just one click or one command installs. > And, for example, using Spyder would be a lot nicer and easier for > writing scripts, that are equivalent to Gauss scripts, than using > commandline tools. > > I fully agree with the objective of getting python/numpy/scipy tools > to get economists, econometricians to switch from gauss or matlab, but > to make it competitive we need enough supporting functions and we need > the speed that some Monte Carlo simulations don`t take days instead of > hours. > > I hope you are successful with getting economists or econ students to > use python. > > Josef > >> Thanks! >> >> Ben >> >> -- >> Ben Smith >> Founder / CSA >> WBP SYSTEMS >> http://www.wbpsystems.com >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From josef.pktd at gmail.com Sun Mar 20 12:02:45 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 20 Mar 2011 12:02:45 -0400 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: On Sun, Mar 20, 2011 at 11:49 AM, wrote: > On Sun, Mar 20, 2011 at 11:44 AM, ? wrote: >> On Sun, Mar 20, 2011 at 11:08 AM, Ben Smith wrote: >>> >>> So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. this looks interesting on this topic: http://www.vwl.uni-mannheim.de/gaudecker/teaching.htm Josef >>> >>> So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? >> >> I think you can package also numpy with py2exe. > > I should have explained this first: > all basic numpy array calculations are in C, extra packages in scipy > are often in fortran. > numpy.linalg uses C, but scipy.linalg uses the fortran libraries that > are the same (LAPACK,..) or similar versions as in GAUSS. numpy.random > is in C, scipy.special for distribution functions is in C and fortran. > > Josef > >> >> Overall I think restricting to pure python is a very bad idea if you >> want to compete with Gauss. >> Even for a minimal translation of Gauss programs I need at least numpy >> and scipy, and statsmodels for the econometrics specific parts. linear >> algebra, optimization and special functions for distributions look >> like a minimum to me, and some scipy.signal for time series analysis, >> and more random numbers than in python`s standard library. >> >> Pure python will be slow for this and I doubt you will get anyone to >> switch from Gauss to pure python. >> Also, I haven`t seen yet a pure python matrix inverse, or linalg solver. >> >> If they want to write their own python programs for analysis and use >> python later on, then they are much better of getting a full python >> distribution, EPD, pythonxy or similar. Binary distributions are >> available and just one click or one command installs. >> And, for example, using Spyder would be a lot nicer and easier for >> writing scripts, that are equivalent to Gauss scripts, than using >> commandline tools. >> >> I fully agree with the objective of getting python/numpy/scipy tools >> to get economists, econometricians to switch from gauss or matlab, but >> to make it competitive we need enough supporting functions and we need >> the speed that some Monte Carlo simulations don`t take days instead of >> hours. >> >> I hope you are successful with getting economists or econ students to >> use python. >> >> Josef >> >>> Thanks! >>> >>> Ben >>> >>> -- >>> Ben Smith >>> Founder / CSA >>> WBP SYSTEMS >>> http://www.wbpsystems.com >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > From lou_boog2000 at yahoo.com Sun Mar 20 12:54:54 2011 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Sun, 20 Mar 2011 09:54:54 -0700 (PDT) Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: <531426.60206.qm@web34406.mail.mud.yahoo.com> I'll add my $0.02 here. Someone mentioned SAGE. I can say that on the Mac the sage package seems to install very easily and reliably. I've done 4 installations on Macs 10.4 to 10.6. You can do them with one command line. They take a few hours, but all have gone flawlessly. The installation contains a LOT of python stuff (including all the packages mentioned here) and you use it just like any other installation except you need to point to the sage folder. There are examples in the documentation. -- Lou Pecora, my views are my own. From alan.isaac at gmail.com Sun Mar 20 13:45:21 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 20 Mar 2011 13:45:21 -0400 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: <4D863D31.4020902@gmail.com> On 3/20/2011 11:08 AM, Ben Smith wrote: > I'd like to do is limit myself to just the functions that > are implemented in python, package it with py2exe and hand > that to anyone that needs it. So, my question, if anyone > knows, what's implemented in python and what depends on > the c libraries? Perhaps a different approach is preferable. Get them to install the Enthought Python Distribution (free for academic use) and provide some GAUSS function look alikes, along this line http://econpy.googlecode.com/svn/trunk/pytrix/pyGAUSS.py fwiw, Alan Isaac From paul.anton.letnes at gmail.com Mon Mar 21 04:30:56 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Mon, 21 Mar 2011 09:30:56 +0100 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> Message-ID: <2C021F99-AC4A-4DF4-8D58-E63DA23878E2@gmail.com> On 20. mars 2011, at 16.08, Ben Smith wrote: > > So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. > > So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? I can testify that on most windows computers python(x,y) will give you everything you need - numpy, scipy, matplotlib, pyqt for GUI design, and much more. The only problem I ever saw was that some people had problems with $PATH not being set properly on windows. But this was on machines that seemed to be full of other problems. Oh, and in my experience, it is easier to run python scripts from the generic windows command line than in the ipython shell. Good luck, Paul From ralf.gommers at googlemail.com Mon Mar 21 05:34:35 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 21 Mar 2011 10:34:35 +0100 Subject: [Numpy-discussion] 1.6: branching and release notes In-Reply-To: References: Message-ID: On Wed, Mar 16, 2011 at 6:45 AM, Ralf Gommers wrote: > > > On Wed, Mar 16, 2011 at 12:02 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Mar 15, 2011 at 9:29 PM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> >>> >>> On Wed, Mar 16, 2011 at 2:31 AM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> >>>> >>>> On Tue, Mar 15, 2011 at 12:26 PM, Mark Wiebe wrote: >>>> >>>> I pushed one more small API change to PyArray_NewLikeArray, adding a >>>>> 'subok' parameter which lets you disable sub-types. The things missing still >>>>> are documentation (maybe others can help?). The Python nditer exposure is >>>>> undocumented, as well as the new parameters to ufuncs (full list: 'casting', >>>>> 'out', 'order', 'subok', 'dtype'). >>>>> >>>> >>> >>>> We should probably postpone the beta by a few days, there are some other >>>> loose ends floating about. >>>> >>>> Besides documentation the only thing I can think of is the structured >>> array non-existing filed segfault thing. And perhaps #1619 would be good to >>> have fixed, but not essential for a beta IMHO. Anything else? >>> >>> >> Just that things that go in at the last moment should be let sit a bit to >> settle. >> >> True. I will create the branch already though, otherwise there will > continue to be things that just went in. > > Things have settled enough I think, planning to update the API version hash one last time and tag beta1 tonight. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From silyko at gmail.com Mon Mar 21 10:25:45 2011 From: silyko at gmail.com (Simon Lyngby Kokkendorff) Date: Mon, 21 Mar 2011 15:25:45 +0100 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <2C021F99-AC4A-4DF4-8D58-E63DA23878E2@gmail.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> <2C021F99-AC4A-4DF4-8D58-E63DA23878E2@gmail.com> Message-ID: Hi Ben, It's very easy to package numpy (and most other modules) with py2exe, which like Dan mentioned above, will include all necessary (also non-python) libraries into a dist-folder. The folder to distribute can of course get quite large if you include a lot of libraries - but I think that only standard libraries and numpy will be below 5 mb. Cheers, Simon On Mon, Mar 21, 2011 at 9:30 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 20. mars 2011, at 16.08, Ben Smith wrote: > > > > > So, in addition to my computer science work, I'm a PhD student in econ. > Right now, the class is using GAUSS for almost everything. This sort of > pisses me off because it means people are building libraries of code that > become valueless when they graduate (because right now we get GAUSS licenses > for free, but it is absurdly expensive later) -- particularly when this is > the only language they know. > > > > So, I had this idea of building some command line tools to do the same > things using the most basic pieces of NumPy (arrays, dot products, transpose > and inverse -- that's it). And it is going great. My problem however is that > I'd like to be able to share these tools but I know I'm opening up a big can > of worms where I have to go around building numpy on 75 peoples computers. > What I'd like to do is limit myself to just the functions that are > implemented in python, package it with py2exe and hand that to anyone that > needs it. So, my question, if anyone knows, what's implemented in python and > what depends on the c libraries? Is this even possible? > > I can testify that on most windows computers python(x,y) will give you > everything you need - numpy, scipy, matplotlib, pyqt for GUI design, and > much more. > > The only problem I ever saw was that some people had problems with $PATH > not being set properly on windows. But this was on machines that seemed to > be full of other problems. > > Oh, and in my experience, it is easier to run python scripts from the > generic windows command line than in the ipython shell. > > Good luck, > Paul > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Mon Mar 21 12:03:37 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 21 Mar 2011 11:03:37 -0500 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: <531426.60206.qm@web34406.mail.mud.yahoo.com> References: <2C0BE92645A04EE392C3A1906D517453@wbpsystems.com> <531426.60206.qm@web34406.mail.mud.yahoo.com> Message-ID: <4D8776D9.5080007@creativetrax.com> On 3/20/11 11:54 AM, Lou Pecora wrote: > I'll add my $0.02 here. Someone mentioned SAGE. I can say that on the Mac the > sage package seems to install very easily and reliably. I've done 4 > installations on Macs 10.4 to 10.6. You can do them with one command line. > They take a few hours, but all have gone flawlessly. The installation contains > a LOT of python stuff (including all the packages mentioned here) and you use it > just like any other installation except you need to point to the sage folder. > There are examples in the documentation. > -- Lou Pecora, my views are my own. On a mac, you can also download the drag-and-drop Mac App. If you download the *-app.dmg, you can just drag and drop the app to your applications folder. Launch Sage by double-clicking on the app (it pulls up your webbrowser with the notebook frontend). You can also launch terminal sessions with Sage, use Sage from the command line, etc. Or you can compile from source, which is I believe what the above instructions are about. Jason From hmgaudecker at gmail.com Mon Mar 21 12:23:50 2011 From: hmgaudecker at gmail.com (Hans-Martin v. Gaudecker) Date: Mon, 21 Mar 2011 17:23:50 +0100 Subject: [Numpy-discussion] What Requires C and what is just python In-Reply-To: References: Message-ID: <6622ACB1-39D3-4F38-82D0-E1483463324A@gmail.com> >>>> So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. That's precisely the reason we just switched our PhD computational econometrics course from GAUSS to NumPy/SciPy/statsmodels/matplotlib :-) Although some of my colleagues are not completely convinced yet... > this looks interesting on this topic: > > http://www.vwl.uni-mannheim.de/gaudecker/teaching.htm > > Josef Well, the only closely related thing there are the 'installation instructions', other than that it's really an adaption of http://software-carpentry.org/ for economists. >>>> So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. I don't think that will be an issue -- most of my students (in either the course Josef linked to above or the econometrics one) had not used anything else than Stata or Matlab before, let alone installed something on their own. Nobody reported any problems when following the above-mentioned guide. The installers work great now, only thing missing currently is a matplotlib installer on 64-bit MacOS. My 2 cents. Cheers, Hans-Martin From bpederse at gmail.com Mon Mar 21 13:10:54 2011 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 21 Mar 2011 11:10:54 -0600 Subject: [Numpy-discussion] moving window product Message-ID: hi, is there a way to take the product along a 1-d array in a moving window? -- similar to convolve, with product in place of sum? currently, i'm column_stacking the array with offsets of itself into window_size columns and then taking the product at axis 1. like:: w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) window_product = np.product(w, axis=1) but then there are the edge effects/array size issues--like those handled in np.convolve. it there something in numpy/scipy that addresses this. or that does the column_stacking with an offset? thanks, -brent From kwgoodman at gmail.com Mon Mar 21 13:19:09 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 21 Mar 2011 10:19:09 -0700 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 10:10 AM, Brent Pedersen wrote: > hi, is there a way to take the product along a 1-d array in a moving > window? -- similar to convolve, with product in place of sum? > currently, i'm column_stacking the array with offsets of itself into > window_size columns and then taking the product at axis 1. > like:: > > ?w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) > ?window_product = np.product(w, axis=1) > > but then there are the edge effects/array size issues--like those > handled in np.convolve. > it there something in numpy/scipy that addresses this. or that does > the column_stacking with an offset? The Bottleneck package has a fast moving window sum (bn.move_sum and bn.move_nansum). You could use that along with >> a = np.random.rand(5) >> a.prod() 0.015877866878931741 >> np.exp(np.log(a).sum()) 0.015877866878931751 Or you could use strides or scipy.ndimage as in https://github.com/kwgoodman/bottleneck/blob/master/bottleneck/slow/move.py From bpederse at gmail.com Mon Mar 21 13:34:01 2011 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 21 Mar 2011 11:34:01 -0600 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 11:19 AM, Keith Goodman wrote: > On Mon, Mar 21, 2011 at 10:10 AM, Brent Pedersen wrote: >> hi, is there a way to take the product along a 1-d array in a moving >> window? -- similar to convolve, with product in place of sum? >> currently, i'm column_stacking the array with offsets of itself into >> window_size columns and then taking the product at axis 1. >> like:: >> >> ?w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) >> ?window_product = np.product(w, axis=1) >> >> but then there are the edge effects/array size issues--like those >> handled in np.convolve. >> it there something in numpy/scipy that addresses this. or that does >> the column_stacking with an offset? > > The Bottleneck package has a fast moving window sum (bn.move_sum and > bn.move_nansum). You could use that along with > >>> a = np.random.rand(5) >>> a.prod() > ? 0.015877866878931741 >>> np.exp(np.log(a).sum()) > ? 0.015877866878931751 > > Or you could use strides or scipy.ndimage as in > https://github.com/kwgoodman/bottleneck/blob/master/bottleneck/slow/move.py > ah yes, of course. thank you. def moving_product(a, window_size, mode="same"): return np.exp(np.convolve(np.log(a), np.ones(window_size), mode)) i'll have a closer look at your strided version in bottleneck as well. From kwgoodman at gmail.com Mon Mar 21 13:57:26 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 21 Mar 2011 10:57:26 -0700 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 10:34 AM, Brent Pedersen wrote: > On Mon, Mar 21, 2011 at 11:19 AM, Keith Goodman wrote: >> On Mon, Mar 21, 2011 at 10:10 AM, Brent Pedersen wrote: >>> hi, is there a way to take the product along a 1-d array in a moving >>> window? -- similar to convolve, with product in place of sum? >>> currently, i'm column_stacking the array with offsets of itself into >>> window_size columns and then taking the product at axis 1. >>> like:: >>> >>> ?w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) >>> ?window_product = np.product(w, axis=1) >>> >>> but then there are the edge effects/array size issues--like those >>> handled in np.convolve. >>> it there something in numpy/scipy that addresses this. or that does >>> the column_stacking with an offset? >> >> The Bottleneck package has a fast moving window sum (bn.move_sum and >> bn.move_nansum). You could use that along with >> >>>> a = np.random.rand(5) >>>> a.prod() >> ? 0.015877866878931741 >>>> np.exp(np.log(a).sum()) >> ? 0.015877866878931751 >> >> Or you could use strides or scipy.ndimage as in >> https://github.com/kwgoodman/bottleneck/blob/master/bottleneck/slow/move.py >> > > ah yes, of course. thank you. > > def moving_product(a, window_size, mode="same"): > ? ?return np.exp(np.convolve(np.log(a), np.ones(window_size), mode)) > > i'll have a closer look at your strided version in bottleneck as well. I don't know what size problem you are working on or if speed is an issue, but here are some timings: >> a = np.random.rand(1000000) >> window_size = 1000 >> timeit np.exp(np.convolve(np.log(a), np.ones(window_size), 'same')) 1 loops, best of 3: 889 ms per loop >> timeit np.exp(bn.move_sum(np.log(a), window_size)) 10 loops, best of 3: 82.5 ms per loop Most all that time is spent in np.exp(np.log(a)): >> timeit bn.move_sum(a, window_size) 100 loops, best of 3: 3.72 ms per loop So I assume if I made a bn.move_prod the time would be around 200x compared to convolve. BTW, you could do the exp inplace: >> timeit b = bn.move_sum(np.log(a), window_size); np.exp(b, b) 10 loops, best of 3: 76.3 ms per loop From washakie at gmail.com Mon Mar 21 14:01:43 2011 From: washakie at gmail.com (John) Date: Mon, 21 Mar 2011 19:01:43 +0100 Subject: [Numpy-discussion] convert dictionary of arrays into single array In-Reply-To: References: <4D80C985.6030205@gmail.com> Message-ID: Hey all, just wanted to let you know that this works nicely: stacked = np.vstack(MyDict.values()) So long as the dictionary only cotains the recarrays. From warren.weckesser at enthought.com Mon Mar 21 14:19:18 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 21 Mar 2011 13:19:18 -0500 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 12:10 PM, Brent Pedersen wrote: > hi, is there a way to take the product along a 1-d array in a moving > window? -- similar to convolve, with product in place of sum? > currently, i'm column_stacking the array with offsets of itself into > window_size columns and then taking the product at axis 1. > like:: > > w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) > window_product = np.product(w, axis=1) > > but then there are the edge effects/array size issues--like those > handled in np.convolve. > it there something in numpy/scipy that addresses this. or that does > the column_stacking with an offset? > > Here's a quick demo of how you might use the function as_strided from the stride_tricks module to do this. ----- import numpy as np from numpy.lib.stride_tricks import as_strided def moving_window(x, length, step=1): # Assume x is a 1D ndarray r = (x.size - length + step) / step nb = x.itemsize strides = (step*nb, nb) windowed_view = as_strided(x, shape=(r, length), strides=strides) return windowed_view if __name__ == "__main__": x = np.linspace(0,4.0,21) print "x =", x w = moving_window(x, length=4, step=2) print "windowed view of x:" print w p = w.prod(axis=-1) print "windowed products:", p ----- Warren > thanks, > -brent > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpederse at gmail.com Mon Mar 21 14:27:41 2011 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 21 Mar 2011 12:27:41 -0600 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 11:57 AM, Keith Goodman wrote: > On Mon, Mar 21, 2011 at 10:34 AM, Brent Pedersen wrote: >> On Mon, Mar 21, 2011 at 11:19 AM, Keith Goodman wrote: >>> On Mon, Mar 21, 2011 at 10:10 AM, Brent Pedersen wrote: >>>> hi, is there a way to take the product along a 1-d array in a moving >>>> window? -- similar to convolve, with product in place of sum? >>>> currently, i'm column_stacking the array with offsets of itself into >>>> window_size columns and then taking the product at axis 1. >>>> like:: >>>> >>>> ?w = np.column_stack(a[i:-window_size+i] for i in range(0, window_size)) >>>> ?window_product = np.product(w, axis=1) >>>> >>>> but then there are the edge effects/array size issues--like those >>>> handled in np.convolve. >>>> it there something in numpy/scipy that addresses this. or that does >>>> the column_stacking with an offset? >>> >>> The Bottleneck package has a fast moving window sum (bn.move_sum and >>> bn.move_nansum). You could use that along with >>> >>>>> a = np.random.rand(5) >>>>> a.prod() >>> ? 0.015877866878931741 >>>>> np.exp(np.log(a).sum()) >>> ? 0.015877866878931751 >>> >>> Or you could use strides or scipy.ndimage as in >>> https://github.com/kwgoodman/bottleneck/blob/master/bottleneck/slow/move.py >>> >> >> ah yes, of course. thank you. >> >> def moving_product(a, window_size, mode="same"): >> ? ?return np.exp(np.convolve(np.log(a), np.ones(window_size), mode)) >> >> i'll have a closer look at your strided version in bottleneck as well. > > I don't know what size problem you are working on or if speed is an > issue, but here are some timings: > >>> a = np.random.rand(1000000) >>> window_size = 1000 >>> timeit np.exp(np.convolve(np.log(a), np.ones(window_size), 'same')) > 1 loops, best of 3: 889 ms per loop >>> timeit np.exp(bn.move_sum(np.log(a), window_size)) > 10 loops, best of 3: 82.5 ms per loop > > Most all that time is spent in np.exp(np.log(a)): > >>> timeit bn.move_sum(a, window_size) > 100 loops, best of 3: 3.72 ms per loop > > So I assume if I made a bn.move_prod the time would be around 200x > compared to convolve. > > BTW, you could do the exp inplace: > >>> timeit b = bn.move_sum(np.log(a), window_size); np.exp(b, b) > 10 loops, best of 3: 76.3 ms per loop > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > my current use-case is to do this 24 times on arrays of about 200K elements. file IO is the major bottleneck. From kwgoodman at gmail.com Mon Mar 21 15:00:31 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 21 Mar 2011 12:00:31 -0700 Subject: [Numpy-discussion] moving window product In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 11:27 AM, Brent Pedersen wrote: > my current use-case is to do this 24 times on arrays of about 200K elements. > file IO is the major bottleneck. Would using h5py or pytables help? I get about 3 ms for a write-read cycle for a 200K array. That's much faster than the convolve speed. >> import h5py >> import time >> f = h5py.File('speed.h5py') >> a = np.random.rand(200000) >> t1 = time.time(); f['a'] = a; b = f['a'][:]; t2 = time.time() >> print "%0.2f ms" % (1000*(t2 - t1)) 3.30 ms From ben.root at ou.edu Mon Mar 21 16:11:46 2011 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 21 Mar 2011 15:11:46 -0500 Subject: [Numpy-discussion] argmin and datetimes Message-ID: Came across an odd behavior with a numpy array of datetimes: >>> import numpy as np >>> from datetime import datetime >>> a = np.empty((9, 20), dtype=datetime) >>> a.fill(datetime.now()) This works: >>> a.min() datetime.datetime(2011, 3, 21, 15, 5, 10, 990733) but, this fails: >>> np.argmin(a) Traceback (most recent call last): File "", line 1, in File "/home/bvr/Programs/numpy/numpy/core/fromnumeric.py", line 754, in argmin return argmin(axis) TypeError: unsupported operand type(s) for -: 'int' and 'datetime.datetime' Is this a bug, or am I just doing datetimes wrong? For reference, I am at commit aada93306acfb4e2eb816faf32652edf8825cf45 Thanks, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Mar 21 16:26:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 21 Mar 2011 15:26:27 -0500 Subject: [Numpy-discussion] argmin and datetimes In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 15:11, Benjamin Root wrote: > Came across an odd behavior with a numpy array of datetimes: > >>>> import numpy as np >>>> from datetime import datetime >>>> a = np.empty((9, 20), dtype=datetime) >>>> a.fill(datetime.now()) > > This works: >>>> a.min() > datetime.datetime(2011, 3, 21, 15, 5, 10, 990733) > > but, this fails: >>>> np.argmin(a) > Traceback (most recent call last): > ? File "", line 1, in > ? File "/home/bvr/Programs/numpy/numpy/core/fromnumeric.py", line 754, in > argmin > ??? return argmin(axis) > TypeError: unsupported operand type(s) for -: 'int' and 'datetime.datetime' > > Is this a bug, or am I just doing datetimes wrong? Heh. x.argmin() is implemented as (0-x).argmax(). It should probably just be implemented. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From SSharma84 at slb.com Mon Mar 21 21:00:27 2011 From: SSharma84 at slb.com (Sachin Kumar Sharma) Date: Tue, 22 Mar 2011 01:00:27 +0000 Subject: [Numpy-discussion] Suggestion about reading text file and extracting data LAS format (newbie questions) Message-ID: <75C2FED246299A478280FA1470EDA4432D92375A@NL0230MBX06N1.DIR.slb.com> BB, I have picked python recently and use it for simple day to day task. I am looking a routine for loading Well Log Las format and storing information as arrays or List. Las format looks like following # LAS format # Project units are specified as depth units #================================================================== ~Version information VERS. 2.0: WRAP. NO: #================================================================== ~Well STRT .ft 1900.0000000 : STOP .ft 6725.0000000 : STEP .ft 0.50000000 : NULL . -999.250000 : COMP. : COMPANY WELL. : WELL FLD. : FIELD LOC. : LOCATION SRVC. : SERVICE COMPANY DATE. Tuesday, March 22 2011 08:31:52 : DATE PROV. : PROVINCE UWI. : UNIQUE WELL ID API. : API NUMBER #================================================================== ~Curve DEPT .ft : DEPTH PHIE .ft3/ft3 : POROSITY SW . : WATER SATURATION VCLfin . : VCL FACIES_NN . : FACIES KairHU .mD : PERM ~Parameter #================================================================== 2711.0000000 0.2031999975 1.0000000000 0.1447000057 5.0000000000 171.17599487 2711.5000000 0.2098000050 1.0000000000 0.1421000063 5.0000000000 199.55209351 2712.0000000 0.2072000057 1.0000000000 0.1570000052 5.0000000000 200.47390747 2712.5000000 0.2063000053 1.0000000000 0.1530999988 5.0000000000 192.77430725 2713.0000000 0.2161999941 1.0000000000 0.1184000000 5.0000000000 208.76330566 2713.5000000 0.2213999927 1.0000000000 0.0970999971 5.0000000000 214.45840454 2714.0000000 0.2188999951 1.0000000000 0.0939999968 5.0000000000 198.89270020 2714.5000000 0.2120999992 1.0000000000 0.1231999993 5.0000000000 192.96949768 2715.0000000 0.2094999999 1.0000000000 0.1480000019 5.0000000000 203.21719360 2715.5000000 0.2054000050 1.0000000000 0.1673000008 6.0000000000 201.36019897 2716.0000000 0.2071000040 1.0000000000 0.1544999927 5.0000000000 197.63549805 2716.5000000 0.2006999999 1.0000000000 0.1601999998 5.0000000000 173.38989258 2717.0000000 0.2045000046 1.0000000000 0.1494999975 5.0000000000 181.11289978 2717.5000000 0.2016000003 1.0000000000 0.1485999972 5.0000000000 167.39660645 2718.0000000 0.2023999989 1.0000000000 0.1332000047 5.0000000000 158.64739990 2718.5000000 0.1915999949 1.0000000000 0.1460999995 5.0000000000 127.54989624 2719.0000000 0.1824000031 1.0000000000 0.1656000018 5.0000000000 110.23919678 2719.5000000 0.1812999994 1.0000000000 0.1741999984 5.0000000000 111.93460083 2720.0000000 0.1774999946 1.0000000000 0.1779000014 5.0000000000 102.91439819 2720.5000000 0.1650000066 1.0000000000 0.1888999939 5.0000000000 76.584999084 2721.0000000 0.1314000040 1.0000000000 0.2365999967 5.0000000000 35.996101379 2721.5000000 0.1238999963 1.0000000000 0.2381000072 5.0000000000 28.215700150 2722.0000000 0.1242000014 1.0000000000 0.2396000028 5.0000000000 28.818500519 2722.5000000 0.1412999928 1.0000000000 0.2210000008 5.0000000000 45.044498444 2723.0000000 0.1543000042 1.0000000000 0.2046999931 5.0000000000 61.145401001 2723.5000000 0.1601999998 1.0000000000 0.2020999938 5.0000000000 71.653701782 2724.0000000 0.1479000002 1.0000000000 0.2287999988 5.0000000000 57.839698792 2724.5000000 0.1274999976 1.0000000000 0.2714999914 7.0000000000 39.379501343 2725.0000000 0.1230000034 1.0000000000 0.2777000070 7.0000000000 35.316699982 2725.5000000 0.1237000003 1.0000000000 0.2709000111 7.0000000000 34.647201538 Blue shows the header precedes the data (shown in green). The file normally contains between 3000-10000 data rows. If someone can share a routine existing for the same or Can advise what is the best way to read these lines. Currently, I edit these files by hand to remove the headers and use something like this to plot X = numpy.loadtxt('ASCII.txt') # X is an array t = X [: ,0] # extract the first column v = X [: ,1] # extract the second column len (t) len (v) plot (t,v) # plot the data Is there a way, I can use readline option to direct to read from where the data starts and pick the name of columns from header like porosity, permeability. Cheers Sachin ************************************************************************ Sachin Kumar Sharma Senior Geomodeler -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Mon Mar 21 23:03:45 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 21 Mar 2011 22:03:45 -0500 Subject: [Numpy-discussion] Suggestion about reading text file and extracting data LAS format (newbie questions) In-Reply-To: <75C2FED246299A478280FA1470EDA4432D92375A@NL0230MBX06N1.DIR.slb.com> References: <75C2FED246299A478280FA1470EDA4432D92375A@NL0230MBX06N1.DIR.slb.com> Message-ID: On Mon, Mar 21, 2011 at 8:00 PM, Sachin Kumar Sharma wrote: > BB, > > > > I have picked python recently and use it for simple day to day task. I am > looking a routine for loading Well Log Las format and storing information as > arrays or List. > > > > Las format looks like following > Sachin, Take a look at pyLASDev: http://pylasdev.sourceforge.net/, or try the attached file for a quick-and-dirty, not full tested LAS reader. There are a couple problems with your sample file. The DATE line in the Well section uses colons in the time (08:31:52), but there should be only one colon in a line. The LAS reader that I wrote avoids that problem by only using the last colon as the delimiter of the description. More importantly, your sample doesn't include the line ~A to begin the log data. This is required. The lines where the data begin should look something like this: #================================================================== ~A 2711.0000000 0.2031999975 1.0000000000 0.1447000057 5.0000000000 171.17599487 Warren > > # LAS format > > # Project units are specified as depth units > > #================================================================== > > ~Version information > > VERS. 2.0: > > WRAP. NO: > > #================================================================== > > ~Well > > STRT .ft 1900.0000000 : > > STOP .ft 6725.0000000 : > > STEP .ft 0.50000000 : > > NULL . -999.250000 : > > COMP. : COMPANY > > WELL. : WELL > > FLD. : FIELD > > LOC. : LOCATION > > SRVC. : SERVICE COMPANY > > DATE. Tuesday, March 22 2011 08:31:52 : DATE > > PROV. : PROVINCE > > UWI. : UNIQUE WELL ID > > API. : API NUMBER > > #================================================================== > > ~Curve > > DEPT .ft : DEPTH > > PHIE .ft3/ft3 : POROSITY > > SW . : WATER SATURATION > > VCLfin . : VCL > > FACIES_NN . : FACIES > > KairHU .mD : PERM > > ~Parameter > > #================================================================== > > > > 2711.0000000 0.2031999975 1.0000000000 0.1447000057 5.0000000000 > 171.17599487 > > 2711.5000000 0.2098000050 1.0000000000 0.1421000063 5.0000000000 > 199.55209351 > > 2712.0000000 0.2072000057 1.0000000000 0.1570000052 5.0000000000 > 200.47390747 > > 2712.5000000 0.2063000053 1.0000000000 0.1530999988 5.0000000000 > 192.77430725 > > 2713.0000000 0.2161999941 1.0000000000 0.1184000000 5.0000000000 > 208.76330566 > > 2713.5000000 0.2213999927 1.0000000000 0.0970999971 5.0000000000 > 214.45840454 > > 2714.0000000 0.2188999951 1.0000000000 0.0939999968 5.0000000000 > 198.89270020 > > 2714.5000000 0.2120999992 1.0000000000 0.1231999993 5.0000000000 > 192.96949768 > > 2715.0000000 0.2094999999 1.0000000000 0.1480000019 5.0000000000 > 203.21719360 > > 2715.5000000 0.2054000050 1.0000000000 0.1673000008 6.0000000000 > 201.36019897 > > 2716.0000000 0.2071000040 1.0000000000 0.1544999927 5.0000000000 > 197.63549805 > > 2716.5000000 0.2006999999 1.0000000000 0.1601999998 5.0000000000 > 173.38989258 > > 2717.0000000 0.2045000046 1.0000000000 0.1494999975 5.0000000000 > 181.11289978 > > 2717.5000000 0.2016000003 1.0000000000 0.1485999972 5.0000000000 > 167.39660645 > > 2718.0000000 0.2023999989 1.0000000000 0.1332000047 5.0000000000 > 158.64739990 > > 2718.5000000 0.1915999949 1.0000000000 0.1460999995 5.0000000000 > 127.54989624 > > 2719.0000000 0.1824000031 1.0000000000 0.1656000018 5.0000000000 > 110.23919678 > > 2719.5000000 0.1812999994 1.0000000000 0.1741999984 5.0000000000 > 111.93460083 > > 2720.0000000 0.1774999946 1.0000000000 0.1779000014 5.0000000000 > 102.91439819 > > 2720.5000000 0.1650000066 1.0000000000 0.1888999939 5.0000000000 > 76.584999084 > > 2721.0000000 0.1314000040 1.0000000000 0.2365999967 5.0000000000 > 35.996101379 > > 2721.5000000 0.1238999963 1.0000000000 0.2381000072 5.0000000000 > 28.215700150 > > 2722.0000000 0.1242000014 1.0000000000 0.2396000028 5.0000000000 > 28.818500519 > > 2722.5000000 0.1412999928 1.0000000000 0.2210000008 5.0000000000 > 45.044498444 > > 2723.0000000 0.1543000042 1.0000000000 0.2046999931 5.0000000000 > 61.145401001 > > 2723.5000000 0.1601999998 1.0000000000 0.2020999938 5.0000000000 > 71.653701782 > > 2724.0000000 0.1479000002 1.0000000000 0.2287999988 5.0000000000 > 57.839698792 > > 2724.5000000 0.1274999976 1.0000000000 0.2714999914 7.0000000000 > 39.379501343 > > 2725.0000000 0.1230000034 1.0000000000 0.2777000070 7.0000000000 > 35.316699982 > > 2725.5000000 0.1237000003 1.0000000000 0.2709000111 7.0000000000 > 34.647201538 > > > > Blue shows the header precedes the data (shown in green). The file normally > contains between 3000-10000 data rows. > > > > If someone can share a routine existing for the same or Can advise what is > the best way to read these lines. Currently, I edit these files by hand to > remove the headers and use something like this to plot > > > > X = numpy.loadtxt('ASCII.txt') # X is an array > > t = X [: ,0] # extract the first column > > v = X [: ,1] # extract the second column > > len (t) > > len (v) > > plot (t,v) # plot the data > > > > Is there a way, I can use readline option to direct to read from where the > data starts and pick the name of columns from header like porosity, > permeability. > > > > Cheers > > > > Sachin > > ************************************************************************* > Sachin Kumar Sharma* > > Senior Geomodeler ** > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: las.py Type: application/octet-stream Size: 2829 bytes Desc: not available URL: From giuseppe.aprea at gmail.com Tue Mar 22 05:13:26 2011 From: giuseppe.aprea at gmail.com (Giuseppe Aprea) Date: Tue, 22 Mar 2011 10:13:26 +0100 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <201008180913.12594.eg@fft.be> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> <201008180913.12594.eg@fft.be> Message-ID: Hi all, I wonder if Peter finally got Gotoblas working with numpy. I am trying with gotoblas 1.13 installed in the same way: $ ls -R .: include lib ./include: goto ./include/goto: blaswrap.h cblas.h clapack.h f2c.h ./lib: libgoto2.a libgoto2_nehalemp-r1.13.a libgoto2_nehalemp-r1.13.so libgoto2.so and numpy 1.5.1 with this site.cfg [DEFAULT] library_dirs = /usr/local/gcc/4.5.2/gcc/lib64:/usr/local/gcc/4.5.2/gcc/lib:/usr/local/gcc/4.5.2/gcc/lib32:/usr/local/gcc/4.5.2/gotoblas2/1.13/lib:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/lib:/usr/local/gcc/4.5.2/fftw/3.2.2/lib include_dirs = /usr/local/gcc/4.5.2/gcc/include:/usr/local/gcc/4.5.2/gotoblas2/1.13/include/goto:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/include:/usr/local/gcc/4.5.2/fftw/3.2.2/include search_static_first = 1 [blas_opt] libraries = goto2 language = fortran [lapack_opt] libraries = goto2 language = fortran [amd] amd_libs = amd [umfpack] umfpack_libs = umfpack [fftw] libraries = fftw3 (I also tried without "_opt" and "language = fortran"); I used goto2 for lapack because I read lapack should be included in libgoto (anyway things do not change using "lapack"). I am quite sure the system is not using my goto-lapack stuff since every time I buid i get: building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources creating build/src.linux-x86_64-2.7/numpy/linalg *### Warning: Using unoptimized lapack ###* * adding 'numpy/linalg/lapack_litemodule.c' to sources.* adding 'numpy/linalg/python_xerbla.c' to sources. adding 'numpy/linalg/zlapack_lite.c' to sources. adding 'numpy/linalg/dlapack_lite.c' to sources. adding 'numpy/linalg/blas_lite.c' to sources. adding 'numpy/linalg/dlamch.c' to sources. adding 'numpy/linalg/f2c_lite.c' to sources. building extension "numpy.random.mtrand" sources creating build/src.linux-x86_64-2.7/numpy/random C compiler: /usr/local/gcc/4.5.2//gcc/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O1 -pthread -fPIC -march=native -mtune=native -I/usr/local/gcc/4.5.2//gcc/include -I/usr/local/gcc/4.5.2//suiteSparse/3.6.0/include -I/usr/local/gcc/4.5.2//fftw/3.2.2/include -fPIC during numpy installation (which ends successfully). Moreover I cannot see any -lgoto2 as I would have expected. Incidentally, I cannot see -lamd, -lumfpack, -lfftw3 (or any reference to amd, umfpack, fftw3) neither, although there seems to be something to handle them in system_info.py. The failure is so complete that I must have done some big mistake but I can't correct my site.cfg even after searching the internet. This seems to be one of the major discussion about this topic so I am asking here for some help, please. Is the problem related with site.cfg or with gotoblas2 installation? Is it true that gotoblas2 hosts a full lapack inside? thank you very much! giuseppe -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.becker at amolf.nl Tue Mar 22 06:28:51 2011 From: n.becker at amolf.nl (Nils Becker) Date: Tue, 22 Mar 2011 11:28:51 +0100 Subject: [Numpy-discussion] np.histogramdd of empty data Message-ID: <4D8879E3.1020603@amolf.nl> Hi, I was wondering why histogram2d and histogramdd raise a ValueError when fed with empty data of the correct dimensions. I came across this as a corner case when calling histogram2d from my own specialized histogram function. In comparison, histogram does handle this case correctly when bins are specified explicitely (obviously automatic binning cannot work without data...) np.histogram([], bins=([0,1])) (array([0]), array([0, 1])) np.histogram2d([],[], bins=([0,1],[0,1])) ValueError: zero-size array to ufunc.reduce without identity np.histogramdd([[],[]], bins=([0,1],[0,1])) ValueError: zero-size array to ufunc.reduce without identity cheers, nils From alessandro.sanginario at polito.it Tue Mar 22 06:37:18 2011 From: alessandro.sanginario at polito.it (Alessandro) Date: Tue, 22 Mar 2011 11:37:18 +0100 Subject: [Numpy-discussion] Appending a numpy array to binary file Message-ID: <201103221137.18997.alessandro.sanginario@polito.it> Hi everybody, I'm trying to append, inside a loop, into a binary file some arrays using numpy.save(file, arr) but wvhen I open it there is only the last array I saved. If I use savetxt it works perfectly. Do you know how can I append data to a binary file inside a loop? Example: x = arange(10) fp = open("TempFile","ab") for i in range(3): save(fp,x) fp.close() Data saved into the file is: [0,1,2,3,4,5,6,7,8,9] but I would like: [0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9 0,1,2,3,4,5,6,7,8,9] Thank you. -- ------------------------------------------------------------------------------ Alessandro Sanginario Politecnico di Torino Department of Electronics C.so Duca degli Abruzzi, 24 10129 Torino - Italy Tel. +39 0115644151 Fax. +39 0115644099 Skype: sangiox ------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Mar 22 11:22:07 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Mar 2011 10:22:07 -0500 Subject: [Numpy-discussion] Appending a numpy array to binary file In-Reply-To: <201103221137.18997.alessandro.sanginario@polito.it> References: <201103221137.18997.alessandro.sanginario@polito.it> Message-ID: On Tue, Mar 22, 2011 at 05:37, Alessandro wrote: > Hi everybody, > > I'm trying to append, inside a loop, into a binary file some arrays using > > numpy.save(file, arr) but wvhen I open it there is only the last array I > saved. > > If I use savetxt it works perfectly. Do you know how can I append data to a > binary file inside a loop? > > Example: > > x = arange(10) > > fp = open("TempFile","ab") > > for i in range(3): > > save(fp,x) > > fp.close() > > Data saved into the file is: [0,1,2,3,4,5,6,7,8,9] but I would like: > [0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9 0,1,2,3,4,5,6,7,8,9] The data is actually saved three times, which you can verify by looking at the actual bytes in the file. The NPY format is self-describing. It contains a header that describes the shape and dtype of the data. So you get three header+data blocks in the file. You cannot implicitly concatenate three different arrays together by repeatedly saving them out to a file using np.save(). Using np.load() once will simply load the first array in the file. You can read all three arrays by doing repeated np.load() calls on the same file object: fp = open('TempFile', 'rb') for i in range(3): print np.load(fp) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Tue Mar 22 13:29:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 22 Mar 2011 18:29:25 +0100 Subject: [Numpy-discussion] git tag question/warning Message-ID: Hi, After tagging 1.6.0b1 and pushing it I found some issues that only showed up with released=True in setup.py. According to the docs at http://www.kernel.org/pub/software/scm/git/docs/git-tag.html there is a sane (just make a new tag) and an insane (delete tag, reuse same tag) way to deal with this. Since the tagged version will actually not compile due to a version string issue in setup.py I want to go the insane way. Apologies. Please correct me if there's a better option. If you pulled from the github repo in the last few hours, please check if you have a new local tag and delete it. Changes to go in https://github.com/rgommers/numpy/tree/fixes-1.6.0b1 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Tue Mar 22 14:30:04 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 22 Mar 2011 11:30:04 -0700 Subject: [Numpy-discussion] git tag question/warning In-Reply-To: References: Message-ID: I noticed that the nditer documentation didn't get linked properly, this fixes it. You might want to cherry-pick the patch I committed to master so it shows up. -Mark On Tue, Mar 22, 2011 at 10:29 AM, Ralf Gommers wrote: > Hi, > > After tagging 1.6.0b1 and pushing it I found some issues that only showed > up with released=True in setup.py. According to the docs at > http://www.kernel.org/pub/software/scm/git/docs/git-tag.html there is a > sane (just make a new tag) and an insane (delete tag, reuse same tag) way to > deal with this. Since the tagged version will actually not compile due to a > version string issue in setup.py I want to go the insane way. Apologies. > Please correct me if there's a better option. If you pulled from the github > repo in the last few hours, please check if you have a new local tag and > delete it. > > Changes to go in https://github.com/rgommers/numpy/tree/fixes-1.6.0b1 > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Tue Mar 22 15:40:03 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Tue, 22 Mar 2011 12:40:03 -0700 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error Message-ID: Hello, I've been trying out intel's compiler with python and so far things have been going pretty well. I managed to compile python 2.7.1 with icc (with help from http://software.intel.com/en-us/forums/showthread.php?t=56652), so distutils automatically defaults to the icc compiler. So far a number of packages work great, so the overall system seems to work fine. However, I'm running into a few problems compiling numpy. I'm using the latest development build from git (rev ac3cba). The first is that some typing for long double isn't getting recognized -- building numpy breaks with the following error (I'll also post a bug report on this if requested). Traceback (most recent call last): File "setup.py", line 201, in setup_package() File "setup.py", line 194, in setup_package configuration=configuration ) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/hoytak/sysroot/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/home/hoytak/sysroot/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/home/hoytak/sysroot/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 442, in generate_config_h rep = check_long_double_representation(config_cmd) File "numpy/core/setup_common.py", line 136, in check_long_double_representation type = long_double_representation(pyod(object)) File "numpy/core/setup_common.py", line 280, in long_double_representation raise ValueError("Could not lock sequences (%s)" % saw) ValueError: Could not lock sequences (None) This happens regardless of whether I use the --compiler=intel option. I'm using one of the new sandy bridge processors (i7-2600k), which may be part of the reason. Looking at the code, it seems this comes from a failure to detect the type info from a particular string. I've attached a dump of the lines variable passed to the long_double_representation function. As a workaround, I set it manually to return 'INTEL_EXTENDED_12_BYTES_LE' with the idea that I could test this using the unit tests later on. When I did that, everything else compiled fine and numpy installed. However, attempting to import numpy gives me this: >>> import numpy Traceback (most recent call last): File "", line 1, in File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/__init__.py", line 137, in import add_newdocs File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in from type_check import * File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in import multiarray ImportError: /home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: npy_half_to_float In case it's relevant: $ ldd /home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/multiarray.so linux-vdso.so.1 => (0x00007fff60fff000) libpython2.7.so.1.0 => /home/hoytak/sysroot/lib/libpython2.7.so.1.0 (0x00007f5916dc3000) libimf.so => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libimf.so (0x00007f59169df000) libsvml.so => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libsvml.so (0x00007f5916337000) libm.so.6 => /lib/libm.so.6 (0x00007f5916093000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f5915e7c000) libintlc.so.5 => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libintlc.so.5 (0x00007f5915d2d000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f5915b10000) libc.so.6 => /lib/libc.so.6 (0x00007f591578c000) libdl.so.2 => /lib/libdl.so.2 (0x00007f5915588000) libutil.so.1 => /lib/libutil.so.1 (0x00007f5915385000) /lib64/ld-linux-x86-64.so.2 (0x00007f591770c000) I also attached the build log here (which is the same for the first error up to the crash). Any suggestions of where to look or where something could have gone wrong? Thanks! -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log.gz Type: application/x-gzip Size: 10983 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: lines_dump.p.gz Type: application/x-gzip Size: 3385 bytes Desc: not available URL: From ckkart at hoc.net Tue Mar 22 16:54:54 2011 From: ckkart at hoc.net (Christian K.) Date: Tue, 22 Mar 2011 21:54:54 +0100 Subject: [Numpy-discussion] fast robus implementation of float() Message-ID: Hi, I wonder if someone has a good solution for a fast conversion of gridded ascii data to ndarray. It should manage ',' as decimal point (on demand) and special windows numbers as 1.#INF. Of course, this is easy to wrap in a small function but I expect it to be slow when the input size is in the Mb range. Regards, Christian From paul.anton.letnes at gmail.com Tue Mar 22 18:18:09 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Tue, 22 Mar 2011 23:18:09 +0100 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> <201008180913.12594.eg@fft.be> Message-ID: I'm no expert, but I just pulled off the scipy+numpy+GotoBLAS2 installation. >From what I gather, the Makefile for libgoto2 downloads and compiles the generic lapack from netlib. It also wraps lapack into libgoto2.so/.a. I believe the idea is as long as the BLAS implementation is fast(TM), the lapack performance will be good. To wit*, what I did was to tell numpy where libgoto2 was: env BLAS=/path/to/libgoto2.so python setup.py install Scipy also wants the path to lapack, which is wrapped inside libgoto2: env BLAS=/path/to/libgoto2.so LAPACK=/path/to/libgoto2.so python setup.py install Afterwards, I added the path to LD_LIBRARY_PATH. This was on a linux cluster, if that matters. At any rate, I can testify that it was not a big job to get numpy and scipy working with goto blas. Good luck, Paul. *) I have notes on this on a different computer, but not available right now. On Tue, Mar 22, 2011 at 10:13 AM, Giuseppe Aprea wrote: > Hi all, > > I wonder if Peter finally got Gotoblas working with numpy. I am trying with > gotoblas 1.13 installed in the same way: > > $ ls -R > .: > include lib > > ./include: > goto > > ./include/goto: > blaswrap.h cblas.h clapack.h f2c.h > > ./lib: > libgoto2.a libgoto2_nehalemp-r1.13.a libgoto2_nehalemp-r1.13.so libgoto2.so > > and numpy 1.5.1 with this site.cfg > > [DEFAULT] > library_dirs = > /usr/local/gcc/4.5.2/gcc/lib64:/usr/local/gcc/4.5.2/gcc/lib:/usr/local/gcc/4.5.2/gcc/lib32:/usr/local/gcc/4.5.2/gotoblas2/1.13/lib:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/lib:/usr/local/gcc/4.5.2/fftw/3.2.2/lib > include_dirs = > /usr/local/gcc/4.5.2/gcc/include:/usr/local/gcc/4.5.2/gotoblas2/1.13/include/goto:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/include:/usr/local/gcc/4.5.2/fftw/3.2.2/include > search_static_first = 1 > [blas_opt] > libraries = goto2 > language = fortran > [lapack_opt] > libraries = goto2 > language = fortran > [amd] > amd_libs = amd > [umfpack] > umfpack_libs = umfpack > [fftw] > libraries = fftw3 > > (I also tried without "_opt" and "language = fortran"); I used goto2 for > lapack because I read lapack should be included in libgoto (anyway things do > not change using "lapack"). I am quite sure the system is not using my > goto-lapack stuff since every time I buid i get: > > building extension "numpy.numarray._capi" sources > building extension "numpy.fft.fftpack_lite" sources > building extension "numpy.linalg.lapack_lite" sources > creating build/src.linux-x86_64-2.7/numpy/linalg > *### Warning: Using unoptimized lapack ###* > * adding 'numpy/linalg/lapack_litemodule.c' to sources.* > adding 'numpy/linalg/python_xerbla.c' to sources. > adding 'numpy/linalg/zlapack_lite.c' to sources. > adding 'numpy/linalg/dlapack_lite.c' to sources. > adding 'numpy/linalg/blas_lite.c' to sources. > adding 'numpy/linalg/dlamch.c' to sources. > adding 'numpy/linalg/f2c_lite.c' to sources. > building extension "numpy.random.mtrand" sources > creating build/src.linux-x86_64-2.7/numpy/random > C compiler: /usr/local/gcc/4.5.2//gcc/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall > -Wstrict-prototypes -O1 -pthread -fPIC -march=native -mtune=native > -I/usr/local/gcc/4.5.2//gcc/include > -I/usr/local/gcc/4.5.2//suiteSparse/3.6.0/include > -I/usr/local/gcc/4.5.2//fftw/3.2.2/include -fPIC > > during numpy installation (which ends successfully). Moreover I cannot see > any -lgoto2 as I would have expected. > Incidentally, I cannot see -lamd, -lumfpack, -lfftw3 (or any reference > to amd, umfpack, fftw3) neither, although there seems to be something to > handle them in system_info.py. The failure is so complete that I must have > done some big mistake but I can't correct my site.cfg even after searching > the internet. This seems to be one of the major discussion about this topic > so I am asking here for some help, please. > Is the problem related with site.cfg or with gotoblas2 installation? Is it > true that gotoblas2 hosts a full lapack inside? > > thank you very much! > > giuseppe > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 22 22:38:04 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 22 Mar 2011 19:38:04 -0700 Subject: [Numpy-discussion] fast robus implementation of float() In-Reply-To: References: Message-ID: <4D895D0C.3070500@noaa.gov> On 3/22/11 1:54 PM, Christian K. wrote: > I wonder if someone has a good solution for a fast conversion of gridded > ascii data to ndarray. the fastest out of the box way is with np.fromfile(input_file, sep=" ", dtype=np.float) It will only read multiple lines if the separater is whitespace, but it's pretty fast if it does. > It should manage ',' as decimal point (on demand) I think that will work ig the locale is set right, though I don't know how to do that "on demand" > and special windows numbers as 1.#INF. I can't remember if it does that -- give it a try. It does use ascii-to-float function written for numpy to handle things like that. > Of course, this is easy to wrap > in a small function but I expect it to be slow when the input size is in > the Mb range. Never expect -- do a simple solution, then see if it's too slow for youre needs. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Mar 23 00:34:13 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Mar 2011 22:34:13 -0600 Subject: [Numpy-discussion] git tag question/warning In-Reply-To: References: Message-ID: On Tue, Mar 22, 2011 at 11:29 AM, Ralf Gommers wrote: > Hi, > > After tagging 1.6.0b1 and pushing it I found some issues that only showed > up with released=True in setup.py. According to the docs at > http://www.kernel.org/pub/software/scm/git/docs/git-tag.html there is a > sane (just make a new tag) and an insane (delete tag, reuse same tag) way to > deal with this. Since the tagged version will actually not compile due to a > version string issue in setup.py I want to go the insane way. Apologies. > Please correct me if there's a better option. If you pulled from the github > repo in the last few hours, please check if you have a new local tag and > delete it. > > Changes to go in https://github.com/rgommers/numpy/tree/fixes-1.6.0b1 > > Why not just make a new tag? Although I suspect this question is a bit late coming... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 23 03:13:59 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 08:13:59 +0100 Subject: [Numpy-discussion] git tag question/warning In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 5:34 AM, Charles R Harris wrote: > > > On Tue, Mar 22, 2011 at 11:29 AM, Ralf Gommers > wrote: >> >> Hi, >> >> After tagging 1.6.0b1 and pushing it I found some issues that only showed >> up with released=True in setup.py. According to the docs at >> http://www.kernel.org/pub/software/scm/git/docs/git-tag.html there is a sane >> (just make a new tag) and an insane (delete tag, reuse same tag) way to deal >> with this. Since the tagged version will actually not compile due to a >> version string issue in setup.py I want to go the insane way. Apologies. >> Please correct me if there's a better option. If you pulled from the github >> repo in the last few hours, please check if you have a new local tag and >> delete it. >> >> Changes to go in https://github.com/rgommers/numpy/tree/fixes-1.6.0b1 >> > > Why not just make a new tag? Although I suspect this question is a bit late > coming... No, still possible. No fundamental reason, it just looks a bit stupid to start out with beta 2. But guess it looks like that either way now. Ralf From m.c.dixon at leeds.ac.uk Wed Mar 23 05:11:54 2011 From: m.c.dixon at leeds.ac.uk (Mark Dixon) Date: Wed, 23 Mar 2011 09:11:54 +0000 (GMT) Subject: [Numpy-discussion] Numpy: controlling ATLAS multithreading Message-ID: Hi, I've built Numpy 1.5.1 against a copy of ATLAS 3.8.3 and LAPACK 3.3.0, it's passing all tests and is looking good. Great. I note that Numpy preferentially links against the multi-threaded ATLAS libraries; however, we sometimes want to constrain a process to a single thread, and sometimes not. I see that ATLAS doesn't have the capability of setting the maximum number of threads used with, say, an environment variable. So, here comes the long-shot question: Does Numpy have the capability to choose between the single-threaded and multi-threaded ATLAS libraries at runtime? Or do I need to make this decision at build time? Thanks, Mark -- ----------------------------------------------------------------- Mark Dixon Email : m.c.dixon at leeds.ac.uk HPC/Grid Systems Support Tel (int): 35429 Information Systems Services Tel (ext): +44(0)113 343 5429 University of Leeds, LS2 9JT, UK ----------------------------------------------------------------- From sturla at molden.no Wed Mar 23 06:42:06 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Mar 2011 11:42:06 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: References: <4D7F81CA.30808@molden.no> <4D80AC48.3020809@molden.no> <4D80ADC6.30507@molden.no> Message-ID: <4D89CE7E.9030809@molden.no> Den 16.03.2011 14:46, skrev Neal Becker: > Also: > * can it adopt external memory? Yes. Using CRAY pointers with libc malloc/free is e.g. a common way to get dynamic memory in Fortran 77. > * can it interwork with numpy? (kinda required for this audience) > Yes, that is why NumPy has f2py :-) - Explicit shape and assumed size arrays can be passed as a pointer to the first element. We can use numpy.f2py, ctypes or Cython for this purpose. I usually put all my Fortran 95 code in modules, and expose them to Python via driver subroutines. - Fortran 2003 has bindings for ISO C that makes this portable. fwrap automates this process, so we don't have to write all the boilerplate code by hand. It is still immature though, e.g. no sujpport for modules last time I checked. Sturla From sturla at molden.no Wed Mar 23 06:14:34 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Mar 2011 11:14:34 +0100 Subject: [Numpy-discussion] Fortran was dead ... [was Re: rewriting NumPy code in C or C++ or similar] In-Reply-To: <201103161210.25058.lists_ravi@lavabit.com> References: <201103161210.25058.lists_ravi@lavabit.com> Message-ID: <4D89C80A.4060901@molden.no> I've been busy lately, so I haven't had time to answer. The idea is of course to use Fortran with Python, not to build a framework in Fortran. I also find C useful to interface with the operating system, and Cython to write C extensions for Python. Also, if the computationally demaning parts are taken care of by libraries (LAPACK et al.), it does not matter from which language we call the performance libraries. Sturla Den 16.03.2011 20:10, skrev Ravi: > Comparing Fortran with C++ is like comparing Matlab with Python. Fortran is > very good at what it does but it does not serve the same audience as C++. The > typical comparisons work like this: > > 1. C++ does not have a standard array type but Fortran does. Python does not > have a standard numerical array type either; numpy is nice, but, it is an > external package. The point is that you pick the external array type that > works well for you in Python/C++ but you use the built-in array type in > Matlab/Fortran. Whether the flexibility is a problem or a god-send is entirely > up to you. For modeling high-performance algorithms in silicon, usually in > fixed-point, Fortran/Matlab are worthless, but C++ (and to some extent, > python) works very well. > > 2. C++ (resp. python+numpy) does not perform as well as Fortran (resp. > Matlab). The issue with C++ is that aliasing is allowed, but virtually all > compilers will allow you to use "restrict" to get almost the same benefits as > Fortran. The analogous python+numpy issue is that it is very hard to create a > good JIT (unladen-swallow was, at best, a partial success) for python while > Matlab has a very nice JIT (and an even better integration with the java > virtual machine). > > 3. "Template metaprogramming makes my head hurt." (Equivalently, "python > generators and decorators make my head hurt".) Neither template > metaprogramming nor python generators/decorators are *required* for most > scientific programming tasks, especially when you use libraries designed to > shield you from such details. However, knowledge and use of those techniques > makes one much more productive in those corner cases where they are useful. > > 4. "I do not need all the extra stuff that C++ (resp. python) provides > compared to Fortran (resp. Matlab)". C++/python are industrial strength > programming languages which serve a large audience, where each interested > niche group can create efficient libraries unhindered by the orientation of > the language itself. Fortran/Matlab are numerical languages with extra general > purpose stuff tacked on. Building large scale Fortran/Matlab programs are > possible (see the Joint Strike Fighter models in Matlab or any large-scale > Fortran application), but the lack of a real programming language is a pain > whose intensity increases exponentially with the size of the codebase. > > Another way it is usually phrased: "I will use Fortran's (resp. Matlab's) > integration with Python (resp. Java) when I need a real programming language." > > 5. "OOP/generic programming are useless for most scientific programming." This > is usually just elitism. (While I personally do not believe that OO is worth > one percent of the hype, I do believe that generic programming, as practiced > in Python& C++, are pretty much our primary hope for reusable software.) When > not elitism, it is a measure of the immaturity of computational science (some > would say scientists), where large repositories of reusable code are few and > far between. OO, generic programming, and functional programming are the only > techniques of which I am aware for building large scale programs with > manageable complexity. > > I would take any Fortran hype with large grains of salt. > > Regards, > Ravi > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From sturla at molden.no Wed Mar 23 07:25:53 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Mar 2011 12:25:53 +0100 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> <201008180913.12594.eg@fft.be> Message-ID: <4D89D8C1.9010209@molden.no> Den 22.03.2011 23:18, skrev Paul Anton Letnes: > I'm no expert, but I just pulled off the scipy+numpy+GotoBLAS2 > installation. From what I gather, the Makefile for libgoto2 downloads > and compiles the generic lapack from netlib. It also wraps lapack into > libgoto2.so/.a . I believe the idea is as long > as the BLAS implementation is fast(TM), the lapack performance will be > good. GotoBLAS replaces a few LAPACK routines where BLAS optimization is not sufficient. Last time I built GotoBLAS2 it came with Netlib LAPACK sources in the tarball. What really matters for LAPACK performance is not even BLAS, but the general matrix multiply routines *GEMM in BLAS. That is why AMD has made a GPU version of ACML where matrix multiplication in BLAS can be deferred to the ATI GPU. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Wed Mar 23 08:23:16 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Wed, 23 Mar 2011 14:23:16 +0200 Subject: [Numpy-discussion] Appending a numpy array to binary file In-Reply-To: References: <201103221137.18997.alessandro.sanginario@polito.it> Message-ID: On 22 March 2011 17:22, Robert Kern wrote: > On Tue, Mar 22, 2011 at 05:37, Alessandro > wrote: >> Hi everybody, >> >> I'm trying to append, inside a loop, into a binary file some arrays using >> >> numpy.save(file, arr) but wvhen I open it there is only the last array I >> saved. >> >> If I use savetxt it works perfectly. Do you know how can I append data to a >> binary file inside a loop? >> >> Example: >> >> x = arange(10) >> >> fp = open("TempFile","ab") >> >> for i in range(3): >> >> save(fp,x) >> >> fp.close() >> >> Data saved into the file is: [0,1,2,3,4,5,6,7,8,9] but I would like: >> [0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9 0,1,2,3,4,5,6,7,8,9] > > The data is actually saved three times... > You can read all > three arrays by doing repeated np.load() calls on the same file > object: > > fp = open('TempFile', 'rb') > for i in range(3): > ? ?print np.load(fp) You can also use numpy.savez if you want to save several arrays and read them back with a single call to numpy.load. Cheers, Scott From amenity at enthought.com Wed Mar 23 08:56:17 2011 From: amenity at enthought.com (Amenity Applewhite) Date: Wed, 23 Mar 2011 07:56:17 -0500 Subject: [Numpy-discussion] SciPy 2011 Call for Papers Message-ID: Hello, SciPy 2011 , the 10th Python in Science conference, will be held July 11 - 16, 2011, in Austin, TX. At this conference, novel applications and breakthroughs made in the pursuit of science using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is preceded by two days of tutorials, during which community experts provide training on several scientific Python packages. *We'd like to invite you to consider presenting at SciPy 2011.* The list of topics that are appropriate for the conference includes (but is not limited to): * new Python libraries for science and engineering; * applications of Python to the solution of scientific or computational problems; * high performance, parallel and GPU computing with Python; * use of Python in science education. *Specialized Tracks* This year we also have two specialized tracks. They will be run concurrent to the main conference. *Python in Data Science Chair: Peter Wang, Streamitive, Inc.* This track focuses on the advantages and challenges of applying Python in the emerging field of "data science". This includes a breadth of technologies, from wrangling realtime data streams from the social web, to machine learning and semantic analysis, to workflow and repository management for large datasets. *Python and Core Technologies Chair: Anthony Scopatz, Enthought, Inc.* In an effort to broaden the scope of SciPy and to engage the larger community of software developers, we are pleased to introduce the _Python & Core Technologies_ track. Talks will cover subjects that are not directly related to science and engineering, yet nonetheless affect scientific computing. Proposals on the Python language, visualization toolkits, web frameworks, education, and other topics are appropriate for this session. *Talk/Paper Submission* We invite you to take part by submitting a talk abstract on the conference website at: http://conference.scipy.org/scipy2011/papers.php Papers are included in the peer-reviewed conference proceedings, to be published online. *Important dates for authors:* Friday, April 15: Tutorial proposals due (remember: stipends will be provided for Tutorial instructors) http://conference.scipy.org/scipy2011/tutorials.php Sunday, April 24: Paper abstracts due Sunday, May 8: Student sponsorship request due http://conference.scipy.org/scipy2011/student.php Tuesday, May 10: Accepted talks announced Monday, May 16: Student sponsorships announced Monday, May 23: Early Registration ends Sunday, June 20: Papers due Monday-Tuesday, July 11 - 12: Tutorials Wednesday-Thursday, July 13 - July 14: Conference Friday-Saturday, July 15 - July 16: Sprints The SciPy 2011 Team @SciPy2011 http://twitter.com/SciPy2011 _________________________ Amenity Applewhite Enthought, Inc. Scientific Computing Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Mar 23 09:26:47 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 14:26:47 +0100 Subject: [Numpy-discussion] Numpy: controlling ATLAS multithreading In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 10:11 AM, Mark Dixon wrote: > Hi, > > I've built Numpy 1.5.1 against a copy of ATLAS 3.8.3 and LAPACK 3.3.0, > it's passing all tests and is looking good. Great. > > I note that Numpy preferentially links against the multi-threaded ATLAS > libraries; however, we sometimes want to constrain a process to a single > thread, and sometimes not. I see that ATLAS doesn't have the capability of > setting the maximum number of threads used with, say, an environment > variable. > > So, here comes the long-shot question: > > Does Numpy have the capability to choose between the single-threaded and > multi-threaded ATLAS libraries at runtime? Or do I need to make this > decision at build time? You need to decide at build time. Ralf From m.c.dixon at leeds.ac.uk Wed Mar 23 09:32:12 2011 From: m.c.dixon at leeds.ac.uk (Mark Dixon) Date: Wed, 23 Mar 2011 13:32:12 +0000 (GMT) Subject: [Numpy-discussion] Numpy: controlling ATLAS multithreading In-Reply-To: References: Message-ID: On Wed, 23 Mar 2011, Ralf Gommers wrote: ... >> Does Numpy have the capability to choose between the single-threaded and >> multi-threaded ATLAS libraries at runtime? Or do I need to make this >> decision at build time? > > You need to decide at build time. ... Thanks Ralf Mark -- ----------------------------------------------------------------- Mark Dixon Email : m.c.dixon at leeds.ac.uk HPC/Grid Systems Support Tel (int): 35429 Information Systems Services Tel (ext): +44(0)113 343 5429 University of Leeds, LS2 9JT, UK ----------------------------------------------------------------- From sturla at molden.no Wed Mar 23 10:28:58 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Mar 2011 15:28:58 +0100 Subject: [Numpy-discussion] Is numpy/scipy linux apt or PYPI installation linked with ACML? In-Reply-To: References: Message-ID: <4D8A03AA.7010909@molden.no> ACML has a C interface for BLAS and LAPACK, but it is different from cblas. For portability, stick with the Fortran 77 interface, particularly for LAPACK where no C interface is specified. One can probably get cblas from Netlib and link with libacml. cblas is just a wrapper for the Fortran routines. Sturla Den 23.01.2011 17:49, skrev Matthieu Brucher: > I think the main issue is that ACML didn't have an official CBLAS > interface, so you have to check if they provide one now. If thy do, it > should be almost easy to link against it. > > Matthieu > > 2011/1/23 David Cournapeau > > > 2011/1/23 Dmitrey >: > > Hi all, > > I have AMD processor and I would like to get to know what's the > easiest way > > to install numpy/scipy linked with ACML. > > Is it possible to link linux apt or PYPI installation linked > with ACML? > > Answer for the same question about MKL also would be useful, > however, AFAIK > > it has commercial license and thus can't be handled in the ways. > > For the MKL, the easiest solution is to get EPD, or to build > numpy/scipy by yourself, although the later is not that easy. For > ACML, I don't know how difficult it is, but I would be surprised if it > worked out of the box. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From tartley at tartley.com Wed Mar 23 10:29:18 2011 From: tartley at tartley.com (Jonathan Hartley) Date: Wed, 23 Mar 2011 14:29:18 +0000 Subject: [Numpy-discussion] newbie question about boolean testing of array equality result Message-ID: <4D8A03BE.9010506@tartley.com> Hey people, I'm writing an application in which we evaluate user-supplied Python expressions. Sometimes, we want to do an equality test on the result of the evaluation, eg: result = eval(...) if result == float('nan'): ... If the result is a numpy array, then this raises a ValueError, since the array equality operator returns a new numpy array, and the coercion of this to a boolean for the if predicate then explicitly raises. Presumably this is well-known? For us, it is undesirable. Am I right to understand that any code which might ever encounter a numpy array therefore can never use an unguarded '/if x == y:/' construction? Is my workaround really to replace every instance of this with '/if not isinstance(x, numpy.array) and x==y:/' ? This pains me, because otherwise this code would have no dependency on numpy. (I can't just prepend 'isinstance(x, float)' because, unlike the example above, we don't always know the type of the equality RHS until runtime.) I can see that there's a pleasing symmetry to have all the array arithmetic operators and comparisons operate in an element-wise manner, but I think it's more important for __eq__ to follow it's usual semantics of returning a boolean. I'd way prefer it if the element-wise equality array generation was exposed as a different method. Have I misunderstood the situation greatly? Thanks for any pointers or suggestions. Jonathan -- Jonathan Hartley tartley at tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 23 10:59:26 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Mar 2011 09:59:26 -0500 Subject: [Numpy-discussion] newbie question about boolean testing of array equality result In-Reply-To: <4D8A03BE.9010506@tartley.com> References: <4D8A03BE.9010506@tartley.com> Message-ID: On Wed, Mar 23, 2011 at 09:29, Jonathan Hartley wrote: > Hey people, > > I'm writing an application in which we evaluate user-supplied Python > expressions. > > Sometimes, we want to do an equality test on the result of the evaluation, > eg: > > ??? result = eval(...) > ??? if result == float('nan'): > ???????? ... Please note that this particular expression will *never* work, even with float objects. float('nan') != float('nan'). It's a quick of floating point semantics. > If the result is a numpy array, then this raises a ValueError, since the > array equality operator returns a new numpy array, and the coercion of this > to a boolean for the if predicate then explicitly raises. Presumably this is > well-known? Yes. > For us, it is undesirable. > > Am I right to understand that any code which might ever encounter a numpy > array therefore can never use an unguarded? 'if x == y:' construction? Is my > workaround really to replace every instance of this with 'if not > isinstance(x, numpy.array) and x==y:' ? This pains me, because otherwise > this code would have no dependency on numpy. (I can't just prepend > 'isinstance(x, float)' because, unlike the example above, we don't always > know the type of the equality RHS until runtime.) def equals(x, y): z = x == y if not isinstance(z, bool): # Or maybe you check for the existence of .all() or .any() depending on which semantics you would like in the presence of numpy arrays. z = False return z > I can see that there's a pleasing symmetry to have all the array arithmetic > operators and comparisons operate in an element-wise manner, but I think > it's more important for __eq__ to follow it's usual semantics of returning a > boolean. I'd way prefer it if the element-wise equality array generation was > exposed as a different method. I'm afraid that it is far too late to make such a change. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Wed Mar 23 12:07:24 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 17:07:24 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 Message-ID: Hi, I am pleased to announce the availability of the first beta of NumPy 1.6.0. Due to the extensive changes in the Numpy core for this release, the beta testing phase will last at least one month. Please test this beta and report any problems on the Numpy mailing list. Sources and binaries can be found at: http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ For (preliminary) release notes see below. Enjoy, Ralf ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py interprets Fortran expression ``size(array, dim)`` as ``shape(array, dim-1)`` which makes it possible to automatically wrap Fortran routines that use two argument ``size`` function in dimension specifications. Before users were forced to apply this mapping manually. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= Changes and improvements in the numpy core ------------------------------------------ ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. Checksums ========= 89f52ae0f0ea84cfcb457298190bee14 release/installers/numpy-1.6.0b1-py2.7-python.org.dmg 8dee06b362540b2c8c033399951e5386 release/installers/numpy-1.6.0b1-win32-superpack-python2.5.exe c1b11bf48037ac8fe025bfd297969840 release/installers/numpy-1.6.0b1-win32-superpack-python2.6.exe 2b005cb359d8123bd5ee3063d9eae3ac release/installers/numpy-1.6.0b1-win32-superpack-python2.7.exe 45627e8f63fe34011817df66722c39a5 release/installers/numpy-1.6.0b1-win32-superpack-python3.1.exe 1d8b214752b19b51ee747a6436ca1d38 release/installers/numpy-1.6.0b1-win32-superpack-python3.2.exe aeab5881974aac595b87a848c0c6344a release/installers/numpy-1.6.0b1.tar.gz 3ffc6e308f9e0614531fa3babcb75544 release/installers/numpy-1.6.0b1.zip From dplepage at gmail.com Wed Mar 23 12:15:08 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Wed, 23 Mar 2011 12:15:08 -0400 Subject: [Numpy-discussion] newbie question about boolean testing of array equality result In-Reply-To: <4D8A03BE.9010506@tartley.com> References: <4D8A03BE.9010506@tartley.com> Message-ID: On Wed, Mar 23, 2011 at 10:29 AM, Jonathan Hartley wrote: > Hey people, > > I'm writing an application in which we evaluate user-supplied Python > expressions. > > Sometimes, we want to do an equality test on the result of the evaluation, > eg: > > ??? result = eval(...) > ??? if result == float('nan'): > ???????? ... > > If the result is a numpy array, then this raises a ValueError, since the > array equality operator returns a new numpy array, and the coercion of this > to a boolean for the if predicate then explicitly raises. Presumably this is > well-known? For us, it is undesirable. > > Am I right to understand that any code which might ever encounter a numpy > array therefore can never use an unguarded? 'if x == y:' construction? Is my > workaround really to replace every instance of this with 'if not > isinstance(x, numpy.array) and x==y:' ? This pains me, because otherwise > this code would have no dependency on numpy. (I can't just prepend > 'isinstance(x, float)' because, unlike the example above, we don't always > know the type of the equality RHS until runtime.) With floating point numbers you usually don't want to do 'if x == y' anyway - it's pretty common for two numbers that should be mathematically the same to differ due to lack of precision, e.g. 0.3 + 0.2 + 0.1 != 0.3 + (0.2 + 0.1) because the first is 0.59999999999999998 and the second is 0.60000000000000009 Typically you want to compare them by checking that their difference is below some threshold, e.g. abs(x-y) < 1e-10. The threshold should depend on the precision: for 32-bit floats, 1e-10 isn't enough (e.g. (np.float32(.3) - np.float32(.2) - np.float32(.1)) < 1e-10 fails). The best threshold also depends on the magnitude of the values, since precision errors are greater between large numbers than between numbers close to zero. The easiest way to do this is to just go ahead and depend on numpy, because it provides a function allclose that does what you want - np.allclose(x,y) is true if x and y are the same shape and their elementwise differences are below a small threshold. Note that there are additional arguments to allclose if you want to control the threshold, but for simple cases allclose(x,y) is probably fine. Also be aware that it does apply numpy broadcasting, so that e.g. np.allclose(x,1) checks if all elements of x are close to 1. You can also use numpy for nan testing: np.isnan(x).any() works if x is a numpy array or if x is a python scalar, list, or tuple (or anything else that can be passed into np.asarray). Other useful checks include np.isneginf, np.isposinf, and np.isinf for checking for infinities, and also np.isfinite which is equivalent to ~(np.isnan | np.isinf). -- Dan Lepage From bsouthey at gmail.com Wed Mar 23 12:29:58 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 23 Mar 2011 11:29:58 -0500 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: <4D8A2006.3090706@gmail.com> On 03/23/2011 11:07 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first beta of NumPy > 1.6.0. Due to the extensive changes in the Numpy core for this > release, the beta testing phase will last at least one month. Please > test this beta and report any problems on the Numpy mailing list. > > Sources and binaries can be found at: > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > [snip] Python 3.1 failed to build it due to import error of _inspect module and Python2.4 failed 'test_iterator.test_iter_broadcasting_errors'. Yet all tests passed on Python versions 2.5, 2.6 and 2.7. Bruce PS I do want to go through the bugs I indicated but I have not found sufficient time. Python3.1 error: Traceback (most recent call last): File "setup.py", line 201, in setup_package() File "setup.py", line 178, in setup_package from numpy.distutils.core import setup File "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/__init__.py", line 22, in import numpy.distutils.ccompiler File "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/ccompiler.py", line 15, in from numpy.distutils.exec_command import exec_command File "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/exec_command.py", line 58, in from numpy.compat import open_latin1 File "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/compat/__init__.py", line 11, in import _inspect ImportError: No module named _inspect Python 2.4 failed test: $ python2.4 -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.6.0b1 NumPy is installed in /usr/local/lib/python2.4/site-packages/numpy Python version 2.4.6 (#1, Sep 13 2010, 15:54:12) [GCC 4.4.4 20100630 (Red Hat 4.4.4-10)] nose version 0.11.2 .............................................................................................................................................................................F...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................K......................K....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS............ ====================================================================== FAIL: test_iterator.test_iter_broadcasting_errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/nose/case.py", line 186, in runTest self.test(*self.arg) File "/usr/local/lib/python2.4/site-packages/numpy/core/tests/test_iterator.py", line 639, in test_iter_broadcasting_errors 'Message "%s" doesn\'t contain operand shape (2,3)' % msg) File "/usr/local/lib/python2.4/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Message "non-broadcastable output operand with shape (%zd,%zd) doesn't match the broadcast shape (%zd,%zd,%zd)" doesn't contain operand shape (2,3) ---------------------------------------------------------------------- Ran 3382 tests in 19.454s FAILED (KNOWNFAIL=3, SKIP=3, failures=1) From ralf.gommers at googlemail.com Wed Mar 23 12:38:23 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 17:38:23 +0100 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <4D8A2006.3090706@gmail.com> References: <4D8A2006.3090706@gmail.com> Message-ID: On Wed, Mar 23, 2011 at 5:29 PM, Bruce Southey wrote: > On 03/23/2011 11:07 AM, Ralf Gommers wrote: >> Hi, >> >> I am pleased to announce the availability of the first beta of NumPy >> 1.6.0. Due to the extensive changes in the Numpy core for this >> release, the beta testing phase will last at least one month. Please >> test this beta and report any problems on the Numpy mailing list. >> >> Sources and binaries can be found at: >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >> For (preliminary) release notes see below. >> >> Enjoy, >> Ralf >> >> >> > [snip] > Python 3.1 failed to build it due to import error of _inspect module and > Python2.4 failed 'test_iterator.test_iter_broadcasting_errors'. Yet all > tests passed on Python versions 2.5, 2.6 and 2.7. > > Bruce > PS I do want to go through the bugs I indicated but I have not found > sufficient time. > > Python3.1 error: > > Traceback (most recent call last): > ? File "setup.py", line 201, in > ? ? setup_package() > ? File "setup.py", line 178, in setup_package > ? ? from numpy.distutils.core import setup > ? File > "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/__init__.py", > line 22, in > ? ? import numpy.distutils.ccompiler > ? File > "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/ccompiler.py", > line 15, in > ? ? from numpy.distutils.exec_command import exec_command > ? File > "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/exec_command.py", > line 58, in > ? ? from numpy.compat import open_latin1 > ? File > "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/compat/__init__.py", > line 11, in > ? ? import _inspect > ImportError: No module named _inspect This is strange. I checked that numpy/compat/_inspect.py is present in the tarball. What build command did you use, and can you give the complete output of that command? Ralf From bsouthey at gmail.com Wed Mar 23 13:00:32 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 23 Mar 2011 12:00:32 -0500 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <4D8A2006.3090706@gmail.com> Message-ID: <4D8A2730.4080908@gmail.com> On 03/23/2011 11:38 AM, Ralf Gommers wrote: > On Wed, Mar 23, 2011 at 5:29 PM, Bruce Southey wrote: >> On 03/23/2011 11:07 AM, Ralf Gommers wrote: >>> Hi, >>> >>> I am pleased to announce the availability of the first beta of NumPy >>> 1.6.0. Due to the extensive changes in the Numpy core for this >>> release, the beta testing phase will last at least one month. Please >>> test this beta and report any problems on the Numpy mailing list. >>> >>> Sources and binaries can be found at: >>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >>> For (preliminary) release notes see below. >>> >>> Enjoy, >>> Ralf >>> >>> >>> >> [snip] >> Python 3.1 failed to build it due to import error of _inspect module and >> Python2.4 failed 'test_iterator.test_iter_broadcasting_errors'. Yet all >> tests passed on Python versions 2.5, 2.6 and 2.7. >> >> Bruce >> PS I do want to go through the bugs I indicated but I have not found >> sufficient time. >> >> Python3.1 error: >> >> Traceback (most recent call last): >> File "setup.py", line 201, in >> setup_package() >> File "setup.py", line 178, in setup_package >> from numpy.distutils.core import setup >> File >> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/__init__.py", >> line 22, in >> import numpy.distutils.ccompiler >> File >> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/ccompiler.py", >> line 15, in >> from numpy.distutils.exec_command import exec_command >> File >> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/exec_command.py", >> line 58, in >> from numpy.compat import open_latin1 >> File >> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/compat/__init__.py", >> line 11, in >> import _inspect >> ImportError: No module named _inspect > This is strange. I checked that numpy/compat/_inspect.py is present in > the tarball. What build command did you use, and can you give the > complete output of that command? > > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Yes, I know that it is there and not sure what is missing! The typical way: $python3.1 setup.py build Also I know that it builds with Python3.1 and Python 3.2 with the latest git pull! But I have update nose before I can test it. Bruce -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: e.log URL: From ralf.gommers at googlemail.com Wed Mar 23 13:25:11 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 18:25:11 +0100 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <4D8A2730.4080908@gmail.com> References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> Message-ID: On Wed, Mar 23, 2011 at 6:00 PM, Bruce Southey wrote: > On 03/23/2011 11:38 AM, Ralf Gommers wrote: >> >> On Wed, Mar 23, 2011 at 5:29 PM, Bruce Southey ?wrote: >>> >>> On 03/23/2011 11:07 AM, Ralf Gommers wrote: >>>> >>>> Hi, >>>> >>>> I am pleased to announce the availability of the first beta of NumPy >>>> 1.6.0. Due to the extensive changes in the Numpy core for this >>>> release, the beta testing phase will last at least one month. Please >>>> test this beta and report any problems on the Numpy mailing list. >>>> >>>> Sources and binaries can be found at: >>>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >>>> For (preliminary) release notes see below. >>>> >>>> Enjoy, >>>> Ralf >>>> >>>> >>>> >>> [snip] >>> Python 3.1 failed to build it due to import error of _inspect module and >>> Python2.4 failed 'test_iterator.test_iter_broadcasting_errors'. Yet all >>> tests passed on Python versions 2.5, 2.6 and 2.7. >>> >>> Bruce >>> PS I do want to go through the bugs I indicated but I have not found >>> sufficient time. >>> >>> Python3.1 error: >>> >>> Traceback (most recent call last): >>> ? File "setup.py", line 201, in >>> ? ? setup_package() >>> ? File "setup.py", line 178, in setup_package >>> ? ? from numpy.distutils.core import setup >>> ? File >>> >>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/__init__.py", >>> line 22, in >>> ? ? import numpy.distutils.ccompiler >>> ? File >>> >>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/ccompiler.py", >>> line 15, in >>> ? ? from numpy.distutils.exec_command import exec_command >>> ? File >>> >>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/exec_command.py", >>> line 58, in >>> ? ? from numpy.compat import open_latin1 >>> ? File >>> >>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/compat/__init__.py", >>> line 11, in >>> ? ? import _inspect >>> ImportError: No module named _inspect >> >> This is strange. I checked that numpy/compat/_inspect.py is present in >> the tarball. What build command did you use, and can you give the >> complete output of that command? >> >> Ralf >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Yes, I know that it is there and not sure what is missing! > The typical way: > $python3.1 setup.py build It runs 2to3, but then for some reason decides not to use the output of that on your system. Is that failing each time? If so, can you look just above where it is failing, and see what's going on before/after switching the path. Relevant code is: old_path = os.getcwd() os.chdir(src_path) sys.path.insert(0, src_path) Just inserting some print statements to look at os.getcwd() before and after and at sys.path may tell you what's going on. Ralf > > Also I know that it builds with Python3.1 and Python 3.2 with the latest git > pull! But I have update nose before I can test it. From Chris.Barker at noaa.gov Mon Mar 7 17:18:11 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 07 Mar 2011 22:18:11 -0000 Subject: [Numpy-discussion] Scientific Software developer wanted in Seattle. Message-ID: <4D7559A1.5070704@noaa.gov> Scientific Software Developer NOAA Emergency Response Division Help us develop our next-generation oil spill transport model. Background: The Emergency Response Division (ERD) of NOAA's Office of Response and Restoration (OR&R) provides scientific expertise to support the response to oil and chemical spills in the coastal environment. We played a major role in the recent Deepwater Horizon oil spill in the Gulf of Mexico. In order to fulfill our mission, we develop many of the software tools and models required to support a response to hazardous material spills. In the wake of the Deepwater horizon incident, we are embarking on a program to develop our next-generation oil spill transport model, taking into account lessons learned from years of response and this major incident. General Characteristics: The incumbent of this position will provide software development services to support the mission of the Emergency Response Division of NOAA's Office of Response and Restoration. As part of his/her efforts, independent evaluation and application of development techniques, algorithms, software architecture, and programming patterns will be required. The incumbent will work with the staff of ERD to provide analysis on user needs and software, GUI, and library design. He/she will be expect to work primarily on site at NOAA's facility in Seattle. Knowledge: The incumbent must be able to apply modern concepts of software engineering and design to the development of computational code, desktop applications, web applications, and libraries. The incumbent will need to be able to design, write, refactor, and implement code for a complex desktop and/or web application and computational library. The incumbent will work with a multi-disciplinary team including scientists, users, and other developers, utilizing software development practices such as usability design, version control, bug and issue tracking, and unit testing. Good communication skills and the knowledge of working as part of a team are required. Direction received: The incumbent will participate on various research and development teams. While endpoints will be identified through Division management and some direct supervision will be provided, the incumbent will be responsible for progressively being able to take input from team meetings and design objectives and propose strategies for reaching endpoints. Typical duties and responsibilities: The incumbent will work with the oil and chemical spill modeling team to improve and develop new tools and models used in fate and transport forecasting. Different components of the project will be written in C++, Python, and Javascript. Education requirement, minimum: Bachelor's degree in a technical discipline. Experience requirement, minimum: One to five years experience in development of complex software systems in one or more full-featured programming languages (C, C++, Java, Python, Ruby, Fortran, etc.) The team requires experience in the following languages/disciplines. Each incumbent will need experience in some subset: * Computational/Scientific programming * Numerical Analysis/Methods * Parallel processing * Desktop GUI * Web services * Web clients: HTML/CSS/Javascript * Python * wxPython * OpenGL * C/C++ * Python--C/C++ integration * Software development team leadership While the incumbent will work on-site at NOAA, directly with the NOAA team, this is a contract position with General Dynamics Information Technology: http://www.gdit.com/default.aspx For more information and to apply, use the GDIT web site: https://secure.resumeware.net/gdns_rw/gdns_web/job_detail.cfm?key=59436&show_cart=0&referredId=20 if that long url doesn't work, try: http://www.resumeware.net/gdns_rw/gdns_web/job_search.cfm and search for job ID: 179178 NOTE: This is a potion being hired by GDIT to work with NOAA, so any questions about salary, benefits, etc, etc should go to GDIT. However, feel free to send me questions about our organization, working conditions, more detail about the nature of the projects etc. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tmp50 at ukr.net Wed Mar 23 13:53:10 2011 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 23 Mar 2011 19:53:10 +0200 Subject: [Numpy-discussion] numpy log2 has bug Message-ID: >>> from numpy import log2, __version__ >>> log2(2**63) Traceback (most recent call last): File "", line 1, in AttributeError: log2 >>> __version__ '2.0.0.dev-1fe8136' (doesn't work with 1.3.0 as well) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Wed Mar 23 14:02:31 2011 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 23 Mar 2011 20:02:31 +0200 Subject: [Numpy-discussion] bug with numpy 2 ** N Message-ID: >>> 2**64 18446744073709551616L >>> 2**array(64) -9223372036854775808 >>> 2**100 1267650600228229401496703205376L >>> 2**array(100) -9223372036854775808 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Mar 23 14:05:32 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 23 Mar 2011 19:05:32 +0100 Subject: [Numpy-discussion] bug with numpy 2 ** N In-Reply-To: References: Message-ID: Hi, I don't thnk this is a bug. You are playign with C integers, not Python integers, and the former are limited. It's a common "feature" in all processors (even DSPs). Matthieu 2011/3/23 Dmitrey > >>> 2**64 > 18446744073709551616L > >>> 2**array(64) > -9223372036854775808 > >>> 2**100 > 1267650600228229401496703205376L > >>> 2**array(100) > -9223372036854775808 > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Mar 23 14:51:49 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 23 Mar 2011 14:51:49 -0400 Subject: [Numpy-discussion] numpy log2 has bug In-Reply-To: References: Message-ID: 2011/3/23 Dmitrey : >>>> from numpy import log2, __version__ >>>> >>>> log2(2**63) > Traceback (most recent call > last): > ? File "", line 1, in > > AttributeError: log2 >>>> __version__ > '2.0.0.dev-1fe8136' > (doesn't work with 1.3.0 as well) >>> np.array([2**63]) array([9223372036854775808], dtype=object) >>> log2(2.**63) 62.999999999999993 >>> log2(2**63) Traceback (most recent call last): File "", line 1, in log2(2**63) AttributeError: log2 integer conversion problem Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From dplepage at gmail.com Wed Mar 23 15:05:21 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Wed, 23 Mar 2011 15:05:21 -0400 Subject: [Numpy-discussion] Make array uncopyable Message-ID: Hi all, Is there a way to mark an array as uncopyable? If you have very large data arrays some numpy functions will cause MemoryErrors because behind the scenes they e.g. transpose the arrays into Fortran order to feed them into Fortran libraries (numpy.linalg.svd is an example of this). It would be great if there were a way to mark an array as "too big to copy" so that an attempt to call transpose() or astype() would raise an exception immediately instead of clobbering the machine's memory first, but I don't know of any flag that does this. I suppose I could always subclass ndarray and redefine transpose(), astype(), etc., but it'd be nice if there were an easier way. Thanks, Dan Lepage From bsouthey at gmail.com Wed Mar 23 15:17:20 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 23 Mar 2011 14:17:20 -0500 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> Message-ID: <4D8A4740.50305@gmail.com> On 03/23/2011 12:25 PM, Ralf Gommers wrote: > On Wed, Mar 23, 2011 at 6:00 PM, Bruce Southey wrote: >> On 03/23/2011 11:38 AM, Ralf Gommers wrote: >>> On Wed, Mar 23, 2011 at 5:29 PM, Bruce Southey wrote: >>>> On 03/23/2011 11:07 AM, Ralf Gommers wrote: >>>>> Hi, >>>>> >>>>> I am pleased to announce the availability of the first beta of NumPy >>>>> 1.6.0. Due to the extensive changes in the Numpy core for this >>>>> release, the beta testing phase will last at least one month. Please >>>>> test this beta and report any problems on the Numpy mailing list. >>>>> >>>>> Sources and binaries can be found at: >>>>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >>>>> For (preliminary) release notes see below. >>>>> >>>>> Enjoy, >>>>> Ralf >>>>> >>>>> >>>>> >>>> [snip] >>>> Python 3.1 failed to build it due to import error of _inspect module and >>>> Python2.4 failed 'test_iterator.test_iter_broadcasting_errors'. Yet all >>>> tests passed on Python versions 2.5, 2.6 and 2.7. >>>> >>>> Bruce >>>> PS I do want to go through the bugs I indicated but I have not found >>>> sufficient time. >>>> >>>> Python3.1 error: >>>> >>>> Traceback (most recent call last): >>>> File "setup.py", line 201, in >>>> setup_package() >>>> File "setup.py", line 178, in setup_package >>>> from numpy.distutils.core import setup >>>> File >>>> >>>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/__init__.py", >>>> line 22, in >>>> import numpy.distutils.ccompiler >>>> File >>>> >>>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/ccompiler.py", >>>> line 15, in >>>> from numpy.distutils.exec_command import exec_command >>>> File >>>> >>>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/distutils/exec_command.py", >>>> line 58, in >>>> from numpy.compat import open_latin1 >>>> File >>>> >>>> "/home/bsouthey/python/numpystuff/numpy-1.6.0b1/numpy/compat/__init__.py", >>>> line 11, in >>>> import _inspect >>>> ImportError: No module named _inspect >>> This is strange. I checked that numpy/compat/_inspect.py is present in >>> the tarball. What build command did you use, and can you give the >>> complete output of that command? >>> >>> Ralf >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> Yes, I know that it is there and not sure what is missing! >> The typical way: >> $python3.1 setup.py build > It runs 2to3, but then for some reason decides not to use the output > of that on your system. Is that failing each time? If so, can you look > just above where it is failing, and see what's going on before/after > switching the path. Relevant code is: > > old_path = os.getcwd() > os.chdir(src_path) > sys.path.insert(0, src_path) > > Just inserting some print statements to look at os.getcwd() before and > after and at sys.path may tell you what's going on. > > Ralf > >> Also I know that it builds with Python3.1 and Python 3.2 with the latest git >> pull! But I have update nose before I can test it. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I can not figure out what is different between git and this version :-( All the paths appear to be the same. Further it continues onwards when I do: $python3.1 -m pdb setup.py build I added these lines to the start of "numpy/compat/__init__.py" import os print(os.getcwd()) /home/bsouthey/python/numpystuff/tmp6/numpy-1.6.0b1/build/py3k This is equivalent to the git version. Unless someone else finds it, I will consider it just my problem and ignore it. Bruce From pav at iki.fi Wed Mar 23 15:32:12 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 23 Mar 2011 19:32:12 +0000 (UTC) Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> <4D8A4740.50305@gmail.com> Message-ID: On Wed, 23 Mar 2011 14:17:20 -0500, Bruce Southey wrote: > I can not figure out what is different between git and this version :-( > > All the paths appear to be the same. > Further it continues onwards when I do: $python3.1 -m pdb setup.py build > > I added these lines to the start of "numpy/compat/__init__.py" import os > print(os.getcwd()) > > /home/bsouthey/python/numpystuff/tmp6/numpy-1.6.0b1/build/py3k This is > equivalent to the git version. > > Unless someone else finds it, I will consider it just my problem and > ignore it. There's some magic in setup.py for generating numpy/version.py, which behaves differently depending whether `.git` is present or not. It might do some stuff that goes wrong --- I fixed it once already, but maybe it's broken again. Pauli From robert.kern at gmail.com Wed Mar 23 15:46:32 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Mar 2011 14:46:32 -0500 Subject: [Numpy-discussion] numpy log2 has bug In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 13:51, wrote: > 2011/3/23 Dmitrey : >>>>> from numpy import log2, __version__ >>>>> >>>>> log2(2**63) >> Traceback (most recent call >> last): >> ? File "", line 1, in >> >> AttributeError: log2 >>>>> __version__ >> '2.0.0.dev-1fe8136' >> (doesn't work with 1.3.0 as well) > >>>> np.array([2**63]) > array([9223372036854775808], dtype=object) > >>>> log2(2.**63) > 62.999999999999993 >>>> log2(2**63) > Traceback (most recent call last): > ?File "", line 1, in > ? ?log2(2**63) > AttributeError: log2 > > integer conversion problem Right. numpy cannot safely convert a long object of that size to a dtype it knows about, so it leaves it as an object array. Most ufuncs operate on object arrays by looking for a method on each element with the name of the ufunc. So np.log2(np.array([x], dtype=object)) will look for x.log2(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bsouthey at gmail.com Wed Mar 23 15:48:07 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 23 Mar 2011 14:48:07 -0500 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> <4D8A4740.50305@gmail.com> Message-ID: <4D8A4E77.9020601@gmail.com> On 03/23/2011 02:32 PM, Pauli Virtanen wrote: > On Wed, 23 Mar 2011 14:17:20 -0500, Bruce Southey wrote: >> I can not figure out what is different between git and this version :-( >> >> All the paths appear to be the same. >> Further it continues onwards when I do: $python3.1 -m pdb setup.py build >> >> I added these lines to the start of "numpy/compat/__init__.py" import os >> print(os.getcwd()) >> >> /home/bsouthey/python/numpystuff/tmp6/numpy-1.6.0b1/build/py3k This is >> equivalent to the git version. >> >> Unless someone else finds it, I will consider it just my problem and >> ignore it. > There's some magic in setup.py for generating numpy/version.py, which > behaves differently depending whether `.git` is present or not. It might > do some stuff that goes wrong --- I fixed it once already, but maybe it's > broken again. > > Pauli > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion That's it! I commented out the setup.py line 'from numpy.version import git_revision as GIT_REVISION' and defined GIT_VERSION. (I reverted to an older nose version for nose) Python3.1 passes all tests but Python3.2 has a failure but I have to cleanup my messes in case it is me. Bruce $ python3.2 -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.6.0b1 NumPy is installed in /usr/local/lib/python3.2/site-packages/numpy Python version 3.2 (r32:88445, Mar 23 2011, 11:36:08) [GCC 4.5.1 20100924 (Red Hat 4.5.1-4)] nose version 3.0.0 ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.....................................................................Warning: invalid value encountered in true_divide Warning: invalid value encountered in true_divide Warning: invalid value encountered in true_divide .................................................................................................................K.................................................................................................K......................K..........................................................................................................STart of distr core ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./usr/local/lib/python3.2/site-packages/numpy/lib/format.py:575: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmpathakb'> mode=mode, offset=offset) ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................../usr/local/lib/python3.2/subprocess.py:460: ResourceWarning: unclosed file <_io.BufferedReader name=3> return Popen(*popenargs, **kwargs).wait() /usr/local/lib/python3.2/subprocess.py:460: ResourceWarning: unclosed file <_io.BufferedReader name=8> return Popen(*popenargs, **kwargs).wait() ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E........ ====================================================================== ERROR: Failure: OSError (/usr/local/lib/python3.2/site-packages/numpy/core/multiarray.pyd: cannot open shared object file: No such file or directory) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.2/site-packages/nose-3.0.0.dev-py3.2.egg/nose/failure.py", line 37, in runTest reraise(self.exc_class, self.exc_val, self.tb) File "/usr/local/lib/python3.2/site-packages/nose-3.0.0.dev-py3.2.egg/nose/_3.py", line 7, in reraise raise exc_class(exc_val).with_traceback(tb) File "/usr/local/lib/python3.2/site-packages/nose-3.0.0.dev-py3.2.egg/nose/loader.py", line 389, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib/python3.2/site-packages/nose-3.0.0.dev-py3.2.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib/python3.2/site-packages/nose-3.0.0.dev-py3.2.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/lib/python3.2/site-packages/numpy/tests/test_ctypeslib.py", line 8, in cdll = load_library('multiarray', np.core.multiarray.__file__) File "/usr/local/lib/python3.2/site-packages/numpy/ctypeslib.py", line 122, in load_library raise exc File "/usr/local/lib/python3.2/site-packages/numpy/ctypeslib.py", line 119, in load_library return ctypes.cdll[libpath] File "/usr/local/lib/python3.2/ctypes/__init__.py", line 415, in __getitem__ return getattr(self, name) File "/usr/local/lib/python3.2/ctypes/__init__.py", line 410, in __getattr__ dll = self._dlltype(name) File "/usr/local/lib/python3.2/ctypes/__init__.py", line 340, in __init__ self._handle = _dlopen(self._name, mode) OSError: /usr/local/lib/python3.2/site-packages/numpy/core/multiarray.pyd: cannot open shared object file: No such file or directory From d.s.seljebotn at astro.uio.no Wed Mar 23 15:54:12 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 23 Mar 2011 20:54:12 +0100 Subject: [Numpy-discussion] Make array uncopyable In-Reply-To: References: Message-ID: <4D8A4FE4.8060409@student.matnat.uio.no> On 03/23/2011 08:05 PM, Daniel Lepage wrote: > Hi all, > Is there a way to mark an array as uncopyable? If you have very > large data arrays some numpy functions will cause MemoryErrors because > behind the scenes they e.g. transpose the arrays into Fortran order to > feed them into Fortran libraries (numpy.linalg.svd is an example of > this). It would be great if there were a way to mark an array as "too > big to copy" so that an attempt to call transpose() or astype() would > raise an exception immediately instead of clobbering the machine's > memory first, but I don't know of any flag that does this. > > I suppose I could always subclass ndarray and redefine transpose(), > astype(), etc., but it'd be nice if there were an easier way. This is a bit OT, but if your problem is simply wanting to fail hard when using too much memory instead of swapping to disk, then on Unix systems you can use "ulimit -v" to limit how much memory your application can use. When I do this the MemoryError is quick and painless. Dag Sverre From ralf.gommers at googlemail.com Wed Mar 23 16:16:38 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 21:16:38 +0100 Subject: [Numpy-discussion] np.histogramdd of empty data In-Reply-To: <4D8879E3.1020603@amolf.nl> References: <4D8879E3.1020603@amolf.nl> Message-ID: On Tue, Mar 22, 2011 at 11:28 AM, Nils Becker wrote: > Hi, > > I was wondering why histogram2d and histogramdd raise a ValueError when > fed with empty data of the correct dimensions. I came across this as a > corner case when calling histogram2d from my own specialized histogram > function. Many function in numpy don't work with empty inputs, no big surprise. > > In comparison, histogram does handle this case correctly when bins are > specified explicitely (obviously automatic binning cannot work without > data...) That seems to be more an accident then anything else, empty input is not checked for or handled in histogram. This fixes it I think (please check, I just used your examples as test cases): https://github.com/rgommers/numpy/tree/histogram-empty Cheers, Ralf > np.histogram([], bins=([0,1])) > (array([0]), array([0, 1])) > > np.histogram2d([],[], bins=([0,1],[0,1])) > ValueError: zero-size array to ufunc.reduce without identity > > np.histogramdd([[],[]], bins=([0,1],[0,1])) > ValueError: zero-size array to ufunc.reduce without identity > > cheers, nils > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dplepage at gmail.com Wed Mar 23 16:24:16 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Wed, 23 Mar 2011 16:24:16 -0400 Subject: [Numpy-discussion] Make array uncopyable In-Reply-To: <4D8A4FE4.8060409@student.matnat.uio.no> References: <4D8A4FE4.8060409@student.matnat.uio.no> Message-ID: On Wed, Mar 23, 2011 at 3:54 PM, Dag Sverre Seljebotn wrote: > On 03/23/2011 08:05 PM, Daniel Lepage wrote: >> Hi all, >> ? ? Is there a way to mark an array as uncopyable? If you have very >> large data arrays some numpy functions will cause MemoryErrors because >> behind the scenes they e.g. transpose the arrays into Fortran order to >> feed them into Fortran libraries (numpy.linalg.svd is an example of >> this). It would be great if there were a way to mark an array as "too >> big to copy" so that an attempt to call transpose() or astype() would >> raise an exception immediately instead of clobbering the machine's >> memory first, but I don't know of any flag that does this. >> >> I suppose I could always subclass ndarray and redefine transpose(), >> astype(), etc., but it'd be nice if there were an easier way. > > This is a bit OT, but if your problem is simply wanting to fail hard > when using too much memory instead of swapping to disk, then on Unix > systems you can use "ulimit -v" to limit how much memory your > application can use. When I do this the MemoryError is quick and painless. Ah, I'd completely forgotten that I could do that. That'll help a lot, thanks! That said, it'd still be cool if there were a way to have numpy warn me of these things, so that I tell the difference between "I ran out of memory because I did something stupid" and "I ran out of memory because numpy was doubling things behind the scenes". What would be *really* cool is if I could get error messages telling me why something wanted to copy my array (e.g. UncopyableException("This function requires Fortran-style arrays")) so that I'd know what I'd need to do to my arrays if I wanted the function to work, but I'm sure applying that to all of numpy would be a gargantuan undertaking. Thanks, Dan Lepage From ralf.gommers at googlemail.com Wed Mar 23 16:29:19 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Mar 2011 21:29:19 +0100 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> <4D8A4740.50305@gmail.com> Message-ID: On Wed, Mar 23, 2011 at 8:32 PM, Pauli Virtanen wrote: > On Wed, 23 Mar 2011 14:17:20 -0500, Bruce Southey wrote: >> I can not figure out what is different between git and this version :-( >> >> All the paths appear to be the same. >> Further it continues onwards when I do: $python3.1 -m pdb setup.py build >> >> I added these lines to the start of "numpy/compat/__init__.py" import os >> print(os.getcwd()) >> >> /home/bsouthey/python/numpystuff/tmp6/numpy-1.6.0b1/build/py3k This is >> equivalent to the git version. >> >> Unless someone else finds it, I will consider it just my problem and >> ignore it. > > There's some magic in setup.py for generating numpy/version.py, which > behaves differently depending whether `.git` is present or not. It might > do some stuff that goes wrong --- I fixed it once already, but maybe it's > broken again. I un-fixed your fix by accident, since I missed why you did that and your fix broke the paver script. Leaving the FULLVERSION line outside write_version_py() but putting the import inside should fix it, only the commit part will be missing when building packages for an unreleased version. Ralf From d.s.seljebotn at astro.uio.no Wed Mar 23 16:36:11 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 23 Mar 2011 21:36:11 +0100 Subject: [Numpy-discussion] Make array uncopyable In-Reply-To: References: <4D8A4FE4.8060409@student.matnat.uio.no> Message-ID: <4D8A59BB.1050609@student.matnat.uio.no> On 03/23/2011 09:24 PM, Daniel Lepage wrote: > On Wed, Mar 23, 2011 at 3:54 PM, Dag Sverre Seljebotn > wrote: >> On 03/23/2011 08:05 PM, Daniel Lepage wrote: >>> Hi all, >>> Is there a way to mark an array as uncopyable? If you have very >>> large data arrays some numpy functions will cause MemoryErrors because >>> behind the scenes they e.g. transpose the arrays into Fortran order to >>> feed them into Fortran libraries (numpy.linalg.svd is an example of >>> this). It would be great if there were a way to mark an array as "too >>> big to copy" so that an attempt to call transpose() or astype() would >>> raise an exception immediately instead of clobbering the machine's >>> memory first, but I don't know of any flag that does this. >>> >>> I suppose I could always subclass ndarray and redefine transpose(), >>> astype(), etc., but it'd be nice if there were an easier way. >> This is a bit OT, but if your problem is simply wanting to fail hard >> when using too much memory instead of swapping to disk, then on Unix >> systems you can use "ulimit -v" to limit how much memory your >> application can use. When I do this the MemoryError is quick and painless. > Ah, I'd completely forgotten that I could do that. That'll help a lot, thanks! > > That said, it'd still be cool if there were a way to have numpy warn > me of these things, so that I tell the difference between "I ran out > of memory because I did something stupid" and "I ran out of memory > because numpy was doubling things behind the scenes". > > What would be *really* cool is if I could get error messages telling > me why something wanted to copy my array (e.g. > UncopyableException("This function requires Fortran-style arrays")) so > that I'd know what I'd need to do to my arrays if I wanted the > function to work, but I'm sure applying that to all of numpy would be > a gargantuan undertaking. Well, most such copies happen in the same NumPy C API function, so a global flag there would go a long way (put perhaps disable the possibility of explicit copying as well...). But I don't think it is implemented. And yes, doing it in any nice way would likely be a major effort. For a quicker workaround that doesn't touch NumPy, you could also implement a context manager in Cython that temporarily sets the ulimit ("man setrlimit", and use the soft limits), and wrap your calls to NumPy in that. Dag Sverre From derek at astro.physik.uni-goettingen.de Wed Mar 23 18:43:50 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Wed, 23 Mar 2011 23:43:50 +0100 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <4D8A6C9C.1010005@gmail.com> References: <90A85509-C029-4684-BE95-91ED9CD45C50@astro.physik.uni-goettingen.de> <4426C763-D075-437A-8C65-4219F40DDA18@astro.physik.uni-goettingen.de> <4D8A6C9C.1010005@gmail.com> Message-ID: <0B87A30E-5EBB-4F8F-847F-B46ABF8397A9@astro.physik.uni-goettingen.de> Hi Bruce, > Sorry as I should I have paid more attention to you first email as I > raised this on the numpy list. (At least with gmail, the reply is to > the > scipy-dev list rather than the numpy list.) > > Based on Pauli's answer I got around it by commenting out a line in > the > setup.py file about the git version: > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/ > 055603.html > thanks, I was not subscribed to numpy-discussion (I realise I better should before going into major testing...)! This patch diff -ruN --exclude='*~' numpy-1.6.0b1.orig/setup.py numpy-1.6.0b1/ setup.py --- numpy-1.6.0b1.orig/setup.py 2011-03-23 09:24:00.000000000 +0100 +++ numpy-1.6.0b1/setup.py 2011-03-23 23:03:19.000000000 +0100 @@ -99,9 +99,10 @@ FULLVERSION = VERSION if os.path.exists('.git'): GIT_REVISION = git_version() -elif os.path.exists('numpy/version.py'): - # must be a source distribution, use existing version file - from numpy.version import git_revision as GIT_REVISION +# does not work in a clean source distribution: +#elif os.path.exists('numpy/version.py'): +# # must be a source distribution, use existing version file +# from numpy.version import git_revision as GIT_REVISION else: GIT_REVISION = "Unknown" did the build for Python3.1 and Python3.2. Cheers, Derek From derek at astro.physik.uni-goettingen.de Wed Mar 23 19:34:12 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 24 Mar 2011 00:34:12 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> Hi again, On 23 Mar 2011, at 17:07, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta of NumPy > 1.6.0. Due to the extensive changes in the Numpy core for this > release, the beta testing phase will last at least one month. Please > test this beta and report any problems on the Numpy mailing list. tests with the fink-installed pythons on MacOS X mostly succeeded, with one failure in python2.4 and a couple of issues seemingly related to PPC floating point accuracy, as below: OSX/Python 2.4 2.5 2.6 2.7 3.1 3.2 10.5/i386 FAIL[1] pass pass pass pass pass 10.5/ppc FAIL[1,2] FAIL[2] FAIL[2] FAIL[2] FAIL[2] FAIL[2] 10.6/x86_64 n/a pass pass pass pass pass Failures: [1] Python 2.4.4 (#1, Jan 5 2011, 03:05:41) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test('full') Running unit tests for numpy NumPy version 1.6.0b1 NumPy is installed in /sw/lib/python2.4/site-packages/numpy Python version 2.4.4 (#1, Jan 5 2011, 03:05:41) [GCC 4.0.1 (Apple Inc. build 5493)] nose version 1.0.0 ........ FAIL: test_iterator.test_iter_broadcasting_errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.4/site-packages/nose/case.py", line 187, in runTest self.test(*self.arg) File "/sw/lib/python2.4/site-packages/numpy/core/tests/ test_iterator.py", line 639, in test_iter_broadcasting_errors 'Message "%s" doesn\'t contain operand shape (2,3)' % msg) File "/sw/lib/python2.4/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Message "non-broadcastable output operand with shape (%zd,%zd) doesn't match the broadcast shape (%zd,%zd,%zd)" doesn't contain operand shape (2,3) [2] identical in Python 2.5.5 (r255:77872, Jan 4 2011, 03:26:36) Python 2.6.6 (r266:84292, Jan 4 2011, 03:36:52) Python 2.7.1 (r271:86832, Dec 17 2010, 19:39:40) Python 3.1.3 (r313:86834, Dec 17 2010, 19:49:56) Python 3.2 (r32:88445, Mar 1 2011, 18:28:16) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.6.0b1 NumPy is installed in /sw/lib/python3.2/site-packages/numpy Python version 3.2 (r32:88445, Mar 1 2011, 18:28:16) [GCC 4.0.1 (Apple Inc. build 5493)] nose version 1.0.0 ..............................................................................................................................................Warning : overflow encountered in add Warning: invalid value encountered in subtract Warning: invalid value encountered in subtract Warning: overflow encountered in add Warning: invalid value encountered in subtract Warning: invalid value encountered in subtract ..................................................................................................................................................................................................................................................................................................................FF ................................................................................................................................................................................................................................K .....................................................................Warning : invalid value encountered in true_divide Warning: invalid value encountered in true_divide Warning: invalid value encountered in true_divide .................................................................................................................K ...K ...........................................................F ..F.....Warning: invalid value encountered in exp .........................K ...SK .S .......S ........................................................................ ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./sw /lib/python3.2/site-packages/numpy/lib/format.py:575: ResourceWarning: unclosed file <_io.BufferedReader name='/scratch/derek/tmp/tmpuqv5id'> mode=mode, offset=offset) ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ......................S........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_numeric.py", line 308, in test_floating_exceptions lambda a,b:a/b, ft_tiny, ft_max) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_numeric.py", line 261, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error 'underflow'. ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/numpy/testing/decorators.py", line 216, in knownfailer return f(*args, **kwargs) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_numeric.py", line 352, in test_floating_exceptions_power np.power, ftype(2), ftype(2**fi.nexp)) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_numeric.py", line 261, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error 'overflow'. ====================================================================== FAIL: test_umath.test_nextafterl ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest self.test(*self.arg) File "/sw/lib/python3.2/site-packages/numpy/testing/decorators.py", line 216, in knownfailer return f(*args, **kwargs) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_umath.py", line 1085, in test_nextafterl return _test_nextafter(np.longdouble) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_umath.py", line 1071, in _test_nextafter assert np.nextafter(one, two) - one == eps AssertionError ====================================================================== FAIL: test_umath.test_spacingl ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest self.test(*self.arg) File "/sw/lib/python3.2/site-packages/numpy/testing/decorators.py", line 216, in knownfailer return f(*args, **kwargs) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_umath.py", line 1110, in test_spacingl return _test_spacing(np.longdouble) File "/sw/lib/python3.2/site-packages/numpy/core/tests/ test_umath.py", line 1094, in _test_spacing assert np.spacing(one) == eps AssertionError ---------------------------------------------------------------------- Ran 3362 tests in 64.845s FAILED (KNOWNFAIL=5, SKIP=4, failures=4) From derek at astro.physik.uni-goettingen.de Wed Mar 23 20:04:32 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 24 Mar 2011 01:04:32 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> Message-ID: <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> On 24 Mar 2011, at 00:34, Derek Homeier wrote: > tests with the fink-installed pythons on MacOS X mostly succeeded, > with one failure in python2.4 and a couple of issues seemingly > related to PPC floating point accuracy, as below: > Probably last update for tonight: with the 'full' test suite, there's one additional failure and error, respectively under 10.5/ppc and 10.6/x86_64 (in all Python versions): PowerPC: FAIL: test_kind.TestKind.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.5/site-packages/nose/case.py", line 187, in runTest self.test(*self.arg) File "/sw/lib/python2.5/site-packages/numpy/f2py/tests/test_kind.py", line 30, in test_all 'selectedrealkind(%s): expected %r but got %r' % (i, selected_real_kind(i), selectedrealkind(i))) File "/sw/lib/python2.5/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: selectedrealkind(16): expected 10 but got 16 Intel-64bit: ERROR: test_assumed_shape.TestAssumedShapeSumExample.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/nose/case.py", line 372, in setUp try_run(self.inst, ('setup', 'setUp')) File "/sw/lib/python3.2/site-packages/nose/util.py", line 478, in try_run return func() File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 352, in setUp module_name=self.module_name) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper memo[key] = func(*a, **kw) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 134, in build_module % (cmd[4:], asstr(out))) RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/precision.f90'] Reading .f2py_f2cmap ... Mapping "real(kind=rk)" to "double" Succesfully applied user defined changes from .f2py_f2cmap running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building extension "_test_ext_module_5403" sources f2py options: [] f2py:> /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403module.c creating /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as creating /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2 Reading fortran codes... Reading file '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90' (format:free) Reading file '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90' (format:free) Reading file '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/precision.f90' (format:free) Post-processing... Block: _test_ext_module_5403 Block: sum Block: fsum Block: sum_with_use Block: precision Post-processing (stage 2)... Block: _test_ext_module_5403 Block: unknown_interface Block: sum Block: fsum Block: sum_with_use Block: precision Building modules... Building module "_test_ext_module_5403"... Creating wrapper for Fortran subroutine "sum"("sum")... Constructing wrapper function "sum"... res = sum(x) Creating wrapper for Fortran function "fsum"("fsum")... Constructing wrapper function "fsum"... res = fsum(x) Creating wrapper for Fortran subroutine "sum_with_use"("sum_with_use")... Constructing wrapper function "sum_with_use"... res = sum_with_use(x) Constructing F90 module support for "precision"... Variables: ik rk Wrote C/API module "_test_ext_module_5403" to file "/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/- [skipping additional compiler output] compiling Fortran sources Fortran f77 compiler: /sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /sw/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops compile options: '-I/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2 -I/sw/lib/python3.2/site-packages/numpy/core/include -I/sw/include/python3.2m -c' extra options: '-J/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/ -I/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/' gfortran:f90: /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90 gfortran:f90: /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90 gfortran:f77: /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403-f2pywrappers.f /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403-f2pywrappers.f:26.21: real :: res 1 Error: Symbol 'res' at (1) already has basic type of REAL /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403-f2pywrappers.f:26.21: real :: res 1 Error: Symbol 'res' at (1) already has basic type of REAL error: Command "/sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops -I/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2 -I/sw/lib/python3.2/site-packages/numpy/core/include -I/sw/include/python3.2m -c -c /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403-f2pywrappers.f -o /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/src.macosx-10.6-x86_64-3.2/_test_ext_module_5403-f2pywrappers.o -J/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/ -I/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpv567as/" failed with exit status 1 ---------------------------------------------------------------------- Ran 3415 tests in 75.507s FAILED (KNOWNFAIL=4, SKIP=1, errors=1) From tartley at tartley.com Wed Mar 23 20:09:21 2011 From: tartley at tartley.com (Jonathan Hartley) Date: Thu, 24 Mar 2011 00:09:21 +0000 Subject: [Numpy-discussion] newbie question about boolean testing of array equality result In-Reply-To: References: <4D8A03BE.9010506@tartley.com> Message-ID: <4D8A8BB1.5000707@tartley.com> Thanks for the responses, folks. Robert, thanks for helping me come to terms with the situation, for pointing out the perils of comparing NaNs, and particularly for the "not isinstance(x, bool)" idea - some variant of this or attribute checking as you suggest sounds like what we'll use. Daniel, thank you also for your thoughtful guidance on floating point numerical accuracy, and the best way to tackle comparisons using numpy. However, I'm reluctant to go down that route - as I say, we otherwise don't have any dependency on Numpy, and I'm reluctant to add one merely so that I can stop numpy arrays from raising exceptions. The sample code I showed you was over-simplified for clarity, perhaps to the point of actually being misleading, for which I apologise. We're comparing the result not just to NaN (for which we're actually, on reflection, using math.is_nan()) but also to many other values, of various types - The example of "== float('nan')" was just a (with hindsight, badly chosen) representative example of that. Best regards, and thanks for putting up with my whining, Jonathan On 23/03/2011 14:59, Robert Kern wrote: > On Wed, Mar 23, 2011 at 09:29, Jonathan Hartley wrote: >> Hey people, >> >> I'm writing an application in which we evaluate user-supplied Python >> expressions. >> >> Sometimes, we want to do an equality test on the result of the evaluation, >> eg: >> >> result = eval(...) >> if result == float('nan'): >> ... > Please note that this particular expression will *never* work, even > with float objects. float('nan') != float('nan'). It's a quick of > floating point semantics. > >> If the result is a numpy array, then this raises a ValueError, since the >> array equality operator returns a new numpy array, and the coercion of this >> to a boolean for the if predicate then explicitly raises. Presumably this is >> well-known? > Yes. > >> For us, it is undesirable. >> >> Am I right to understand that any code which might ever encounter a numpy >> array therefore can never use an unguarded 'if x == y:' construction? Is my >> workaround really to replace every instance of this with 'if not >> isinstance(x, numpy.array) and x==y:' ? This pains me, because otherwise >> this code would have no dependency on numpy. (I can't just prepend >> 'isinstance(x, float)' because, unlike the example above, we don't always >> know the type of the equality RHS until runtime.) > def equals(x, y): > z = x == y > if not isinstance(z, bool): > # Or maybe you check for the existence of .all() or .any() > depending on which semantics you would like in the presence of numpy > arrays. > z = False > return z > >> I can see that there's a pleasing symmetry to have all the array arithmetic >> operators and comparisons operate in an element-wise manner, but I think >> it's more important for __eq__ to follow it's usual semantics of returning a >> boolean. I'd way prefer it if the element-wise equality array generation was >> exposed as a different method. > I'm afraid that it is far too late to make such a change. > -- Jonathan Hartley tartley at tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley From SSharma84 at slb.com Thu Mar 24 03:27:28 2011 From: SSharma84 at slb.com (Sachin Kumar Sharma) Date: Thu, 24 Mar 2011 07:27:28 +0000 Subject: [Numpy-discussion] how to loop read input over various file with same extension in a folder (newbie question) Message-ID: <75C2FED246299A478280FA1470EDA4432D9269F9@NL0230MBX06N1.DIR.slb.com> BB, Currently I am reading a text file and storing the output as an array a1, some arithmetic operation on it and plotting it. I want to do the same over 1000 files. How to loop it and how to store the input in different arrays like a2, a3 and so on. Also advise what is the best way to TAG or assign a comment to an array to store which file it has read the data. Cheers Sachin ************************************************************************ Sachin Kumar Sharma Senior Geomodeler -------------- next part -------------- An HTML attachment was scrubbed... URL: From SSharma84 at slb.com Thu Mar 24 03:42:47 2011 From: SSharma84 at slb.com (Sachin Kumar Sharma) Date: Thu, 24 Mar 2011 07:42:47 +0000 Subject: [Numpy-discussion] how to delete a particular column or row from numpy array based on some rule or value Message-ID: <75C2FED246299A478280FA1470EDA4432D926A09@NL0230MBX06N1.DIR.slb.com> BB, In a numpy array of m x n size, I would like to delete few rows when a given element in that row is 'nan' or say any other value. For e.g. as in example given below, if I wish to delete a row when the 3rd element of row is zero, or if 3rd, 4th, 5th element are zero or either of 3rd, 4th and 5th element is zero. array([[ 1900. , nan, nan, nan, nan, nan], [ 1900.5, nan, nan, nan, nan, nan], [ 1901. , nan, nan, nan, nan, nan], ..., [ 6724. , nan, nan, nan, nan, nan], [ 6724.5, nan, nan, nan, nan, nan], [ 6725. , nan, nan, nan, nan, nan]]) Cheers Sachin ************************************************************************ Sachin Kumar Sharma Senior Geomodeler -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Thu Mar 24 03:53:34 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Thu, 24 Mar 2011 08:53:34 +0100 Subject: [Numpy-discussion] how to loop read input over various file with same extension in a folder (newbie question) In-Reply-To: <75C2FED246299A478280FA1470EDA4432D9269F9@NL0230MBX06N1.DIR.slb.com> References: <75C2FED246299A478280FA1470EDA4432D9269F9@NL0230MBX06N1.DIR.slb.com> Message-ID: <3F538A99-76CE-4AE4-BD41-D6A47568FE7C@gmail.com> One way is to use python dicts. Pseudocode: arrays = {} integer = 0 for file in allfiles: integer +=1 data = file.read() arrays['a' + '%d' % integer] = data I would consider using the filename as the key instead of a1, a2, etc. That way you have that information readily available as well. Good luck, Paul. On 24. mars 2011, at 08.27, Sachin Kumar Sharma wrote: > BB, > > Currently I am reading a text file and storing the output as an array a1, some arithmetic operation on it and plotting it. I want to do the same over 1000 files. > How to loop it and how to store the input in different arrays like a2, a3 and so on. > > Also advise what is the best way to TAG or assign a comment to an array to store which file it has read the data. > > Cheers > > Sachin > > ************************************************************************ > Sachin Kumar Sharma > Senior Geomodeler > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From SSharma84 at slb.com Thu Mar 24 03:55:36 2011 From: SSharma84 at slb.com (Sachin Kumar Sharma) Date: Thu, 24 Mar 2011 07:55:36 +0000 Subject: [Numpy-discussion] how to loop read input over various file with same extension in a folder (newbie question) In-Reply-To: <3F538A99-76CE-4AE4-BD41-D6A47568FE7C@gmail.com> References: <75C2FED246299A478280FA1470EDA4432D9269F9@NL0230MBX06N1.DIR.slb.com> <3F538A99-76CE-4AE4-BD41-D6A47568FE7C@gmail.com> Message-ID: <75C2FED246299A478280FA1470EDA4432D926A63@NL0230MBX06N1.DIR.slb.com> Paul, Appreciate your reply. Cheers Sachin ************************************************************************ Sachin Kumar Sharma Senior Geomodeler -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Paul Anton Letnes Sent: Thursday, March 24, 2011 3:54 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] how to loop read input over various file with same extension in a folder (newbie question) One way is to use python dicts. Pseudocode: arrays = {} integer = 0 for file in allfiles: integer +=1 data = file.read() arrays['a' + '%d' % integer] = data I would consider using the filename as the key instead of a1, a2, etc. That way you have that information readily available as well. Good luck, Paul. On 24. mars 2011, at 08.27, Sachin Kumar Sharma wrote: > BB, > > Currently I am reading a text file and storing the output as an array a1, some arithmetic operation on it and plotting it. I want to do the same over 1000 files. > How to loop it and how to store the input in different arrays like a2, a3 and so on. > > Also advise what is the best way to TAG or assign a comment to an array to store which file it has read the data. > > Cheers > > Sachin > > ************************************************************************ > Sachin Kumar Sharma > Senior Geomodeler > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From pearu.peterson at gmail.com Thu Mar 24 04:11:50 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Thu, 24 Mar 2011 10:11:50 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 24, 2011 at 2:04 AM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > On 24 Mar 2011, at 00:34, Derek Homeier wrote: > > > tests with the fink-installed pythons on MacOS X mostly succeeded, > > with one failure in python2.4 and a couple of issues seemingly > > related to PPC floating point accuracy, as below: > > > Probably last update for tonight: with the 'full' test suite, there's one > additional failure and error, respectively under 10.5/ppc and > 10.6/x86_64 (in all Python versions): > > PowerPC: > > FAIL: test_kind.TestKind.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.5/site-packages/nose/case.py", line 187, in runTest > self.test(*self.arg) > File "/sw/lib/python2.5/site-packages/numpy/f2py/tests/test_kind.py", line > 30, in test_all > 'selectedrealkind(%s): expected %r but got %r' % (i, > selected_real_kind(i), selectedrealkind(i))) > File "/sw/lib/python2.5/site-packages/numpy/testing/utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError: selectedrealkind(16): expected 10 but got 16 > Regarding this test failure, could you hack the numpy/f2py/tests/test_kind.py script by adding the following code for i in range(20): print '%s -> %s, %s' % (i, selected_real_kind(i), selectedrealkind(i)) and send me the output? Also, what Fortran compiler version has been used to build the test modules? > Intel-64bit: > ERROR: test_assumed_shape.TestAssumedShapeSumExample.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python3.2/site-packages/nose/case.py", line 372, in setUp > try_run(self.inst, ('setup', 'setUp')) > File "/sw/lib/python3.2/site-packages/nose/util.py", line 478, in try_run > return func() > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 352, > in setUp > module_name=self.module_name) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 73, > in wrapper > memo[key] = func(*a, **kw) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 134, > in build_module > % (cmd[4:], asstr(out))) > RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/precision.f90'] > Reading .f2py_f2cmap ... Mapping "real(kind=rk)" to "double" Hmm, this should not happen as real(kind=rk) should be mapped to "float". It seems that you have .f2py_f2cmap file lying around. Could you remove it and try again. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Thu Mar 24 04:46:26 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Thu, 24 Mar 2011 10:46:26 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 24, 2011 at 10:11 AM, Pearu Peterson wrote: > > Regarding this test failure, could you hack the > numpy/f2py/tests/test_kind.py script by adding the following code > > for i in range(20): > print '%s -> %s, %s' % (i, selected_real_kind(i), selectedrealkind(i)) > > and send me the output? Also, what Fortran compiler version has been used > to build the test modules? > Even better, you can report the result to http://projects.scipy.org/numpy/ticket/1767 Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dileepkunjaai at gmail.com Thu Mar 24 05:06:35 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Thu, 24 Mar 2011 14:36:35 +0530 Subject: [Numpy-discussion] how to delete a particular column or row from numpy array based on some rule or value In-Reply-To: <75C2FED246299A478280FA1470EDA4432D926A09@NL0230MBX06N1.DIR.slb.com> References: <75C2FED246299A478280FA1470EDA4432D926A09@NL0230MBX06N1.DIR.slb.com> Message-ID: On Thu, Mar 24, 2011 at 1:12 PM, Sachin Kumar Sharma wrote: > BB, > > > > In a numpy array of m x n size, I would like to delete few rows when a > given element in that row is ?nan? or say any other value. > > > > For e.g. as in example given below, if I wish to delete a row when the 3 > rd element of row is zero, or if 3rd, 4th, 5th element are zero or either > of 3rd, 4th and 5th element is zero. > > array([[ 1900. , nan, nan, nan, nan, nan], > > [ 1900.5, nan, nan, nan, nan, nan], > > [ 1901. , nan, nan, nan, nan, nan], > > ..., > > [ 6724. , nan, nan, nan, nan, nan], > > [ 6724.5, nan, nan, nan, nan, nan], > > [ 6725. , nan, nan, nan, nan, nan]]) > > Cheers > > > > Sachin* * > > ************************************************************************* > Sachin Kumar Sharma* > > Senior Geomodeler ** > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I think u can use numpy.loadtext command http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Thu Mar 24 06:00:30 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 12:00:30 +0200 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs Message-ID: hi, when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs I tried to install numpy from PYPI where 1.5.1 seesm to be present, but somehow it involves 1.3.0 instead: $ sudo easy_install numpy install_dir /usr/local/lib/python2.6/dist-packages/ Searching for numpy Best match: numpy 1.3.0 Adding numpy 1.3.0 to easy-install.pth file only after aptitude remove python-numpy version 1.5.1. is involved by easy_install, but it fails: $ sudo easy_install numpy .... Adding numpy 1.5.1 to easy-install.pth file Installing f2py script to /usr/local/bin Installed /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg Processing dependencies for numpy Finished processing dependencies for numpy /tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py:251: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", line 251, in clean_up_temporary_directory ImportError: No module named numpy.distutils Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", line 251, in clean_up_temporary_directory ImportError: No module named numpy.distutils I have Linux KUBUNTU 10.10 D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Thu Mar 24 06:53:55 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 12:53:55 +0200 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output Message-ID: >>> from numpy import inf, array >>> inf*0 nan (ok) >>> array(inf) * 0.0 StdErr: Warning: invalid value encountered in multiply nan My cycled calculations yields this thousands times slowing computations and making text output completely non-readable. >>> from numpy import __version__ >>> __version__ '2.0.0.dev-1fe8136' D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.cognacc at gmail.com Thu Mar 24 06:51:50 2011 From: michael.cognacc at gmail.com (Mic J) Date: Thu, 24 Mar 2011 11:51:50 +0100 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs In-Reply-To: References: Message-ID: Isnt [K]Ubuntu updated each 6 month? 2011/3/24 Dmitrey : > hi, > when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs There will always be bugs, but numpy 1.3 is a stable release, unless there is a bug that affects what your doing right now? If you find a bug that prevents you from from your specific work, better report that bug, if you haven't already. > I tried to install numpy from PYPI where 1.5.1 seesm to be present, but > somehow it involves 1.3.0 instead: > > $ sudo easy_install numpy > install_dir /usr/local/lib/python2.6/dist-packages/ > Searching for numpy > Best match: numpy 1.3.0 > Adding numpy 1.3.0 to easy-install.pth file > > only after aptitude remove python-numpy version 1.5.1. is involved by > easy_install, but it fails: > > $ sudo easy_install numpy > .... > Adding numpy 1.5.1 to easy-install.pth file > Installing f2py script to /usr/local/bin > > Installed > /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg > Processing dependencies for numpy > Finished processing dependencies for numpy > /tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py:251: > RuntimeWarning: Parent module 'numpy.distutils' not found while handling > absolute import I see an *absolute* import, maybe easy_install is different from where ubuntu expects numpy to be installed? i think the folder distutils is specific for ubuntu ? try and remove manually the egg in /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg then do $ sudo easy_install -U numpy -U is for update, maybe you have to run without -U first? > Error in atexit._run_exitfuncs: > Traceback (most recent call last): > ? File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs > ??? func(*targs, **kargs) > ? File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", > line 251, in clean_up_temporary_directory > ImportError: No module named numpy.distutils > Error in sys.exitfunc: > Traceback (most recent call last): > ? File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs > ??? func(*targs, **kargs) > ? File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", > line 251, in clean_up_temporary_directory > ImportError: No module named numpy.distutils > > I have Linux KUBUNTU 10.10 > > D. Good question.. ? easy_install should be distribution non-specific(as far as i know) see also this thread, even though its old. (late 2009) http://old.nabble.com/numpy-1.3.0-eggs-with-python2.6-seem-broken-on-osx,-and-linux-td26551531.html so maybe then download only the egg(with easy_install), then enter the folder and do sudo python2.6 setup.py install Dont you have python 2.7 also on ubuntu? regards mic From e.antero.tammi at gmail.com Thu Mar 24 07:02:19 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 24 Mar 2011 13:02:19 +0200 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: Hi 2011/3/24 Dmitrey > >>> from numpy import inf, array > >>> inf*0 > nan > > (ok) > > >>> array(inf) * 0.0 > StdErr: Warning: invalid value encountered in multiply > nan > > My cycled calculations yields this thousands times slowing computations and > making text output completely non-readable. > Would old= seterr(invalid= 'ignore') be sufficient for you? Regards, eat > > >>> from numpy import __version__ > >>> __version__ > '2.0.0.dev-1fe8136' > > D. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Thu Mar 24 07:24:43 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 13:24:43 +0200 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: Hi 2011/3/24 Dmitrey >>> from numpy import inf, array >>> inf*0 nan (ok) >>> array(inf) * 0.0 StdErr: Warning: invalid value encountered in multiply nan My cycled calculations yields this thousands times slowing computations and making text output completely non-readable. Would old= seterr(invalid= 'ignore') be sufficient for you? yes for me, but I'm not sure for all those users who use my soft. Maybe it will hide some bugs in their objective functions and nonlinear constraints in numerical optimization and nonlinear equation systems. D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Thu Mar 24 07:28:00 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 13:28:00 +0200 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs In-Reply-To: References: Message-ID: Isnt [K]Ubuntu updated each 6 month? 2011/3/24 Dmitrey < tmp50 at ukr.net >: > hi, > when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs There will always be bugs, but numpy 1.3 is a stable release, unless there is a bug that affects what your doing right now? If you find a bug that prevents you from from your specific work, better report that bug, if you haven't already. > I tried to install numpy from PYPI where 1.5.1 seesm to be present, but > somehow it involves 1.3.0 instead: > > $ sudo easy_install numpy > install_dir /usr/local/lib/python2.6/dist-packages/ > Searching for numpy > Best match: numpy 1.3.0 > Adding numpy 1.3.0 to easy-install.pth file > > only after aptitude remove python-numpy version 1.5.1. is involved by > easy_install, but it fails: > > $ sudo easy_install numpy > .... > Adding numpy 1.5.1 to easy-install.pth file > Installing f2py script to /usr/local/bin > > Installed > /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg > Processing dependencies for numpy > Finished processing dependencies for numpy > /tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py:251: > RuntimeWarning: Parent module 'numpy.distutils' not found while handling > absolute import I see an *absolute* import, maybe easy_install is different from where ubuntu expects numpy to be installed? i think the folder distutils is specific for ubuntu ? try and remove manually the egg in /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg then do $ sudo easy_install -U numpy -U is for update, maybe you have to run without -U first? > Error in atexit._run_exitfuncs: > Traceback (most recent call last): > File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs > func(*targs, **kargs) > File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", > line 251, in clean_up_temporary_directory > ImportError: No module named numpy.distutils > Error in sys.exitfunc: > Traceback (most recent call last): > File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs > func(*targs, **kargs) > File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", > line 251, in clean_up_temporary_directory > ImportError: No module named numpy.distutils > > I have Linux KUBUNTU 10.10 > > D. Good question.. ? easy_install should be distribution non-specific(as far as i know) see also this thread, even though its old. (late 2009) http://old.nabble.com/numpy-1.3.0-eggs-with-python2.6-seem-broken-on-osx,-and-linux-td26551531.html so maybe then download only the egg(with easy_install), then enter the folder and do sudo python2.6 setup.py install Dont you have python 2.7 also on ubuntu? regards mic _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Thanks for all those instructions, however, personally I don't need them, I have sucseeded with my own manipulations and even if I wouldn't I always can build numpy/scipy from sources. I mere care for quality and easibility of numpy installation for ordinary non-skilled users. They may just try installation, see that it's buggy and move away to use another soft. D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Thu Mar 24 07:38:42 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 24 Mar 2011 12:38:42 +0100 Subject: [Numpy-discussion] [NumPy-Tickets] [NumPy] #971: numpy.memmap 'offset' parameter docs are almost entirely wrong. In-Reply-To: <026.3efff8747b994514f447acc84c4f4fdc@scipy.org> References: <017.ffa2a7653d4da49c365af4cde463d946@scipy.org> <026.3efff8747b994514f447acc84c4f4fdc@scipy.org> Message-ID: Hi, I got an "Internal Error" message when trying to log into the Trac page. My comment: Could there at least be a note added to the current memmap doc string, stating that there is a problem with the current doc; also a link to http://projects.scipy.org/numpy/ticket/97 would be much better than nothing. My 2 cents, Sebastian Haase On Thu, Mar 24, 2011 at 1:58 AM, wrote: > #971: numpy.memmap 'offset' parameter docs are almost entirely wrong. > ---------------------------+------------------------------------------------ > ?Reporter: ?0ion9 ? ? ? ? ?| ? ? ? Owner: ?pv > ? ? Type: ?defect ? ? ? ? | ? ? ?Status: ?new > ?Priority: ?normal ? ? ? ? | ? Milestone: ?Unscheduled > Component: ?Documentation ?| ? ? Version: ?devel > ?Keywords: ? ? ? ? ? ? ? ? | > ---------------------------+------------------------------------------------ > Changes (by mwiebe): > > ?* milestone: ?=> Unscheduled > > > -- > Ticket URL: > NumPy > My example project > _______________________________________________ > NumPy-Tickets mailing list > NumPy-Tickets at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-tickets > From tmp50 at ukr.net Thu Mar 24 07:55:09 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 13:55:09 +0200 Subject: [Numpy-discussion] argmin and argmax without nan Message-ID: hi, is there any way to get argmin and argmax of an array w/o nans? Currently I have >>> from numpy import * >>> argmax([10,nan,100]) 1 >>> argmin([10,nan,100]) 1 But it's not the values I would like to get. The walkaround I use: get all indeces of nans, replace them by -inf, get argmax, replace them by inf, get argmin. Is there any better way? (BTW, I invoke argmin/argmax along of a chosen axis) D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Mar 24 09:19:13 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 14:19:13 +0100 Subject: [Numpy-discussion] argmin and argmax without nan In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey : > hi, > is there any way to get argmin and argmax of an array w/o nans? > Currently I have >>>> from numpy import * >>>> argmax([10,nan,100]) > 1 >>>> argmin([10,nan,100]) > 1 > But it's not the values I would like to get. > > The walkaround I use: get all indeces of nans, replace them by -inf, get > argmax, replace them by inf, get argmin. > Is there any better way? (BTW, I invoke argmin/argmax along of a chosen > axis) > D. In [3]: np.nanargmax([10, np.nan, 100]) Out[3]: 2 In [4]: np.nanargmin([10, np.nan, 100]) Out[4]: 0 Ralf From michael.cognacc at gmail.com Thu Mar 24 09:19:09 2011 From: michael.cognacc at gmail.com (Mic J) Date: Thu, 24 Mar 2011 14:19:09 +0100 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs In-Reply-To: References: Message-ID: > Thanks for all those instructions, however, personally I don't need them, I > have sucseeded with my own manipulations and even if I wouldn't I always can > build numpy/scipy from sources. > I mere care for quality and easibility of numpy installation for ordinary > non-skilled users. They may just try installation, see that it's buggy and > move away to use another soft. Rest assured that "non-ordinary" users like me also get irritated when we have to do extra work to get something to work/install when we want to try something out, its a waste of our time and resources if we have to do it often. Those problems might "always" be there in some degree, i have numpy installed on debian, fedora, and archlinux. Where there was no problems with the easy_install or package system install on those system (this time) But it will always happen at some time or another (as it did for me with install of other software via easy_install, rst2pdf and sphinx comes to mind) But that is a bug with the installation system not with the numpy package(you said 1.3.1 nupy had many bugs !?). A new user might move on to something other, but the reality is that problems like that crop up at some time, unless you have relatively static install. If the user moves on then he probably doesnt have a specific need for numpy, but just for something that can do the same job for his task. (it's unfortunate that he/we can't try it out without messing about). > D. mic From ralf.gommers at googlemail.com Thu Mar 24 09:28:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 14:28:29 +0100 Subject: [Numpy-discussion] [NumPy-Tickets] [NumPy] #971: numpy.memmap 'offset' parameter docs are almost entirely wrong. In-Reply-To: References: <017.ffa2a7653d4da49c365af4cde463d946@scipy.org> <026.3efff8747b994514f447acc84c4f4fdc@scipy.org> Message-ID: On Thu, Mar 24, 2011 at 12:38 PM, Sebastian Haase wrote: > Hi, > I got an "Internal Error" message when trying to log into the Trac page. That happens a lot, the solution is just to retry. > My comment: > Could there at least be a note added to the current memmap doc string, > stating that there is a problem with the current doc; also a link to > http://projects.scipy.org/numpy/ticket/971 > would be much better than nothing. If you could review the proposed replacement text (or even put it in a git branch) that would be very helpful. If the ticket submitter is correct and there is no bug, only documentation to be fixed, it would be good to get that in for 1.6.0 Ralf > On Thu, Mar 24, 2011 at 1:58 AM, ? wrote: >> #971: numpy.memmap 'offset' parameter docs are almost entirely wrong. >> ---------------------------+------------------------------------------------ >> ?Reporter: ?0ion9 ? ? ? ? ?| ? ? ? Owner: ?pv >> ? ? Type: ?defect ? ? ? ? | ? ? ?Status: ?new >> ?Priority: ?normal ? ? ? ? | ? Milestone: ?Unscheduled >> Component: ?Documentation ?| ? ? Version: ?devel >> ?Keywords: ? ? ? ? ? ? ? ? | >> ---------------------------+------------------------------------------------ >> Changes (by mwiebe): >> >> ?* milestone: ?=> Unscheduled >> >> >> -- >> Ticket URL: >> NumPy >> My example project >> _______________________________________________ >> NumPy-Tickets mailing list >> NumPy-Tickets at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-tickets >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Thu Mar 24 09:33:55 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 24 Mar 2011 14:33:55 +0100 Subject: [Numpy-discussion] how to delete a particular column or row from numpy array based on some rule or value In-Reply-To: References: <75C2FED246299A478280FA1470EDA4432D926A09@NL0230MBX06N1.DIR.slb.com> Message-ID: <9A45D350-D83E-4B57-96C5-C0865E533872@astro.physik.uni-goettingen.de> > In a numpy array of m x n size, I would like to delete few rows when > a given element in that row is ?nan? or say any other value. > > > For e.g. as in example given below, if I wish to delete a row when > the 3rd element of row is zero, or if 3rd, 4th, 5th element are zero > or either of 3rd, 4th and 5th element is zero. > > array([[ 1900. , nan, nan, nan, nan, nan], > > [ 1900.5, nan, nan, nan, nan, nan], > > [ 1901. , nan, nan, nan, nan, nan], > > ..., > > [ 6724. , nan, nan, nan, nan, nan], > > [ 6724.5, nan, nan, nan, nan, nan], > > [ 6725. , nan, nan, nan, nan, nan]]) > > Cheers > > > Sachin > > > > I think u can use numpy.loadtext command > http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html > > > -- > DILEEPKUMAR. R > J R F, IIT DELHI Wouldn't that require to transform the array to a text representation first? If you have an existing array, you can use the numpy test and logical functions to index it. E.g. myarr[:,2] == 0 selects the rows with 3rd column zero, np.isnan(myarr[:,3]) those with 4th column NaN and to select only rows where neither is the case: clean = myarr[np.logical_not( (myarr[:,2]==0) | (np.isnan(darr[:,3])) )] - combine as required with the '&' or '|' operators (there does not seem to be a shorthand for logical_not...), and check np.isnan, np.isinf, np.isfinite - those should allow you to select what you need. HTH, Derek From ralf.gommers at googlemail.com Thu Mar 24 09:45:18 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 14:45:18 +0100 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey : > > Hi > > 2011/3/24 Dmitrey >> >> >>> from numpy import inf, array >> >>> inf*0 >> nan >> >> (ok) >> >> >>> array(inf) * 0.0 >> StdErr: Warning: invalid value encountered in multiply >> nan >> >> My cycled calculations yields this thousands times slowing computations >> and making text output completely non-readable. > > Would old= seterr(invalid= 'ignore') be sufficient for you? > > yes for me, but I'm not sure for all those users who use my soft. Maybe it > will hide some bugs in their objective functions and nonlinear constraints > in numerical optimization and nonlinear equation systems. If we do what you request in your message subject your users will have the same issue. You can wrap (parts of) your code in something like: olderr = seterr(invalid= 'ignore') seterr(**olderr) Ralf From kwgoodman at gmail.com Thu Mar 24 09:51:22 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 24 Mar 2011 06:51:22 -0700 Subject: [Numpy-discussion] argmin and argmax without nan In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 6:19 AM, Ralf Gommers wrote: > 2011/3/24 Dmitrey : >> hi, >> is there any way to get argmin and argmax of an array w/o nans? >> Currently I have >>>>> from numpy import * >>>>> argmax([10,nan,100]) >> 1 >>>>> argmin([10,nan,100]) >> 1 >> But it's not the values I would like to get. >> >> The walkaround I use: get all indeces of nans, replace them by -inf, get >> argmax, replace them by inf, get argmin. >> Is there any better way? (BTW, I invoke argmin/argmax along of a chosen >> axis) >> D. > > In [3]: np.nanargmax([10, np.nan, 100]) > Out[3]: 2 > > In [4]: np.nanargmin([10, np.nan, 100]) > Out[4]: 0 And if speed is an issue (it usually isn't) you can use the nanargmax from Bottleneck: >> a = np.random.rand(10000) >> a[a > 0.5] = np.nan >> timeit np.nanargmax(a) 10000 loops, best of 3: 127 us per loop >> import bottleneck as bn >> timeit bn.nanargmax(a) 100000 loops, best of 3: 12.4 us per loop From tmp50 at ukr.net Thu Mar 24 09:59:22 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 15:59:22 +0200 Subject: [Numpy-discussion] argmin and argmax without nan In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey < tmp50 at ukr.net >: > hi, > is there any way to get argmin and argmax of an array w/o nans? > Currently I have >>>> from numpy import * >>>> argmax([10,nan,100]) > 1 >>>> argmin([10,nan,100]) > 1 > But it's not the values I would like to get. > > The walkaround I use: get all indeces of nans, replace them by -inf, get > argmax, replace them by inf, get argmin. > Is there any better way? (BTW, I invoke argmin/argmax along of a chosen > axis) > D. In [3]: np.nanargmax([10, np.nan, 100]) Out[3]: 2 In [4]: np.nanargmin([10, np.nan, 100]) Out[4]: 0 Ralf Thanks, I thought np.argnanmin should do that but those funcs were absent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Thu Mar 24 10:17:15 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Thu, 24 Mar 2011 10:17:15 -0400 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 9:45 AM, Ralf Gommers wrote: > 2011/3/24 Dmitrey : >> >> Hi >> >> 2011/3/24 Dmitrey >>> >>> >>> from numpy import inf, array >>> >>> inf*0 >>> nan >>> >>> (ok) >>> >>> >>> array(inf) * 0.0 >>> StdErr: Warning: invalid value encountered in multiply >>> nan >>> >>> My cycled calculations yields this thousands times slowing computations >>> and making text output completely non-readable. >> >> Would old= seterr(invalid= 'ignore') be sufficient for you? >> >> yes for me, but I'm not sure for all those users who use my soft. Maybe it >> will hide some bugs in their objective functions and nonlinear constraints >> in numerical optimization and nonlinear equation systems. > > If we do what you request in your message subject your users will have > the same issue. > > You can wrap (parts of) your code in something like: > ?olderr = seterr(invalid= 'ignore') > ? > ?seterr(**olderr) > Also, as Robert pointed out to me before np.errstate is a context-manager for ignoring these warnings. I often wrap optimization code with it. In [3]: np.array(np.inf)*0. Warning: invalid value encountered in multiply Out[3]: nan In [4]: with np.errstate(all='ignore'): ...: np.array(np.inf)*0. ...: ...: Out[4]: nan In [5]: np.array(np.inf)*0. Warning: invalid value encountered in multiply Out[5]: nan Skipper From e.antero.tammi at gmail.com Thu Mar 24 10:52:48 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 24 Mar 2011 16:52:48 +0200 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: Hi On Thu, Mar 24, 2011 at 4:17 PM, Skipper Seabold wrote: > On Thu, Mar 24, 2011 at 9:45 AM, Ralf Gommers > wrote: > > 2011/3/24 Dmitrey : > >> > >> Hi > >> > >> 2011/3/24 Dmitrey > >>> > >>> >>> from numpy import inf, array > >>> >>> inf*0 > >>> nan > >>> > >>> (ok) > >>> > >>> >>> array(inf) * 0.0 > >>> StdErr: Warning: invalid value encountered in multiply > >>> nan > >>> > >>> My cycled calculations yields this thousands times slowing computations > >>> and making text output completely non-readable. > >> > >> Would old= seterr(invalid= 'ignore') be sufficient for you? > >> > >> yes for me, but I'm not sure for all those users who use my soft. Maybe > it > >> will hide some bugs in their objective functions and nonlinear > constraints > >> in numerical optimization and nonlinear equation systems. > > > > If we do what you request in your message subject your users will have > > the same issue. > > > > You can wrap (parts of) your code in something like: > > olderr = seterr(invalid= 'ignore') > > > > seterr(**olderr) > > > > Also, as Robert pointed out to me before np.errstate is a > context-manager for ignoring these warnings. I often wrap optimization > code with it. > I didn't realize that its context-manager, but that's really create! (Perhaps documents could reflect that.) Regards, eat > > In [3]: np.array(np.inf)*0. > Warning: invalid value encountered in multiply > Out[3]: nan > > In [4]: with np.errstate(all='ignore'): > ...: np.array(np.inf)*0. > ...: > ...: > Out[4]: nan > > In [5]: np.array(np.inf)*0. > Warning: invalid value encountered in multiply > Out[5]: nan > > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Thu Mar 24 10:59:58 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Thu, 24 Mar 2011 10:59:58 -0400 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 10:52 AM, eat wrote: > Hi > > On Thu, Mar 24, 2011 at 4:17 PM, Skipper Seabold >> Also, as Robert pointed out to me before np.errstate is a >> context-manager for ignoring these warnings. I often wrap optimization >> code with it. > > I didn't realize that its?context-manager, but that's really?create! > (Perhaps documents could reflect that.) > Regards, > eat I added errstate to the see also of seterr: http://docs.scipy.org/numpy/docs/numpy.core.numeric.seterr/ Skipper From ralf.gommers at googlemail.com Thu Mar 24 11:49:53 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 16:49:53 +0100 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <4D8A2006.3090706@gmail.com> <4D8A2730.4080908@gmail.com> <4D8A4740.50305@gmail.com> Message-ID: On Wed, Mar 23, 2011 at 9:29 PM, Ralf Gommers wrote: > On Wed, Mar 23, 2011 at 8:32 PM, Pauli Virtanen wrote: >> On Wed, 23 Mar 2011 14:17:20 -0500, Bruce Southey wrote: >>> I can not figure out what is different between git and this version :-( >>> >>> All the paths appear to be the same. >>> Further it continues onwards when I do: $python3.1 -m pdb setup.py build >>> >>> I added these lines to the start of "numpy/compat/__init__.py" import os >>> print(os.getcwd()) >>> >>> /home/bsouthey/python/numpystuff/tmp6/numpy-1.6.0b1/build/py3k This is >>> equivalent to the git version. >>> >>> Unless someone else finds it, I will consider it just my problem and >>> ignore it. >> >> There's some magic in setup.py for generating numpy/version.py, which >> behaves differently depending whether `.git` is present or not. It might >> do some stuff that goes wrong --- I fixed it once already, but maybe it's >> broken again. > > I un-fixed your fix by accident, since I missed why you did that and > your fix broke the paver script. Leaving the FULLVERSION line outside > write_version_py() but putting the import inside should fix it, only > the commit part will be missing when building packages for an > unreleased version. It's a little ugly, but this seems to work for all use cases: https://github.com/rgommers/numpy/commit/a9fb1be2 Can someone please review that? Ralf From robert.kern at gmail.com Thu Mar 24 12:11:28 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Mar 2011 11:11:28 -0500 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey : >>>> from numpy import inf, array >>>> inf*0 > nan > > (ok) > >>>> array(inf) * 0.0 > StdErr: Warning: invalid value encountered in multiply > nan > > My cycled calculations yields this thousands times slowing computations and > making text output completely non-readable. > >>>> from numpy import __version__ >>>> __version__ > '2.0.0.dev-1fe8136' We really should change the default to 'warn' for numpy 2.0. Maybe even for numpy 1.6. We've talked about it before, and I think most people were in favor. We just never pulled the trigger. Devs, what say you? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Thu Mar 24 12:25:52 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 17:25:52 +0100 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 5:11 PM, Robert Kern wrote: > 2011/3/24 Dmitrey : >>>>> from numpy import inf, array >>>>> inf*0 >> nan >> >> (ok) >> >>>>> array(inf) * 0.0 >> StdErr: Warning: invalid value encountered in multiply >> nan >> >> My cycled calculations yields this thousands times slowing computations and >> making text output completely non-readable. >> >>>>> from numpy import __version__ >>>>> __version__ >> '2.0.0.dev-1fe8136' > > We really should change the default to 'warn' for numpy 2.0. Maybe > even for numpy 1.6. We've talked about it before, and I think most > people were in favor. We just never pulled the trigger. Old thread on this topic: http://thread.gmane.org/gmane.comp.python.numeric.general/35664 > > Devs, what say you? Works for me, also for 1.6. Cheers, Ralf > -- > Robert Kern From tmp50 at ukr.net Thu Mar 24 14:02:58 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 24 Mar 2011 20:02:58 +0200 Subject: [Numpy-discussion] argmin and argmax without nan In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 6:19 AM, Ralf Gommers < ralf.gommers at googlemail.com > wrote: > 2011/3/24 Dmitrey < tmp50 at ukr.net >: >> hi, >> is there any way to get argmin and argmax of an array w/o nans? >> Currently I have >>>>> from numpy import * >>>>> argmax([10,nan,100]) >> 1 >>>>> argmin([10,nan,100]) >> 1 >> But it's not the values I would like to get. >> >> The walkaround I use: get all indeces of nans, replace them by -inf, get >> argmax, replace them by inf, get argmin. >> Is there any better way? (BTW, I invoke argmin/argmax along of a chosen >> axis) >> D. > > In [3]: np.nanargmax([10, np.nan, 100]) > Out[3]: 2 > > In [4]: np.nanargmin([10, np.nan, 100]) > Out[4]: 0 And if speed is an issue (it usually isn't) you can use the nanargmax from Bottleneck: >> a = np.random.rand(10000) >> a[a > 0.5] = np.nan >> timeit np.nanargmax(a) 10000 loops, best of 3: 127 us per loop >> import bottleneck as bn >> timeit bn.nanargmax(a) 100000 loops, best of 3: 12.4 us per loop For some problems some % speedup could be yielded. Are there any plans for merging bottleneck into numpy? Also, are those benchmarks valid for ordinary numpy only or numpy with MKL/ACML or it doesn't matter? If I have huge arrays and multicore CPU, will numpy with MKL/ACML or something else involve parallel computations with numpy funcs like amin, amax, argmin, nanargmin etc? D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 24 14:12:55 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Mar 2011 13:12:55 -0500 Subject: [Numpy-discussion] argmin and argmax without nan In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey : > Are there any plans for merging?bottleneck into numpy? No plans, but no particular opposition. bottleneck is a good place to experiment with these optimizations. When they settle, it might be worth folding them back in. > Also, are those benchmarks valid for ordinary numpy only or numpy with > MKL/ACML or it doesn't matter? Doesn't matter. > If I have huge arrays and multicore CPU, will numpy with MKL/ACML or > something else involve parallel computations with numpy funcs like amin, > amax, argmin, nanargmin etc? No. MKL/ACML focus on parallelizing expensive computation like matrix-matrix multiplication, not things like finding the minimum elements of an array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From nadavh at visionsense.com Thu Mar 24 15:06:56 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 24 Mar 2011 12:06:56 -0700 Subject: [Numpy-discussion] Partial least squares Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> I am looking for a partial least sqaures code refactoring for two (X,Y) matrices. I found the following, but they not not work for me: 1. MDP: Factors only one matrix (am I wrong?) 2. pychem: Windows only code (I use Linux) 3. chemometrics from Koders: I get a singularity error. 4. pca_module (By Risvik): same problem as MDP Any suggestion? Nadav. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Thu Mar 24 15:15:12 2011 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Thu, 24 Mar 2011 20:15:12 +0100 Subject: [Numpy-discussion] Partial least squares In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> Message-ID: 2011/3/24 Nadav Horesh : > I am looking for a partial least sqaures code refactoring for two (X,Y) > matrices. I found the following, but they not not work for me: > 1. MDP: Factors only one matrix (am I wrong?) > 2. pychem: Windows only code (I use Linux) > 3. chemometrics from Koders: I get a singularity error. > 4. pca_module (By Risvik): same problem as MDP > > Any suggestion? There is one in the master of scikits.learn: https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/pls.py https://github.com/scikit-learn/scikit-learn/blob/master/examples/plot_pls.py -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From ralf.gommers at googlemail.com Thu Mar 24 15:47:24 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 24 Mar 2011 20:47:24 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 24, 2011 at 12:34 AM, Derek Homeier wrote: > Hi again, > > On 23 Mar 2011, at 17:07, Ralf Gommers wrote: > >> I am pleased to announce the availability of the first beta of NumPy >> 1.6.0. Due to the extensive changes in the Numpy core for this >> release, the beta testing phase will last at least one month. Please >> test this beta and report any problems on the Numpy mailing list. > > tests with the fink-installed pythons on MacOS X mostly succeeded, > with one failure in python2.4 and a couple of issues seemingly > related to PPC floating point accuracy, as below: > > OSX/Python ? ? ?2.4 ? ? ? ? ? ? ? ? ? ? 2.5 ? ? ? ? ? ? 2.6 ? ? ? ? ? ? 2.7 ? ? ? ? ? ? 3.1 ? ? ? ? ? ? 3.2 > 10.5/i386 ? ? ? FAIL[1] ? ? ? ? ? ? ? ? pass ? ?pass ? ?pass ? ?pass ? ?pass > 10.5/ppc ? ? ? ?FAIL[1,2] ? ? ? FAIL[2] FAIL[2] FAIL[2] FAIL[2] FAIL[2] > 10.6/x86_64 ? ? n/a ? ? ? ? ? ? pass ? ?pass ? ?pass ? ?pass ? ?pass > > Failures: > [1] > Python 2.4.4 (#1, Jan ?5 2011, 03:05:41) > [GCC 4.0.1 (Apple Inc. build 5493)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > ?>>> import numpy > ?>>> numpy.test('full') > Running unit tests for numpy > NumPy version 1.6.0b1 > NumPy is installed in /sw/lib/python2.4/site-packages/numpy > Python version 2.4.4 (#1, Jan ?5 2011, 03:05:41) [GCC 4.0.1 (Apple > Inc. build 5493)] > nose version 1.0.0 > ........ > FAIL: test_iterator.test_iter_broadcasting_errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > ? File "/sw/lib/python2.4/site-packages/nose/case.py", line 187, in > runTest > ? ? self.test(*self.arg) > ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/ > test_iterator.py", line 639, in test_iter_broadcasting_errors > ? ? 'Message "%s" doesn\'t contain operand shape (2,3)' % msg) > ? File "/sw/lib/python2.4/site-packages/numpy/testing/utils.py", line > 34, in assert_ > ? ? raise AssertionError(msg) > AssertionError: Message "non-broadcastable output operand with shape > (%zd,%zd) doesn't match the broadcast shape (%zd,%zd,%zd)" doesn't > contain operand shape (2,3) Mark just committed a fix for this, could you test the current 1.6.x branch to see if that worked and report on http://projects.scipy.org/numpy/ticket/1780? Thanks, Ralf From samtygier at yahoo.co.uk Thu Mar 24 15:50:07 2011 From: samtygier at yahoo.co.uk (sam tygier) Date: Thu, 24 Mar 2011 19:50:07 +0000 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs In-Reply-To: References: Message-ID: <4D8BA06F.2050107@yahoo.co.uk> On 24/03/11 10:00, Dmitrey wrote: > hi, > when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs > > I have Linux KUBUNTU 10.10 > > D. (k)ubuntu 11.04 (to be released in april) will have at numpy 1.5.1 (or possible newer) https://launchpad.net/ubuntu/+source/python-numpy also scipy will go from 0.7.2 -> 0.8.0 and python itself from 2.6.6 -> 2.7.1 sam From gael.varoquaux at normalesup.org Thu Mar 24 16:03:38 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 24 Mar 2011 21:03:38 +0100 Subject: [Numpy-discussion] Partial least squares In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> Message-ID: <20110324200338.GA5594@phare.normalesup.org> On Thu, Mar 24, 2011 at 12:06:56PM -0700, Nadav Horesh wrote: > I am looking for a partial least sqaures code refactoring for two (X,Y) > matrices. I found the following, but they not not work for me: We have had a PLS code contributed in the scikits learn very recently: http://scikit-learn.sourceforge.net/dev/modules/pls.html (note that these are the docs for the unreleased development version). You will need to check out and install the development version: http://scikit-learn.sourceforge.net/dev/developers/index.html#retrieving-the-latest-code The docs are very light (bad, bad developers :$ ), and as I am not myself a user of PLS I have a hard time judging the quality of the implementation and its usability, but hopefully it should do what you are looking for. An example can be found on http://scikit-learn.sourceforge.net/dev/auto_examples/plot_pls.html HTH, Ga?l From gael.varoquaux at normalesup.org Thu Mar 24 16:04:17 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 24 Mar 2011 21:04:17 +0100 Subject: [Numpy-discussion] Partial least squares In-Reply-To: References: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> Message-ID: <20110324200416.GB5594@phare.normalesup.org> On Thu, Mar 24, 2011 at 08:15:12PM +0100, Olivier Grisel wrote: > 2011/3/24 Nadav Horesh : > > I am looking for a partial least sqaures code refactoring for two (X,Y) > > matrices. I found the following, but they not not work for me: > > 1. MDP: Factors only one matrix (am I wrong?) > > 2. pychem: Windows only code (I use Linux) > > 3. chemometrics from Koders: I get a singularity error. > > 4. pca_module (By Risvik): same problem as MDP > > Any suggestion? > There is one in the master of scikits.learn: > https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/pls.py > https://github.com/scikit-learn/scikit-learn/blob/master/examples/plot_pls.py Olivier shoots faster than I do :) G From nadavh at visionsense.com Thu Mar 24 16:57:46 2011 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 24 Mar 2011 13:57:46 -0700 Subject: [Numpy-discussion] Partial least squares In-Reply-To: <20110324200416.GB5594@phare.normalesup.org> References: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> , <20110324200416.GB5594@phare.normalesup.org> Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D525E@VA3DIAXVS361.RED001.local> Yes, he is. I'll work on it early next week, and post any comment I'll have. Documentation quality: The code doc refer to an excellent reference [Wegelin et al. 2000], so no real problem here. If the reference is critical I would suggest on of the following: 1. Put a link to the document. 2. Is it possible to copy the paper to the project site? As the paper becomes old (on the internet time scale), it becomes volatile, and may vanish while the project is still alive. Nadav. ________________________________________ From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Gael Varoquaux [gael.varoquaux at normalesup.org] Sent: 24 March 2011 22:04 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Partial least squares On Thu, Mar 24, 2011 at 08:15:12PM +0100, Olivier Grisel wrote: > 2011/3/24 Nadav Horesh : > > I am looking for a partial least sqaures code refactoring for two (X,Y) > > matrices. I found the following, but they not not work for me: > > 1. MDP: Factors only one matrix (am I wrong?) > > 2. pychem: Windows only code (I use Linux) > > 3. chemometrics from Koders: I get a singularity error. > > 4. pca_module (By Risvik): same problem as MDP > > Any suggestion? > There is one in the master of scikits.learn: > https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/pls.py > https://github.com/scikit-learn/scikit-learn/blob/master/examples/plot_pls.py Olivier shoots faster than I do :) G _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From thouis at gmail.com Thu Mar 24 17:40:45 2011 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Thu, 24 Mar 2011 22:40:45 +0100 Subject: [Numpy-discussion] Question about repeated values in X in A[X] = Y Message-ID: I have done some work on scipy.ndimage.measurements, and was adding to it recently when I noticed I had made use of the idiom A[X] = Y with repeated indices in X, relying on the behavior that the last X,Y pair at each unique X is the one that is assigned in A (last = last by position in X). I thought I had found documentation of this behavior, somewhere, but when another developer asked, I was unable to locate it (perhaps I just forgot the google keywords I used last time). So, before I rely too much on this behavior, I thought I should make sure it was: - correct, - reasonable future-proof, - and documented (if the first two are true). Thanks, Thouis Jones From gael.varoquaux at normalesup.org Thu Mar 24 18:32:27 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 24 Mar 2011 23:32:27 +0100 Subject: [Numpy-discussion] Partial least squares In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D525E@VA3DIAXVS361.RED001.local> References: <26FC23E7C398A64083C980D16001012D1CE03D525A@VA3DIAXVS361.RED001.local> <20110324200416.GB5594@phare.normalesup.org> <26FC23E7C398A64083C980D16001012D1CE03D525E@VA3DIAXVS361.RED001.local> Message-ID: <20110324223227.GG5594@phare.normalesup.org> On Thu, Mar 24, 2011 at 01:57:46PM -0700, Nadav Horesh wrote: > The code doc refer to an excellent reference [Wegelin et al. 2000], so > no real problem here. If the reference is critical I would suggest on > of the following: > 1. Put a link to the document. Do you have one? I'd be happy to add one. > 2. Is it possible to copy the paper to the project site? As the paper > becomes old (on the internet time scale), it becomes volatile, and may > vanish while the project is still alive. I don't really want to do this: it is not the scikit's learn goal to maintain a library of documents. However, if you want to give us an executive summary of the document (respecting the copyright, of course, so copy and paste should be avoided). I'd be very happy to integrate it to the documentation. Once again, I am not doing this myself right now because I am not an expert of PLS, and I don't trust myself to do it right. I would have to understand the usecases a bit more, which I feasible, but takes time. Your feedback is much appreciated, Ga?l From derek at astro.physik.uni-goettingen.de Thu Mar 24 19:44:09 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 25 Mar 2011 00:44:09 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> Message-ID: <087D6977-AD04-40F0-83DE-4A9639AC9BCE@astro.physik.uni-goettingen.de> On 24.03.2011, at 9:11AM, Pearu Peterson wrote: > > Intel-64bit: > ERROR: test_assumed_shape.TestAssumedShapeSumExample.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python3.2/site-packages/nose/case.py", line 372, in setUp > try_run(self.inst, ('setup', 'setUp')) > File "/sw/lib/python3.2/site-packages/nose/util.py", line 478, in try_run > return func() > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 352, in setUp > module_name=self.module_name) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper > memo[key] = func(*a, **kw) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line 134, in build_module > % (cmd[4:], asstr(out))) > RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90', '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/precision.f90'] > Reading .f2py_f2cmap ... > Mapping "real(kind=rk)" to "double" > > Hmm, this should not happen as real(kind=rk) should be mapped to "float". > It seems that you have .f2py_f2cmap file lying around. Could you remove it and try again. Yes, it's in the tarball and was installed together with the f2py tests! tar tzvf /sw/src/numpy-1.6.0b1.tar.gz | grep f2py_f2 -rw-r--r-- zouzoujing/staff 29 2011-03-15 06:22 numpy-1.6.0b1/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap cat /sw/lib/python3.2/site-packages/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap dict(real=dict(rk="double")) But still after removing all installed instances of the file and even removing it from the tarball prior to rebuilding numpy, the test apparently keeps re-creating it in its own tmpdir: building extension "_test_ext_module_5403" sources f2py options: [] f2py:> /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403module.c creating /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum creating /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7 getctype: "real(kind=rk)" is mapped to C "float" (to override define dict(real = dict(rk="")) in /private/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpPo744G/.f2py_f2cmap file). ... gfortran:f77: /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f:26.21: real :: res 1 Error: Symbol 'res' at (1) already has basic type of REAL /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f:26.21: real :: res 1 Error: Symbol 'res' at (1) already has basic type of REAL ... f2py is the one installed with this numpy version, gfortran is COLLECT_GCC=/sw/bin/gfortran COLLECT_LTO_WRAPPER=/sw/lib/gcc4.5/libexec/gcc/x86_64-apple-darwin10.6.0/4.5.2/lto-wrapper Ziel: x86_64-apple-darwin10.6.0 Konfiguriert mit: ../gcc-4.5.2/configure --prefix=/sw --prefix=/sw/lib/gcc4.5 --mandir=/sw/share/man --infodir=/sw/lib/gcc4.5/info --enable-languages=c,c++,fortran,objc,obj-c++,java --with-gmp=/sw --with-libiconv-prefix=/sw --with-ppl=/sw --with-cloog=/sw --with-mpc=/sw --with-system-zlib --x-includes=/usr/X11R6/include --x-libraries=/usr/X11R6/lib --program-suffix=-fsf-4.5 --enable-lto Thread-Modell: posix gcc-Version 4.5.2 (GCC) Any more ideas? Thanks, Derek From david at silveregg.co.jp Thu Mar 24 23:22:07 2011 From: david at silveregg.co.jp (David) Date: Fri, 25 Mar 2011 12:22:07 +0900 Subject: [Numpy-discussion] Ticket closing policy Message-ID: <4D8C0A5F.5010501@silveregg.co.jp> Hi Mark, hi all, I noticed you did a lot of cleaning in the bug trackers, thank you for helping there, this is sorely needed. However, I noticed quite a few tickets were closed as wontfix even though they are valid. I understand the concern of getting many languishing tickets, but one should not close valid tickets either. Also, in my experience, issues specific to win32 should not be closed because they work on Linux - I remember the fastputmask issue was still there not so long ago (but could not understand what was going on, unfortunately). cheers, David From pearu.peterson at gmail.com Fri Mar 25 00:58:30 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 25 Mar 2011 06:58:30 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: <087D6977-AD04-40F0-83DE-4A9639AC9BCE@astro.physik.uni-goettingen.de> References: <579879E3-94E0-4400-8CE1-62D6FD843362@astro.physik.uni-goettingen.de> <249E13F2-7EDE-4A98-8A6E-6D4A9C5374B7@astro.physik.uni-goettingen.de> <087D6977-AD04-40F0-83DE-4A9639AC9BCE@astro.physik.uni-goettingen.de> Message-ID: On Fri, Mar 25, 2011 at 1:44 AM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > On 24.03.2011, at 9:11AM, Pearu Peterson wrote: > > > > Intel-64bit: > > ERROR: test_assumed_shape.TestAssumedShapeSumExample.test_all > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python3.2/site-packages/nose/case.py", line 372, in setUp > > try_run(self.inst, ('setup', 'setUp')) > > File "/sw/lib/python3.2/site-packages/nose/util.py", line 478, in > try_run > > return func() > > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line > 352, in setUp > > module_name=self.module_name) > > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line > 73, in wrapper > > memo[key] = func(*a, **kw) > > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/util.py", line > 134, in build_module > > % (cmd[4:], asstr(out))) > > RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_free.f90', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/foo_use.f90', > '/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpfiy1jn/precision.f90'] > > Reading .f2py_f2cmap ... > > Mapping "real(kind=rk)" to "double" > > > > Hmm, this should not happen as real(kind=rk) should be mapped to "float". > > It seems that you have .f2py_f2cmap file lying around. Could you remove > it and try again. > > Yes, it's in the tarball and was installed together with the f2py tests! > Indeed, f2py tests suite contains the .f2py_f2cmap file. Its effect should be local to the corresponding test but it seems not. I'll look into it.. > building extension "_test_ext_module_5403" sources > f2py options: [] > f2py:> > /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403module.c > creating /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum > creating > /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7 > getctype: "real(kind=rk)" is mapped to C "float" (to override define > dict(real = dict(rk="")) in > /private/var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpPo744G/.f2py_f2cmap > file). > ... > gfortran:f77: > /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f > > /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f:26.21: > > real :: res > 1 > Error: Symbol 'res' at (1) already has basic type of REAL > > /var/folders/DC/DC7g9UNr2RWkb++8ZSn1J+++0Dk/-Tmp-/tmpS4KGum/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers.f:26.21: > > real :: res > 1 > Error: Symbol 'res' at (1) already has basic type of REAL > ... > > f2py is the one installed with this numpy version, gfortran is > COLLECT_GCC=/sw/bin/gfortran > > COLLECT_LTO_WRAPPER=/sw/lib/gcc4.5/libexec/gcc/x86_64-apple-darwin10.6.0/4.5.2/lto-wrapper > Ziel: x86_64-apple-darwin10.6.0 > Konfiguriert mit: ../gcc-4.5.2/configure --prefix=/sw > --prefix=/sw/lib/gcc4.5 --mandir=/sw/share/man --infodir=/sw/lib/gcc4.5/info > --enable-languages=c,c++,fortran,objc,obj-c++,java --with-gmp=/sw > --with-libiconv-prefix=/sw --with-ppl=/sw --with-cloog=/sw --with-mpc=/sw > --with-system-zlib --x-includes=/usr/X11R6/include > --x-libraries=/usr/X11R6/lib --program-suffix=-fsf-4.5 --enable-lto > Thread-Modell: posix > gcc-Version 4.5.2 (GCC) > > Can you send me the _test_ext_module_5403-f2pywrappers.f file. It should exist there when the compilation fails. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Fri Mar 25 03:08:45 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 25 Mar 2011 00:08:45 -0700 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error In-Reply-To: References: Message-ID: Okay, so here's a follow up on my progress. Apologies in advance for the long email here, but I'd like to be thorough about this before I forget. For the sake of completeness, here's my setup. I'm running python 2.7.1 compiled from source with icc. I'm running ubuntu 10.10 on one of intel's new processors (a i7-2600). The goal is to compile numpy and scipy both with intel's compiler and intel's mkl. I finally got numpy to compile with icc / ifort with pretty much all of the tests passing. It's a bit of work, partly cause I was trying to be an optimization junky, but I thought I'd share my discoveries. Scipy also compiles, but with some errors (which are likely due to me not configuring f2py correctly). First, I wanted to compile things with intel's interprocedural optimization enabled, and that seems to work, but only if -O2 is used for the compiling stage and -O1 is used for the linking stage. If -O3 is given for the compiling stage, then the einsum test goes into some sort of infinite loop and hangs. If -O2 or -O3 are given for the linker, then there are random other segfaults (I forget where). However, with these optimization levels, things are stable. Also, if I turn off -ipo, then -O3 works fine for compiling. I'm not sure if this reflects bugs in the flags I'm passing to the intel compiler or in icc/ifort itself. Second, to use -ipo, it's critical that xiar is used instead of ar to create object archives. This needed to be changed in fcompiler/intel.py and intelccompiler.py. I've attached a diff of these files that gives working options for me. I don't know if these options are set in the correct place or not, but perhaps they would be helpful: The essence of it is the following (from intelccompiler.py) linker_flags = '-O1 -ipo -openmp -lpthread -fno-alias -xHOST -fPIC ' compiler_opt_flags = '-static -ipo -xHOST -O2 -fPIC -DMKL_LP64 -mkl -wd188 -g -fno-alias ' icc_run_string = 'icc ' + compiler_opt_flags icpc_run_string = 'icpc ' + compiler_opt_flags linker_run_string = 'icc ' + linker_flags + ' -shared ' with the rest of this diff setting these options. In this case, the -openmp and -lpthread are required for linking with the threaded layer of the MKL. This could possibly be ripped out of there. Also, the -fno-alias is critical for the c compiler -- random segfaults and memory corruptions occur without it. The -DMKL_LP64 is to ensure proper linking with the lp64 (32 bit indices) part of mkl, instead of the ilp64 (64 bit indices). The latter isn't supported by the lapack_lite module -- things compile, but don't work. -mkl may or may not help things. For the fortran side, this was the compiler string: compiler_opt_flags = '-static -ipo -xHOST -fPIC -DMKL_LP64 -mkl -wd188 -g -fno-alias -O3' Here you don't need the -fno-alias and -O3 seems to work. Third, it was a bit of a pain to figure out how to get the linking/detection done correctly, as somehow order matters, and it was easy to get undefined symbols, runtime errors, etc. Very annoying. In the end, my site.cfg file looked like this: [DEFAULT] library_dirs=/usr/intel/current/mkl/lib/intel64 include_dirs=/usr/intel/current/mkl/include mkl_libs = mkl_rt, mkl_core, mkl_intel_thread, mkl_intel_lp64 blas_libs = mkl_blas95_lp64 lapack_libs = mkl_lapack95_lp64 [lapack_opt] library_dirs=/usr/intel/current/mkl/lib/intel64 include_dirs=/usr/intel/current/mkl/include/intel64/lp64 libraries = mkl_lapack95_lp64 [blas_opt] library_dirs = /usr/intel/current/mkl/lib/intel64 include_dirs = /usr/intel/current/mkl/include/intel64/lp64 libraries = mkl_blas95_lp64 where /usr/intel/current/ points to my intel install location. It's critical that the mkl_libs are given in that order. I didn't find another combination that worked. Finally, I attached my bash setup script for environment variables. I don't know how much of a role those play in things, but I had them in place when things started working, so I should put them here. Now, on to scipy. With all these options in place, scipy compiles fine. However, there are two problems, and these don't seem to go away at any optimization level. I'm looking for suggestions. I'm guessing it's some sort of configuration error. 1) The CloughTocher2DInterpolator segfaults every time it's called to interpret values. I couldn't manage to track it down -- it's in the cython code somewhere -- but I can give more details next time, I disabled it for now. 2) f2py isn't getting the interfaces right. When I run the test suite, I get about 250 errors, all of the form: ValueError: failed to create intent(cache|hide)|optional array-- must have defined dimensions but got (5,5,) and so on, with different tuples on the end. Other than these errors, everything seemed to work great. What might I be doing wrong there? Thanks! -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: setupenv.sh Type: application/x-sh Size: 1706 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: intel_compiler_stuff.diff Type: text/x-patch Size: 5033 bytes Desc: not available URL: From ralf.gommers at googlemail.com Fri Mar 25 05:56:08 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 25 Mar 2011 10:56:08 +0100 Subject: [Numpy-discussion] Ticket closing policy In-Reply-To: <4D8C0A5F.5010501@silveregg.co.jp> References: <4D8C0A5F.5010501@silveregg.co.jp> Message-ID: On Fri, Mar 25, 2011 at 4:22 AM, David wrote: > Hi Mark, hi all, > > I noticed you did a lot of cleaning in the bug trackers, thank you for > helping there, this is sorely needed. > > However, I noticed quite a few tickets were closed as wontfix even > though they are valid. I checked all the wontfix tickets and reopened two, but I agreed with the others. Never closing tickets if they are valid (even though it's clear they will not be resolved anymore) is not a useful policy either IMHO. For example, the bug report of a build issue of numpy 1.0.1 on IRIX could still be valid, but no more info or help on this ticket for 3 years. So close as wontfix. If a new IRIX user comes along and this is still an issue, it can be reopened or a new ticket can be opened. Cheers, Ralf > I understand the concern of getting many > languishing tickets, but one should not close valid tickets either. > Also, in my experience, issues specific to win32 should not be closed > because they work on Linux - I remember the fastputmask issue was still > there not so long ago (but could not understand what was going on, > unfortunately). > > cheers, > > David From cournape at gmail.com Fri Mar 25 06:36:47 2011 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Mar 2011 19:36:47 +0900 Subject: [Numpy-discussion] when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs In-Reply-To: References: Message-ID: 2011/3/24 Dmitrey : > hi, > when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs We do not decide what Ubuntu and/or debian are putting into their release. You should send bug report to those distributions bug trackers if you want to see things improve for those distributions. cheers, David From dileepkunjaai at gmail.com Fri Mar 25 07:58:50 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Fri, 25 Mar 2011 17:28:50 +0530 Subject: [Numpy-discussion] help me Message-ID: Dear sir, I have a numpy array,,. in which i have to choose only different values only, For example Let A= [ 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25.] then the answer is [20, 21. 22. 23. 24. 25.] ie choose only different values Please help me. -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.deval at gmail.com Fri Mar 25 08:02:32 2011 From: miguel.deval at gmail.com (Miguel de Val-Borro) Date: Fri, 25 Mar 2011 13:02:32 +0100 Subject: [Numpy-discussion] help me In-Reply-To: References: Message-ID: <20110325120231.GB21956@poincare> On Fri, Mar 25, 2011 at 05:28:50PM +0530, dileep kunjaai wrote: > Dear sir, > I have a numpy array,,. in which i have to choose only different values > only, > For example > > Let A= [ 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. > 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. > 23. 24. 25.] > then > the answer is [20, 21. 22. 23. 24. 25.] ie choose only different values Try the numpy.unique function: >>> numpy.unique(A) array([20, 21. 22. 23. 24. 25.]) Miguel From dileepkunjaai at gmail.com Fri Mar 25 08:08:06 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Fri, 25 Mar 2011 17:38:06 +0530 Subject: [Numpy-discussion] help me In-Reply-To: <20110325120231.GB21956@poincare> References: <20110325120231.GB21956@poincare> Message-ID: Thank you sir,,,,,,, Thank you very much..... On Fri, Mar 25, 2011 at 5:32 PM, Miguel de Val-Borro wrote: > On Fri, Mar 25, 2011 at 05:28:50PM +0530, dileep kunjaai wrote: > > Dear sir, > > I have a numpy array,,. in which i have to choose only different values > > only, > > For example > > > > Let A= [ 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. > > 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. > > 23. 24. 25.] > > then > > the answer is [20, 21. 22. 23. 24. 25.] ie choose only different > values > > Try the numpy.unique function: > > >>> numpy.unique(A) > array([20, 21. 22. 23. 24. 25.]) > > Miguel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Mar 25 08:23:11 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 25 Mar 2011 13:23:11 +0100 Subject: [Numpy-discussion] So... Message-ID: <4D8C892F.6000207@molden.no> FFTs should keep the GIL locked, because sharing the interpreter is not funny. Array indices should be sizeof(void*), because following C standard and Python C API is lame. Median should be calculated in O(n log n) instead of O(n) time, because redundant sorting is gymnatics for the CPU. But at least something happened to the tickets, after a year or two :-) Sturla From pav at iki.fi Fri Mar 25 08:33:33 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 25 Mar 2011 12:33:33 +0000 (UTC) Subject: [Numpy-discussion] So... References: <4D8C892F.6000207@molden.no> Message-ID: Fri, 25 Mar 2011 13:23:11 +0100, Sturla Molden wrote: > FFTs should keep the GIL locked, because sharing the interpreter is not > funny. > Array indices should be sizeof(void*), because following C standard and > Python C API is lame. > Median should be calculated in O(n log n) instead of O(n) time, because > redundant sorting is gymnatics for the CPU. That the ticket was changed "unscheduled" does not mean it will not be implemented when someone finds the extra time for it. That npy_intp will not be redefined as ssize_t does not mean that the type of array indices could not be changed. From sturla at molden.no Fri Mar 25 08:40:54 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 25 Mar 2011 13:40:54 +0100 Subject: [Numpy-discussion] So... In-Reply-To: References: <4D8C892F.6000207@molden.no> Message-ID: <4D8C8D56.6070407@molden.no> Den 25.03.2011 13:33, skrev Pauli Virtanen: > That npy_intp will not be redefined as ssize_t does not mean that the > type of array indices could not be changed. By the way, what is the resonable array index for AMD64, where 32-bit is the native offset size but size_t is 64 bit? Sturla From pav at iki.fi Fri Mar 25 08:50:57 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 25 Mar 2011 12:50:57 +0000 (UTC) Subject: [Numpy-discussion] So... References: <4D8C892F.6000207@molden.no> <4D8C8D56.6070407@molden.no> Message-ID: Fri, 25 Mar 2011 13:40:54 +0100, Sturla Molden wrote: > Den 25.03.2011 13:33, skrev Pauli Virtanen: >> That npy_intp will not be redefined as ssize_t does not mean that the >> type of array indices could not be changed. > > By the way, what is the resonable array index for AMD64, where 32-bit is > the native offset size but size_t is 64 bit? Well, whatever the address space is. Since 64-bit programs on amd64 have in principle 64 bits of address space, size_t should be OK. Pauli From charlesr.harris at gmail.com Fri Mar 25 11:06:49 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Mar 2011 09:06:49 -0600 Subject: [Numpy-discussion] loadtxt/savetxt tickets Message-ID: Hi All, Could someone with an interest in loadtxt/savetxt look through the associated tickets? A search on the tickets using either of those keys will return fairly lengthy lists. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Mar 25 12:10:07 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 25 Mar 2011 17:10:07 +0100 Subject: [Numpy-discussion] HP-UX users, please test distutils changes Message-ID: Hi, Here are some updated compile flags for HP-UX that were submitted on the bug tracker. It would be nice to have someone confirm this works: https://github.com/rgommers/numpy/tree/hp Thanks, Ralf From wickedgrey at gmail.com Fri Mar 25 13:00:24 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Fri, 25 Mar 2011 10:00:24 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? Message-ID: I'm working on getting support for 16-bit floats into cython and have run into what seems like a numpy bug (I'm a new contributor to both projects, so bear with me ;). https://github.com/wickedgrey/numpy/commit/29f9f1b709cc2c346b8514859c58a761df80f031 Adding NPY_HALF to the switch statement on line 350 of numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it needs to be spelled NPY_FLOAT16 or something similar), but I have no idea how to test the change. Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request? Thanks, Eli From pav at iki.fi Fri Mar 25 13:21:40 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 25 Mar 2011 17:21:40 +0000 (UTC) Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? References: Message-ID: Fri, 25 Mar 2011 10:00:24 -0700, Eli Stevens (Gmail) wrote: > I'm working on getting support for 16-bit floats into cython and have > run into what seems like a numpy bug (I'm a new contributor to both > projects, so bear with me ;). > > https://github.com/wickedgrey/numpy/ commit/29f9f1b709cc2c346b8514859c58a761df80f031 > > Adding NPY_HALF to the switch statement on line 350 of > numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it > needs to be spelled NPY_FLOAT16 or something similar), but I have no > idea how to test the change. The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec. What can be done instead is exporting the half-float items (and any other non-3118 types) as plain bytes. I think this is the way to go. On the Cython side, you'd need to detect when you are working with Numpy arrays, and get the half-float type information from the Numpy dtype rather than from the exported buffer. -- Pauli Virtanen From mwwiebe at gmail.com Fri Mar 25 13:28:08 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 25 Mar 2011 10:28:08 -0700 Subject: [Numpy-discussion] Ticket closing policy In-Reply-To: <4D8C0A5F.5010501@silveregg.co.jp> References: <4D8C0A5F.5010501@silveregg.co.jp> Message-ID: On Thu, Mar 24, 2011 at 8:22 PM, David wrote: > Hi Mark, hi all, > > I noticed you did a lot of cleaning in the bug trackers, thank you for > helping there, this is sorely needed. > > However, I noticed quite a few tickets were closed as wontfix even > though they are valid. I understand the concern of getting many > languishing tickets, but one should not close valid tickets either. > Also, in my experience, issues specific to win32 should not be closed > because they work on Linux - I remember the fastputmask issue was still > there not so long ago (but could not understand what was going on, > unfortunately). > I understand your concern, but I think since dealing with bug reports requires subjective judgement calls, it's going to be a bit messy. That's why the bug tracker timeline is there to let people see what changes have been made and peer review those choices. In the fastputmask case, it was about the test suite crashing on win32 as well as a memory leak on Linux. I commented about testing the latter, but could have also added a justification for the former that no win32 test suite crash reports were coming in during the beta testing. Regardless, please add to the discussion on any bugs I've modified where you believe I've erred, this is a collaborative effort! Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Mar 25 13:35:30 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 25 Mar 2011 10:35:30 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen wrote: > Fri, 25 Mar 2011 10:00:24 -0700, Eli Stevens (Gmail) wrote: > > I'm working on getting support for 16-bit floats into cython and have > > run into what seems like a numpy bug (I'm a new contributor to both > > projects, so bear with me ;). > > > > https://github.com/wickedgrey/numpy/ > commit/29f9f1b709cc2c346b8514859c58a761df80f031 > > > > Adding NPY_HALF to the switch statement on line 350 of > > numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it > > needs to be spelled NPY_FLOAT16 or something similar), but I have no > > idea how to test the change. > > The buffer interface cannot be used to export the half-float types, since > the type is not specified in PEP 3118. Numpy cannot unilaterally add > nonstandard format codes to the spec. > That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From wickedgrey at gmail.com Fri Mar 25 14:12:10 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Fri, 25 Mar 2011 11:12:10 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen wrote: > The buffer interface cannot be used to export the half-float types, since > the type is not specified in PEP 3118. Numpy cannot unilaterally add > nonstandard format codes to the spec. > > What can be done instead is exporting the half-float items (and any other > non-3118 types) as plain bytes. I think this is the way to go. What about as uint16s? Won't plain bytes have endian issues? case NPY_HALF: if (_append_char(str, 'H')) return -1; break; Something like that? I am also still curious about where unit tests for things like this should go. > On the Cython side, you'd need to detect when you are working with Numpy > arrays, and get the half-float type information from the Numpy dtype > rather than from the exported buffer. Hrm, okay. I'll have to follow up with the Cython folks about that. I hope that my fears about that ending up being a sweeping change are unfounded (having already gone through and made the changes to unilaterally add support for 'e' buffer types, it seems like it's baked in; we'll see). :-/ Thanks, Eli From wickedgrey at gmail.com Fri Mar 25 14:14:42 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Fri, 25 Mar 2011 11:14:42 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: > That said, I think starting a discussion with the Python core developers > about the float16 type is worthwhile. There might be interest in supporting > the float16 type in the struct interface, something that would be required > as part of amending PEP 3118. That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes. Eli From mwwiebe at gmail.com Fri Mar 25 14:28:31 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 25 Mar 2011 11:28:31 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) wrote: > On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: > > That said, I think starting a discussion with the Python core developers > > about the float16 type is worthwhile. There might be interest in > supporting > > the float16 type in the struct interface, something that would be > required > > as part of amending PEP 3118. > > That's something that wouldn't see production until Python 3.3, right? > I'm imagining that the buffer spec in use wouldn't change during the > 2.7.x series, unless it's unusually accepting of changes. That's true, but explaining the hoops being jumped through to support a new data type across multiple Python plugins could help evolve the spec if necessary, and they may have some good suggestions. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 25 14:32:00 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Mar 2011 12:32:00 -0600 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 12:28 PM, Mark Wiebe wrote: > On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) < > wickedgrey at gmail.com> wrote: > >> On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: >> > That said, I think starting a discussion with the Python core developers >> > about the float16 type is worthwhile. There might be interest in >> supporting >> > the float16 type in the struct interface, something that would be >> required >> > as part of amending PEP 3118. >> >> That's something that wouldn't see production until Python 3.3, right? >> I'm imagining that the buffer spec in use wouldn't change during the >> 2.7.x series, unless it's unusually accepting of changes. > > > That's true, but explaining the hoops being jumped through to support a new > data type across multiple Python plugins could help evolve the spec if > necessary, and they may have some good suggestions. > > Last I looked, support for PEP 3118 in Python 3 was stuck in the doldrums. Might be worth another ping. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Mar 25 14:41:12 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 25 Mar 2011 11:41:12 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 11:28 AM, Mark Wiebe wrote: > On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) < > wickedgrey at gmail.com> wrote: > >> On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: >> > That said, I think starting a discussion with the Python core developers >> > about the float16 type is worthwhile. There might be interest in >> supporting >> > the float16 type in the struct interface, something that would be >> required >> > as part of amending PEP 3118. >> >> That's something that wouldn't see production until Python 3.3, right? >> I'm imagining that the buffer spec in use wouldn't change during the >> 2.7.x series, unless it's unusually accepting of changes. > > > That's true, but explaining the hoops being jumped through to support a new > data type across multiple Python plugins could help evolve the spec if > necessary, and they may have some good suggestions. > Thinking about that a bit more, maybe a convention added to PEP 3118 for custom types would be useful. For float16, labeling it as 'H' makes sense, then it could be followed by metadata identifying the custom type, like 'H{float16}'. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Fri Mar 25 16:32:58 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 25 Mar 2011 13:32:58 -0700 Subject: [Numpy-discussion] ANN: TreeDict 0.12, a lightweight library for hierarchical parameter/data management Message-ID: Hello, I thought the numpy/scipy community might be particularly interested in this library, so here goes: I'm pleased to announce a release of the next version of TreeDict, a dictionary-like, hierarcical python container to simplify the bookkeeping surrounding parameters, variables and data. It aims to be fast, lightweight, intuitive, feature-rich and stable. While intended for general python coding, it includes a number of features particularly useful for scientific programming. It is similar in basic functionality to MATLAB structures in terms of concise syntax and implicit branch creation, which, I've found, makes organizing parameters/data much easier than with other methods. In addition, though, TreeDict implements all the methods of regular dictionaries, pickling, fast non-intersecting hashing for efficient cache lookups, manipulations on the tree structure, a system for forward referencing branches to make lists of parameters more readable, BSD license, and a full suite of unit tests. More info can be found at http://www.stat.washington.edu/~hoytak/code/treedict/index.html. Questions or comments or feedback are welcome. Thanks! --Hoyt P.S. One thought regarding scipy -- since this library is fairly similar to matlab structs, it may be easy to integrate this with the current matlab io routines to translate things between the two more easily. In this direction, I've added in options to two methods, fromdict and convertTo, to make conversion to/from the existing python representations of the matlab objects. Discussion/comments on this point are welcome. ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From hoytak at stat.washington.edu Fri Mar 25 16:46:41 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 25 Mar 2011 13:46:41 -0700 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error In-Reply-To: References: Message-ID: Okay, last update. I finally have got everything to work. It turns out the problems that I had earlier with f2py were due to intel's -ipo flag. So the only place this flag works is with the C++ code, not fortran or c. Also, I forgot to mention -- the qhull_a.h method has a workaround for some aspect of intel's compiler that is no longer needed and in fact causes an error. It's for a macro that simply suppresses unused variable warnings. In my opinion, it could be removed, as it's only used two places, and scipy spits out enough warnings that that is hardly an issue. Thus my change was around line 102 in qhul_a.h. Replace #if defined(__INTEL_COMPILER) && !defined(QHULL_OS_WIN) template inline void qhullUnused(T &x) { (void)x; } # define QHULL_UNUSED(x) qhullUnused(x); #else # define QHULL_UNUSED(x) (void)x; #endif with #define QHULL_UNUSED(x) Also, I could still not get the CloughTocher2DInterpolator to not segfault. Thus I had to disable it by raising an exception in the init method. With this in place, everything compiles and the unit tests pretty much all run, with 5 failures mostly due to numerical accuracy stuff and 9 errors due to the interpolator. In summary, my final environment variables that give the flags for compiling stuff are: export FLAGS='-xHOST -static -fPIC -g -fltconsistency' export CFLAGS="$FLAGS -O2 -fno-alias" export CPPFLAGS="$FLAGS -fno-alias -ipo -O3" export CXXFLAGS="$CPPFLAGS" export FFLAGS="$FLAGS -O3" export F77FLAGS="$FFLAGS" export F90FLAGS="$FFLAGS" export LDFLAGS="-xHOST -O1 -openmp -lpthread -fPIC" And the arguments given to the fortran compiler in fcompiler/intel.py are: compiler_opt_flags = '-static -xHOST -fPIC -DMKL_LP64 -mkl -g -O3' I'd be happy to answer any more questions about the process as needed. Now, back to my real work. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From pav at iki.fi Fri Mar 25 17:30:23 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 25 Mar 2011 21:30:23 +0000 (UTC) Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error References: Message-ID: On Fri, 25 Mar 2011 13:46:41 -0700, Hoyt Koepke wrote: [clip] > Also, I could still not get the CloughTocher2DInterpolator to not > segfault. Backtrace would be useful here. It's probably best to recompile with "-O0" and some debug flags enabled in the compiler to get something reasonable out. -- Pauli Virtanen From hoytak at stat.washington.edu Fri Mar 25 19:14:52 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 25 Mar 2011 16:14:52 -0700 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error In-Reply-To: References: Message-ID: Okay, even better news. After a little bit of probing around, it turns out that some of the errors, namely the one in the interpolator, were due to intel's handling of floating point values at higher optimization levels. Thus the two rather important flags, if you wish to avoid difficult-to-explain segfaults, are -fp-model strict ( disables unsafe fp optimizations) -fno-alias (C & C++, not fortran) Now here's a summary of the other errors: -ipo -- okay for linker / c++, but not elsewhere. For C, it causes the long_double_representation() function in setup_common() to fail (I'll note this on the corresponding ticket), and causes f2py to fail to generate correct wrappers. -O3 -- okay for C++ / fortran, but not C. For C, it causes einsum to hang. Thus the highest optimization levels I could find in which everything compiled and ran were: Fortran: -static -DMKL_LP64 -mkl -xHOST -O3 -fPIC -fp-model strict C: -static -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict C++: -static -DMKL_LP64 -mkl -xHOST -O3 -fno-alias -fPIC -fp-model strict Linker: -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict -openmp -lpthread Enjoy, --Hoyt On Fri, Mar 25, 2011 at 2:30 PM, Pauli Virtanen wrote: > On Fri, 25 Mar 2011 13:46:41 -0700, Hoyt Koepke wrote: > [clip] >> Also, I could still not get the CloughTocher2DInterpolator to not >> segfault. > > Backtrace would be useful here. It's probably best to recompile with > "-O0" and some debug flags enabled in the compiler to get something > reasonable out. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From nwagner at iam.uni-stuttgart.de Sat Mar 26 06:34:53 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sat, 26 Mar 2011 11:34:53 +0100 Subject: [Numpy-discussion] ValueError: Unknown format code 'g' for object of type 'str' Message-ID: >>> numpy.__version__ '2.0.0.dev-10db259' ====================================================================== ERROR: Test the str.format method with NumPy scalar types ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", line 183, in runTest self.test(*self.arg) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/tests/test_print.py", line 223, in test_scalar_format assert_equal(fmat.format(val), fmat.format(valtype(val)), ValueError: Unknown format code 'g' for object of type 'str' From paul.anton.letnes at gmail.com Sat Mar 26 08:11:46 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sat, 26 Mar 2011 13:11:46 +0100 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: Message-ID: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> Hi! I have had a look at the list of numpy.loadtxt tickets. I have never contributed to numpy before, so I may be doing stupid things - don't be afraid to let me know! My opinions are my own, and in detail, they are: 1752: I attach a possible patch. FWIW, I agree with the request. The patch is written to be compatible with the fix in ticket #1562, but I did not test that yet. 1731: This seems like a rather trivial feature enhancement. I attach a possible patch. 1616: The suggested patch seems reasonable to me, but I do not have a full list of what objects loadtxt supports today as opposed to what this patch will support. 1562: I attach a possible patch. This could also be the default behavior to my mind, since the function caller can simply call numpy.squeeze if needed. Changing default behavior would probably break old code, however. 1458: The fix suggested in the ticket seems reasonable, but I have never used record arrays, so I am not sure of this. 1445: Adding this functionality could break old code, as some old datafiles may have empty lines which are now simply ignored. I do not think the feature is a good idea. It could rather be implemented as a separate function. 1107: I do not see the need for this enhancement. In my eyes, the usecols kwarg does this and more. Perhaps I am misunderstanding something here. 1071: It is not clear to me whether loadtxt is supposed to support missing values in the fashion indicated in the ticket. 1163: 1565: These tickets seem to have the same origin of the problem. I attach one possible patch. The previously suggested patches that I've seen will not correctly convert floats to ints, which I believe my patch will. I hope you find this useful! Is there some way of submitting the patches for review in a more convenient fashion than e-mail? Cheers, Paul. -------------- next part -------------- A non-text attachment was scrubbed... Name: 1562.patch Type: application/octet-stream Size: 1515 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1163.patch Type: application/octet-stream Size: 1015 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1731.patch Type: application/octet-stream Size: 1399 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1752.patch Type: application/octet-stream Size: 631 bytes Desc: not available URL: -------------- next part -------------- On 25. mars 2011, at 16.06, Charles R Harris wrote: > Hi All, > > Could someone with an interest in loadtxt/savetxt look through the associated tickets? A search on the tickets using either of those keys will return fairly lengthy lists. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Sat Mar 26 09:36:47 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Mar 2011 13:36:47 +0000 (UTC) Subject: [Numpy-discussion] loadtxt/savetxt tickets References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> Message-ID: Hi, Thanks! On Sat, 26 Mar 2011 13:11:46 +0100, Paul Anton Letnes wrote: [clip] > I hope you find this useful! Is there some way of submitting the patches > for review in a more convenient fashion than e-mail? You can attach them on the trac to each ticket. That way they'll be easy to find later on. Pauli From derek at astro.physik.uni-goettingen.de Sat Mar 26 10:20:32 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Sat, 26 Mar 2011 15:20:32 +0100 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> Message-ID: <20A47259-DCF2-4119-A81F-78A8F3100749@astro.physik.uni-goettingen.de> Hi, On 26 Mar 2011, at 14:36, Pauli Virtanen wrote: > > On Sat, 26 Mar 2011 13:11:46 +0100, Paul Anton Letnes wrote: > [clip] >> I hope you find this useful! Is there some way of submitting the >> patches >> for review in a more convenient fashion than e-mail? > > You can attach them on the trac to each ticket. That way they'll be > easy > to find later on. I've got some comments on 1562, and I'd attach a revised patch then - just a general question: should I then change "Milestone" to 1.6.0 and "Version" to 'devel'? > 1562: > I attach a possible patch. This could also be the default > behavior to my mind, since the function caller can simply call > numpy.squeeze if needed. Changing default behavior would probably > break old code, Seems the fastest solution unless someone wants to change numpy.squeeze as well. But the present patch does not call np.squeeze any more at all, so I propose to restore that behaviour for X.ndim > ndmin to remain really backwards compatible. It also seems easier to code when making the default ndmin=0. Cheers, Derek From derek at astro.physik.uni-goettingen.de Sat Mar 26 10:48:56 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Sat, 26 Mar 2011 15:48:56 +0100 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: <20A47259-DCF2-4119-A81F-78A8F3100749@astro.physik.uni-goettingen.de> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <20A47259-DCF2-4119-A81F-78A8F3100749@astro.physik.uni-goettingen.de> Message-ID: <508D7502-1C87-41DB-8CEA-A4EB56487232@astro.physik.uni-goettingen.de> Hi again, On 26 Mar 2011, at 15:20, Derek Homeier wrote: >> >> 1562: >> I attach a possible patch. This could also be the default >> behavior to my mind, since the function caller can simply call >> numpy.squeeze if needed. Changing default behavior would probably >> break old code, > > Seems the fastest solution unless someone wants to change > numpy.squeeze > as well. But the present patch does not call np.squeeze any more at > all, so I > propose to restore that behaviour for X.ndim > ndmin to remain really > backwards > compatible. It also seems easier to code when making the default > ndmin=0. I've got another somewhat general question: since it would probably be nice to have a test for this, I found one could simply add something along the lines of assert_equal(a.shape, x.shape) to test_io.py - test_shaped_dtype(self) or should one generally create a new test for such things (might still be better in this case, since test_shaped_dtype does not really test different ndim)? Cheers, Derek From paul.anton.letnes at gmail.com Sat Mar 26 10:53:20 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sat, 26 Mar 2011 15:53:20 +0100 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: <508D7502-1C87-41DB-8CEA-A4EB56487232@astro.physik.uni-goettingen.de> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <20A47259-DCF2-4119-A81F-78A8F3100749@astro.physik.uni-goettingen.de> <508D7502-1C87-41DB-8CEA-A4EB56487232@astro.physik.uni-goettingen.de> Message-ID: Hi Derek! On 26. mars 2011, at 15.48, Derek Homeier wrote: > Hi again, > > On 26 Mar 2011, at 15:20, Derek Homeier wrote: >>> >>> 1562: >>> I attach a possible patch. This could also be the default >>> behavior to my mind, since the function caller can simply call >>> numpy.squeeze if needed. Changing default behavior would probably >>> break old code, >> >> Seems the fastest solution unless someone wants to change >> numpy.squeeze >> as well. But the present patch does not call np.squeeze any more at >> all, so I >> propose to restore that behaviour for X.ndim > ndmin to remain really >> backwards >> compatible. It also seems easier to code when making the default >> ndmin=0. > > I've got another somewhat general question: since it would probably be > nice to > have a test for this, I found one could simply add something along the > lines of > > assert_equal(a.shape, x.shape) > > to test_io.py - test_shaped_dtype(self) > or should one generally create a new test for such things (might still > be better > in this case, since test_shaped_dtype does not really test different > ndim)? > > Cheers, > Derek It would be nice to see your patch. I uploaded all of mine as mentioned. I'm no testing expert, but I am sure someone else will comment on it. Paul. From sourceforge.numpy at user.fastmail.fm Sat Mar 26 13:10:42 2011 From: sourceforge.numpy at user.fastmail.fm (Hugo Gagnon) Date: Sat, 26 Mar 2011 13:10:42 -0400 Subject: [Numpy-discussion] Array views Message-ID: <1301159442.11145.1434128601@webmail.messagingengine.com> Hello, Say I have a few 1d arrays and one 2d array which columns I want to be the 1d arrays. I also want all the a's arrays to share the *same data* with the b array. If I call my 1d arrays a1, a2, etc. and my 2d array b, then b[:,0] = a1[:] b[:,1] = a2[:] ... won't work because apparently copying occurs. I tried it the other way around i.e. a1 = b[:,0] a2 = b[:,1] ... and it works but that doesn't help me for my problem. Is there a way to reformulate the first code snippet above but with shallow copying? Thanks, -- Hugo Gagnon -- Hugo Gagnon From pav at iki.fi Sat Mar 26 13:12:22 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Mar 2011 17:12:22 +0000 (UTC) Subject: [Numpy-discussion] Array views References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: On Sat, 26 Mar 2011 13:10:42 -0400, Hugo Gagnon wrote: [clip] > a1 = b[:,0] > a2 = b[:,1] > ... > > and it works but that doesn't help me for my problem. Is there a way to > reformulate the first code snippet above but with shallow copying? No. You need an 2-D array to "own" the data. The second way is the approach to use if you want to share the data. -- Pauli Virtanen From srean.list at gmail.com Sat Mar 26 13:32:24 2011 From: srean.list at gmail.com (srean) Date: Sat, 26 Mar 2011 12:32:24 -0500 Subject: [Numpy-discussion] Array views In-Reply-To: <1301159442.11145.1434128601@webmail.messagingengine.com> References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: Hi, I am also interested in this. In my application there is a large 2d array, lets call it 'b' to keep the notation consistent in the thread. b's columns need to be recomputed often. Ideally this re-computation happens in a function. Lets call that function updater(b, col_index): The simplest example is where updater(b, col_index) is a matrix vector multiply, where the matrix or the vector changes. Is there anyway apart from using ufuncs that I can make updater() write the result directly in b and not create a new temporary column that is then copied into b ? Say for the matrix vector multiply example. I can write the matrix vector product in terms of ufuncs but will lose out in terms of speed. In the best case scenario I would like to maintain 'b' in a csr sparse matrix form, as 'b' participates in a matrix vector multiply. I think csr would be asking for too much, but even ccs should help. I dont want to clutter this thread with the sparsity issues though, any solution to the original question or pointers to solutions would be appreciated. Thanks --srean On Sat, Mar 26, 2011 at 12:10 PM, Hugo Gagnon < sourceforge.numpy at user.fastmail.fm> wrote: > Hello, > > Say I have a few 1d arrays and one 2d array which columns I want to be > the 1d arrays. > I also want all the a's arrays to share the *same data* with the b > array. > If I call my 1d arrays a1, a2, etc. and my 2d array b, then > > b[:,0] = a1[:] > b[:,1] = a2[:] > ... > > won't work because apparently copying occurs. > I tried it the other way around i.e. > > a1 = b[:,0] > a2 = b[:,1] > ... > > and it works but that doesn't help me for my problem. > Is there a way to reformulate the first code snippet above but with > shallow copying? > > Thanks, > -- > Hugo Gagnon > -- > Hugo Gagnon > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Sat Mar 26 14:31:10 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 26 Mar 2011 11:31:10 -0700 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: <4D8E30EE.1010401@noaa.gov> On 3/26/11 10:32 AM, srean wrote: > I am also interested in this. In my application there is a large 2d > array, lets call it 'b' to keep the notation consistent in the thread. > b's columns need to be recomputed often. Ideally this re-computation > happens in a function. Lets call that function updater(b, col_index): > The simplest example is where > updater(b, col_index) is a matrix vector multiply, where the matrix or > the vector changes. > > Is there anyway apart from using ufuncs that I can make updater() > write the result directly in b and not create a new temporary column > that is then copied into b ? Say for the matrix vector multiply example. Probably not -- the trick is that when an array is a view of a slice of another array, it may not be laid out in memory in a way that other libs (like LAPACK, BLAS, etc) require, so the data needs to be copied to call those routines. To understand all this, you'll need to study up a bit on how numpy arrays lay out and access the memory that they use: they use a concept of "strided" memory. It's very powerful and flexible, but most other numeric libs can't use those same data structures. I"m not sure what a good doc is to read to learn about this -- I learned it from messing with the C API. TAke a look at any docs that talk about "strides", and maybe playing with the "stride tricks" tools will help. A simple example: In [3]: a = np.ones((3,4)) In [4]: a Out[4]: array([[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]) In [5]: a.flags Out[5]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False So a is a (3,4) array, stored in C_contiguous fashion, jsut like a "regular old C array". A lib expecting data in this fashion could use the data pointer just like regular C code. In [6]: a.strides Out[6]: (32, 8) this means is is 32 bytes from the start of one row to the next, and 8 bytes from the start of one element to the next -- which makes sense for a 64bit double. In [7]: b = a[:,1] In [10]: b Out[10]: array([ 1., 1., 1.]) so b is a 1-d array with three elements. In [8]: b.flags Out[8]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False but it is NOT C_Contiguous - the data is laid out differently that a standard C array. In [9]: b.strides Out[9]: (32,) so this means that it is 32 bytes from one element to the next -- for a 8 byte data type. This is because the elements are each one element in a row of the a array -- they are not all next to each other. A regular C library generally won't be able to work with data laid out like this. HTH, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Sat Mar 26 14:36:18 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 26 Mar 2011 11:36:18 -0700 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: <4D8E3222.5050408@noaa.gov> On 3/26/11 10:12 AM, Pauli Virtanen wrote: > On Sat, 26 Mar 2011 13:10:42 -0400, Hugo Gagnon wrote: > [clip] >> a1 = b[:,0] >> a2 = b[:,1] >> ... >> >> and it works but that doesn't help me for my problem. Is there a way to >> reformulate the first code snippet above but with shallow copying? > > No. You need an 2-D array to "own" the data. The second way is the > approach to use if you want to share the data. exactly -- but to clarify, it's not just about ownership, it's about layout of the data in memory. the data in a numpy array needs to be laid out in memory as one block, with consitent strides from one element to the next, one row to the next, etc. When you create an array from scratch (like your 2-d array here), you get one big block of memory. If you create each row separately, they each have their own block of memory that are unrelated -- there is no way to put those together into one block with consistent strides. So you need to create that big block first (the 2-d array), then you can reference parts of it for each row. See my previous note for a bit more discussion. Oh, and maybe the little presentation and sample code I gave to the Seattle Python Interest group will help: http://www.seapig.org/November2010Notes -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From srean.list at gmail.com Sat Mar 26 14:50:24 2011 From: srean.list at gmail.com (srean) Date: Sat, 26 Mar 2011 13:50:24 -0500 Subject: [Numpy-discussion] Array views In-Reply-To: <4D8E30EE.1010401@noaa.gov> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> Message-ID: Hi Christopher, thanks for taking the time to reply at length. I do understand the concept of striding in general but was not familiar with the Numpy way of accessing that information. So thanks for pointing me to .flag and .stride. That said, BLAS/LAPACK do have apis that take the stride length into account. But for sparse arrays I think its a hopeless situation. That is a bummer, because sparse is what I need. Oh well, I will probably do it in C++ -- srean p.s. I hope top posting is not frowned upon here. If so, I will keep that in mind in my future posts. On Sat, Mar 26, 2011 at 1:31 PM, Christopher Barker wrote: > > Probably not -- the trick is that when an array is a view of a slice of > another array, it may not be laid out in memory in a way that other libs > (like LAPACK, BLAS, etc) require, so the data needs to be copied to call > those routines. > > To understand all this, you'll need to study up a bit on how numpy > arrays lay out and access the memory that they use: they use a concept > of "strided" memory. It's very powerful and flexible, but most other > numeric libs can't use those same data structures. I"m not sure what a > good doc is to read to learn about this -- I learned it from messing > with the C API. TAke a look at any docs that talk about "strides", and > maybe playing with the "stride tricks" tools will help. > > A simple example: > > In [3]: a = np.ones((3,4)) > > In [4]: a > Out[4]: > array([[ 1., 1., 1., 1.], > [ 1., 1., 1., 1.], > [ 1., 1., 1., 1.]]) > > In [5]: a.flags > Out[5]: > C_CONTIGUOUS : True > F_CONTIGUOUS : False > OWNDATA : True > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > So a is a (3,4) array, stored in C_contiguous fashion, jsut like a > "regular old C array". A lib expecting data in this fashion could use > the data pointer just like regular C code. > > In [6]: a.strides > Out[6]: (32, 8) > > this means is is 32 bytes from the start of one row to the next, and 8 > bytes from the start of one element to the next -- which makes sense for > a 64bit double. > > > In [7]: b = a[:,1] > > In [10]: b > Out[10]: array([ 1., 1., 1.]) > > so b is a 1-d array with three elements. > > In [8]: b.flags > Out[8]: > C_CONTIGUOUS : False > F_CONTIGUOUS : False > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > but it is NOT C_Contiguous - the data is laid out differently that a > standard C array. > > In [9]: b.strides > Out[9]: (32,) > > so this means that it is 32 bytes from one element to the next -- for a > 8 byte data type. This is because the elements are each one element in a > row of the a array -- they are not all next to each other. A regular C > library generally won't be able to work with data laid out like this. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Mar 26 15:13:43 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Mar 2011 19:13:43 +0000 (UTC) Subject: [Numpy-discussion] Array views References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: On Sat, 26 Mar 2011 12:32:24 -0500, srean wrote: [clip] > Is there anyway apart from using ufuncs that I can make updater() write > the result directly in b and not create a new temporary column that is > then copied into b? Say for the matrix vector multiply example. I can > write the matrix vector product in terms of ufuncs but will lose out in > terms of speed. Well, you can e.g. write def updater(b, col_idx): b[:,col_idx] *= 3 # <- modifies b[:,col_idx] in place And ditto for sparse matrices --- but maybe this is not what you asked. If you want to have control over temporaries, you can make use of the out= argument of ufuncs (`numpy.dot` will gain it in 1.6.1 --- you can call LAPACK routines from scipy.lib in the meantime, if your data is in Fortran order). Also numexpr is probably able to write the output directly to a given array --- using it is an alternative way to avoid temporaries, and probably easier to write than doing things via the out= arguments. For sparse matrices, things then depend on how they are laid out in memory. You can probably alter the `.data` attribute of the arrays directly, if you know how the underlying representation works. -- Pauli Virtanen From pav at iki.fi Sat Mar 26 15:34:34 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Mar 2011 19:34:34 +0000 (UTC) Subject: [Numpy-discussion] Array views References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: On Sat, 26 Mar 2011 19:13:43 +0000, Pauli Virtanen wrote: [clip] > If you want to have control over temporaries, you can make use of the > out= argument of ufuncs (`numpy.dot` will gain it in 1.6.1 --- you can > call LAPACK routines from scipy.lib in the meantime, if your data is in > Fortran order). Like so: # Fortran-order for efficient DGEMM -- each column must be contiguous A = np.random.randn(4,4).copy('F') b = np.random.randn(4,10).copy('F') def updater(b, col_idx): # This will work in Numpy 1.6.1 dot(A, b[:,col_idx].copy(), out=b[:,col_idx]) In the meantime you can do A = np.random.randn(4,4).copy('F') b = np.random.randn(4,10).copy('F') from scipy.lib.blas import get_blas_funcs gemm, = get_blas_funcs(['gemm'], [A, b]) # get correct type func def updater(b, col_idx): bcol = b[:,col_idx] c = gemm(1.0, A, bcol.copy(), 0.0, bcol, overwrite_c=True) assert c is bcol # check that it didn't make copies! Note that DGEMM and `dot` cannot do in-place multiplication -- at least the BLAS library I have fails when the B and C arguments point to the same memory, so you'll anyway end up with one temporary. (This has nothing to do with Scipy -- same occurs in Fortran). -- Pauli Virtanen From srean.list at gmail.com Sat Mar 26 16:16:13 2011 From: srean.list at gmail.com (srean) Date: Sat, 26 Mar 2011 15:16:13 -0500 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: Ah! very nice. I did not know that numpy-1.6.1 supports in place 'dot', and neither the fact that you could access the underlying BLAS functions like so. This is pretty neat. Thanks. Now I at least have an idea how the sparse version might work. If I get time I will probably give numpy-1.6.1 a shot. I already have the MKL libraries thanks to free version of epd for students. On Sat, Mar 26, 2011 at 2:34 PM, Pauli Virtanen wrote: > > Like so: > > # Fortran-order for efficient DGEMM -- each column must be contiguous > A = np.random.randn(4,4).copy('F') > b = np.random.randn(4,10).copy('F') > > def updater(b, col_idx): > # This will work in Numpy 1.6.1 > dot(A, b[:,col_idx].copy(), out=b[:,col_idx]) > > In the meantime you can do > > A = np.random.randn(4,4).copy('F') > b = np.random.randn(4,10).copy('F') > > from scipy.lib.blas import get_blas_funcs > gemm, = get_blas_funcs(['gemm'], [A, b]) # get correct type func > > def updater(b, col_idx): > bcol = b[:,col_idx] > c = gemm(1.0, A, bcol.copy(), 0.0, bcol, overwrite_c=True) > assert c is bcol # check that it didn't make copies! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srean.list at gmail.com Sat Mar 26 16:19:17 2011 From: srean.list at gmail.com (srean) Date: Sat, 26 Mar 2011 15:19:17 -0500 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> Message-ID: On Sat, Mar 26, 2011 at 3:16 PM, srean wrote: > > Ah! very nice. I did not know that numpy-1.6.1 supports in place 'dot', > In place is perhaps not the right word, I meant "in a specified location" -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 26 16:25:25 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Mar 2011 14:25:25 -0600 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <20A47259-DCF2-4119-A81F-78A8F3100749@astro.physik.uni-goettingen.de> <508D7502-1C87-41DB-8CEA-A4EB56487232@astro.physik.uni-goettingen.de> Message-ID: On Sat, Mar 26, 2011 at 8:53 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > Hi Derek! > > On 26. mars 2011, at 15.48, Derek Homeier wrote: > > > Hi again, > > > > On 26 Mar 2011, at 15:20, Derek Homeier wrote: > >>> > >>> 1562: > >>> I attach a possible patch. This could also be the default > >>> behavior to my mind, since the function caller can simply call > >>> numpy.squeeze if needed. Changing default behavior would probably > >>> break old code, > >> > >> Seems the fastest solution unless someone wants to change > >> numpy.squeeze > >> as well. But the present patch does not call np.squeeze any more at > >> all, so I > >> propose to restore that behaviour for X.ndim > ndmin to remain really > >> backwards > >> compatible. It also seems easier to code when making the default > >> ndmin=0. > > > > I've got another somewhat general question: since it would probably be > > nice to > > have a test for this, I found one could simply add something along the > > lines of > > > > assert_equal(a.shape, x.shape) > > > > to test_io.py - test_shaped_dtype(self) > > or should one generally create a new test for such things (might still > > be better > > in this case, since test_shaped_dtype does not really test different > > ndim)? > > > > Cheers, > > Derek > > It would be nice to see your patch. I uploaded all of mine as mentioned. > I'm no testing expert, but I am sure someone else will comment on it. > > I put all these patches together at https://github.com/charris/numpy/tree/loadtxt-savetxt. Please pull from there to continue work on loadtxt/savetxt so as to avoid conflicts in the patches. One of the numpy tests is failing, I assume from patch conflicts, and more tests for the tickets are needed in any case. Also, new keywords should be added to the end, not put in the middle of existing keywords. I haven't reviewed the patches, just tried to get them organized. Also, I have Derek as the author on all of them, that can be changed if it is decided the credit should go elsewhere ;) Thanks for the work you all have been doing on these tickets. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Sat Mar 26 16:44:14 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Sat, 26 Mar 2011 21:44:14 +0100 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> Message-ID: <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Hi Paul, having had a look at the other tickets you dug up, > My opinions are my own, and in detail, they are: > 1752: > I attach a possible patch. FWIW, I agree with the request. The > patch is written to be compatible with the fix in ticket #1562, but > I did not test that yet. Tested, see also my comments on Trac. > 1731: > This seems like a rather trivial feature enhancement. I attach a > possible patch. Agreed. Haven't tested it though. > 1616: > The suggested patch seems reasonable to me, but I do not have a > full list of what objects loadtxt supports today as opposed to what > this patch will support. > 1562: > I attach a possible patch. This could also be the default > behavior to my mind, since the function caller can simply call > numpy.squeeze if needed. Changing default behavior would probably > break old code, however. See comments on Trac as well. > 1458: > The fix suggested in the ticket seems reasonable, but I have > never used record arrays, so I am not sure of this. There were some issues with Python3, and I also had some general reservations as noted on Trac - basically, it makes 'unpack' equivalent to transposing for 2D-arrays, but to splitting into fields for 1D-recarrays. My question was, what's going to happen when you get to 2D-recarrays? Currently this is not an issue since loadtxt can only read 2D regular or 1D structured arrays. But this might change if the data block functionality (see below) were to be implemented - data could then be returned as 3D arrays or 2D structured arrays... Still, it would probably make most sense (or at least give the widest functionality) to have 'unpack=True' always return a list or iterator over columns. > 1445: > Adding this functionality could break old code, as some old > datafiles may have empty lines which are now simply ignored. I do > not think the feature is a good idea. It could rather be implemented > as a separate function. > 1107: > I do not see the need for this enhancement. In my eyes, the > usecols kwarg does this and more. Perhaps I am misunderstanding > something here. Agree about #1445, and the bit about 'usecols' - 'numcols' would just provide a shorter call to e.g. read the first 20 columns of a file (well, not even that much over 'usecols=range(20)'...), don't think that justifies an extra argument. But the 'datablocks' provides something new, that a number of people seem to miss from e.g. gnuplot (including me, actually ;-). And it would also satisfy the request from #1445 without breaking backwards compatibility. I've been wondering if could instead specify the separator lines through the parameter, e.g. "blocksep=['None', 'blank','invalid']", not sure if that would make it more useful... > 1071: > It is not clear to me whether loadtxt is supposed to support > missing values in the fashion indicated in the ticket. In principle it should at least allow you to, by the use of converters as described there. The problem is, the default delimiter is described as 'any whitespace', which in the present implementation obviously includes any number of blanks or tabs. These are therefore treated differently from delimiters like ',' or '&'. I'd reckon there are too many people actually relying on this behaviour to silently change it (e.g. I know plenty of tables with columns separated by either one or several tabs depending on the length of the previous entry). But the tab is apparently also treated differently if explicitly specified with "delimiter='\t'" - and in that case using a converter ? la {2: lambda s: float(s or 'Nan')} is working for fields in the middle of the line, but not at the end - clearly warrants improvement. I've prepared a patch working for Python3 as well. > 1163: > 1565: > These tickets seem to have the same origin of the problem. I > attach one possible patch. The previously suggested patches that > I've seen will not correctly convert floats to ints, which I believe > my patch will. +1, though I am a bit concerned that prompting to raise a ValueError for every element could impede performance. I'd probably still enclose it into an if issubclass(typ, np.uint64) or issubclass(typ, np.int64): just like in npio.patch. I also thought one might switch to int(float128(x)) in that case, but at least for the given examples float128 cannot convert with more accuracy than float64 (even on PowerPC ;-). There were some dissenting opinions that trying to read a float into an int should generally throw an exception though... And Chuck just beat me... On 26 Mar 2011, at 21:25, Charles R Harris wrote: > I put all these patches together at https://github.com/charris/numpy/tree/loadtxt-savetxt > . Please pull from there to continue work on loadtxt/savetxt so as > to avoid conflicts in the patches. One of the numpy tests is > failing, I assume from patch conflicts, and more tests for the > tickets are needed in any case. Also, new keywords should be added > to the end, not put in the middle of existing keywords. > > I haven't reviewed the patches, just tried to get them organized. > Also, I have Derek as the author on all of them, that can be changed > if it is decided the credit should go elsewhere ;) Thanks for the > work you all have been doing on these tickets. Thanks, I'll have a look at the new ticket and try to get that organized! Cheers, Derek From mwwiebe at gmail.com Sat Mar 26 19:58:58 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 26 Mar 2011 16:58:58 -0700 Subject: [Numpy-discussion] ValueError: Unknown format code 'g' for object of type 'str' In-Reply-To: References: Message-ID: It turns out that Python 2.6 complex doesn't implement __format__, and that results in the problem. http://web.archiveorange.com/archive/v/jA6s92Ni29ENZpi4rpz5 I've disabled the complex formatting tests for 2.6 in commit 7d436cc8994f9efbc512. -Mark On Sat, Mar 26, 2011 at 3:34 AM, Nils Wagner wrote: > >>> numpy.__version__ > '2.0.0.dev-10db259' > > ====================================================================== > ERROR: Test the str.format method with NumPy scalar types > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", > line 183, in runTest > self.test(*self.arg) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", > line 146, in skipper_func > return f(*args, **kwargs) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/tests/test_print.py", > line 223, in test_scalar_format > assert_equal(fmat.format(val), > fmat.format(valtype(val)), > ValueError: Unknown format code 'g' for object of type > 'str' > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dileepkunjaai at gmail.com Sun Mar 27 03:24:29 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Sun, 27 Mar 2011 12:54:29 +0530 Subject: [Numpy-discussion] Please help me Message-ID: Dear sir, Can we take a numpy array excluding particular columns Example: A=[[ 20. 71. 57.5] [ 21. 71. 56. ] [ 22. 71. 74.3] [ 23. 71. 66. ] [ 24. 71. 96.5] [ 25. 71. 61.7] [ 20. 72. 72.9] (Here i am give up the last column) The answer should be [[ 20. 71. ] [ 21. 71. ] [ 22. 71. ] [ 23. 71. ] [ 24. 71. ] [ 25. 71. ] [ 20. 72. ] Please help me ............................... -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sun Mar 27 03:48:46 2011 From: andyfaff at gmail.com (andrew nelson) Date: Sun, 27 Mar 2011 18:48:46 +1100 Subject: [Numpy-discussion] Please help me In-Reply-To: References: Message-ID: I think A[:,0:2] should give you what you want. On 27/03/2011, at 6:24 PM, dileep kunjaai wrote: > Dear sir, > Can we take a numpy array excluding particular columns > Example: > A=[[ 20. 71. 57.5] > [ 21. 71. 56. ] > [ 22. 71. 74.3] > [ 23. 71. 66. ] > [ 24. 71. 96.5] > [ 25. 71. 61.7] > [ 20. 72. 72.9] > (Here i am give up the last column) > > The answer should be [[ 20. 71. ] > [ 21. 71. ] > [ 22. 71. ] > [ 23. 71. ] > [ 24. 71. ] > [ 25. 71. ] > [ 20. 72. ] > > Please help me ............................... > > -- > DILEEPKUMAR. R > J R F, IIT DELHI > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _________________ Dr Andrew Nelson andyfaff at gmail.com _________________ From dileepkunjaai at gmail.com Sun Mar 27 03:56:42 2011 From: dileepkunjaai at gmail.com (dileep kunjaai) Date: Sun, 27 Mar 2011 13:26:42 +0530 Subject: [Numpy-discussion] Please help me In-Reply-To: References: Message-ID: Thank you sir thank you very much...... sir i got a new flexible command: >>> delete(a, s_[1:3], axis=1) # remove columns 1 and 2 http://www.scipy.org/Numpy_Example_List#head-2413c07da8954330701674ff8e950c4505665be1 On Sun, Mar 27, 2011 at 1:18 PM, andrew nelson wrote: > I think A[:,0:2] should give you what you want. > > On 27/03/2011, at 6:24 PM, dileep kunjaai wrote: > > > Dear sir, > > Can we take a numpy array excluding particular columns > > Example: > > A=[[ 20. 71. 57.5] > > [ 21. 71. 56. ] > > [ 22. 71. 74.3] > > [ 23. 71. 66. ] > > [ 24. 71. 96.5] > > [ 25. 71. 61.7] > > [ 20. 72. 72.9] > > (Here i am give up the last column) > > > > The answer should be [[ 20. 71. ] > > [ 21. 71. ] > > [ 22. 71. ] > > [ 23. 71. ] > > [ 24. 71. ] > > [ 25. 71. ] > > [ 20. 72. ] > > > > Please help me ............................... > > > > -- > > DILEEPKUMAR. R > > J R F, IIT DELHI > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _________________ > Dr Andrew Nelson > andyfaff at gmail.com > _________________ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- DILEEPKUMAR. R J R F, IIT DELHI -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Mar 27 05:56:07 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 27 Mar 2011 11:56:07 +0200 Subject: [Numpy-discussion] histogram normed/density keyword (#1628) Message-ID: Hi all, For the 1.6 release #1628 needs to be resolved. A while ago there was a discussion about the normed keyword in histogram, which ATM has changed behavior compared to numpy 1.5.1. The preferred fix as I read the discussion (http://thread.gmane.org/gmane.comp.python.numeric.general/39746/focus=40089) was to leave normed alone and add a new keyword density. It would be helpful if someone can volunteer to fix this. Thanks, Ralf From paul.anton.letnes at gmail.com Sun Mar 27 06:09:18 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 27 Mar 2011 12:09:18 +0200 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On 26. mars 2011, at 21.44, Derek Homeier wrote: > Hi Paul, > > having had a look at the other tickets you dug up, > >> My opinions are my own, and in detail, they are: >> 1752: >> I attach a possible patch. FWIW, I agree with the request. The >> patch is written to be compatible with the fix in ticket #1562, but >> I did not test that yet. > > Tested, see also my comments on Trac. Great! >> 1731: >> This seems like a rather trivial feature enhancement. I attach a >> possible patch. > > Agreed. Haven't tested it though. Great! >> 1616: >> The suggested patch seems reasonable to me, but I do not have a >> full list of what objects loadtxt supports today as opposed to what >> this patch will support. Looks like you got this one. Just remember to make it compatible with #1752. Should be easy. >> 1562: >> I attach a possible patch. This could also be the default >> behavior to my mind, since the function caller can simply call >> numpy.squeeze if needed. Changing default behavior would probably >> break old code, however. > > See comments on Trac as well. Your patch is better, but there is one thing I disagree with. 808 if X.ndim < ndmin: 809 if ndmin == 1: 810 X.shape = (X.size, ) 811 elif ndmin == 2: 812 X.shape = (X.size, 1) The last line should be: 812 X.shape = (1, X.size) If someone wants a 2D array out, they would most likely expect a one-row file to come out as a one-row array, not the other way around. IMHO. >> 1458: >> The fix suggested in the ticket seems reasonable, but I have >> never used record arrays, so I am not sure of this. > > There were some issues with Python3, and I also had some general > reservations > as noted on Trac - basically, it makes 'unpack' equivalent to > transposing for 2D-arrays, > but to splitting into fields for 1D-recarrays. My question was, what's > going to happen > when you get to 2D-recarrays? Currently this is not an issue since > loadtxt can only > read 2D regular or 1D structured arrays. But this might change if the > data block > functionality (see below) were to be implemented - data could then be > returned as > 3D arrays or 2D structured arrays... Still, it would probably make > most sense (or at > least give the widest functionality) to have 'unpack=True' always > return a list or iterator > over columns. OK, I don't know recarrays, as I said. >> 1445: >> Adding this functionality could break old code, as some old >> datafiles may have empty lines which are now simply ignored. I do >> not think the feature is a good idea. It could rather be implemented >> as a separate function. >> 1107: >> I do not see the need for this enhancement. In my eyes, the >> usecols kwarg does this and more. Perhaps I am misunderstanding >> something here. > > Agree about #1445, and the bit about 'usecols' - 'numcols' would just > provide a > shorter call to e.g. read the first 20 columns of a file (well, not > even that much > over 'usecols=range(20)'...), don't think that justifies an extra > argument. > But the 'datablocks' provides something new, that a number of people > seem > to miss from e.g. gnuplot (including me, actually ;-). And it would > also satisfy the > request from #1445 without breaking backwards compatibility. > I've been wondering if could instead specify the separator lines > through the > parameter, e.g. "blocksep=['None', 'blank','invalid']", not sure if > that would make > it more useful... What about writing a separate function, e.g. loadblocktxt, and have it separate the chunks and call loadtxt for each chunk? Just a thought. Another possibility would be to write a function that would let you load a set of text files in a directory, and return a dict of datasets, one per file. One could write a similar save-function, too. They would just need to call loadtxt/savetxt on a per-file basis. >> 1071: >> It is not clear to me whether loadtxt is supposed to support >> missing values in the fashion indicated in the ticket. > > In principle it should at least allow you to, by the use of converters > as described there. > The problem is, the default delimiter is described as 'any > whitespace', which in the > present implementation obviously includes any number of blanks or > tabs. These > are therefore treated differently from delimiters like ',' or '&'. I'd > reckon there are > too many people actually relying on this behaviour to silently change it > (e.g. I know plenty of tables with columns separated by either one or > several > tabs depending on the length of the previous entry). But the tab is > apparently also > treated differently if explicitly specified with "delimiter='\t'" - > and in that case using > a converter ? la {2: lambda s: float(s or 'Nan')} is working for > fields in the middle of > the line, but not at the end - clearly warrants improvement. I've > prepared a patch > working for Python3 as well. Great! >> 1163: >> 1565: >> These tickets seem to have the same origin of the problem. I >> attach one possible patch. The previously suggested patches that >> I've seen will not correctly convert floats to ints, which I believe >> my patch will. > > +1, though I am a bit concerned that prompting to raise a ValueError > for every > element could impede performance. I'd probably still enclose it into an > if issubclass(typ, np.uint64) or issubclass(typ, np.int64): > just like in npio.patch. I also thought one might switch to > int(float128(x)) in that > case, but at least for the given examples float128 cannot convert with > more > accuracy than float64 (even on PowerPC ;-). > There were some dissenting opinions that trying to read a float into > an int should > generally throw an exception though... > > And Chuck just beat me... I am sure someone has been using this functionality to convert floats to ints. Changing will break their code. I am not sure how big a deal that would be. Also, I am of the opinion that one should _first_ write a program that works _correctly_, and only afterwards worry about performance. Even so, using numpy.int64 would be better, because it would give a sensible error message. > On 26 Mar 2011, at 21:25, Charles R Harris wrote: > >> I put all these patches together at https://github.com/charris/numpy/tree/loadtxt-savetxt >> . Please pull from there to continue work on loadtxt/savetxt so as >> to avoid conflicts in the patches. One of the numpy tests is >> failing, I assume from patch conflicts, and more tests for the >> tickets are needed in any case. Also, new keywords should be added >> to the end, not put in the middle of existing keywords. >> >> I haven't reviewed the patches, just tried to get them organized. >> Also, I have Derek as the author on all of them, that can be changed >> if it is decided the credit should go elsewhere ;) Thanks for the >> work you all have been doing on these tickets. > > Thanks, I'll have a look at the new ticket and try to get that > organized! With some luck, all the loadtxt tickets should be closed in short time :-) Best, Paul. From sturla at molden.no Sun Mar 27 14:54:52 2011 From: sturla at molden.no (Sturla Molden) Date: Sun, 27 Mar 2011 20:54:52 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D8E30EE.1010401@noaa.gov> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> Message-ID: <4D8F87FC.4010103@molden.no> Den 26.03.2011 19:31, skrev Christopher Barker: > To understand all this, you'll need to study up a bit on how numpy > arrays lay out and access the memory that they use: they use a concept > of "strided" memory. It's very powerful and flexible, but most other > numeric libs can't use those same data structures. With the ISO C bindings in Fortran 2003, we can get a Fortran pointer from a C pointer. This means it is actually possible to pass "strided memory" from NumPy to Fortran libraries in a portable way. A Fortran pointer is more similar to a NumPy view array than a C pointer. If I am not mistaken, fwrap can generate the boiler-plate Cython and Fortran code. Sturla From ralf.gommers at googlemail.com Sun Mar 27 15:21:09 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 27 Mar 2011 21:21:09 +0200 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error In-Reply-To: References: Message-ID: Hi Hoyt, Thanks for the thorough description of getting everything to work. On Sat, Mar 26, 2011 at 12:14 AM, Hoyt Koepke wrote: > Okay, even better news. ?After a little bit of probing around, it > turns out that some of the errors, namely the one in the interpolator, > were due to intel's handling of floating point values at higher > optimization levels. ?Thus the two rather important flags, if you wish > to avoid difficult-to-explain segfaults, are > > -fp-model strict ?( disables unsafe fp optimizations) > -fno-alias (C & C++, not fortran) > > Now here's a summary of the other errors: > > -ipo -- okay for linker / c++, but not elsewhere. ?For C, it causes > the long_double_representation() function in setup_common() to fail > (I'll note this on the corresponding ticket), and causes f2py to fail > to generate correct wrappers. Would it be enough to put a comment in the intelccompiler.py code not to use the -ipo flag, or do you think there's something to fix here? Doesn't seem to be a new problem by the way: http://cens.ioc.ee/pipermail/f2py-users/2006-January/001229.html > > -O3 -- okay for C++ / fortran, but not C. ?For C, it causes einsum to hang. -O3 is the default optimization level, so this is a bug I guess. There's also another report in #1378 that -O3 doesn't work with numpy 1.4.0 (which does not have einsum). Should it be lowered to -O2 by default? > Thus the highest optimization levels I could find in which everything > compiled and ran were: > > Fortran: -static -DMKL_LP64 -mkl -xHOST -O3 -fPIC ?-fp-model strict > C: ? ? -static -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict > C++: -static -DMKL_LP64 -mkl -xHOST -O3 -fno-alias -fPIC -fp-model strict > Linker: -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict > -openmp -lpthread I'm not sure which of those flags would be appropriate as a default in distutils, perhaps only fp-model-strict? If you could help put together a patch for numpy.distutils, that would be very helpful I think. The rest of your description could be put at http://scipy.org/Installing_SciPy/Linux. Ralf From hoytak at stat.washington.edu Sun Mar 27 15:46:03 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Sun, 27 Mar 2011 12:46:03 -0700 Subject: [Numpy-discussion] Compiling numpy using icc gets missing library error In-Reply-To: References: Message-ID: > Thanks for the thorough description of getting everything to work. You're welcome. I'm glad people find it helpful :-). >> -ipo -- okay for linker / c++, but not elsewhere. ?For C, it causes >> the long_double_representation() function in setup_common() to fail >> (I'll note this on the corresponding ticket), and causes f2py to fail >> to generate correct wrappers. > > Would it be enough to put a comment in the intelccompiler.py code not > to use the -ipo flag, or do you think there's something to fix here? > Doesn't seem to be a new problem by the way: > http://cens.ioc.ee/pipermail/f2py-users/2006-January/001229.html Well, to be honest, it's debatable how this should be fixed, if at all. The issue is that the intermediate object files and archives are in their own format -- hence the need for xiar instead of ar -- and I suspect that that's what's messing up f2py and the long_double_representation() stuff (I don't actually know how either really works, so I could be wrong here). Reverse engineering those formats doesn't make sense. Now it's possible that there may be a better way of determining long_double_representation(), but I don't know. IMO it's not worth it. >> -O3 -- okay for C++ / fortran, but not C. ?For C, it causes einsum to hang. > > -O3 is the default optimization level, so this is a bug I guess. > There's also another report in #1378 that -O3 doesn't work with numpy > 1.4.0 (which does not have einsum). Should it be lowered to -O2 by > default? I'm not sure what's going on here. I think it is likely a bug in icc brought out by the really low-level code in einsum. However, the documentation said that the extra optimizations enabled by -O3 are likely not going to help except in large, predictable loops where more aggressive loop transformations are doable and help. Thus my vote is to put in O3 for C++ and fortran, but O2 on C (because of this bug). However, if the einsum bug gets fixed, nice! (BTW, I think it's also be possible to change per-file optimization settings with intel specific pragmas, but I don't know for sure. If so, that could be a workaround.) >> Fortran: -static -DMKL_LP64 -mkl -xHOST -O3 -fPIC ?-fp-model strict >> C: ? ? -static -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict >> C++: -static -DMKL_LP64 -mkl -xHOST -O3 -fno-alias -fPIC -fp-model strict >> Linker: -DMKL_LP64 -mkl -xHOST -O2 -fno-alias -fPIC -fp-model strict >> -openmp -lpthread > > I'm not sure which of those flags would be appropriate as a default in > distutils, perhaps only fp-model-strict? If you could help put > together a patch for numpy.distutils, that would be very helpful I > think. The rest of your description could be put at > http://scipy.org/Installing_SciPy/Linux. -fp-model strict and -fno-alias, as it's analogous to the -fno-strict-aliasing gcc flag required by python. The -static flag is probably optional. -xHOST is the same as gcc's --march=native, so it seems like the nature of the distutils system isn't appropriate. I'll try to get a patch together for the distutils. However, I'd want someone to review it since I'm not that confident in my knowledge of the distutils code. I can also try to turn this into a more complete description for the wiki. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From d.s.seljebotn at astro.uio.no Mon Mar 28 03:34:11 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 28 Mar 2011 09:34:11 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D8F87FC.4010103@molden.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> Message-ID: <4D9039F3.3030608@student.matnat.uio.no> On 03/27/2011 08:54 PM, Sturla Molden wrote: > Den 26.03.2011 19:31, skrev Christopher Barker: > >> To understand all this, you'll need to study up a bit on how numpy >> arrays lay out and access the memory that they use: they use a concept >> of "strided" memory. It's very powerful and flexible, but most other >> numeric libs can't use those same data structures. > With the ISO C bindings in Fortran 2003, we can get a Fortran pointer > from a C pointer. This means it is actually possible to pass "strided > memory" from NumPy to Fortran libraries in a portable way. A Fortran > pointer is more similar to a NumPy view array than a C pointer. If I am > not mistaken, fwrap can generate the boiler-plate Cython and Fortran code. fwrap currently ensures that array are Fortran-contiguous. Supporting strided arrays has been in the back of our minds but not much thought has been given it really... What would we do exactly -- pass the entire underlying buffer to Fortran and then re-slice it Fortran side? That's more complicated than doing a copy Python-side, so unless the Fortran compiler actually supports keeping the array strided throughout the code it's not worth it. I'm not aware of the rules here -- basically, if you keep an array assumed-shape, it can be passed around in non-contiguous form in Fortran? And it will be copied whenever it is passed as an explicit-shape array? Dag Sverre From sturla at molden.no Mon Mar 28 06:55:59 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 12:55:59 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D9039F3.3030608@student.matnat.uio.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> Message-ID: <4D90693F.1080702@molden.no> Den 28.03.2011 09:34, skrev Dag Sverre Seljebotn: > What would we do exactly -- pass the entire underlying buffer to Fortran > and then re-slice it Fortran side? Pass a C pointer to the first element along with shape and strides, get a Fortran pointer using c_f_pointer, then reslice the Fortran pointer to a second Fortran pointer. Remember to argsort strides and dimentions on strides in ascending order. > That's more complicated than doing a > copy Python-side, so unless the Fortran compiler actually supports > keeping the array strided throughout the code it's not worth it. I'm not > aware of the rules here -- basically, if you keep an array > assumed-shape, it can be passed around in non-contiguous form in > Fortran? The Fortran standard does not specify this. A compiler is free to make a copy (copy-in copy-out), which is why a function call can invalidate a pointer. A compiler can decide to make a local copy, pass a dope array struct, or even remap virtual memory. It might e.g. depend on optimization rules, such as whether to favour speed or size. We cannot instruct a Fortran compiler to use strided memory, but we can allow it to use strided memory if it wants. > And it will be copied whenever it is passed as an > explicit-shape array? That is also compiler dependent. The standard just says that array indexing shall work. There is no requirement that explicit-shape arrays are contiguous. A compiler is free to use strided memory if it wants to here as well. Most compilers will assume that explicit-shape and assumed-size arrays are contiguous and passed as a pointer to the first element. But the standard does not require this, which is why there were no portable way of interfacing C and Fortran prior to Fortran 2003. Only a C array using Fortran 2003 C bindings is assumed contiguous by the standard. Apart from that, the standard does not care about the binary representation. f2py actually depends on knowing implementation details for common Fortran compilers, not a strandard C interface. I seem to remember that some Fortran 77 compilers even used virtual memory remapping to make strided memory to look contiguous to the callee, instead of making a local copy. That would be an alternative to a dope array, while presering a simple interface to C. So the answer is this is messy, the Fortran standard only require indexing to work as expected, and Fortran compilers can do whatever they want to achieve this. Just to repeat myself, "we cannot instruct a Fortran compiler to use strided memory, but we can allow it to use strided memory if it wants." It is indeed easier to make a local contiguous copy when calling Fortran. My experience when working with C is that "copy-in copy-out" is faster for computation than strided memory, particularly when it can fit in CPU cache. So just making a copy can be a good strategy, but it is depending on small array size. Generally I would say that it is a bad idea to try to be smarter than a Fortran compiler when it comes to decisions on array access. Those that make Fortran compilers have spent man years tuning this. So my preference is to just tell Fortran that memory is strided, and let it do whatever it wants. Sturla From d.s.seljebotn at astro.uio.no Mon Mar 28 08:28:41 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 28 Mar 2011 14:28:41 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D90693F.1080702@molden.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> Message-ID: <4D907EF9.2020909@student.matnat.uio.no> On 03/28/2011 12:55 PM, Sturla Molden wrote: > Den 28.03.2011 09:34, skrev Dag Sverre Seljebotn: >> What would we do exactly -- pass the entire underlying buffer to Fortran >> and then re-slice it Fortran side? > Pass a C pointer to the first element along with shape and strides, get > a Fortran pointer using c_f_pointer, then reslice the Fortran pointer to > a second Fortran pointer. > > Remember to argsort strides and dimentions on strides in ascending order. > >> That's more complicated than doing a >> copy Python-side, so unless the Fortran compiler actually supports >> keeping the array strided throughout the code it's not worth it. I'm not >> aware of the rules here -- basically, if you keep an array >> assumed-shape, it can be passed around in non-contiguous form in >> Fortran? > The Fortran standard does not specify this. A compiler is free to make a > copy (copy-in copy-out), which is why a function call can invalidate a > pointer. A compiler can decide to make a local copy, pass a dope array > struct, or even remap virtual memory. It might e.g. depend on > optimization rules, such as whether to favour speed or size. We cannot > instruct a Fortran compiler to use strided memory, but we can allow it > to use strided memory if it wants. > >> And it will be copied whenever it is passed as an >> explicit-shape array? > That is also compiler dependent. The standard just says that array > indexing shall work. There is no requirement that explicit-shape arrays > are contiguous. A compiler is free to use strided memory if it wants to > here as well. Most compilers will assume that explicit-shape and > assumed-size arrays are contiguous and passed as a pointer to the first > element. But the standard does not require this, which is why there were > no portable way of interfacing C and Fortran prior to Fortran 2003. Only > a C array using Fortran 2003 C bindings is assumed contiguous by the > standard. Apart from that, the standard does not care about the binary > representation. > > f2py actually depends on knowing implementation details for common > Fortran compilers, not a strandard C interface. > > I seem to remember that some Fortran 77 compilers even used virtual > memory remapping to make strided memory to look contiguous to the > callee, instead of making a local copy. That would be an alternative to > a dope array, while presering a simple interface to C. > > So the answer is this is messy, the Fortran standard only require > indexing to work as expected, and Fortran compilers can do whatever they > want to achieve this. Just to repeat myself, "we cannot instruct a > Fortran compiler to use strided memory, but we can allow it to use > strided memory if it wants." > > It is indeed easier to make a local contiguous copy when calling > Fortran. My experience when working with C is that "copy-in copy-out" is > faster for computation than strided memory, particularly when it can fit > in CPU cache. So just making a copy can be a good strategy, but it is > depending on small array size. Generally I would say that it is a bad > idea to try to be smarter than a Fortran compiler when it comes to > decisions on array access. Those that make Fortran compilers have spent > man years tuning this. So my preference is to just tell Fortran that > memory is strided, and let it do whatever it wants. Sure, I realize that it is not standard. I'm mostly wondering whether major Fortran compilers support working with strided memory in practice (defined as you won't get out-of-memory-errors when passing around huge strided array subset). If no compilers actually support it in practice, the priority for this functionality in Fwrap can be put fairly low. I totally agree it's the right thing to do, it's a question of priorities. Dag Sverre From sturla at molden.no Mon Mar 28 11:01:46 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 17:01:46 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D907EF9.2020909@student.matnat.uio.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> Message-ID: <4D90A2DA.5060400@molden.no> Den 28.03.2011 14:28, skrev Dag Sverre Seljebotn: > > Sure, I realize that it is not standard. I'm mostly wondering whether > major Fortran compilers support working with strided memory in practice > (defined as you won't get out-of-memory-errors when passing around huge > strided array subset). I'll try to clarify this: ** Most Fortran 90 compilers (and beyond) supports strided memory for assumed-shape and deferred-shape arrays. That is, arrays declared like a(:,:), and/or with 'allocatable' or 'pointer' attribute. They are usually passed as a 'dope array' C struct, i.e. similar to NumPy's PyArrayObject. These arrays are not a part of Fortran 77, but passing them to Fortran 77 subroutines is required to work. The compiler is free to make a local copy if it wants. ** Most Fortran 77 compilers (and beyond) assume explicit-shape and assumed-size arrays are contiguous blocks of memory. That is, arrays declared like a(m,n) or a(m,*). They are usually passed as a pointer to the first element. These are the only type of Fortran arrays f2py supports. ** Most Fortran compilers will make a temporary copy when passing a non-contiguous array section to a subroutine expecting an explicit-shape or assumed-shape array. ** All Fortran 2003 compilers will assume a C array is contiguous. Ok? Sturla From sturla at molden.no Mon Mar 28 11:06:12 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 17:06:12 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D90A2DA.5060400@molden.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> Message-ID: <4D90A3E4.7070003@molden.no> Den 28.03.2011 17:01, skrev Sturla Molden: > > ** Most Fortran compilers will make a temporary copy when passing a > non-contiguous array section to a subroutine expecting an explicit-shape > or assumed-shape array. Sorry, typo. The latter should be "assumed-size array". Sturla From sturla at molden.no Mon Mar 28 11:17:43 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 17:17:43 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: <4D907EF9.2020909@student.matnat.uio.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> Message-ID: <4D90A697.7030707@molden.no> Den 28.03.2011 14:28, skrev Dag Sverre Seljebotn: > > Sure, I realize that it is not standard. I'm mostly wondering whether > major Fortran compilers support working with strided memory in practice > (defined as you won't get out-of-memory-errors when passing around huge > strided array subset). On 64-bit systems one could also use virtual memory (re)mapping for this, i.e. a contiguous block of virtual memory could alias a strided block of virtual memory, avoiding a local copy. That could be convinient for passing huge arrays to Fortran 77 subroutines, e.g. LAPACK, for which most implementations require arrays to be contiguous. Sturla From giuseppe.aprea at gmail.com Mon Mar 28 12:34:17 2011 From: giuseppe.aprea at gmail.com (Giuseppe Aprea) Date: Mon, 28 Mar 2011 18:34:17 +0200 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> <201008180913.12594.eg@fft.be> Message-ID: Dear Paul Anton, thanks a lot for your suggestion. I was also successful with [blas] libraries = blas library_dirs = $PREFIX/gotoblas2/lib [lapack] libraries = lapack library_dirs = $PREFIX/gotoblas2/lib but I had compile clapack as a shared library and place a symbolic link ln -s $PREFIX/gotoblas2/lib/libgoto2.so $PREFIX/gotoblas2/lib/libblas.so I think your solution works better (the only difference is that I had to define LAPACK variable before installing numpy to avoid that warning about an optimized lapack lib missing), but what about source dirs? do you also set BLAS_SRC and LAPACK_SRC? how? cheers On Tue, Mar 22, 2011 at 11:18 PM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > I'm no expert, but I just pulled off the scipy+numpy+GotoBLAS2 > installation. From what I gather, the Makefile for libgoto2 downloads and > compiles the generic lapack from netlib. It also wraps lapack into > libgoto2.so/.a. I believe the idea is as long as the BLAS implementation > is fast(TM), the lapack performance will be good. > > To wit*, what I did was to tell numpy where libgoto2 was: > env BLAS=/path/to/libgoto2.so python setup.py install > Scipy also wants the path to lapack, which is wrapped inside libgoto2: > env BLAS=/path/to/libgoto2.so LAPACK=/path/to/libgoto2.so python setup.py > install > Afterwards, I added the path to LD_LIBRARY_PATH. This was on a linux > cluster, if that matters. At any rate, I can testify that it was not a big > job to get numpy and scipy working with goto blas. > > Good luck, > Paul. > > *) I have notes on this on a different computer, but not available right > now. > > > On Tue, Mar 22, 2011 at 10:13 AM, Giuseppe Aprea > wrote: > >> Hi all, >> >> I wonder if Peter finally got Gotoblas working with numpy. I am trying >> with gotoblas 1.13 installed in the same way: >> >> $ ls -R >> .: >> include lib >> >> ./include: >> goto >> >> ./include/goto: >> blaswrap.h cblas.h clapack.h f2c.h >> >> ./lib: >> libgoto2.a libgoto2_nehalemp-r1.13.a libgoto2_nehalemp-r1.13.so libgoto2.so >> >> and numpy 1.5.1 with this site.cfg >> >> [DEFAULT] >> library_dirs = >> /usr/local/gcc/4.5.2/gcc/lib64:/usr/local/gcc/4.5.2/gcc/lib:/usr/local/gcc/4.5.2/gcc/lib32:/usr/local/gcc/4.5.2/gotoblas2/1.13/lib:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/lib:/usr/local/gcc/4.5.2/fftw/3.2.2/lib >> include_dirs = >> /usr/local/gcc/4.5.2/gcc/include:/usr/local/gcc/4.5.2/gotoblas2/1.13/include/goto:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/include:/usr/local/gcc/4.5.2/fftw/3.2.2/include >> search_static_first = 1 >> [blas_opt] >> libraries = goto2 >> language = fortran >> [lapack_opt] >> libraries = goto2 >> language = fortran >> [amd] >> amd_libs = amd >> [umfpack] >> umfpack_libs = umfpack >> [fftw] >> libraries = fftw3 >> >> (I also tried without "_opt" and "language = fortran"); I used goto2 for >> lapack because I read lapack should be included in libgoto (anyway things do >> not change using "lapack"). I am quite sure the system is not using my >> goto-lapack stuff since every time I buid i get: >> >> building extension "numpy.numarray._capi" sources >> building extension "numpy.fft.fftpack_lite" sources >> building extension "numpy.linalg.lapack_lite" sources >> creating build/src.linux-x86_64-2.7/numpy/linalg >> *### Warning: Using unoptimized lapack ###* >> * adding 'numpy/linalg/lapack_litemodule.c' to sources.* >> adding 'numpy/linalg/python_xerbla.c' to sources. >> adding 'numpy/linalg/zlapack_lite.c' to sources. >> adding 'numpy/linalg/dlapack_lite.c' to sources. >> adding 'numpy/linalg/blas_lite.c' to sources. >> adding 'numpy/linalg/dlamch.c' to sources. >> adding 'numpy/linalg/f2c_lite.c' to sources. >> building extension "numpy.random.mtrand" sources >> creating build/src.linux-x86_64-2.7/numpy/random >> C compiler: /usr/local/gcc/4.5.2//gcc/bin/gcc -DNDEBUG -g -fwrapv -O3 >> -Wall -Wstrict-prototypes -O1 -pthread -fPIC -march=native -mtune=native >> -I/usr/local/gcc/4.5.2//gcc/include >> -I/usr/local/gcc/4.5.2//suiteSparse/3.6.0/include >> -I/usr/local/gcc/4.5.2//fftw/3.2.2/include -fPIC >> >> during numpy installation (which ends successfully). Moreover I cannot see >> any -lgoto2 as I would have expected. >> Incidentally, I cannot see -lamd, -lumfpack, -lfftw3 (or any reference >> to amd, umfpack, fftw3) neither, although there seems to be something to >> handle them in system_info.py. The failure is so complete that I must have >> done some big mistake but I can't correct my site.cfg even after searching >> the internet. This seems to be one of the major discussion about this topic >> so I am asking here for some help, please. >> Is the problem related with site.cfg or with gotoblas2 installation? Is it >> true that gotoblas2 hosts a full lapack inside? >> >> thank you very much! >> >> giuseppe >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Mon Mar 28 13:12:35 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Mon, 28 Mar 2011 20:12:35 +0300 Subject: [Numpy-discussion] Array views In-Reply-To: <4D90A2DA.5060400@molden.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> Message-ID: On Mon, Mar 28, 2011 at 6:01 PM, Sturla Molden wrote > > > I'll try to clarify this: > > ** Most Fortran 77 compilers (and beyond) assume explicit-shape and > assumed-size arrays are contiguous blocks of memory. That is, arrays > declared like a(m,n) or a(m,*). They are usually passed as a pointer to > the first element. These are the only type of Fortran arrays f2py supports. > FYI, f2py in numpy 1.6.x supports also assumed shape arrays. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From wickedgrey at gmail.com Mon Mar 28 13:45:11 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Mon, 28 Mar 2011 10:45:11 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) wrote: > On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: >> That said, I think starting a discussion with the Python core developers >> about the float16 type is worthwhile. There might be interest in supporting >> the float16 type in the struct interface, something that would be required >> as part of amending PEP 3118. > > That's something that wouldn't see production until Python 3.3, right? > ?I'm imagining that the buffer spec in use wouldn't change during the > 2.7.x series, unless it's unusually accepting of changes. Actually, thinking about this more, if the PEP gets updated with a backwards-compatible change, numpy could support the newer version, even on older versions of Python that don't officially, right? How quickly could something like the 'e' type be added to the PEP? Eli From butterw at gmail.com Mon Mar 28 15:34:19 2011 From: butterw at gmail.com (butterw at gmail.com) Date: Mon, 28 Mar 2011 19:34:19 +0000 Subject: [Numpy-discussion] stable sort on a recarray ? Message-ID: <0022158df61ba44ffc049f900732@google.com> What is the recommended way to perform a stable sort on a recarray / structured array ? In the following example, I want to sort on name, while retaining the existing order in the case of equal values: values = [('a', 1), ('a', 0), ('b', 2)] dtype = [('name', 'S10'), ('val', 'i4')] >>> a = np.array(values, dtype=dtype) array([('a', 1), ('a', 0), ('b', 2)], dtype=[('name', '|S10'), ('val', '>> np.sort(a, order='name') array([('a', 0), ('a', 1), ('b', 2)], dtype=[('name', '|S10'), ('val', ' From sturla at molden.no Mon Mar 28 15:44:49 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 21:44:49 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> Message-ID: <4D90E531.1000608@molden.no> Den 28.03.2011 19:12, skrev Pearu Peterson: > > FYI, f2py in numpy 1.6.x supports also assumed shape arrays. How did you do that? Chasm-interop, C bindings from F03, or marshalling through explicit-shape? Can f2py pass strided memory from NumPy to Fortran? Sturla From mwwiebe at gmail.com Mon Mar 28 18:54:53 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Mon, 28 Mar 2011 15:54:53 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Mon, Mar 28, 2011 at 10:45 AM, Eli Stevens (Gmail) wrote: > On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) > wrote: > > On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe wrote: > >> That said, I think starting a discussion with the Python core developers > >> about the float16 type is worthwhile. There might be interest in > supporting > >> the float16 type in the struct interface, something that would be > required > >> as part of amending PEP 3118. > > > > That's something that wouldn't see production until Python 3.3, right? > > I'm imagining that the buffer spec in use wouldn't change during the > > 2.7.x series, unless it's unusually accepting of changes. > > Actually, thinking about this more, if the PEP gets updated with a > backwards-compatible change, numpy could support the newer version, > even on older versions of Python that don't officially, right? > > How quickly could something like the 'e' type be added to the PEP? You've got to get the attention and buy-in from the right people for this to happen. I found this with a brief search: http://www.python.org/dev/peps/pep-0001/#reporting-pep-bugs-or-submitting-pep-updates -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Tue Mar 29 01:13:43 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 29 Mar 2011 08:13:43 +0300 Subject: [Numpy-discussion] Array views In-Reply-To: <4D90E531.1000608@molden.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> <4D90E531.1000608@molden.no> Message-ID: On Mon, Mar 28, 2011 at 10:44 PM, Sturla Molden wrote: > Den 28.03.2011 19:12, skrev Pearu Peterson: > > > > FYI, f2py in numpy 1.6.x supports also assumed shape arrays. > > How did you do that? Chasm-interop, C bindings from F03, or marshalling > through explicit-shape? > The latter. Basically, if you have subroutine foo(a) real a(:) end then f2py automatically creates a wrapper subroutine subroutine wrapfoo(a, n) real a(n) integer n !f2py intent(in) :: a !f2py intent(hide) :: n = shape(a,0) interface subroutine foo(a) real a(:) end end interface call foo(a) end that can be wrapped with f2py in ordinary way. > Can f2py pass strided memory from NumPy to Fortran? > > No. I haven't thought about it. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Mar 29 01:18:38 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 29 Mar 2011 01:18:38 -0400 Subject: [Numpy-discussion] stable sort on a recarray ? In-Reply-To: <0022158df61ba44ffc049f900732@google.com> References: <0022158df61ba44ffc049f900732@google.com> Message-ID: On Mon, Mar 28, 2011 at 3:34 PM, wrote: > What is the recommended way to perform a stable sort on a recarray / > structured array ? > > In the following example, I want to sort on name, while retaining the > existing order in the case of equal values: > values = [('a', 1), ('a', 0), ('b', 2)] > dtype = [('name', 'S10'), ('val', 'i4')] >>>> a = np.array(values, dtype=dtype) > array([('a', 1), ('a', 0), ('b', 2)], > dtype=[('name', '|S10'), ('val', '>>> np.sort(a, order='name') > array([('a', 0), ('a', 1), ('b', 2)], > dtype=[('name', '|S10'), ('val', '>> np.argsort(np.ones(10)) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> np.argsort(np.r_[2*np.ones(5),np.ones(5)]) array([5, 6, 7, 8, 9, 0, 1, 2, 3, 4]) according to the docs only mergesort is stable, so sorting only on one column should work, something like this might work x = array([('a', 0), ('a', 1), ('b', 2)], dtype=[('name', '|S10'), ('val', ' _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Tue Mar 29 02:29:04 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 29 Mar 2011 02:29:04 -0400 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 Message-ID: numpy/lib/test_io.py only uses StringIO in the test, no actual csv file If I give the filename than I get a TypeError: Can't convert 'bytes' object to str implicitly from the statsmodels mailing list example >>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) > Traceback (most recent call last): > File "", line 1, in > data = recfromtxt(open('./star98.csv', "U"), delimiter=",", > skip_header=1, dtype=float) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", > line 1633, in recfromtxt > output = genfromtxt(fname, **kwargs) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", > line 1181, in genfromtxt > first_values = split_line(first_line) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", > line 206, in _delimited_splitter > line = line.split(self.comments)[0].strip(asbytes(" \r\n")) > TypeError: Can't convert 'bytes' object to str implicitly > > line 1184 in npyio (py32 sourcefile) > > if isinstance(fname, str): > fhd = np.lib._datasource.open(fname, 'U') > > seems to be the culprit for my case changing to binary solved this problem for me fhd = np.lib._datasource.open(fname, 'Ub') (I still have other errors but don't know yet where they are coming from.) Almost all problem with porting statsmodels to python 3.2 so far are input related, mainly reading csv files which are heavily used in the tests. All the "real" code seems to work fine with numpy and scipy (and matplotlib so far) for python 3.2 Josef From pearu.peterson at gmail.com Tue Mar 29 03:35:45 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 29 Mar 2011 10:35:45 +0300 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> <4D90E531.1000608@molden.no> Message-ID: On Tue, Mar 29, 2011 at 8:13 AM, Pearu Peterson wrote: > > > On Mon, Mar 28, 2011 at 10:44 PM, Sturla Molden wrote: > >> Den 28.03.2011 19:12, skrev Pearu Peterson: >> > >> > FYI, f2py in numpy 1.6.x supports also assumed shape arrays. >> >> How did you do that? Chasm-interop, C bindings from F03, or marshalling >> through explicit-shape? >> > > The latter. > Basically, if you have > > subroutine foo(a) > real a(:) > end > > then f2py automatically creates a wrapper subroutine > > subroutine wrapfoo(a, n) > real a(n) > integer n > !f2py intent(in) :: a > !f2py intent(hide) :: n = shape(a,0) > interface > subroutine foo(a) > real a(:) > end > end interface > call foo(a) > end > > that can be wrapped with f2py in ordinary way. > > >> Can f2py pass strided memory from NumPy to Fortran? >> >> > No. I haven't thought about it. > > Now, after little bit of thinking and testing, I think supporting strided arrays in f2py is easily doable. For the example above, f2py just must generate the following wrapper subroutine subroutine wrapfoo(a, stride, n) real a(n) integer n, stride !f2py intent(in) :: a !f2py intent(hide) :: n = shape(a,0) !f2py intent(hide) :: stride = getstrideof(a) interface subroutine foo(a) real a(:) end end interface call foo(a(1:stride:n)) end Now the question is, how important this feature would be? How high I should put it in my todo list? If there is interest, the corresponding numpy ticket should be assigned to me. Best regards, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Tue Mar 29 04:03:56 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Mar 2011 10:03:56 +0200 Subject: [Numpy-discussion] Array views In-Reply-To: References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> <4D90E531.1000608@molden.no> Message-ID: <4D91926C.9070005@student.matnat.uio.no> On 03/29/2011 09:35 AM, Pearu Peterson wrote: > > > On Tue, Mar 29, 2011 at 8:13 AM, Pearu Peterson > > wrote: > > > > On Mon, Mar 28, 2011 at 10:44 PM, Sturla Molden > wrote: > > Den 28.03.2011 19:12, skrev Pearu Peterson: > > > > FYI, f2py in numpy 1.6.x supports also assumed shape arrays. > > How did you do that? Chasm-interop, C bindings from F03, or > marshalling > through explicit-shape? > > > The latter. > Basically, if you have > > subroutine foo(a) > real a(:) > end > > then f2py automatically creates a wrapper subroutine > > subroutine wrapfoo(a, n) > real a(n) > integer n > !f2py intent(in) :: a > !f2py intent(hide) :: n = shape(a,0) > interface > subroutine foo(a) > real a(:) > end > end interface > call foo(a) > end > > that can be wrapped with f2py in ordinary way. > > Can f2py pass strided memory from NumPy to Fortran? > > > No. I haven't thought about it. > > > Now, after little bit of thinking and testing, I think supporting > strided arrays in f2py > is easily doable. For the example above, f2py just must generate the > following wrapper subroutine > > subroutine wrapfoo(a, stride, n) > real a(n) > integer n, stride > !f2py intent(in) :: a > !f2py intent(hide) :: n = shape(a,0) > !f2py intent(hide) :: stride = getstrideof(a) > interface > subroutine foo(a) > real a(:) > end > end interface > call foo(a(1:stride:n)) > end I think it should be a(1:n*stride:stride) or something. Dag Sverre -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Tue Mar 29 04:09:50 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 29 Mar 2011 11:09:50 +0300 Subject: [Numpy-discussion] Array views In-Reply-To: <4D91926C.9070005@student.matnat.uio.no> References: <1301159442.11145.1434128601@webmail.messagingengine.com> <4D8E30EE.1010401@noaa.gov> <4D8F87FC.4010103@molden.no> <4D9039F3.3030608@student.matnat.uio.no> <4D90693F.1080702@molden.no> <4D907EF9.2020909@student.matnat.uio.no> <4D90A2DA.5060400@molden.no> <4D90E531.1000608@molden.no> <4D91926C.9070005@student.matnat.uio.no> Message-ID: On Tue, Mar 29, 2011 at 11:03 AM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > > I think it should be a(1:n*stride:stride) or something. > > Yes, it was my typo and I assumed that n is the length of the original array. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Mar 29 04:16:00 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 29 Mar 2011 04:16:00 -0400 Subject: [Numpy-discussion] npz load on numpy 3 Message-ID: more python 3.2 fun a npz file saved with python 2.6 (I guess) that I try to read with python 3.2 I have no clue, since I never use .npz files >>> arr = np.load(r"..\scikits\statsmodels\tsa\vector_ar\tests\results\vars_results.npz") >>> arr >>> dir(arr) ['__class__', '__contains__', '__del__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_files', 'close', 'f', 'fid', 'files', 'items', 'iteritems', 'iterkeys', 'keys', 'zip'] >>> arr.keys() ['causality', 'orthirf', 'detomega', 'nirfs', 'loglike', 'stderr', 'crit', 'phis', 'nahead', 'totobs', 'type', 'obs', 'irf', 'coefs'] >>> arr['irf'] Traceback (most recent call last): File "", line 1, in File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", line 222, in __getitem__ return format.read_array(value) File "C:\Programs\Python32\lib\site-packages\numpy\lib\format.py", line 449, in read_array array = pickle.load(fp) UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: ordinal not in range(128) >>> Any ideas ? Josef From pav at iki.fi Tue Mar 29 04:27:52 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 29 Mar 2011 08:27:52 +0000 (UTC) Subject: [Numpy-discussion] npz load on numpy 3 References: Message-ID: Tue, 29 Mar 2011 04:16:00 -0400, josef.pktd wrote: > Traceback (most recent call last): > File "", line 1, in > File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", > line 222, in __getitem__ > return format.read_array(value) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\format.py", > line 449, in read_array > array = pickle.load(fp) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: > ordinal not in range(128) [clip] > Any ideas ? There's some open() call that opens the file in text mode rather than in binary mode, I guess. Pauli From pav at iki.fi Tue Mar 29 04:37:00 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 29 Mar 2011 08:37:00 +0000 (UTC) Subject: [Numpy-discussion] npz load on numpy 3 References: Message-ID: Tue, 29 Mar 2011 08:27:52 +0000, Pauli Virtanen wrote: > Tue, 29 Mar 2011 04:16:00 -0400, josef.pktd wrote: >> Traceback (most recent call last): >> File "", line 1, in >> File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >> line 222, in __getitem__ >> return format.read_array(value) >> File "C:\Programs\Python32\lib\site-packages\numpy\lib\format.py", >> line 449, in read_array >> array = pickle.load(fp) >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: >> ordinal not in range(128) > [clip] >> Any ideas ? > > There's some open() call that opens the file in text mode rather than in > binary mode, I guess. Ah, that's not it. The problem is that pickled Numpy arrays are not backward compatible between Python 2 and 3 because of the string vs. unicode change --- the pickle.load() call should specify an encoding eg. pickle.load(fp, encoding='latin1'). This needs to be wrapped in a try-expect block so that it tries to load it with encoding='latin1' if the first attempt fails. -- Pauli Virtanen From ater1980 at gmail.com Tue Mar 29 05:00:35 2011 From: ater1980 at gmail.com (Alex Ter-Sarkissov) Date: Tue, 29 Mar 2011 22:00:35 +1300 Subject: [Numpy-discussion] random number genration Message-ID: If I want to generate a string of random bits with equal probability I run random.randint(0,2,size). What if I want a specific proportion of bits? In other words, each bit is 1 with probability p<1/2 and 0 with probability q=1-p? thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Tue Mar 29 06:29:42 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 29 Mar 2011 13:29:42 +0300 Subject: [Numpy-discussion] random number genration In-Reply-To: References: Message-ID: Hi, On Tue, Mar 29, 2011 at 12:00 PM, Alex Ter-Sarkissov wrote: > If I want to generate a string of random bits with equal probability I run > > random.randint(0,2,size). > > What if I want a specific proportion of bits? In other words, each bit is 1 > with probability p<1/2 and 0 with probability q=1-p? > Would it be sufficient to: In []: bs= ones(1e6, dtype= int) In []: bs[randint(0, 1e6, 1e5)]= 0 In []: bs.sum()/ 1e6 Out[]: 0.904706 Regards, eat > > thanks > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Tue Mar 29 06:42:11 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 29 Mar 2011 13:42:11 +0300 Subject: [Numpy-discussion] random number genration In-Reply-To: References: Message-ID: On Tue, Mar 29, 2011 at 1:29 PM, eat wrote: > Hi, > > On Tue, Mar 29, 2011 at 12:00 PM, Alex Ter-Sarkissov wrote: > >> If I want to generate a string of random bits with equal probability I run >> >> >> random.randint(0,2,size). >> >> What if I want a specific proportion of bits? In other words, each bit is >> 1 with probability p<1/2 and 0 with probability q=1-p? >> > Would it be sufficient to: > In []: bs= ones(1e6, dtype= int) > In []: bs[randint(0, 1e6, 1e5)]= 0 > In []: bs.sum()/ 1e6 > Out[]: 0.904706 > Or: In []: bs= ones(1e6, dtype= int) In []: bs[rand(1e6)< 1./ 10]= 0 In []: bs.sum()/ 1e6 Out[]: 0.89983 > > Regards, > eat > >> >> thanks >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Mar 29 08:56:33 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 14:56:33 +0200 Subject: [Numpy-discussion] random number genration In-Reply-To: References: Message-ID: <4D91D701.4050900@molden.no> Den 29.03.2011 11:00, skrev Alex Ter-Sarkissov: > If I want to generate a string of random bits with equal probability I > run > > random.randint(0,2,size). > > What if I want a specific proportion of bits? In other words, each bit > is 1 with probability p<1/2 and 0 with probability q=1-p? Does this work you? import numpy as np def randombits(n, p): b = (np.random.rand(n*8).reshape((n,8)) < p).astype(int) return (b << range(8)).sum(axis=1).astype(np.uint8) Sturla From sturla at molden.no Tue Mar 29 09:06:36 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 15:06:36 +0200 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D91D701.4050900@molden.no> References: <4D91D701.4050900@molden.no> Message-ID: <4D91D95C.6080106@molden.no> Den 29.03.2011 14:56, skrev Sturla Molden: > import numpy as np > def randombits(n, p): > b = (np.random.rand(n*8).reshape((n,8))< p).astype(int) > return (b<< range(8)).sum(axis=1).astype(np.uint8) n is the number of bytes. If you prefer to count in bits: def randombits(n, p): assert(n%8 == 0) b = (np.random.rand(n).reshape((n>>3,8))< p).astype(int) return (b<< range(8)).sum(axis=1).astype(np.uint8) Sturla From eric at depagne.org Tue Mar 29 09:29:35 2011 From: eric at depagne.org (=?utf-8?q?=C3=89ric_Depagne?=) Date: Tue, 29 Mar 2011 15:29:35 +0200 Subject: [Numpy-discussion] np.histogram on arrays. Message-ID: <201103291529.35478.eric@depagne.org> Hi all. Sorry if this question has already been asked. I've searched the archive, but could not find anything related, so here is my question. I'm using np.histogram on a 4000x4000 array, each with 200 bins. I do that on both dimensions, meaning I compute 8000 histograms. It takes around 5 seconds (which is of course quite fast). I was wondering why np.histogram does not accept an axis parameter so that it could work directly on the array without me having to write a loop. Or maybe did I miss some parameters using np.histogram. Thanks. ?ric. Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne eric at depagne.org From e.antero.tammi at gmail.com Tue Mar 29 09:42:51 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 29 Mar 2011 16:42:51 +0300 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: <201103291529.35478.eric@depagne.org> References: <201103291529.35478.eric@depagne.org> Message-ID: Hi, On Tue, Mar 29, 2011 at 4:29 PM, ?ric Depagne wrote: > Hi all. > > Sorry if this question has already been asked. I've searched the archive, > but > could not find anything related, so here is my question. > > I'm using np.histogram on a 4000x4000 array, each with 200 bins. I do that > on > both dimensions, meaning I compute 8000 histograms. It takes around 5 > seconds > (which is of course quite fast). > > I was wondering why np.histogram does not accept an axis parameter so that > it > could work directly on the array without me having to write a loop. > > Or maybe did I miss some parameters using np.histogram. > FWIW, have you considered to use http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogramdd.html#numpy.histogramdd Regards, eat > > Thanks. > > ?ric. > > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dplepage at gmail.com Tue Mar 29 09:46:05 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Tue, 29 Mar 2011 09:46:05 -0400 Subject: [Numpy-discussion] random number genration In-Reply-To: References: Message-ID: On Tue, Mar 29, 2011 at 5:00 AM, Alex Ter-Sarkissov wrote: > If I want to generate a string of random bits with equal probability I run > > random.randint(0,2,size). > > What if I want a specific proportion of bits? In other words, each bit is 1 > with probability p<1/2 and 0 with probability q=1-p? x = (np.random.random(size) < p) Setting p = .5 should produce the same results as np.random.randint(0,2,size). Note that this gives you an array of bools, not ints; use x.astype(int) if integers are important (or x.astype(np.uint8) if memory is an issue). HTH, Dan Lepage From eric at depagne.org Tue Mar 29 10:13:36 2011 From: eric at depagne.org (=?iso-8859-15?q?=C9ric_Depagne?=) Date: Tue, 29 Mar 2011 16:13:36 +0200 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: References: <201103291529.35478.eric@depagne.org> Message-ID: <201103291613.36520.eric@depagne.org> > FWIW, have you considered to use > http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogramdd.html# > numpy.histogramdd > > Regards, > eat > I tried, but I get a /usr/lib/pymodules/python2.6/numpy/lib/function_base.pyc in histogramdd(sample, bins, range, normed, weights) 370 # Reshape is used so that overlarge arrays 371 # will raise an error. --> 372 hist = zeros(nbin, float).reshape(-1) 373 374 # Compute the sample indices in the flattened histogram matrix. ValueError: sequence too large; must be smaller than 32 so I suspect my array is too big for histogramdd ?ric. -- Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne eric at depagne.org From sturla at molden.no Tue Mar 29 10:49:05 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 16:49:05 +0200 Subject: [Numpy-discussion] random number genration In-Reply-To: References: Message-ID: <4D91F161.2090100@molden.no> Den 29.03.2011 15:46, skrev Daniel Lepage: > > x = (np.random.random(size)< p) This will not work. A boolean array is not compactly stored, but an array of bytes. Only the first bit 0 is 1 with probability p, bits 1 to 7 bits are 1 with probability 0. We thus have to do this 8 times for each byte, shift left by range(8), and combine them with binary or. Also the main use of random bits is crypto, which requires the use of /dev/urandom or CrypGenRandom instead of Mersenne Twister (np.random.rand). Sturla From sturla at molden.no Tue Mar 29 10:52:20 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 16:52:20 +0200 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D91F161.2090100@molden.no> References: <4D91F161.2090100@molden.no> Message-ID: <4D91F224.5030003@molden.no> Den 29.03.2011 16:49, skrev Sturla Molden: "Only the first bit 0 is 1 with probability p, bits 1 to 7 bits are 1 with probability 0." That should read: "Only bit 0 is 1 with probability p, bits 1 to 7 are 1 with probability 0." Sorry :) Sturla From robert.kern at gmail.com Tue Mar 29 11:08:51 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 29 Mar 2011 10:08:51 -0500 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D91F161.2090100@molden.no> References: <4D91F161.2090100@molden.no> Message-ID: On Tue, Mar 29, 2011 at 09:49, Sturla Molden wrote: > Den 29.03.2011 15:46, skrev Daniel Lepage: >> >> x = (np.random.random(size)< ?p) > > This will not work. A boolean array is not compactly stored, but an > array of bytes. Only the first bit 0 is 1 with probability p, bits 1 to > 7 bits are 1 with probability 0. We thus have to do this 8 times for > each byte, shift left by range(8), and combine them with binary or. It's not clear that the OP really meant bits rather than just bools. Judging by the example that he tried first, it's likely that he just wants bools (or even just 0s and 1s) and not a real string of bits compacted into bytes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From e.antero.tammi at gmail.com Tue Mar 29 11:16:17 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 29 Mar 2011 18:16:17 +0300 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: <201103291613.36520.eric@depagne.org> References: <201103291529.35478.eric@depagne.org> <201103291613.36520.eric@depagne.org> Message-ID: Hi, On Tue, Mar 29, 2011 at 5:13 PM, ?ric Depagne wrote: > > FWIW, have you considered to use > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogramdd.html# > > numpy.histogramdd > > > > Regards, > > eat > > > I tried, but I get a > /usr/lib/pymodules/python2.6/numpy/lib/function_base.pyc in > histogramdd(sample, bins, range, normed, weights) > 370 # Reshape is used so that overlarge arrays > > 371 # will raise an error. > > --> 372 hist = zeros(nbin, float).reshape(-1) > 373 > 374 # Compute the sample indices in the flattened histogram matrix. > > > ValueError: sequence too large; must be smaller than 32 > > so I suspect my array is too big for histogramdd > So it seems that you give your array directly to histogramdd (asking a 4000D histogram!). Surely that's not what you are trying to achieve. Can you elaborate more on your objectives? Perhaps some code (slow but working) to demonstrate the point. Regards, eat > > ?ric. > -- > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Mar 29 12:59:55 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 18:59:55 +0200 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D91F161.2090100@molden.no> References: <4D91F161.2090100@molden.no> Message-ID: <4D92100B.7080308@molden.no> Den 29.03.2011 16:49, skrev Sturla Molden: > > This will not work. A boolean array is not compactly stored, but an > array of bytes. Only the first bit 0 is 1 with probability p, bits 1 to > 7 bits are 1 with probability 0. We thus have to do this 8 times for > each byte, shift left by range(8), and combine them with binary or. > Also the main use of random bits is crypto, which requires the use of > /dev/urandom or CrypGenRandom instead of Mersenne Twister (np.random.rand). Here's a cleaned-up one for those who might be interested :-) Sturla import numpy as np import os def randombits(n, p, intention='numerical'): """ Returns an array with packed bits drawn from n Bernoulli trials with successrate p. """ assert (intention in ('numerical','crypto')) # number of bytes m = (n >> 3) + 1 if n % 8 else n >> 3 if intention == 'numerical': # Mersenne Twister rflt = np.random.rand(m*8) else: # /dev/urandom on Linux, Apple, et al., # CryptGenRandom on Windows rflt = np.frombuffer(os.urandom(m*64),dtype=np.uint64) rflt = rflt / float(2**64) b = (rflt.reshape((m,8))> (m*8 - n)) return b def bitcount(a, bytewise=False): """ Count the number of set bits in an array of np.uint8. """ assert(a.dtype == np.uint8) c = a[:,np.newaxis].repeat(8, axis=1) >> range(8) & 0x01 return (c.sum(axis=1) if bytewise else c.sum()) if __name__ == '__main__': b = randombits(int(1e6), .1, intent='numerical') print bitcount(b) # should be close to 100000 b = randombits(int(1e6), .1, intent='crypto') print bitcount(b) # should be close to 100000 From warren.weckesser at enthought.com Tue Mar 29 13:10:34 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 29 Mar 2011 12:10:34 -0500 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D92100B.7080308@molden.no> References: <4D91F161.2090100@molden.no> <4D92100B.7080308@molden.no> Message-ID: On Tue, Mar 29, 2011 at 11:59 AM, Sturla Molden wrote: > Den 29.03.2011 16:49, skrev Sturla Molden: > > > > This will not work. A boolean array is not compactly stored, but an > > array of bytes. Only the first bit 0 is 1 with probability p, bits 1 to > > 7 bits are 1 with probability 0. We thus have to do this 8 times for > > each byte, shift left by range(8), and combine them with binary or. > > Also the main use of random bits is crypto, which requires the use of > > /dev/urandom or CrypGenRandom instead of Mersenne Twister > (np.random.rand). > > > Here's a cleaned-up one for those who might be interested :-) > How about adding it to http://www.scipy.org/Cookbook? Warren > > Sturla > > > > import numpy as np > import os > > > def randombits(n, p, intention='numerical'): > """ > Returns an array with packed bits drawn from n Bernoulli > trials with successrate p. > """ > assert (intention in ('numerical','crypto')) > # number of bytes > m = (n >> 3) + 1 if n % 8 else n >> 3 > if intention == 'numerical': > # Mersenne Twister > rflt = np.random.rand(m*8) > else: > # /dev/urandom on Linux, Apple, et al., > # CryptGenRandom on Windows > rflt = np.frombuffer(os.urandom(m*64),dtype=np.uint64) > rflt = rflt / float(2**64) > b = (rflt.reshape((m,8)) # pack the bits > b = (b< # zero the trailing m*8 - n bits > b[-1] &= (0xFF >> (m*8 - n)) > return b > > > def bitcount(a, bytewise=False): > """ > Count the number of set bits in an array of np.uint8. > """ > assert(a.dtype == np.uint8) > c = a[:,np.newaxis].repeat(8, axis=1) >> range(8) & 0x01 > return (c.sum(axis=1) if bytewise else c.sum()) > > > if __name__ == '__main__': > > b = randombits(int(1e6), .1, intent='numerical') > print bitcount(b) # should be close to 100000 > b = randombits(int(1e6), .1, intent='crypto') > print bitcount(b) # should be close to 100000 > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Tue Mar 29 13:10:34 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 29 Mar 2011 12:10:34 -0500 Subject: [Numpy-discussion] random number genration In-Reply-To: <4D92100B.7080308@molden.no> References: <4D91F161.2090100@molden.no> <4D92100B.7080308@molden.no> Message-ID: On Tue, Mar 29, 2011 at 11:59 AM, Sturla Molden wrote: > Den 29.03.2011 16:49, skrev Sturla Molden: > > > > This will not work. A boolean array is not compactly stored, but an > > array of bytes. Only the first bit 0 is 1 with probability p, bits 1 to > > 7 bits are 1 with probability 0. We thus have to do this 8 times for > > each byte, shift left by range(8), and combine them with binary or. > > Also the main use of random bits is crypto, which requires the use of > > /dev/urandom or CrypGenRandom instead of Mersenne Twister > (np.random.rand). > > > Here's a cleaned-up one for those who might be interested :-) > How about adding it to http://www.scipy.org/Cookbook? Warren > > Sturla > > > > import numpy as np > import os > > > def randombits(n, p, intention='numerical'): > """ > Returns an array with packed bits drawn from n Bernoulli > trials with successrate p. > """ > assert (intention in ('numerical','crypto')) > # number of bytes > m = (n >> 3) + 1 if n % 8 else n >> 3 > if intention == 'numerical': > # Mersenne Twister > rflt = np.random.rand(m*8) > else: > # /dev/urandom on Linux, Apple, et al., > # CryptGenRandom on Windows > rflt = np.frombuffer(os.urandom(m*64),dtype=np.uint64) > rflt = rflt / float(2**64) > b = (rflt.reshape((m,8)) # pack the bits > b = (b< # zero the trailing m*8 - n bits > b[-1] &= (0xFF >> (m*8 - n)) > return b > > > def bitcount(a, bytewise=False): > """ > Count the number of set bits in an array of np.uint8. > """ > assert(a.dtype == np.uint8) > c = a[:,np.newaxis].repeat(8, axis=1) >> range(8) & 0x01 > return (c.sum(axis=1) if bytewise else c.sum()) > > > if __name__ == '__main__': > > b = randombits(int(1e6), .1, intent='numerical') > print bitcount(b) # should be close to 100000 > b = randombits(int(1e6), .1, intent='crypto') > print bitcount(b) # should be close to 100000 > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From butterw at gmail.com Tue Mar 29 14:33:12 2011 From: butterw at gmail.com (butterw at gmail.com) Date: Tue, 29 Mar 2011 18:33:12 +0000 Subject: [Numpy-discussion] stable sort on a recarray ? Message-ID: <90e6ba615562f1337d049fa34a92@google.com> >> sortind = np.argsort(x['name'], kind='mergesort'); x[sortind] The indirect sorting method that was suggested works for doing stable sort on recarrays / structured arrays based on a single-column. # It is necessary to specify kind='mergesort' because argsort is not stable: > np.argsort(np.ones(100)) array([ 0, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 73, 51, 74, 76, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 75, 50, 49, 48, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 22, 23, 24, 25, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 98, 37, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 36, 99]) Any suggestions on how to achieve stable sort based on multiple columns with numpy ? a = array([('a', 1, 1), ('a', 0, 1), ('a', 0, 0), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '>> argsort(a, order=('name', 'x') array([2, 1, 0, 3]) # we get the expected result but the sort is not stable, desired result was: array([2, 0, 1, 3]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Mar 29 14:40:55 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 29 Mar 2011 13:40:55 -0500 Subject: [Numpy-discussion] stable sort on a recarray ? In-Reply-To: <90e6ba615562f1337d049fa34a92@google.com> References: <90e6ba615562f1337d049fa34a92@google.com> Message-ID: On Tue, Mar 29, 2011 at 13:33, wrote: > Any suggestions on how to achieve stable sort based on multiple columns with > numpy ? http://docs.scipy.org/doc/numpy/reference/generated/numpy.lexsort.html#numpy.lexsort It uses mergesort for stability. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From butterw at gmail.com Tue Mar 29 15:21:04 2011 From: butterw at gmail.com (Peter Butterworth) Date: Tue, 29 Mar 2011 21:21:04 +0200 Subject: [Numpy-discussion] stable sort on a recarray ? Message-ID: np.lexsort does the job for both the single or multi-column stable sort cases, thanks. a = np.array([('a', 1, 1), ('a', 0, 1), ('a', 0, 0), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '> sortind array([1, 2, 0, 3], dtype=int64) >> a[sortind] array([('a', 0, 1), ('a', 0, 0), ('a', 1, 1), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '> http://docs.scipy.org/doc/numpy/reference/generated/numpy.lexsort.html#numpy.lexsort . It is not mentioned on that page that lexsort is a stable sort. . no structured array / recarray example is given . it also states that "Structured arrays are sorted lexically by argsort", but fails to mention that the resulting sort is not stable. -- thanks, peter butterworth From ralf.gommers at googlemail.com Tue Mar 29 16:14:31 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 29 Mar 2011 22:14:31 +0200 Subject: [Numpy-discussion] histogram normed/density keyword (#1628) In-Reply-To: References: Message-ID: On Sun, Mar 27, 2011 at 11:56 AM, Ralf Gommers wrote: > Hi all, > > For the 1.6 release #1628 needs to be resolved. A while ago there was > a discussion about the normed keyword in histogram, which ATM has > changed behavior compared to numpy 1.5.1. The preferred fix as I read > the discussion (http://thread.gmane.org/gmane.comp.python.numeric.general/39746/focus=40089) > was to leave normed alone and add a new keyword density. > > It would be helpful if someone can volunteer to fix this. I've done the above, can some review this: https://github.com/rgommers/numpy/tree/histogram-density-kw Thanks, Ralf From stefan at sun.ac.za Tue Mar 29 18:16:04 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 30 Mar 2011 00:16:04 +0200 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Sun, Mar 27, 2011 at 12:09 PM, Paul Anton Letnes wrote: > I am sure someone has been using this functionality to convert floats to ints. Changing will break their code. I am not sure how big a deal that would be. Also, I am of the opinion that one should _first_ write a program that works _correctly_, and only afterwards worry about performance. While I'd agree in most cases, keep in mind that np.loadtxt is supposed to be a fast but simpler alternative to np.genfromtxt. If np.loadtxt becomes much slower, there's not much need to keep these separate any longer. Regards St?fan From rowen at uw.edu Tue Mar 29 18:22:31 2011 From: rowen at uw.edu (Russell E. Owen) Date: Tue, 29 Mar 2011 15:22:31 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 References: Message-ID: In article , Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first beta of NumPy > 1.6.0. Due to the extensive changes in the Numpy core for this > release, the beta testing phase will last at least one month. Please > test this beta and report any problems on the Numpy mailing list. > > Sources and binaries can be found at: > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ > For (preliminary) release notes see below. Great! FYI: it works for me on MacOS X 10.5.8 with python.org python 2.6.6: python setup.py build --fcompiler=gnu95 python setup.py install cd .. python -Wd -c 'import numpy; numpy.test()' NumPy version 1.6.0b1 NumPy is installed in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pack ages/numpy Python version 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1 (Apple Inc. build 5493)] nose version 0.11.4 ... Ran 3399 tests in 25.474s OK (KNOWNFAIL=3, SKIP=1) -- Russell From matthew.brett at gmail.com Tue Mar 29 21:39:54 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 29 Mar 2011 18:39:54 -0700 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 In-Reply-To: References: Message-ID: Hi, On Mon, Mar 28, 2011 at 11:29 PM, wrote: > numpy/lib/test_io.py ? ?only uses StringIO in the test, no actual csv file > > If I give the filename than I get a ?TypeError: Can't convert 'bytes' > object to str implicitly > > > from the statsmodels mailing list example > >>>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?data = recfromtxt(open('./star98.csv', "U"), delimiter=",", >> skip_header=1, dtype=float) >> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >> line 1633, in recfromtxt >> ? ?output = genfromtxt(fname, **kwargs) >> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >> line 1181, in genfromtxt >> ? ?first_values = split_line(first_line) >> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", >> line 206, in _delimited_splitter >> ? ?line = line.split(self.comments)[0].strip(asbytes(" \r\n")) >> TypeError: Can't convert 'bytes' object to str implicitly Is the right fix for this to open a 'filename' passed to genfromtxt, as 'binary' (bytes)? If so I will submit a pull request with a fix and a test, Best, Matthew From eric at depagne.org Wed Mar 30 03:04:39 2011 From: eric at depagne.org (=?iso-8859-15?q?=C9ric_Depagne?=) Date: Wed, 30 Mar 2011 09:04:39 +0200 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: References: <201103291529.35478.eric@depagne.org> <201103291613.36520.eric@depagne.org> Message-ID: <201103300904.39731.eric@depagne.org> Hi. Sorry for not having been clearer. I'll explain a little bit. I have 4k x 4k images that I want to analyse. I turn them into numpy arrays so I have 4k x 4k np.array. My analysis starts with determining the bias level. To do that, I compute for each line, and then for each row, an histogram. So I compute 8000 histograms. Here is the code I've used sofar: for i in range(self.data.shape[0]): #Compute an histogram along the columns # Gets counts and bounds self.countsC[i], self.boundsC[i] = np.histogram(data[i], bins=self.bins) for i in range(self.data.shape[1]): # Do the same, along the rows. self.countsR[i], self.boundsR[i] = np.histogram(data[:,i], bins=self.bins) And data.shape is (4000,4000). If histogram had an axis parameter, I could avoid the loop and I guess it would be faster. ?ric. > So it seems that you give your array directly to histogramdd (asking a > 4000D histogram!). Surely that's not what you are trying to achieve. Can > you elaborate more on your objectives? Perhaps some code (slow but > working) to demonstrate the point. > > Regards, > eat > Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne eric at depagne.org From thouis at gmail.com Wed Mar 30 05:48:51 2011 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Wed, 30 Mar 2011 11:48:51 +0200 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: <201103300904.39731.eric@depagne.org> References: <201103291529.35478.eric@depagne.org> <201103291613.36520.eric@depagne.org> <201103300904.39731.eric@depagne.org> Message-ID: How about something like this: # numpy 1.6 def rowhist(A, bins=100): assert (bins > 0) assert isinstance(bins, int) rownum = np.arange(A.shape[0]).reshape((-1, 1)).astype(int) * bins intA = (bins * (A - A.min()) / float(A.max() - A.min())).astype(int) intA[intA == bins] = bins - 1 return np.bincount((intA + rownum).flatten(), minlength=(A.shape[0]).reshape((A.shape[0], bins)) # numpy 1.5 def rowhist(A, bins=100): assert (bins > 0) assert isinstance(bins, int) rownum = np.arange(A.shape[0]).reshape((-1, 1)).astype(int) * bins intA = (bins * (A - A.min()) / float(A.max() - A.min())).astype(int) intA[intA == bins] = bins - 1 counts = np.zeros(A.shape[0] * bins) bc = np.bincount((intA + rownum).flatten()) counts[:len(bc)] = bc return counts.reshape((A.shape[0], bins)) On Wed, Mar 30, 2011 at 09:04, ?ric Depagne wrote: > Hi. > > Sorry for not having been clearer. I'll explain a little bit. > > I have 4k x 4k images that I want to analyse. I turn them into numpy arrays so > I have 4k x 4k np.array. > > My analysis starts with determining the bias level. To do that, I compute for > each line, and then for each row, an histogram. > So I compute 8000 histograms. > > Here is the code I've used sofar: > > ? ? ? ?for i in range(self.data.shape[0]): > ? ? ? ? ? #Compute an histogram along the columns > ? ? ? ? ? # Gets counts and bounds > ? ? ? ? ? ?self.countsC[i], self.boundsC[i] = np.histogram(data[i], > bins=self.bins) > ? ? ? ?for i in range(self.data.shape[1]): > ? ? ? ? ? ?# Do the same, along the rows. > ? ? ? ? ? ?self.countsR[i], self.boundsR[i] = np.histogram(data[:,i], > bins=self.bins) > > And data.shape is (4000,4000). > > If histogram ?had an axis parameter, I could avoid the loop and I guess it > would be faster. > > ?ric. >> So it seems that you give your array directly to histogramdd (asking a >> 4000D histogram!). Surely that's not what you are trying to achieve. Can >> you elaborate more on your objectives? Perhaps some code (slow but >> working) to demonstrate the point. >> >> Regards, >> eat >> > > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne ? ? ? ? ? ? ? ? ? ? ? ? ? ?eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From e.antero.tammi at gmail.com Wed Mar 30 06:26:38 2011 From: e.antero.tammi at gmail.com (eat) Date: Wed, 30 Mar 2011 13:26:38 +0300 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: <201103300904.39731.eric@depagne.org> References: <201103291529.35478.eric@depagne.org> <201103291613.36520.eric@depagne.org> <201103300904.39731.eric@depagne.org> Message-ID: Hi, On Wed, Mar 30, 2011 at 10:04 AM, ?ric Depagne wrote: > Hi. > > Sorry for not having been clearer. I'll explain a little bit. > > I have 4k x 4k images that I want to analyse. I turn them into numpy arrays > so > I have 4k x 4k np.array. > > My analysis starts with determining the bias level. To do that, I compute > for > each line, and then for each row, an histogram. > So I compute 8000 histograms. > > Here is the code I've used sofar: > > for i in range(self.data.shape[0]): > #Compute an histogram along the columns > # Gets counts and bounds > self.countsC[i], self.boundsC[i] = np.histogram(data[i], > bins=self.bins) > for i in range(self.data.shape[1]): > # Do the same, along the rows. > self.countsR[i], self.boundsR[i] = np.histogram(data[:,i], > bins=self.bins) > > And data.shape is (4000,4000). > > If histogram had an axis parameter, I could avoid the loop and I guess it > would be faster. > Well I guess, for a slight performance improvement, you could create your own streamlined histogrammer. But, in order to better grasp your situation it would be beneficial to know how the counts and bounds are used later on. Just wondering if this kind massive histogramming could be somehow avoided totally. Regards, eat > > ?ric. > > So it seems that you give your array directly to histogramdd (asking a > > 4000D histogram!). Surely that's not what you are trying to achieve. Can > > you elaborate more on your objectives? Perhaps some code (slow but > > working) to demonstrate the point. > > > > Regards, > > eat > > > > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed Mar 30 06:30:46 2011 From: andyfaff at gmail.com (andrew nelson) Date: Wed, 30 Mar 2011 21:30:46 +1100 Subject: [Numpy-discussion] Question regarding concatenate/vstack. In-Reply-To: References: <4D8A03BE.9010506@tartley.com> Message-ID: <3B611BE5-703F-45BE-8A5D-014FA558B38C@gmail.com> Dear List, I have a quick question regarding vstack and concatenate. In the docs for vstack it says that: np.concatenate(tup, axis=0) should be equivalent to: np.vstack(tup) However, I tried this out and it doesn't seem to be case, i.e. >>> np.vstack((np.arange(5.), np.arange(5.))) array([[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]]) >>> np.concatenate((np.arange(5.),np.arange(5.)), axis=0) array([ 0., 1., 2., 3., 4., 0., 1., 2., 3., 4.]) These aren't the same. Maybe I'm missing something? regards, Andrew. From eric at depagne.org Wed Mar 30 06:44:13 2011 From: eric at depagne.org (=?iso-8859-15?q?=C9ric_Depagne?=) Date: Wed, 30 Mar 2011 12:44:13 +0200 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: References: <201103291529.35478.eric@depagne.org> <201103300904.39731.eric@depagne.org> Message-ID: <201103301244.13721.eric@depagne.org> > > Well I guess, for a slight performance improvement, you could create your > own streamlined histogrammer. > > But, in order to better grasp your situation it would be beneficial to know > how the counts and bounds are used later on. Just wondering if this kind > massive histogramming could be somehow avoided totally. Indeed. Here's what I do. My images come from CCD, and as such, the zero level in the image is not the true zero level, but is the true zero + the background noise of each pixels. By doing the histogram, I plan on detecting what is the most common value per row. Once I have the most common value, I can derive the interval where most of the values are (the index of the largest occurence is easily obtained by sorting the counts, and I take a slice [index_max_count,index_max_count+1] in the second array given by the histogram). Then, I take the mean value of this interval and I assume it is the value of the bias for my row. I do this procedure both on the row and columns as a sanity check. And I know this procedure will not work if on any row/column there is a lot of signal and very little bias. I'll fix that afterwards ;-) ?ric. > > Regards, > eat > Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne eric at depagne.org From gruben at bigpond.net.au Wed Mar 30 07:42:11 2011 From: gruben at bigpond.net.au (gary ruben) Date: Wed, 30 Mar 2011 22:42:11 +1100 Subject: [Numpy-discussion] Question regarding concatenate/vstack. In-Reply-To: <3B611BE5-703F-45BE-8A5D-014FA558B38C@gmail.com> References: <4D8A03BE.9010506@tartley.com> <3B611BE5-703F-45BE-8A5D-014FA558B38C@gmail.com> Message-ID: You're right, they are not equivalent. vstack will happily create an array of higher rank than the parts it is stacking, whereas concatenate requires the arrays it is working with to already be at least 2d, so the equivalent is np.concatenate((np.arange(5.)[newaxis],np.arange(5.)[newaxis]), axis=0) or np.concatenate((np.atleast_2d(np.arange(5.)),np.atleast_2d(np.arange(5.))), axis=0) Gary R. On Wed, Mar 30, 2011 at 9:30 PM, andrew nelson wrote: > Dear List, > I have a quick question regarding vstack and concatenate. > In the docs for vstack it says that: > > np.concatenate(tup, axis=0) > > should be equivalent to: > > np.vstack(tup) > > However, I tried this out and it doesn't seem to be case, i.e. > >>>> np.vstack((np.arange(5.), np.arange(5.))) > array([[ 0., ?1., ?2., ?3., ?4.], > ? ? ? [ 0., ?1., ?2., ?3., ?4.]]) > >>>> np.concatenate((np.arange(5.),np.arange(5.)), axis=0) > array([ 0., ?1., ?2., ?3., ?4., ?0., ?1., ?2., ?3., ?4.]) > > These aren't the same. Maybe I'm missing something? > > regards, > Andrew. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Wed Mar 30 12:35:33 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 30 Mar 2011 18:35:33 +0200 Subject: [Numpy-discussion] Question regarding concatenate/vstack. In-Reply-To: References: <4D8A03BE.9010506@tartley.com> <3B611BE5-703F-45BE-8A5D-014FA558B38C@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 1:42 PM, gary ruben wrote: > You're right, they are not equivalent. vstack will happily create an > array of higher rank than the parts it is stacking, whereas > concatenate requires the arrays it is working with to already be at > least 2d, so the equivalent is > np.concatenate((np.arange(5.)[newaxis],np.arange(5.)[newaxis]), axis=0) > or > np.concatenate((np.atleast_2d(np.arange(5.)),np.atleast_2d(np.arange(5.))), > axis=0) This is fixed in the docstring now. Ralf > On Wed, Mar 30, 2011 at 9:30 PM, andrew nelson wrote: >> Dear List, >> I have a quick question regarding vstack and concatenate. >> In the docs for vstack it says that: >> >> np.concatenate(tup, axis=0) >> >> should be equivalent to: >> >> np.vstack(tup) >> >> However, I tried this out and it doesn't seem to be case, i.e. >> >>>>> np.vstack((np.arange(5.), np.arange(5.))) >> array([[ 0., ?1., ?2., ?3., ?4.], >> ? ? ? [ 0., ?1., ?2., ?3., ?4.]]) >> >>>>> np.concatenate((np.arange(5.),np.arange(5.)), axis=0) >> array([ 0., ?1., ?2., ?3., ?4., ?0., ?1., ?2., ?3., ?4.]) >> >> These aren't the same. Maybe I'm missing something? >> >> regards, >> Andrew. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Wed Mar 30 13:02:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 30 Mar 2011 19:02:25 +0200 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 3:39 AM, Matthew Brett wrote: > Hi, > > On Mon, Mar 28, 2011 at 11:29 PM, ? wrote: >> numpy/lib/test_io.py ? ?only uses StringIO in the test, no actual csv file >> >> If I give the filename than I get a ?TypeError: Can't convert 'bytes' >> object to str implicitly >> >> >> from the statsmodels mailing list example >> >>>>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ? ?data = recfromtxt(open('./star98.csv', "U"), delimiter=",", >>> skip_header=1, dtype=float) >>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>> line 1633, in recfromtxt >>> ? ?output = genfromtxt(fname, **kwargs) >>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>> line 1181, in genfromtxt >>> ? ?first_values = split_line(first_line) >>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", >>> line 206, in _delimited_splitter >>> ? ?line = line.split(self.comments)[0].strip(asbytes(" \r\n")) >>> TypeError: Can't convert 'bytes' object to str implicitly > > Is the right fix for this to open a 'filename' passed to genfromtxt, > as 'binary' (bytes)? > > If so I will submit a pull request with a fix and a test, Seems to work and is what was intended I think, see Pauli's changes/notes in commit 0f2e7db0. This is ticket #1607 by the way. Cheers, Ralf From joonpyro at gmail.com Wed Mar 30 13:12:18 2011 From: joonpyro at gmail.com (Joon Ro) Date: Wed, 30 Mar 2011 12:12:18 -0500 Subject: [Numpy-discussion] Warning: invalid value encountered in true_divide? Message-ID: Hi, After numpy upgrade, I started to get "Warning: invalid value encountered in true_divide," when I run a code which did now show any warning previously. What does it mean and where should I look to fix this? It does not stop my debugger so I could not identify where the message was from. Thank you, Joon -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 30 13:19:24 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Mar 2011 12:19:24 -0500 Subject: [Numpy-discussion] Warning: invalid value encountered in true_divide? In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 12:12, Joon Ro wrote: > Hi, > After numpy upgrade, I started to get "Warning: invalid value encountered in > true_divide," when I run a code which did now show any warning?previously. > What does it mean and where should I look to fix this? It means that a NaN popped up in a division somewhere. It always was there, but some previous versions of numpy had the warnings unintentionally silenced. > It does not stop my > debugger so I could not identify where the message was from. You can use np.seterr() to change how these warnings are printed. In particular, you can cause an exception to be raised so that you can use a debugger to locate the source. http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From matthew.brett at gmail.com Wed Mar 30 13:37:45 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 30 Mar 2011 10:37:45 -0700 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 In-Reply-To: References: Message-ID: Hi, On Wed, Mar 30, 2011 at 10:02 AM, Ralf Gommers wrote: > On Wed, Mar 30, 2011 at 3:39 AM, Matthew Brett wrote: >> Hi, >> >> On Mon, Mar 28, 2011 at 11:29 PM, ? wrote: >>> numpy/lib/test_io.py ? ?only uses StringIO in the test, no actual csv file >>> >>> If I give the filename than I get a ?TypeError: Can't convert 'bytes' >>> object to str implicitly >>> >>> >>> from the statsmodels mailing list example >>> >>>>>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) >>>> Traceback (most recent call last): >>>> ?File "", line 1, in >>>> ? ?data = recfromtxt(open('./star98.csv', "U"), delimiter=",", >>>> skip_header=1, dtype=float) >>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>>> line 1633, in recfromtxt >>>> ? ?output = genfromtxt(fname, **kwargs) >>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>>> line 1181, in genfromtxt >>>> ? ?first_values = split_line(first_line) >>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", >>>> line 206, in _delimited_splitter >>>> ? ?line = line.split(self.comments)[0].strip(asbytes(" \r\n")) >>>> TypeError: Can't convert 'bytes' object to str implicitly >> >> Is the right fix for this to open a 'filename' passed to genfromtxt, >> as 'binary' (bytes)? >> >> If so I will submit a pull request with a fix and a test, > > Seems to work and is what was intended I think, see Pauli's > changes/notes in commit 0f2e7db0. > > This is ticket #1607 by the way. Thanks for making a ticket. I've submitted a pull request for the fix and linked to it from the ticket. The reason I asked whether this was the correct fix was: imagine I'm working with a non-latin default encoding, and I've opened a file: fobj = open('my_nonlatin.txt', 'rt') in python 3.2. That might contain numbers and non-latin text. I can't pass that into 'genfromtxt' because it will give me this error above. I can pass it is as binary but then I'll get garbled text. Should those functions also allow unicode-providing files (perhaps with binary as default for speed)? See you, Matthew From ralf.gommers at googlemail.com Wed Mar 30 14:12:18 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 30 Mar 2011 20:12:18 +0200 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 7:37 PM, Matthew Brett wrote: > Hi, > > On Wed, Mar 30, 2011 at 10:02 AM, Ralf Gommers > wrote: >> On Wed, Mar 30, 2011 at 3:39 AM, Matthew Brett wrote: >>> Hi, >>> >>> On Mon, Mar 28, 2011 at 11:29 PM, ? wrote: >>>> numpy/lib/test_io.py ? ?only uses StringIO in the test, no actual csv file >>>> >>>> If I give the filename than I get a ?TypeError: Can't convert 'bytes' >>>> object to str implicitly >>>> >>>> >>>> from the statsmodels mailing list example >>>> >>>>>>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) >>>>> Traceback (most recent call last): >>>>> ?File "", line 1, in >>>>> ? ?data = recfromtxt(open('./star98.csv', "U"), delimiter=",", >>>>> skip_header=1, dtype=float) >>>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>>>> line 1633, in recfromtxt >>>>> ? ?output = genfromtxt(fname, **kwargs) >>>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", >>>>> line 1181, in genfromtxt >>>>> ? ?first_values = split_line(first_line) >>>>> ?File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", >>>>> line 206, in _delimited_splitter >>>>> ? ?line = line.split(self.comments)[0].strip(asbytes(" \r\n")) >>>>> TypeError: Can't convert 'bytes' object to str implicitly >>> >>> Is the right fix for this to open a 'filename' passed to genfromtxt, >>> as 'binary' (bytes)? >>> >>> If so I will submit a pull request with a fix and a test, >> >> Seems to work and is what was intended I think, see Pauli's >> changes/notes in commit 0f2e7db0. >> >> This is ticket #1607 by the way. > > Thanks for making a ticket. ?I've submitted a pull request for the fix > and linked to it from the ticket. > > The reason I asked whether this was the correct fix was: > > imagine I'm working with a non-latin default encoding, and I've opened a file: > > fobj = open('my_nonlatin.txt', 'rt') > > in python 3.2. ?That might contain numbers and non-latin text. ? I > can't pass that into 'genfromtxt' because it will give me this error > above. ?I can pass it is as binary but then I'll get garbled text. I admit the string/bytes thing is still a little confusing to me, but isn't that always going to be a problem (even with python 2.x)? There's no way for genfromtxt to know what the encoding of an arbitrary file is. So your choices are garbled text or an error. Garbled text is better. It may help to explicitly say in the docstring that this is an ASCII routine (as it does in the source code). Ralf > Should those functions also allow unicode-providing files (perhaps > with binary as default for speed)? From pav at iki.fi Wed Mar 30 14:32:34 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 30 Mar 2011 18:32:34 +0000 (UTC) Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 References: Message-ID: On Wed, 30 Mar 2011 10:37:45 -0700, Matthew Brett wrote: [clip] > imagine I'm working with a non-latin default encoding, and I've opened a > file: > > fobj = open('my_nonlatin.txt', 'rt') > > in python 3.2. That might contain numbers and non-latin text. I can't > pass that into 'genfromtxt' because it will give me this error above. I > can pass it is as binary but then I'll get garbled text. That's the way it also works on Python 2. The text is not garbled -- it's just in some binary representation that you can later on decode to unicode: >>> np.array(['asd']).view(np.chararray).decode('utf-8') array([u'asd'], dtype=' Should those functions also allow unicode-providing files (perhaps with > binary as default for speed)? Nobody has yet asked for this feature as far as I know, so I guess the need for it is pretty low. Personally, I don't think going unicode makes much sense here. First, it would be a Py3-only feature. Second, there is a real need for it only when dealing with multibyte encodings, which are seldom used these days with utf-8 rightfully dominating. -- Pauli Virtanen From bsouthey at gmail.com Wed Mar 30 15:37:08 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 30 Mar 2011 14:37:08 -0500 Subject: [Numpy-discussion] Old tickets Message-ID: <4D938664.6040404@gmail.com> Hi, This followup on tickets that I had previously indicated. So I want to thank Mark, Ralph and any people for going over those! For those that I followed I generally agreed with the outcome. Ticket 301: 'Make power and divide return floats from int inputs (like true_divide)' http://projects.scipy.org/numpy/ticket/301 Invalid because the output dtype is the same as the input dtype unless you override using the dtype argument: >>> np.power(3, 1, dtype=np.float128).dtype dtype('float128') Alternatively return a float and indicate in the docstring that the output dtype can be changed. Ticket 354: 'Possible inconsistency in 0-dim and scalar empty array types' http://projects.scipy.org/numpy/ticket/354 Invalid because an empty array is not the same as an empty string. Ticket 1071: 'loadtxt fails if the last column contains empty value' http://projects.scipy.org/numpy/ticket/1071 Invalid mainly because loadtxt states that 'Each row in the text file must have the same number of values.' So of cause loadtxt must fail when there are missing values. Ticket 1374: 'Ticket 628 not fixed for Solaris (polyfit uses 100% CPU and does not stop)' http://projects.scipy.org/numpy/ticket/1374 Unless this can be verified it should be set as needs_info. Bruce From matthew.brett at gmail.com Wed Mar 30 15:48:18 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 30 Mar 2011 12:48:18 -0700 Subject: [Numpy-discussion] bug in genfromtxt for python 3.2 In-Reply-To: References: Message-ID: Hi, On Wed, Mar 30, 2011 at 11:32 AM, Pauli Virtanen wrote: > On Wed, 30 Mar 2011 10:37:45 -0700, Matthew Brett wrote: > [clip] >> imagine I'm working with a non-latin default encoding, and I've opened a >> file: >> >> fobj = open('my_nonlatin.txt', 'rt') >> >> in python 3.2. ?That might contain numbers and non-latin text. ? I can't >> pass that into 'genfromtxt' because it will give me this error above. ?I >> can pass it is as binary but then I'll get garbled text. > > That's the way it also works on Python 2. The text is not garbled -- it's > just in some binary representation that you can later on decode to > unicode: > >>>> np.array(['asd']).view(np.chararray).decode('utf-8') > array([u'asd'], > ? ? ?dtype=' > Granted, utf-16 and the ilk might be problematic. > >> Should those functions also allow unicode-providing files (perhaps with >> binary as default for speed)? > > Nobody has yet asked for this feature as far as I know, so I guess the > need for it is pretty low. > > Personally, I don't think going unicode makes much sense here. First, it > would be a Py3-only feature. Second, there is a real need for it only > when dealing with multibyte encodings, which are seldom used these days > with utf-8 rightfully dominating. It's not a feature I need, but then, I'm afraid all the languages I've been taught are latin-1. Oh, except I learnt a tiny bit of Greek. But I don't use it for work :) I suppose the annoyances would be: 1) Probably temporary surprise that genfromtxt(open('my_file.txt', 'rt')) generates this error 2) Having to go back over returned arrays decoding stuff for utf-8 3) Wrong results for other encodings Maybe the best way is a graceful warning on entry to the routine? Best, Matthew From ralf.gommers at googlemail.com Wed Mar 30 17:03:01 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 30 Mar 2011 23:03:01 +0200 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 5:25 PM, Ralf Gommers wrote: > On Thu, Mar 24, 2011 at 5:11 PM, Robert Kern wrote: >> 2011/3/24 Dmitrey : >>>>>> from numpy import inf, array >>>>>> inf*0 >>> nan >>> >>> (ok) >>> >>>>>> array(inf) * 0.0 >>> StdErr: Warning: invalid value encountered in multiply >>> nan >>> >>> My cycled calculations yields this thousands times slowing computations and >>> making text output completely non-readable. >>> >>>>>> from numpy import __version__ >>>>>> __version__ >>> '2.0.0.dev-1fe8136' >> >> We really should change the default to 'warn' for numpy 2.0. Maybe >> even for numpy 1.6. We've talked about it before, and I think most >> people were in favor. We just never pulled the trigger. > > Old thread on this topic: > http://thread.gmane.org/gmane.comp.python.numeric.general/35664 >> >> Devs, what say you? > > Works for me, also for 1.6. Hi, just pinging this issue. If this is to happen for 1.6 it should go in the next beta (probably this weekend, only waiting for the genfromtxt issue to be resolved). Some more input would be good. As would a patch. Thanks, Ralf From ben.root at ou.edu Wed Mar 30 17:26:40 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 30 Mar 2011 16:26:40 -0500 Subject: [Numpy-discussion] Old tickets In-Reply-To: <4D938664.6040404@gmail.com> References: <4D938664.6040404@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 2:37 PM, Bruce Southey wrote: > Hi, > This followup on tickets that I had previously indicated. So I want to > thank Mark, Ralph and any people for going over those! > > For those that I followed I generally agreed with the outcome. > > Ticket 301: 'Make power and divide return floats from int inputs (like > true_divide)' > http://projects.scipy.org/numpy/ticket/301 > Invalid because the output dtype is the same as the input dtype unless > you override using the dtype argument: > >>> np.power(3, 1, dtype=np.float128).dtype > dtype('float128') > Alternatively return a float and indicate in the docstring that the > output dtype can be changed. > > FWIW, Just thought I'd note (on a python 2.6 system): >>> import numpy as np >>> a = np.array([1, 2, 3, 4]) >>> a.dtype dtype('int32') >>> 2 / a array([2, 1, 0, 0]) >>> from __future__ import division >>> 2 / a array([ 2. , 1. , 0.66666667, 0.5 ]) So, numpy already does this when true division is imported (and therefore consistent with whatever the python environment does), and python currently also returns integers for exponentials when both inputs are integers. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Wed Mar 30 17:52:56 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Wed, 30 Mar 2011 23:52:56 +0200 Subject: [Numpy-discussion] Old tickets In-Reply-To: References: <4D938664.6040404@gmail.com> Message-ID: On 30 Mar 2011, at 23:26, Benjamin Root wrote: > Ticket 301: 'Make power and divide return floats from int inputs (like > true_divide)' > http://projects.scipy.org/numpy/ticket/301 > Invalid because the output dtype is the same as the input dtype unless > you override using the dtype argument: > >>> np.power(3, 1, dtype=np.float128).dtype > dtype('float128') > Alternatively return a float and indicate in the docstring that the > output dtype can be changed. > > > FWIW, > > Just thought I'd note (on a python 2.6 system): > > >>> import numpy as np > >>> a = np.array([1, 2, 3, 4]) > >>> a.dtype > dtype('int32') > >>> 2 / a > array([2, 1, 0, 0]) > >>> from __future__ import division > >>> 2 / a > array([ 2. , 1. , 0.66666667, 0.5 ]) > > So, numpy already does this when true division is imported (and > therefore consistent with whatever the python environment does), and > python currently also returns integers for exponentials when both > inputs are integers. I'd agree, and in my view power(3, -1) is well defined as 1 / 3 - also, in future (or Python3) >>> a/2 array([ 0.5, 1. , 1.5, 2. ]) >>> a//2 array([0, 1, 1, 2], dtype=int32) so I think at least a**n should follow integer math rules; depends on whether we want np.power to behave differently from ** (if they are internally handled separately at all)... Not sure if I understand the overload suggestion in the ticket, but maybe a solution could be using the output argument (if an explicit optional dtype is not an option): >>> b = np.zeros(2, dtype=np.int32) >>> np.power(np.arange(1,3),-2, b) array([1, 0]) >>> b = np.zeros(2) >>> np.power(np.arange(1,3),-2, b) array([ 1., 0.]) ^^ this could be changed to array([ 1., 0.25]) Cheers, Derek From derek at astro.physik.uni-goettingen.de Wed Mar 30 18:09:33 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 31 Mar 2011 00:09:33 +0200 Subject: [Numpy-discussion] Old tickets In-Reply-To: <4D938664.6040404@gmail.com> References: <4D938664.6040404@gmail.com> Message-ID: Hi, On 30 Mar 2011, at 21:37, Bruce Southey wrote: > Ticket 1071: 'loadtxt fails if the last column contains empty value' > http://projects.scipy.org/numpy/ticket/1071 > Invalid mainly because loadtxt states that 'Each row in the text file > must have the same number of values.' So of cause loadtxt must fail > when > there are missing values. I don't follow the line of argument - see my comment on the ticket. This covers cases where missing values could always have been caught by the user - "Converters can also be used to provide a default value for missing data: ``converters = {3: lambda s: float(s or 0)}``." The ticket simply addresses the issue that "delimiter='\t'" is treated differently from other delimiters if (and only if) the missing value is the last item in the row. Cheers, Derek From robert.kern at gmail.com Wed Mar 30 18:51:32 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Mar 2011 17:51:32 -0500 Subject: [Numpy-discussion] should get rid of the annoying numpy STDERR output In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 16:03, Ralf Gommers wrote: > On Thu, Mar 24, 2011 at 5:25 PM, Ralf Gommers > wrote: >> On Thu, Mar 24, 2011 at 5:11 PM, Robert Kern wrote: >>> We really should change the default to 'warn' for numpy 2.0. Maybe >>> even for numpy 1.6. We've talked about it before, and I think most >>> people were in favor. We just never pulled the trigger. >> >> Old thread on this topic: >> http://thread.gmane.org/gmane.comp.python.numeric.general/35664 >>> >>> Devs, what say you? >> >> Works for me, also for 1.6. > > Hi, just pinging this issue. If this is to happen for 1.6 it should go > in the next beta (probably this weekend, only waiting for the > genfromtxt issue to be resolved). > > Some more input would be good. As would a patch. Patch: https://github.com/numpy/numpy/pull/65 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From wickedgrey at gmail.com Wed Mar 30 19:42:44 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Wed, 30 Mar 2011 16:42:44 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) wrote: > Can anyone please give me some suggestions on how to go about writing > a unit test for this? ?Or should I just submit a pull request? I've gotten a bit of positive feedback to adding the 'e' type to the struct module on the python-ideas list (per my understanding, not before python 3.3, but I don't think that should hinder adoption in other libraries), so I'd like to ask again about unit testing a change like this. Can anyone offer some advice for where to start? Also, what kind of timeframe / cutoff am I looking at to get this into 1.6.0 or 1.6.x? Thanks, Eli From charlesr.harris at gmail.com Wed Mar 30 22:53:14 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Mar 2011 20:53:14 -0600 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 26. mars 2011, at 21.44, Derek Homeier wrote: > > > Hi Paul, > > > > having had a look at the other tickets you dug up, > > > >> My opinions are my own, and in detail, they are: > >> 1752: > >> I attach a possible patch. FWIW, I agree with the request. The > >> patch is written to be compatible with the fix in ticket #1562, but > >> I did not test that yet. > > > > Tested, see also my comments on Trac. > > Great! > > >> 1731: > >> This seems like a rather trivial feature enhancement. I attach a > >> possible patch. > > > > Agreed. Haven't tested it though. > > Great! > > >> 1616: > >> The suggested patch seems reasonable to me, but I do not have a > >> full list of what objects loadtxt supports today as opposed to what > >> this patch will support. > > Looks like you got this one. Just remember to make it compatible with > #1752. Should be easy. > > >> 1562: > >> I attach a possible patch. This could also be the default > >> behavior to my mind, since the function caller can simply call > >> numpy.squeeze if needed. Changing default behavior would probably > >> break old code, however. > > > > See comments on Trac as well. > > Your patch is better, but there is one thing I disagree with. > 808 if X.ndim < ndmin: > 809 if ndmin == 1: > 810 X.shape = (X.size, ) > 811 elif ndmin == 2: > 812 X.shape = (X.size, 1) > The last line should be: > 812 X.shape = (1, X.size) > If someone wants a 2D array out, they would most likely expect a one-row > file to come out as a one-row array, not the other way around. IMHO. > > >> 1458: > >> The fix suggested in the ticket seems reasonable, but I have > >> never used record arrays, so I am not sure of this. > > > > There were some issues with Python3, and I also had some general > > reservations > > as noted on Trac - basically, it makes 'unpack' equivalent to > > transposing for 2D-arrays, > > but to splitting into fields for 1D-recarrays. My question was, what's > > going to happen > > when you get to 2D-recarrays? Currently this is not an issue since > > loadtxt can only > > read 2D regular or 1D structured arrays. But this might change if the > > data block > > functionality (see below) were to be implemented - data could then be > > returned as > > 3D arrays or 2D structured arrays... Still, it would probably make > > most sense (or at > > least give the widest functionality) to have 'unpack=True' always > > return a list or iterator > > over columns. > > OK, I don't know recarrays, as I said. > > >> 1445: > >> Adding this functionality could break old code, as some old > >> datafiles may have empty lines which are now simply ignored. I do > >> not think the feature is a good idea. It could rather be implemented > >> as a separate function. > >> 1107: > >> I do not see the need for this enhancement. In my eyes, the > >> usecols kwarg does this and more. Perhaps I am misunderstanding > >> something here. > > > > Agree about #1445, and the bit about 'usecols' - 'numcols' would just > > provide a > > shorter call to e.g. read the first 20 columns of a file (well, not > > even that much > > over 'usecols=range(20)'...), don't think that justifies an extra > > argument. > > But the 'datablocks' provides something new, that a number of people > > seem > > to miss from e.g. gnuplot (including me, actually ;-). And it would > > also satisfy the > > request from #1445 without breaking backwards compatibility. > > I've been wondering if could instead specify the separator lines > > through the > > parameter, e.g. "blocksep=['None', 'blank','invalid']", not sure if > > that would make > > it more useful... > > What about writing a separate function, e.g. loadblocktxt, and have it > separate the chunks and call loadtxt for each chunk? Just a thought. Another > possibility would be to write a function that would let you load a set of > text files in a directory, and return a dict of datasets, one per file. One > could write a similar save-function, too. They would just need to call > loadtxt/savetxt on a per-file basis. > > >> 1071: > >> It is not clear to me whether loadtxt is supposed to support > >> missing values in the fashion indicated in the ticket. > > > > In principle it should at least allow you to, by the use of converters > > as described there. > > The problem is, the default delimiter is described as 'any > > whitespace', which in the > > present implementation obviously includes any number of blanks or > > tabs. These > > are therefore treated differently from delimiters like ',' or '&'. I'd > > reckon there are > > too many people actually relying on this behaviour to silently change it > > (e.g. I know plenty of tables with columns separated by either one or > > several > > tabs depending on the length of the previous entry). But the tab is > > apparently also > > treated differently if explicitly specified with "delimiter='\t'" - > > and in that case using > > a converter ? la {2: lambda s: float(s or 'Nan')} is working for > > fields in the middle of > > the line, but not at the end - clearly warrants improvement. I've > > prepared a patch > > working for Python3 as well. > > Great! > > >> 1163: > >> 1565: > >> These tickets seem to have the same origin of the problem. I > >> attach one possible patch. The previously suggested patches that > >> I've seen will not correctly convert floats to ints, which I believe > >> my patch will. > > > > +1, though I am a bit concerned that prompting to raise a ValueError > > for every > > element could impede performance. I'd probably still enclose it into an > > if issubclass(typ, np.uint64) or issubclass(typ, np.int64): > > just like in npio.patch. I also thought one might switch to > > int(float128(x)) in that > > case, but at least for the given examples float128 cannot convert with > > more > > accuracy than float64 (even on PowerPC ;-). > > There were some dissenting opinions that trying to read a float into > > an int should > > generally throw an exception though... > > > > And Chuck just beat me... > > I am sure someone has been using this functionality to convert floats to > ints. Changing will break their code. I am not sure how big a deal that > would be. Also, I am of the opinion that one should _first_ write a program > that works _correctly_, and only afterwards worry about performance. Even > so, using numpy.int64 would be better, because it would give a sensible > error message. > > > On 26 Mar 2011, at 21:25, Charles R Harris wrote: > > > >> I put all these patches together at > https://github.com/charris/numpy/tree/loadtxt-savetxt > >> . Please pull from there to continue work on loadtxt/savetxt so as > >> to avoid conflicts in the patches. One of the numpy tests is > >> failing, I assume from patch conflicts, and more tests for the > >> tickets are needed in any case. Also, new keywords should be added > >> to the end, not put in the middle of existing keywords. > >> > >> I haven't reviewed the patches, just tried to get them organized. > >> Also, I have Derek as the author on all of them, that can be changed > >> if it is decided the credit should go elsewhere ;) Thanks for the > >> work you all have been doing on these tickets. > > > > Thanks, I'll have a look at the new ticket and try to get that > > organized! > > With some luck, all the loadtxt tickets should be closed in short time :-) > > Can some one bring me up to date on which tickets have been dealt with? Also, how do they divide into bug fixes and enhancements? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbellem at gmail.com Thu Mar 31 01:27:32 2011 From: sbellem at gmail.com (Sylvain Bellemare) Date: Wed, 30 Mar 2011 22:27:32 -0700 Subject: [Numpy-discussion] Willing to contribute to SciPy & NumPy ... Message-ID: Hi, I would like to seriously start contributing to NumPy and/or SciPy, as much as I possibly can. Because NumPy & SciPy are new to me, I'm not so sure whether I could be of some help. But if my help is welcomed, and if it makes sense to ask: "What could I best contribute to?", then perhaps some background information about myself would be useful ... *ACADEMICS* I completed an undergraduate degree in Computer Science, along with some mathematics and physics courses. For the mathematics, I did some Calculus, Linear Algebra, Differential Equations, Fourier series, Fourier & Laplace transform. In Physics, I did up to and including a first course in Quantum Mechanics. I also did some undergraduate research work, that involved computational high energy physics (proton-proton collision). The work was done with some Mathematica packages (FeynArts ). *INDUSTRY* I currently work in the software industry and for the past few months, have been performing both software development and system administration tasks. In terms of object-oriented programming, my main experience is with Java. I'm a beginner with Python. I used Python at work to automate the reverse-engineering process of some Java programs. Career-wise, one of my goals is to go into scientific computing, and use Python as the main programming language, and to perhaps pursue graduate studies in Computational Physics, hence my current interest for contributing to NumPy & SciPy. If you wish to know more about me I have a LinkedIn profile at http://www.linkedin.com/in/sylvainbellemare. *DOCUMENTATION* Documentation-wise, I can use LaTeX. For instance, my resume, which is attached with this email, was written with LaTeX. It's noteworthy to mention that I like to work on documentation, as long as it balances with coding work [?]. If that makes any sense to you and you have ideas on projects that I could contribute to, please let me know, and I'll be very happy to do my very best to contribute. Best regards, -Sylvain -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 328.png Type: image/png Size: 569 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sylvain.bellemare.pdf Type: application/pdf Size: 134849 bytes Desc: not available URL: From scott.sinclair.za at gmail.com Thu Mar 31 04:04:58 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 31 Mar 2011 10:04:58 +0200 Subject: [Numpy-discussion] Willing to contribute to SciPy & NumPy ... In-Reply-To: References: Message-ID: On 31 March 2011 07:27, Sylvain Bellemare wrote: > ?I would like to seriously start contributing to NumPy and/or SciPy, as much as I possibly can. I'm sure that your help will be welcomed! A good place to get started is helping out with documentation (see http://docs.scipy.org/numpy/Front%20Page/). SciPy has plenty of work required - you'll probably learn your way into the code that way. Another place to look is http://www.scipy.org/Developer_Zone. It's worthwhile learning how to work with Git and Github if you want to get patches accepted more easily. Cheers, Scott From cournape at gmail.com Thu Mar 31 05:19:39 2011 From: cournape at gmail.com (David Cournapeau) Date: Thu, 31 Mar 2011 18:19:39 +0900 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 7:22 AM, Russell E. Owen wrote: > In article > , > ?Ralf Gommers wrote: > >> Hi, >> >> I am pleased to announce the availability of the first beta of NumPy >> 1.6.0. Due to the extensive changes in the Numpy core for this >> release, the beta testing phase will last at least one month. Please >> test this beta and report any problems on the Numpy mailing list. >> >> Sources and binaries can be found at: >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >> For (preliminary) release notes see below. I see a segfault on Ubuntu 64 bits for the test TestAssumedShapeSumExample in numpy/f2py/tests/test_assumed_shape.py. Am I the only one seeing it ? David From e.antero.tammi at gmail.com Thu Mar 31 05:28:12 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 31 Mar 2011 12:28:12 +0300 Subject: [Numpy-discussion] np.histogram on arrays. In-Reply-To: <201103301244.13721.eric@depagne.org> References: <201103291529.35478.eric@depagne.org> <201103300904.39731.eric@depagne.org> <201103301244.13721.eric@depagne.org> Message-ID: Hi, On Wed, Mar 30, 2011 at 1:44 PM, ?ric Depagne wrote: > > > > Well I guess, for a slight performance improvement, you could create your > > own streamlined histogrammer. > > > > But, in order to better grasp your situation it would be beneficial to > know > > how the counts and bounds are used later on. Just wondering if this kind > > massive histogramming could be somehow avoided totally. > Indeed. > Here's what I do. > My images come from CCD, and as such, the zero level in the image is not > the > true zero level, but is the true zero + the background noise of each > pixels. > By doing the histogram, I plan on detecting what is the most common value > per > row. > Once I have the most common value, I can derive the interval where most of > the > values are (the index of the largest occurence is easily obtained by > sorting > the counts, and I take a slice [index_max_count,index_max_count+1] in the > second array given by the histogram). > Then, I take the mean value of this interval and I assume it is the value > of > the bias for my row. > > I do this procedure both on the row and columns as a sanity check. > And I know this procedure will not work if on any row/column there is a lot > of > signal and very little bias. I'll fix that afterwards ;-) > Perhaps something along these lines will help you: from numpy import histogram, linspace, r_ def hist2(a, nob): bins= linspace(a.min(), a.max(), nob+ 1) d= linspace(0, bins[-1]* a.shape[0], a.shape[0])[:, None] b= (a+ d).ravel() bbins= (bins[:-1]+ d).ravel() bbins= r_[bbins, bbins[-1]+ 1] counts, _= histogram(b, bbins) return counts.reshape(-1, nob), bins It has two disadvantages 1) needs more memory and 2) "global" bins (which although should be quite straightforward to enhance if needed). Regards, eat > > ?ric. > > > > > > Regards, > > eat > > > > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ater1980 at gmail.com Thu Mar 31 05:28:23 2011 From: ater1980 at gmail.com (Alex Ter-Sarkissov) Date: Thu, 31 Mar 2011 22:28:23 +1300 Subject: [Numpy-discussion] random number genration Message-ID: thanks Sturl, your second code works well although I had to divide the vector elements through 128 to get just the binaries -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Thu Mar 31 05:37:24 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Thu, 31 Mar 2011 12:37:24 +0300 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 12:19 PM, David Cournapeau wrote: > On Wed, Mar 30, 2011 at 7:22 AM, Russell E. Owen wrote: > > In article > > , > > Ralf Gommers wrote: > > > >> Hi, > >> > >> I am pleased to announce the availability of the first beta of NumPy > >> 1.6.0. Due to the extensive changes in the Numpy core for this > >> release, the beta testing phase will last at least one month. Please > >> test this beta and report any problems on the Numpy mailing list. > >> > >> Sources and binaries can be found at: > >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ > >> For (preliminary) release notes see below. > > I see a segfault on Ubuntu 64 bits for the test > TestAssumedShapeSumExample in numpy/f2py/tests/test_assumed_shape.py. > Am I the only one seeing it ? > > The test work here ok on Ubuntu 64 with numpy master. Could you try the maintenance/1.6.x branch where the related bugs are fixed. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Thu Mar 31 06:00:33 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 31 Mar 2011 12:00:33 +0200 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: On 31 March 2011 11:37, Pearu Peterson wrote: > > > On Thu, Mar 31, 2011 at 12:19 PM, David Cournapeau > wrote: >> >> On Wed, Mar 30, 2011 at 7:22 AM, Russell E. Owen wrote: >> > In article >> > , >> > ?Ralf Gommers wrote: >> > >> >> Hi, >> >> >> >> I am pleased to announce the availability of the first beta of NumPy >> >> 1.6.0. Due to the extensive changes in the Numpy core for this >> >> release, the beta testing phase will last at least one month. Please >> >> test this beta and report any problems on the Numpy mailing list. >> >> >> >> Sources and binaries can be found at: >> >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ >> >> For (preliminary) release notes see below. >> >> I see a segfault on Ubuntu 64 bits for the test >> TestAssumedShapeSumExample in numpy/f2py/tests/test_assumed_shape.py. >> Am I the only one seeing it ? >> > > The test work here ok on Ubuntu 64 with numpy master. Could you try the > maintenance/1.6.x branch where the related bugs are fixed. For what it's worth, the maintenance/1.6.x branch works for me on 64-bit Ubuntu: (numpy-1.6.x)scott at godzilla:~$ python -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.6.0b2.dev-a172fd6 NumPy is installed in /home/scott/.virtualenvs/numpy-1.6.x/lib/python2.6/site-packages/numpy Python version 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) [GCC 4.4.5] nose version 1.0.0 ---------------------------------------------------------------------- Ran 3406 tests in 16.889s OK (KNOWNFAIL=3, SKIP=4) Cheers, Scott From pearu.peterson at gmail.com Thu Mar 31 06:18:34 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Thu, 31 Mar 2011 13:18:34 +0300 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 1:00 PM, Scott Sinclair wrote: > On 31 March 2011 11:37, Pearu Peterson wrote: > > > > > > On Thu, Mar 31, 2011 at 12:19 PM, David Cournapeau > > wrote: > >> > >> On Wed, Mar 30, 2011 at 7:22 AM, Russell E. Owen wrote: > >> > In article > >> > , > >> > Ralf Gommers wrote: > >> > > >> >> Hi, > >> >> > >> >> I am pleased to announce the availability of the first beta of NumPy > >> >> 1.6.0. Due to the extensive changes in the Numpy core for this > >> >> release, the beta testing phase will last at least one month. Please > >> >> test this beta and report any problems on the Numpy mailing list. > >> >> > >> >> Sources and binaries can be found at: > >> >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ > >> >> For (preliminary) release notes see below. > >> > >> I see a segfault on Ubuntu 64 bits for the test > >> TestAssumedShapeSumExample in numpy/f2py/tests/test_assumed_shape.py. > >> Am I the only one seeing it ? > >> > > > > The test work here ok on Ubuntu 64 with numpy master. Could you try the > > maintenance/1.6.x branch where the related bugs are fixed. > > For what it's worth, the maintenance/1.6.x branch works for me on 64-bit > Ubuntu: > > (numpy-1.6.x)scott at godzilla:~$ python -c "import numpy; numpy.test()" > You might want to run python -c "import numpy; numpy.test('full')" as the corresponding test is decorated as slow. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.becker at amolf.nl Thu Mar 31 06:33:17 2011 From: n.becker at amolf.nl (Nils Becker) Date: Thu, 31 Mar 2011 12:33:17 +0200 Subject: [Numpy-discussion] np.histogramdd of empty data In-Reply-To: References: <4D9435C8.5040709@amolf.nl> Message-ID: <4D94586D.4080300@amolf.nl> Hi Ralf, I cloned numpy/master and played around a little. when giving the bins explicitely, now histogram2d and histogramdd work as expected in all tests i tried. However, some of the cases with missing bin specification appear somewhat inconsistent. The first question is if creating arbitrary bins for empty data and empty bin specification is better than raising an Exception: Specifically: numpy.histogram2d([],[],bins=[0,0]) > (array([ 0., 0.]), array([ 0.]), array([ 0.])) numpy.histogram([],bins=0) > ValueError: zero-size array to minimum.reduce without identity so 1-d and 2-d behave not quite the same. also, these work (although with arbitrary bin edges): numpy.histogram2d([],[],bins=[1,1]) > (array([ 0., 0.]), array([ 0., 1.]), array([ 0., 1.])) numpy.histogram2d([],[],bins=[0,1]) > (array([ 0., 0.]), array([ 0.]), array([ 0., 1.])) while this raises an error: numpy.histogram([],bins=1) > ValueError: zero-size array to minimum.reduce without identity another thing with non-empty data: numpy.histogram([1],bins=1) > (array([1]), array([ 0.5, 1.5])) numpy.histogram([1],bins=0) > (array([], dtype=int64), array([ 0.5])) while numpy.histogram2d([1],[1],bins=A) > ValueError: zero-size array to minimum.reduce without identity (here A==[0,0] or A==[0,1] but not A==[1,1] which gives a result) Nils From scott.sinclair.za at gmail.com Thu Mar 31 07:03:12 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 31 Mar 2011 13:03:12 +0200 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: On 31 March 2011 12:18, Pearu Peterson wrote: > > > On Thu, Mar 31, 2011 at 1:00 PM, Scott Sinclair > wrote: >> >> For what it's worth, the maintenance/1.6.x branch works for me on 64-bit >> Ubuntu: >> >> (numpy-1.6.x)scott at godzilla:~$ python -c "import numpy; numpy.test()" > > You might want to run > > ?python -c "import numpy; numpy.test('full')" > > as the corresponding test is decorated as slow. python -c "import numpy; numpy.test('full')" Running unit tests for numpy NumPy version 1.6.0b2.dev-a172fd6 NumPy is installed in /home/scott/.virtualenvs/numpy-1.6.x/lib/python2.6/site-packages/numpy Python version 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) [GCC 4.4.5] nose version 1.0.0 Ran 3423 tests in 28.713s OK (KNOWNFAIL=3, SKIP=4) Cheers, Scott From palla74 at gmail.com Thu Mar 31 07:30:31 2011 From: palla74 at gmail.com (Palla) Date: Thu, 31 Mar 2011 13:30:31 +0200 Subject: [Numpy-discussion] EuroPython 2011: call for paper is ending - Please spread the word Message-ID: Hi all, I'm Francesco and I am writing on behalf of "Python Italia APS", a no-profit association promoting EuroPython conference. (www.europython.eu) Europython End of Call for Presentations is April 6th. I'd like to ask to you to forward this mail to anyone that you feel may be interested. We're looking for proposals on every aspects of Python: programming from novice to advanced levels, applications and frameworks, or how you have been involved in introducing Python into your organisation. **First-time speakers are especially welcome**; EuroPython is a community conference and we are eager to hear about your experience. If you have friends or colleagues who have something valuable to contribute, twist their arms to tell us about it! Presenting at EuroPython ------------------------ We will accept a broad range of presentations, from reports on academic and commercial projects to tutorials and case studies. As long as the presentation is interesting and potentially useful to the Python community, it will be considered for inclusion in the programme. Can you show the conference-goers something new and useful? Can you show attendees how to: use a module? Explore a Python language feature? Package an application? If so, consider submitting a talk. Talks and hands-on trainings ---------------------------- There are two different kind of presentations that you can give as a speaker at EuroPython: * **Regular talk**. These are standard "talk with slides", allocated in slots of 45, 60 or 90 minutes, depending on your preference and scheduling constraints. A Q&A session is held at the end of the talk. * **Hands-on training**. These are advanced training sessions for a smaller audience (10-20 people), to dive into the subject with all details. These sessions are 4-hours long, and audience will be strongly encouraged to bring a laptop to experiment. They should be prepared with less slides and more source code. If possible, trainers will also give a short "teaser talk" of 30 minutes the day before the training, to tease delegates into attending the training. In the talk submission form, we assume that you intend to give a regular talk on the subject, but you will be asked if you are available for also doing a hands-on training on the very same subject. Speakers that will give a hands-on training are rewarded with a **free entrance** to EuroPython to compensate for the longer preparation required, and might also be eligible for a speaking fee (which we cannot confirm at the moment). Topics and goals ---------------- Specific topics for EuroPython presentations include, but are not limited to: - Core Python - Other implementations: Jython, IronPython, PyPy, and Stackless - Python libraries and extensions - Python 3.x migration - Databases - Documentation - GUI Programming - Game Programming - Network Programming - Open Source Python projects - Packaging Issues - Programming Tools - Project Best Practices - Embedding and Extending - Science and Math - Web-based Systems Presentation goals usually are some of the following: - Introduce audience to a new topic they are unaware of - Introduce audience to new developments on a well-known topic - Show audience real-world usage scenarios for a specific topic (case study) - Dig into advanced and relatively-unknown details on a topic - Compare different options in the market on a topic Community-based talk voting --------------------------- This year, for the first time in EuroPython history, the talk voting process is fully public. Every partecipant gains the right to vote for talks submitted during the Call For Papers, as soon as they commit to their presence at the conference by buying a ticket. See all the details in the talk voting[1] page. Contacts -------- For any further question, feel free to contact the organizers at info at pycon.it. Thank you! [1]: http://ep2011.europython.eu/talk-voting -- ->PALLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Mar 31 10:42:13 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 16:42:13 +0200 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 31, 2011 at 4:53 AM, Charles R Harris wrote: > > > On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes > wrote: >> >> On 26. mars 2011, at 21.44, Derek Homeier wrote: >> >> > Hi Paul, >> > >> > having had a look at the other tickets you dug up, >> > >> >> My opinions are my own, and in detail, they are: >> >> 1752: >> >> ? I attach a possible patch. FWIW, I agree with the request. The >> >> patch is written to be compatible with the fix in ticket #1562, but >> >> I did not test that yet. >> > >> > Tested, see also my comments on Trac. >> >> Great! >> >> >> 1731: >> >> ? This seems like a rather trivial feature enhancement. I attach a >> >> possible patch. >> > >> > Agreed. Haven't tested it though. >> >> Great! >> >> >> 1616: >> >> ? The suggested patch seems reasonable to me, but I do not have a >> >> full list of what objects loadtxt supports today as opposed to what >> >> this patch will support. >> >> Looks like you got this one. Just remember to make it compatible with >> #1752. Should be easy. >> >> >> 1562: >> >> ? I attach a possible patch. This could also be the default >> >> behavior to my mind, since the function caller can simply call >> >> numpy.squeeze if needed. Changing default behavior would probably >> >> break old code, however. >> > >> > See comments on Trac as well. >> >> Your patch is better, but there is one thing I disagree with. >> 808 ? ?if X.ndim < ndmin: >> 809 ? ? ? ?if ndmin == 1: >> 810 ? ? ? ? ? ?X.shape = (X.size, ) >> 811 ? ? ? ?elif ndmin == 2: >> 812 ? ? ? ? ? ?X.shape = (X.size, 1) >> The last line should be: >> 812 ? ? ? ? ? ?X.shape = (1, X.size) >> If someone wants a 2D array out, they would most likely expect a one-row >> file to come out as a one-row array, not the other way around. IMHO. >> >> >> 1458: >> >> ? The fix suggested in the ticket seems reasonable, but I have >> >> never used record arrays, so I am not sure ?of this. >> > >> > There were some issues with Python3, and I also had some general >> > reservations >> > as noted on Trac - basically, it makes 'unpack' equivalent to >> > transposing for 2D-arrays, >> > but to splitting into fields for 1D-recarrays. My question was, what's >> > going to happen >> > when you get to 2D-recarrays? Currently this is not an issue since >> > loadtxt can only >> > read 2D regular or 1D structured arrays. But this might change if the >> > data block >> > functionality (see below) were to be implemented - data could then be >> > returned as >> > 3D arrays or 2D structured arrays... Still, it would probably make >> > most sense (or at >> > least give the widest functionality) to have 'unpack=True' always >> > return a list or iterator >> > over columns. >> >> OK, I don't know recarrays, as I said. >> >> >> 1445: >> >> ? Adding this functionality could break old code, as some old >> >> datafiles may have empty lines which are now simply ignored. I do >> >> not think the feature is a good idea. It could rather be implemented >> >> as a separate function. >> >> 1107: >> >> ? I do not see the need for this enhancement. In my eyes, the >> >> usecols kwarg does this and more. Perhaps I am misunderstanding >> >> something here. >> > >> > Agree about #1445, and the bit about 'usecols' - 'numcols' would just >> > provide a >> > shorter call to e.g. read the first 20 columns of a file (well, not >> > even that much >> > over 'usecols=range(20)'...), don't think that justifies an extra >> > argument. >> > But the 'datablocks' provides something new, that a number of people >> > seem >> > to miss from e.g. gnuplot (including me, actually ;-). And it would >> > also satisfy the >> > request from #1445 without breaking backwards compatibility. >> > I've been wondering if could instead specify the separator lines >> > through the >> > parameter, e.g. "blocksep=['None', 'blank','invalid']", not sure if >> > that would make >> > it more useful... >> >> What about writing a separate function, e.g. loadblocktxt, and have it >> separate the chunks and call loadtxt for each chunk? Just a thought. Another >> possibility would be to write a function that would let you load a set of >> text files in a directory, and return a dict of datasets, one per file. One >> could write a similar save-function, too. They would just need to call >> loadtxt/savetxt on a per-file basis. >> >> >> 1071: >> >> ? ? ?It is not clear to me whether loadtxt is supposed to support >> >> missing values in the fashion indicated in the ticket. >> > >> > In principle it should at least allow you to, by the use of converters >> > as described there. >> > The problem is, the default delimiter is described as 'any >> > whitespace', which in the >> > present implementation obviously includes any number of blanks or >> > tabs. These >> > are therefore treated differently from delimiters like ',' or '&'. I'd >> > reckon there are >> > too many people actually relying on this behaviour to silently change it >> > (e.g. I know plenty of tables with columns separated by either one or >> > several >> > tabs depending on the length of the previous entry). But the tab is >> > apparently also >> > treated differently if explicitly specified with "delimiter='\t'" - >> > and in that case using >> > a converter ? la {2: lambda s: float(s or 'Nan')} is working for >> > fields in the middle of >> > the line, but not at the end - clearly warrants improvement. I've >> > prepared a patch >> > working for Python3 as well. >> >> Great! >> >> >> 1163: >> >> 1565: >> >> ? These tickets seem to have the same origin of the problem. I >> >> attach one possible patch. The previously suggested patches that >> >> I've seen will not correctly convert floats to ints, which I believe >> >> my patch will. >> > >> > +1, though I am a bit concerned that prompting to raise a ValueError >> > for every >> > element could impede performance. I'd probably still enclose it into an >> > if issubclass(typ, np.uint64) or issubclass(typ, np.int64): >> > just like in npio.patch. I also thought one might switch to >> > int(float128(x)) in that >> > case, but at least for the given examples float128 cannot convert with >> > more >> > accuracy than float64 (even on PowerPC ;-). >> > There were some dissenting opinions that trying to read a float into >> > an int should >> > generally throw an exception though... >> > >> > And Chuck just beat me... >> >> I am sure someone has been using this functionality to convert floats to >> ints. Changing will break their code. I am not sure how big a deal that >> would be. Also, I am of the opinion that one should _first_ write a program >> that works _correctly_, and only afterwards worry about performance. Even >> so, using numpy.int64 would be better, because it would give a sensible >> error message. >> >> > On 26 Mar 2011, at 21:25, Charles R Harris wrote: >> > >> >> I put all these patches together at >> >> https://github.com/charris/numpy/tree/loadtxt-savetxt >> >> . Please pull from there to continue work on loadtxt/savetxt so as >> >> to avoid conflicts in the patches. One of the numpy tests is >> >> failing, I assume from patch conflicts, and more tests for the >> >> tickets are needed in any case. Also, new keywords should be added >> >> to the end, not put in the middle of existing keywords. >> >> >> >> I haven't reviewed the patches, just tried to get them organized. >> >> Also, I have Derek as the author on all of them, that can be changed >> >> if it is decided the credit should go elsewhere ;) Thanks for the >> >> work you all have been doing on these tickets. >> > >> > Thanks, I'll have a look at the new ticket and try to get that >> > organized! >> >> With some luck, all the loadtxt tickets should be closed in short time :-) >> > > Can some one bring me up to date on which tickets have been dealt with? > Also, how do they divide into bug fixes and enhancements? If you look in Trac under "All Tickets by Milestone" you'll find all nine tickets together under 1.6.0. Five are bug fixes, four are enhancements. There are some missing tests, but all tickets have proposed patches. Cheers, Ralf From ralf.gommers at googlemail.com Thu Mar 31 11:00:10 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 17:00:10 +0200 Subject: [Numpy-discussion] Old tickets In-Reply-To: References: <4D938664.6040404@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 11:52 PM, Derek Homeier wrote: > > On 30 Mar 2011, at 23:26, Benjamin Root wrote: > >> Ticket 301: 'Make power and divide return floats from int inputs (like >> true_divide)' >> http://projects.scipy.org/numpy/ticket/301 >> Invalid because the output dtype is the same as the input dtype unless >> you override using the dtype argument: >> ?>>> np.power(3, 1, dtype=np.float128).dtype >> dtype('float128') >> Alternatively return a float and indicate in the docstring that the >> output dtype can be changed. >> >> >> FWIW, >> >> Just thought I'd note (on a python 2.6 system): >> >> >>> import numpy as np >> >>> a = np.array([1, 2, 3, 4]) >> >>> a.dtype >> dtype('int32') >> >>> 2 / a >> array([2, 1, 0, 0]) >> >>> from __future__ import division >> >>> 2 / a >> array([ 2. ? ? ? ?, ?1. ? ? ? ?, ?0.66666667, ?0.5 ? ? ? ]) >> >> So, numpy already does this when true division is imported (and >> therefore consistent with whatever the python environment does), and >> python currently also returns integers for exponentials when both >> inputs are integers. > > I'd agree, and in my view power(3, -1) is well defined as 1 / 3 - > also, in future (or Python3) > > ?>>> a/2 > array([ 0.5, ?1. , ?1.5, ?2. ]) > ?>>> a//2 > array([0, 1, 1, 2], dtype=int32) The ticket is about the functions np.divide and np.power, not / and **. This currently does not work the same, unlike what's said above: >>> from __future__ import division >>> x = np.arange(4) + 1 >>> 2 / x array([ 2. , 1. , 0.66666667, 0.5 ]) >>> np.divide(2, x) array([2, 1, 0, 0]) The power and divide functions should not check the "from __future__ import division", just return floats IMHO. This is what's expected by most users, and much more useful than integer math. > so I think at least a**n should follow integer math rules; depends on > whether we want > np.power to behave differently from ** (if they are internally handled > separately at all)... > Not sure if I understand the overload suggestion in the ticket I don't understand that either, it'll just lead to more confusion. Cheers, Ralf From bsouthey at gmail.com Thu Mar 31 11:03:11 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 31 Mar 2011 10:03:11 -0500 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Wed, Mar 30, 2011 at 9:53 PM, Charles R Harris wrote: > > > On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes > wrote: >> >> On 26. mars 2011, at 21.44, Derek Homeier wrote: >> >> > Hi Paul, >> > >> > having had a look at the other tickets you dug up, >> > [snip] >> >> >> 1071: >> >> ? ? ?It is not clear to me whether loadtxt is supposed to support >> >> missing values in the fashion indicated in the ticket. >> > >> > In principle it should at least allow you to, by the use of converters >> > as described there. >> > The problem is, the default delimiter is described as 'any >> > whitespace', which in the >> > present implementation obviously includes any number of blanks or >> > tabs. These >> > are therefore treated differently from delimiters like ',' or '&'. I'd >> > reckon there are >> > too many people actually relying on this behaviour to silently change it >> > (e.g. I know plenty of tables with columns separated by either one or >> > several >> > tabs depending on the length of the previous entry). But the tab is >> > apparently also >> > treated differently if explicitly specified with "delimiter='\t'" - >> > and in that case using >> > a converter ? la {2: lambda s: float(s or 'Nan')} is working for >> > fields in the middle of >> > the line, but not at the end - clearly warrants improvement. I've >> > prepared a patch >> > working for Python3 as well. >> >> Great! >> This is an invalid ticket because the docstring clearly states that in 3 different, yet critical places, that missing values are not handled here: "Each row in the text file must have the same number of values." "genfromtxt : Load data with missing values handled as specified." " This function aims to be a fast reader for simply formatted files. The `genfromtxt` function provides more sophisticated handling of, e.g., lines with missing values." Really I am trying to separate the usage of loadtxt and genfromtxt to avoid unnecessary duplication and confusion. Part of this is historical because loadtxt was added in 2007 and genfromtxt was added in 2009. So really certain features of loadtxt have been 'kept' for backwards compatibility purposes yet these features can be 'abused' to handle missing data. But I really consider that any missing values should cause loadtxt to fail. The patch is incorrect because it should not include a space in the split() as indicated in the comment by the original reporter. Of course a corrected patch alone still is not sufficient to address the problem without the user providing the correct converter. Also you start to run into problems with multiple delimiters (such as one space versus two spaces) so you start down the path to add all the features that duplicate genfromtxt. Bruce From ralf.gommers at googlemail.com Thu Mar 31 11:08:27 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 17:08:27 +0200 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 31, 2011 at 5:03 PM, Bruce Southey wrote: > On Wed, Mar 30, 2011 at 9:53 PM, Charles R Harris > wrote: >> >> >> On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes >> wrote: >>> >>> On 26. mars 2011, at 21.44, Derek Homeier wrote: >>> >>> > Hi Paul, >>> > >>> > having had a look at the other tickets you dug up, >>> > > [snip] >>> >>> >> 1071: >>> >> ? ? ?It is not clear to me whether loadtxt is supposed to support >>> >> missing values in the fashion indicated in the ticket. >>> > >>> > In principle it should at least allow you to, by the use of converters >>> > as described there. >>> > The problem is, the default delimiter is described as 'any >>> > whitespace', which in the >>> > present implementation obviously includes any number of blanks or >>> > tabs. These >>> > are therefore treated differently from delimiters like ',' or '&'. I'd >>> > reckon there are >>> > too many people actually relying on this behaviour to silently change it >>> > (e.g. I know plenty of tables with columns separated by either one or >>> > several >>> > tabs depending on the length of the previous entry). But the tab is >>> > apparently also >>> > treated differently if explicitly specified with "delimiter='\t'" - >>> > and in that case using >>> > a converter ? la {2: lambda s: float(s or 'Nan')} is working for >>> > fields in the middle of >>> > the line, but not at the end - clearly warrants improvement. I've >>> > prepared a patch >>> > working for Python3 as well. >>> >>> Great! >>> > This is an invalid ticket because the docstring clearly states that in > 3 different, yet critical places, that missing values are not handled > here: > > "Each row in the text file must have the same number of values." > "genfromtxt : Load data with missing values handled as specified." > ?" ? This function aims to be a fast reader for simply formatted files. ?The > ? ?`genfromtxt` function provides more sophisticated handling of, e.g., > ? ?lines with missing values." > > Really I am trying to separate the usage of loadtxt and genfromtxt to > avoid unnecessary duplication and confusion. Part of this is > historical because loadtxt was added in 2007 and genfromtxt was added > in 2009. So really certain features of loadtxt have been ?'kept' for > backwards compatibility purposes yet these features can be 'abused' to > handle missing data. But I really consider that any missing values > should cause loadtxt to fail. I agree with you Bruce, but it would be easier to discuss this on the tickets instead of here. Could you add your comments there please? Ralf > The patch is incorrect because it should not include a space in the > split() as indicated in the comment by the original reporter. Of > course a corrected patch alone still is not sufficient to address the > problem without the user providing the correct converter. Also you > start to run into problems with multiple delimiters (such as one space > versus two spaces) so you start down the path to add all the features > that duplicate genfromtxt. From bsouthey at gmail.com Thu Mar 31 11:10:29 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 31 Mar 2011 10:10:29 -0500 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: <4D949965.2010202@gmail.com> On 03/31/2011 10:08 AM, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 5:03 PM, Bruce Southey wrote: >> On Wed, Mar 30, 2011 at 9:53 PM, Charles R Harris >> wrote: >>> >>> On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes >>> wrote: >>>> On 26. mars 2011, at 21.44, Derek Homeier wrote: >>>> >>>>> Hi Paul, >>>>> >>>>> having had a look at the other tickets you dug up, >>>>> >> [snip] >>>>>> 1071: >>>>>> It is not clear to me whether loadtxt is supposed to support >>>>>> missing values in the fashion indicated in the ticket. >>>>> In principle it should at least allow you to, by the use of converters >>>>> as described there. >>>>> The problem is, the default delimiter is described as 'any >>>>> whitespace', which in the >>>>> present implementation obviously includes any number of blanks or >>>>> tabs. These >>>>> are therefore treated differently from delimiters like ',' or '&'. I'd >>>>> reckon there are >>>>> too many people actually relying on this behaviour to silently change it >>>>> (e.g. I know plenty of tables with columns separated by either one or >>>>> several >>>>> tabs depending on the length of the previous entry). But the tab is >>>>> apparently also >>>>> treated differently if explicitly specified with "delimiter='\t'" - >>>>> and in that case using >>>>> a converter ? la {2: lambda s: float(s or 'Nan')} is working for >>>>> fields in the middle of >>>>> the line, but not at the end - clearly warrants improvement. I've >>>>> prepared a patch >>>>> working for Python3 as well. >>>> Great! >>>> >> This is an invalid ticket because the docstring clearly states that in >> 3 different, yet critical places, that missing values are not handled >> here: >> >> "Each row in the text file must have the same number of values." >> "genfromtxt : Load data with missing values handled as specified." >> " This function aims to be a fast reader for simply formatted files. The >> `genfromtxt` function provides more sophisticated handling of, e.g., >> lines with missing values." >> >> Really I am trying to separate the usage of loadtxt and genfromtxt to >> avoid unnecessary duplication and confusion. Part of this is >> historical because loadtxt was added in 2007 and genfromtxt was added >> in 2009. So really certain features of loadtxt have been 'kept' for >> backwards compatibility purposes yet these features can be 'abused' to >> handle missing data. But I really consider that any missing values >> should cause loadtxt to fail. > I agree with you Bruce, but it would be easier to discuss this on the > tickets instead of here. Could you add your comments there please? > > Ralf > 'Easier' seems a contradiction when you have use captcha... Sure I will add more comments there. Bruce From ben.root at ou.edu Thu Mar 31 11:41:11 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 31 Mar 2011 10:41:11 -0500 Subject: [Numpy-discussion] Old tickets In-Reply-To: References: <4D938664.6040404@gmail.com> Message-ID: On Thu, Mar 31, 2011 at 10:00 AM, Ralf Gommers wrote: > On Wed, Mar 30, 2011 at 11:52 PM, Derek Homeier > wrote: > > > > On 30 Mar 2011, at 23:26, Benjamin Root wrote: > > > >> Ticket 301: 'Make power and divide return floats from int inputs (like > >> true_divide)' > >> http://projects.scipy.org/numpy/ticket/301 > >> Invalid because the output dtype is the same as the input dtype unless > >> you override using the dtype argument: > >> >>> np.power(3, 1, dtype=np.float128).dtype > >> dtype('float128') > >> Alternatively return a float and indicate in the docstring that the > >> output dtype can be changed. > >> > >> > >> FWIW, > >> > >> Just thought I'd note (on a python 2.6 system): > >> > >> >>> import numpy as np > >> >>> a = np.array([1, 2, 3, 4]) > >> >>> a.dtype > >> dtype('int32') > >> >>> 2 / a > >> array([2, 1, 0, 0]) > >> >>> from __future__ import division > >> >>> 2 / a > >> array([ 2. , 1. , 0.66666667, 0.5 ]) > >> > >> So, numpy already does this when true division is imported (and > >> therefore consistent with whatever the python environment does), and > >> python currently also returns integers for exponentials when both > >> inputs are integers. > > > > I'd agree, and in my view power(3, -1) is well defined as 1 / 3 - > > also, in future (or Python3) > > > > >>> a/2 > > array([ 0.5, 1. , 1.5, 2. ]) > > >>> a//2 > > array([0, 1, 1, 2], dtype=int32) > > The ticket is about the functions np.divide and np.power, not / and > **. This currently does not work the same, unlike what's said above: > > >>> from __future__ import division > >>> x = np.arange(4) + 1 > >>> 2 / x > array([ 2. , 1. , 0.66666667, 0.5 ]) > >>> np.divide(2, x) > array([2, 1, 0, 0]) > > The power and divide functions should not check the "from __future__ > import division", just return floats IMHO. This is what's expected by > most users, and much more useful than integer math. > > Hmm, I didn't notice that distinction. So long as there is a consistent difference between the function and the operator, I am fine with that. > > > so I think at least a**n should follow integer math rules; depends on > > whether we want > > np.power to behave differently from ** (if they are internally handled > > separately at all)... > > Not sure if I understand the overload suggestion in the ticket > > I don't understand that either, it'll just lead to more confusion. > > Did a little further investigation of this (I hadn't thought of negative powers), and now I am getting thoroughly confused... >>> import numpy as np >>> a = np.array([1, 2]) >>> a.dtype dtype('int32') >>> 1**2 1 >>> 2**2 4 >>> 1**-2 1.0 >>> 2**-2 0.25 >>> a**2 array([1, 4]) >>> a**-2 array([1, 0]) >>> np.power(a, 2) array([1, 4]) >>> np.power(a, -2) array([1, 0]) So, in this case, the ** operator and np.power are identical, but diverges from the behavior of python's operator. FWIW, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 31 11:48:43 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Mar 2011 09:48:43 -0600 Subject: [Numpy-discussion] Old tickets In-Reply-To: References: <4D938664.6040404@gmail.com> Message-ID: On Thu, Mar 31, 2011 at 9:00 AM, Ralf Gommers wrote: > On Wed, Mar 30, 2011 at 11:52 PM, Derek Homeier > wrote: > > > > On 30 Mar 2011, at 23:26, Benjamin Root wrote: > > > >> Ticket 301: 'Make power and divide return floats from int inputs (like > >> true_divide)' > >> http://projects.scipy.org/numpy/ticket/301 > >> Invalid because the output dtype is the same as the input dtype unless > >> you override using the dtype argument: > >> >>> np.power(3, 1, dtype=np.float128).dtype > >> dtype('float128') > >> Alternatively return a float and indicate in the docstring that the > >> output dtype can be changed. > >> > >> > >> FWIW, > >> > >> Just thought I'd note (on a python 2.6 system): > >> > >> >>> import numpy as np > >> >>> a = np.array([1, 2, 3, 4]) > >> >>> a.dtype > >> dtype('int32') > >> >>> 2 / a > >> array([2, 1, 0, 0]) > >> >>> from __future__ import division > >> >>> 2 / a > >> array([ 2. , 1. , 0.66666667, 0.5 ]) > >> > >> So, numpy already does this when true division is imported (and > >> therefore consistent with whatever the python environment does), and > >> python currently also returns integers for exponentials when both > >> inputs are integers. > > > > I'd agree, and in my view power(3, -1) is well defined as 1 / 3 - > > also, in future (or Python3) > > > > >>> a/2 > > array([ 0.5, 1. , 1.5, 2. ]) > > >>> a//2 > > array([0, 1, 1, 2], dtype=int32) > > The ticket is about the functions np.divide and np.power, not / and > **. This currently does not work the same, unlike what's said above: > > >>> from __future__ import division > >>> x = np.arange(4) + 1 > >>> 2 / x > array([ 2. , 1. , 0.66666667, 0.5 ]) > >>> np.divide(2, x) > array([2, 1, 0, 0]) > > The power and divide functions should not check the "from __future__ > import division", just return floats IMHO. This is what's expected by > most users, and much more useful than integer math There is true_divide for that. In [4]: np.true_divide(2, x) Out[4]: array([ 2. , 1. , 0.66666667, 0.5 ]) true_divide is what is called by the '/' operator in Python3. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Mar 31 12:04:13 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 31 Mar 2011 09:04:13 -0700 Subject: [Numpy-discussion] Old tickets In-Reply-To: References: <4D938664.6040404@gmail.com> Message-ID: <4D94A5FD.4040002@noaa.gov> On 3/31/11 8:41 AM, Benjamin Root wrote: > The ticket is about the functions np.divide and np.power, not / and > **. This currently does not work the same, unlike what's said above: > Hmm, I didn't notice that distinction. So long as there is a consistent > difference between the function and the operator, I am fine with that. > > whether we want > > np.power to behave differently from ** (if they are internally handled > > separately at all)... Please, no! the operator and functions should behave the same. I certainly always expect them to. For instance, I often write code with operators, then replace that with the functions so that i can add an "out" parameter for performance reasons -- I would be very surprised if the function were defined differently -- it would be a source of surprising bugs. > So, in this case, the ** operator and np.power are identical, but > diverges from the behavior of python's operator. I think consistency within numpy is more important that consistency with python -- I expect differences like this from python (long integers, different data types, etc) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From orion at cora.nwra.com Thu Mar 31 12:33:32 2011 From: orion at cora.nwra.com (Orion Poplawski) Date: Thu, 31 Mar 2011 10:33:32 -0600 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc Message-ID: <4D94ACDC.40701@cora.nwra.com> I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. It appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc (see below for examples). This seems wrong. Thoughts? /usr/lib64/python2.7/site-packages/doc/cython/MANIFEST /usr/lib64/python2.7/site-packages/doc/cython/Makefile /usr/lib64/python2.7/site-packages/doc/cython/README.txt /usr/lib64/python2.7/site-packages/doc/cython/c_numpy.pxd /usr/lib64/python2.7/site-packages/doc/cython/c_python.pxd /usr/lib64/python2.7/site-packages/doc/cython/numpyx.pyx /usr/lib64/python2.7/site-packages/doc/cython/run_test.py /usr/lib64/python2.7/site-packages/doc/cython/run_test.pyc /usr/lib64/python2.7/site-packages/doc/cython/run_test.pyo /usr/lib64/python2.7/site-packages/doc/cython/setup.py /usr/lib64/python2.7/site-packages/doc/cython/setup.pyc /usr/lib64/python2.7/site-packages/doc/cython/setup.pyo /usr/lib64/python2.7/site-packages/doc/pyrex/MANIFEST /usr/lib64/python2.7/site-packages/doc/pyrex/Makefile /usr/lib64/python2.7/site-packages/doc/pyrex/README.txt /usr/lib64/python2.7/site-packages/doc/pyrex/c_numpy.pxd /usr/lib64/python2.7/site-packages/doc/pyrex/c_python.pxd /usr/lib64/python2.7/site-packages/doc/pyrex/notes /usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.c /usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.pyx /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.py /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyc /usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyo /usr/lib64/python2.7/site-packages/doc/pyrex/setup.py /usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyc /usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyo /usr/lib64/python2.7/site-packages/doc/swig/Array.i /usr/lib64/python2.7/site-packages/doc/swig/Array1.cxx /usr/lib64/python2.7/site-packages/doc/swig/Array1.h -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From ralf.gommers at googlemail.com Thu Mar 31 13:00:31 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 19:00:31 +0200 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: <4D94ACDC.40701@cora.nwra.com> References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 6:33 PM, Orion Poplawski wrote: > I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. ?It > appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc > (see below for examples). ?This seems wrong. ?Thoughts? That is indeed wrong. It's caused by the following in setup.py: # we want these files also in binaries/installed files, so it belongs here # instead of in Manifest.in config.add_data_files(('doc/cython/'), ('doc/pyrex/'), ('doc/swig/')) Maybe those three dirs should be moved from doc/ to numpy/doc/ in the source tree? Ralf > > ? ?/usr/lib64/python2.7/site-packages/doc/cython/MANIFEST > ? ?/usr/lib64/python2.7/site-packages/doc/cython/Makefile > ? ?/usr/lib64/python2.7/site-packages/doc/cython/README.txt > ? ?/usr/lib64/python2.7/site-packages/doc/cython/c_numpy.pxd > ? ?/usr/lib64/python2.7/site-packages/doc/cython/c_python.pxd > ? ?/usr/lib64/python2.7/site-packages/doc/cython/numpyx.pyx > ? ?/usr/lib64/python2.7/site-packages/doc/cython/run_test.py > ? ?/usr/lib64/python2.7/site-packages/doc/cython/run_test.pyc > ? ?/usr/lib64/python2.7/site-packages/doc/cython/run_test.pyo > ? ?/usr/lib64/python2.7/site-packages/doc/cython/setup.py > ? ?/usr/lib64/python2.7/site-packages/doc/cython/setup.pyc > ? ?/usr/lib64/python2.7/site-packages/doc/cython/setup.pyo > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/MANIFEST > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/Makefile > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/README.txt > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/c_numpy.pxd > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/c_python.pxd > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/notes > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.c > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/numpyx.pyx > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/run_test.py > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyc > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/run_test.pyo > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/setup.py > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyc > ? ?/usr/lib64/python2.7/site-packages/doc/pyrex/setup.pyo > ? ?/usr/lib64/python2.7/site-packages/doc/swig/Array.i > ? ?/usr/lib64/python2.7/site-packages/doc/swig/Array1.cxx > ? ?/usr/lib64/python2.7/site-packages/doc/swig/Array1.h > > > -- > Orion Poplawski > Technical Manager ? ? ? ? ? ? ? ? ? ? 303-415-9701 x222 > NWRA/CoRA Division ? ? ? ? ? ? ? ? ? ?FAX: 303-415-9702 > 3380 Mitchell Lane ? ? ? ? ? ? ? ? ?orion at cora.nwra.com > Boulder, CO 80301 ? ? ? ? ? ? ?http://www.cora.nwra.com > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Thu Mar 31 13:02:41 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 31 Mar 2011 19:02:41 +0200 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On 31 Mar 2011, at 17:03, Bruce Southey wrote: > This is an invalid ticket because the docstring clearly states that in > 3 different, yet critical places, that missing values are not handled > here: > > "Each row in the text file must have the same number of values." > "genfromtxt : Load data with missing values handled as specified." > " This function aims to be a fast reader for simply formatted > files. The > `genfromtxt` function provides more sophisticated handling of, > e.g., > lines with missing values." > > Really I am trying to separate the usage of loadtxt and genfromtxt to > avoid unnecessary duplication and confusion. Part of this is > historical because loadtxt was added in 2007 and genfromtxt was added > in 2009. So really certain features of loadtxt have been 'kept' for > backwards compatibility purposes yet these features can be 'abused' to > handle missing data. But I really consider that any missing values > should cause loadtxt to fail. > OK, I was not aware of the design issues of loadtxt vs. genfromtxt - you could probably say also for historical reasons since I have not used genfromtxt much so far. Anyway the docstring statement "Converters can also be used to provide a default value for missing data:" then appears quite misleading, or an invitation to abuse, if you will. This should better be removed from the documentation then, or users explicitly discouraged from using converters instead of genfromtxt (I don't see how you could completely prevent using converters in this way). > The patch is incorrect because it should not include a space in the > split() as indicated in the comment by the original reporter. Of The split('\r\n') alone caused test_dtype_with_object(self) to fail, probably because it relies on stripping the blanks. But maybe the test is ill- formed? > course a corrected patch alone still is not sufficient to address the > problem without the user providing the correct converter. Also you > start to run into problems with multiple delimiters (such as one space > versus two spaces) so you start down the path to add all the features > that duplicate genfromtxt. Given that genfromtxt provides that functionality more conveniently, I agree again users should be encouraged to use this instead of converters. But the actual tab-problem causes in fact an issue not related to missing values at all (well, depending on what you call a missing value). I am describing an example on the ticket. Cheers, Derek From mwwiebe at gmail.com Thu Mar 31 13:06:52 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 31 Mar 2011 10:06:52 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) wrote: > On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) > wrote: > > Can anyone please give me some suggestions on how to go about writing > > a unit test for this? Or should I just submit a pull request? > > I've gotten a bit of positive feedback to adding the 'e' type to the > struct module on the python-ideas list (per my understanding, not > before python 3.3, but I don't think that should hinder adoption in > other libraries), so I'd like to ask again about unit testing a change > like this. Can anyone offer some advice for where to start? > The tests for the buffer protocol are in numpy/core/tests/test_multiarray.py, starting at line 1847. It does some round-trip testing of all the types through the memoryview object. You'll also need to change the _dtype_from_pep3118 in numpy/core/_internal.py, called from the _descriptor_from_pep3118_format function, so that NumPy can accept the half type from a buffer as well. Also, what kind of timeframe / cutoff am I looking at to get this into > 1.6.0 or 1.6.x? Since the changes to NumPy are pretty small, I think it should be ok to get into 1.6.0. How to handle it is up to Ralf, however, and it would probably be best to get it into beta 2 which I believe he would like to release over the weekend (please chime in to correct me). -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Mar 31 13:32:03 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 19:32:03 +0200 Subject: [Numpy-discussion] np.histogramdd of empty data In-Reply-To: References: <4D9435C8.5040709@amolf.nl> <4D94586D.4080300@amolf.nl> Message-ID: ---------- Forwarded message ---------- From: Ralf Gommers Date: Thu, Mar 31, 2011 at 7:31 PM Subject: Re: [Numpy-discussion] np.histogramdd of empty data To: Nils Becker On Thu, Mar 31, 2011 at 12:33 PM, Nils Becker wrote: > Hi Ralf, > > I cloned numpy/master and played around a little. > > when giving the bins explicitely, now histogram2d and histogramdd work > as expected in all tests i tried. > > > However, some of the cases with missing bin specification appear > somewhat inconsistent. > > The first question is if creating arbitrary bins for empty data and > empty bin specification is better than raising an Exception: > > Specifically: Bins of size 0 should give a meaningful error, I was just fixing that as part of #1788 in https://github.com/rgommers/numpy/tree/ticket-1788-histogramdd > numpy.histogram2d([],[],bins=[0,0]) >> (array([ 0., ?0.]), array([ 0.]), array([ 0.])) Now gives: ? ?ValueError: Element at index 0 in `bins` should be a positive integer. > numpy.histogram([],bins=0) >> ValueError: zero-size array to minimum.reduce without identity Now gives: ? ?ValueError: `bins` should be a positive integer. > so 1-d and 2-d behave not quite the same. > > also, these work (although with arbitrary bin edges): > > numpy.histogram2d([],[],bins=[1,1]) >> (array([ 0., ?0.]), array([ 0., ?1.]), array([ 0., ?1.])) > > numpy.histogram2d([],[],bins=[0,1]) >> (array([ 0., ?0.]), array([ 0.]), array([ 0., ?1.])) > > while this raises an error: > > numpy.histogram([],bins=1) >> ValueError: zero-size array to minimum.reduce without identity Now gives: ? ?(array([0]), array([ 0., ?1.])) > another thing with non-empty data: > > numpy.histogram([1],bins=1) >> (array([1]), array([ 0.5, ?1.5])) That is the correct answer. > numpy.histogram([1],bins=0) >> (array([], dtype=int64), array([ 0.5])) Now gives: ? ?ValueError: `bins` should be a positive integer. > while > > numpy.histogram2d([1],[1],bins=A) >> ValueError: zero-size array to minimum.reduce without identity > > (here A==[0,0] or A==[0,1] but not A==[1,1] which gives a result) Same sensible errors now, telling you bins elements shouldn't be 0. Cheers, Ralf From ralf.gommers at googlemail.com Thu Mar 31 13:40:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 19:40:39 +0200 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 7:06 PM, Mark Wiebe wrote: > On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) > wrote: >> >> On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) >> wrote: >> > Can anyone please give me some suggestions on how to go about writing >> > a unit test for this? ?Or should I just submit a pull request? >> >> I've gotten a bit of positive feedback to adding the 'e' type to the >> struct module on the python-ideas list (per my understanding, not >> before python 3.3, but I don't think that should hinder adoption in >> other libraries), so I'd like to ask again about unit testing a change >> like this. ?Can anyone offer some advice for where to start? > > The tests for the buffer protocol are in > numpy/core/tests/test_multiarray.py, starting at line 1847. It does some > round-trip testing of all the types through the memoryview object. > You'll also need to change the _dtype_from_pep3118 in > numpy/core/_internal.py, called from the _descriptor_from_pep3118_format > function, so that NumPy can accept the half type from a buffer as well. >> >> Also, what kind of timeframe / cutoff am I looking at to get this into >> 1.6.0 or 1.6.x? > > Since the changes to NumPy are pretty small, I think it should be ok to get > into 1.6.0. How to handle it is up to Ralf, however, and it would probably > be best to get it into beta 2 which I believe he would like to release over > the weekend (please chime in to correct me). It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment. It would need some testing with other libraries that use the numpy buffer protocol. Also, reading back the thread, Pauli seemed to disagree with this. Ralf From ben.root at ou.edu Thu Mar 31 14:00:19 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 31 Mar 2011 13:00:19 -0500 Subject: [Numpy-discussion] np.histogramdd of empty data In-Reply-To: References: <4D9435C8.5040709@amolf.nl> <4D94586D.4080300@amolf.nl> Message-ID: On Thu, Mar 31, 2011 at 12:32 PM, Ralf Gommers wrote: > ---------- Forwarded message ---------- > From: Ralf Gommers > Date: Thu, Mar 31, 2011 at 7:31 PM > Subject: Re: [Numpy-discussion] np.histogramdd of empty data > To: Nils Becker > > > On Thu, Mar 31, 2011 at 12:33 PM, Nils Becker wrote: > > Hi Ralf, > > > > I cloned numpy/master and played around a little. > > > > when giving the bins explicitely, now histogram2d and histogramdd work > > as expected in all tests i tried. > > > > > > However, some of the cases with missing bin specification appear > > somewhat inconsistent. > > > > The first question is if creating arbitrary bins for empty data and > > empty bin specification is better than raising an Exception: > > > > Specifically: > > Bins of size 0 should give a meaningful error, I was just fixing that > as part of #1788 in > https://github.com/rgommers/numpy/tree/ticket-1788-histogramdd > > > numpy.histogram2d([],[],bins=[0,0]) > >> (array([ 0., 0.]), array([ 0.]), array([ 0.])) > > Now gives: > ValueError: Element at index 0 in `bins` should be a positive integer. > > > numpy.histogram([],bins=0) > >> ValueError: zero-size array to minimum.reduce without identity > > Now gives: > ValueError: `bins` should be a positive integer. > > > > so 1-d and 2-d behave not quite the same. > > > > also, these work (although with arbitrary bin edges): > > > > numpy.histogram2d([],[],bins=[1,1]) > >> (array([ 0., 0.]), array([ 0., 1.]), array([ 0., 1.])) > > > > numpy.histogram2d([],[],bins=[0,1]) > >> (array([ 0., 0.]), array([ 0.]), array([ 0., 1.])) > > > > while this raises an error: > > > > numpy.histogram([],bins=1) > >> ValueError: zero-size array to minimum.reduce without identity > > Now gives: > (array([0]), array([ 0., 1.])) > > Just for consistency's sake, maybe the same should be done for np.bincount() and np.digitize() for empty data (but known bins)? I don't know if your fix to histogram does this or not, but the latest pull from numpy master doesn't do this. Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Thu Mar 31 14:15:08 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 31 Mar 2011 11:15:08 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 7:06 PM, Mark Wiebe wrote: > > On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) < > wickedgrey at gmail.com> > > wrote: > >> > >> On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) > >> wrote: > >> > Can anyone please give me some suggestions on how to go about writing > >> > a unit test for this? Or should I just submit a pull request? > >> > >> I've gotten a bit of positive feedback to adding the 'e' type to the > >> struct module on the python-ideas list (per my understanding, not > >> before python 3.3, but I don't think that should hinder adoption in > >> other libraries), so I'd like to ask again about unit testing a change > >> like this. Can anyone offer some advice for where to start? > > > > The tests for the buffer protocol are in > > numpy/core/tests/test_multiarray.py, starting at line 1847. It does some > > round-trip testing of all the types through the memoryview object. > > You'll also need to change the _dtype_from_pep3118 in > > numpy/core/_internal.py, called from the _descriptor_from_pep3118_format > > function, so that NumPy can accept the half type from a buffer as well. > >> > >> Also, what kind of timeframe / cutoff am I looking at to get this into > >> 1.6.0 or 1.6.x? > > > > Since the changes to NumPy are pretty small, I think it should be ok to > get > > into 1.6.0. How to handle it is up to Ralf, however, and it would > probably > > be best to get it into beta 2 which I believe he would like to release > over > > the weekend (please chime in to correct me). > > It would definitely need to get into beta 2, and even then I'm a > little hesitant to push in such a change at the last moment. It would > need some testing with other libraries that use the numpy buffer > protocol. Also, reading back the thread, Pauli seemed to disagree with > this. My reading of Pauli's thoughts was that putting it in unilaterally is undesirable, something I definitely agree with. I think with Eli doing the legwork of getting input and acceptance from the relevant parties, we should help him out as much as possible. Not getting this change into 1.6 makes the Cython support much more difficult because of their design based around the buffer protocol. Looking at the thread on python-users, I see mostly concern that the type be standard or have precedent elsewhere, which as an IEEE standard it definitely satisfies. The other question is on the chosen letter - we picked 'e' as 'h' was taken and it is close to 'f' and 'd', the other float types. If a letter change is deemed necessary, it would be good to get that into 1.6 as well, since this kind of change is easy now, but much more difficult later. For testing, Eli implementing the Cython support seems like a good start to me. Another nice test would be to get PIL's OpenEXR support to export a half buffer to NumPy. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 31 14:32:10 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 31 Mar 2011 13:32:10 -0500 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 12:00, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 6:33 PM, Orion Poplawski wrote: >> I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. ?It >> appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc >> (see below for examples). ?This seems wrong. ?Thoughts? > > That is indeed wrong. It's caused by the following in setup.py: > > ? ?# we want these files also in binaries/installed files, so it belongs here > ? ?# instead of in Manifest.in > ? ?config.add_data_files(('doc/cython/'), > ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/pyrex/'), > ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/swig/')) > > Maybe those three dirs should be moved from doc/ to numpy/doc/ in the > source tree? It's not clear to me how they were included in the Windows .exe installer originally (the original ticket that led to this change[1088]). I don't think it was through the generic setup.py. The proper fix would probably be something specific to each binary installer. [1088] http://projects.scipy.org/numpy/ticket/1088 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From wickedgrey at gmail.com Thu Mar 31 14:37:54 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Thu, 31 Mar 2011 11:37:54 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers wrote: > It would definitely need to get into beta 2, and even then I'm a > little hesitant to push in such a change at the last moment. If I miss the 1.6.0b2 cutoff, would a change like this be appropriate for 1.6.1, or will it need to wait until 1.7.0? I wish I'd known that half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd have started working on this a few weeks earlier. :) > It would > need some testing with other libraries that use the numpy buffer > protocol. I am currently working on a patch for cython to accept half-floats. ATM, it is able to accept the data from numpy (though it does not yet handle the bit-fiddling to expand the values into a float32 on read, nor collapse it from a float32 back into a float16 on write). The code change to numpy was literally a one-line change to add NPY_HALF to a switch statement (though I haven't tried to import half floats yet). Is one other library enough, or will I need to find another library to patch as well? I think that OpenCL would probably be my next choice, since I think that we'll eventually want float16 support there too. > Also, reading back the thread, Pauli seemed to disagree with > this. On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen wrote: > The buffer interface cannot be used to export the half-float types, since > the type is not specified in PEP 3118. Numpy cannot unilaterally add > nonstandard format codes to the spec. I'm currently in the process of submitting a patch to CPython to update the struct module to handle half-floats. So far, the response on python-ideas has been generally positive (a couple +1s from people with commit bits, GVR has chimed in on the thread with an aside (but not an objection), etc.). Unless I'm misunderstanding the objection, making half-floats an official part of the spec (the discussion on python-ideas explicitly notes that numpy driving force behind the change) would satisfy Pauli's concern. Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today. Thanks! Eli From ralf.gommers at googlemail.com Thu Mar 31 14:39:23 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 20:39:23 +0200 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 8:32 PM, Robert Kern wrote: > On Thu, Mar 31, 2011 at 12:00, Ralf Gommers wrote: >> On Thu, Mar 31, 2011 at 6:33 PM, Orion Poplawski wrote: >>> I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. ?It >>> appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc >>> (see below for examples). ?This seems wrong. ?Thoughts? >> >> That is indeed wrong. It's caused by the following in setup.py: >> >> ? ?# we want these files also in binaries/installed files, so it belongs here >> ? ?# instead of in Manifest.in >> ? ?config.add_data_files(('doc/cython/'), >> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/pyrex/'), >> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/swig/')) >> >> Maybe those three dirs should be moved from doc/ to numpy/doc/ in the >> source tree? > > It's not clear to me how they were included in the Windows .exe > installer originally (the original ticket that led to this > change[1088]). I don't think it was through the generic setup.py. The > proper fix would probably be something specific to each binary > installer. > > [1088] http://projects.scipy.org/numpy/ticket/1088 I'm not sure how this was done at first (maybe someone remembers?), it kind of hard to search for that in the history. But something specific for each way of installing sounds fragile. The first question to be answered is where these files should be installed to. The current location is obviously wrong. If it is under numpy/doc/swig/ etc. then moving them there in the source tree itself seems logical. Those dirs are not used to build the user and reference guides, so there's no need for them to stay where they are. Ralf From bsouthey at gmail.com Thu Mar 31 14:39:45 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 31 Mar 2011 13:39:45 -0500 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: <4D94CA71.5050606@gmail.com> On 03/31/2011 12:02 PM, Derek Homeier wrote: > On 31 Mar 2011, at 17:03, Bruce Southey wrote: > >> This is an invalid ticket because the docstring clearly states that in >> 3 different, yet critical places, that missing values are not handled >> here: >> >> "Each row in the text file must have the same number of values." >> "genfromtxt : Load data with missing values handled as specified." >> " This function aims to be a fast reader for simply formatted >> files. The >> `genfromtxt` function provides more sophisticated handling of, >> e.g., >> lines with missing values." >> >> Really I am trying to separate the usage of loadtxt and genfromtxt to >> avoid unnecessary duplication and confusion. Part of this is >> historical because loadtxt was added in 2007 and genfromtxt was added >> in 2009. So really certain features of loadtxt have been 'kept' for >> backwards compatibility purposes yet these features can be 'abused' to >> handle missing data. But I really consider that any missing values >> should cause loadtxt to fail. >> > OK, I was not aware of the design issues of loadtxt vs. genfromtxt - > you could probably say also for historical reasons since I have not > used genfromtxt much so far. > Anyway the docstring statement "Converters can also be used to > provide a default value for missing data:" > then appears quite misleading, or an invitation to abuse, if you will. > This should better be removed from the documentation then, or users > explicitly discouraged from using converters instead of genfromtxt > (I don't see how you could completely prevent using converters in > this way). > >> The patch is incorrect because it should not include a space in the >> split() as indicated in the comment by the original reporter. Of > The split('\r\n') alone caused test_dtype_with_object(self) to fail, > probably > because it relies on stripping the blanks. But maybe the test is ill- > formed? > >> course a corrected patch alone still is not sufficient to address the >> problem without the user providing the correct converter. Also you >> start to run into problems with multiple delimiters (such as one space >> versus two spaces) so you start down the path to add all the features >> that duplicate genfromtxt. > Given that genfromtxt provides that functionality more conveniently, > I agree again users should be encouraged to use this instead of > converters. > But the actual tab-problem causes in fact an issue not related to > missing > values at all (well, depending on what you call a missing value). > I am describing an example on the ticket. > > Cheers, > Derek > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I am really not disagreeing that much with you. Rather that, as you have shown, it is very easy to increase the complexity of examples that loadtxt does not handle. By missing value I mean "when no data value is stored for the variable in the current observation" (via Wikipedia) since encoded missing values (such as '.', 'NA' and 'NaN') can be recovered. Bruce From ralf.gommers at googlemail.com Thu Mar 31 14:45:32 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 20:45:32 +0200 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) wrote: > On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers > wrote: >> It would definitely need to get into beta 2, and even then I'm a >> little hesitant to push in such a change at the last moment. > > If I miss the 1.6.0b2 cutoff, would a change like this be appropriate > for 1.6.1, or will it need to wait until 1.7.0? ?I wish I'd known that > half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd > have started working on this a few weeks earlier. ?:) There likely will not be a 1.7.0, if things with 2.0 go according to plan. It probably wouldn't be a very appropriate change for 1.6.1. > >> It would >> need some testing with other libraries that use the numpy buffer >> protocol. > > I am currently working on a patch for cython to accept half-floats. > ATM, it is able to accept the data from numpy (though it does not yet > handle the bit-fiddling to expand the values into a float32 on read, > nor collapse it from a float32 back into a float16 on write). ?The > code change to numpy was literally a one-line change to add NPY_HALF > to a switch statement (though I haven't tried to import half floats > yet). ?Is one other library enough, or will I need to find another > library to patch as well? ?I think that OpenCL would probably be my > next choice, since I think that we'll eventually want float16 support > there too. If it works with Cython I think that's enough. With other libraries I did not mean that you should write patches for those, just some testing to make sure the change is 100% backwards compatible. >> Also, reading back the thread, Pauli seemed to disagree with >> this. > > On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen wrote: >> The buffer interface cannot be used to export the half-float types, since >> the type is not specified in PEP 3118. Numpy cannot unilaterally add >> nonstandard format codes to the spec. > > I'm currently in the process of submitting a patch to CPython to > update the struct module to handle half-floats. ?So far, the response > on python-ideas has been generally positive (a couple +1s from people > with commit bits, GVR has chimed in on the thread with an aside (but > not an objection), etc.). ?Unless I'm misunderstanding the objection, > making half-floats an official part of the spec (the discussion on > python-ideas explicitly notes that numpy driving force behind the > change) would satisfy Pauli's concern. > > Next up for me is to get a patch onto the CPython issue tracker, but > as soon as I have that done, I'll start working on adding unit tests > to my numpy fork. ?I anticipate being able to get that done today. Sounds good. Cheers, Ralf From robert.kern at gmail.com Thu Mar 31 14:52:40 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 31 Mar 2011 13:52:40 -0500 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 13:39, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 8:32 PM, Robert Kern wrote: >> On Thu, Mar 31, 2011 at 12:00, Ralf Gommers wrote: >>> On Thu, Mar 31, 2011 at 6:33 PM, Orion Poplawski wrote: >>>> I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. ?It >>>> appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc >>>> (see below for examples). ?This seems wrong. ?Thoughts? >>> >>> That is indeed wrong. It's caused by the following in setup.py: >>> >>> ? ?# we want these files also in binaries/installed files, so it belongs here >>> ? ?# instead of in Manifest.in >>> ? ?config.add_data_files(('doc/cython/'), >>> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/pyrex/'), >>> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/swig/')) >>> >>> Maybe those three dirs should be moved from doc/ to numpy/doc/ in the >>> source tree? >> >> It's not clear to me how they were included in the Windows .exe >> installer originally (the original ticket that led to this >> change[1088]). I don't think it was through the generic setup.py. The >> proper fix would probably be something specific to each binary >> installer. >> >> [1088] http://projects.scipy.org/numpy/ticket/1088 > > I'm not sure how this was done at first (maybe someone remembers?), it > kind of hard to search for that in the history. But something specific > for each way of installing sounds fragile. The first question to be > answered is where these files should be installed to. The current > location is obviously wrong. Well, the answer is different for each platform. That's why it ought to be up to each binary installer. > If it is under numpy/doc/swig/ etc. then moving them there in the > source tree itself seems logical. Those dirs are not used to build the > user and reference guides, so there's no need for them to stay where > they are. Linux distributions start to complain when data files (especially documentation data files not used at runtime) are placed into the Python packages. I would argue that it's the wrong place for them on *any* platform. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From scopatz at gmail.com Thu Mar 31 15:04:47 2011 From: scopatz at gmail.com (Anthony Scopatz) Date: Thu, 31 Mar 2011 14:04:47 -0500 Subject: [Numpy-discussion] Willing to contribute to SciPy & NumPy ... In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 3:04 AM, Scott Sinclair wrote: > On 31 March 2011 07:27, Sylvain Bellemare wrote: > > I would like to seriously start contributing to NumPy and/or SciPy, as > much as I possibly can. > > I'm sure that your help will be welcomed! > > A good place to get started is helping out with documentation (see > http://docs.scipy.org/numpy/Front%20Page/). Actually, according to the Jacob Kaplan-Moss's talk at PyCon this year ( http://pycon.blip.tv/file/4881071/) (with which I am inclined to agree), documentation is probably not the best way to get started. Do check out the developer zone, maybe search the the open tickets and see if there are any bugs you want to tackle. Maybe write some tests for untested code you find. If you have questions, don't hesitate to ask. And don't forget to test and document your own code! Be Well Anthony > SciPy has plenty of work > required - you'll probably learn your way into the code that way. > Another place to look is http://www.scipy.org/Developer_Zone. It's > worthwhile learning how to work with Git and Github if you want to get > patches accepted more easily. > > Cheers, > Scott > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Mar 31 15:08:51 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 31 Mar 2011 21:08:51 +0200 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 8:52 PM, Robert Kern wrote: > On Thu, Mar 31, 2011 at 13:39, Ralf Gommers wrote: >> On Thu, Mar 31, 2011 at 8:32 PM, Robert Kern wrote: >>> On Thu, Mar 31, 2011 at 12:00, Ralf Gommers wrote: >>>> On Thu, Mar 31, 2011 at 6:33 PM, Orion Poplawski wrote: >>>>> I'm looking at updating the numpy package in Fedora rawhide to 1.6.0b1. ?It >>>>> appears to install a bunch of suff in /usr/lib64/python2.7/site-packages/doc >>>>> (see below for examples). ?This seems wrong. ?Thoughts? >>>> >>>> That is indeed wrong. It's caused by the following in setup.py: >>>> >>>> ? ?# we want these files also in binaries/installed files, so it belongs here >>>> ? ?# instead of in Manifest.in >>>> ? ?config.add_data_files(('doc/cython/'), >>>> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/pyrex/'), >>>> ? ? ? ? ? ? ? ? ? ? ? ? ?('doc/swig/')) >>>> >>>> Maybe those three dirs should be moved from doc/ to numpy/doc/ in the >>>> source tree? >>> >>> It's not clear to me how they were included in the Windows .exe >>> installer originally (the original ticket that led to this >>> change[1088]). I don't think it was through the generic setup.py. The >>> proper fix would probably be something specific to each binary >>> installer. >>> >>> [1088] http://projects.scipy.org/numpy/ticket/1088 >> >> I'm not sure how this was done at first (maybe someone remembers?), it >> kind of hard to search for that in the history. But something specific >> for each way of installing sounds fragile. The first question to be >> answered is where these files should be installed to. The current >> location is obviously wrong. > > Well, the answer is different for each platform. That's why it ought > to be up to each binary installer. > >> If it is under numpy/doc/swig/ etc. then moving them there in the >> source tree itself seems logical. Those dirs are not used to build the >> user and reference guides, so there's no need for them to stay where >> they are. > > Linux distributions start to complain when data files (especially > documentation data files not used at runtime) are placed into the > Python packages. I would argue that it's the wrong place for them on > *any* platform. But they're not documentation files. The line is a bit blurry, but something like the numpy.i SWIG interface face is meant for using numpy, not documenting usage of SWIG and numpy (maybe most similar to what's in core/include/ ?). The actual documentation of how to use numpy.i is included in the reference guide, and is not present in that dir. Perhaps to make clear this is not documentation, put it in something like numpy/interfacing/swig/, etc.? Ralf From robert.kern at gmail.com Thu Mar 31 15:12:49 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 31 Mar 2011 14:12:49 -0500 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 14:08, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 8:52 PM, Robert Kern wrote: >> Linux distributions start to complain when data files (especially >> documentation data files not used at runtime) are placed into the >> Python packages. I would argue that it's the wrong place for them on >> *any* platform. > > But they're not documentation files. The line is a bit blurry, but > something like the numpy.i SWIG interface face is meant for using > numpy, not documenting usage of SWIG and numpy (maybe most similar to > what's in core/include/ ?). Yeah, and they hate that, too. As the originator of that idea, I'm loathe to push it any further. :-) > The actual documentation of how to use > numpy.i is included in the reference guide, and is not present in that > dir. > > Perhaps to make clear this is not documentation, put it in something > like numpy/interfacing/swig/, etc.? Well, they're meant to be copied into your own code, which is why they end up under a doc/ directory. Lots of things like this tend to end up in doc/ directories. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From orion at cora.nwra.com Thu Mar 31 15:13:31 2011 From: orion at cora.nwra.com (Orion Poplawski) Date: Thu, 31 Mar 2011 13:13:31 -0600 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: <4D94D25B.3090909@cora.nwra.com> On 03/31/2011 01:08 PM, Ralf Gommers wrote: > But they're not documentation files. The line is a bit blurry, but > something like the numpy.i SWIG interface face is meant for using > numpy, not documenting usage of SWIG and numpy (maybe most similar to > what's in core/include/ ?). The actual documentation of how to use > numpy.i is included in the reference guide, and is not present in that > dir. > > Perhaps to make clear this is not documentation, put it in something > like numpy/interfacing/swig/, etc.? How are these files used? Are they used by the python code directly? Just used by people wanting to use swig/cython/pyrex to interface with numpy? If the latter, they don't belong in python/site-packages but something lib /usr/share/numpy. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From orion at cora.nwra.com Thu Mar 31 15:16:20 2011 From: orion at cora.nwra.com (Orion Poplawski) Date: Thu, 31 Mar 2011 13:16:20 -0600 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> Message-ID: <4D94D304.1050401@cora.nwra.com> On 03/31/2011 01:12 PM, Robert Kern wrote: > Well, they're meant to be copied into your own code, which is why they > end up under a doc/ directory. Lots of things like this tend to end up > in doc/ directories. > perhaps /usr/share/doc/numpy then. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From robert.kern at gmail.com Thu Mar 31 15:21:23 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 31 Mar 2011 14:21:23 -0500 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: <4D94D304.1050401@cora.nwra.com> References: <4D94ACDC.40701@cora.nwra.com> <4D94D304.1050401@cora.nwra.com> Message-ID: On Thu, Mar 31, 2011 at 14:16, Orion Poplawski wrote: > On 03/31/2011 01:12 PM, Robert Kern wrote: >> Well, they're meant to be copied into your own code, which is why they >> end up under a doc/ directory. Lots of things like this tend to end up >> in doc/ directories. > > perhaps /usr/share/doc/numpy then. This is up to the various Linux packagers of numpy. They have their own tools to do this according to their own policies. Doing *anything* in our setup.py to install these files will just make their jobs harder. That's why I think that this needs to be handled in the binary installers, not the setup.py. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From orion at cora.nwra.com Thu Mar 31 15:25:31 2011 From: orion at cora.nwra.com (Orion Poplawski) Date: Thu, 31 Mar 2011 13:25:31 -0600 Subject: [Numpy-discussion] 1.6.0b1 installs stuff in /usr/lib/python2.7/site-packages/doc In-Reply-To: References: <4D94ACDC.40701@cora.nwra.com> <4D94D304.1050401@cora.nwra.com> Message-ID: <4D94D52B.1030607@cora.nwra.com> On 03/31/2011 01:21 PM, Robert Kern wrote: > On Thu, Mar 31, 2011 at 14:16, Orion Poplawski wrote: >> On 03/31/2011 01:12 PM, Robert Kern wrote: >>> Well, they're meant to be copied into your own code, which is why they >>> end up under a doc/ directory. Lots of things like this tend to end up >>> in doc/ directories. >> >> perhaps /usr/share/doc/numpy then. > > This is up to the various Linux packagers of numpy. They have their > own tools to do this according to their own policies. Doing *anything* > in our setup.py to install these files will just make their jobs > harder. That's why I think that this needs to be handled in the binary > installers, not the setup.py. > Fedora already copies everything in doc/* to /usr/share/doc/numpy-%{version}, so yeah, no need to put it anywhere. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From Chris.Barker at noaa.gov Thu Mar 31 16:29:23 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 31 Mar 2011 13:29:23 -0700 Subject: [Numpy-discussion] Willing to contribute to SciPy & NumPy ... In-Reply-To: References: Message-ID: <4D94E423.7060208@noaa.gov> On 3/31/11 12:04 PM, Anthony Scopatz wrote: > A good place to get started is helping out with documentation (see > http://docs.scipy.org/numpy/Front%20Page/). > > > Actually, according to the Jacob Kaplan-Moss's talk at PyCon this year > (http://pycon.blip.tv/file/4881071/) Nice talk -- thanks for the link. > (with which I am inclined to agree), documentation is probably not the > best way to get started. I agree as well. However, I think newbies can be a great help in identifying holes in the documentation -- so when documentation is missing or unclear, ask here, and then you'll have something to contribute. -Chris > Do check out the developer zone, maybe search the the open tickets and > see if there are any bugs you want to tackle. > Maybe write some tests for untested code you find. Good ideas as well, of course. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ben.root at ou.edu Thu Mar 31 16:42:59 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 31 Mar 2011 15:42:59 -0500 Subject: [Numpy-discussion] Willing to contribute to SciPy & NumPy ... In-Reply-To: <4D94E423.7060208@noaa.gov> References: <4D94E423.7060208@noaa.gov> Message-ID: On Thu, Mar 31, 2011 at 3:29 PM, Christopher Barker wrote: > On 3/31/11 12:04 PM, Anthony Scopatz wrote: > > > (with which I am inclined to agree), documentation is probably not the > > best way to get started. > > I agree as well. However, I think newbies can be a great help in > identifying holes in the documentation -- so when documentation is > missing or unclear, ask here, and then you'll have something to contribute. > > +1 on this. The biggest problem with developers writing documentation for tools like NumPy is that we have already become accustom to the quirks and assumptions of the tool. Fresh minds are perfect for identifying problem areas in the docs. The problem I have seen, though, is that it becomes so easy to just answer the user's question and then just move on. We really need to better remind ourselves that each time someone has to ask an "easy" question, it is a failing with our documentation. "What if it was already in the docs", you ask? Then the question is, why couldn't they find it (or recognize it)? "What if they didn't read the documentation", you say? Then the question becomes how did our documentation fail to be read? Just food for thought... Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Thu Mar 31 21:40:29 2011 From: david at silveregg.co.jp (David) Date: Fri, 01 Apr 2011 10:40:29 +0900 Subject: [Numpy-discussion] [SciPy-Dev] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: <4D952D0D.7020409@silveregg.co.jp> On 03/31/2011 06:37 PM, Pearu Peterson wrote: > > > On Thu, Mar 31, 2011 at 12:19 PM, David Cournapeau > wrote: > > On Wed, Mar 30, 2011 at 7:22 AM, Russell E. Owen > wrote: > > In article > > >, > > Ralf Gommers > wrote: > > > >> Hi, > >> > >> I am pleased to announce the availability of the first beta of NumPy > >> 1.6.0. Due to the extensive changes in the Numpy core for this > >> release, the beta testing phase will last at least one month. Please > >> test this beta and report any problems on the Numpy mailing list. > >> > >> Sources and binaries can be found at: > >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b1/ > >> For (preliminary) release notes see below. > > I see a segfault on Ubuntu 64 bits for the test > TestAssumedShapeSumExample in numpy/f2py/tests/test_assumed_shape.py. > Am I the only one seeing it ? > > The test work here ok on Ubuntu 64 with numpy master. Could you try the > maintenance/1.6.x branch where the related bugs are fixed. I did test that as well, and got the same issue, but could not reproduce it on another machine. I do get the error every time on my main work machine, though. I will look more into it, but it is most likely something due to my machine, cheers, David From pivanov314 at gmail.com Thu Mar 31 21:42:52 2011 From: pivanov314 at gmail.com (Paul Ivanov) Date: Thu, 31 Mar 2011 18:42:52 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1 In-Reply-To: References: Message-ID: <20110401014252.GT3401@ykcyc> Ralf Gommers, on 2011-03-23 17:07, wrote: > Please test this beta and report any problems on the Numpy > mailing list. Hi, I reopened http://projects.scipy.org/numpy/ticket/1768, as 'test_chisquare' still fails for me on a 32 bit machine. Lowering the precision from 14 to 13 decimals makes the test pass, however. Other than that: NumPy version 1.6.0b1 NumPy is installed in /home/pi/.local/lib/python2.6/site-packages/numpy Python version 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) [GCC 4.4.3] nose version 0.11.1 --snip-- Ran 3399 tests in 23.052s FAILED (KNOWNFAIL=3, SKIP=4, failures=1) best, -- Paul Ivanov 314 address only used for lists, off-list direct email at: http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From charlesr.harris at gmail.com Thu Mar 31 22:34:35 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Mar 2011 20:34:35 -0600 Subject: [Numpy-discussion] loadtxt/savetxt tickets In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: On Thu, Mar 31, 2011 at 8:42 AM, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 4:53 AM, Charles R Harris > wrote: > > > > > > On Sun, Mar 27, 2011 at 4:09 AM, Paul Anton Letnes > > wrote: > >> > If you look in Trac under "All Tickets by Milestone" you'll find all > nine tickets together under 1.6.0. Five are bug fixes, four are > enhancements. There are some missing tests, but all tickets have > proposed patches. > > OK. I changed 1562 to enhancement because it adds a keyword. With that change the current status looks like this. Bug Fixes: 1163 -- convert int64 correctly 1458 -- make loadtxt unpack structured arrays 1071 -- loadtxt fails if the last column contains empty value, under discussion 1565 -- duplicate of 1163 Enhancements: 1107 -- support for blocks of data, adds two keywords. 1562 -- add ndmin keyword to aid in getting correct dimensions, doesn't apply on top of previous. 1616 -- remove use of readline so input isn't restricted to files. 1731 -- allow loadtxt to read given number of rows, adds keyword. 1752 -- return empty array when empty file encountered, conflicts with 1616. Some of this might should go into genfromtxt. None of the patches have tests. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wickedgrey at gmail.com Thu Mar 31 23:12:11 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Thu, 31 Mar 2011 20:12:11 -0700 Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug? In-Reply-To: References: Message-ID: On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers wrote: > On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) > wrote: >> Next up for me is to get a patch onto the CPython issue tracker, but >> as soon as I have that done, I'll start working on adding unit tests >> to my numpy fork. ?I anticipate being able to get that done today. > > Sounds good. CPython issue here: http://bugs.python.org/issue11734 My numpy changes so far: https://github.com/wickedgrey/numpy/compare/23965b6b665985390d7a...maintenance%2F1.6.x >From my last commit message: "Note that my changes to _internal were mostly of the "make it work" variety; I'm not really understanding the wider scope. Very much needs review by wiser eyes." I'd also appreciate some feedback on the unit test coverage; the test_roundtrip_half function uncovered a problem that the other two didn't (which surprised me), and I'd like to make sure that there aren't more gaps in the tests that I've written. Thanks for all the help! Eli