Mailman 3 May 2013 - NumPy-Discussion

Consistency of numpy’s behavior with arrays of dimension 0
by Sylvain Corlay 08 May '13

08 May '13

Hello everyone, 0d arrays are very convenient because they allow us to write functions that are valid for both arrays and scalar-like argument, thanks to Boolean indexing. However, most special functions in numpy (and scipy) and most Boolean operators, when called on 0d arrays, return scalars rather than 0d arrays, which is quite annoying. For example, numpy.exp called on a 0d array containing a float number returns a float, rather than a 0d array, and if x is a 0d array, x > 0 returns a Boolean, rather than a 0d array containing a Boolean. What I would expect is the following. -If x is a 0d array containing float, I expect numpy.exp(x) to return a 0d array, and x>0 to return a 0d array containing a Boolean. -If x is a scalar, numpy.exp(x) returns the expected scalar type, and x>0 returns the expected Boolean. Here is an example of a simple function that suffers from this issue (a corrected version is proposed later) import numpy from scipy.stats import norm def normal_time_value(sig, m, strikes): """ The three arguments are array-like and have the same shape. Consider a random variable G ~ N (m , sig^2) The function returns E[(G-K)+] - (E[G]-K)_+ which is also equal to E[(K-G)+] - (K-E[G])_+ """ sig = numpy.array(sig) strikes = numpy.array(strikes) m = numpy.array(m) tv = numpy.zeros(strikes.shape) tv[sig < 0] = numpy.nan # sig must be nonnegative non0 = sig > 0.0 dev = numpy.where(non0, (m - strikes) / sig, numpy.nan) tv[non0] = numpy.where(strikes > m, (m - strikes) * norm.cdf(dev) + sig * norm.pdf(dev), (strikes - m) * norm.cdf(-dev) + sig * norm.pdf(dev))[non0] return tv This function does not work with scalars or 0d arrays. To make it work, we need to modify it in the following fashion: reconvert intermediate results to 0d array to take advantage of the Boolean indexing. import numpy from scipy.stats import norm def normal_time_value(sig, m, strikes): """ The three arguments are array-like and have the same shape. Consider a random variable G ~ N (m , sig^2) The function returns E[(G-K)+] - (E[G]-K)_+ which is also equal to E[(K-G)+] - (K-E[G])_+ """ sig = numpy.array(sig) strikes = numpy.array(strikes) m = numpy.array(m) tv = numpy.zeros(strikes.shape) tv[numpy.array(sig < 0)] = numpy.nan # sig must be nonnegative non0 = numpy.array(sig > 0.0) dev = numpy.where(non0, (m - strikes) / sig, numpy.nan) tv[non0] = numpy.where(numpy.array(strikes > m), (m - strikes) * norm.cdf(dev) + sig * norm.pdf(dev), (strikes - m) * norm.cdf(-dev) + sig * norm.pdf(dev))[non0] return tv This problem also affects functions like logical_and, logical_or and logical_not, which all return numpy.bool_ type rather than 0d array of dtype bool. Travis Oliphant opened a ticket in track regarding the same issue some time ago: http://projects.scipy.org/numpy/ticket/823 It was transfered to github https://github.com/numpy/numpy/issues/1421. Best, Sylvain

1 0

ANN: Spyder v2.2
by Pierre Raybaut 08 May '13

08 May '13

Hi all, On the behalf of Spyder's development team ( http://code.google.com/p/spyderlib/people/list), I'm pleased to announce that Spyder v2.2 has been released and is available for Windows XP/Vista/7/8, GNU/Linux and MacOS X: http://code.google.com/p/spyderlib/. This release represents 18 months of development since v2.1 and introduces major enhancements and new features: * Full support for IPython v0.13, including the ability to attach to existing kernels * New MacOS X application * Much improved debugging experience * Various editor improvements for code completion, zooming, auto insertion, and syntax highlighting * Better looking and faster Object Inspector * Single instance mode * Spanish tranlation of the interface * And many other changes: http://code.google.com/p/spyderlib/wiki/ChangeLog This is the last release to support Python 2.5: * Spyder 2.2 supports Python 2.5 to 2.7 * Spyder 2.3 will support Python 2.7 and Python 3 * (Spyder 2.1.14dev4 is a development release which already supports Python 3) See also https://code.google.com/p/spyderlib/downloads/list. Spyder is a free, open-source (MIT license) interactive development environment for the Python language with advanced editing, interactive testing, debugging and introspection features. Originally designed to provide MATLAB-like features (integrated help, interactive console, variable explorer with GUI-based editors for dictionaries, NumPy arrays, ...), it is strongly oriented towards scientific computing and software development. Thanks to the `spyderlib` library, Spyder also provides powerful ready-to-use widgets: embedded Python console (example: http://packages.python.org/guiqwt/_images/sift3.png), NumPy array editor (example: http://packages.python.org/guiqwt/_images/sift2.png), dictionary editor, source code editor, etc. Description of key features with tasty screenshots can be found at: http://code.google.com/p/spyderlib/wiki/Features Don't forget to follow Spyder updates/news: * on the project website: http://code.google.com/p/spyderlib/ * and on our official blog: http://spyder-ide.blogspot.com/ Last, but not least, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment! (http://code.google.com/p/spyderlib/wiki/NoteForContributors) Enjoy! -Pierre

2 1

ANN: WinPython 2.7.4.1 and 3.3.1.1, 32 and 64bit
by Pierre Raybaut 08 May '13

08 May '13

Hi all, I am pleased to announce that four new versions of WinPython have been released yesterday with Python 2.7.4 and 3.3.1, 32 and 64 bits. Many packages have been added or upgraded (see the automatically-generated changelogs): * WinPython 2.7.4.1 32bit, 64bit -- including new Spyder v2.2: https://code.google.com/p/winpython/wiki/ChangeLog_27 * WinPython 3.3.1.1 32bit, 64bit: https://code.google.com/p/winpython/wiki/ChangeLog_33 Special thanks to Christoph Gohlke for building most of the binary packages bundled in WinPython. WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.3) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel), and adding shortcuts to Windows Start menu. Enjoy!

1 0

CfP 2013 Workshop on Middleware for HPC and Big Data Systems (MHPC'13)
by MHPC 2013 07 May '13

07 May '13

we apologize if you receive multiple copies of this message =================================================================== CALL FOR PAPERS 2013 Workshop on Middleware for HPC and Big Data Systems MHPC '13 as part of Euro-Par 2013, Aachen, Germany =================================================================== Date: August 27, 2012 Workshop URL: http://m-hpc.org Springer LNCS SUBMISSION DEADLINE: May 31, 2013 - LNCS Full paper submission (rolling abstract submission) June 28, 2013 - Lightning Talk abstracts SCOPE Extremely large, diverse, and complex data sets are generated from scientific applications, the Internet, social media and other applications. Data may be physically distributed and shared by an ever larger community. Collecting, aggregating, storing and analyzing large data volumes presents major challenges. Processing such amounts of data efficiently has been an issue to scientific discovery and technological advancement. In addition, making the data accessible, understandable and interoperable includes unsolved problems. Novel middleware architectures, algorithms, and application development frameworks are required. In this workshop we are particularly interested in original work at the intersection of HPC and Big Data with regard to middleware handling and optimizations. Scope is existing and proposed middleware for HPC and big data, including analytics libraries and frameworks. The goal of this workshop is to bring together software architects, middleware and framework developers, data-intensive application developers as well as users from the scientific and engineering community to exchange their experience in processing large datasets and to report their scientific achievement and innovative ideas. The workshop also offers a dedicated forum for these researchers to access the state of the art, to discuss problems and requirements, to identify gaps in current and planned designs, and to collaborate in strategies for scalable data-intensive computing. The workshop will be one day in length, composed of 20 min paper presentations, each followed by 10 min discussion sections. Presentations may be accompanied by interactive demonstrations. TOPICS Topics of interest include, but are not limited to: - Middleware including: Hadoop, Apache Drill, YARN, Spark/Shark, Hive, Pig, Sqoop, HBase, HDFS, S4, CIEL, Oozie, Impala, Storm and Hyrack - Data intensive middleware architecture - Libraries/Frameworks including: Apache Mahout, Giraph, UIMA and GraphLab - NG Databases including Apache Cassandra, MongoDB and CouchDB/Couchbase - Schedulers including Cascading - Middleware for optimized data locality/in-place data processing - Data handling middleware for deployment in virtualized HPC environments - Parallelization and distributed processing architectures at the middleware level - Integration with cloud middleware and application servers - Runtime environments and system level support for data-intensive computing - Skeletons and patterns - Checkpointing - Programming models and languages - Big Data ETL - Stream processing middleware - In-memory databases for HPC - Scalability and interoperability - Large-scale data storage and distributed file systems - Content-centric addressing and networking - Execution engines, languages and environments including CIEL/Skywriting - Performance analysis, evaluation of data-intensive middleware - In-depth analysis and performance optimizations in existing data-handling middleware, focusing on indexing/fast storing or retrieval between compute and storage nodes - Highly scalable middleware optimized for minimum communication - Use cases and experience for popular Big Data middleware - Middleware security, privacy and trust architectures DATES Papers: Rolling abstract submission May 31, 2013 - Full paper submission July 8, 2013 - Acceptance notification October 3, 2013 - Camera-ready version due Lightning Talks: June 28, 2013 - Deadline for lightning talk abstracts July 15, 2013 - Lightning talk notification August 27, 2013 - Workshop Date TPC CHAIR Michael Alexander (chair), TU Wien, Austria Anastassios Nanos (co-chair), NTUA, Greece Jie Tao (co-chair), Karlsruhe Institut of Technology, Germany Lizhe Wang (co-chair), Chinese Academy of Sciences, China Gianluigi Zanetti (co-chair), CRS4, Italy PROGRAM COMMITTEE Amitanand Aiyer, Facebook, USA Costas Bekas, IBM, Switzerland Jakob Blomer, CERN, Switzerland William Gardner, University of Guelph, Canada José Gracia, HPC Center of the University of Stuttgart, Germany Zhenghua Guom, Indiana University, USA Marcus Hardt, Karlsruhe Institute of Technology, Germany Sverre Jarp, CERN, Switzerland Christopher Jung, Karlsruhe Institute of Technology, Germany Andreas Knüpfer - Technische Universität Dresden, Germany Nectarios Koziris, National Technical University of Athens, Greece Yan Ma, Chinese Academy of Sciences, China Martin Schulz - Lawrence Livermore National Laboratory Viral Shah, MIT Julia Group, USA Dimitrios Tsoumakos, Ionian University, Greece Zhifeng Yun, Louisiana State University, USA PAPER PUBLICATION Accepted full papers will be published in the Springer LNCS series. The best papers of the workshop -- after extension and revision -- will be published in a Special Issue of the Springer Journal of Scalable Computing. PAPER SUBMISSION Papers submitted to the workshop will be reviewed by at least two members of the program committee and external reviewers. Submissions should include abstract, key words, the e-mail address of the corresponding author, and must not exceed 10 pages, including tables and figures at a main font size no smaller than 11 point. Submission of a paper should be regarded as a commitment that, should the paper be accepted, at least one of the authors will register and attend the conference to present the work. The format must be according to the Springer LNCS Style. Initial submissions are in PDF; authors of accepted papers will be requested to provide source files. Format Guidelines: http://www.springer.de/comp/lncs/authors.html Style template: ftp://ftp.springer.de/pub/tex/latex/llncs/latex2e/llncs2e.zip Abstract Registration - Submission Link: http://edas.info/newPaper.php?c=14763 LIGHTNING TALKS Talks are strictly limited to 5 minutes. They can be used to gain early feedback on ongoing research, for demonstrations, to present research results, early research ideas, perspectives and positions of interest to the community. Lightning talks should spark discussion with presenters making themselves available following the lightning talk track. DURATION: Workshop Duration is one day. GENERAL INFORMATION The workshop will be held as part of Euro-Par 2013. Euro-Par 2013: http://www.europar2013.org

1 0

Numpy beginner tutorial
by Nicolas Rougier 07 May '13

07 May '13

Hello everybody, I've written a numpy beginner tutorial that is available from: http://www.loria.fr/~rougier/teaching/numpy/numpy.html It has been designed around cellular automata to try to make it fun. While writing it, I tried to compile a set of exercises and make them progressively harder. For advanced levels, I thought the easiest way would be to extract simple questions (but more importantly answers) from this very mailing list in order to gather them on a single page. The goal would be both to offer a quick reference for new (and old users) and to provide also a set of exercices for those who teach. However, it's a bit harder than I thought since the mailing list is huge. I made a separate page for this: http://www.loria.fr/~rougier/teaching/numpy.100/index.html (Sources are http://www.loria.fr/~rougier/teaching/numpy.100/index.rst) (The level names came from an old-game: Dungeon Master) In order to extract questions/answers and I would need some help, if you have some free time to spare... If you remember having asked or answered a (short) problem, could you send a link to the relevant post (the one with the answer), or better, write directly the formated entry. Here is an example: #. Find indices of non-zero elements from [1,2,0,0,4,0] .. code:: python # Author: Somebody print np.nonzero([1,2,0,0,4,0]) If you can provide the (assumed) level of the answer, that would be even better. Nicolas

3 3

searchsorted descending arrays
by Bago 07 May '13

07 May '13

I submitted a patch a little while ago, https://github.com/numpy/numpy/pull/3107, which gave the searchsorted function the ability to search arrays sorted in descending order. At the time my approach was to detect the sortorder of the array by comparing the first and last elements. This works pretty well in most cases, but fails in one notable case. After giving it some thought, I think the best way to add searching of descending arrays to numpy would be by adding a keyword to the searchsorted function. I wanted to know what you guys thought of this before updating the pr. I would like to add something like the following to numpy: A = [10, 9, 2, 1] np.searchsorted(A, 5, sortorder='descending') the other option would be to auto-detect the order, but then this case might surprise some users: A = [0, 0, 0] A = np.sort(A)[::-1] print np.searchsorted(A, [1, -1]) # [3, 0] This might surprise a user who expects to be searching a descending array Bago

3 2

mpfr integration into numpy?
by Funky Dev 06 May '13

06 May '13

Hi, I've got a project in which it turns out we need much higher precision than even __float128 and playing around with a few alternatives mpfr seems to be the highest performing possibility. So I've starting writing a cythonized class mpfr_array which provides array-like functionality but with mpfr_t as a "primitive" type. While this is doable it feels very much like re-inventing the wheel as I'm basically rewriting some part of numpy's functionality. So I was wondering, as I think this is a very standard use case, is there any interest in adding mpfr_t as a "primitive" dtype to numpy? I know one can define new dtype's but I don't want it to by a python object since then there will be a large python overhead for all operations (i.e. dot products will happen in python, not C). mpfr_t is a very natural dtype to add as its fast, C-based and supports general precision. I have to admit complete ignorance to numpy's internals but as I'm writing my own verison of such a class I would be happy to work with anyone more versed in numpy than myself to extend numpy with built-in mpfr_t support. On a related note, if this was done, would it automatically work with functionality such as numpy.linalg.inv(), etc...? In principle such functions could have been written with macros to be more type-flexible (i.e. an ADD(a,b,c) mapping to e.g. a=b+c for floats or to mpfr_add(a, b, c) for mpfr_t) but I suspect this is not the case. thanks, Sheer

3 2

Matching line points with grid coordinates numerically
by Bakhtiyor Zokhidov 06 May '13

06 May '13

Dear Python users, I am having difficulty with numerically scaling to match line coordinates vs grid cell size coordinates. I want to calculate the following function: F = distance_of_crossed_line x intersected_cell_value The problem here is that when I calculate crossed_line_length in line coordinates that will unmatch vs grid coordinate, which is also another x,y with step e.g., dx=dy=2.5 each grid size. I want to do numerical calculation , say, F(distance, intersected_grid_value) function where, intersected_grid_value - values in intersected grid, distance - intersected_line_length (given below or can be found http://stackoverflow.com/questions/16377826/distance-for-each-intersected-p… ) import numpy as np import scipy as sp def distance_of_crossed_line(x0, x1, y0, y1): # slope m = (y1 - y0) / (x1 - x0) # Boundary of the selected points x_ceil = np.ceil(min(x0, x1)) x_floor = np.floor(max(x0, x1)) y_ceil = np.ceil(min(y0, y1)) y_floor = np.floor(max(y0, y1)) # calculate all intersected x coordinate x = np.arange(x_ceil, x_floor + 1) y = m * (x - x0) + y0 ax = zip(x, y) # calculate all intersected y coordinate y = np.arange(y_ceil, y_floor + 1) x = (y - y0) / m + x0 ax.extend(zip(x, y)) ax.append((x0, y0)) ax.append((x1, y1)) ax.sort() # Transpose ax = np.array(ax).T # Calculate difference of intersections in X dist_x = np.diff(ax[0]) # Calculate difference of intersections in Y dist_y = np.diff(ax[1]) return np.sqrt(dist_x**2 + dist_y**2) # PLEASE, note that line points are different from 2D array axis. they should be matched with each other. # 2D array. d_array = np.array[[4.5, 4.5, 4.5, 3.4, 2.5],[ 3.9, 4.5, 5.2, 4.5, 3.4],[3.9, 3.9, 2.5, 2.2, 1.9]] # Two sample points as a line x = np.array([ -80, -40 ]) y = np.array([ 60, 55 ]) # The problem: F = intersected_line_length * array_grid_values_where_line_crossed_area * It is not necessary for me to overlay lines onto grid cells properly, JUST, I need to calculate numerically accurate F function Thanks for the answer and guidance in advance, -- Bakhtiyor Zokhidov

1 0

numpy ring buffer
by Daniele Nicolodi 06 May '13

06 May '13

Hello, numpy arrays are great for interfacing python with libraries that expect continuous memory buffers for data passing. However, libraries interfacing to data acquisition hardware often use those buffers as ring buffers where, once the buffer has been filled with data, new data will be written overwriting the data at the beginning of the buffer. The application is supposed to have read this data meanwhile. Efficiently processing the data in the ring buffer requires addressing efficiently the content of the ring buffer. In C or Cython this is very easy, just compute the wrap around when accessing the single elements of the array: buffer = np.empty(size) adc.setup(buffer) start = 0 for n in adc.read(): # data processing for i in range(start, start + n): element = buffer[start % size] .... start += n My current approach to do the same thing in Python is to use the np.roll() function to "linearize" the buffer access: buffer = np.empty(size) adc.setup(buffer) start = 0 for n in adc.read(): data = np.roll(buffer, -start)[:n] start += n # data processing process(data) Since np.roll() returns a view on the array i suppose this is very efficient. Does anyone have a better idea on how to do this? Thank you. Cheers, Daniele

3 6

Any plans for windows 64-bit installer for 1.7?
by Matthew Brett 06 May '13

06 May '13

Hi, I see there is no Windows 64 bit installer for the 1.7 rc1. Is there any prospect of a 64 bit installer for the full release? Can I help? I have a Windows 7 64 bit machine that I use as a build slave; I am happy to give access. Cheers, Matthew

13 57