Mailman 3 December 2012 - NumPy-Discussion

Re: [Numpy-discussion] Efficient way of binning points and, applying functions to these groups
by Eric Emsellem Dec. 27, 2012

Dec. 27, 2012

Thanks Ralf! this module looks great in fact. I didn't know it existed, and in fact It is only available in Scipy 0.11.0 (had to install from source since an Ubuntu 12.04 bin is not available). Too bad that the User-defined function only accepts one single array. If that function should take more input you need to rely on a trick to basically duplicate the input coordinates and concatenate the input arrays you need. But apart from the fact that the programme looks much cleaner now, I just tested it and it is rather SLOW in fact. Since I have to repeat this 2 or 3 times (I have in fact 3 functions to apply), it takes about 20 seconds for a full test, while with the changes I made (see below) it takes now 3 seconds or so [I am talking about the real code, not the example I give below]. So I managed to speed up things a bit by doing two things: - keeping the first loop but replacing the second one with a loop ONLY on bins which contains the right number of points - and more importantly not addressing the full array at each loop iteration but first selecting the right points (to reduce the size). So something as shown below. it is still a factor of 2 slower than the idl routine and I have no clue why. I will analyse it further. The idl routine has similar loops etc, so there is no reason for this. if anybody has an idea .... THANKS (using cython is a bit too much on my side - being a low-profile python developer. As for numba, will have a look) and thanks again for your help! Eric ======================================================= import numpy as np x = np.random.random(1000) y = np.random.random(1000) data = np.random.random(1000) # I have now a 2D grid given by let's say 10x10 grid points: nx = 11 ny = 21 lx = linspace(0,1,nx) ly = linspace(0,1,ny) gx, gy = np.meshgrid(lx, ly) # So my set of 2D bins are (not needed in the solution I present but just for clarity) bins = np.dstack((gx.ravel(), gy.ravel()))[0] # Now I want to have the list of points in each bin and # if the number of points in that bin is larger than 10, apply (dummy) function func1 (see below) # If less than 10, apply (dummy) function func2 so (dum?) # if 0, do nothing # for two dummy functions like (for example): def func1(x) : return x.mean() def func2(x) : return x.std() h = histogram2d(x, y, bins=[lx, ly])[0] digitX = np.digitize(x, lx) digitY = np.digitize(y, ly) # create the output array, with -999 values to make sure I see which ones are not filled in result = np.zeros_like(h) - 999 for i in range(nx-1) : selX = (digitX == i+1) dataX = data[selX] selH10 = np.where(h > 10) selH0 = np.where((h > 0) & (h <= 10)) for j in selH10 : selectionofpoints = (digitY == j+1) result[i,j] = func1(data[selectionofpoints]) for j in selH0 : selectionofpoints = (digitY == j+1) result[i,j] = func2(data[selectionofpoints]) > Hi! > > I am looking for an efficient way of doing some simple binning of points > and then applying some functions to points within each bin. > That's exactly what scipy.stats.binned_statistic does: http://docs.scipy.org/doc/scipy-dev/reference/generated/scipy.stats.binned_… binned_statistic uses np.digitize as well, but I'm not sure that in your code below digitize is the bottleneck - the nested for-loop looks like the more likely suspect. Ralf

1 0

Efficient way of binning points and applying functions to these groups
by Eric Emsellem Dec. 26, 2012

Dec. 26, 2012

Hi! I am looking for an efficient way of doing some simple binning of points and then applying some functions to points within each bin. I have tried several ways, including crude looping over the indices, or using digitize (see below) but I cannot manage to get it as efficient as I need it to be. I have a comparison with a similar (although complex) code in idl, and thought I would ask the forum. In idl there is a way to "invert" an histogram and get a reverse set of indices (via the histogram function) which seems to do the trick (or maybe it is faster for another reason). Below I provide a (dummy) example of what I wish to achieve. Any hint on how to do this EFFICIENTLY using numpy is most welcome. I need to speed things up quite a bit (at the moment, the version I have, see below, is 10 times slower than the more complex idl routine..., I must be doing something wrong!). thanks!! Eric ======================================================== # I have a random set of data points in 2D with coordinates x,y : import numpy as np x = np.random.random(1000) y = np.random.random(1000) # I have now a 2D grid given by let's say 10x10 grid points: nx = 11 ny = 21 lx = linspace(0,1,nx) ly = linspace(0,1,ny) gx, gy = np.meshgrid(lx, ly) # So my set of 2D bins are (not needed in the solution I present but just for clarity) bins = np.dstack((gx.ravel(), gy.ravel()))[0] # Now I want to have the list of points in each bin and # if the number of points in that bin is larger than 10, apply (dummy) function func1 (see below) # If less than 10, apply (dummy) function func2 so (dum?) # if 0, do nothing # for two dummy functions like (for example): def func1(x) : return x.mean() def func2(x) : return x.std() # One solution would be to use digitize in 1D and histogram in 2D (don't need gx, gy for this one): h = histogram2d(x, y, bins=[lx, ly])[0] digitX = np.digitize(x, lx) digitY = np.digitize(y, ly) # create the output array, with -999 values to make sure I see which ones are not filled in result = np.zeros_like(h) - 999 for i in range(nx-1) : for j in range(ny-1) : selectionofpoints = (digitX == i+1) & (digitY == j+1) if h[i,j] > 10 : result[i,j] = func1(x[selectionofpoints]) elif h[i,j] > 0 : result[i,j] = func2(x[selectionofpoints])

3 2

Many failing doctests - release blocker? Enable for default test runs?
by Matthew Brett Dec. 24, 2012

Dec. 24, 2012

Hi, I noticed that enabling the doctests on the 1.7.x maintenance branch caused lots and lots of doctest failures. (np-devel)[mb312@blair ~/dev_trees]$ python -c 'import numpy as np; np.test(doctests=True)' 1.7.0rc1.dev-1e8fcdf Running unit tests and doctests for numpy NumPy version 1.7.0rc1.dev-1e8fcdf NumPy is installed in /Users/mb312/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy Python version 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1 (Apple Inc. build 5493)] nose version 1.1.2 ... Ran 3839 tests in 59.928s FAILED (KNOWNFAIL=4, SKIP=4, errors=23, failures=175) The doctests also throw up somewhere round 10 matplotlib plots, so presumably those would fail as well on a machine without a display without forcing the import of an 'Agg' backend or similar. I have never checked the doctests on Python 3. Has anyone run those recently? For the projects I work on most, we enable doctests for the default test run - as in 'doctests=True' by default in the numpy testing machinery. Do ya'll see any disadvantage in doing that for numpy? In case someone gets to this before I do, we've also got some logic for doing conditional skips of doctests when optional packages are not available such as matplotlib, inspired by something similar in IPython: https://github.com/nipy/nipy/blob/master/nipy/testing/doctester.py#L193 If Christmas allows I'll send a pull request with something like that in the next few days. Cheers, Matthew

3 5

Future of numpy (was: DARPA funding for Blaze and passing the NumPy torch)
by Matthew Brett Dec. 24, 2012

Dec. 24, 2012

Hi, [Travis wrote...] > My strong suggestion is that development discussions of the project continue on > this list with consensus among the active participants being the goal for > development. I don't think 100% consensus is a rigid requirement --- but > certainly a super-majority should be the goal, and serious changes should not be > made with out a clear consensus. I would pay special attention to > under-represented people (users with intense usage of NumPy but small voices on > this list). There are many of them. If you push me for specifics then at > this point in NumPy's history, I would say that if Chuck, Nathaniel, and Ralf > agree on a course of action, it will likely be a good thing for the project. I > suspect that even if only 2 of the 3 agree at one time it might still be a good > thing (but I would expect more detail and discussion). There are others whose > opinion should be sought as well: Ondrej Certik, Perry Greenfield, Robert Kern, > David Cournapeau, Francesc Alted, and Mark Wiebe to name a few. For some > questions, I might even seek input from people like Konrad Hinsen and Paul > Dubois --- if they have time to give it. I will still be willing to offer my > view from time to time and if I am asked. Thank you for starting this discussion. I am more or less offline at the moment in Cuba and flying, but I hope very much this will be an opportunity for a good discussion on the best way forward for numpy. Travis - I think you are suggesting that there should be no one person in charge of numpy, and I think this is very unlikely to work well. Perhaps there are good examples of well-led projects where there is not a clear leader, but I can't think of any myself at the moment. My worry would be that, without a clear leader, it will be unclear how decisions are made, and that will make it very hard to take strategic decisions. I would like to humbly suggest the following in the hope that it spurs discussion. As first pass, Ralf, Chuck and Nathaniel decide on a core group of people that will form the succession committee. Maybe this could be the list of people you listed above. Ralf, Chuck and Nathaniel then ask for people to wish to propose themselves as the leader of numpy. Anyone proposing themselves to lead numpy would remove themselves from the succession committee. The proposed leaders of numpy write a short manifesto saying why they are the right choice for the job, and what they intend to do if elected. The manifestos and issues arising are discussed in public on the mailing list - the equivalent of an online presidential debate. In due course - say after 2 weeks or after the discussion seems to be dying out - the succession committee votes on the leader. I propose that these votes should be public, but I can see the opposite argument. How does that sound? Best, Matthew

5 10

help with f2py
by Aronne Merrelli Dec. 23, 2012

Dec. 23, 2012

Hi, I'm trying to run f2py and running into some trouble. Starting from http://www.scipy.org/Cookbook/F2Py, and the very simple 'Wrapping Hermite Polynomial' example, I can get the pyf file created with no issues. The system I am using is RedHat linux, and has several Fortran compilers: $ f2py -c --help-fcompiler <snip> Fortran compilers found: --fcompiler=g95 G95 Fortran Compiler (0.92) --fcompiler=gnu GNU Fortran 77 compiler (3.4.6) --fcompiler=gnu95 GNU Fortran 95 compiler (4.1.2) --fcompiler=intelem Intel Fortran Compiler for 64-bit apps (11.1) All of these will successfully create the .so file except for g95, but when I try to import into python I get this ImportError for any of the other three compilers: In [5]: import hermite ImportError: ./hermite.so: undefined symbol: c06ebf_ If I look at the shared object I find that symbol here: $ nm hermite.so <snip> 00000000000043c0 T array_from_pyobj U c06eaf_ U c06ebf_ And that about hits my limit of compiler knowledge, as I am pretty much a novice with these things. Any ideas on what is going wrong here, or suggestions of things to try? Thanks, Aronne

2 1

Re: [Numpy-discussion] [pystatsmodels] Re: ANN: pandas 0.10.0 released
by Chang She Dec. 21, 2012

Dec. 21, 2012

On Dec 21, 2012, at 3:27 PM, Collin Sellman <collin.sellman(a)gmail.com> wrote: > Thanks, Wes and team. I've been looking through the new features, but haven't found any documentation on the integration with the Google Analytics API. I was just in the midst of trying to pull data into Pandas from GA in v.0.9.0, so would love to try what you built in .10. > > -Collin > > On Monday, December 17, 2012 10:19:49 AM UTC-7, Wes McKinney wrote: > hi all, > > I'm super excited to announce the pandas 0.10.0 release. This is > a major release including a new high performance file reading > engine with tons of new user-facing functionality as well, a > bunch of work on the HDF5/PyTables integration layer, > much-expanded Unicode support, a new option/configuration > interface, integration with the Google Analytics API, and a wide > array of other new features, bug fixes, and performance > improvements. I strongly recommend that all users get upgraded as > soon as feasible. Many performance improvements made are quite > substantial over 0.9.x, see vbenchmarks at the end of the e-mail. > > As of this release, we are no longer supporting Python 2.5. Also, > this is the first release to officially support Python 3.3. > > Note: there are a number of minor, but necessary API changes that > long-time pandas users should pay attention to in the What's New. > > Thanks to all who contributed to this release, especially Chang > She, Yoval P, and Jeff Reback (and everyone else listed in the > commit log!). > > As always source archives and Windows installers are on PyPI. > > What's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html > Installers: http://pypi.python.org/pypi/pandas > > $ git log v0.9.1..v0.10.0 --pretty=format:%aN | sort | uniq -c | sort -rn > 246 Wes McKinney > 140 y-p > 99 Chang She > 45 jreback > 18 Abraham Flaxman > 17 Jeff Reback > 14 locojaydev > 11 Keith Hughitt > 5 Adam Obeng > 2 Dieter Vandenbussche > 1 zach powers > 1 Luke Lee > 1 Laurent Gautier > 1 Ken Van Haren > 1 Jay Bourque > 1 Donald Curtis > 1 Chris Mulligan > 1 alex arsenovic > 1 A. Flaxman > > Happy data hacking! > > - Wes > > What is it > ========== > pandas is a Python package providing fast, flexible, and > expressive data structures designed to make working with > relational, time series, or any other kind of labeled data both > easy and intuitive. It aims to be the fundamental high-level > building block for doing practical, real world data analysis in > Python. > > Links > ===== > Release Notes: http://github.com/pydata/pandas/blob/master/RELEASE.rst > Documentation: http://pandas.pydata.org > Installers: http://pypi.python.org/pypi/pandas > Code Repository: http://github.com/pydata/pandas > Mailing List: http://groups.google.com/group/pydata > > Performance vs. v0.9.0 > ====================== > > Benchmarks from https://github.com/pydata/pandas/tree/master/vb_suite > Ratio < 1 means that v0.10.0 is faster > > v0.10.0 v0.9.0 ratio > name > unstack_sparse_keyspace 1.2813 144.1262 0.0089 > groupby_frame_apply_overhead 20.1520 337.3330 0.0597 > read_csv_comment2 25.3097 363.2860 0.0697 > groupbym_frame_apply 75.1554 504.1661 0.1491 > frame_iteritems_cached 0.0711 0.3919 0.1815 > read_csv_thou_vb 35.2690 191.9360 0.1838 > concat_small_frames 12.9019 55.3561 0.2331 > join_dataframe_integer_2key 5.8184 21.5823 0.2696 > series_value_counts_strings 5.3824 19.1262 0.2814 > append_frame_single_homogenous 0.3413 0.9319 0.3662 > read_csv_vb 18.4084 46.9500 0.3921 > read_csv_standard 12.0651 29.9940 0.4023 > panel_from_dict_all_different_indexes 73.6860 158.2949 0.4655 > frame_constructor_ndarray 0.0471 0.0958 0.4918 > groupby_first 3.8502 7.1988 0.5348 > groupby_last 3.6962 6.7792 0.5452 > panel_from_dict_two_different_indexes 50.7428 86.4980 0.5866 > append_frame_single_mixed 1.2950 2.1930 0.5905 > frame_get_numeric_data 0.0695 0.1119 0.6212 > replace_fillna 4.6349 7.0540 0.6571 > frame_to_csv 281.9340 427.7921 0.6590 > replace_replacena 4.7154 7.1207 0.6622 > frame_iteritems 2.5862 3.7463 0.6903 > series_align_int64_index 29.7370 41.2791 0.7204 > join_dataframe_integer_key 1.7980 2.4303 0.7398 > groupby_multi_size 31.0066 41.7001 0.7436 > groupby_frame_singlekey_integer 2.3579 3.1649 0.7450 > write_csv_standard 326.8259 427.3241 0.7648 > groupby_simple_compress_timing 41.2113 52.3993 0.7865 > frame_fillna_inplace 16.2843 20.0491 0.8122 > reindex_fillna_backfill 0.1364 0.1667 0.8181 > groupby_multi_series_op 15.2914 18.6651 0.8193 > groupby_multi_cython 17.2169 20.4420 0.8422 > frame_fillna_many_columns_pad 14.9510 17.5114 0.8538 > panel_from_dict_equiv_indexes 25.8427 29.9682 0.8623 > merge_2intkey_nosort 19.0755 22.1138 0.8626 > sparse_series_to_frame 167.8529 192.9920 0.8697 > reindex_fillna_pad 0.1410 0.1617 0.8720 > merge_2intkey_sort 44.7863 51.3315 0.8725 > reshape_stack_simple 2.6698 3.0502 0.8753 > groupby_indices 7.2264 8.2314 0.8779 > sort_level_one 4.3845 4.9902 0.8786 > sort_level_zero 4.3362 4.9198 0.8814 > write_store 16.0587 18.2042 0.8821 > frame_reindex_both_axes 0.3726 0.4183 0.8907 > groupby_multi_different_numpy_functions 13.4164 15.0509 0.8914 > index_int64_intersection 25.3705 28.1867 0.9001 > groupby_frame_median 7.7491 8.6011 0.9009 > frame_drop_dup_na_inplace 2.6290 2.9155 0.9017 > dataframe_reindex_columns 0.3052 0.3372 0.9049 > join_dataframe_index_multi 20.5651 22.6893 0.9064 > frame_ctor_list_of_dict 101.7439 112.2260 0.9066 > groupby_pivot_table 18.4551 20.3184 0.9083 > reindex_frame_level_align 0.9644 1.0531 0.9158 > stat_ops_level_series_sum_multiple 7.3637 8.0230 0.9178 > write_store_mixed 38.2528 41.6604 0.9182 > frame_reindex_both_axes_ix 0.4550 0.4950 0.9192 > stat_ops_level_frame_sum_multiple 8.1975 8.9055 0.9205 > panel_from_dict_same_index 25.7938 28.0147 0.9207 > groupby_series_simple_cython 5.1310 5.5624 0.9224 > frame_sort_index_by_columns 41.9577 45.1816 0.9286 > groupby_multi_python 54.9727 59.0400 0.9311 > datetimeindex_add_offset 0.2417 0.2584 0.9356 > frame_boolean_row_select 0.2905 0.3100 0.9373 > frame_reindex_axis1 2.9760 3.1742 0.9376 > stat_ops_level_series_sum 2.3382 2.4937 0.9376 > groupby_multi_different_functions 14.0333 14.9571 0.9382 > timeseries_timestamp_tzinfo_cons 0.0159 0.0169 0.9397 > stats_rolling_mean 1.6904 1.7959 0.9413 > melt_dataframe 1.5236 1.6181 0.9416 > timeseries_asof_single 0.0548 0.0582 0.9416 > frame_ctor_nested_dict_int64 134.3100 142.6389 0.9416 > join_dataframe_index_single_key_bigger 15.6578 16.5949 0.9435 > stat_ops_level_frame_sum 3.2475 3.4414 0.9437 > indexing_dataframe_boolean_rows 0.2382 0.2518 0.9459 > timeseries_asof_nan 10.0433 10.6006 0.9474 > frame_reindex_axis0 1.4403 1.5184 0.9485 > concat_series_axis1 69.2988 72.8099 0.9518 > join_dataframe_index_single_key_small 6.8492 7.1847 0.9533 > dataframe_reindex_daterange 0.4054 0.4240 0.9562 > join_dataframe_index_single_key_bigger 6.4616 6.7578 0.9562 > timeseries_timestamp_downsample_mean 4.5849 4.7787 0.9594 > frame_fancy_lookup 2.5498 2.6544 0.9606 > series_value_counts_int64 2.5569 2.6581 0.9619 > frame_fancy_lookup_all 30.7510 31.8465 0.9656 > index_int64_union 82.2279 85.1500 0.9657 > indexing_dataframe_boolean_rows_object 0.4809 0.4977 0.9662 > frame_ctor_nested_dict 91.6129 94.8122 0.9663 > stat_ops_series_std 0.2450 0.2533 0.9673 > groupby_frame_cython_many_columns 3.7642 3.8894 0.9678 > timeseries_asof 10.4352 10.7721 0.9687 > series_ctor_from_dict 3.7707 3.8749 0.9731 > frame_drop_dup_inplace 3.0007 3.0746 0.9760 > timeseries_large_lookup_value 0.0242 0.0248 0.9764 > read_table_multiple_date_baseline 1201.2930 1224.3881 0.9811 > dti_reset_index 0.6339 0.6457 0.9817 > read_table_multiple_date 2600.7280 2647.8729 0.9822 > reindex_frame_level_reindex 0.9524 0.9674 0.9845 > reindex_multiindex 1.3483 1.3685 0.9853 > frame_insert_500_columns 102.1249 103.4329 0.9874 > frame_drop_duplicates 19.3780 19.6157 0.9879 > reindex_daterange_backfill 0.1870 0.1889 0.9899 > stats_rank2d_axis0_average 25.0480 25.2801 0.9908 > series_align_left_monotonic 13.1929 13.2558 0.9953 > timeseries_add_irregular 22.4635 22.5122 0.9978 > read_store_mixed 13.4398 13.4560 0.9988 > lib_fast_zip 11.1289 11.1354 0.9994 > match_strings 0.3831 0.3833 0.9995 > read_store 5.5526 5.5290 1.0043 > timeseries_sort_index 22.7172 22.5976 1.0053 > timeseries_1min_5min_mean 0.6224 0.6175 1.0079 > stats_rank2d_axis1_average 14.6569 14.5339 1.0085 > reindex_daterange_pad 0.1886 0.1867 1.0102 > timeseries_period_downsample_mean 6.4241 6.3480 1.0120 > frame_drop_duplicates_na 19.3303 19.0970 1.0122 > stats_rank_average_int 23.3569 22.9996 1.0155 > lib_fast_zip_fillna 14.1394 13.8473 1.0211 > index_datetime_intersection 17.2626 16.8986 1.0215 > timeseries_1min_5min_ohlc 0.7054 0.6891 1.0237 > stats_rank_average 31.3440 30.3845 1.0316 > timeseries_infer_freq 10.9854 10.6439 1.0321 > timeseries_slice_minutely 0.0637 0.0611 1.0418 > index_datetime_union 17.9083 17.1640 1.0434 > series_align_irregular_string 89.9470 85.1344 1.0565 > series_constructor_ndarray 0.0127 0.0119 1.0742 > indexing_panel_subset 0.5692 0.5214 1.0917 > groupby_apply_dict_return 46.3497 42.3220 1.0952 > reshape_unstack_simple 3.2901 2.9089 1.1310 > timeseries_to_datetime_iso8601 4.2305 3.6015 1.1746 > frame_to_string_floats 53.6217 37.2041 1.4413 > reshape_pivot_time_series 170.4340 107.9068 1.5795 > sparse_frame_constructor 6.2714 3.5053 1.7891 > datetimeindex_normalize 37.2718 6.9329 5.3761 > > Columns: test_name | target_duration [ms] | baseline_duration [ms] | ratio Hi Collin, I didn't add it to the official docs because of the authentication step complicating the doc build, but you can reference this brief blog post I wrote here: http://quantabee.wordpress.com/2012/12/17/google-analytics-pandas/ Best, Chang

1 0

Travis-CI stopped supporting Python 3.1, but added 3.3
by Ondřej Čertík Dec. 21, 2012

Dec. 21, 2012

Hi, I noticed that the 3.1 tests are now failing. After clarification with the Travis guys: https://groups.google.com/d/topic/travis-ci/02iRu6kmwY8/discussion I've submitted a fix to our .travis.yml (and backported to 1.7): https://github.com/numpy/numpy/pull/2850 https://github.com/numpy/numpy/pull/2851 In case you were wondering. Do we need to support Python 3.1? We could in principle test 3.1 just like we test 2.4. I don't know if it is worth the pain. Ondrej

3 3

NaN (Not a Number) occurs in calculation of complex number for Bessel functions
by Happyman Dec. 21, 2012

Dec. 21, 2012

DEAR PYTHON USERS DO MATHEMATICAL FUNCTIONS HAVE LIMITATION IN PYTHON in comparison with other programming languages ???? I have two mathematical functions: from scipy.special import sph_jn, sph_jnyn 1) sph_jn (n, z) ---> n is float, z is complex number for example: a,b=sph_jn ( 2.0 , 5+0.4j ) gives the following result: >>> a array( [ - 0.20416243 + 0.03963597j, - 0.10714653 - 0.06227716j, 0.13731305 - 0.07165432j ] ) >>>b array( [ 0.10714653 + 0.06227716j, -0.15959617 + 0.06098154j, -0.18559289 - 0.01300886j ] ) 2) sph_jnyn(n , x) --> n-float, x - float ,for example: c,d,e,f=sph_jnyn(2.0 , 3.0) >>> c array( [ 0.04704 , 0.3456775, 0.2986375 ] ) >>> d array( [-0.3456775 , -0.18341166, 0.04704 ] ) >>> e array( [ 0.3299975 , 0.06295916, -0.26703834 ] ) >>> f array( [ -0.06295916, 0.28802472, 0.3299975 ] ) PROBLEM IS HERE!!! BUT , IF I GIVE ( it is necessary value for my program ): a , b = sph_jn ( 536 , 2513.2741228718346 + 201.0619298974676j ) I would like to see even very very deep comments as specific as possible!!!!!!

4 10

Byte aligned arrays
by Henry Gomersall Dec. 21, 2012

Dec. 21, 2012

I've written a few simple cython routines for assisting in creating byte-aligned numpy arrays. The point being for the arrays to work with SSE/AVX code. https://github.com/hgomersall/pyFFTW/blob/master/pyfftw/utils.pxi The change recently has been to add a check on the CPU as to what flags are supported (though it's not complete, I should make the default return 0 or something). It occurred to me that this is something that (a) other people almost certainly need and are solving themselves and (b) I lack the necessary platforms to test all the possible CPU/OS combinations to make sure something sensible happens in all cases. Is this something that can be rolled into Numpy (the feature, not my particular implementation or interface - though I'd be happy for it to be so)? Regarding (b), I've written a test case that works for Linux on x86-64 with GCC (my platform!). I can test it on 32-bit windows, but that's it. Is ARM supported by Numpy? Neon would be great to include as well. What other platforms might need this? Cheers, Henry

7 38

DARPA funding for Blaze and passing the NumPy torch
by Travis Oliphant Dec. 21, 2012

Dec. 21, 2012

Hello all, There is a lot happening in my life right now and I am spread quite thin among the various projects that I take an interest in. In particular, I am thrilled to publicly announce on this list that Continuum Analytics has received DARPA funding (to the tune of at least $3 million) for Blaze, Numba, and Bokeh which we are writing to take NumPy, SciPy, and visualization into the domain of very large data sets. This is part of the XDATA program, and I will be taking an active role in it. You can read more about Blaze here: http://blaze.pydata.org. You can read more about XDATA here: http://www.darpa.mil/Our_Work/I2O/Programs/XDATA.aspx I personally think Blaze is the future of array-oriented computing in Python. I will be putting efforts and resources next year behind making that case. How it interacts with future incarnations of NumPy, Pandas, or other projects is an interesting and open question. I have no doubt the future will be a rich ecosystem of interoperating array-oriented data-structures. I invite anyone interested in Blaze to participate in the discussions and development at https://groups.google.com/a/continuum.io/forum/#!forum/blaze-dev or watch the project on our public GitHub repo: https://github.com/ContinuumIO/blaze. Blaze is being incubated under the ContinuumIO GitHub project for now, but eventually I hope it will receive its own GitHub project page later next year. Development of Blaze is early but we are moving rapidly with it (and have deliverable deadlines --- thus while we will welcome input and pull requests we won't have a ton of time to respond to simple queries until at least May or June). There is more that we are working on behind the scenes with respect to Blaze that will be coming out next year as well but isn't quite ready to show yet. As I look at the coming months and years, my time for direct involvement in NumPy development is therefore only going to get smaller. As a result it is not appropriate that I remain as "head steward" of the NumPy project (a term I prefer to BFD12 or anything else). I'm sure that it is apparent that while I've tried to help personally where I can this year on the NumPy project, my role has been more one of coordination, seeking funding, and providing expert advice on certain sections of code. I fundamentally agree with Fernando Perez that the responsibility of care-taking open source projects is one of stewardship --- something akin to public service. I have tried to emulate that belief this year --- even while not always succeeding. It is time for me to make official what is already becoming apparent to observers of this community, namely, that I am stepping down as someone who might be considered "head steward" for the NumPy project and officially leaving the development of the project in the hands of others in the community. I don't think the project actually needs a new "head steward" --- especially from a development perspective. Instead I see a lot of strong developers offering key opinions for the project as well as a great set of new developers offering pull requests. My strong suggestion is that development discussions of the project continue on this list with consensus among the active participants being the goal for development. I don't think 100% consensus is a rigid requirement --- but certainly a super-majority should be the goal, and serious changes should not be made with out a clear consensus. I would pay special attention to under-represented people (users with intense usage of NumPy but small voices on this list). There are many of them. If you push me for specifics then at this point in NumPy's history, I would say that if Chuck, Nathaniel, and Ralf agree on a course of action, it will likely be a good thing for the project. I suspect that even if only 2 of the 3 agree at one time it might still be a good thing (but I would expect more detail and discussion). There are others whose opinion should be sought as well: Ondrej Certik, Perry Greenfield, Robert Kern, David Cournapeau, Francesc Alted, and Mark Wiebe to name a few. For some questions, I might even seek input from people like Konrad Hinsen and Paul Dubois --- if they have time to give it. I will still be willing to offer my view from time to time and if I am asked. Greg Wilson (of Software Carpentry fame) asked me recently what letter I would have written to myself 5 years ago. What would I tell myself to do given the knowledge I have now? I've thought about that for a bit, and I have some answers. I don't know if these will help anyone, but I offer them as hopefully instructive: 1) Do not promise to not break the ABI of NumPy --- and in fact emphasize that it will be broken at least once in the 1.X series. NumPy was designed to add new data-types --- but not without breaking the ABI. NumPy has needed more data-types and still needs even more. While it's not beautifully simple to add new data-types, it can be done. But, it is impossible to add them without breaking the ABI in some fashion. The desire to add new data-types *and* keep ABI compatibility has led to significant pain. I think the ABI non-breakage goal has been amplified by the poor state of package management in Python. The fact that it's painful for someone to update their downstream packages when an upstream ABI breaks (on Windows and Mac in particular) has put a lot of unfortunate pressure on this community. Pressure that was not envisioned or understood when I was writing NumPy. (As an aside: This is one reason Continuum has invested resources in building the conda tool and a completely free set of binary packages called Anaconda CE which is becoming more and more usable thanks to the efforts of Bryan Van de Ven and Ilan Schnell and our testing team at Continuum. The conda tool: http://docs.continuum.io/conda/index.html is open source and BSD licensed and the next release will provide the ability to build packages, build indexes on package repositories and interface with pip. Expect a blog-post in the near future about how cool conda is!). 2) Don't create array-scalars. Instead, make the data-type object a meta-type object whose instances are the items returned from NumPy arrays. There is no need for a separate array-scalar object and in fact it's confusing to the type-system. I understand that now. I did not understand that 5 years ago. 3) Special-case small arrays to avoid the memory indirection and look at PDL so that generalized ufuncs are supported from the beginning. 4) Define missing-value data-types and labels on the dimensions and arrays 5) Define a standard "dictionary of NumPy arrays" interface as the basic "structure of arrays" concept to go with the "array of structures" that structured arrays provide. 6) Start work on SQL interface to NumPy arrays *now* Additional comments I would make to someone today: 1) Most of NumPy should be written in Python with Numba used as the compiler (particularly as soon as Numba gets the ability to create Python extension modules which is in the next release). 2) There are still many, many optimizations that can be made in NumPy run-time (especially in the face of modern hardware). I will continue to be available to answer questions and I may chime in here and there on pull requests. However, most of my time for NumPy will be on administrative aspects of the project where I will continue to take an active interest. To help make sure that this happens in a transparent way, I would like to propose that "administrative" support of the project be left to the NumFOCUS board of which I am currently 1 of 9 members. The other board members are currently: Ralf Gommers, Anthony Scopatz, Andy Terrel, Prabhu Ramachandran, Fernando Perez, Emmanuelle Gouillart, Jarrod Millman, and Perry Greenfield. While NumFOCUS basically seeks to promote and fund the entire scientific Python stack, I think it can also play a role in helping to administer some of the core projects which the board members themselves have a personal interest in. By administrative support, I mean decisions like "what should be done with any NumPy IP or web-domains" or "what kind of commercially-related ads or otherwise should go on the NumPy home page", or "what should be done with the NumPy github account", etc. --- basically anything that requires an executive decision that is not directly development related. I don't expect there to be many of these decisions. But, when they show up, I would like them to be made in as transparent and public of a way as possible. In practice, the way I see this working is that there are members of the NumPy community who are (like me) particularly interested in admin-related questions and serve on a NumPy team in the NumFOCUS organization. I just know I'll be attending NumFOCUS board meetings, and I would like to help move administrative decisions forward with NumPy as part of the time I spend thinking about NumFOCUS. If people on this list would like to play an active role in those admin discussions, then I would heartily welcome them into NumFOCUS membership where they would work with interested members of the NumFOCUS board (like me and Ralf) to help direct that organization. I would really love to have someone from this list volunteer to serve on the NumPy team as part of the NumFOCUS project. I am certainly going to be interested in the opinions of people who are active participants on this list and on GitHub pages for NumPy on anything admin related to NumPy, and I expect Ralf would also be very interested in those views. One admin discussion that I will bring up in another email (as this one is already too long) is about making 2 or 3 lists for NumPy such as numpy-admin(a)numpy.org, numpy-dev(a)numpy.org, and numpy-users@numpy-org. Just because I'll be spending more time on Blaze, Numba, Bokeh, and the PyData ecosystem does not mean that I won't be around for NumPy. I will continue to promote NumPy. My involvement with Continuum connects me to NumPy as Continuum continues to offer commercial support contracts for NumPy (and SciPy and other open source projects). Continuum will also continue to maintain its Github NumPy project which will contain pull requests from our company that we are working to get into the mainline branch. Continuum will also continue to provide resources for release-management of NumPy (we have been funding Ondrej in this role for the past 6 months --- though I would like to see this happen through NumFOCUS in the future even if Continuum provides much of the money). We also offer optimized versions of NumPy in our commercial Anaconda distribution (Anaconda CE is free and open source). Also, I will still be available for questions and help (I'm not disappearing --- just making it clear that I'm stepping back into an occasional NumPy developer role). It has been extremely gratifying to see the number of pull-requests, GitHub-conversations, and code contributions increase this year. Even though the 1.7 release has taken a long time to stabilize, there have been a lot of people participating in the discussion and in helping to track down the problems, figure out what to do, and fix them. It even makes it possible for people to think about 1.7 as a long-term release. I will continue to hope that the spirit of openness, tolerance, respect, and gratitude continue to permeate this mailing list, and that we continue to seek to resolve any differences with trust and mutual respect. I know I have offended people in the past with quick remarks and actions made sometimes in haste without fully realizing how they might be taken. But, I also know that like many of you I have always done the very best I could for moving Python for scientific computing forward in the best way I know how. Thank you for the great memories. If you will forgive a little sentiment: My daughter who is in college now was 3 years old when I began working with this community and went down a road that would lead to my involvement with SciPy and NumPy. I have marked the building of my family and the passage of time with where the Python for Scientific Computing Community was at. Like many of you, I have given a great deal of attention and time to building this community. That sacrifice and time has led me to love what we have created. I know that I leave this segment of the community with the tools in better hands than mine. I am hopeful that NumPy will continue to be a useful array library for the Python community for many years to come even as we all continue to build new tools for the future. Very best regards, -Travis

6 8