Mailman 3 December 2011 - NumPy-Discussion

Problem with changes to f2py
by John Salvatier Dec. 29, 2011

Dec. 29, 2011

Hi Numpy users! I maintain the boundary value problem solver package scikits.bvp_solver<http://pypi.python.org/pypi/scikits.bvp_solver>. It's had problems with f2py for a while, and I am not sure where they are coming from. I made this stackoverflow post<http://stackoverflow.com/questions/7707414/problems-compiling-with-changes-…>some time ago, but I didn't get any solutions. Here are the details: I am trying to update my package scikits.bvp_solver (source here<https://github.com/jsalvatier/scikits.bvp_solver/>) and I have run into some problems with f2py generated files. The files 'bvp_solverf-f2pywrappers2.f90' and 'bvp_solverfmodule.c' which were generated in 2009 allow the package to be built in place with "python setup.py build_ext --inplace" but if I delete them and try to rebuild I get the error scikits/bvp_solver/lib/bvp_solverf-f2pywrappers2.f90:218.48: use guess_3_wrap__user__routines 1 Fatal Error: Can't open module file 'guess_3_wrap__user__routines.mod' for reading at (1): No such file or directory scikits/bvp_solver/lib/bvp_solverf-f2pywrappers2.f90:11.19: The file bvp_interface.pyf specifies a python module guess_3_wrap__user__routines, but I do not see this show up in the f2py generated modules. The part the adds this use statement does not appear in the old version of the file. I am having difficulty figuring out how to fix this issue. Can anyone offer advice? What are the major changes to f2py in the last two years? I appreciate all clues on this issue. Thank you in advance, John

1 0

fft help
by Burlen Loring Dec. 29, 2011

Dec. 29, 2011

Hi I have an image I need to do an fft on, I tried numpy.fft but results are not what I expected, and differ from matlab. My input image is a weird size, 5118x1279, I think numpy fft is not liking it. In numpy the fft appears to be computed multiple times and tiled across the output image. In other words the pattern I see in matlab fft is tiled repeatedly over numpy fft output. Any idea on what I'm doing wrong? you can see repeated pattern in the top panel of this image which also has the input in the bottom panel. http://old.nabble.com/file/p33047057/fft_uex.png fft_uex.png tx

2 4

Re: [Numpy-discussion] NumPy Governance
by Joe Harrington Dec. 29, 2011

Dec. 29, 2011

I had intended to stay quiet in this discussion since I am not a core developer and also no longer even lead the doc project. However, I've watched two organizations go very wrong very fast recently. Both were similar in structure to this one. I've done some study as a result and there are some lessons to learn. The committee-of-doers-and-executive format is very effective when it works. It's worked well for numpy and many other OSS projects. Benevolence and commitment of the leader are made likely by choosing someone who is already doing a lot and whom everyone knows. Yet it can still go wrong. By studying the failures, as well as the successes, we can build in some safeguards. So, I'll tell the story of Pack 608. (It's the next four paragraphs if you want to skip it.) This is a Cub Scout Pack, which for those who don't already know is a group of 20-100 boys, ages 6-10, who participate in a variety of indoor and outdoor activities that are designed to build good character (whether they do is a different discussion). Our group has about 80 boys. It is run by parents who volunteer their time, and it is led by a Pack Committee, with a Chair and a Chartered Organization Representative. The latter is an overlord who can overrule anything but by design usually observes from the sidelines and only steps in when things go wrong. He represents the organization (usually a church or community service organization) that owns the pack. There is also a Cubmaster, who is the face of the organization to the boys. He reports to the Committee but is a really big dog there, often as big as the Chair. Sometimes bigger. Running a group of 80 boys doing a dozen activities a year is a huge job, way too big for any individual with a job, so finding and developing parent volunteers is hard. In our group, one married couple, extremely committed, got involved and began doing a huge number of tasks. Whenever anyone slacked off, they took over that job. They did those jobs extremely well. As our top leadership moved on (their boys turned 11 and became Boy Scouts), eventually we looked around and the clear choice for the Chair and Cubmaster were these two. We were not exited about installing a married couple in two of three key posts, but we had few options. We didn't want to do it, but nobody else stepped up. It was the biggest mistake we ever made. Even though everything this couple did was, in their eyes, in the best interest of the boys and the group, their style was so overbearing and their opinions so inflexible that there were serious verbal conflicts and other leaders began to quit. Some non-leaders complained, the Chartered Org. Rep. called a meeting, and these two were forced to step down. But, by this time, they were doing so many tasks that between them and the other departed leaders, we had lost EIGHT critical leadership roles (for those who know Scouts, these were Committee Chair, Cubmaster, Outdoors Chair, Events Chair, Advancement Chair, Fundraising Chair, and two Den Leaders). The pack nearly folded, but we managed to pull in a few parents (including me) who were neither eager to get close to this group nor really ready to spend a lot of time on it. We aren't yet up to where we used to be a year ago, but we're surviving. It's been brutal on our personal and work time, however. It would have been much easier had we stepped up before the fiasco. While such near-collapses are infrequent in small organizations, they happen often enough that many readers have probably experienced one. They can happen out of poor choice of leaders, feelings of possessiveness, differences of vision among stakeholders, and a host of other causes. The point here is to build into any organization plan not just the structure that will let it succeed, but also the safeguards that will reduce the chances of failure by catching and correcting organizational health problems early. With OSS, we have an ultimate safeguard, which is that anyone can fork the source and start a new org. But, as we've seen in this project, forks are *very* damaging and can take years to recover from, so it makes sense to think about safeguards in this organization, too. There are organizations of organizations that provide advice on this topic. Among the key suggestions: - Have uninvolved oversight. In Pack 608's case this was the Chartered Org. Rep. For us it could be an external board with some community reps. This oversight itself can be abused, so it must be limited to removing problem leaders and calling an election, or even just to calling an election. It should probably NOT extend to changing decisions on the code, other than forcing a vote of reconsideration. - Rotate key posts. When people do a job forever, they can become possessive of it. Not everyone does, but it's common. The job becomes tailored to the person, others don't know how it's done, key practices go undocumented, and if that person leaves or is temporarily unavailable at a key time or if a change needs to be made and that person disagrees, there's trouble. Feeling part ownership is good. Feeling full possession is not. The lack of central ownership is a big piece of why we can trust OSS in a way we can't trust commercial software. So, rotate key jobs. - Ensure that critical authorities (keys, passwords, certificates, title to important property like a web domain, signatory authority on a bank account) are never held by just one or even two people. Make sure the oversight entity has ultimate access. - Have multiple sources of key resources, like labor and money, so that the threat of taking them away is not debilitating. This means not letting one person hold too many jobs, too. - Have a well defined voting procedure and rules of order. You don't have to use them all the time when consensus is clear. They can be slow and cumbersome. But, when there is conflict, they ensure fairness. - Have a no-confidence vote procedure that removes a problem Chair. The rules for this need to be carefully thought out. - Have the consent group, and not the outgoing Chair, choose the next Chair. There are many reasons a Chair moves on, including lack of time, becoming tired of the job, and differences of opinion with the consent group. Especially in the latter case, you don't want the former Chair to influence the future. - Have open meetings with posted minutes. Keep no secrets (except passwords, etc.). As Ondrej pointed out, identifying the potential leaders isn't hard, in our case, but I think those in the consent group should include more than just coders. There has been a lot of work on the docs, for example, though not as much recently. Leaders of key client packages should be involved. Representatives of key user classes (astronomers, neuroscientists, statisticians, numerical programming teachers, newbies, etc.) would be important. Perhaps it's ok to let most coding questions go to the coders, but significant decisions that could change the direction of the effort should be referred to the wider group of stakeholders. That could be the mailing list, but then voting becomes vague. I would suggest including some number of core coders and an equal number of community representatives. Or perhaps the community reps become the oversight board, which also includes a small minority of coding reps. One method that works well for continuity is to elect a Vice-Chair who shadows the Chair and is involved in decisions for a period, then becomes Chair. The Past Chair can be a formal position, too, whose role it is to provide advice. This also eases the transition out of the top job. The Past Chair should be a resource only, and have no special authority. There are lots of sample organizational bylaws available online that can serve as a template. If we join an existing organization, e.g., for fundraising, they likely have requirements that our bylaws need to meet. To sum it up, the BDFL system is great if you have a truly BD. It's most likely to find one in the founder of an effort. Once they step down, be wary of centralizing too much authority. If one out of four leaders is less than fully committed, competent, and benevolent, it's trouble. We should at least have oversight and procedures that can remove a problem leader. Numpy is at a difficult stage, where we like the freedom of not having written rules and procedures, but we're at risk if we don't. Much larger and it's clear we need them, and vice versa. The time we'll wish we had them is when we have a leadership crisis. The chances of having one are small, but not vanishingly so. --jh--

1 0

NumPy Governance
by Travis Oliphant Dec. 28, 2011

Dec. 28, 2011

Hi everyone, There have been some wonderfully vigorous discussions over the past few months that have made it clear that we need some clarity about how decisions will be made in the NumPy community. When we were a smaller bunch of people it seemed easier to come to an agreement and things pretty much evolved based on (mostly) consensus and who was available to actually do the work. There is a need for a more clear structure so that we know how decisions will get made and so that code can move forward while paying attention to the current user-base. There has been a "steering committee" structure for SciPy in the past, and I have certainly been prone to lump both NumPy and SciPy together given that I have a strong interest in and have spent a great amount of time working on both projects. Others have also spent time on both projects. However, I think it is critical at this stage to clearly separate the projects and define a governing structure that is fair and agreeable for NumPy. SciPy has multiple modules and will probably need structure around each module independently. For now, I wanted to open up a discussion to see what people thought about NumPy's governance. My initial thoughts: * discussions happen as they do now on the mailing list * a small group of developers (5-11) constitute the "board" and major decisions are made by vote of that group (not just simple majority --- needs at least 2/3 +1 votes). * votes are +1/+0/-0/-1 * if a topic is difficult to resolve it is moved off the main list and discussed on a separate "board" mailing list --- these should be rare, but parts of the NA discussion would probably qualify * This board mailing list is "publically" viewable but only board members may post. * The board is renewed and adjusted each year --- based on nomination and 2/3 vote of the current board until board is at 11. * The chairman of the board is voted by a majority of the board and has veto power unless over-ridden by 3/4 of the board. * Petitions to remove people off the board can be made by 50+ independent reverse nominations (hopefully people will just withdraw if they are no longer active). All of these points are open for discussion. I just thought I would start the conversation. I will be much more active this next year with NumPy and will be very interested in the direction NumPy is taking. I'm hoping to discern by this conversation, who else is very interested in the direction of NumPy so that the first board can be formally constituted. Best regards, -Travis

12 19

How's our broadcasting?
by Jordi Gutiérrez Hermoso Dec. 28, 2011

Dec. 28, 2011

Just FYI, the next stable release of Octave (3.6) will have broadcasting. I used Numpy as an inspiration. Here is the WIP manual for it: http://jordi.platinum.linux.pl/octave.html/Broadcasting.html#Broadcasting I want to thank Numpy both for the inspiration and for any comments you may have on this. I'm pretty excited that Octave will now have this feature. Thanks, - Jordi G. H. GNU Octave developer

3 3

Indexing empty dimensions with empty arrays
by Jordi Gutiérrez Hermoso Dec. 28, 2011

Dec. 28, 2011

I have been instructed to bring this issue to the mailing list: http://projects.scipy.org/numpy/ticket/1994 TIA, - Jordi G. H.

5 14

Functions vs Methods
by Jaidev Deshpande Dec. 28, 2011

Dec. 28, 2011

Hi It is said that function calls are expensive. Does that mean one must use available methods instead? Eg. x is a NumPy array, and I need its transpose Should I use >>> x.T or >>> numpy.transpose(T) ? Does it matter which one I'm using ? If not, under what conditions does it become important to think about this distinction? Thanks

3 2

trouble building numpy 1.6.1 on Scientific Linux 5
by Russell E. Owen Dec. 28, 2011

Dec. 28, 2011

I'm trying to build numpy 1.6.1 on Scientific Linux 5 but the unit tests claim the wrong version of fortran was used. I thought I knew how to avoid that, but it's not working. I don't have atlas (this needs to run on a lot of similar-but-not-identical machines). I believe blas and lapack were built against gfortran: -bash-3.2$ ldd /usr/lib64/libblas.so linux-vdso.so.1 => (0x00007fff4bffd000) libm.so.6 => /lib64/libm.so.6 (0x00002ab26a0c8000) libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002ab26a34c000) libc.so.6 => /lib64/libc.so.6 (0x00002ab26a5e3000) /lib64/ld-linux-x86-64.so.2 (0x0000003b2ba00000) -bash-3.2$ ldd /usr/lib64/liblapack.so linux-vdso.so.1 => (0x00007fffe97fd000) libblas.so.3 => /usr/lib64/libblas.so.3 (0x00002b6438d75000) libm.so.6 => /lib64/libm.so.6 (0x00002b6438fca000) libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002b643924d000) libc.so.6 => /lib64/libc.so.6 (0x00002b64394e4000) /lib64/ld-linux-x86-64.so.2 (0x0000003b2ba00000) The sysadmins have provided a gcc 4.4.0 compiler that I access using symlinks on my $PATH: -bash-3.2$ which gcc g++ gfortran ~/local/bin/gcc ~/local/bin/g++ ~/local/bin/gfortran -bash-3.2$ ls -l ~/local/bin lrwxrwxrwx 1 rowen astro 14 Oct 28 2010 g++ -> /usr/bin/g++44 lrwxrwxrwx 1 rowen astro 14 Oct 28 2010 gcc -> /usr/bin/gcc44 lrwxrwxrwx 1 rowen astro 19 Dec 5 16:40 gfortran -> /usr/bin/gfortran44 -bash-3.2$ gfortran --version GNU Fortran (GCC) 4.4.0 20090514 (Red Hat 4.4.0-6) Copyright (C) 2009 Free Software Foundation, Inc. For this log I used a home-bulit python 2.6.5 that is widely used. However, I've tried it with other builds of python that are on our system, as well, with no better success (including a Python 2.7.2). -bash-3.2$ which python /astro/apps/pkg/python64/bin/python -bash-3.2$ python Python 2.6.5 (r265:79063, Aug 4 2010, 11:27:53) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. numpy seems to see gfortran when it builds: -bash-3.2$ python setup.py build --fcompiler=gnu95 Running from numpy source directory.non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: ... NOT AVAILABLE atlas_blas_threads_info: ... NOT AVAILABLE atlas_blas_info: ... NOT AVAILABLE blas_info: ... FOUND: libraries = ['blas'] library_dirs = ['/usr/lib64'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib64'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: ... NOT AVAILABLE NOT AVAILABLE atlas_threads_info: ... NOT AVAILABLE atlas_info: ... NOT AVAILABLE /astro/users/rowen/build/numpy-1.6.1/numpy/distutils/system_info.py:1330: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /astro/apps/lsst_w12_sl5/Linux64/external/python/2.7.2+2/lib ... FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib64'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib64'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.linux-x86_64-2.7 creating build/src.linux-x86_64-2.7/numpy creating build/src.linux-x86_64-2.7/numpy/distutils building library "npymath" sources customize Gnu95FCompiler Found executable /astro/users/rowen/local/bin/gfortran # I install it in an out-of-the-way location just so I can test it -bash-3.2$ python setup.py install --home=~/local ... -bash-3.2$ cd -bash-3.2$ python >>> import numpy >>> numpy.__path__ ['/astro/users/rowen/local/lib/python/numpy'] >>> numpy.test() Running unit tests for numpy NumPy version 1.6.1 NumPy is installed in /astro/users/rowen/local/lib/python/numpy Python version 2.6.5 (r265:79063, Aug 4 2010, 11:27:53) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] nose version 0.11.4 .... ====================================================================== FAIL: test_lapack (test_build.TestF77Mismatch) ---------------------------------------------------------------------- Traceback (most recent call last): File "/astro/users/rowen/local/lib/python/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/astro/users/rowen/local/lib/python/numpy/linalg/tests/test_build.py", line 50, in test_lapack information.""") AssertionError: Both g77 and gfortran runtimes linked in lapack_lite ! This is likely to cause random crashes and wrong results. See numpy INSTALL.txt for more information. ---------------------------------------------------------------------- Ran 3533 tests in 13.400s Any suggestions on how to fix this? -- Russell

3 9

Returning tuple of numpy arrays from a Numpy C-extension?
by Åke Kullenberg Dec. 27, 2011

Dec. 27, 2011

I have put together a few c extensions following the documentation on http://www.scipy.org/Cookbook/C_Extensions/NumPy_arrays. There is however one thing that stumps me. To illustrate with a simple code snippet, the test function below multiplies the input numpy double array by two. So far so good. But how about if I want the function to return a tuple of two numpy arrays so I could do 'a, b = myCLib.test(c)' from a Python script? It's straight forward to convert C data structures to Python objects with Py_BuildValue, but how can I do this for numpy arrays instead? static PyObject * test(PyObject *self, PyObject *args) { PyArrayObject *py_in, *py_out; double *in, *out; int i, n, dims[2]; if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &py_in)) return NULL; if (NULL == py_in || not_doublevector(py_in)) return NULL; n = py_in->dimensions[0]; dims[0] = n; dims[1] = 1; py_out = (PyArrayObject *) PyArray_FromDims(1, dims, NPY_DOUBLE); in = pyvector_to_Carrayptrs(py_in); out = pyvector_to_Carrayptrs(py_out); for (i=0; i<n; i++) { out[i] = in[i] * 2.0; } return PyArray_Return(py_out); }

2 2

hello
by pearu.peterson＠gmail.com Dec. 27, 2011

Dec. 27, 2011

The message contains Unicode characters and has been sent as a binary attachment.

1 0