Mailman 3 May 2012 - NumPy-Discussion

CEP 1001 - Custom PyTypeObject extensions
by Dag Sverre Seljebotn 11 May '12

11 May '12

This comes from a refactor of the work on CEP 1000: It's a pre-PEP with a hack we can use *today*, that allows 3rd party libraries to agree on extensions to the PyTypeObject structure. http://wiki.cython.org/enhancements/cep1001 As hinted in my other recent thread, I believe this will also be good medicine as NumPy moves forward, so that we stop worrying about "ndarray" vs. C extensions, and rather talk about which PyTypeObject extensions are supported. Technical discussion about this on the Cython list please. Dag

1 0

spurious space in printing record arrays?
by Benjamin Root 11 May '12

11 May '12

Just noticed this in the output from printing some numpy record arrays: [[('2008081712', -24, -78.0, 20.100000381469727, 45.0, -999.0, 0.0)] [ ('2008081718', -18, -79.5999984741211, 20.700000762939453, 45.0, -999.0, 0.0)] [ ('2008081800', -12, -80.30000305175781, 21.100000381469727, 45.0, -999.0, 0.0)] [ ('2008081806', -6, -80.80000305175781, 21.899999618530273, 45.0, -999.0, 0.0)] [ ('2008081812', 0, -81.19999694824219, 23.200000762939453, 50.0, -999.0, 1002.0)]] [[ ('2008081812', 0, -81.19999694824219, 23.200000762939453, 50.0, -999.0, 0.0)] [('2008081815', 3, -81.5, 23.600000381469727, 50.0, -999.0, 1003.0)] [ ('2008081900', 12, -81.80000305175781, 24.600000381469727, 55.0, -999.0, 0.0)] [ ('2008081912', 24, -82.0999984741211, 26.200000762939453, 65.0, -999.0, 0.0)] [('2008082000', 36, -82.0, 27.799999237060547, 50.0, -999.0, 0.0)] [ ('2008082012', 48, -81.80000305175781, 29.299999237060547, 40.0, -999.0, 0.0)] [('2008082112', 72, -81.5, 31.5, 35.0, -999.0, 0.0)] [('2008082212', 96, -81.5, 33.599998474121094, 25.0, -999.0, 0.0)] [('2008082312', 120, -82.5, 35.5, 20.0, -999.0, 0.0)]] On my 80-character wide terminal window, each line that gets wrapped also has an extra space after the inner square bracket. Coincidence? Using v1.6.1 I don't think it is a big problem... just odd. Thanks, Ben Root

1 0

read line mixed with string and number?
by Chao YUE 10 May '12

10 May '12

Dear all, I have files which contain lines like this: 30516F5 Sep 1985 1-Day Rain Trace 0.2 3.2 Trace 0.0 0.0 0.0 0.0 0.2 0.0 Trace 29.2 0.0 0.0 0.0 0.0 1.8 30516F5 Sep 1985 1-Day Snow Trace 0.0 0.0 0.0 14.8 10.1 Trace 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Trace Trace 0.0 30516F5 Sep 1985 1-Day Pcpn. Trace 0.2 3.2 Trace 18.9 9.8 Trace 0.0 0.2 0.0 Trace 29.2 0.0 0.0 0.0 Trace 1.8 0.0 30516F5 May 1986 Max. Temp. Misg Misg Misg Misg Misg Misg 9.0 8.0 8.0 0.0 6.0 1.0 1.0 -3.0 3. 30516F5 May 1986 Min. Temp. Misg Misg Misg Misg Misg Misg Misg -1.0 -2.0 -6.0 -5.0 -5.0 -3.0 -7.0 -6.0 -5.0 -3.0 different columns were separated by blank spaces. with the first column as sitename, second as month name, then year, then variable name and data. I want to read them line by line into a list, and then connect all the numerical data within one year into a list, and then combining different year data into one masked ndarray, in this process, I check the flags (Trace, Misg, etc.) and replace them as unique values (or missing values). and then begin to analyse the data. each file contains only one site, it can be big or small depending on the number of years. I don't know what's the good way to do this job. what I am thinking is to read one file line by line, and then divide this line by blank space, and replace special flag. but during this process, I need to do type conversion. any suggestion would be appreciated. Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0

[SciPy-Dev] Announce: scikit-learn v0.11
by Gael Varoquaux 10 May '12

10 May '12

On behalf of Andy Mueller, our release manager, I am happy to announce the 0.11 release of scikit-learn. This release includes some major new features such as randomized sparse models, gradient boosted regression trees, label propagation and many more. The release also has major improvements in the documentation and in stability. Details can be found on the [1]what's new page. We also have a new page with [2]video tutorials on machine learning with scikit-learn and different aspects of the package. Sources and windows binaries are available on sourceforge, through pypi (http://pypi.python.org/pypi/scikit-learn/0.11) or can be installed directly using pip: pip install -U scikit-learn Thanks again to all the contributors who made this release possible. Cheers, Gaël 1. http://scikit-learn.org/stable/whats_new.html 2. http://scikit-learn.org/stable/presentations.html

2 2

Re: [Numpy-discussion] Documentation roles in the numpy/scipy documentation editor
by Joe Harrington 10 May '12

10 May '12

We considered lowering the review standard near the end of my direct involvement in the doc project but decided not to. You didn't mention any benefit to the proposed changes, so while I'm not active in the doc project anymore, let me relate our decision. It's often the case that docstrings get written fast, and it's usually the case that they're written by a single person, who has a single perspective. We wanted to make docs that were professional, that could be placed next to the manuals for IDL, Matlab, etc. without embarrassment. So, we set up a system similar to academic publishing. Every docstring would be seen by two sets of critical eyes, and for major X.0 releases we'd pay a proofreader to spend a few days to polish off the English and get the style totally consistent. At the same time, we needed to get something decent in every docstring fast, so we made that the priority. About the time we achieved that, money ran out. So, lots of docstrings are in "needs review" or even "being edited" status. But that doesn't mean money will never come again. Indeed, there are now several companies basing their services around this software. If someone does want to make the docs professional, say for numpy 2.0 or 3.0 or whatever, or as part of a larger system for sale, then they have a system in place that can do it. The purpose of the review statuses is to identify how close a docstring is to publishable. However, there is no consequence to the statuses: a docstring gets included in the release no matter its status. But, you do know which docstrings need what kind of work. So, what's the benefit of changing what the statuses mean, or eliminating them? I think it may only be that the writers feel better. The users don't even see the statuses as they're not listed in the release. Tim felt that docs should be continually edited, not "finished". I agree, especially if the underlying routine or surrounding docs get changed. But the system is designed to encourage this! Here's how: Say most/all routines get genuine "proofed" status. That's great, but it's not the end of the line by any means. If someone comes along and edits a "proofed" docstring, that docstring then automatically "needs review" once again, to ensure that a mistake was not inserted. Now you know what to look at when checking things over before a release (since there can't be unit tests for docs). From the history, you also know it was once proofed, so reviewing and proofing it is very easy just by looking at the diffs. So, the system encourages and accounts for continual edits while allowing a professional product to be produced for a particular release. The way to move forward is to declare that the goal is to get all docs to some status, say "needs review" (that was our initial goal, and the only one we achieved, more or less). Then, go after the docs that don't have that, like the new polynomial docs. If someone wants to publish a manual, the goal becomes "proofed", and there's more work to do. It DOES make sense to give the reviewer role to more people. Just make sure they take care in their reviews, so the statuses continue to have meaning. Otherwise what's the point? --jh-- On Mon, 7 May 2012 22:14:56, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: On Mon, May 7, 2012 at 7:37 PM, Tim Cera <tim(a)cerazone.net> wrote: >> I think we should change the roles established for the Numpy/Scipy >> documentation editors because they do not work as intended. >> >> For reference they are described here: >> http://docs.scipy.org/numpy/Front%20Page/ >> >> Basically there aren't that many active people to support being split into >> the roles as described which has led to a backlog of 'Needs review' >> docstrings and only one 'Proofed' docstring. I think that many of these >> docstrings are good enough, just that not enough people have put themselves >> out front as so knowledgeable about a certain topic to label docstrings as >> 'Reviewed' or 'Proofed'. >> >> You're right. I think at some point the goal shifted from getting >everything to "proofed" to getting everything to "needs review". > > >> Here are the current statistics for numpy docstrings: >> Current %Count Needs editing17 279 Being written / Changed4 62 Needs >> review76 1235 Needs review (revised)2 35 Needs work (reviewed)0 3Reviewed (needs proof) >> 0 0 Proofed0 1 Unimportant? 1793 >> >> The "needs editing" category actually contains mostly docstrings that are >quite good, but were recently created and never edited in the doc wiki. The >% keeps on growing. Bumping all polynomial docstrings up to "needs review" >would be a good start here to make the % reflect the actual status. > >> >> I have thought about some solutions in no particular order: >> >> * Get rid of the 'Reviewer' and 'Proofer' roles. >> * Assign all 'Editors', the 'Reviewer', and 'Proofer' privileges. >> * People start out as 'Editors', and then become 'Reviewers', and >> 'Proofers' based on some editing metric. >> >> For full disclosure, I would be generous with a 'Reviewed' label if given >> the authority because philosophically I think there should be a point where >> the docstring is 'Good enough' and it should be expected to have a life of >> continually small improvements rather that a point when it is 'Done'. >> > >This makes sense to me. > > >> Regardless of what decision is made, the single 'Proofed' docstring should >> be available for editing. I can't even find what it is. I imagine that it >> should be on the docstring page at http://docs.scipy.org/numpy/docs/ >> >> It used to be there - maybe the stats got confused.

1 0

Announce: scikit-learn v0.11
by Gael Varoquaux 08 May '12

08 May '12

On behalf of Andy Mueller, our release manager, I am happy to announce the 0.11 release of scikit-learn. This release includes some major new features such as randomized sparse models, gradient boosted regression trees, label propagation and many more. The release also has major improvements in the documentation and in stability. Details can be found on the [1]what's new page. We also have a new page with [2]video tutorials on machine learning with scikit-learn and different aspects of the package. Sources and windows binaries are available on sourceforge, through pypi (http://pypi.python.org/pypi/scikit-learn/0.11) or can be installed directly using pip: pip install -U scikit-learn Thanks again to all the contributors who made this release possible. Cheers, Gaël 1. http://scikit-learn.org/stable/whats_new.html 2. http://scikit-learn.org/stable/presentations.html

1 0

Issue Tracking
by Travis Oliphant 08 May '12

08 May '12

Hey all, We have been doing some investigation of various approaches to issue tracking. The last time the conversation left this list was with Ralf's current list of preferences as: 1) Redmine 2) Trac 3) Github Since that time, Maggie who has been doing a lot of work settting up various issue tracking tools over the past couple of months, has set up a redmine instance and played with it. This is a possibility as a future issue tracker. However, today I took a hard look at what the IPython folks are doing with their issue tracker and was very impressed by the level of community integration that having issues tracked by Github provides. Right now, we have a major community problem in that there are 3 conversations taking place (well at least 2 1/2). One on Github, one on this list, and one on the Trac and it's accompanying wiki. I would like to propose just using Github's issue tracker. This just seems like the best move overall for us at this point. I like how the Pull Request mechanism integrates with the issue tracking. We could setup a Redmine instance but this would just re-create the same separation of communities that currently exists with the pull-requests, the mailing list, and the Trac pages. Redmine is nicer than Trac, but it's still a separate space. We need to make Github the NumPy developer hub and not have it spread throughout several sites. The same is true of SciPy. I think if SciPy also migrates to use Github issues, then together with IPython we can really be a voice that helps Github. I will propose to NumFOCUS that the Foundation sponsor migration of the Trac to Github for NumPy and SciPy. If anyone would like to be involved in this migration project, please let me know. Comments, concerns? -Travis

10 37

numpy_quaternion: OK on numpy 1.5.0, fails on 1.6.1
by Tom Aldcroft 08 May '12

08 May '12

Sorry to bother again, but I am running into an issue with the numpy quaternion dtype on numpy 1.6.1 : $ python ActivePython 2.7.1.4 (ActiveState Software Inc.) based on Python 2.7.1 (r271:86832, Feb 7 2011, 11:30:54) [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import quaternion >>> q = numpy.quaternion(1,0,0,0) >>> q quaternion(1, 0, 0, 0) >>> q2 = numpy.quaternion(1,0,0,0) >>> q * q2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: ufunc 'multiply' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe' >>> numpy.__version__ '1.6.1' Using numpy 1.5.0 on the same platform the multiplication works and I get the expected result. For my near-term goal of getting tests in place I can just use 1.5.0, but if anyone has an idea of the problem I would appreciate help. Thanks, Tom

3 4

Documentation roles in the numpy/scipy documentation editor
by Tim Cera 08 May '12

08 May '12

I think we should change the roles established for the Numpy/Scipy documentation editors because they do not work as intended. For reference they are described here: http://docs.scipy.org/numpy/Front%20Page/ Basically there aren't that many active people to support being split into the roles as described which has led to a backlog of 'Needs review' docstrings and only one 'Proofed' docstring. I think that many of these docstrings are good enough, just that not enough people have put themselves out front as so knowledgeable about a certain topic to label docstrings as 'Reviewed' or 'Proofed'. Here are the current statistics for numpy docstrings: Current%CountNeeds editing17279Being written / Changed462Needs review761235Needs review (revised)235Needs work (reviewed)03Reviewed (needs proof)00Proofed01 Unimportant–1793 I have thought about some solutions in no particular order: * Get rid of the 'Reviewer' and 'Proofer' roles. * Assign all 'Editors', the 'Reviewer', and 'Proofer' privileges. * People start out as 'Editors', and then become 'Reviewers', and 'Proofers' based on some editing metric. For full disclosure, I would be generous with a 'Reviewed' label if given the authority because philosophically I think there should be a point where the docstring is 'Good enough' and it should be expected to have a life of continually small improvements rather that a point when it is 'Done'. Regardless of what decision is made, the single 'Proofed' docstring should be available for editing. I can't even find what it is. I imagine that it should be on the docstring page at http://docs.scipy.org/numpy/docs/ Kindest regards, Tim

2 1

How to run NumPy's tests with coverage?
by Chris Ball 07 May '12

07 May '12

Hi, I'm trying to figure out how to run NumPy's tests with coverage enabled (i.e. numpy.test(coverage=True) ). I can run the tests successfully like this: $ git clone git://github.com/numpy/numpy.git [...] $ cd numpy/ $ python setup.py build_ext -i [...] $ cd .. # (avoid running from source directory) $ export PYTHONPATH=numpy/ $ python Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. python> import numpy python> numpy.test() Running unit tests for numpy NumPy version 1.7.0.dev-259fff8 NumPy is installed in [...]/numpy Python version 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] nose version 0.11.1 [...] Ran 3710 tests in 27.654s OK (KNOWNFAIL=3, SKIP=6) <nose.result.TextTestResult run=3710 errors=0 failures=0> However, if I try to run the tests with coverage, I get lots of errors (and seven more tests are run than without coverage): python> numpy.test(coverage=True) Running unit tests for numpy NumPy version 1.7.0.dev-259fff8 NumPy is installed in [...]/numpy Python version 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] nose version 0.11.1 Could not locate executable icc Could not locate executable ecc [...]/numpy/numarray/alter_code2.py:12: UserWarning: numpy.numarray.alter_code2 is not working yet. warnings.warn("numpy.numarray.alter_code2 is not working yet.") [...]/numpy/oldnumeric/alter_code2.py:26: UserWarning: numpy.oldnumeric.alter_code2 is not working yet. warnings.warn("numpy.oldnumeric.alter_code2 is not working yet.") [...] ====================================================================== ERROR: Failure: ImportError (No module named waflib.Configure) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/build_utils/waf.py", line 4, in <module> import waflib.Configure ImportError: No module named waflib.Configure ====================================================================== ERROR: Failure: ImportError (No module named numscons.numdist) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/core/scons_support.py", line 21, in <module> from numscons.numdist import process_c_str as process_str ImportError: No module named numscons.numdist ====================================================================== ERROR: Failure: ImportError (No module named numscons) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/core/setupscons.py", line 8, in <module> from numscons import get_scons_build_dir ImportError: No module named numscons ====================================================================== ERROR: test_multiarray.TestNewBufferProtocol.test_roundtrip ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest self.test(*self.arg) File "[...]/numpy/core/tests/test_multiarray.py", line 2233, in test_roundtrip assert_raises(ValueError, self._check_roundtrip, x) File "[...]/numpy/testing/utils.py", line 1053, in assert_raises return nose.tools.assert_raises(*args,**kwargs) File "/usr/lib/python2.6/unittest.py", line 336, in failUnlessRaises callableObj(*args, **kwargs) File "[...]/numpy/core/tests/test_multiarray.py", line 2167, in _check_roundtrip y = np.asarray(x) File "[...]/numpy/core/tests/test_multiarray.py", line 2167, in _check_roundtrip y = np.asarray(x) File "/usr/lib/python2.6/dist-packages/coverage.py", line 322, in t self.c[(f.f_code.co_filename, f.f_lineno)] = 1 RuntimeWarning: tp_compare didn't return -1 or -2 for exception ====================================================================== ERROR: Failure: ImportError (No module named np.core.fromnumeric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/ma/timer_comparison.py", line 6, in <module> import np.core.fromnumeric as fromnumeric ImportError: No module named np.core.fromnumeric ====================================================================== ERROR: Failure: AttributeError ('module' object has no attribute '__revision__') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/ma/version.py", line 9, in <module> revision = [core.__revision__.split(':')[-1][:-1].strip(), AttributeError: 'module' object has no attribute '__revision__' ====================================================================== ERROR: Failure: ImportError (The convolve package is not installed. It can be downloaded by checking out the latest source from http://svn.scipy.org/svn/scipy/trunk/Lib/stsci or by downloading and installing all of SciPy from http://www.scipy.org. ) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/numarray/convolve.py", line 14, in <module> raise ImportError(msg) ImportError: The convolve package is not installed. It can be downloaded by checking out the latest source from http://svn.scipy.org/svn/scipy/trunk/Lib/stsci or by downloading and installing all of SciPy from http://www.scipy.org. ====================================================================== ERROR: Failure: ImportError (The image package is not installed It can be downloaded by checking out the latest source from http://svn.scipy.org/svn/scipy/trunk/Lib/stsci or by downloading and installing all of SciPy from http://www.scipy.org. ) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "[...]/numpy/numarray/image.py", line 14, in <module> raise ImportError(msg) ImportError: The image package is not installed It can be downloaded by checking out the latest source from http://svn.scipy.org/svn/scipy/trunk/Lib/stsci or by downloading and installing all of SciPy from http://www.scipy.org. ====================================================================== FAIL: test_blasdot.test_dot_3args ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest self.test(*self.arg) File "[...]/numpy/core/tests/test_blasdot.py", line 52, in test_dot_3args assert_equal(sys.getrefcount(r), 2) File "[...]/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 3 DESIRED: 2 ====================================================================== FAIL: test_dot_3args (test_multiarray.TestDot) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/numpy/core/tests/test_multiarray.py", line 1698, in test_dot_3args assert_equal(sys.getrefcount(r), 2) File "[...]/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 3 DESIRED: 2 [...] Ran 3717 tests in 31.447s FAILED (KNOWNFAIL=3, SKIP=6, errors=8, failures=2) <nose.result.TextTestResult run=3717 errors=8 failures=2> Anyone know what I'm doing wrong? I'm using Ubuntu 10.04 LTS in case that matters. Thanks, Chris

3 3