Mailman 3 November 2008 - SciPy-Dev

scikit for dae
by Benny Malengier Nov. 6, 2008

Nov. 6, 2008

With all the talk of integration of new code and the difficulties involved, I decided to rewrite my patch to scipy.integrate to add a dae solver, as a scikit instead: scikits.odes Is this acceptable? If allowed as a scikit I could add some further backends (I'm thinking of 2). It would ease my hacking on it, allow collaboration, and enable more extensive testing (only linux 64bit here). The code would be more accessible too being available with easy_install, instead of a patch set. The patch is here: http://cage.ugent.be/~bm/progs.html Benny PS: If somebody needs Krylov preconditioners in dae, I could interface that in the ddaspk backend, but as I have no experience with this, somebody interested should construct a test case, and collaborate with me. ddaspk is in essense two programs in one, one with Krylov, one without.

3 5

a modest proposal for technology previews
by Jarrod Millman Nov. 6, 2008

Nov. 6, 2008

Hey, I have been thinking about how to best get useful, widely-needed, high-quality code with a good, stable API into scipy without creating an unnecessary burden on developers or early adopters. Unfortunately, I don't have time to fully flesh out what I have been thinking; but I went ahead and started writing a SEP: http://projects.scipy.org/scipy/scipy/browser/trunk/doc/seps/technology-pre… Please note that I don't intend this to replace scikits or other staging grounds. I imagine that a project could easily start as a scikit and mature there. Then a number of developer decide that it belongs in scipy proper. Rather than just working in a branch until the code is ready for release to the world. This mechanism would allow any additional incubator for code maturation and development. I would love to hear everyone's initial thoughts, suggestions, and ideas. I will try and incorporate these comments into the SEP and then we can discuss whether we should accept, reject, or defer the SEP. I have been kicking the idea around a bit with Fernando, Chris, Stefan, and others; so I can't claim the idea is mine. I am just trying to write it up. If anyone once to help out, the SEP is checked into the scipy trunk. I plan to work on it more later tonight to flesh out some of the ideas better. But in the spirit of "release early, release often" ... have at it. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

10 20

Dealing with precision problem: warning vs raise
by David Cournapeau Nov. 6, 2008

Nov. 6, 2008

Hi, I was looking at bug #651, related to finding the roots of a polynomial with too small coefficients. Currently, the small values are discarded by the function in #651, which I find a bid surprising (it may not be obvious to the user why it gets no zero for a polynomial of order N). Wouldn't it be better to raise an exception or a at least a warning instead (I was thinking about doing that in numpy.roots instead, but that may be too drastic, as it would break some code - maybe it is desirable, though, as the values in that case are more or less meaningless ) cheers, David

2 2

Old SciPy wiki content?
by Jonathan Guyer Nov. 5, 2008

Nov. 5, 2008

Many ages ago, I posted some benchmarking results on sparse solvers to the old SciPy wiki. http://thread.gmane.org/gmane.comp.python.scientific.devel/2243/focus=3192 When the site got moved, I never did figure out where it had gone to and never made a big enough fuss about it at the time. There's an item on http://www.scipy.org/MigratingFromPlone about moving it, but it's given low priority and the link to http://old.scipy.org/wikis/featurerequests/SparseSolvers gives a 404. Is this content anywhere anymore?

1 0

toward scipy 1.0
by jh＠physics.ucf.edu Nov. 5, 2008

Nov. 5, 2008

This message is about cleaning up scipy and releasing 1.0. Tied up in this topic is the issue of how code is brought into scipy. We recognize that there is work to do before scipy is 1.0-ready, and much of that work is not merely adding to what is there. So, I also think that we need to solve the problems that got us here before 1.0. Since this is a message about solving a problem in our process, some may think I am placing blame or don't support our developers. So, let me state at the outset that we could not have come as far as we have without a lot of dedicated work on many parts, particularly the current set of developers. Further, they have all recognized and spoken about the organizational issues we have had, and about greater community input as a route out. All I'm intending here is to move us in that direction. "Jarrod Millman" <millman(a)berkeley.edu> wrote in another thread: > I imagine that a project could easily start as a > scikit and mature there. Then a number of developer decide that it > belongs in scipy proper. I think this is part of the problem. We need to define a clear route to inclusion for new packages. That route should include a period of community review and then a vote, and not merely be the decision of a few developers in private discussion. This should be a formalized process, so that if we have a disagreement, the result is clear and we all agree to live with it. I feel tht the small number of people involved in decisions, and the lack of community review, has brought scipy's organization to where it is today. We are all human. As recently as the 1.2.0 numpy release we had significant API changes angling for inclusion at the last minute, and only last-minute community outcry let cooler heads prevail. To solve this problem, we need to open the process and make it more deliberate and deliberative. We don't need scipy to be nimble, we need it to be stable and well thought-out. >From discussions with Stefan, I understand that some parts of scipy are not maintained and others don't hang together or mesh well with the rest of the package. Some module names don't make sense (scipy.stsci was recently pointed out). Docs are arcane (stats) or lacking. We now have a decade or more of use experience with scipy and its predecessors, enough to make something coherent and long-term stable out of it, but this refactoring has not yet happened. So, I propose a community reassessment of scipy and a refactoring and doc effort before 1.0. The goal is that in 6.0 your 1.0-based code still works well, and we look at it and say, "That structure really stood the test of time. It still makes sense today." Here's what I propose: 1. reassess what's there: - break it into components for discussion purposes - decide for each whether it's: - used - self-consistent - complete at some level - maintained - documented for normal users - a build problem - well integrated into the rest of scipy This part of the process would be done by small teams and would result in a short report for each package recommending whether it is: - worth keeping as is - needs specific work (docs, build stuff, tests, etc.) - needs a maintainer - needs to be refactored, removed, or merged into something else Any component for which a team does not volunteer to do the review would be a good candidate for removal based on lack of use. 2. hunt for looming incompatible API changes: - in the code as it exists - that result from the reassessment above These would be collected on a page(s) for specific community review and comment. 3. community comments on the collection of reports online and looks for consensus on any overall restructuring 4. do the work resulting from the reassessment Yes, this is the hard part. Likely we will pick up some new developers/maintainers from step 1 and they pick up much of the work. 5. present the refactored package in a declared-unstable release (0.7.99?) 6. stabilize it and release it as 0.8 7. run a doc marathon based on that release (I can pay one or more writers) 8. allow significant additions, if any, until 0.9, but only after they have been scikits for a year *and* pass a community-based acceptance process 9. release scipy 1.0 as a cleaned-up package with community testing and full docs, but no significant new code after 0.9 10. include new code in future releases after they have been scikits for a year *and* pass a community-based acceptance process I don't expect most scikits to be accepted after 1 year, and expect that some never will be. I think this is a 2-3-year process, of which steps 1-4 could happen in 6-8 months, say by next June. My guess is that, except for docs, most of the components would come through pretty much unchanged, but a few would get a real working over or be removed. We'd need a web tool similar to the doc wiki for this. Or, we could use a wiki and a lot of discipline. We would also need to define some sort of voting process after the assessment period. Whoever gets to vote, I propose a 2/3 majority required for inclusion of a new package, and an 80% majority for a major API change (which waits for the next major release). There is much detail to fill in, such as what this community review process really looks like, not to mention whether any of this is a good idea at all. Let me know your thoughts. --jh--

4 4

ValueError: Buffer datatype mismatch (rejecting on 'l')
by Nils Wagner Nov. 4, 2008

Nov. 4, 2008

Hi all, scipy.test() results in Ran 2595 tests in 47.866s FAILED (KNOWNFAIL=2, errors=26, failures=2) <nose.result.TextTestResult run=2595 errors=26 failures=2> ====================================================================== ERROR: test_kdtree.test_random_compiled.test_approx ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 363, in setUp try_run(self.inst, ('setup', 'setUp')) File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/util.py", line 453, in try_run return func() File "/usr/local/lib64/python2.5/site-packages/scipy/spatial/tests/test_kdtree.py", line 133, in setUp self.kdtree = cKDTree(self.data) File "ckdtree.pyx", line 229, in scipy.spatial.ckdtree.cKDTree.__init__ (scipy/spatial/ckdtree.c:1039) ValueError: Buffer datatype mismatch (rejecting on 'l') Nils

1 0

Generating SciPy Sphinx HTML
by Damian Eads Nov. 4, 2008

Nov. 4, 2008

Hi, I'm at the Berkeley sprint now trying to fix a few doc bugs. Can anyone point me to instructions or a script for generating Sphinx HTML documentation from the RST docstrings? Please advise. Thank you! Damian ----------------------------------------------------- Damian Eads Ph.D. Student Jack Baskin School of Engineering, UCSC E2-489 1156 High Street Machine Learning Lab Santa Cruz, CA 95064 http://www.soe.ucsc.edu/~eads

4 6

scipy.stsci to scikits?
by Christopher Hanley Nov. 3, 2008

Nov. 3, 2008

Would there be any opposition is moving the scipy.stsci package to scikits? As a reminder, scipy.stsci contains the convolve and image modules that were once distributed with numarray. Thoughts? Opinions? A secondary question. If I have commit privileges for scipy do I need a separate account for scikit access? Thanks, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

6 7

Definition of gammaln(x) for negative x
by G-J van Rooyen Nov. 3, 2008

Nov. 3, 2008

Hey everyone Ticket #737 refers: ----- Gamma in negative for negative x whose floor value is odd. As such, gammaln does not make sense for those values (while staying in the real domain, at least). scipy.special.gammaln returns bogus values: import numpy as np from scipy.special import gamma, gammaln print np.log(gamma(-0.5)) print gammaln(-0.5) Returns nan in the first case (expected) and 1.26551212348 in the second (totally meaningless value). ----- The info line for gammaln reads: * gammaln -- Log of the absolute value of the gamma function. With this definition of gammaln, the function actually works fine, since np.log(abs(gamma(-0.5))) is in fact 1.2655. However, this seems to be an unusual definition for gammaln. What is the best way to fix it? Options: 1) Keep it as it is, with gammaln(x) = ln|gamma(x)| 2) Change it so that it returns NaN for negative values of gamma(x) (i.e. negative x whose floor value is odd) 3) Change it to always give NaN for negative values of x (Matlab's approach) 4) Have it return complex values for negative logarithms Which is best? G-J

5 10

scipy.distance
by Jarrod Millman Nov. 3, 2008

Nov. 3, 2008

Hey, Damian Eads is joining the Berkeley sprint today and we were hoping to make a change that was discussed at the SciPy conference and on the list a few times. As part of his hierarchical clustering work, Damian wrote a module for a bunch of different distance functions: http://scipy.org/scipy/scipy/browser/trunk/scipy/cluster/distance.py This is obviously much more generic and useful than the domain of clustering. So we were proposing to make distance a top-level subpackage called: scipy.distance Before he made the change, I wanted to run it by the list one last time and see if anyone had any thoughts, comments, or suggestions. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

9 15