distributions.ncf.fit -- never converges?
Since 0.9.0 (not present in 0.7.2), distributions.ncf.fit seems to never converge. could anyone check in development tree if the below call ever returns (I was waiting for minutes... with 0.7.2 it was taking just a moment): python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));' Thanks in advance! -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
On Wed, Apr 13, 2011 at 10:34 AM, Yaroslav Halchenko <lists@onerussian.com> wrote:
Since 0.9.0 (not present in 0.7.2), distributions.ncf.fit seems to never converge. could anyone check in development tree if the below call ever returns (I was waiting for minutes... with 0.7.2 it was taking just a moment):
python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));'
I only have 0.9.0, I forgot to kill the process and 1 hour 22 minutes of cpu time later it still hadn't returned. At least a maximum iteration or maximum function evaluation should have stopped it. Josef
Thanks in advance!
-- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Thanks Josef for the confirmation and making your computer sweat ;) I have built current master *In [3]: scipy.__version__ Out[3]: '0.10.0.dev' the same story. apparently it is known to some degree -- ncf is skipped the unittesting: $> git grep -A 10 'TestFitMethod' scipy/stats/tests/test_distributions.py:class TestFitMethod(TestCase): scipy/stats/tests/test_distributions.py- skip = ['ncf'] scipy/stats/tests/test_distributions.py- scipy/stats/tests/test_distributions.py- @dec.slow scipy/stats/tests/test_distributions.py- def test_fit(self): scipy/stats/tests/test_distributions.py- for func, dist, args, alpha in test_all_distributions(): scipy/stats/tests/test_distributions.py- if dist in self.skip: scipy/stats/tests/test_distributions.py- continue $> git show ba349771 commit ba349771afebb1473a961139570da770ada59032 Author: Travis Oliphant <oliphant@enthought.com> Date: Tue Jun 1 07:59:20 2010 +0000 Add tests for fit method including fixing input parameters. ;-) :-/ On Wed, 13 Apr 2011, josef.pktd@gmail.com wrote:
python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));'
I only have 0.9.0, I forgot to kill the process and 1 hour 22 minutes of cpu time later it still hadn't returned. At least a maximum iteration or maximum function evaluation should have stopped it.
Josef -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
just in case it would come useful in troubleshooting -- gets stuck in cumfnc_ (not modified since 2007) and blunt guess could be that reason is could be related to cdffnc1_wrap (dfn=-nan(0x8000000000000), dfd=-nan(0x8000000000000), ... ? *(gdb) bt full 10 #0 0x00007f69103a44eb in cumfnc_ () from /home/yoh/proj/scipy/install/lib/python2.6/site-packages/scipy/special/_cephes.so No symbol table info available. #1 0x00007f691039fab0 in cdffnc_ () from /home/yoh/proj/scipy/install/lib/python2.6/site-packages/scipy/special/_cephes.so No symbol table info available. #2 0x00007f6910363b84 in cdffnc1_wrap (dfn=-nan(0x8000000000000), dfd=-nan(0x8000000000000), nc=1, f=0.5) at scipy/special/cdf_wrappers.c:254 which = 1 q = 8.4886169591766247e-317 p = <value optimized out> bound = <value optimized out> status = 6 #3 0x00007f69103668fe in PyUFunc_dddd_d (args=0x7f691067a750, dimensions=<value optimized out>, steps=0x7ff80000, func=<value optimized out>) at scipy/special/ufunc_extras.c:510 i = 1 ip1 = 0xf6e2c0 "" ip2 = 0x19df1e0 "" ip3 = 0x19ee0c0 "" ip4 = 0x190c7b0 "" op = 0x17dedf0 "\b\357\374\023i\177" n = 10 #4 0x00007f6913257209 in ?? () from /usr/lib/pymodules/python2.6/numpy/core/umath.so No symbol table info available. #5 0x00007f6913257d13 in ?? () from /usr/lib/pymodules/python2.6/numpy/core/umath.so No symbol table info available. #6 0x000000000041ef47 in PyObject_Call (func=<numpy.ufunc at remote 0x14dba50>, arg=<unknown at remote 0x43680000>, kw=<unknown at remote 0x7ff80000>) at ../Objects/abstract.c:2492 result = < at remote 0xedd180> call = 0x7f6913257cb0 #7 0x00000000004a72b8 in do_call (f= Frame 0x19e4b60, for file /home/yoh/proj/scipy/install/lib/python2.6/site-packages/scipy/stats/distributions.py, line 4241, in _cdf (self=<ncf_gen(a=<float at remote 0x15f1a88>, moment_type=1, b=<float at remote 0xefa5d0>, vecfunc=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890640>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891f50>, name='ncf', xb=<float at remote 0x1589d58>, xa=<float at remote 0x1589d70>, m=<float at remote 0x1589d28>, vecentropy=<vectorize(nin=4, nin_wo_defaults=None, thefunc=<instancemethod at remote 0x18906e0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891fd0>, numargs=3, generic_moment=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890fa0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1893090>, veccdf=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890e10>, otypes='d', lastcallargs=0, ufunc=None, __doc_...(truncated), throwflag=<value optimized out>) at ../Python/ceval.c:3968 callargs = (<numpy.float64 at remote 0x1643710>, <numpy.float64 at remote 0x1553990>, <numpy.float64 at remote 0x18d1b80>, <numpy.ndarray at remote 0x19f4590>) kwdict = 0x0 #8 call_function (f= Frame 0x19e4b60, for file /home/yoh/proj/scipy/install/lib/python2.6/site-packages/scipy/stats/distributions.py, line 4241, in _cdf (self=<ncf_gen(a=<float at remote 0x15f1a88>, moment_type=1, b=<float at remote 0xefa5d0>, vecfunc=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890640>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891f50>, name='ncf', xb=<float at remote 0x1589d58>, xa=<float at remote 0x1589d70>, m=<float at remote 0x1589d28>, vecentropy=<vectorize(nin=4, nin_wo_defaults=None, thefunc=<instancemethod at remote 0x18906e0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891fd0>, numargs=3, generic_moment=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890fa0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1893090>, veccdf=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890e10>, otypes='d', lastcallargs=0, ufunc=None, __doc_...(truncated), throwflag=<value optimized out>) at ../Python/ceval.c:3773 func = <numpy.ufunc at remote 0x14dba50> w = <value optimized out> nk = <value optimized out> n = <value optimized out> pfunc = 0x19e4d00 #9 PyEval_EvalFrameEx (f= Frame 0x19e4b60, for file /home/yoh/proj/scipy/install/lib/python2.6/site-packages/scipy/stats/distributions.py, line 4241, in _cdf (self=<ncf_gen(a=<float at remote 0x15f1a88>, moment_type=1, b=<float at remote 0xefa5d0>, vecfunc=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890640>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891f50>, name='ncf', xb=<float at remote 0x1589d58>, xa=<float at remote 0x1589d70>, m=<float at remote 0x1589d28>, vecentropy=<vectorize(nin=4, nin_wo_defaults=None, thefunc=<instancemethod at remote 0x18906e0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1891fd0>, numargs=3, generic_moment=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890fa0>, otypes='d', lastcallargs=0, ufunc=None, __doc__=None, nout=None) at remote 0x1893090>, veccdf=<vectorize(nin=4, nin_wo_defaults=1, thefunc=<instancemethod at remote 0x1890e10>, otypes='d', lastcallargs=0, ufunc=None, __doc_...(truncated), throwflag=<value optimized out>) at ../Python/ceval.c:2412 sp = 0x19e4d08 stack_pointer = <value optimized out> next_instr = 0x1615229 "S" opcode = <value optimized out> oparg = <value optimized out> why = <value optimized out> err = <value optimized out> x = <value optimized out> v = <value optimized out> w = <value optimized out> u = <value optimized out> t = <value optimized out> stream = <value optimized out> freevars = 0x19e4d00 retval = 0x0 tstate = 0xedd180 co = 0x1614828 instr_ub = -1 instr_lb = 0 instr_prev = -1 first_instr = 0x1615214 "t" names = ('special', 'ncfdtr') consts = (None,) (More stack frames follow...) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
2011/4/13 Yaroslav Halchenko <lists@onerussian.com>:
Thanks Josef for the confirmation and making your computer sweat ;)
I have built current master
*In [3]: scipy.__version__ Out[3]: '0.10.0.dev'
the same story.
The default starting parameters have changed. If your example worked before, then it's possible that it works with explicit starting parameters. (I haven't figured out yet how to make my new computer sweat.) Josef
apparently it is known to some degree -- ncf is skipped the unittesting:
$> git grep -A 10 'TestFitMethod' scipy/stats/tests/test_distributions.py:class TestFitMethod(TestCase): scipy/stats/tests/test_distributions.py- skip = ['ncf'] scipy/stats/tests/test_distributions.py- scipy/stats/tests/test_distributions.py- @dec.slow scipy/stats/tests/test_distributions.py- def test_fit(self): scipy/stats/tests/test_distributions.py- for func, dist, args, alpha in test_all_distributions(): scipy/stats/tests/test_distributions.py- if dist in self.skip: scipy/stats/tests/test_distributions.py- continue
$> git show ba349771 commit ba349771afebb1473a961139570da770ada59032 Author: Travis Oliphant <oliphant@enthought.com> Date: Tue Jun 1 07:59:20 2010 +0000
Add tests for fit method including fixing input parameters.
;-) :-/
On Wed, 13 Apr 2011, josef.pktd@gmail.com wrote:
python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));'
I only have 0.9.0, I forgot to kill the process and 1 hour 22 minutes of cpu time later it still hadn't returned. At least a maximum iteration or maximum function evaluation should have stopped it.
Josef -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On 04/13/2011 12:33 PM, josef.pktd@gmail.com wrote:
Thanks Josef for the confirmation and making your computer sweat ;)
I have built current master
*In [3]: scipy.__version__ Out[3]: '0.10.0.dev'
the same story. The default starting parameters have changed. If your example worked before, then it's possible that it works with explicit starting
2011/4/13 Yaroslav Halchenko<lists@onerussian.com>: parameters.
(I haven't figured out yet how to make my new computer sweat.)
Josef
apparently it is known to some degree -- ncf is skipped the unittesting:
$> git grep -A 10 'TestFitMethod' scipy/stats/tests/test_distributions.py:class TestFitMethod(TestCase): scipy/stats/tests/test_distributions.py- skip = ['ncf'] scipy/stats/tests/test_distributions.py- scipy/stats/tests/test_distributions.py- @dec.slow scipy/stats/tests/test_distributions.py- def test_fit(self): scipy/stats/tests/test_distributions.py- for func, dist, args, alpha in test_all_distributions(): scipy/stats/tests/test_distributions.py- if dist in self.skip: scipy/stats/tests/test_distributions.py- continue
$> git show ba349771 commit ba349771afebb1473a961139570da770ada59032 Author: Travis Oliphant<oliphant@enthought.com> Date: Tue Jun 1 07:59:20 2010 +0000
Add tests for fit method including fixing input parameters.
;-) :-/
On Wed, 13 Apr 2011, josef.pktd@gmail.com wrote:
python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));' I only have 0.9.0, I forgot to kill the process and 1 hour 22 minutes of cpu time later it still hadn't returned. At least a maximum iteration or maximum function evaluation should have stopped it. Josef -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev I did not run it very long because I saw this message: /usr/lib64/python2.7/site-packages/scipy/stats/distributions.py:1882: RuntimeWarning: invalid value encountered in double_scalars Lhat = muhat - Shat*mu
It seems that line 1874 (mu, mu2 = self.stats(*args,**{'moments':'mv'})) give mu and mu2 as inf's. So Lhat is nan and there are better uses for that cpu power than spinning around infinities ... Bruce
On Wed, 13 Apr 2011, Bruce Southey wrote:
I did not run it very long because I saw this message: /usr/lib64/python2.7/site-packages/scipy/stats/distributions.py:1882: RuntimeWarning: invalid value encountered in double_scalars Lhat = muhat - Shat*mu
It seems that line 1874 (mu, mu2 = self.stats(*args,**{'moments':'mv'})) give mu and mu2 as inf's. So Lhat is nan and there are better uses for that cpu power than spinning around infinities ...
lucky you -- I got none of that warning. Although I must confess, scipy 0.9.0 is really "talkative" without any way to shut it really up: $> dpkg -L python-scipy | grep '\.py' | xargs grep -i "print.*warn" | head /usr/share/pyshared/scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." /usr/share/pyshared/scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." ... $> dpkg -L python-scipy | grep '\.py' | xargs grep -i "print.*warn" | wc -l 132 great to see that future 0.9.1 will be saner: $> git describe --tags v0.4.3-4868-g9b5db2c $> git grep -i 'print[^f].*war' | wc -l 47 -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
On Wed, Apr 13, 2011 at 1:58 PM, Bruce Southey <bsouthey@gmail.com> wrote:
On 04/13/2011 12:33 PM, josef.pktd@gmail.com wrote:
Thanks Josef for the confirmation and making your computer sweat ;)
I have built current master
*In [3]: scipy.__version__ Out[3]: '0.10.0.dev'
the same story. The default starting parameters have changed. If your example worked before, then it's possible that it works with explicit starting
2011/4/13 Yaroslav Halchenko<lists@onerussian.com>: parameters.
(I haven't figured out yet how to make my new computer sweat.)
Josef
apparently it is known to some degree -- ncf is skipped the unittesting:
$> git grep -A 10 'TestFitMethod' scipy/stats/tests/test_distributions.py:class TestFitMethod(TestCase): scipy/stats/tests/test_distributions.py- skip = ['ncf'] scipy/stats/tests/test_distributions.py- scipy/stats/tests/test_distributions.py- @dec.slow scipy/stats/tests/test_distributions.py- def test_fit(self): scipy/stats/tests/test_distributions.py- for func, dist, args, alpha in test_all_distributions(): scipy/stats/tests/test_distributions.py- if dist in self.skip: scipy/stats/tests/test_distributions.py- continue
$> git show ba349771 commit ba349771afebb1473a961139570da770ada59032 Author: Travis Oliphant<oliphant@enthought.com> Date: Tue Jun 1 07:59:20 2010 +0000
Add tests for fit method including fixing input parameters.
;-) :-/
On Wed, 13 Apr 2011, josef.pktd@gmail.com wrote:
python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));' I only have 0.9.0, I forgot to kill the process and 1 hour 22 minutes of cpu time later it still hadn't returned. At least a maximum iteration or maximum function evaluation should have stopped it. Josef -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev I did not run it very long because I saw this message: /usr/lib64/python2.7/site-packages/scipy/stats/distributions.py:1882: RuntimeWarning: invalid value encountered in double_scalars Lhat = muhat - Shat*mu
It seems that line 1874 (mu, mu2 = self.stats(*args,**{'moments':'mv'})) give mu and mu2 as inf's. So Lhat is nan and there are better uses for that cpu power than spinning around infinities ...
"To infinity and the nans beyond." It looks like mean and variance are only finite if the degrees of freedom for the ncf, dfn, dfd, are large enough, so I guess the starting values should be large. I think _fitstart could default back to loc=0, scale=1 if np.any(~np.isfinite(self.fit_loc_scale(data, *args))) This would remove at least one source of infs and nans. Yaroslav, Did you ever get a good estimate in your example with older versions? With numpy 1.3, scipy 0.72 and with fully specified starting values in scipy 0.9, the estimation results are the same as the starting values. So the estimation doesn't move and does not produce any useful results. (I don't have scipy 0.8 right now.) Also as a note: ncf doesn't have the pdf defined. I removed it a long time ago because it was buggy, and didn't figure out how to fix it. So any maximum likelihood estimation has to work with the generic pdf. Josef
Bruce
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Hi Josef, Sorry -- I have missed the reply/question On Fri, 15 Apr 2011, josef.pktd@gmail.com wrote:
Yaroslav, Did you ever get a good estimate in your example with older versions? With numpy 1.3, scipy 0.72 and with fully specified starting values in scipy 0.9, the estimation results are the same as the starting values. So the estimation doesn't move and does not produce any useful results. (I don't have scipy 0.8 right now.)
not sure -- those distributions were not of particular interest for me, so in my evil match_distribution script, their absence among the winners didn't trigger my interest ;) But before it wasn't getting stuck for sure -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
participants (3)
-
Bruce Southey -
josef.pktd＠gmail.com -
Yaroslav Halchenko