<p dir="ltr">On 1 Oct 2013 17:34, "Charles R Harris" <<a href="mailto:charlesr.harris@gmail.com">charlesr.harris@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
><br>
> On Tue, Oct 1, 2013 at 10:19 AM, <<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>> wrote:<br>
>><br>
>> On Tue, Oct 1, 2013 at 10:47 AM, Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>> wrote:<br>
>> > On Tue, Oct 1, 2013 at 3:20 PM, Charles R Harris<br>
>> > <<a href="mailto:charlesr.harris@gmail.com">charlesr.harris@gmail.com</a>> wrote:<br>
>> >><br>
>> >><br>
>> >><br>
>> >> On Tue, Oct 1, 2013 at 8:12 AM, Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>> wrote:<br>
>> >>><br>
>> >>> [switching subject to break out from the giant 1.8.0rc1 thread]<br>
>> >>><br>
>> >>> On Tue, Oct 1, 2013 at 2:52 PM, Charles R Harris<br>
>> >>> <<a href="mailto:charlesr.harris@gmail.com">charlesr.harris@gmail.com</a>> wrote:<br>
>> >>> ><br>
>> >>> ><br>
>> >>> ><br>
>> >>> > On Tue, Oct 1, 2013 at 7:25 AM, Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>> wrote:<br>
>> >>> >><br>
>> >>> >> On Tue, Oct 1, 2013 at 1:56 PM, Charles R Harris<br>
>> >>> >> <<a href="mailto:charlesr.harris@gmail.com">charlesr.harris@gmail.com</a>> wrote:<br>
>> >>> >> > On Tue, Oct 1, 2013 at 4:43 AM, Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>><br>
>> >>> >> > wrote:<br>
>> >>> >> >><br>
>> >>> >> >> On Mon, Sep 30, 2013 at 10:51 PM, Christoph Gohlke <<a href="mailto:cgohlke@uci.edu">cgohlke@uci.edu</a>><br>
>> >>> >> >> wrote:<br>
>> >>> >> >> > 2) Bottleneck 0.7.0<br>
>> >>> >> >> ><br>
>> >>> >> >> ><br>
>> >>> >> >> ><br>
>> >>> >> >> > <a href="https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701">https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701</a><br>
>> >>> >> >><br>
>> >>> >> >> I can't tell if these are real bugs in numpy, or tests checking that<br>
>> >>> >> >> bottleneck is bug-for-bug compatible with old numpy and we just<br>
>> >>> >> >> fixed<br>
>> >>> >> >> some bugs, or what. It's clearly something to do with the<br>
>> >>> >> >> nanarg{max,min} rewrite -- @charris, do you know what's going on<br>
>> >>> >> >> here?<br>
>> >>> >> >><br>
>> >>> >> ><br>
>> >>> >> > Yes ;) The previous behaviour of nanarg for all-nan axis was to cast<br>
>> >>> >> > nan<br>
>> >>> >> > to<br>
>> >>> >> > intp when the result was an array, and return nan when a scalar. The<br>
>> >>> >> > current<br>
>> >>> >> > behaviour is to return the most negative value of intp as an error<br>
>> >>> >> > marker in<br>
>> >>> >> > both cases and raise a warning. It is a change in behavior, but I<br>
>> >>> >> > think<br>
>> >>> >> > one<br>
>> >>> >> > that needs to be made.<br>
>> >>> >><br>
>> >>> >> Ah, okay! I kind of lost track of the nanfunc changes by the end there.<br>
>> >>> >><br>
>> >>> >> So for the bottleneck issue, it sounds like the problem is just that<br>
>> >>> >> bottleneck is still emulating the old numpy behaviour in this corner<br>
>> >>> >> case, which isn't really a problem. So we don't really need to worry<br>
>> >>> >> about that, both behaviours are correct, just maybe out of sync.<br>
>> >>> >><br>
>> >>> >> I'm a little dubious about this "make up some weird value that will<br>
>> >>> >> *probably* blow up if people try to use it without checking, and also<br>
>> >>> >> raise a warning" thing, wouldn't it make more sense to just raise an<br>
>> >>> >> error? That's what exceptions are for? I guess I should have said<br>
>> >>> >> something earlier though...<br>
>> >>> >><br>
>> >>> ><br>
>> >>> > I figure the blowup is safe, as we can't allocate arrays big enough that<br>
>> >>> > the<br>
>> >>> > minimum intp value would be a valid index. I considered raising an<br>
>> >>> > error,<br>
>> >>> > and if there is a consensus the behavior could be changed. Or we could<br>
>> >>> > add a<br>
>> >>> > keyword to determine the behavior.<br>
>> >>><br>
>> >>> Yeah, the intp value can't be a valid index, so that covers 95% of<br>
>> >>> cases, but I'm worried about that other 5%. It could still pass<br>
>> >>> silently as the endpoint of a slice, or participate in some sort of<br>
>> >>> integer arithmetic calculation, etc. I assume you also share this<br>
>> >>> worry to some extent or you wouldn't have put in the warning ;-).<br>
>> >>><br>
>> >>> I guess the bigger question is, why would we *not* use the standard<br>
>> >>> method for signaling an exceptional condition here, i.e., exceptions?<br>
>> >>> That way we're 100% guaranteed that if people aren't prepared to<br>
>> >>> handle it then they'll at least know something has gone wrong, and if<br>
>> >>> they are prepared to handle it then it's very easy and standard, just<br>
>> >>> use try/except. Right now I guess you have to check for the special<br>
>> >>> value, but also do something to silence warnings, but just for that<br>
>> >>> one line? Sounds kind of complicated...<br>
>> >><br>
>> >><br>
>> >> The main reason was for the case of multiple axis, where some of the results<br>
>> >> would be valid and others not. The simple thing might be to raise an<br>
>> >> exception but keep the current return values so that users could determine<br>
>> >> where the problem occurred.<br>
>> ><br>
>> > Oh, duh, yes, right, now I remember this discussion. Sorry for being slow.<br>
>> ><br>
>> > In the past we've *always* raised in error in the multiple axis case,<br>
>> > right? Has anyone ever complained? Wanting to get all<br>
>> > nanargmax/nanargmin results, of which some might be errors, without<br>
>> > just writing a loop, seems like a pretty exotic case to me, so I'm not<br>
>> > sure we should optimize for it at the expense of returning<br>
>> > possibly-misleading results in the scalar case.<br>
>> ><br>
>> > Like (I think) you say, we could get the best of both worlds by<br>
>> > encoding the results in the same way we do right now, but then raise<br>
>> > an exception and attach the results to the exception so they can be<br>
>> > retrieved if wanted. Kind of cumbersome, but maybe good?<br>
>> ><br>
>> > This is a more general problem though of course -- we've run into it<br>
>> > in the gufunc linalg code too, where there's some question about you<br>
>> > do in e.g. chol() if some sub-matrices are positive-definite and some<br>
>> > are not.<br>
>> ><br>
>> > Off the top of my head the general solution might be to define a<br>
>> > MultiError exception type that has a standard generic format for<br>
>> > describing such things. It'd need a mask saying which values were<br>
>> > valid, rather than encoding them directly into the return values --<br>
>> > otherwise we have the problem where nanargmax wants to use INT_MIN,<br>
>> > chol wants to use NaN, and maybe the next function along doesn't have<br>
>> > any usable flag value available at all. So probably more thought is<br>
>> > needed before nailing down exactly how we handle such "partial" errors<br>
>> > for vectorized functions.<br>
>> ><br>
>> > In the short term (i.e., 1.8.0), maybe we should defer this discussion<br>
>> > by simply raising a regular ValueError for nanarg functions on all<br>
>> > errors? That's not a regression from 1.7, since 1.7 also didn't<br>
>> > provide any way to get at partial results in the event of an error,<br>
>> > and it leaves us in a good position to solve the more general problem<br>
>> > later.<br>
>><br>
>> Can we make the error optional in these cases?<br>
>><br>
>> like np.seterr for zerodivision, invalid, or floating point errors<br>
>> that allows ignore and raise<br>
>> np.seterr(linalg='ignore')<br>
>><br>
>> I don't know about nanarg, but thinking about some applications for<br>
>> gufunc linalg code.<br>
>><br>
>> In some cases I might require for example invertibility of all<br>
>> matrices and raise if one fails,<br>
>> in other case I would be happy with nans, and just sum the results<br>
>> with nansum for example or replace them by some fill value.<br>
>><br>
> I'm thinking warnings might be more flexible than exceptions:<br>
><br>
> with warnings.catch_warnings():<br>
> warnings.simplefilter('error')<br>
> ...</p>
<p dir="ltr">Sure. Passing in a callback or just leaving the function out and telling people to implement it themselves would be even more flexible :-). But we have to trade off complexity of usage, complexity of teaching people how to do stuff (nobody knows how to use catch_warnings, we only know because we started writing warning tests just in the last year or so), usefulness in common situations, etc. The warnings api doesn't give you any way to pass results out, you still need a separate channel to say what failed and what succeeded (and maybe for the failures to say what the different failures are).</p>
<p dir="ltr">Anyway this back and forth still supprts my main suggestion for *right* now, which is that this is sufficiently nonobvious that with 1.8 breathing down our necks we should start with the safe behaviour and then work up from there.</p>
<p dir="ltr">-n</p>