[SciPy-Dev] Adding logsoftmax function to scipy.special

Sat Dec 7 11:58:33 EST 2019

Hi Dominik and Ralf,

Thank you for your prompt response.
I got it!

Best,
Takuya

2019年12月7日(土) 13:48 Ralf Gommers <ralf.gommers at gmail.com>:

>
>
> On Fri, Dec 6, 2019 at 8:43 PM Dominik Stańczak <stanczakdominik at gmail.com>
> wrote:
>
>> Hey,
>>
>> your function may not show up if there's an __all__ variable in that
>> module that lists importable functions and you didn't add your function to
>> it. Sphinx goes by that one if it's available.
>>
>> Not sure that's relevant, but it's likely to be.
>>
>> Cheers,
>> Dominik
>>
>> On Sat, Dec 7, 2019, 05:34 Takuya Koumura <koumura at cycentum.com> wrote:
>>
>>> Hi,
>>>
>>> I implemented log_softmax and I wrote a docstring for it. Can anyone
>>> tell me how to add the function name to the scipy.special index page (
>>> https://docs.scipy.org/doc/scipy/reference/special.html)?  Is it Okay
>>> to directly edit the docstring in special/__init__.py?
>>>
>>
> Yes, you need to add your function to __init__.py, right under `softmax`
> looks like the correct place.
>
> And indeed, as Dominic suggests, you also need to add the function name to
> `__all__` in the file where you added it - `__all__` is a list of all
> public names in each file.
>
>
>> I wonder “.. autosummary::  :toctree: generated/” indicates it was
>>> generated by Sphinx?
>>>
>>
> No, that is a directive to tell Sphinx it should take the functions listed
> below it, and generated links and doc pages for those in the html/pdf
> output.
>
> Cheers,
> Ralf
>
>
>
>>
>>> Best regards,
>>> Takuya
>>>
>>>
>>> 2019年11月18日(月) 17:38 Takuya Koumura <koumura at cycentum.com>:
>>>
>>>> Hi Ralf, Josh, and all,
>>>>
>>>> Thank you for your positive comments and raising potential issues.
>>>>
>>>> Having read Josh’s comment, I conducted several experiments and found
>>>> there indeed is a case where cancellation occurs in my original
>>>> implementation, in which log_softmax(x) = x - logsumexp(x) = x - (m +
>>>> log(sum(exp(x - m)))) where m = max(x).
>>>>
>>>> Calculating s = x - m first and using it to calculate log_softmax(x) =
>>>> s - logsumexp(s) should decrease the chance of cancellation because it does
>>>> not involve subtraction other than x - m. Note that s is always <=0, so s -
>>>> log(sum(exp(s))) is actually addition of negative values.
>>>> Also, reusing s may be a bit more efficient in computational time.
>>>>
>>>> Best,
>>>> Takuya
>>>>
>>>>
>>>> 2019年11月18日(月) 3:51 Joshua Wilson <josh.craig.wilson at gmail.com>:
>>>>
>>>>> > I would like to implement logsoftmax(x) as x-logsumexp(x)
>>>>>
>>>>> The function seems reasonable to add, but I am not so sure about that
>>>>> implementation, as it appears that it could lead to cancellation.
>>>>> Note, for example, that because of the scaling `logsumexp` does to
>>>>> prevent overflow, with that implementation you will end up adding and
>>>>> subtracting the component `x_i` of the largest magnitude when
>>>>> computing the `i`th component of the results array, see e.g.
>>>>>
>>>>>
>>>>> https://github.com/scipy/scipy/blob/master/scipy/special/_logsumexp.py#L124
>>>>>
>>>>> Off the top of my head, it is not obvious to me that situations
>>>>> similar to that will not lead to cancellation errors.
>>>>>
>>>>> Of course, I might be wrong, but my overall point is that in order to
>>>>> use a particular implementation we should have a reasonable
>>>>> understanding of its theoretical properties, so I would like to see
>>>>> some exploration of that before proceeding.
>>>>>
>>>>> - Josh
>>>>>
>>>>> On Fri, Nov 15, 2019 at 5:17 PM Ralf Gommers <ralf.gommers at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi Takuya,
>>>>> >
>>>>> >
>>>>> > On Tue, Nov 12, 2019 at 9:05 PM Takuya Koumura <koumura at cycentum.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hello,
>>>>> >>
>>>>> >> I raised a GitHub issue (#11058) and was suggested to post it to
>>>>> scipy-dev.
>>>>> >>
>>>>> >> I’m considering to send a PR to add logsoftmax function in
>>>>> scipy.special. Before that, I would like to hear your opinion (partly
>>>>> because it’s my first time to send a PR to scipy).
>>>>> >
>>>>> >
>>>>> > Welcome! Thanks for proposing that. logsoftmax is fairly popular at
>>>>> least in deep learning, so it makes sense I think, and we already have a
>>>>> bunch of other log* functions in scipy.special.
>>>>> >
>>>>> > I noticed that both PyTorch and Tensorflow name this function
>>>>> `log_softmax` rather than `logsoftmax`. The latter would be a little more
>>>>> consistent with other functions (although we also have `special.log_ndtr`),
>>>>> while the former is consistent with other implementations of the same
>>>>> functionality. I'd be okay with either, with a slight preference for
>>>>> `log_softmax`.
>>>>> >
>>>>> >>
>>>>> >> I would like to implement logsoftmax(x) as x-logsumexp(x).
>>>>> Actually, special.softmax(x) = np.exp(x-logsumexp(x)), so it is trivial for
>>>>> those who read the source code of softmax, but I think including logsoftmax
>>>>> as a separate function will be useful for other users. Logsoftmax is more
>>>>> accurate with inputs that make softmax saturate, eg: When x=[1000, 0],
>>>>> np.log(softmax(x))=[0, -Inf] (maybe depending on the floating point
>>>>> precision), while logsoftmax(x)=[0, -1000].
>>>>> >>
>>>>> >> I am planning to add the new function at the bottom of
>>>>> special/_logsumexp.py following the softmax function, and add some unit
>>>>> tests in special/test/test_logsumexp.py. If you have comments, I’d
>>>>> appreciate any.
>>>>> >
>>>>> >
>>>>> > That seems like a good place.
>>>>> >
>>>>> > Cheers,
>>>>> > Ralf
>>>>> >
>>>>> >>
>>>>> >> Best wishes,
>>>>> >> --
>>>>> >> Takuya KOUMURA
>>>>> >> koumura at cycentum.com
>>>>> >> _______________________________________________
>>>>> >> SciPy-Dev mailing list
>>>>> >> SciPy-Dev at python.org
>>>>> >> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>> >
>>>>> > _______________________________________________
>>>>> > SciPy-Dev mailing list
>>>>> > SciPy-Dev at python.org
>>>>> > https://mail.python.org/mailman/listinfo/scipy-dev
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Takuya KOUMURA
>>>> koumura at cycentum.com
>>>>
>>>
>>>
>>> --
>>> --
>>> Takuya KOUMURA
>>> koumura at cycentum.com
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>

-- 
--
Takuya KOUMURA
koumura at cycentum.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20191208/535c677f/attachment.html>