<div dir="ltr">Hi Gregory,<br><div class="gmail_extra"><br><div class="gmail_quote">2017-01-16 20:28 GMT+01:00 Gregory P. Smith <span dir="ltr"><<a href="mailto:greg@krypto.org" target="_blank">greg@krypto.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p dir="ltr">Is there a good reason not to detect single expression multiply adds and just emit a new FMA bytecode?</p></blockquote><div>Yes ;-) See below.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">Is our goal for floats to strictly match the result of the same operations coded in unoptimized C using doubles?</p></blockquote><div><br></div><div>I think it should be. This determinism is a feature, i.e. it is of value to some,</div><div>although not to everybody.</div><div>The cost of this determinism if a possible loss of performance, but as I already mentioned</div><div>in an earlier mail, I do not believe this cost would be observable in pure Python code.</div><div>And anyway, people who care about numerical performance to that extent are all using Numpy.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">Or can we be more precise on occasion?</p></blockquote><div><br></div><div>Being more precise on occasion is only valuable if the occasion can be predicted/controlled by the programmer.</div><div>(In this I assume you are not proposing that x*y+z would be guaranteed to produce an FMA on *all* platforms,</div><div>even those lacking a hardware FMA. That would be very expensive.)</div><div> </div><div>Generally speaking, there are two reasons why people may *not* want an FMA operation.</div><div>1. They need their results to be reproducible across compilers/platforms. (the most common reason)</div><div>2. The correctness of their algorithm depends on the intermediate rounding step being done.</div><div><br></div><div>As an example of the second, take for example the cross product of two 2D vectors:</div><div><br></div><div>def cross(a, b):</div><div> return a[0]*b[1] - b[0] * a[1]</div><div><br></div><div>In exact mathematics, this operation has the property that cross(a, b) == -cross(b,a).</div><div>In the current Python implementation, this property is preserved. </div><div>Synthesising an FMA would destroy it.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">I guess a similar question may be asked of all C compilers as they too could emit an FMA instruction on such expressions... If they don't do it by default, that suggests we match them and not do it either.</p></blockquote><div><br></div><div>C99 has defined #pragma's to let the programmer indicate if they care about the strict FP model or not.</div><div>So in C99 I can express the following three options:</div><div><br></div><div>1. I need an FMA, give it to me even if it needs to be emulated expensively in software:</div><div> fma(x, y, z)</div><div><br></div><div>2. I do NOT want an FMA, please do intermediate rounding:</div><div> #pragma STDC FP_CONTRACT OFF</div><div> x*y + z</div><div><br></div><div><div>3. I don't care if you do intermediate rounding or not, just give me what is fastest:</div><div> #pragma STDC FP_CONTRACT ON</div><div> x*y + z</div></div><div><br></div><div>Note that a conforming compiler can simply ignore FP_CONTRACT as long as it never</div><div>generates an FMA for "x*y+z". This is what GCC does in -std mode.</div><div>It's what I would recommend for Python.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">Regardless +1 on adding math.fma() either way as it is an expression of precise intent.</p></blockquote><div><br></div><div>Yep.</div><div><br></div><div>Stephan</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-HOEnZb"><font color="#888888">
<p dir="ltr">-gps</p></font></span><div class="gmail-HOEnZb"><div class="gmail-h5">
<br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 16, 2017, 10:44 AM David Mertz <<a href="mailto:mertz@gnosis.cx" target="_blank">mertz@gnosis.cx</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail-m_2034495978117050036gmail_msg">My understanding is that NumPy does NOT currently support a direct FMA operation "natively." However, higher-level routines like `numpy.linalg.solve` that are linked to MKL or BLAS DO take advantage of FMA within the underlying libraries.</div><div class="gmail_extra gmail-m_2034495978117050036gmail_msg"></div><div class="gmail_extra gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"><div class="gmail_quote gmail-m_2034495978117050036gmail_msg">On Mon, Jan 16, 2017 at 10:06 AM, Guido van Rossum <span dir="ltr" class="gmail-m_2034495978117050036gmail_msg"><<a href="mailto:gvanrossum@gmail.com" class="gmail-m_2034495978117050036gmail_msg" target="_blank">gvanrossum@gmail.com</a>></span> wrote:<br class="gmail-m_2034495978117050036gmail_msg"><blockquote class="gmail_quote gmail-m_2034495978117050036gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto" class="gmail-m_2034495978117050036gmail_msg">Does numpy support this?<br class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036gmail_msg">--Guido (mobile)</div></div><div class="gmail-m_2034495978117050036m_-9113847478803286698HOEnZb gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036m_-9113847478803286698h5 gmail-m_2034495978117050036gmail_msg"><div class="gmail_extra gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"><div class="gmail_quote gmail-m_2034495978117050036gmail_msg">On Jan 16, 2017 7:27 AM, "Stephan Houben" <<a href="mailto:stephanh42@gmail.com" class="gmail-m_2034495978117050036gmail_msg" target="_blank">stephanh42@gmail.com</a>> wrote:<br type="attribution" class="gmail-m_2034495978117050036gmail_msg"><blockquote class="gmail_quote gmail-m_2034495978117050036gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail-m_2034495978117050036gmail_msg">Hi Steve,<div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div><div class="gmail-m_2034495978117050036gmail_msg">Very good!</div><div class="gmail-m_2034495978117050036gmail_msg">Here is a version which also handles the nan's, infinities,</div><div class="gmail-m_2034495978117050036gmail_msg">negative zeros properly.</div><div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div><div class="gmail-m_2034495978117050036gmail_msg">===============</div><div class="gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036gmail_msg">import math</div><div class="gmail-m_2034495978117050036gmail_msg">from fractions import Fraction</div><div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div><div class="gmail-m_2034495978117050036gmail_msg">def fma2(x, y, z):</div><div class="gmail-m_2034495978117050036gmail_msg"> if math.isfinite(x) and math.isfinite(y) and math.isfinite(z):</div><div class="gmail-m_2034495978117050036gmail_msg"> result = float(Fraction(x)*Fraction(y) + Fraction(z))</div><div class="gmail-m_2034495978117050036gmail_msg"> if not result and not z:</div><div class="gmail-m_2034495978117050036gmail_msg"> result = math.copysign(result, x*y+z)</div><div class="gmail-m_2034495978117050036gmail_msg"> else:</div><div class="gmail-m_2034495978117050036gmail_msg"> result = x * y + z</div><div class="gmail-m_2034495978117050036gmail_msg"> assert not math.isfinite(result)</div><div class="gmail-m_2034495978117050036gmail_msg"> return result</div><div class="gmail-m_2034495978117050036gmail_msg">===========================</div><div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div><div class="gmail-m_2034495978117050036gmail_msg">Stephan</div><div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div></div></div><div class="gmail_extra gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"><div class="gmail_quote gmail-m_2034495978117050036gmail_msg">2017-01-16 12:04 GMT+01:00 Steven D'Aprano <span dir="ltr" class="gmail-m_2034495978117050036gmail_msg"><<a href="mailto:steve@pearwood.info" class="gmail-m_2034495978117050036gmail_msg" target="_blank">steve@pearwood.info</a>></span>:<br class="gmail-m_2034495978117050036gmail_msg"><blockquote class="gmail_quote gmail-m_2034495978117050036gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Jan 16, 2017 at 11:01:23AM +0100, Stephan Houben wrote:<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
[...]<br class="gmail-m_2034495978117050036gmail_msg">
<span class="gmail-m_2034495978117050036gmail_msg">> So the following would not be a valid FMA fallback<br class="gmail-m_2034495978117050036gmail_msg">
><br class="gmail-m_2034495978117050036gmail_msg">
> double bad_fma(double x, double y, double z) {<br class="gmail-m_2034495978117050036gmail_msg">
> return x*y + z;<br class="gmail-m_2034495978117050036gmail_msg">
> }<br class="gmail-m_2034495978117050036gmail_msg">
</span>[...]<br class="gmail-m_2034495978117050036gmail_msg">
<span class="gmail-m_2034495978117050036gmail_msg">> Upshot: if we want to provide a software fallback in the Python code, we<br class="gmail-m_2034495978117050036gmail_msg">
> need to do something slow and complicated like musl does.<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
</span>I don't know about complicated. I think this is pretty simple:<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
from fractions import Fraction<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
def fma(x, y, z):<br class="gmail-m_2034495978117050036gmail_msg">
# Return x*y + z with only a single rounding.<br class="gmail-m_2034495978117050036gmail_msg">
return float(Fraction(x)*Fraction(y) + Fraction(z))<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
When speed is not the number one priority and accuracy is important,<br class="gmail-m_2034495978117050036gmail_msg">
its hard to beat the fractions module.<br class="gmail-m_2034495978117050036gmail_msg">
<span class="gmail-m_2034495978117050036m_-9113847478803286698m_-6458649721746544018m_2404065159388076217HOEnZb gmail-m_2034495978117050036gmail_msg"><font color="#888888" class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg">
<br class="gmail-m_2034495978117050036gmail_msg">
--<br class="gmail-m_2034495978117050036gmail_msg">
Steve<br class="gmail-m_2034495978117050036gmail_msg">
</font></span><div class="gmail-m_2034495978117050036m_-9113847478803286698m_-6458649721746544018m_2404065159388076217HOEnZb gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036m_-9113847478803286698m_-6458649721746544018m_2404065159388076217h5 gmail-m_2034495978117050036gmail_msg">______________________________<wbr>_________________<br class="gmail-m_2034495978117050036gmail_msg">
Python-ideas mailing list<br class="gmail-m_2034495978117050036gmail_msg">
<a href="mailto:Python-ideas@python.org" class="gmail-m_2034495978117050036gmail_msg" target="_blank">Python-ideas@python.org</a><br class="gmail-m_2034495978117050036gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br class="gmail-m_2034495978117050036gmail_msg">
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a><br class="gmail-m_2034495978117050036gmail_msg">
</div></div></blockquote></div><br class="gmail-m_2034495978117050036gmail_msg"></div>
<br class="gmail-m_2034495978117050036gmail_msg">______________________________<wbr>_________________<br class="gmail-m_2034495978117050036gmail_msg">
Python-ideas mailing list<br class="gmail-m_2034495978117050036gmail_msg">
<a href="mailto:Python-ideas@python.org" class="gmail-m_2034495978117050036gmail_msg" target="_blank">Python-ideas@python.org</a><br class="gmail-m_2034495978117050036gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br class="gmail-m_2034495978117050036gmail_msg">
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a><br class="gmail-m_2034495978117050036gmail_msg"></blockquote></div></div>
</div></div><br class="gmail-m_2034495978117050036gmail_msg">______________________________<wbr>_________________<br class="gmail-m_2034495978117050036gmail_msg">
Python-ideas mailing list<br class="gmail-m_2034495978117050036gmail_msg">
<a href="mailto:Python-ideas@python.org" class="gmail-m_2034495978117050036gmail_msg" target="_blank">Python-ideas@python.org</a><br class="gmail-m_2034495978117050036gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br class="gmail-m_2034495978117050036gmail_msg">
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a><br class="gmail-m_2034495978117050036gmail_msg"></blockquote></div><br class="gmail-m_2034495978117050036gmail_msg"><br clear="all" class="gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036gmail_msg"><br class="gmail-m_2034495978117050036gmail_msg"></div></div><div class="gmail_extra gmail-m_2034495978117050036gmail_msg">-- <br class="gmail-m_2034495978117050036gmail_msg"><div class="gmail-m_2034495978117050036m_-9113847478803286698gmail_signature gmail-m_2034495978117050036gmail_msg">Keeping medicines from the bloodstreams of the sick; food <br class="gmail-m_2034495978117050036gmail_msg">from the bellies of the hungry; books from the hands of the <br class="gmail-m_2034495978117050036gmail_msg">uneducated; technology from the underdeveloped; and putting <br class="gmail-m_2034495978117050036gmail_msg">advocates of freedom in prisons. Intellectual property is<br class="gmail-m_2034495978117050036gmail_msg">to the 21st century what the slave trade was to the 16th.<br class="gmail-m_2034495978117050036gmail_msg"></div>
</div>
______________________________<wbr>_________________<br class="gmail-m_2034495978117050036gmail_msg">
Python-ideas mailing list<br class="gmail-m_2034495978117050036gmail_msg">
<a href="mailto:Python-ideas@python.org" class="gmail-m_2034495978117050036gmail_msg" target="_blank">Python-ideas@python.org</a><br class="gmail-m_2034495978117050036gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br class="gmail-m_2034495978117050036gmail_msg">
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" class="gmail-m_2034495978117050036gmail_msg" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a></blockquote></div>
</div></div><br>______________________________<wbr>_________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a><br></blockquote></div><br></div></div>