<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Another idea might be to make most text a unique color that then could be not included in some image diffs. Sure there may be the occasional image that uses that colour but a few missed pixels won’t matter too much. <br><br><div id="AppleMailSignature">Sent from my iPhone</div><div><br>On Dec 11, 2017, at 6:20 PM, Elliott Sales de Andrade <<a href="mailto:quantum.analyst@gmail.com">quantum.analyst@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="auto"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 11 December 2017 at 00:44, Elliott Sales de Andrade <span dir="ltr"><<a href="mailto:quantum.analyst@gmail.com" target="_blank">quantum.analyst@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="auto"><div dir="ltr"><div dir="auto">Hi all,<div dir="auto"><br></div><div dir="auto">Downstream in Fedora (and maybe Debian), they are running into issues with testing and text. Fedora 26 has FreeType 2.7.1 and Fedora 27 & Rawhide has FreeType 2.8. Fedora 25 uses 2.6.5, but it will be EOL in the next week. Many other distros are also transitioning to these newer FreeType as well [1] and I think anaconda recently added 2.8 too.</div><div dir="auto"><br></div><div>With 2.7.1, a few tests fail (rms < 1) and it is straightforward to patch that [2]. With 2.8 though, over 800 tests fail [3] ranging up to ~80 rms [4]. This is a bit harder to paper over.</div><div dir="auto"><br></div><div dir="auto">I see a few ways to mitigate the problem, with varying advantages/disadvantages:</div><div dir="auto"><br></div><div dir="auto">1. Bundle the older version in the Matplotlib package like we do with tests. I don't really believe this to be a viable option for downstream, but I'm just mentioning it to be thorough. There are already a few (minor) security issues in the one we test against.</div><div dir="auto">2. Inject older FreeType just to run tests on the package. Again I don't like this idea. The point of running tests is to be sure that the version in a distro works *in that distro*. Testing with something a user could never install seems useless.</div><div dir="auto">3. Re-create all our current and future test images with 2.8. While this is most future-proof, adding over 800 images is going to bloat the repo quite a bit.</div><div dir="auto">4. Create some sort of side repo with test images for other FreeType releases. This would reduce bloat in the main repo but be somewhat more work. Thus I'd only suggest doing so for tags.</div><div dir="auto"><br></div></div></div></div></div></blockquote><div><br></div><div>One more thing I forgot to mention is ImageHash [5] which is used by Iris for image tests using the following strategy [6]:<br></div><div><blockquote>
<ul><li>using a perceptual 'image hash' of the outputs<a href="https://github.com/JohannesBuchner/imagehash" target="_blank"></a> as the basis for checking
test results.</li><li>storing the hashes of 'known accepted results' for each test in a
database in the repo<code></code></li><li>storing associated reference images for each hash value in a separate public
repository, <a href="https://github.com/SciTools/test-images-scitools" target="_blank"></a>allowing human-eye judgement of 'valid equivalent' results.</li><li>a new version of the 'iris/tests/idiff.py' assists in comparing proposed
new 'correct' result images with the existing accepted ones.</li></ul>
</blockquote></div><div>While this does reduce load in the main repo itself, it does increase the cognitive load for developers. Iris has a small core group of developers and much fewer drive-by contributions compared to Matplotlib, so I'm not sure we want to be doing this full idea. (Note also their repo is LGPL3, so please don't copy anything from there.) Using ImageHash might still be useful instead of RMS, though it may generalize things too much.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="auto"><div dir="ltr"><div dir="auto"><div dir="auto">I dislike the first two options as they would be repetitive across distros (unless they just stopped testing altogether), but the last two are not without work for us.</div><div dir="auto"><br></div><div dir="auto">Opinions? Alternative ideas?</div><div dir="auto"><br></div><div dir="auto">[1] <a href="https://repology.org/metapackage/freetype/versions" target="_blank">https://repology.org/metap<wbr>ackage/freetype/versions</a></div><div dir="auto">[2] <a href="https://github.com/QuLogic/matplotlib/commit/cfdc835923407810bd087f60332cdc8cdcb23f05" target="_blank">https://github.com/QuLogic<wbr>/matplotlib/commit/cfdc8359234<wbr>07810bd087f60332cdc8cdcb23f05</a></div><div dir="auto">[3] <a href="https://kojipkgs.fedoraproject.org//work/tasks/3137/23623137/build.log" target="_blank">https://kojipkgs.fedoraproject<wbr>.org//work/tasks/3137/23623137<wbr>/build.log</a><br></div><div dir="auto">[4] <a href="https://gist.github.com/QuLogic/477055a847a44cd444a0932432acffd1" target="_blank">https://gist.github.com/QuLogi<wbr>c/477055a847a44cd444a0932432ac<wbr>ffd1</a></div></div></div></div></div></blockquote></div></div><div class="gmail_extra">[5] <a href="https://pypi.python.org/pypi/ImageHash" target="_blank">https://pypi.python.org/pypi/I<wbr>mageHash</a><br></div><div class="gmail_extra">[6] <a href="https://github.com/SciTools/iris/blob/master/docs/iris/src/developers_guide/graphics_tests.rst#graphics-testing-strategy" target="_blank">https://github.com/SciTools/ir<wbr>is/blob/master/docs/iris/src/<wbr>developers_guide/graphics_test<wbr>s.rst#graphics-testing-strateg<wbr>y</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">-- <br><div class="m_-1486571873875372636m_5555398372463660224gmail_signature">Elliott</div>
</div></div></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Matplotlib-devel mailing list</span><br><span><a href="mailto:Matplotlib-devel@python.org">Matplotlib-devel@python.org</a></span><br><span><a href="https://mail.python.org/mailman/listinfo/matplotlib-devel">https://mail.python.org/mailman/listinfo/matplotlib-devel</a></span><br></div></blockquote></body></html>