Don't worry about this one, if compile time increase on other platforms is minor, it'll be fine for aarch64 too. We have limited TravisCI credits (actual status of that is a little unclear), so no need to burn them for this. We anyway should be moving CI providers for aarch64 at some point, probably to
https://www.drone.io/