Is there a way to do this if you have automerge set? I don't see a way to
control the commit message in that case, but I could easily be missing
something.
Yes, you need to edit the PR description. Automerge will take the PR
description as the commit message.
