Is there a way to do this if you have automerge set? I don't see a way to control the commit message in that case, but I could easily be missing something.

Yes, you need to edit the PR description. Automerge will take the PR description as the commit message.