I think that's the only option, I tried to pay him off but the quarters just turned into a whistle
You had to talk to him multiple times, not click on the quarters in your inventory, I can see how you got confused, it happened to me a few times as well.
