You really can't think of a different way to handle it?
I can think of many, each one of them worse.
Play out each circumstance in your head, assuming the ref has heard nothing, cameras/mics picked nothing up and it's one guys word against another. I can't think of another course of action that wouldn't open up a very big can of worms.
When it comes to rules, you have to assume the worst of people IMO. If you open a loophole, people will take advantage I reckon.