I understand what you're saying, but it just seems to subjective. I've hit, killed and retrieved game that didn't in the least act like they were hit, but most certainly were.
I've also been on hunts where it appeared due to the animal's reaction, it was hit. But only to never find blood or any other evidence including a dead animal to support that.
So I don't see it as an issue of relying on anyone's word or trusting one over another. Certainly it's possible relying only on blood that a true hit results in no charge when there should be one, but it also prevents true misses being wrongly charged.