Good point - and I think while you're right in saying "but you shouldn't" there are exceptions. I've done this a couple of times, only if two users made the exact same point, one a while before another. Maybe you'd argue I shouldn't have upvoted one and downvoted another as unfair karma wise to the later poster, but my decision to do so was based on improving where the comments are positioned (readers don't need to see the same view twice high up), not based on rewarding or punishing users.