You were totally right. The man was totally wrong. And, if the people there knew you at all and said nothing to him, even more shame on them.