Ribble Valley MP Evans, 57, the former deputy speaker of the House of Commons, denies indecently assaulting the man 10 years ago.
At Preston Crown Court, the man, who can not be identified for legal reasons, said Evans knew he was openly gay and was “testing the water” but overstepped the mark on the night in question.
He said he was drinking with Evans and another woman - a parliamentary press adviser - when Evans put his hand down the waistband of his trousers and rested his fingers on his buttocks.
The man said he knew Evans had a reputation and when he felt his hand he thought: “Oh God, it’s happening to me now.”
He stepped away and told the woman he was with: “If he does it again I’m going to punch him.”
But she advised him: “You can’t punch the secretary of state for Wales” suggesting it would be in the newspapers as Evans occupied a front bench position in Parliament.
After the alleged assault the man said he and his friend put Evans in a taxi.
He told the court he did not see the incident as a crime and forgave Evans. “It was just Nigel being drunk Nigel”, he told the court.
He said he saw Evans in Parliament after he had been arrested on suspicion of rape and could not believe the allegations.
He said: “I said, ‘Do you want to come for a drink? Stay strong.’”
The man never complained about what had happened to him and still considered Evans a friend although he had been annoyed at the incident.
However when he was approached by police during the course of the investigation he agreed to testify.
Evans denies charges against seven men.