After a goalless first half, the home side took control of the game with a goal on the break from Chris Almond and a Michael White penalty.
But Vince felt the opening goal had followed a foul on Paul Evans and the second should never have been a spot kick.
"It is so frustrating," he said, "What can you do when officials make decisions like that?
"I have never blamed the referees this season because I always feel it is a little bit of a cop out.
"But he has made two absolutely ludicrous decisions.
"There was a clear foul on Paul Evans, we were loaded on the attack and they have broken away.
"It was never a penalty and if it was a foul it was outside the area.
"The referee is 15 yards away, facing the play and the linesman gave it.
"It is an absolute joke, it really is.
"We work hard all week in training and these idiots - and they are idiots - ruin what we are trying to achieve.
"Why don't we get these decisions?
"There is nothing you can do. You can flick it round and say you should not rely on referees, but I am sorry they influence games.
"I did not think the referee was bad on Saturday but he has made two decisions that have cost us goals."
Town hit back late in the game through Mark Peers, whose long range shot slipped through the hands of home keeper Ryan McMahon.
It was not enough to get anything from the game although Vince felt his side deserved something.
"Was the performance disappointing?" he said. "I have watched a game where we have controlled the first half.
"I thought we got the ball behind them, did everything we should have done.
"We have created but we haven't had a shot on goal.
"I am not disappointed with the performance. I am disappointed that we have not got the result.
"The lads worked hard as a team. Their endeavour was great.
"We took the game to them and I thought we did well.
"If we have not done well, I am sorry, but I am not watching the same game as anybody else."
Town suffered another injury blow at Skelmersdale when Ritchie Sutton limped out of the action after 33 minutes with a hamstring problem.
"We think Ritchie will be out for the rest of the season," he said. "We knew it was a slight risk to play him.
"We are down to the bare bones but we have to get on with it."
Town are down to sixth after Saturday's defeat and Curzon Ashton, the team who replaced them in the play-off places, are in action at Warrington tonight.