The Shaymen conceded after just 33 seconds to Kenny Davis’ goal before Kevin Roberts was sent-off in the second-half and Michael Cheek doubled the hosts’ lead.
Harvey was left frustrated at how the night went for his team, and wants to put the game behind him as quickly as possible.
“We didn’t start at all,” he reflected. “They scored in the first minute, the first attack.
“We looked really sluggish. It was just a really poor, poor night for us. One to put in the bin and forget about.
“We started slowly. We had a long trip down here and I think we were still on the coach, weren’t sharp at all.
“That puts you on the back foot straight away.”
Harvey was bemused at referee Alex Young’s decision to send-off Roberts for two bookable offences, and felt the official didn’t take the conditions into account.
“I couldn’t believe it,” said the Town boss. “From where I was it looked like he’d won the ball.
“It’s a shame because our discipline has been very good and it’s not very often we pick up a booking never mind a sending-off.
“On a stinking, horrible pitch, to keep your feet is extremely difficult.
“I don’t think there was anything malicious throughout the whole game.
“That’s what happens, you end up talking about referees.
“It wasn’t a good night for us. We had a slow start, then went down to 10-men and it was always a battle then.
“Having said all of that, if Jordan Burrow puts that away when he goes through one-on-one with the keeper, that would have given us the last 15 minutes with plenty to go for and put them on the back foot.”
Harvey also felt Town should have had a penalty for a foul on Jake Hibbs, instead of which the referee booked the Halifax midfielder for diving.
“Those things are out of your control,” he said.
“Hibbsy’s been brought down, it’s blatant, it’s obvious, but we didn’t get it.”
The Town boss also felt the Braintree pitch made things difficult for his team.
He added: “It’s the same for both teams but they play on it every other week so they know how to play on there.
“The pitch was a really poor standard, the boys are slipping all over the place.
“And I don’t think the referee’s allowed anything. I don’t know how many bookings and fouls he gave - it was just stop-start.
“It wasn’t free-flowing - it was a difficult night for all of us.”