The incident took place last week outside their premises on Hopwood Lane.
It means less mobile veterans and visitors don't have access to the building for the foreseeable future as it is not safe underfoot.
Paul Tait, vice chair of the Halifax Central branch of the Royal British Legion, said: "All we know is that it happened during the last week as with it being a long weekend there has been nobody in the office since the 13th, so it could have happened any time.
"I'm absolutely astounded and infuriated that they have done this outside a charity building and this is following on from a theft of over £15,000 worth of pin badges last October after they were delivered to Hopwood Hall and then later disappeared.
"The charity will be out of pocket by close to £20,000 by the time costs come in to repair the damage that has been caused and it is very unlikely that they can be replaced with like for like with the really high cost of Yorkshire stone right now, hence why it has been stolen.
"I am struggling to comprehend how they have had the audacity to steal from a charity.
"Now a Grade II listed building will never be the same again as the charity won't be able to fund replacing it like for like so we will have to source somewhere where we can get it replaced with modern paving and get it sorted as soon as possible to enable disabled access to the building without people having to go all the way round to the back of the building.
"I reported it to the police but it's unlikely that it will go any further despite the fact I mentioned a fair number of boot prints in the underlying sand that they may have been able to match if they ever had any suspects.
"Some nice person has also fly-tipped a load of rubble at the back of the building."
In a statement, West Yorkshire Police said: "On Wednesday 20 April, police received a report that a number of Yorkshire Stone slabs had been taken from a courtyard area outside an address on Hopwood Lane, Halifax.
"Enquiries are ongoing and anyone with information is asked to contact police on 101 quoting crime reference 13220210182."