There's also the town's failure to capitalise on the higher education boom of the mid-nineties by not establishing a university here when everyone knows students equal disposable income.
And, of course, the horrific transport links – ie a train service that makes walking to Leeds seem attractive – don't help.
But when it comes to reasons why Halifax has seemingly stagnated as neighbouring towns like Huddersfield have boomed of late, let's blame the Piece Hall.
Nice building and all, obvious historic value, reasonable architectural worth, good setting for last year's Doghouse music festival. But, let's be clear, this run-down market-place with a bunch of tat shops is not a jewel in Calderdale's crown. It's a thorn.
Here's the thing: the Piece Hall was pretty much out of date from its inception, and its design has never come back in fashion.
The place was thrown up in 1779 to house small-scale cloth merchants who had dominated the town's economy for 300 years.
Bad timing though, because within three decades the industrial revolution had killed most of those traders off along with the idea the Piece Hall could be a profitable wool trading post. Which sparked in those early 1800s the same question still being asked: what should be done with it?
Its past is one long stream of failed ideas to attract people there, less a monument to history, more to lack of imagination.
So, what should be done?
As shops and public space, it doesn't work.
People like swathes of grass and water and open promenades, when they're relaxing. They don't like cobbles in a prison-like courtyard bathed in shade.
And when it comes to shopping they like bright and airy, not dark and pokey and slightly intimidating when you're inside and the daylight recedes through portcullis windows.
What's worse, the Piece Hall sits right at the gateway to town just above the station meaning any potential development there has to fit in – presumably by looking equally dark and imposing and redundant.
It means it's stopping possible investment.
Forget crown on head, this is albatross round neck territory.
Not, I should make clear, that I advocate demolishing a Grade I listed building. Or replacing it, say, with a massive Arndale-style centre. That option has already been considered. Back in the seventies. It was rejected. Obviously.
No, clearly it needs to stay but be turned into something more valuable than shops and space few people like. Got me thinking: those tiny rooms and the pokey courtyard are like half the halls of residence you see.
Perhaps Halifax could still build that uni and attract those students – and all that disposable income – and the Piece Hall could be at the centre of it all. Or, more likely, perhaps that's just another in the constant stream of bad ideas this folly has inspired. God bless it.
colin.drury@halifaxcourier.co.uk