Put simply: High-end audio is always expensive, audiophile gear isn't.
For example, the recently introduced ELAC Debut B5 speaker that sells for a mere $230 (£250) per pair is an outstanding audiophile speaker, but it will never, ever be mistaken for a high-end speaker. The Magnepan 3.7i flat-panel speaker is an audiophile speaker that's also a bona-fide high-end speaker, it sells for $5,995 per pair in the US, £6.990 in the UK.
High-end audio, like high-end cars, watches, cameras, clothes, wines, etc., are first and foremost, premium-priced products that offer extreme levels of quality and/or performance. A Camaro ZL1 may be a really fast car, but it's not in the same league of quick as a high-end car like aor . Extreme performance never comes cheap, but the good news is high-end audio is a hell of a lot more affordable than high-end cars!
I cover the full gamut of quality audio, from budget audiophile gems like the Andrew Jones Pioneer SP-BS22-LR bookshelf speakers that sell for $129 per pair in the US (£160 in the UK), to high-end exotics like thes ($79,750 per pair). The FiiO X5 2nd Gen portable high-resolution music player ($349, £289) is an audiophile player, but it's not a high-end design. The ($2,495, £1,400) looks, feels and sounds way better, and it exemplifies high-end design ethos.
High price is no guarantee of great sound, not by a long shot. Still, all of the very best audio I've heard, and I've heard a lot, is always extremely expensive. So it is price and performance that distinguish high-end audio from audiophile gear. High-end designers, freed from the budgetary constraints placed on more moderately priced audio components can explore the outer limits of performance and use the best quality materials.
So how are audiophile products such as speakers different from mainstream speakers? There's no strict design criteria I can point to, but audiophile components are designed with sound quality as a top priority. Mainstream product designers are more concerned with adding features, minimizing size, brand recognition, styling and more, than optimizing sound quality. It's no wonder mainstream designs are hugely more popular than the audiophile alternatives that sell for near the same price. Sound quality isn't a major factor for most buyers.
The way I see it audiophile gear comes in a wide range of prices from downright affordable to crazy expensive, but high-end products are, by definition, expensive.