It's an issue of training, not hearing per se. Audiophiles are interested in certain concepts and therefore pay attention to differences in imaging, how phase affects things, etc. They might notice a sense of depth with one recording that's missing from another, or how in one, a musician's position seems more discernible.
But it's not really hearing. If a mother of identical twins has no problem telling them apart, we don't say that she has better vision. We know that she can apply her visual skills based on her knowledge. Audio is the same.
If you learn to listen critically, you can hear all sorts of differences that you might otherwise take for granted. Even if you have hearing damage, you might find that you are more sensitive to distortion and higher quality equipment might be even more helpful for you than for somebody else. Or not.
What people forget is that many of the measurements that we use, such as frequency response, or distortion level, etc, are all ultimately based on listening tests. At one point, somebody determined what a "normal" person should hear, and came up with standards based on that. People listened. Scientists determined what they could or could not hear based on their responses. The numbers were used to quantify this.
But it's like picking a wife based on height, weight, and shoe size. Audio is such a complex phenomenon, and real life audio systems vary in complex ways, especially when it comes to transducers. So you can't objectively say what's better if one speaker is off by a certain amount here and there, and another is off by little bits in other places. You have to listen.
The bottom line is that if you can't tell an MP3 from a WAV file or can't tell a pressed CD from a copy, then none of this matters. You are probably better off. Ultimately, it's the music that matters and not the system. If you enjoy it, then don't worry about the rest.