Audiophile MP3 players, by the numbers
Donald Bell gives up all the test results and analysis in his top picks for high-fidelity MP3 players.
When Jasmine and I evaluate MP3 players for CNET reviews, we always try to spend a few sentences describing any noticeable audio performance characteristics we detect during our subjective testing. We'll play around with all of the gadget's different EQ and sound enhancement options, listen back on our reference headphones, and run through a playlist of familiar music. We're only human, however, and hearing loss, ear wax, head congestion, and hangovers can skew our perceptions of audio quality from day to day. Thankfully, we have Eric Franklin.
Eric works in our CNET Labs and tests the audio quality and battery life of our MP3 players as part of his job. Unlike Jasmine and myself, Eric never actually listens to the MP3 players we review. Instead, he reformats the player, makes sure the latest firmware is installed, and transfers over a few white noise and sine wave audio files. Finally, he connects a Audio Precision ATS-2 Audio Analyzer to the headphone output of the MP3 player to reveal details such as the MP3 player's signal-to-noise ratio (SNR), total harmonic distortion (THD+N), frequency response deviation, maximum power output, and stereo crosstalk.
We've been testing our MP3 players this way for more than a year, but you wouldn't know it from reading our reviews because we've never included the data. Here's why:
Most MP3 players spit out the same results during testing, especially when you consider that a variance of +/- 5dB in each testing measure is indistinguishable to the human ear. Most of the variances we see between players aren't worth mentioning.
When we talk about "audio quality," most people assume we mean how pleasing something sounds to your ear. Unfortunately, our tests don't measure how pleasant an MP3 player audio chip sounds, but how accurately it performs. Paradoxically, not all accurate players sound great, and some of our favorite-sounding MP3 players have a few technical shortcomings.
I expect to hear some backlash on this point, but I'll put it out there anyway. These are "portable" music players. They are used as a means to shoehorn music into all the circumstances of your life that are the least ideal for appreciating the finer nuances of recorded audio. When you're on the bus, or at the gym, or driving in your car, the ambient noises all around you mask enough frequencies to make our lab measurements inconsequential. Personally, I think it's much more practical to know that an MP3 player has enough EQ and sound enhancement options to make a bass line cut through subway noise, than to know how accurately a given player reproduces frequencies in the inaudible 22kHz range. If we were testing home hi-fis, that's one thing, but portable audio players are a different beast.
That said, I'm an audio nerd at heart, and when I pored over the past year's audio test results, I couldn't help but share the high scores. As an extended version of the product comparison I posted last week, I've included the charts and full results of our top performers.
1.
I was not expecting a Creative MP3 player to come out on top of our list of cleanest-sounding MP3 players (much less, the top two). Don't get me wrong; Jasmine and I have long sung the praises of the excellent EQ and sound enhancement features Creative builds into their MP3 players. You just don't expect an MP3 player that's capable of boosting low frequencies to skull-rattling levels to also be the player with the most desirable audio traits for EQ-eschewing purists. I guess it shouldn't have come as a surprise that one of the leading manufacturers of computer audio cards would know a thing or two about digital audio performance. The Zen didn't have the flattest frequency response of the bunch, but it had an unmatched triple threat score of a 1.47dB frequency response deviation average, -82.27dB THD+N average, and a -83.62dB SNR. The Zen also gets bonus points for having its THD+N and SNR levels so close to one another.
2.
Creative's Zen Stone Plus couldn't quite strike the same overall balance as its bigger brother, but for less than $60, who can complain? With a -0.65dB frequency response deviation average, -78.95dB THD+N, and a -84.75dB SNR, the Stone Plus arguably performed better than the Zen depending on which measure you place more weight in. Still, we feel a difference of 4dB in the total harmonic distortion reading between the Zen and the Stone means more than the 1dB differences in FRD and SNR.
Then again, what's 4dB if you're saving $50 over the Zen?
3.
The iPod sound quality debate has followed the king of MP3 players from the very beginning. Love it or hate it, I had my money on Apple to win for the best technical sound quality. With an frequency response deviation of -1.56dB, a THD+N of -69.26dB, and an SNR of -84.42dB, the iPod Classic is nothing to turn up your nose at when it comes to clean audio reproduction. Other iPod models came close, but the Classic ruled roost among Apple's offerings. Support for the Apple Lossless music format, up to 160Gb of storage, and a hackable line-output through the dock connection, give the Classic other advantages over the competition.
4.
We pride ourselves on not judging a book by its cover, but admittedly, our expectations were fairly low for Best Buy's in-house brand Insignia Pilot MP3 player. We couldn't have been more dumbfounded when the Pilot NS-4V24 Bluetooth MP3 player gave an unbelievably flat frequency response, with a deviation average of just -0.44dB. Its total harmonic distortion average came in at a decent -62.38dB, while its SNR impressed us with an average of -87.3 dB--the best signal-to-noise ratio we have on record. Plus, it has stereo Bluetooth audio streaming built-in, and support for MP3, WAV, WMA, WMAPro, Audible, Ogg Vorbis, and Protected WMA file formats.
5.
Like the Insignia Pilot, we were not expecting much from the Sansa Clip. Why on earth would SanDisk pack one of their best sound chips into an MP3 player costing less than $50? The Clip's -1.36dB frequency response deviation average is slightly better than the iPod Classic, and a closer inspection of the graph reveals absolutely no frequency dips between the audible range of 20Hz-22kHz. A total harmonic distortion score of -65.25dB and an SNR of -84.78dB makes the Clip a top pick for budget audiophiles who have already put themselves in debt with their $500 headphones.
6.
I had Eric test the Sony NWZ-S718F with its noise-cancellation feature switched off, and its DSEE high-frequency enhancement feature turned on and turned off. It turns out that the Walkman performed better with DSEE engaged, giving us the supreme frequency response deviation reading of -0.28dB (or -0.77 with DSEE off). Throw in a THD+N rating of -63.57dB and an SNR of -83.25dB and Sony just slips out of the top five. Subjectively speaking, however, the NWZ-S718F is at the top of my audio-lover list. The noise-canceling headphones integrated into the NWZ-S718F allow you to hear more detail in your music under real world conditions.
Down, but not out...the Zune
There are only six slots in our CNET product comparisons, so I didn't have room to fit the Microsoft Zune. The
The world is not flat
It's been fun to geek out on all these numbers, but ultimately I don't recommend anyone buy an MP3 player based on stats alone. My best piece of advice: if audio quality is important to you, trust your ears over anything else. Your perception of what sounds good or bad is unique to you, shaped over years of listening habits and a personal musical taste.
For instance, one of our favorite-sounding MP3 players, the
So don't miss the forest for the trees, folks. Music is supposed to be fun, not math, and the best part of owning an MP3 player is weaving music into your messy, noisy life. That said, I hope I've given those unapologetically obsessed portable audiophiles something to chew on. If you'd like to see more of this kind of analysis in our formal MP3 player reviews, let me here you in the comments section. Also, let me know if there's another audio performance measure beyond frequency response, THD+N, or SNR, that you really want our labs to test for.