Audiophile MP3 players, by the numbers

Donald Bell gives up all the test results and analysis in his top picks for high-fidelity MP3 players.

When Jasmine and I evaluate MP3 players for CNET reviews, we always try to spend a few sentences describing any noticeable audio performance characteristics we detect during our subjective testing. We'll play around with all of the gadget's different EQ and sound enhancement options, listen back on our reference headphones, and run through a playlist of familiar music. We're only human, however, and hearing loss, ear wax, head congestion, and hangovers can skew our perceptions of audio quality from day to day. Thankfully, we have Eric Franklin.

Are you the kind of portable audio purist who wouldn't think twice about dropping $150 on a portable heaphone amplifier for your MP3 player? HeadRoom

Eric works in our CNET Labs and tests the audio quality and battery life of our MP3 players as part of his job. Unlike Jasmine and myself, Eric never actually listens to the MP3 players we review. Instead, he reformats the player, makes sure the latest firmware is installed, and transfers over a few white noise and sine wave audio files. Finally, he connects a Audio Precision ATS-2 Audio Analyzer to the headphone output of the MP3 player to reveal details such as the MP3 player's signal-to-noise ratio (SNR), total harmonic distortion (THD+N), frequency response deviation, maximum power output, and stereo crosstalk.

We've been testing our MP3 players this way for more than a year, but you wouldn't know it from reading our reviews because we've never included the data. Here's why:

  • It's boring.

    Most MP3 players spit out the same results during testing, especially when you consider that a variance of +/- 5dB in each testing measure is indistinguishable to the human ear. Most of the variances we see between players aren't worth mentioning.

  • It's misleading.

    When we talk about "audio quality," most people assume we mean how pleasing something sounds to your ear. Unfortunately, our tests don't measure how pleasant an MP3 player audio chip sounds, but how accurately it performs. Paradoxically, not all accurate players sound great, and some of our favorite-sounding MP3 players have a few technical shortcomings.

  • Who cares?

    I expect to hear some backlash on this point, but I'll put it out there anyway. These are "portable" music players. They are used as a means to shoehorn music into all the circumstances of your life that are the least ideal for appreciating the finer nuances of recorded audio. When you're on the bus, or at the gym, or driving in your car, the ambient noises all around you mask enough frequencies to make our lab measurements inconsequential. Personally, I think it's much more practical to know that an MP3 player has enough EQ and sound enhancement options to make a bass line cut through subway noise, than to know how accurately a given player reproduces frequencies in the inaudible 22kHz range. If we were testing home hi-fis, that's one thing, but portable audio players are a different beast.

  • That said, I'm an audio nerd at heart, and when I pored over the past year's audio test results, I couldn't help but share the high scores. As an extended version of the product comparison I posted last week, I've included the charts and full results of our top performers.

    1. Creative Zen

    Photo of Creative Zen MP3 player.
    The Creative Zen took the top score in our CNET Labs audio test results. Corinne Schulze/CNET Networks

    I was not expecting a Creative MP3 player to come out on top of our list of cleanest-sounding MP3 players (much less, the top two). Don't get me wrong; Jasmine and I have long sung the praises of the excellent EQ and sound enhancement features Creative builds into their MP3 players. You just don't expect an MP3 player that's capable of boosting low frequencies to skull-rattling levels to also be the player with the most desirable audio traits for EQ-eschewing purists. I guess it shouldn't have come as a surprise that one of the leading manufacturers of computer audio cards would know a thing or two about digital audio performance. The Zen didn't have the flattest frequency response of the bunch, but it had an unmatched triple threat score of a 1.47dB frequency response deviation average, -82.27dB THD+N average, and a -83.62dB SNR. The Zen also gets bonus points for having its THD+N and SNR levels so close to one another.

    Frequency response test chart of the Creative Zen MP3 player.
    The Creative Zen offered a frequency response deviation of -1.47dB during testing, with a very gradual drop under 60Hz and over 22kHz.

    2. Creative Zen Stone Plus

    Photo of Creative Zen Stone Plus MP3 player.
    The Creative Zen Stone Plus delivers clean sound in a small, affordable package.

    Creative's Zen Stone Plus couldn't quite strike the same overall balance as its bigger brother, but for less than $60, who can complain? With a -0.65dB frequency response deviation average, -78.95dB THD+N, and a -84.75dB SNR, the Stone Plus arguably performed better than the Zen depending on which measure you place more weight in. Still, we feel a difference of 4dB in the total harmonic distortion reading between the Zen and the Stone means more than the 1dB differences in FRD and SNR.

    Then again, what's 4dB if you're saving $50 over the Zen?

    Frequency response test chart of the Creative Zen Stone Plus MP3 player.
    The Creative Zen Stone Plus scored better than its big brother during our frequency response deviation test, scoring an FRD average of just -0.65dB. While the majority of frequencies skimmed right along the 0dB midline, sloping at the upper and lower (inaudible) frequency ranges is a little more pronounced than the larger Zen player. CNET Networks

    3. Apple iPod Classic

    Photo of iPod Classic MP3 player.
    The mighty iPod Classic, in third place?

    The iPod sound quality debate has followed the king of MP3 players from the very beginning. Love it or hate it, I had my money on Apple to win for the best technical sound quality. With an frequency response deviation of -1.56dB, a THD+N of -69.26dB, and an SNR of -84.42dB, the iPod Classic is nothing to turn up your nose at when it comes to clean audio reproduction. Other iPod models came close, but the Classic ruled roost among Apple's offerings. Support for the Apple Lossless music format, up to 160Gb of storage, and a hackable line-output through the dock connection, give the Classic other advantages over the competition.

    Frequency response test chart of the Apple iPod Classic MP3 player.
    Among all four models of the iPod, the iPod Classic performed the best during testing, scoring an average frequency response deviation rating of -1.56dB. While the lower frequencies roll off down below 40Hz, the upper ranges stay steady up to 22kHz. CNET Networks

    4. Insignia Pilot

    Photo of the Insignia Pilot MP3 player.
    The Insignia Pilot? We were as surprised as you are.

    We pride ourselves on not judging a book by its cover, but admittedly, our expectations were fairly low for Best Buy's in-house brand Insignia Pilot MP3 player. We couldn't have been more dumbfounded when the Pilot NS-4V24 Bluetooth MP3 player gave an unbelievably flat frequency response, with a deviation average of just -0.44dB. Its total harmonic distortion average came in at a decent -62.38dB, while its SNR impressed us with an average of -87.3 dB--the best signal-to-noise ratio we have on record. Plus, it has stereo Bluetooth audio streaming built-in, and support for MP3, WAV, WMA, WMAPro, Audible, Ogg Vorbis, and Protected WMA file formats.

    Frequency response test chart of the Insignia Pilot NS-4V24 MP3 player.
    Out of nowhere comes the Insignia Pilot MP3 player with an impressive frequency response deviation average of -0.44dB. Sure, it dips at the inaudible extremes, but between 90Hz and 20kHz the reading stays glued to the 0dB midline. CNET Networks

    5. SanDisk Sansa Clip

    Photo of SanDisk Sansa Clip MP3 player
    SanDisk Sansa Clip

    Like the Insignia Pilot, we were not expecting much from the Sansa Clip. Why on earth would SanDisk pack one of their best sound chips into an MP3 player costing less than $50? The Clip's -1.36dB frequency response deviation average is slightly better than the iPod Classic, and a closer inspection of the graph reveals absolutely no frequency dips between the audible range of 20Hz-22kHz. A total harmonic distortion score of -65.25dB and an SNR of -84.78dB makes the Clip a top pick for budget audiophiles who have already put themselves in debt with their $500 headphones.

    Frequency response test chart of the SanDisk Sansa Clip MP3 player.
    SanDisk MP3 players are not known for their audio quality, but the superaffordable Sansa Clip is a notable exception. With a frequency response deviation average of -1.36dB that plots steady from 20Hz-22kHz and an admirable signal to noise ratio, the Clip can hold its own. CNET Networks

    6. Sony Walkman NWZ-S718F

    Photo of Sony Walkman MP3 player.
    Sony makes the cut, with a little performance enhancement.

    I had Eric test the Sony NWZ-S718F with its noise-cancellation feature switched off, and its DSEE high-frequency enhancement feature turned on and turned off. It turns out that the Walkman performed better with DSEE engaged, giving us the supreme frequency response deviation reading of -0.28dB (or -0.77 with DSEE off). Throw in a THD+N rating of -63.57dB and an SNR of -83.25dB and Sony just slips out of the top five. Subjectively speaking, however, the NWZ-S718F is at the top of my audio-lover list. The noise-canceling headphones integrated into the NWZ-S718F allow you to hear more detail in your music under real world conditions.

    Frequency response test chart of the Sony Walkman NWZ-S718F MP3 player.
    The Editors' Choice award-winning Sony Walkman NWZ-S718F with its DSEE high-frequency enhancement setting turned on. Ironically, when the setting was active, frequencies above 20kHz appear to be completely shelved. The active DSEE setting produced a better overall frequency response deviation average of -0.28dB, however, with DSEE disengaged the frequency response curve is continuous up through 22kHz. CNET Networks

    Down, but not out...the Zune

    There are only six slots in our CNET product comparisons, so I didn't have room to fit the Microsoft Zune. The 80Gb Zune scored a -1.03dB on frequency response, and an excellent -84.99dB on SNR, but had relatively average scores on total harmonic distortion, grabbing a -59.25dB. Arguably, the Zune's high capacity and support for WMA Lossless music playback might be more important to audio purists than a few dB of THD.

    Frequency response test chart of the Microsoft Zune MP3 player.
    The Microsoft Zune nearly broke the top list, especially with its frequency response deviation average of -1.03. CNET Networks

    The world is not flat

    It's been fun to geek out on all these numbers, but ultimately I don't recommend anyone buy an MP3 player based on stats alone. My best piece of advice: if audio quality is important to you, trust your ears over anything else. Your perception of what sounds good or bad is unique to you, shaped over years of listening habits and a personal musical taste.

    For instance, one of our favorite-sounding MP3 players, the Cowon D2, tested poorly because it has a musically flattering EQ curve engineered into it. So what? It sounds great, supports all kinds of lossless audio formats, and has one of the best EQ sections you can buy on a portable device.

    Frequency response test chart of the Cowon D2 MP3 player.
    To give some perspective, here's the frequency response chart of the Cowon D2, an MP3 player whose sound we can't get enough of. Many manufacturers (not just Cowon) deliberately engineer their MP3 players to sweeten certain frequencies. Being flat isn't everything. CNET Networks

    So don't miss the forest for the trees, folks. Music is supposed to be fun, not math, and the best part of owning an MP3 player is weaving music into your messy, noisy life. That said, I hope I've given those unapologetically obsessed portable audiophiles something to chew on. If you'd like to see more of this kind of analysis in our formal MP3 player reviews, let me here you in the comments section. Also, let me know if there's another audio performance measure beyond frequency response, THD+N, or SNR, that you really want our labs to test for.

     

    ARTICLE DISCUSSION

    Conversation powered by Livefyre

    Don't Miss
    Hot Products
    Trending on CNET

    Hot on CNET

    The Next Big Thing

    Consoles go wide and far beyond gaming with power and realism.