How CNET tests storage devices
Curious to find out how CNET comes up with the numbers for a review's performance chart? Dive in!
Testing a storage device is more difficult than you may expect. They may not be particularly glamorous, but storage devices serve a vital function in your digital life. Making sure they perform efficiently is important, which is why I wrote this post detailing how CNET tests the devices that we review.
Before I begin, though, it's essential that you have a good understanding of. So click over to read that series if you want to learn more.
There are three types of mainstream consumer-grade storage devices: internal, external, and network. Generally, we test them by transferring data from one place to another, but the exact method varies depending on the nature of the device's usage and capability.
Internal drives are drives that are inside a computer, such as a laptop or a desktop. As of now, we only review standard internal drives that come in either the 2.5-inch (laptop) or 3.5-inch (desktop) design. Nonstandard and proprietary internal drives typically aren't available to general consumers.
We test internal drives under two common scenarios: when they're used as a secondary drive for extra storage and backup, and when they serve as the main drive that hosts the operating system.
Test system: The test system for internal drives is a custom-built desktop running mid- and high-end components that supports both SATA 3 and SATA 2, the two most popular interfaces for internal drives. We configure the test operating system with the desktop and we use Acronis True Image software to make sure that every test is done with identical system configurations. We test all drives using SATA 3, though we also use SATA 2 to determine if the drive is backward compatible.
As secondary drive When it's used as a secondary device, we perform write and read tests on the internal device by copying 57GB of mixed data between it and the system's main drive. We determine the speed by dividing the total data (in MB) by the number of seconds it takes to complete the task. We run each test more than once for consistency.
Note: To ensure that the main drive is not a bottleneck for the data connection, we use a device that's faster than the drive that we're testing. So for example, if the reviewed unit is a regular hard drive, we'll use a solid-state drive (SSD) as the main drive. And in the case that a faster drive isn't available, we'll use a second identical unit of the reviewed drive.
As the main drive This is the most popular usage scenario for an internal drive, especially SSDs. In this test, we re-image the drive we're reviewing with the configured operating system and test the device by itself. Then, we copy roughly 57GB of mixed data from one place to another (so it reads and writes at the same time). Here again we determine the speed by dividing the total data (in MB) by the number of seconds it takes to complete the task. Also, we do more than one trial to ensure a consistent data rate. Note that this test represents the drive's real-world data transfer performance.
Note: Because SSDs have a limited number of program/erase cycles (aka the number of times you can write to the drive), SSD owners should refrain from testing drives at home.
After data transfer, we test the system running on the reviewed drive to determine how long it takes to boot, shut down, and resume from sleep mode. All of these tests are recorded in seconds.
Finally, we test different applications and games to note if the reviewed drive affects their performance. These tests, however, are anecdotal given that it's difficult to consistently and meaningfully quantify the difference between difference drives.
External storage devices
External storage devices, also known as direct-attached storage devices, are the external drives or external RAID storage systems that connect to a computer using a peripheral standard. Currently, the most popular standards are USB 3.0 and Thunderbolt, but we test reviewed devices with all peripheral standards they support, including FireWire, eSATA, and USB 2.0. In fact, we'll test a USB 3.0 device with USB 2.0 since the two are compatible and many older computers don't support USB 3.0. Finally, though all external storage devices share one test in common, certain Thunderbolt devices get two extra tests of their own.
Test system: Legacy external storage devices (USB, FireWire, eSATA) are tested with a desktop that runs Windows 7 Ultimate 64-bit. And when applicable, a USB 3.0 add-in card or an eSATA add-in card is installed. Thunderbolt devices are tested with a MacBook Pro late 2011 model, powered by an SATA 3 solid-state drive and Mac OS 10.7.Common test This test represent the most common task of the external storage device: transferring data from (the write test) and to (the read test) a computer test system. For the Thunderbolt connection, we again use roughly 57GB of data. For the rest of the connection types, however, we use only 7GB of data. Also, because Thunderbolt devices tend to be much faster and have more capacity, they require a larger amount of data to show their real performance. We run each test more than once to make sure the score is consistent.
Thunderbolt-only tests Since Thunderbolt is the first peripheral connection type that's faster than the standard shared by internal drives, we put multiple-volume Thunderbolt storage devices through two extra tests: unit to unit and self read and write (we exclude single-volume Thunderbolt storage devices from the extra tests because their speeds and capacities are similar to the performance of USB 3.0 storage devices). Both tests are performed with minimal involvement from the test machine.
Unit to unit This test is designed to show the best possible performance of a device. To complete it, we daisy chain two Thunderbolt storage devices (either two identical units or a reviewed unit and one faster) together and copy 57GB of mixed data between them.
Self read and write In contrast, this test determines the real-world usage for anyone who owns just one Thunderbolt storage device. We copy the 57GB data between two locations within the storage device itself so that it reads and writes at the same time.
Network storage devices
Network-attached storage (NAS) devices are those that are connected to a network, either via a network cable or Wi-Fi, instead of to a computer directly. They offer the same amount of storage and features to all the connected devices within the local network.
The test system: The test system for network storage devices is similar to what we use for legacy external storage devices. This time around, however, we connect the computer to a Gigabit router using a Gigabit connection. Also, we connect the reviewed NAS device to the same router using a Gigabit cable connection. Finally, a map network drive linked to a shared folder on the NAS device is created on the test computer for testing.
We conduct the performance test by copying 7GB of data from the test system to the storage device (write test) and then back (read test).