How we test: Desktops

Most modern computers are now fast enough that you don't really need to question their ability to browse the Web, run word-processing or spreadsheet programs, or perform basic photo editing. If those common tasks won't put newer hardware to work, you can still find challenges out there, particularly thanks to the gigabytes of media files we've accumulated on our various portable storage devices. Tasks like rendering HD video files, converting large batches of high-quality digital images, or even performing multiple media-oriented processes at once can be beyond the capabilities of lower-end systems. For professionals or enthusiasts, it's also important to know which computer is best at those tasks at a given price. Throw in the ever-evolving PC gaming landscape, and performance testing remains an important part of the computer reviewing process.

Test environment

We run all of our desktop tests from our lab in New York City. We conduct all tests with the desktop disconnected from any network and with wireless networking and Bluetooth adapters disabled. If a system includes an integrated display, we set the screen to its native resolution; otherwise, we set the operating system resolution to 1,920x1,080 pixels. For all application testing, the desktop's power management settings are set to allow the system to run at maximum speed, with no components permitted to power down.

On Windows systems, we install all available critical Windows Updates, then turn off automatic updating; on Mac systems, we install all available software updates. For Windows systems, we also disable System Protection system restore, remote desktop, hard-drive defragging, Windows Defender, and Windows Firewall. We uninstall antivirus software because of its potential to interrupt our tests, and we also disable any screensavers.

We also disable the following settings under Windows 7's Performance/Visual effects controls:

• Animate controls and elements inside windows
• Fade or slide menu into view
• Fade or slide tool tips into view
• Fade out menu items after closing

Occasionally, some desktops arrive for testing at CNET Labs with nonstandard system settings, such as overclocked CPUs, GPUs, or graphics memory. To review such a system, we require that the manufacturer make the same settings clearly available on its purchasing Web site. We also require the system to pass 20 runs of the LinX system stability test, with the memory setting adjusted to the highest available option.

Application tests

Unless otherwise specified, we run all performance tests three times. We report the average of the three scores that are within +/- 5 percent of each other. If scores consistently fluctuate outside of the +/- 5 percent range, we run additional iterations of the test and instead report the overall average of all the test runs, throwing out the highest and lowest scores.

Using iTunes, we time how long it takes to convert 19 320Kbps MP3 tracks to 128Kbps AAC files, totaling 169MB. This test almost exclusively exercises a system's CPU capabilities. Apple iTunes supports multithreading, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.

Using our own custom Action file, we time how long it takes for Adobe Photoshop CS3 to execute the Action file on a collection of seven 12.7MB camera raw image files captured from an 8.2-megapixel camera. The Action file represents the automated tasks that a portrait or wedding photographer might undertake to prepare black-and-white proofs for a client, such as running the Unsharp Mask, Lens Correction, and Dust and Scratches filters, as well as reducing image noise and converting the images to grayscale JPEGs.

This test primarily exercises a system's CPU, memory, and chipset subsystem, but it also uses the graphics and hard-drive subsystems to a degree. Some of the filters we use in the Photoshop CS3 test can use multithreading, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.

Adobe Photoshop CS5 image-processing test

The CS5 test lets us measure the impact of 64-bit processing support in both Windows 7 and OS X. Certain CS5 filters, such as the lens correction filter, also benefit from GPU computing, an operation that relies on the computer's graphics card for processing assistance, and that's becoming more common in multimedia applications in particular.

We conduct our Photoshop CS5 image-processing test similarly to the CS3 version. We use the same image files as in the CS3 test, but with the color bit depth adjusted from 8-bit to 16-bit. We also increase the cache tile size from 128K to 1,024K.

We use Apple's QuickTime to convert a high-definition source video using QuickTime's "Movie to iPod" selection. The source file is an H.264-encoded, 30 frames per second (fps), 1,920x1,072 pixels, 302MB MOV file. While the video conversion takes place in the foreground, iTunes converts a group of 128Kbps MP3 files into 128Kbps AAC files.

This test's score is based on how long it takes a system to perform only the QuickTime conversion. The iTunes conversion taking place in the background is designed to significantly increase the overall CPU workload and to create a true multitasking environment. This test exercises nearly every major subsystem, including the CPU, the memory, and the hard drive. Desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.

Cinebench is a 3D-rendering test based on Maxon's 3D animation application, Cinema 4D. This test almost exclusively exercises a system's CPU capabilities. Cinebench 11.5 supports multithreading up to 64 processing threads, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.

3D games tests

We typically run our formal gaming tests when a desktop has a discrete graphics card. There is no single PC game that can determine the overall gaming performance outlook for a Windows desktop, which is why we rely on a suite of old and new titles, as well as a synthetic benchmark, to help paint the overall picture. We've taken some of the most challenging titles from recent years, on the theory that if a PC can play these games, it can play similar games with comparable performance. For desktops with integrated graphics chips, we rely on anecdotal testing using games that are less computationally challenging (although not necessarily any less fun).

Crysis test

Crysis might be from 2007, but it also remains one of the most challenging games available. We run this benchmark using the Crysis Benchmarking Tool developed by MadBoris. We run three iterations of the Harbor test, run at two display resolutions: 1,280x1,024 pixels and 1,600x1,280 pixels. Although Crysis supports DirectX 10, we run the test in DirectX 9 mode. This test still presents a challenge for modern gaming PCs, and remains relevant particularly considering the number of multiplatform games that still use DirectX 9.

Far Cry 2 test

Far Cry 2 is nearing the end of its life cycle. We run the test with the built-in benchmark tool using the Ranch Long fly-through in DirectX 10.1 mode, at 1,440x900 pixels and 1,920x1,200 pixels. Most PCs with discrete graphics cards can run this test with little difficulty. While it was useful several years ago, most new PC games are bypassing DirectX 10.1 for DirectX 11.

Metro 2033

Metro 2033 is a DirectX 11-based shooter that presents newer gaming PCs with a major performance challenge. It uses many newer graphics and game physics processing techniques. As this test tends to favor (but still challenge) Nvidia graphics cards, we hope to replace it with a more GPU-agnostic game soon. For now, we use this test to give higher-end gaming PCs a real challenge, and we run the built-in benchmark at 1,920x1,080 pixels and 2,560x1,600 pixels.

3D Mark 11

FutureMark's 3DMark 11 benchmark is the most recent addition to our test suite. Based on a series of gamelike 3D scenes, this latest synthetic gaming test incorporates DirectX 11 and GPU physics in a vendor agnostic software environment. We run only the combined CPU and GPU test, and we use the Extreme test and Performance preset settings at 1,920x1,080 pixels (with 16x anisotropic filtering turned on in the Performance run), and the Entry-level preset at 1,680x1,050 pixels.

Anecdotal testing

In addition to our established gaming tests, we will often run Call of Duty 4 or Portal 2 on OS X-based systems to compare their performance between Mac generations. For PCs that don't have graphics cards, we will often simply test with recent, popular PC games, like Portal 2 and Dragon Age 2, to see if the games are playable at various resolutions and image quality settings.

Power consumption testing

In addition to speed-based performance testing, we also conduct a series of power efficiency tests on all desktops. We test and tabulate results using the Environmental Protection Agency's Energy Star 5.0 qualification procedures, but we also add in a load test, and we use that information to estimate the annual cost to operate a desktop based on the national average energy cost of $0.1135 per kilowatt hour. We measure the power draw of each system using with a Chroma Digital Power Meter Model 66202.