How we test: Laptops
Learn about CNET's testing procedures for laptops.
Battery life and application performance are two of the more important performance-based attributes of a laptop computer, and they can significantly influence each other. While most laptops modulate processor speed and other resource-intensive components such as screen brightness to save energy, models with more powerful processors or larger displays typically sacrifice battery life for raw power. CNET Labs' tests are designed to gauge a laptop's expected battery life as well as application and gaming performance using many of the same applications you might use on a regular basis, such as Windows Media Player or DVD Player (on the Mac), Apple iTunes, and Adobe Photoshop. Our test results, performance charts, and analysis compare a laptop's performance with that of other systems in its class.
When a laptop arrives at CNET Labs for testing, we try to make as few changes as possible to its operating system and software configuration. There are a number of basic changes we must make, however, in order to ensure that certain system settings won't adversely impact a system's performance. The adjustments we make also allow us to compare the performance of different laptops on as even a playing field as possible. Before installing our tests, we uninstall any preinstalled programs that will be used by our benchmarks, ensuring that we always test with the correct version of our test applications.On Windows systems, we install all available critical Windows Updates, and then turn off automatic updating; on Mac systems, we install all available software updates. For Windows systems, we also disable System Protection (Vista) or System Restore (XP), Problem Reporting (Vista) or Error Reporting (XP), Remote Assistance, Remote Desktop, and all Windows Security Center alerts.
For Vista systems, we also disable the following Visual Effects:
- Animate controls and elements inside windows
- Animate windows when minimizing and maximizing
- Fade or slide menus into view
- Fade or slide ToolTips into view
- Fade out menu items after clicking
- Slide open combo boxes
- Slide taskbar buttons
We run all tests with the laptop disconnected from any network and with wireless networking and Bluetooth (if any) disabled. Unless the particular test being run requires security software to be present, we deactivate all security applications while the tests are running. The system's screen resolution is set to the native resolution of the laptop and at a 32-bit color depth. For all application testing, the laptop's power management settings are set to allow the system to run at maximum speed, with no components permitted to power down. The laptop's screensaver is also disabled.
Unless otherwise specified, all tests are run a minimum of three times. Following the initial run on Windows systems, we defragment the system's hard drive using Diskeeper 2007 and force Windows to run its ProcessIdleTasks routine so that the system is fully optimized for the applications used by the tests. We report the average of the three scores that are within +/-5 percent of each other. If scores consistently fluctuate outside of the +/-5 percent range, we run additional iterations of the test and instead report the overall average of all the test runs, throwing out the highest and lowest scores.
Testing laptop battery life is as much art as it is science. The criteria that a manufacturer uses to measure battery life can be very different from what others, including CNET Labs, use to measure it. There are numerous variables that can impact battery life, such as CPU utilization and screen brightness. What is most important is that the same criteria be used to measure battery life on all laptops that are being compared to one another. Our laptop battery performance test is designed to measure the expected battery life of a laptop playing back a full-screen DivX-converted movie. This scenario can be a fairly battery-intensive task and a popular laptop activity.
We first set the brightness of each laptop screen to 60 to 70 nits (or 60cd/m² to 70cd/m²), using a Konica Minolta LS-100 luminance meterto calibrate laptop brightness. If a laptop supports autodim, we disable the feature in order to ensure that the laptop remains at the same brightness level thought testing. We enable adaptive CPU, throttling those laptops that include this feature as user configurable.
The system volume is set to 100 percent; the media playback software's volume is set to 50 percent; and headphones are plugged into the laptop's headphone jack.
For Windows Vista systems, we test with a customized version of Vista's Balanced power scheme. The system, display, and hard disk are not allowed to sleep; the processor state is allowed to scale from 0 to 100 percent; the system is set to remain on until the battery level is 0 percent; and all alarms are disabled.
For Windows XP systems, we test with a customized version of XP's Portable/Laptop power scheme. The system, display, and hard disk are not allowed to sleep, and all alarms are disabled.
For laptops that use a power conservation utility other than the one built into Windows, we choose a profile and custom settings that comes closest to the above schemes.
For Mac laptops, we set the computer, display, and hard disk never to sleep.
For Windows laptops, we test using whichever playback codec is already installed. If one is not installed, we install and use the codec from InterVideo's WinDVD 8. By default, Mac laptops already include the necessary software to play movies. The media playback software we use on Windows laptops is Windows Media Player 11; we use DVD Player on Mac laptops. The video we use is a DivX conversion of the Collector's Edition of King Kong.
We time how long the laptop's power lasts before its battery dies. We set the file to repeat from the beginning in case the power lasts longer than the movie. We run this test only twice if the two scores are +/-5 percent of each other. Performance is reported in minutes.
We use Apple's QuickTime to convert a high-definition source video using QuickTime's "Movie to iPod" selection. The source file is an H.264-encoded, 30fps, 1,920x1,072, 302MB MOV file. While the video conversion takes place in the foreground, iTunes converts a group of 128Kbps MP3 files into 128Kbps AAC files. This test's score is based on how long it takes a system to perform only the QuickTime conversion. The iTunes conversion taking place in the background is designed to significantly increase the overall CPU workload and to create a true multitasking environment. This test exercises nearly every major subsystem, including the CPU, the memory, and the hard drive. Desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.
Using our own custom Action file, we time how long it takes for Adobe Photoshop CS3 to execute the Action file on a collection of seven, 12.7MB Camera RAW image files captured from an 8.2-megapixel camera. The Action file represents the automated tasks that a portrait or wedding photographer might undertake to prepare black-and-white proofs for a client, such as running the Unsharp Mask, Lens Correction, and Dust and Scratches filters; as well as reducing image noise and converting the images to grayscale JPEGs. This test primarily exercises a system's CPU, memory, and chipset subsystem, but it also utilizes the graphics and hard drive subsystems to a degree. Some of the filters we use in the Photoshop CS3 test can use multithreading, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.
Using iTunes, we time how long it takes to convert 19, 320Kbps MP3 tracks to 128Kbps AAC files, totaling 169MB. This test almost exclusively exercises a system's CPU capabilities. Apple iTunes supports multithreading, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.
Cinebench is a 3D rendering test based on Maxon's 3D animation application, Cinema 4D. This test almost exclusively exercises a system's CPU capabilities. Cinebench supports multithreading up to 16 CPU cores, so desktops with multicore CPUs are likely to perform better than comparable systems that use CPUs with fewer cores or single-core CPUs.
3D games tests
Quake 4, a fairly demanding 3D game developed by Raven Software, is one of the few that is available for both Windows and Mac platforms. The game uses the OpenGLengine of Doom 3, which features its own lighting model (where lighting effects and shadows are generated in real time). The result is impressive graphics with realistic lighting and shadows that respond accurately to moving objects. After installing the retail game, we patch it to version 1.2, which is available for download here on GameSpot. Quake 4 settings:
- Video quality: High quality
- Full screen: Yes
- High-quality special effects: Yes
- Enable shadows: Yes
- Enable specular: Yes
- Enable bump maps: Yes
- Vertical sync: No
- Multiple CPU/core: Yes (if available)
We play back our own custom netdemo of actual gameplay usingQuake 4's Bloodwork map. The test generates an average frame rate score, reported in frames per second (fps); a higher frame rate is better. We run the test at resolutions of 1,024x768; 1,280x1,024; 1,600x1,200; and 2,048x1,536, with antialiasing at 4x and anisotropic filtering at 8x, using the game's internal settings; therefore, the graphics chip's driver interface settings for antialiasing and anisotropic filtering are set to application controlled. We run the test with vSync off. Occasionally, we run this test at additional resolutions and settings, which we note in the review.
F.E.A.R. is a very demanding 3D game that can bring many systems to their knees. Developed by Monolith Productions, F.E.A.R.is a DirectX 9-based game that uses its own proprietary graphic-and-physics engine. As it is a DX9 game, this test is only run on Windows XP and Vista-based systems. The game features complex particle effects, realistic physics, and real-time lighting- and shadow-effects models. After installing the retail game, we patch it to version 1.08, which is available for download here on GameSpot.
- Single player physics: Maximum
- Multiplayer physics: Maximum
- Max software sounds: Maximum
- Particle bouncing: Maximum
- Shell casings: On
- World detail: Maximum
- Corpse detail: Maximum
- Effects detail: Maximum
- Model decals: Maximum
- Water resolution: Maximum
- Reflections and displays: Maximum
- Volumetric lights: On
- Volumetric light density: Maximum
- FSAA: 4x *
- Light Detail: Maximum
- Enable shadows: On
- Shadow detail: Maximum
- Soft shadows: Off *
- Texture filtering: Anisotropic 8x
- Texture resolution: Maximum
- Videos: Maximum
- Pixel doubling: Off
- DX8 shadows: Off
- Shaders: Maximum
* Note that when we run the F.E.A.R. test at a resolution of 1,024x768, we turn Soft shadows on and turn FSAA off.
We play back the game's built-in performance test at resolutions of 1,024x768; 1,280x1,024; 1,600x1,200; and 2,048x1,536, with antialiasing at 4x and anisotropic filtering at 8x. We run the test with the graphics chip's driver interface settings for antialiasing and anisotropic filtering set to application controlled and with vSync off. The test generates an average frame rate score, reported in frames per second (fps); a higher frame rate is better.
In addition to the benchmark tests mentioned above, we run additional tests that are designed to evaluate the performance of the CPU, the memory, and the hard drive subsystems of a Windows desktop. If any of the results from these additional tests are especially relevant to a particular system, we comment on these findings in the review. We are constantly evaluating new tools to assist us in this process. At present, the benchmark we use for our anecdotal testing is SiSoftware Sandra.