How we test: Dishwashers
You want to know how well your new dishwasher is going to get the coffee stains out of your mugs? So do we. Here's our process for putting dishwashers through the wringer.
Why do some dishwashers cost $400 and others $2,000? Both just wash the dishes. Can there really be that much of a difference? Dishwashers are certainly not created equal, with different methods of distributing water, arranging dishes and getting your dinnerware clean while keeping water use, power use and sound to a minimum. Testing how well dishwashers strike this balance requires a meticulous process in controlled conditions.
That's what we do here at our CNET Appliances office in Louisville, Kentucky. Our tests are modeled after the tests of the Association of Home Appliance Manufacturers (AHAM) and the International Electrotechnical Commission (IEC). Those are the standardized trials run by manufacturers when testing their own machines. To create CNET's tests, we combined pieces of those tests, consulted with experts at various manufacturers and added a few touches of our own.
Here are the steps of our unique testing process. Over the course of a week, we task the machines with things they can't do, so we can discover the boundaries of what they can, and ultimately find out which dishwashers are worth your money.
Setting the table
A lot of outside factors affect how a dishwasher performs. For example, high ambient humidity will make it much harder for any dishwasher to dry your dishes. And if your home suffers from low water pressure, there's only so much a dishwasher can do to mitigate the issue, and you'll likely have some leftover residue.
For our reviews, we've built a dishwasher lab that eliminates as many environmental variables as possible so that our reviews will be consistent throughout the year.
To that end, our dishwasher testing lab can control and monitor:
- Temperature
- Humidity
- Power
- Water temperature
- Water pressure
- Water flow rate
We control temperature and humidity with independent systems for the room designed to maintain our set temperature plus or minus 5 degrees F and our relative humidity percentage plus or minus 10. For power, we use an automatic voltage regulator to maintain our set point of 120 VAC plus or minus 2 percent.
For water, we have two separate temperature-conditioned water loops. One chilled to 55 degrees Fahrenheit (12.8 Celsius) and one heated to 140 degrees F (60 C). We mix these two water sources using electronic valves that adjust the mixture 60 times per second in order to maintain a constant inlet temperature of 120 degrees F (48.9 C). We also use variable speed pumps in combination with pressure-reducing valves to maintain both static and dynamic water pressures of 35 PSI at each test station.
For troubleshooting, we measure the water inlet temperature, drain temperature and flow rate so we can see if anything goes wrong during the course of our tests.
A choice selection of dishes
We spend most of our testing time with a dishwasher gauging how well it cleans and dries dishes. Our reviews also consider factors such as noise level, energy consumption, design, ease of use and extra features. Quite a few bells and whistles separate low-end and high-end models, but it's usually not hard to figure out which dishwasher is fancier. It can be quite challenging to figure out which one actually cleans better.
So, we spend more than a week with each machine hooked into our dish lab, and run it through an arduous set of trials designed to see what it cleans well and what it doesn't.
We start each test by cleaning 112 items, both dishes and flatware. That's not the actual test -- we still have a lot to do before we load them all into the dishwasher and hit Start. First we have to examine all our test items to be sure they're perfectly clean and dry. We want to make sure that the only dirt on the dishes when we load them into the dishwasher is the dirt we put there.
Here are the items we use for each test:
- 5 porcelain dinner plates
- 5 melamine dinner plates
- 5 porcelain small plates
- 5 melamine small plates
- 6 porcelain bowls
- 4 melamine bowls
- 1 porcelain serving bowl
- 1 Melamine serving bowl
- 1 porcelain platter
- 1 melamine platter
- 4 porcelain coffee cups
- 4 melamine coffee cups
- 4 porcelain saucers
- 4 glass drinking glasses
- 4 plastic drinking glasses
- 4 wine glasses
- 10 knives
- 20 teaspoons
- 2 serving spoons
- 2 serving forks
- 10 dinner forks
- 10 salad forks
We include a mix of materials such as porcelain and plastic as they absorb heat differently.
We use some aspects of both the AHAM and IEC tests mentioned above in our testing, as well as some of our own ideas. For example, neither IEC nor AHAM includes wine glasses. We do because wine glasses are unusually shaped and can reveal a lot about how a dishwasher arranges its tines and gets water into narrower cavities.
The mix of porcelain and plastic (melamine is a durable type of plastic) illustrates how well a dishwasher cleans and dries dishes of different materials. We also include some normal-sized coffee cups along with the tiny AHAM coffee cups that look like they belong in a Jane Austen novel.
We took a similar approach when selecting the foods to coat the dishes, taking some ideas from AHAM and some from IEC, and mixing them with some of our own.
We keep our foods on a rolling cart so we can move them quickly when we apply them to the various dishes.
Getting dirty
The next part's fun, because after all of the clean and dry dishes are accounted for and neatly set out, we slather them with a bunch of different types of food. Each dinner plate, for example, needs four different foods, or soils, spread on it -- egg yolk, spinach, honey, and macaroni and cheese.
We're not imagining many people eat a regular dinner of eggs, honey, spinach and mac & cheese, but again, we need to control the variables, and each soil we use reveals something about how the dishwasher performs. Egg and honey are sticky, while spinach and mac & cheese put stress on the dishwasher's filter.
The large plates get coated with egg, spinach, mac & cheese and honey.
A dishwasher will generally run several cleaning cycles in which it sprays the dishes, uses detergent to clean them and then rinses them. If the filter doesn't work well, the spinach might come off the first time the plate gets sprayed, then get redeposited onto another dish during a later cycle. Even though we only use spinach on large plates, we often find little green spots on the inside of the coffee mugs when we inspect the dishes after the run.
You can generally expect to get a better filter if you pay more for your machine, but we want to make sure that's true with our tests.
We soil our dishes as follows:
-Dinner plates: A quarter of the surface is covered in egg, spinach, mac & cheese and honey.
-Small plates: Half of the surface is covered in raspberry preserves, and the other half is coated with buttery mashed potatoes.
-Bowls: We coat the inside of three porcelain bowls entirely with oatmeal, and three with chili. Two melamine bowls get oatmeal, two get chili.
-Serving bowls: Each bowl gets half a coat of chili and half a coat of oatmeal.
-Serving platters and knives: These two go together. We scoop peanut butter onto a corner of each platter, run the blades of the knives through the peanut butter, then place five knives on each platter, letting the blades touch and dirty the platter rims with more peanut butter.
-Serving forks and dinner forks: We brush their tines with egg, and place them on the egg quadrants of the dinner plates.
-Salad forks: We rub the smaller forks in the mac & cheese, and put them on that quadrant of the dinner plates.
-Teaspoons: Half the spoons get a coat of raspberry preserves, the other half get mashed potatoes and all go on their matching soils on the small plates.
-Serving spoons: Rounding out the flatware, we cover one serving spoon in chili, the other in oatmeal, and put both in the matching half of a serving bowl.
-Coffee cups: We fill each cup roughly to the halfway point with coffee, then slowly pour it out while rotating the cup so the entire inside gets coated.
-Saucers: Similarly, we pour coffee into the saucer, stir it around, then pour it out.
-Wine glasses: We coast these the same way as the coffee cups, only with red wine.
-Drinking glasses: Again, we coat these as we do the coffee cups, but using tomato juice for two of the glass glasses and two plastic glasses, and grape juice for the other four.
After we finish dirtying all 112 items, we set them aside and let the soil dry on them for exactly 24 hours. That drying time forces the dishwasher to work hard to remove the dirt, again allowing us to see what it does well and what it doesn't.
Once they're dirty, we let the dishes sit for 24 hours. The Platemate allows us to stack them without placing them right on top of each other and ruining our work.
For added fun, we prepare, measure and spread all 13 soils across all 112 items within 30 minutes. Our goal is to give every dish exactly 24 hours to sit from the time we dirty it to the time we wash it, but obviously, the first dish we soil will sit for slightly longer than the last. So, we keep ourselves to 30 minutes of coordinated food-spreading craziness to keep that variance to a minimum, timed from when we first interact with the food to when the last dish is dirty. The smell in the room after we've prepared and spread all of these foods simultaneously is that much more motivation to get the process done quickly.
Loading the dishes
After 24 hours, it's finally time for the dishwasher to get to work. Beforehand, we've run the dishwasher once, and filled it with the manufacturer's recommended rinse aid. Next, we load the dishes following the recommended loading pattern of the manufacturer and add 20 grams of Cascade Complete detergent.
We follow the manufacturer's recommended loading pattern when putting the dishes into the dishwasher.
We set the dishwasher to run its normal cycle with no add-ons like an extra rinse or dry. This allows us to directly compare results from one machine to the next, as cycles other than normal can vary significantly from dishwasher to dishwasher.
During the couple of hours the dishwasher takes to do its job., our computers are still working hard, monitoring and collecting data on the room and water conditions. Once the end-of-cycle signal sounds, we set a timer for 30 minutes to give the dishwasher a chance to help itself on its dry score with the residual steam and heat from the cycle. After 30 minutes, we pull out the dishes and see how it did.
Once the dishes are loaded, we run the normal cycle on the dishwasher. Pictured is the GE PDT750SSFSS.
Making the grade
With all lights turned off except for our grading light -- a 4-foot LED array set to a specific color temperature -- we hold each dish up against a black backdrop, carefully examining every side and angle. We then assign a numerical value to each dish based on the number and size of dirty spots we find. A small spot counts as a 1, three small spots count as a 3 and a big spot might count as a 5. If a dish is completely clean it gets a 0, while the dirty score for a dish maxes out at 9. At the same time, we look for water drops and water marks, and give a separate numerical value based on how many of those we see.
Once we have two scores, a dry score and a clean score for each dish, we enter both into an equation to calculate the total percent dryness and cleanliness. We use the AHAM score system for cleanliness and IEC's scoring system for our dry testing.
We examine each dish under a light and against a black backdrop.
And that's the end of one test. Now we can use this numerical grading to compare this dishwasher with others. We also now know what kind of soils the dishwasher left behind, and on what types of dishes. If spots of honey remain, for example, it could mean the spray arms don't effectively hit all large plates with enough force to scrape them away. (We know any leftover honey wasn't because of lack of water pressure, again, because that's something we regulate.)
Then, we run this same test two more times to make sure we're getting consistent results and eliminate the possibility of human error as much as possible. If any of our results fall more than 5 percentage points apart from the rest, we redo that test.
But before we soil the dishes for Test 2, we come back to that very first step -- we wash the dishes. Yes, we just washed them, but inevitably there's crud left over, so we clean them again, and wipe them down and inspect them. We can't start again until all 112 items are completely clean and dry. We also run a quick empty cycle on the dishwasher being tested and clean its filter. Then we can start the second test. Roughly a week later, we're fully finished testing a dishwasher.
At that point, we average the three clean and dry scores. Occasionally, we average more than three scores. Again, we don't count results that aren't within five percentage points of the rest, but every now and again, a fourth test at say 67% gives us four matching results if the first three were 62%, 63%, and 69%, so we average all four.
Then, we break the scores down into a subcategories for dinnerware (plates, bowls, serving platters and the like), silverware and glassware, then average out the amount of time the dishwasher spent getting to these results. We pair those numbers with the manufacturer's listed energy usage and sound rating, and have a complete picture of a dishwasher's performance.
Here are the results for the GE PDT750SSFSS.
The leftovers -- sound, energy, features and usability
During the course of a dishwasher's stay in our lab, we run it a few times outside of the regimented tests. We always allow a dishwasher to cool down for more than 8 hours before an official run, so residual heat doesn't affect the results. But during the 24-hour period in which we leave soiled dishes to set before cleaning, we have a window in which to play around with the machines, usually right after their official test of the day.
If the schedule works out, we'll use the dishwasher being tested to clean up after itself -- putting the dishes it didn't quite get clean in an official test back in, but this time running a heavier cycle or an auto cycle. We'll also use the dishwasher to wash the plastic containers, brushes and spatulas we use to soil the dishes. The unusual shape of the various pots and pieces of Tupperware shows how well the tines can fit differently shaped dishes.
While we test, we make note of how easy a dishwasher is to load, and how useful the features are.
Between these side tests, we evaluate how easy it is to fold down or shift the tines, how helpful the various features are and how well the organizational pieces work.
Once we're done with our official cleaning tests, if a dishwasher has an "extra dry" option or something similar, we run anecdotal tests to see how well it works. We load in the same set of 112 dishes, but this time without any dirt on them. Then, we grade the dishes the same way, but only check for dryness. As with our main clean and dry test, we run this extra dry test three times and average the results.
We don't actually test for how loud the dishwasher is. That's an important factor for some people; we simply use the manufacturer's listed decibel rating for our reviews, and we note if anything sounds out of place. We take the same approach with energy draw, and use the appliances' Energy Star info or similarly certified data.
The finishing touches
To put a bow on the process, we gather all of the information we learned about the dishwasher over the course of our week with it, and rate the dishwasher overall on a 10-point scale. Well, the overall 10-point rating is actually based on a weighted average of four different 10-point subratings -- performance, features, usability and design. All four subratings factor in value, so a $500 machine that can keep up with a $1,000 unit is going to get a better score.
Features gets 30 percent of the weight as this score includes noise level, and all of the extra bells and whistles that might make one machine attractive such as a third rack, extra cycle options or specific zones for specific dishes. The Design and Usability score are each worth 15 percent. You'll be looking at your dishwasher often, but it does sit under your counter. And how easy it is to load and unload can certainly make a difference, but once you learn the quirks of a dishwasher, you'll most likely find it easy enough to get used to.
The Performance score factors in both the clean and dry score, and accounts for 40 percent of the average. This category, representing our longest and most meticulous test, gets the most weight because above all, we're looking for a dishwasher that can wash dishes. Whether a dishwasher is worth the money to you will depend on what you're looking for, but we do our best to combine regimented performance tests with more casual runs to give you a complete picture of what a dishwasher does well, and what it doesn't.