DHS fudged test results, watchdog agency says

A new report by the U.S. Government Accountability Office charges that the Department of Homeland Security used biased methods to enhance performance results in tests on a new generation of radiation detectors meant to protect U.S. ports.

At stake are $1.2 billion in contracts to produce advanced spectroscopic portal (ASP) monitors and thousands of lives should they fail to work.

Experts from four national laboratories were consulted prior to publication of the report (PDF) by the GAO, the nonpartisan audit and investigative arm of Congress, which was released yesterday.

The agency found that the DHS' Domestic Nuclear Detection Office "used biased test methods that enhanced the performance of ASPs." Specifically, it conducted preliminary tests and then allowed contractors access to the results, which they then used to adjust systems accordingly.

It is "highly unlikely that such favorable conditions" would be found in a real-world situation, the GAO report deadpanned.

Portals in use today detect radiation but cannot distinguish between different types. This leads to expensive and time-consuming delays at ports of entry when customs officers respond to false alarms, according to the Domestic Nuclear Detection Office. To remedy this, DHS sponsored research on new technology to enhance detection capabilities at the nation's ports. In 2006, it awarded contracts to three companies based on performance tests in Nevada the previous year: Raytheon, Thermo Electron and Canberra Industries.

The GAO, however, was not convinced that any "additional detection capability provided by the ASPs was worth the considerable additional costs." The accounting agency found that the DHS had no sound basis for spending taxpayer money and "relied on assumptions of anticipated performance instead of actual test data." It recommended further testing and a rigorous cost-benefit analysis.

It wasn't the first time that problems had been found in the procurement process. In a March 2007 report (PDF), the GAO concluded that DHS' decision to procure and deploy the new equipment was not supported by the cost and suggested that the department come up with some "objective" assessments of ASP capability.

The question was whether the new equipment, at six times the cost of current models, was better able to detect radiation through different masking materials, such as a lead. The GAO charged that the Domestic Nuclear Detection Office did not test portal limitations or make any effort to replicate the material that would be used to mask a radiation source from detection, a "critical oversight in DNDO's original test plan." Instead, the detection office is attempting to get off the hook by substituting what are essentially computer simulations that are not comparable with "actual testing with nuclear and masking materials," according to the GAO.

The GAO recommended that production of the new portal monitors be delayed until the DHS provides a "sound analytical basis for its decision to purchase and deploy the new technology."