Feds hampered by incomplete MPAA piracy data

Last summer, not long after the U.S. government began a review of how piracy affects consumers and the nation's economy, the feds went to the major movie studios for help.

Representatives from the Government Accountability Office (GAO), Congress' investigative arm, asked the studios for information about a survey the studios had commissioned (PDF) that concluded piracy and counterfeiting cost the film industry $6.1 billion in 2005.

But the GAO never got all of the information it requested from the Motion Picture Association of America, according to GAO administrators, including Loren Yager, the author of the summary report that ensued and director of the GAO's International Affairs and Trade efforts. The agency said as much in the report: "It is difficult based on the information provided in the study to determine how the authors handled key assumptions."

Without the materials, government analysts couldn't properly evaluate the MPAA's 2005 survey. Last week, the GAO issued the results of its year-long study. Researchers there found that many of the claims copyright owners have made about piracy's effects on their businesses were based on unreliable research. The findings sent shock waves through tech and media circles and may have damaged the credibility of the MPAA, the trade group of the six largest film studios as well as other copyright owners.

And it doesn't get any better the deeper you dig. Had the GAO's analysts looked under the hood of the MPAA's research, they would have discovered that the findings were problem plagued almost from the start.

Fred von Lohmann, senior attorney at the Electronic Frontier Foundation challenged the film industry to reveal its data and, once it has done so, to then bankroll independent analysis.

"I think this shows the film industry is unwilling to stand behind their numbers when they are called into question," von Lohmann said. "I thought the GAO's report was very well done because it was willing to admit this kind of data is hard to get. The GAO didn't say that piracy didn't cause any harm. They just said the data was pretty inadequate. It's a much harder problem to solve than copyright industries have pretended."

A spokesman for the MPAA said that the group turned over everything the GAO requested and fully cooperated with the study.

"The MPAA met with the GAO last summer and had extensive, detailed follow-up conversations," said Howard Gantman, an MPAA spokesman, about the 2005 piracy study. "We believed it was extremely important that the GAO understand the serious threat from piracy to job creation, innovation, and economic growth."

Yager wanted to note that the agency received extensive cooperation from the studios and that they were under no obligation to cooperate. And why wouldn't the MPAA want to cooperate? The GAO's study on piracy was a result of a congressional plan to help improve antipiracy efforts. The GAO was there to help groups such as the MPAA and Recording Industry Association of America.

A shaky foundation
When the MPAA released the results of its survey in May 2006, it said that in addition to the $6.1 billion that piracy cost the U.S. film industry, piracy cost the worldwide film sector $18.2 billion. The study claimed college students were responsible for 44 percent of the film industry's losses. LEK Consultancy, the "international strategy firm" that conducted the survey, reported that the study was done over 18 months and surveyed more than 20,600 consumers in 22 countries "using focus groups and telephone, Internet, and in-person interviews."

But missing from the report was how LEK came to the dollar figures. The report did not detail the methodology or what assumptions LEK researchers made. Apparently, the GAO wasn't the first to ask to see the MPAA and LEK's details about its report. Ken Fisher, a reporter from the blog Ars Technica, noted in 2006 that the MPAA hadn't disclosed important background about the research.

"Normally you'd check the study, but in this case, the study can't be studied," Fisher wrote.

In 2008, more than 18 months later, the MPAA acknowledged that because of human error, the study had made erroneous statements. Instead of 44 percent, the films pirated by college students were responsible for only 15 percent of the film industry's revenue losses.

Coming up with more credible research might not be a simple matter, according to the GAO's Yager.

Yager said in an interview with CNET that the GAO received a lot of pressure from Congress to do its own study but leaders there concluded it just wasn't feasible. They focused on mining the available research for answers. He said the GAO was concerned that many of the most widely used numbers about the size of piracy in the United States were "wildly inaccurate."

"I think you can do a pretty good job of measuring piracy on a product basis," Yager said. "But from there to go to an industrywide problem you have to make assumptions. And from there to go to a national number...I think that may be impossible."

The GAO said the problem of quantifying piracy over multiple industries across broad geographic boundaries has stumped great minds in Europe as well.

He said the Paris-based Organization for Economic Cooperation and Development took a stab at doing its own research and was thwarted. "They've got a lot of top economists and, frankly, they came up pretty empty," he said. "I think they were all a bit surprised and disappointed about how much they put into it and how little they got out."

Meanwhile, the Obama administration has already promised the entertainment sector antipiracy support and is looking to move forward with the proposed global Anti-Counterfeiting Trade Agreement, or ACTA. The president and Congress could conclude that there's plenty of anecdotal and other evidence to support increased efforts in this area.

But there's no getting around that any entertainment company or industry that attempts to put a dollar figure on piracy or tries to tie piracy to the country's larger economic woes now has a high bar to overcome.