No, half of The Last Jedi haters were not Russian trolls
Commentary: Numerous reports say The Last Jedi negativity was influenced by Russian bots and trolls, but the reality is far different.
If you've spent any time online, you've no doubt heard the news: Research shows that a huge majority of fan hostility toward the oft-maligned The Last Jedi was the work of Russian bots and trolls.
In this day and age, embroiled as we are in the "culture wars", it's tempting to take this news at face value. To scream voicelessly into the social media void.
But when you break down the numbers and look deeper, the reality is stunningly different: Star Wars does not have a Russian troll problem and the fan backlash is far more complex than the reports might have you believe.
If you're not up to speed, a prepublication research paper, written by Morten Bay at the University of Southern California, analyzed the sentiment toward The Last Jedi by studying tweets directed at Rian Johnson between the release of the film on Dec. 13, 2017 and July 20, 2018.
Bay is a research fellow at the Center for the Digital Future at USC and has previously written on social media policy in the Trump era, Russian bots, fake news, journalism in social media and social media ethics. He is also a Star Wars fan.
It's safe to say The Last Jedi is one of the most divisive films of the past 12 months. The discourse has been vitriolic -- harassment campaigns saw Kelly Marie Tran, who plays Rose Tico, leave social media and one group of fans petitioned to remake The Last Jedi altogether.
Research like this can be important in highlighting some of the issues with the politicization of social media and pop culture -- and how they may be used to influence political discourse. However, though the goals are noble, the research is not definitive -- a fact that seems to have been lost in the media firestorm it set off.
So, instead, let's dig deeper on the research.
One in 10
Bay's paper stated that "50.9% of those tweeting negatively" about The Last Jedi were likely to be politically motivated or not even human. This was the major takeaway for a panoply of media outlets that then stated most of the negative sentiment directed at The Last Jedi came from Russian trolls.
But that is not the truth -- or at least, it distorts the truth.
Director of The Last Jedi Rian Johnson.
Bay collected 1,273 tweets using Twitter's Advanced Search function, all of which were tweeted to Rian Johnson's account (@rianjohnson) over a seven-month period, post-release.
After "cleaning" the dataset, Bay finished with 967 tweets. He then "manually" determined whether a tweet was negative, positive or neutral. Ultimately, deciding the sentiment of any particular tweet was left up to Bay. To separate the negative tweets out even more, Bay would search the accounts with high activity for terms such as "Trump" or "SJW" to determine their political stance.
Of the 967 tweets analyzed, 206 expressed "a negative sentiment" toward the film and its director.
Of the 206 negative comments, 61 were real people reported to have a political agenda, 11 were bots and only 33 appeared to be trolls. Of those 33, just 16 appeared to possess characteristics consistent with Russian troll accounts. In reality, less than one in 10 tweets were from Russian trolls -- nowhere near the 50 percent being widely reported.
A less exciting story
There are a few issues here. The first is that Bay's collection method relies only on tweets directed at Rian Johnson. Other accounts related to the film, such as that of Luke Skywalker -- @HamillHimself -- who has almost triple the amount of followers and assumedly a far greater reach than Johnson, were not analyzed. This significantly limits the power of the analysis.
Notably, the research did in fact catch the eye of The Last Jedi director himself, causing him to remark "what the top-line describes is consistent with my experience online." Of course, it would be consistent with his experience online, because the research paper directly used Johnson's tweets as the source of their data. It quite literally analyzed his experience online.
While it is true that half of the negative tweets (105 of the 206) were classified as politically motivated, bots, trolls or sock puppets, the reporting that ensued diluted this message and conflated all of these separate negative reactions together. This cherry-picking helps tell a story, but it muddies the waters.
None of that is to say this was Bay's intention. The 38 page research paper explains his methodology in-depth and with great clarity. He draws conclusions based on the dataset he has acquired and even states that it has "limitations" and the study is of a "less-than-comprehensive nature". In his conclusion, he remarks that the assertions made within his paper must be considered only within the "limited scope of the data set".
He knows that makes for a less exciting story.
"Having worked as a journalist for many years, I know how the game works," he explains.
Bay is "moderately disappointed in some of the major media brands" that ran articles without taking the time to get a little deeper. He understands that some of his findings were buried because they produce a less enticing lede than "Russian Troll Army invading the Star Wars galaxy."
That's just not what his research suggests.
"The suspected Russian trolls are so few that it is basically the normal amount of Russian trolls you would expect to be present in a high-profile online debate."
A hive of scum and villainy
I'm not here to tell you that the reaction to The Last Jedi did not contain a myriad of trolls, bullies or bots. It's plain for all to see that the Star Wars fandom continues to tear itself in two online, trading barbs, almost a year after The Last Jedi was released.
That conversation has fused with alt-right politics, diversity in the media and Trump-era social media discourse. A very small minority of the fandom was involved in that conversation, at least on Twitter in this one specific instance, appears to be artificially constructed.
More importantly, it is impossible to make generalized, sweeping statements about the state of the Star Wars discourse from this one particular study. It is, by Bay's own admissions, limited in the conclusions that it can draw. It is hampered by a proportionately small sample size and prone to bias. It's misleading to say "half" when not every single Star Wars "hater" was used in the dataset.
And to be clear, it's not bad research. But the real story, according to Bay, shows that "American political activists have started using the same tactics as the Russians to insert themselves into any sort of debate on social media where there's a rift that can be widened."
His work confirms the idea that pop culture debates on social media can be politicized and potentially used for strategic purposes. He also suggests that pop culture fandoms are another place to look when trying to decipher how political messaging can be propagated online.
The truth is simple: The majority of people who read about Bay's research will never go and read the study from top-to-bottom.
The information will be presented to them and ingested through second-hand osmosis. Via a tweet, via outlets trying to capture a snapshot of the research that most appeals to their audience. There's no hard analysis of the methods, no interrogation of the dataset. And that approach only serves to inflame the discourse even more.
Ultimately, it is ironic that in an era when fake news and misinformation are so rife, Bay's study found widespread appeal via a media-driven narrative that was far, far away from the truth.
iHate: CNET looks at how intolerance is taking over the internet.
Infowars and Silicon Valley: Everything you need to know about the tech industry's free speech debate.