At first glance, Misrad Sulejmanovic could be your typical YouTuber. The 25-year-old from Bosnia and Herzegovina has a following of about 3,300 subscribers on his channel. He goes by the handle Dinaric Wolf.
Sulejmanovic (not to be confused with a Bosnian soccer player of the same name) mostly posts reactions to videos by Geography Now, an educational channel with more than 2 million subscribers. The reaction videos follow the same format: The original Geography Now video plays in the main screen, while Sulejmanovic comments in a smaller screen at the bottom left. He sits at a desk alone in a dark room and chats into a microphone. He kids around and, like many YouTubers, occasionally rambles. In a recent video, he joked that he hadn't posted anything in a while because he'd been away fighting World War III, but nobody remembered because the army had to "neuralize" everyone. Nerdy humor you'll see all over YouTube.
To find Sulejmanovic's more widely seen work, however, you would've needed to look elsewhere on the platform.
Until last week, he was the face of a channel called, simply, Breaking News, which served up pro-Trump and conservative-leaning videos. A sampling of recent clips included: "BREAKING: Trump Just Made One Bold Move – Obama Must Scream"; "After Abysmal Super Tuesday...Bernie Sanders Admits Defeat"; and "They Did It! – Supreme Court Ends It For Dems."
Sulejmanovic's name wasn't anywhere on the channel. But in the videos, he was there at the bottom left corner of the screen, same picture-in-picture format as on his own channel, sitting alone at the same desk and speaking into the same mic. In every clip, he read a script that could've been written for a news anchor, albeit an obviously right-wing one. In some of the videos, the words "Breaking News" flashed on the screen, as if that segment had interrupted regularly scheduled programming. The clips had a bizarre aesthetic, halfway between formal news brief and casual vlog. "Hi, and welcome back to our YouTube channel," he said in perfect American English as each video began. Then he launched into the faux-news script.
Sulejmanovic, it turns out, had been an unwitting cog in a sprawling disinformation operation that appears to have stretched across at least four countries to generate ad revenue by posting false and divisive content aimed at American viewers, a CNET investigation, conducted in partnership with the Atlantic Council, has found. His videos also appeared on another channel, called News 24H. The two channels were just small parts of a larger group of more than a dozen channels with similar names, including American News Today, Breaking News 24-7 and Breaking Story, among others.
To find out more about the origin of the channels, CNET approached the Atlantic Council's Digital Forensic Research Lab, a nonpartisan organization based in Washington, DC. The lab also works with Facebook, as well as Google sister company Jigsaw, to combat and analyze disinformation. Most recently, it helped uncover a false campaign on Facebook by telecommunications companies in Vietnam and Myanmar aimed at discrediting telecom rivals.
The channels, which YouTube removed after CNET inquired about them, were likely designed to exploit the video platform's advertising program and take advantage of an American appetite for partisan content. Google, which owns YouTube, said its Threat Analysis Group, as well as YouTube's own teams, saw no evidence the channels were part of a foreign political influence operation. Instead, the company said, it was a spamming effort with channels operating out of different parts of the world, aimed at making money.
The strategy shows that the vastness of the internet does more than help disseminate false content widely. It also makes the creation of disinformation more efficient. The Breaking News channel and others that were taken down said they were based in the US, but contact information on a few of the pages, such as an email address and Twitter links, suggested at least some of the channels had ties to people in Vietnam. At least two of the channels used Fiverr, a Tel Aviv-based marketplace for freelancers, to find Sulejmanovic, the Bosnian voice-over actor. And the inauthentic content -- the term social media platforms use for posts containing false information or created by people who lie about their identity -- was uploaded to San Bruno, California-based YouTube, which distributed it across the planet. Think of it as the globalization of disinformation, touching platforms and people in far-flung corners of the world.
The operation demonstrates how easy and cheap it is to exploit the opportunities tech platforms offer and hook audiences on videos about hot-button issues. In several cases, the channels amplified content from fringe, right-wing websites, like Patriot Pulse and American Patriot Daily. The channels employed simple yet effective techniques to evade platform safeguards. And though tech giants like Google and Facebook often take the brunt of the blame, this operation makes clear that smaller players in the internet ecosystem, like Fiverr, can be enablers of the process.
The group of channels is far from alone. During the 2016 US presidential election, a Macedonian village turned false news into a cottage industry, using Facebook and Google to generate ad dollars. The Internet Research Agency, a Kremlin-linked troll farm, used Twitter and Facebook to post divisive messages that ran along America's racial fault lines. Just last week, Facebook and Twitter removed more than 200 accounts linked to Russia that operated out of Ghana and Nigeria.
When I spoke with Sulejmanovic last week while the channels were still active, he seemed unaware that Breaking News was creating disinformation. He thought he was simply doing a job. "I just get work," he said. "I get paid."
He said he's dispassionate about the political viewpoints expressed in the videos, even if they're hyperpartisan. He'll read a script as long as it isn't "racist or neo-Nazi shit." He continued, "But if it's Republican or Democrat, that's fine for me."
After we spoke, Sulejmanovic appeared to have a change of heart, emailing me to say he wouldn't work for Breaking News anymore now that he knew the channel is inauthentic. "I honestly just thought it was Republican news," he wrote. "But if it could be considered as fake news then I want nothing to do with it."
This week, YouTube took down at least 20 of the channels. "Upon review, our teams determined that this is spam behavior emanating from channels operating out of several regions," Ivy Choi, a YouTube spokeswoman, said in a statement. "We've taken action against these accounts and will continue our work to remove spam from the platform."
Social media companies are still under fire for the way they handled the last election cycle. As the 2020 primary season rolls on in the states, Silicon Valley's giants face intense scrutiny to ensure their platforms aren't abused again for election interference. Ahead of the Iowa caucuses last month, YouTube spelled out new policies regarding deepfake videos and other forms of sophisticated disinformation. The video service said it would take down "technically manipulated or doctored" videos, as well as content that attempts to mislead people about voting or census issues, such as when and where to vote.
Experts worry that domestic propagandists have absorbed the lessons learned four years ago. The threat, they say, is that disinformation is moving from foreign to domestic sources. These could be media pundits who don't lie about their identity but spread false and misleading information that goes viral online.
The Breaking News channel and the others that were taken down, however, show that exploitation methods similar to those of the Macedonian teens still thrive online. Some headlines from the channels were flat-out false, while others were just sensational and intentionally misleading. None of the viewpoints were extremist. Instead, the videos were engineered to inflame existing political tensions for profit. If your politics lean right, you might've found yourself nodding in agreement. If they lean left, you might've shaken your head. The videos often used Trump's nicknames for his political opponents. Calling out the Democrats and their partners in "the fake news media" was a familiar refrain in the clips. But above all, they were designed to rack up viewership.
"The amplification of divisive, toxic information is very effective," said Gideon Blocq, CEO of VineSight, a company that uses artificial intelligence to track the spread of viral disinformation. "If you have a topic that's already divisive and happens to be true, and you can use it to amplify inauthentic activity, that strategy is used a lot."
YouTube classified the network as spam, which is one of the most common policy violations on the platform. Last quarter, YouTube removed 3 million videos and nearly 2 million channels for breaking spamming rules, the company said.
In terms of reach, the Breaking News channel and its counterparts weren't huge. Most videos with Sulejmanovic's voice-overs got between 5,000 and 40,000 views. It's unclear how much revenue they may have generated. Still, Breaking News had advertisers including Intuit's TurboTax; VRBO, a vacation rental company; GoDaddy, a web domain service; Polaris, an off-road vehicle maker; and MasterClass, an online education provider. Ads supporting Bernie Sanders, the Vermont senator trying to gain the Democratic Party nomination, also appeared before some videos about his campaign.
Before YouTube took down the channel, a MasterClass representative said that the ad placement wasn't "intentional" and that the company was working to remove it. The other companies, as well as the Sanders campaign, didn't respond to requests for comment. YouTube said it issues credits to advertisers when it finds spam that wasn't detected by the company's automated systems.
At least one, but not all, of the channels in the group appeared to operate out of Vietnam. In one case, an email address associated with the Breaking News channel belonged to a person in Ho Chi Minh City, according to a resume tracked down by Kanishk Karan, a research associate at the Atlantic Council's Digital Forensic Research Lab. Karan worked with Iain Robertson, the lab's deputy managing editor, to analyze the channels. The resume said that person's previous job experience included translating documents from English to Vietnamese, as well as creating Facebook and Google pages for marketing purposes. The person's phone number is operated through Viettel, a Vietnamese telecom company.
When I called the number at 9 a.m. in Ho Chi Minh, a woman answered in fluent, if heavily accented, English. "I'm walking now," she said. "Can you email me?" Then she gave me an email address that matched the one Karan had identified. A few minutes later, she replied to an email I had earlier sent, saying she wasn't involved with the Breaking News YouTube channel. "There is a mistake," she wrote. She confirmed, though, that she uses the email address listed in the channel's About section, but said she didn't know how it got there.
Other clues linked the channels to Vietnam. Videos on the News 24H channel, one of the pages that features Sulejmanovic's voice-overs, were all about right-wing American politics, except two of the three oldest videos on the page. In those clips, from December, a woman sat in front of the camera speaking Vietnamese. In each of the videos, she made small talk for two hours, noting the time of day or what fruit she was eating.
On several of the channels, the "Featured Channels" label on the right-hand side of the About section was listed in Vietnamese. Channel creators can change that language setting. A Twitter account listed in the About section of the American News Today channel also tweets out video links in Vietnamese.
I first learned about the channels after watching a video by Destin Sandlin, a popular YouTuber who runs the science channel SmarterEveryDay. In March 2019, he posted about a bizarre set of videos on YouTube that all had the same title: "After Trump sends note to Ginsberg." The clips chide the "liberal media" for hiding something about Supreme Court Justice Ruth Bader Ginsberg's health and suggest the justice will retire. The videos were meant to look like the kinds of legitimate news videos that were created to be shared on social news feeds, popularized by sites like Upworthy or Mic.com.
But the clips contained some bright red flags: They all used the same script, read by a robotic voice. The grammar was wrong, and the videos used clip art rather than professional graphics. Essentially, they were all the same video, each slightly tweaked and the pitch of the robot voice altered every time. One version contained a rotating globe graphic. The videos zoomed in on photographs at different speeds. The tiny differences, however, served a broader purpose.
The subtle differences were meant to fool YouTube's AI tools, which scan for and remove banned content. YouTube's software is trained to take down duplicates of videos that aren't allowed on the site. But the minor changes in each video allowed the clips to slip past YouTube's filters. (The videos appear to have been taken down since Sandlin's post.)
Eventually, the channels changed their strategy. Instead of using robotic speech reader software to recite the script, the channel started using a voice-over actor. In one of the videos, as the actor read the script, I noticed a Fiverr logo appear at the top right corner of the screen. I scanned Fiverr's marketplace and found Sulejmanovic's profile, where he also goes by the handle dinaricwolf. His account sports a "Fiverr's Choice" badge, an endorsement by the platform based on "quality and delivery."
"The act of deception is not taken lightly in our marketplace," Fiverr spokeswoman Holly Steffy said in a statement. "When a seller makes a report about a request that breaches our terms, we have an in-house Marketplace Integrity team who work constantly to investigate and take the appropriate action."
The account that solicited Sulejmanovic's services, which went by the handle Ngquyt, is no longer on the platform. Steffy declined to answer specific questions about the account, citing privacy concerns.
This isn't the first time Fiverr, which went public on the New York Stock Exchange in June, has played a supporting role in an online controversy. In 2017, PewDiePie, the popular Swedish YouTuber, hired two men from the platform to unfurl a banner that read "Death to All Jews." Two years before that, Amazon sued more than 1,100 freelancers on Fiverr for offering their services to write fake five-star product reviews.
The American News Today channel posted videos with the same script as Sulejmanovic's voice-overs, but read by a different actor who remained off camera. For example, on Nov. 19, both Breaking News and American News Today published videos titled "Adam Schiff is going to want to go into hiding when he sees the results of this poll." The script matches the text of an article on a website called the Patriot Pulse, which bills itself as "the true voice of the American right" and is based in Smithfield, Virginia.
It's unclear how many channels were working in concert, though others appear to be using the same tactics. Another video with Sulejmanovic's voice-over, titled "Joe Biden Is Now Under Criminal Investigation for One Corrupt Ultimatum," was similar to a video on a channel called US Flash News Report, with the same title. That video, though, uses a different person on-camera and slightly different wording, but some of the same phrases. Sulejmanovic's script matches a story from a right-wing website called American Patriot Daily.
Patriot Pulse and American Patriot Daily didn't respond to requests for comment.
Adding a voice-over actor to the mix makes for an awkward-looking video but helps to exploit the YouTube Partner Program, which allows video creators to earn advertising revenue, said Karan, the Atlantic Council's research associate. Using a live actor on camera -- instead of the previous slideshow format -- could make a more "presentable case" to both YouTube and viewers that the content is legitimate, he said. The strategy gives us a glimpse at how bad actors could be evolving their methods to outsmart tech platforms' monetization policies, as well as escape their safeguards.
When I spoke to Sulejmanovic over Skype last week, he sat by himself in the same dark room I've seen in dozens of videos. He was polite and helpful.
He lives in Bosnia and Herzegovina, but grew up in Virginia, where he lived for 11 years. Because of Fiverr's privacy policies, he doesn't know the names of the people behind the channel. All Sulejmanovic knew, he said, was that the client was listed as being in the US. He emphasized that he does lots of voice-over work for other YouTubers and audiobook authors, not just those right-wing channels.
Sulejmanovic said he started doing the Breaking News videos last summer, though he couldn't remember exactly when. He didn't know how many videos he'd done, but said he got a new order after every 100 videos. He also wouldn't say how much money he made; his Fiverr account says his services start at $5, a common price on the platform. After he quit, the Breaking News channel found other voice-over actors to read its scripts.
I asked Sulejmanovic why his handle is Dinaric Wolf. The first part is a tribute to the Dinaric Alps, the mountain range that stretches from Italy, through Bosnia and Herzegovina, and down to Albania. The wolf part, he said, is because, "When I work alone, I work best."
Sulejmanovic thought of himself as a lone wolf, just a random freelancer to be found on Fiverr. It turned out he was part of something much bigger. The disinformation network he unknowingly helped bring to life spanned the world far beyond him, apparently from Vietnam to California.
He may have not realized it, but he was never working alone. ●