South Korea will hold local elections on June 3, and for the first time will enforce two laws aiming to curb the use of AI deepfakes to support political campaigns. The big question is: will it be enough?
While not the most dramatic issue facing voters, deepfakes remain a problem for elections worldwide. In 2024, some New Hampshire voters received a robocall claiming to be from then-US President Joe Biden asking residents not to vote in the state primary. South Korea, meanwhile, has faced everything from fake videos of political candidates on social media to AI-generated television news reports and more.
Leading up to South Korea’s presidential election next year, a video was widely shared which falsely showed President Lee Jae-myung ending his hunger strike. At the time, he was not only the election frontrunner but also the opposition party leader.
Deepfakes have been around for years, but their quality and access have increased alongside the sophistication of generative AI.
Brian Long, CEO and co-founder of Adaptive Security, a firm that specializes in threat awareness training for things like phishing emails and deepfakes, tells Dark Reading that two years ago, generating a convincing deepfake required real technical skill.
“Today, consumer tools produce convincing audio, video, and synthetic text in minutes,” he says.
Long says the June 3 elections will be the first real stress test of South Korea’s full regulatory framework. Clear legal infrastructure, he says, allows investigators to act faster while giving platforms clearer obligations to remove violating content. The problem exists in the channels that regulators can’t clearly reach.
“Deepfakes distributed through encrypted messaging apps, targeted SMS campaigns, and direct voice calls move faster than any fact-checker. By the time a platform removes a clip, it has already reached the people it was designed to reach,” he says. “Laws address the supply side. Awareness addresses the demand side. Voters who approach political audio and video with the same skepticism they bring to a suspicious phone call are significantly harder to manipulate. Building that instinct at scale, before election season, is where most countries are still in early days.”
South Korea: A Unique Deepfake Test Case
South Korea is unique in that it has taken steps to restrict and penalize the use of deepfakes prior to elections at a codified scale. Article 82-8 of the Public Official Election Act, established in 2023, prohibits the use of virtual sounds, images, or videos that are difficult to distinguish from reality for 90 days prior to election day. This is specifically to prevent individuals or campaigns from generating deepfakes intended to influence the outcome of an election, such as content clearly labeled as generated or content created to encourage voting.
A person who violates Article 82-8 “shall be punished by imprisonment with labor for not more than 7 years or by a fine of not less than 10 million won but not more than 50 million won,” according to the law (approximately between $6,700 USD and $33,500 USD).
In addition to this, there’s also the AI Basic Act, established early last year and put into effect this January. While the legislation is substantial and aims to regulate many aspects of AI, one relevant portion comes in the form of Article 31, paragraph 3.
“Where AI systems generate virtual audio, images, or video that are difficult to distinguish from real content, AI business operators shall notify or indicate to users, in a manner that allows clear recognition, that such content has been generated by an AI system,” the law reads. “Provided, that where such outputs constitute, or form part of, artistic or creative works, such notification or indication may be made in a manner that does not interfere with their exhibition or enjoyment.”
The AI Basic Act also, broadly speaking, requires businesses generating deepfake AI content to make clear that content is AI generated, such as through a watermark. Any business that does not transparently disclose the use of deepfake content as described by the act can face an administrative fine not exceeding 30 million Korean won (approximately $20,000 USD).
To facilitate its investigations, South Korea’s National Police Agency (KNPA) developed a deepfake detection tool it deployed in 2024.
It’s notable that a country is putting their best foot forward to combat deepfakes, especially South Korea, which has had to combat a surge in AI-generated sexually explicit content in recent years. A 2023 report from identity theft research firm Security Hero found that globally, “South Korean singers and actresses constitute 53% of the individuals featured in deepfake pornography and are the most commonly targeted group.”
Combined, this marks an early example of comprehensive AI law at the national level.
For comparison, in the US there is no comprehensive law explicitly regulating the use of deepfakes to influence elections, but multiple states such as California and Texas do. The FBI and Department of Justice can also investigate deepfakes under current laws related to election interference and identity theft.
Europe has the EU AI Act, which requires that users of an AI system that manipulates images, audio, or video that resembles a person, object, place, or entity in real life must disclose that the content has been artificially generated or manipulated.
Deepfake Regulations Just a Start
As a deterrent, results have thus been mixed. Leading up to its local election season in 2024, South Korea’s National Election Commission said it caught 129 deepfakes in violation of the election laws between Jan. 29 and Feb. 16. And this year, before local elections in June, reports suggest South Korean election officials are still actively fighting deepfakes on a day-to-day basis.
Dr. Hany Farid, co-founder at GetReal Security (a vendor that specializes in deepfake detection), says that South Korea deserves credit for moving faster than most governments. But he says regulation only sets a floor — not a ceiling — on solving the issue of deepfakes.
“The core issue is speed and asymmetry,” he says. “A realistic deepfake of a presidential or mayoral candidate can be produced using publicly available tools and can spread across the Internet within seconds. The legal machinery to detect it, investigate it, and remove it operates on timescales of days or weeks. That gap is where the damage is done. By the time an election commission confirms something is synthetic, it may have been viewed millions of times.”
Social platforms are difficult to police, and threat actors operating outside the country face no practical liability. While regulations are helpful, Farid adds, they’re not sufficient on their own. Organizations need a layered approach, such as ways of holding social media platforms accountable, using detection infrastructure (which South Korea does), and promoting media literacy.
Farid says the US is in a difficult position going into its midterms this fall, as the country has a patchwork of state laws that vary enormously in scope, enforcement, and constitutional durability.
To combat this, he says, the government must bake detection into the election security apparatus and hold platforms accountable for the content economics that make deepfake disinformation profitable to spread. The use of provenance tools can also authenticate where content originated from. Some companies like TikTok and Meta apply automatic AI markers to certain content according to standards established by the Coalition for Content Provenance and Authenticity (C2PA).
Adaptive Security’s Long says the US will have an opportunity to build on what South Korea has learned.
“A federal framework with clear timelines, labeling requirements, and enforceable penalties would give every jurisdiction consistent ground rules heading into the 2026 midterms,” the CEO says. “The work done now sets the conditions for what voters experience in November.”
