LOS ANGELES: Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films.
AI may be about to change all that.
Start-up Flawless AI, co-founded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.
The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.
Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film "Heist", starring Robert De Niro.
"I hate dubbing as it stands," he told Reuters. "You have to change so many things to try and catch sync. You're changing words filmmakers and performers have thought about so deeply. They're thrown out to find a different word that fits, but it never really does."
Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.
The tool worked to translate a De Niro scene in "Heist".
"It's able to essentially take an 'ooh' sound from De Niro 20 minutes earlier and place that in a different moment in the film," explained Mann. "It measures at the same time and blends it so that the performance is the same but it's a different mouth movement."
Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.
Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.
"From a filmmaking point of view, you're going to see the rise of a much more diverse range of stars," he said.
There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.
"When it's a common thing, we're not going to think about it anymore and we'll just enjoy the movies," he said.
Reuters
Thu May 20 2021

The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French. Filepic/REUTERS

China promises Latin America 'trustworthy' ties as Trump lays out demands
Beijing looks to improve its foothold in a region historically under the US sphere of influence.

Major risks loom as Trump upends US Russia policy
Washington's approach changes dramatically with the initial meeting between US and Russian negotiators.

Pope Francis has double pneumonia, complicating his treatment, Vatican says
Doctors say a polymicrobial infection occurs when two or more micro-organisms are involved and can be caused by bacteria, viruses, or fungi.

Philippines' VP Duterte challenges impeachment before Supreme Court
Sara Duterte makes her first legal move to fight the case that could lead to her removal and lead to a lifetime ban from public office.

Motor racing - Hamilton steps out in red at unprecedented 10 team F1 launch
I feel so full of life, so much energy, because everything's new and I'm just focused on what is up ahead, Hamilton says.

UAE tells US' Rubio it rejects displacement of Palestinians
The UAE's stance matters as its one of four Arab nations that normalised ties with Israel during Trumps first term.

Taylor Swift wins recording artist of the year crown for fifth time
The megastar's "The Tortured Poets Department" record also topped four IFPI charts, the IFPI says.

More than $50 billion needed to rebuild Gaza, World Bank joint assessment says
Years of rebuilding work, including clearing unexploded ordinance and millions of tonnes of rubble lie ahead.

Malaysia’s AI data centre development to be impacted by recent US restrictions - Liew
Liew Chin Tong says the restrictions would not affect the development of the domestic data centre ecosystem as a whole.

Sweden's top court rejects Greta Thunberg lawsuit on climate action
A court cannot decide that the parliament or the government must take any specific action, the Supreme Court says.
![[COLUMNIST] Trump’s Gaza (pipe) dream [COLUMNIST] Trump’s Gaza (pipe) dream](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-10/51728873693_EMIR.jpg)
[COLUMNIST] Trump’s Gaza (pipe) dream
The Gaza plan is dangerous and counter-productive as it's merely another recipe for regional instability.

Sivasangari eyes new career-high in world rankings
After climbing to a career-best world number seven, national women's squash ace S. Sivasangari is determined to keep rising.

Outburst against Steven Sim: DAP will probe if there's a complaint - Loke
DAP secretary-general Anthony Loke says the disciplinary committee has not received any reports related to the incident so far.

US judge will not block Elon Musk from firing federal workers, accessing data
The case raises questions about Musk's apparent unchecked authority as a top deputy to President Donald Trump.

Dozens of false killer whales stranded on remote Australian beach
Rescue crews trying to save the majority of the large dolphins that survived the ordeal, according to authorities.

Trump restricts AP access over Gulf of Mexico issue
The AP, citing editorial standards, says it would continue to use the gulf's established name.

China to fly back 200 nationals rescued from Myanmar scam centres, Thai official says
This is part of a multinational effort to crack down on illegal online operations run from the region.

Somber Tiger Woods back in action in TGL loss to New York GC
Tiger Woods makes his first start since the death of his mother when he competed in a TGL match at Palm Beach Gardens.

Brazil's ex-President Bolsonaro charged in alleged coup plot
A total of 34 people were charged, including several military officials.

Trump and Musk target tax enforcers, rocket scientists, bank regulators for job cuts
Concerns grow over transparency, oversight, and Musk's influence on government operations.