LOS ANGELES: Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films.
AI may be about to change all that.
Start-up Flawless AI, co-founded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.
The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.
Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film "Heist", starring Robert De Niro.
"I hate dubbing as it stands," he told Reuters. "You have to change so many things to try and catch sync. You're changing words filmmakers and performers have thought about so deeply. They're thrown out to find a different word that fits, but it never really does."
Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.
The tool worked to translate a De Niro scene in "Heist".
"It's able to essentially take an 'ooh' sound from De Niro 20 minutes earlier and place that in a different moment in the film," explained Mann. "It measures at the same time and blends it so that the performance is the same but it's a different mouth movement."
Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.
Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.
"From a filmmaking point of view, you're going to see the rise of a much more diverse range of stars," he said.
There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.
"When it's a common thing, we're not going to think about it anymore and we'll just enjoy the movies," he said.
Reuters
Thu May 20 2021
The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French. Filepic/REUTERS
COP29 climate summit draft proposes rich countries pay $250 billion per year
The draft finance deal criticised by both developed and developing nations.
Bomb squad sent to London's Gatwick Airport after terminal evacuation
This was following the discovery of a suspected prohibited item in luggage.
Kelantan urges caution amidst northeast monsoon rains
Kelantan has reminded the public in the state to refrain from outdoor activities with the arrival of the Northeast Monsoon season.
Former New Zealand PM Jacinda Ardern receives UN leadership award
Former New Zealand prime minister Jacinda Ardern was given a global leadership award by the United Nations Foundation.
ICC'S arrest warrants for Netanyahu, Gallant an apt decision - PM
The decision of the ICC to issue arrest warrants against Benjamin Netanyahu and Yoav Gallant is apt, said Datuk Seri Anwar Ibrahim.
KTMB provides two additional ETS trains for Christmas, school holidays
KTMB will provide two additional ETS trains for the KL Sentral-Padang Besar route and return trips in conjunction with the holidays.
BNM'S international reserves rise to USD118 bil as at Nov 15, 2024
Malaysia's international reserves rose to US$118.0 billion as at Nov 15, 2024, up from US$117.6 billion on Oct 30, 2024.
Findings by dark energy researchers back Einstein's conception of gravity
The findings announced are part of a years-long study of the history of the cosmos focusing upon dark energy.
NRES responds to Rimbawatch press release on COP29
The Ministry of Natural Resources and Environmental Sustainability (NRES) wishes to offer the following clarifications to the issues raised.
Online Safety Bill and Anti-Cyberbullying Laws must carefully balance rights and protections
The Online Safety Advocacy Group (OSAG) stands united with people in Malaysia in the fight against serious online harms.
Malaysia's inflation at 1.9 pct in Oct 2024 - DOSM
Malaysia's inflation rate for October 2024 has increased to 1.9 per cent, up from 1.8 per cent in September this year.
Saudi Arabia showcases Vision 2030 goals at Airshow China 2024
For the first time, Saudi Arabia is participating in the China International Aviation & Aerospace Exhibition held recently in Zhuhai.
King Charles' coronation cost GBP 71mil, govt accounts show
The coronation of Britain's King Charles cost taxpayers GBP72 million (US$90 million), official accounts have revealed.
Couple and associate charged with trafficking 51.9 kg of meth
A married couple and a man were charged in the Magistrate's Court here today with trafficking 51.974 kilogrammes of Methamphetamine.
PDRM to consult AGC in completing Teoh Beng Hock investigation
The police may seek new testimony from existing witnesses for additional insights into the investigation of Teoh Beng Hock's death.
Thai court rejects petition over ex-PM Thaksin's political influence
Thailand's Constitutional Court rejects a petition seeking to stop Thaksin Shinawatra from interfering in the running the Pheu Thai party.
Abidin takes oath of office as Sungai Bakap assemblyman
The State Assemblyman for Sungai Bakap, Abidin Ismail, was sworn in today at the State Assembly building, Lebuh Light.
UPNM cadet officer charged with injuring junior, stomping on him with spike boots
A cadet officer at UPNM pleaded not guilty to a charge of injuring his junior by stomping on the victim's stomach with spike boots.
How Indian billionaire Gautam Adani's alleged bribery scheme took off and unraveled
The indictment was unsealed on Nov. 20, prompting a $27 billion plunge in Adani Group companies' market value.
Elon Musk blasts Australia's planned ban on social media for children
Several countries have already vowed to curb social media use by children through legislation, but Australia's policy could become one of the most stringent.