Longtime opposition leader Anwar Ibrahim and former Prime Minister Muhyiddin Yassin each said they could form a government with support from other parties, whom they did not identify.
A record number of Malaysians voted, hoping to end a spell of political uncertainty at a time of slowing economic growth and rising inflation in the Southeast Asian nation, which has had three prime ministers in as many years.
The instability reflects a political transformation in a country that has for decades been one of the most stable in a region that has had its share of military coups, violent political upheavals, and insurgencies.
Saturday's election saw the further decline of a force that dominated politics from independence in 1957 until 2018, as well as gains by an Islamist party that has called for sharia law.
Forming a government may require the involvement of Malaysia's king, whose largely ceremonial role includes the power to appoint as prime minister a lawmaker he believes will command a majority when no coalition can do so on its own.
The palace on Sunday instructed the parties to each present the name of a lawmaker it thinks has majority by 2 p.m. (0600 GMT) on Monday.
Anwar's Pakatan Harapan coalition won 82 lower house seats, short of the 112 majority but slightly ahead of Muhyiddin Yassin's alliance with 73.
Prime Minister Ismail Sabri Yaakob's Barisan Nasional alliance - whose United Malays National Organisation (UMNO) had long been Malaysia's dominant political force - suffered its worst electoral defeat ever, winning just 30 of the 178 seats it vied for.
As voters rejected UMNO and the multi-ethnic Barisan coalition it leads for the second election in a row, Muhyiddin's Perikatan Nasional coalition, in its first national contest, pulled support from Barisan's traditional base.
'MORE DIVIDED'
"I think what we learned here is that the country is more divided," said Asrul Hadi Abdullah Sani, deputy managing director at political risk consultancy BowerGroupAsia.
"With Perikatan Nasional making inroads into UMNO's vote bank, it shows that there are three legitimate coalitions in the future of Malaysian politics."
Fighting against UMNO, which he had once led, Mahathir Mohamad, 97 and Malaysia's longest-serving premier, lost his seat in his first election defeat in 53 years.
A key winner in the election was the Islamist PAS party in the Perikatan coalition, securing the largest number of seats of any single party. Race and religion are divisive issues in Malaysia, where the mostly Muslim Malays are the majority of the population, with sizable ethnic Chinese and Indian minorities.
Smaller political blocs based in Sabah and Sarawak states on Borneo island, which have sought greater autonomy, could potentially play decisive roles if they aligned with either coalition.
Muhyiddin said he had met with Sarawak's state leader and was in discussions with other parties on forming a government.
If Anwar should become premier, it would be a remarkable transformation for a politician who in 25 years has gone from heir apparent of Mahathir to a prisoner convicted of sodomy to the leading opposition figure. He says the sodomy and corruption charges were politically motivated.
After release from prison in 2018, he joined Muhyiddin and mentor-turned-foe-turned-ally Mahathir Mohamad to defeat Barisan, ending its six-decade rule, amid public anger at the government over the multibillion-dollar 1MDB scandal.
That coalition collapsed after 22 months due to infighting over a promise by Mahathir to hand the premiership to Anwar.
Muhyiddin briefly became premier, but his administration collapsed last year, paving the way for Barisan's return to power with Ismail at the helm.
