The Barisan Nasional alliance will be in opposition, incumbent prime minister Ismail Sabri Yaakob said on Twitter.
It is unclear who will form the government, as without Barisan's support, neither candidate can reach the simple majority required.
The decision could now be made by King Al-Sultan Abdullah, who had given political parties until 2 p.m. (0600 GMT) on Tuesday to put together alliances needed for a majority.
The deadline passed without any immediate announcement from the palace and the PM candidates.
No political bloc won the majority on its own in the Saturday election.
The election and the ensuing turmoil prolongs political instability in the Southeast Asian nation, which has had three prime ministers in as many years, and risks delays to policy decisions needed to galvanise an economic recovery.
The uncertainty hit the Kuala Lumpur stock market .KLSE, which fell for a second day on Tuesday. Significant election gains by an Islamist party added to investors' fears, notably over policies on gambling and alcohol consumption.
Its electoral gains have also raised fears in multicultural Malaysia, which has significant ethnic Chinese and ethnic Indian minorities following other faiths. The PAS Islamist party has advocated for sharia law.
Malaysian police cautioned the country's social media users to refrain from posting "provocative" content on race and religion after the divisive election.
Anwar's progressive coalition entered negotiations with Barisan Nasional, Anwar's longtime rival, on Monday to discuss a potential alliance.
Barisan, Malaysia's dominant political force governed since independence from the British in 1957 until 2018 when it was ousted over widespread corruption allegations.
Muhyiddin's conservative Malay Muslim alliance - which includes the Islamist party - reiterated on Monday that he had majority support, though he did not identify his backers.
Anwar's coalition won the most seats in the Saturday election with 82, while Muhyiddin's bloc won 73. They need 112 - a simple majority - to form a government.
Barisan won only 30 seats - its worst electoral performance - but will play a pivotal role in deciding who forms the government as its support is needed for both Anwar and Muhyiddin to get to 112.
It would be another astonishing turnaround in Malaysian politics if Anwar and Barisan forged an alliance: as opposition chief, Anwar has spent much of his career trying to oust Barisan.
For the 2018 election, Anwar tied up with mentor-turned-foe Mahathir Mohamad to defeat Barisan. But their alliance collapsed in 22 months, and the two have since fallen out again.
