As Malaysians headed out to vote in a country that has seen three prime ministers in as many years, opinion polls were forecasting Anwar's alliance would take the most seats in parliament but fail to reach the majority needed to form a government.
But Muhyiddin's new alliance, which includes a Malay-centric conservative party and an Islamist party that has touted shariah law, made strong gains.
The other main contender - Prime Minister Ismail Sabri Yaakob's ruling Barisan Nasional coalition - was losing ground in traditional strongholds to Muhyiddin's bloc, the results showed.
As of 1630 GMT, the Election Commission had announced results for 123 of the 222 parliamentary seats. Muhyiddin's Perikatan Nasional alliance won 42 seats, while Anwar's multi-racial coalition won 36.
Perikatan was a junior partner in Ismail's coalition government. If the result is close, the two could come together again to block Anwar.
If Anwar clinches the top job, it would cap a remarkable journey for a politician who in 25 years has gone from heir apparent, to the premiership, to a prisoner convicted of sodomy, to the country's leading opposition figure.
Polls closed at 1000 GMT, with final results expected in the next few hours.
Malaysia's veteran leader Mahathir Mohamad, meanwhile, was dealt his first election defeat in 53 years in a blow that could mark the end of a seven-decade political career. He lost his seat to Muhyiddin's alliance.
PROGRESSIVE VS CONSERVATIVE
Oh Ei Sun, a senior fellow with Singapore's Institute of International Affairs, said the early results showed a polarisation between the progressive and conservative forces.
Race and religion are divisive issues in Malaysia, where the Muslim ethnic-Malay population make up the majority, while ethnic Chinese and Indians are the minorities.
"In order to form the next government, Anwar must win an outright majority. Otherwise, even if he wins the most number of seats, the Malay-Muslim coalitions will come together to form the next government," Oh said.
At least 70% of Malaysia's 21.1 million eligible voters had cast their ballots by 4 local Pacific time (0800 GMT), the Election Commission said. It has not given the final tally.
Voter turnout in the previous election was one of the highest at 82%, but given the bigger pool of voters in this poll, Saturday's turnout had already surpassed the prior election by nearly 2 million voters.
Higher turnouts typically tend to favour the opposition.
'HOPE FOR CHANGE'
Opinion polls showed significant numbers of undecided voters in the days before the vote.
Ismail said early on Saturday his coalition was targeting a simple majority, but would be open to working with others if it failed to do so. Anwar said he was cautiously confident.
The top issues are the economy, along with corruption, as several leaders from the Barisan Nasional coalition face graft accusations. Malaysians are also frustrated with the political instability, seen as hampering development efforts.
"I hope there's a change in the government," Ismat Abdul Rauf, a 64-year-old retiree, told Reuters. "There are many issues that need to be addressed - the economy, the wealth of the country, the people (guilty of) wrongdoing who are not being prosecuted."
Barisan, dominated by the United Malays National Organisation, governed for 60 years, from independence until 2018, while Perikatan is a new bloc that has emerged as a strong third force with Malay voters' backing.
Anwar was released from prison in 2018 after joining with old foe Mahathir and Muhyiddin to defeat Barisan for the first time in Malaysia's history, amid public anger at the government over the multibillion-dollar 1MDB scandal.
That coalition collapsed after 22 months in power due to infighting over a promise by Mahathir to hand the premiership to Anwar. Muhyiddin briefly became premier, but his administration collapsed last year, paving the way for Barisan's return to power with Ismail at the helm.
