Rescuers were searching for 40 Indonesians including women and children on Saturday after a boat carrying them home to celebrate the end of the Islamic fasting month sank off Malaysia.

Two ships, four speedboats and two helicopters have been dispatched to scour the sea off southern Johor state to look for the missing, said Amran Daud, an official with the Malaysian Maritime Enforcement Agency (MMEA).

The boat, thought to be carrying 44 passengers, sank amid high waves Thursday night around 13 nautical miles (24 kilometres) off the coast, he said, adding four were rescued on Friday.

It is thought the Indonesians chose to travel on the boat because they were working illegally in Malaysia and wished to bypass border controls on their trip home.

The boat sank roughly three hours into its journey from Tanjung Sedili on the state's east coast to Indonesia's Batam island with its passengers hoping to return for Eid al-Fitr, the most important Muslim holiday, Amran said.

"The condition of the boat was believed to be questionable," Amran said, adding that authorities were still investigating the cause of the incident.

Three of the survivors were rescued by passing fishermen, while another was saved by authorities who started search and rescue efforts after being alerted by the fisherman.

"Only four of those on board were rescued by fishermen and MMEA after floating 15 hours in the sea," Amran said.

Another maritime official, Hairi Nizam, added those rescued were found clinging on to plastic drums in the water. None of them were wearing life jackets.

After being sent to hospital for treatment, the four men, aged between 26 and 31, were now in custody while the incident is being investigated, Hairi said.

Boating accidents off Malaysia's coast are common as thousands of people from poorer regional neighbours, such as Indonesia and Myanmar, risk journeys in flimsy boats to work illegally in the relatively affluent country.

They fill low-paying jobs shunned by locals on plantations, construction sites and in factories. Staying in Malaysia illegally is punishable by jail and caning for men.

Many Indonesians try to leave the country during Ramadan to return home to celebrate Eid al-Fitr with their families. The holiday begins next Thursday.

Authorities said last month they were beefing up patrols to prevent illegal immigrants from making the sea voyage across the waterway separating Malaysia and Indonesia.

Ferries can reach Batam island from Johor within 1.5 hours.

An Indonesian woman died and seven people went missing after their wooden boat overturned in mid-July, also off Johor state and heading to Batam island.

The boat, which was smuggling them out of the country, suffered an engine failure. Twenty-seven people were rescued.