The move brings to light another of the multiple disputes in the strategic waterway, most of which China claims sovereignty over, with Beijing involved in frequent altercations with the Philippines and sporadic rows with Vietnam.
The most contested features are around the Spratly archipelago, where China, Taiwan, Malaysia, Vietnam, Brunei and the Philippines all have various claims and degrees of occupation.
Malaysia's letter was sent to Vietnam's foreign ministry in early October but has so far received no reply, the two officials said, declining to be identified more precisely because the matter was sensitive.
The complaint was over Vietnam's alleged artificial expansion of the Barque Canada Reef, an islet in the Spratlys where Vietnam has built numerous infrastructure, according to satellite images analysed by the Center for Strategic and International Studies, a Washington-based think tank, which were released last month.
In late October Radio Free Asia reported that Vietnam was also building an airstrip on the reef.
Vietnam's foreign ministry did not reply to a request for comment. Malaysia's foreign ministry did not comment.
The letter preceded those publications and only criticised the enlargement of the islet, not the building of infrastructure, one of the officials said.
The tiny Spratly islands have seen significant construction in recent years as countries seek to bolster their territorial claims and prove they can sustain human habitation on dozens of islets and features.
China's activities have attracted the most attention, with seven islands built on submerged reefs, some equipped with runways, docks, control towers and missile batteries.
Though complaints between Malaysia and Vietnam over territory are rare, Malaysia has taken issue regularly over what it says is encroachment by Vietnamese fishermen into its Exclusive Economic Zone, leading to the arrest of some crew.
