Washington has declared the beheading of an American journalist a "terrorist attack", upping the stakes in its confrontation with the Islamic State (IS) jihadist group, as Shiite militiamen gunned down 70 people at an Iraqi Sunni mosque.

The apparent revenge attack at the mosque in Diyala province Friday will increase already significant anger among Iraq's Sunni Arab minority with the Shiite-led government, undermining an anti-fighters drive that requires Sunni cooperation to succeed.

It came as the US, which is carrying out airstrikes against IS, ramped up its rhetoric over the grisly killing of journalist James Foley, which was carried out by the jihadist group and shown in a video posted online.

In Washington, Deputy National Security Advisor Ben Rhodes said the beheading of Foley "represents a terrorist attack against our country".

Rhodes also said that paying ransoms to free hostages is "not the right policy", confirming Washington's long-standing position amid claims from IS that other countries had paid to have their nationals freed.

In an unanimous statement Friday, the UN Security Council strongly condemned Foley's murder as "heinous and cowardly".

'We found a massacre'

Army and police officers said the attack on the Musab bin Omair Mosque in Diyala Friday came after Shiite militiamen were killed in clashes, while other sources said it followed a roadside bomb near one of their patrols.

Doctors and the officers put the toll from the attack, in which worshippers were sprayed with machine gun fire, at 70 dead and 20 wounded.

Two officers had earlier blamed IS for the attack, but the preponderance of accounts point to Shiite militiamen.

The government turned to militiamen to bolster its flagging forces during the IS offensive, sparking a resurgence of groups involved in brutal sectarian killings in past years that will be difficult to dislodge.

Ibrahim Aziz Ali, whose 25-year-old nephew was among those killed, told AFP he and other residents heard gunfire and rushed to the mosque, where they were fired on by snipers.

"We found a massacre" at the mosque, he said.

Five vehicles with images of revered Shiite Imam Hussein were parked at the mosque, Ali said, adding that residents clashed with the militiamen who withdrew when the Iraqi army arrived.

Iraqi premier designate Haidar al-Abadi issued a statement calling for unity and condemning the killings, which may complicate the already-contentious process of forming the country's next government.

US Vice President Joe Biden said Friday that Washington would back a federal system in Iraq.

Writing in a Washington Post opinion piece, Biden pointed to "functioning federalism" as an approach to breach the divisions in the country.

Biden is a longtime supporter of the plan under which Iraq would be divided into three semi-autonomous regions for Shiites, Sunnis and Kurds, respectively.

'Beyond anything we have seen'

Biden also said the US was prepared to "further enhance" its support of Iraq's fight against IS.

The United States began an air campaign against IS in Iraq on April 8, and has since conducted 93 air strikes, including three against militants in the area of the Mosul dam, the country's largest, on Friday.

Pentagon chiefs warned of the dangers of IS and said operations against it in Syria may be needed.

"They marry ideology and a sophistication of strategic and tactical military prowess," US Defence Secretary Chuck Hagel.

"This is beyond anything we have seen."

General Martin Dempsey, chairman of the Joint Chiefs of Staff, spoke of a "very long contest" that could be won only with regional support and that of "the 20 million disenfranchised Sunnis that happen to reside between Damascus and Baghdad".

He was referring to the alienation of many Sunni Arab Muslims from Iraq's government and the Alawite-dominated regime in Syria.

US Secretary of State John Kerry, in a telephone call to Iraqi counterpart Hoshyar Zebari, offered his condolences for the "countless" Iraqis who have fallen victim to IS.

Both leaders "recognized that Iraq is on the front line in the war against ISIL and that Iraq, the United States, the region, and the international community must stand together to face this threat", according to the US State Department.

'We don't pay ransoms'

Foley's killing has stoked Western fears that territory seized by the militants in Syria and Iraq could become a launchpad for a new round of global terror attacks.

The US State Department estimates that about 12,000 foreign fighters from at least 50 countries are in Syria.

Foley, a 40-year-old freelance journalist, was kidnapped in northern Syria in November 2012. His employer GlobalPost said his captors had demanded a 100-million-euro ($132-million) ransom.

GlobalPost CEO Philip Balboni said his team had never taken the demand seriously, and US State Department deputy spokeswoman Marie Harf insisted: "We do not pay ransoms."

His captors had also sent Foley's family a taunting and rambling email threatening to kill him.

In the execution video, released online, a black-clad militant said Foley was killed to avenge US air strikes against IS.

The man, speaking with a clear south London accent, paraded a second US reporter, Steven Sotloff, in front of the camera and said he too would die unless President Barack Obama changes course.