Murad Halimmuddin, 49, along with his son Abu Daud, 25, pleaded guilty to terrorism-related charges in the high court, said prosecutor Shukor Abu Bakar. Daud was jailed for 12 years.
"The court was told that Murad had returned from Syria last year," Shukor told AFP, adding he was spiritual leader of a group called "Fisabilillah".
Murad and his son "had admitted in court that they planned to kidnap politicians, raid army camps for weapons and overthrow the government," he said.
All the suspects were arrested by police in April.
"They planned to launch jihad (an armed struggle) and change the current political system which they claim is unIslamic," the prosecutor said.
Three of the four who pleaded not guilty are Malaysians including two army officers - Nor Azmi and Mohamad Yusri - and one is Indonesian. They will appear in court again on August 28.
Authorities in the Muslim-majority country have expressed increasing alarm at the threat of Islamic militancy in the wake of the bloody jihad by Islamic State (IS) in Syria and Iraq.
Shukor said Murad spent about four months from August 2014 in Syria fighting alongside IS before returning to Malaysia last November.
Malaysia has traditionally observed a moderate form of Islam and authorities keep a tight lid on militancy.
But the government has increasingly warned that Malaysian recruits to the IS cause could return home with its radical ideology.
A senior counter-terrorism official with the police Special Branch, Ayob Khan Mydin Pitchay, said they had arrested 108 people with suspected links to or sympathies with IS, or those who had sought to travel to Syria or Iraq.
Malaysia has introduced a new anti-terrorism bill which allows authorities to detain suspects for potentially unlimited periods without trial.