The unarmed group, reported by local media to be suspected followers of a Malaysian Muslim figure claiming royal descent, were stopped by police guards in the confrontations early morning yesterday at the palace in Kuala Lumpur.
A police official called the claim by the group's leader "unbelievable and out of the ordinary".
"His claim doesn't make sense. He claimed to have a letter of appointment from the Philippines," Kuala Lumpur police official Ku Chin Wah told reporters.
He said they were being investigated under a statute that forbids challenging the king's authority, which can bring life imprisonment, and for illegal assembly.
The eight detained males and two females included a child aged 11, he said.
Ku was earlier quoted by Malaysian media as saying they were armed only with flags and documents purportedly laying out the royal claim.
The Star newspaper said one of the documents called for the formation of a three-million-strong army to prepare for the coming of the Mahdi, a prophesied redeemer of Islam.
The sultans trace their lineages back to Malay sultanates of the 15th century.
The current king is Sultan Abdul Halim Mu'adzam Shah, 85.
Despite a merely ceremonial role, the king commands great respect from the public, especially majority Malays.
Malaysia occasionally sees security scares from small Islamic sects or cults.
In one of the biggest, a martial-arts sect that taught members they were invulnerable to bullets disguised themselves as soldiers and stole more than 100 weapons from two military armouries in 2000.