The fighting erupted at dawn after an assault by the jihadist Islamic State of Iraq and the Levant on posts held by al-Qaeda affiliate al-Nusra Front and its allies.
The clashes centred around the border town of Albu Kamal, from which ISIL was expelled from earlier this year, said the Syrian Observatory for Human Rights.
It said 51 combatants were killed.
The bodies of another 10 fighters, apparently executed by ISIL, were found at two sites near Albu Kamal, the Observatory said, adding there were reports of other summary executions in the same area.
The clashes prompted soldiers on the Iraqi side of the border to reinforce their positions.
The border crossing itself on the Syrian side remained in the hands of the mainstream rebel Free Syrian Army, according to a rebel chief and an AFP reporter on the Iraqi side.
