News of the exodus from Kekenu, Budur, Yoyo and Mile 90 villages came as Niger hosted a meeting on how to fight the rebels as concern mounted at their threat to regional security.
Also yesterday in neighbouring Cameroon, Boko Haram insurgents engaged in "heavy clashes" with government soldiers in the far northern border village of Bonderi, though a security source said it was "too early to give a casualty toll".
Boko Haram fighters attacked Baga on January 3, looting and burning down homes and businesses in the town and at least 16 surrounding villages on the shores of Lake Chad.
Hundreds of people, if not more, are feared to have been killed, although there is no official confirmation of the toll as the town is still in rebel hands, residents said.
Security analysts said the attack, in which a regional military base was captured, potentially puts the group in a strong strategic position to strike southwards and launch cross-border raids.
Abubakar Gamandi, head of the Borno State fisherman's union, said residents from the affected villages told him Boko Haram fighters had visited "and asked people to leave -- or else".
One woman who fled Baga to the Borno State capital, Maiduguri, on Monday confirmed that she joined the crowds fleeing the four villages.
"When we came to Mile 90, we found it almost empty with some remaining residents staying behind to pick up personal belongings," said Ma'agana Butari.
The villages lie some 40 kilometres (25 miles) south of Baga and although there was no confirmation that Boko Haram had moved in, it will likely raise fears that the group plans to push south.
Boko Haram, which is fighting for a hardline Islamic state in northeast Nigeria, was founded in Maiduguri in 2002 but was driven out in 2013 after a state of emergency was declared.
