The protest which surrounded the parliament building was held as the nationalist premier tries to force through parliament a set of controversial bills making the changes.
The bills are a pet project of Abe, who says Japan can no longer shy away from its responsibility to help safeguard regional stability, and must step out from under the security umbrella provided by the United States.
The draft legislation would broaden the remit of Japan's well-equipped and well-trained armed forces.
Opponents of the bills accuse Abe of trying to move the country away from pacifism, while three scholars summoned to parliament testified this month that the bills are unconstitutional.
Japan's constitution was imposed by US occupying forces in the aftermath of World War II, but its war-renouncing Article Nine is held dear by many people.
"Don't destroy Article 9," read banners at the rally, which organisers said attracted some 25,000 people.
The legislation, which would overhaul 10 security-related laws and create a new one, would also pave the way for the military to deploy abroad on non-combat missions such as disaster relief and UN peacekeeping missions.
They also say Japan can defend allies "in situations where there is a clear risk that Japan's existence is threatened and its people's rights... Are compromised through an attack on a country which has a close relationship with Japan".
