Beijing's deployment of its first deep sea rig was the latest in a series of provocative actions aimed at asserting its sovereignty in the South China that have raised tensions with Vietnam, the Philippines and other claimants.
The United States shares many of the regional concerns about China's actions in the seas, which are potentially rich in gas and oil.
Last week, President Barack Obama signed a new defence pact with the Philippines aimed at reassuring allies in the region of American backing as they wrangle with Beijing's growing economic and military might.
It said ships entering a 3-mile (4.8-kilometres) radius around the area are prohibited.
Vietnam's foreign ministry said the area where the rig was stationed lay within Vietnam's exclusive economic zone and continental shelf as defined by the 1982 UN Convention on the Law of the Sea.
"All foreign activities in Vietnam's seas without Vietnam's permission are illegal and invalid," the ministry said in a statement. "Vietnam resolutely protests them."
Many analysts believe China is embarking on a strategy of gradually pressing its claims in the water by seeing what it can get away with, believing that its much smaller neighbours will be unable or unwilling to stop them.
Vietnam has accused Chinese ships of cutting cables to its exploration vessels and harassing fishermen, as has the Philippines.
Chinese assertiveness puts Vietnam's authoritarian government in difficult position domestically because anger at China, an ideological ally, runs deep in the country. This is exploited by dissident movements, who accuse the government of being unwilling to speak out against Beijing.
"This act by China is much more dangerous than previous actions such as cutting the exploration cable or fishing bans," he said.