A pause in the yen's depreciation and caution ahead of a European Central Bank meeting also helped drive the benchmark Nikkei average 0.9% lower. The ECB is to hold a policy meeting later in the day that could impact the currency market.
The Nikkei closed at 11,357.07, retreating from a high of 11,498.42 struck on Wednesday. Trading volume stood at 4.04 billion shares, marking the highest level since March 15, 2011.
Shares of Sony Corp advanced ahead of its quarterly results while Mazda Motor Corp rose on better full-year earnings guidance.
The broader Topix was flat at 969.18 in heavy trade, with 5.14 billion shares changing hands, its second highest on the record.
"Investors are taking profits on recent gainers but at the same time, some of them are buying on dips on hopes that there is an upside in a longer term," said Hiroaki Hiwada, a senior strategist at Toyo Securities.
The Nikkei has gained 31% since mid-November, driven by the yen weakness after Prime Minister Shinzo Abe began calling for aggressive easing in his election campaign.
But with investors cautioned against such steep rises in the market, a correction is widely expected, analysts said.
"Hopes for 'Abenomics' are supporting the mood, but investors are also sensitive to the currency moves, so right now, even small uncertainty on Europe can be a reason to pull back," said Hiroichi Nishi, an assistant general manager at SMBC Nikko Securities.
The yen was quoted at 93.65 to the dollar on Thursday after hitting a 33-month low of 94.08 yen in the previous session.
Nikon was the top weighted loser in the Nikkei and draggged down the precision machinery sector, which dropped 5.6%.
"Nikon's sharp cut in its outlook startled us ... I'm not surprised if investors are dumping the stock immediately and even shorting it as there are better stocks to buy," said Mitsushige Akino, chief fund manager at Ichiyoshi Asset Management.
Mazda, Sony offer respite
Bucking the market's overall weakness, Sony gained 2.6% on expectations for strong results, which was released after the market close.
After the closing bell, the maker of PlayStation and Bravia TVs posted a third-quarter operating profit after a loss a year ago, but maintained its full year operating profit forecast of 130 billion yen.
Among other gainers, Mazda Motor soared 12% after the carmaker raised its operating profit forecast for the year ending March 31 by 80% to 45 billion yen, which would be its highest since the year ended March 2008.
Shares of Mazda, which exports about 80% of its cars, were the most-traded by turnover on the main board.