Japanese Prime Minister Shinzo Abe and Mongolian President Tsakhiagiin Elbegdorj announced the agreement in a joint statement.
Under the deal that covers a range of products including beef and cars, all Mongolian exports to Japan and 96 per cent of Japanese products sent to Mongolia will be exempt from tariffs within the next decade, officials said.
The agreement could help foster stability in the region, a foreign ministry official added, as Japan works to resolve the case of Japanese citizens abducted during the Cold War by North Korean agents.
"(We're) hoping to deepen ties with Mongolia to work together on keeping stability in Northeast Asia," the official said.
The trade deal includes a so-called investor-state dispute settlement (ISDS) clause, which allows firms to pursue compensation claims if they think government policy has damaged their investment.
The clause has been criticised for giving firms too much power over public policy.
"It's important that the ISDS clause is included in the deal, as it encourages Japanese companies to invest in Mongolia," a senior trade ministry official told AFP on condition of anonymity.
Japan-Mongolia trade stood at 31.20 billion yen (USD 307 million) last year, far behind Tokyo's massive trade relationships with the United States and China.
In April Japan reached a broad trade agreement with Australia and it is working on a wide-ranging deal with the European Union and the so-called Trans-Pacific Partnership, a vast Pacific-wide deal led by the United States.
Resource-rich Mongolia, wedged between Russia and China, has seen big economic growth in recent years as global firms eye its vast natural resources.
Japan earlier this month lifted some of its own sanctions on North Korea following the apparent deal.
