How a piece of hardware took down Japan's $6-trillion stock market

At 7:04 am on an autumn Thursday in Tokyo, the stewards of the world’s third-largest equity market realised they had a problem.

A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution.

The full-day shutdown that ensued was the longest since the exchange switched to a fully electronic trading system in 1999. It drew criticism from market participants and authorities and shone a spotlight on a lesser-discussed vulnerability in the world’s financial plumbing — not software or security risks but the danger when one of hundreds of pieces of hardware that make up a trading system decides to give up the ghost.

The TSE’s Arrowhead system launched to much fanfare in 2010, billed as a modern-day solution after a series of outages on an older system embarrassed the exchange in the 2000s. The “arrow” symbolizes speed of order processing, while the “head” suggests robustness and reliability, according to the exchange.

The system of roughly 350 servers that process buy and sell orders had a few hiccups but no major outages in its first decade.

That all changed on Thursday, when a piece of hardware called the No. 1 shared disk device, one of two square-shaped data-storage boxes, detected a memory error. These devices store management data used across the servers, and distribute information, such as commands and ID and password combinations, for terminals that monitor trades.

When the error happened, the system should have carried out what’s called a failover — an automatic switching to the No. 2 device. But for reasons the exchange’s executives couldn’t explain, that process also failed. 

That had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders.