NSE technical glitch shows switchover to back-up site isn't easy

A technical glitch faced by the National Stock Exchange (NSE) on Monday and data breach at banks earlier this year have raised queries on the ability of the financial sector to quickly respond to and resolve major cyber security incidents.

The regulatory framework mandates important institutions such as stock exchanges and banks to have business continuity plans (BCP), disaster recovery sites and computer emergency response teams (CERTs). These are part of the contingency plans to ensure that businesses can continue with little or no disruption even in the event of a disaster.

However, recent incidents suggest these safeguards perform well in simulated conditions and mock drills, but often fall short when actual threats emerge. Though no major financial loss has been reported so far due to the three-hour trading halt at the NSE, the time and the number of attempts it took for the bourse to get back to normalcy did not go down well with the financial community, especially brokers.

The bourse’s statements have suggested that there was no such ‘disaster’ to move things to a disaster recovery site.

“A BCP is invoked during a disaster such as hardware failure and connectivity-related issues. Preliminary assessment indicated a software problem. Second, the system was expected to be rectified quickly and shifting BCP site would have taken longer,” the exchange had said.

According to experts, a switch to the disaster recovery site can rarely be without disruption. “There has to be a substantial geographical distance between the primary and the back-up site. Therefore, the switchover could take hours. Therefore, an exchange or a bank first tries to locally resolve the issue even if it takes a few hours,” said an industry expert.

“The technical glitch was a localised cash market problem. The technical team was trying to rectify and bring the market up fairly quickly as the problem took more time than anticipated,” Ravi Varanasi, chief of business development, NSE, told Business Standard.

Typically, a disaster plan is handled by a crisis management team consisting of the senior-most management. Given the flurry of exits in the NSE in recent times, even such simple things could have led to a loss of precious time, experts said.

Ashok Chawla, the non-executive chairman, mentioned “legacy technology issues” in a letter explaining the “black swan” event to the bourse’s employees.

“Companies or institutions using disaster recovery sites should define the recovery time objective. If your recovery time objective is half an hour, then you have to keep a warm site,” said Venkat Nippani, partner at Grant Thornton LLP.

Nippani, who specialises on cyber security issues, underlines the importance of periodic checks to disaster recovery plans. The frequency of such checks and updates should be higher in case of critical organisations such as banks and exchanges.

In January, a little over a month after J Ravichandran took charge as interim chief, the bourse had conducted a two-day live trading session from its disaster recovery site. In a circular issued after this exercise, the NSE had claimed that its “business continuity policy” is aimed at having a systematic approach to deal with business disruptions to protect market integrity.

However, Monday’s disruption left the broking community puzzled as to what constitutes to a “disaster”.

The Securities and Exchange Board of India (Sebi) has asked the NSE to have a review of their BCP and submit a detailed plan as to what measures are going to be taken to avoid such recurrences.

Nippani of Grant Thornton said the central government had moved to set up sectoral CERTs for segments such as energy, communications and BFSI (banking, financial services and insurance) to better deal with sector specific threats. Even within the BFSI space, the levels of preparedness required for technology intensive areas such as exchanges and banks are higher. “Banking is a priority sector. The level of complexity is far higher,” he said.

The banking sector was recently spooked by incidents of data breach and attempts of cyber-heist. Banks, too, are required to have a disaster management mechanism in place, in their individual capacity.

According to Dhananjaya Tambe, chief general manager for IT operations at State Bank of India, banks always maintain two parallel systems in their IT infrastructure. There is one primary system that interacts daily with the bank and customers and there is another hidden secondary system that fires up and replaces the primary system whenever the first one is down.

The transition has to be seamless, but may take time to interchange depending upon controls and permission put in place, said banking sector officials.