Nasdaq says software bug, Arca problems led to trading outage

By John McCrank

NEW YORK (Reuters) - Nasdaq OMX Group said on Thursday Nasdaq's three-hour trading halt last week was due to an internal software bug and other technology issues triggered by problems at NYSE Euronext's Arca exchange that led to the failure of a key backup system.

ARCA is the all-electronic exchange owned by NYSE, the parent of Nasdaq's rival, the New York Stock Exchange.

Nasdaq said connection problems between Arca and the Nasdaq-run system that receives all traffic on quotes and orders for stocks on the exchange, known as the Securities Information Processor, or SIP, led the processor to become overwhelmed.

The exchange was then forced to halt trading in its listed stocks.

"Nasdaq OMX is deeply disappointed in the events of August 22 and our performance is unacceptable to our members, issuers and the investing public," the exchange said. "While getting to 100 percent performance in all of our activities, including our technology is difficult - it is our objective."

New York-based Nasdaq said it was in the process of identifying potential design changes to strengthen the SIP's resiliency, "including architectural improvements, information security, disaster recovery plans and capacity parameters."

The exchange plans to present its initial recommendations related to the changes to the SIP governing committee, made up of U.S. exchanges and the Financial Industry Regulatory Authority, within 30 days.

Nasdaq said that on the morning of August 22, Arca connected and disconnected to the SIP more than 20 times, eating up the capacity of the system. It said the SIP's capacity was further eroded as Arca sent a stream of inaccurate stock symbols to the SIP, generating numerous rejection messages.

Each data port connecting to the SIP can handle 10,000 messages per second, Nasdaq said. But in this case, the traffic from Arca was more than double that, the exchange said.

"The confluence of these events vastly exceeded the SIP's planned capacity, which caused its failure and then revealed a latent flaw in the SIP's software code," the report said.

This software flaw prevented the processor's built-in redundancy capabilities from resetting properly and delayed the return of system messages. With the system degraded, the exchange said it decided to halt trading to ensure fair market conditions.

Nasdaq said it took 30 minutes to resolve the problem and then nearly three more hours to test and evaluate scenarios to re-open the market in a fair and orderly manner.

(Reporting by John McCrank; Editing by Kenneth Barry)