As a data deluge grows, companies rethink storage

John Hayes, cleareyed and wild-haired, stood before his silent creation. Big as a slim refrigerator, it held 16 petabytes of data, roughly equal to 16 billion thick books.

"People are going to have to think about things to put into this," he said, surrounded by the clutter of his office at a Silicon Valley company called Pure Storage. "But that won't take long - there's a demand for data that nobody was ready for."

Each month, the world's one billion cellphones throw out 18 exabytes of data, equal to 1,100 of Hayes's boxes. There are also millions of sensors in things ranging from cars and appliances to personal fitness trackers and cameras.

IBM estimates that by 2020 we will have 44 zettabytes - the thousandfold number next up from exabytes - generated by all those devices. It is so much information that Big Blue is staking its future on so-called machine learning and artificial intelligence, two kinds of pattern-finding software built to cope with all that information.

Making storage products has long been a major part of the tech industry. It has also been one of the dullest, with little in the way of innovation. Now the surge in data is leading both start-ups and some of tech's biggest companies to rethink how they approach the problem.

Pure, co-founded by Hayes, a 38-year-old former video karaoke engineer, is one of several companies trying new approaches.

Hayes's box, which was unveiled on Monday, holds five times as much data as a conventional storage device, thanks to a combination of so-called flash storage technology and clever engineering. Sometime in 2017, he said, it will hold twice that much, as Pure tweaks the product. Power consumption, the company says, is 4 per cent of the current standard.

While traditional storage works much like a record player, with data stored on and fetched from spinning disks, flash storage gets data directly on and off a semiconductor. Flash storage has become common in consumer tech products like smartphones.

The new Pure system is also built in blades, which act like building blocks. Add a blade, and the overall system reconfigures to raise efficiency. There is space on the device for additional processing, so a blade could easily be added.

"It's not just about Google and Facebook," said Joe Unsworth, an analyst with Gartner. "Health care, manufacturing and natural resources companies can all justify owning this much storage. In 10 years, a big sanitation company with sensors on its Dumpsters to manage pickups could have tens of petabytes."

Pure is hardly alone in working on new ways to handle the world's immense data deluge. The industry leader, EMC, has both traditional and advanced storage, including flash. EMC was bought by Dell in October for $67 billion. "If Dell hadn't bought them, I'd wonder what Dell was going to do," Unsworth said.

Netflix, which offers streaming video to 75 million people in 190 countries, ships dozens of custom-built appliances a week to hundreds of local Internet service providers. The newest, a mix of flash and traditional storage, can store five petabytes - enough, the company figures, to hold almost anything from its catalogue that anyone, anywhere would want to watch, for now.

The company analyses data to figure out both the kind of content it should offer next, and the Internet's capacity to stream all that entertainment. "We arguably have the most insight of anybody," said Ken Florance, vice president of content delivery at Netflix. "There are networks in Kenya that are better than some networks in the US"

Capturing and analysing masses of data already touches improbable corners, like championship auto racing. Mercedes AMG Petronas, the racing arm of the Daimler car company, is an early customer of the new Pure blades.

Mercedes AMG has about 300 sensors on each high-performance car. Much the way human athletes are regulated for substances, racing officials now limit the amount of computation cars can use, including wind tunnel simulations between races, so it pays to move data quickly.

"They audit the amount of processing we use," said Matt Harris, the head of information technology for Mercedes AMG. "If we can cut a few seconds off each job with faster storage, that's an extra simulation a month."

Pure is in a race of its own. The company went public in October, and since then its stock has fallen one-third from the highs reached soon after. The new flash blade product line will not be important for revenue this year, but the company's chief executive, Scott Dietzen, hopes it will calm disappointed investors.

"No one can look at all their data anymore; they need algorithms just to decide what to look at," he said, adding that he planned to offer 10 times the performance of competing products, at a lower cost. "This market is at least as large as our original storage business, and a lot bigger over time."

Hayes didn't set out to shake up a once-staid storage world. In 2006, he was a software engineer at a company that did online karaoke talent searches, when it was purchased by Yahoo.

When Yahoo killed the product, he became a promoter of Yahoo's social networks, then quit in 2009. His old bosses at the karaoke company, backed by venture capitalists, were looking at a way to use flash and needed a good coder.

The first Pure Storage product came out in 2012.

By mid-2013, he was thinking about how he would build his product for a world of falling memory prices, and rapidly rising amounts of data inside cloud computing systems.

Working with Dietzen and investors, he set up a venture inside Pure but away from the rest of the company, hiring a team of experienced people from places like Oracle, Microsoft Research and Facebook. Dietzen hopes investors will now see where some formerly unexplained research and development money has gone.

"He got people you couldn't recruit if you were a start-up - midcareer - who wouldn't work at a place that was going to run out of money, but wanted to be in something small and challenging," said Par Botes, an early hire who was chief technology officer at EMC. Like others in the effort, Botes has a LinkedIn profile that says only that he is at a "stealth mode start-up technology company."

From an original group of five people, the quiet project inside Pure has grown to 80, and Hayes sees a need for many more.

"I've talked to companies planning for hundreds of petabytes," he said. "Even a body camera on a cop adds up data pretty quickly."

©The New York Times News Service