I found the real Pied Piper and it is nowhere near Silicon Valley

92 percent of the world's data was created in the last two years alone, says Tech in Asia

"Data creation is exploding," says Gavin Belsen. “With all the selfies and useless files people refuse to delete on the cloud, 92 percent of the world’s data was created in the last two years alone. At the current rate, the world’s data storage capacity will be overtaken by next spring. It will be nothing short of a catastrophe. Data shortages, data rationing, data black markets. Someone’s compression will save the world from data-geddon, and it sure as hell better be Nucleus and not goddamn Pied Piper!”

Belsen is a fictional character, of course, from the HBO show Silicon Valley. But like a lot of the show’s satire, his over-the-top rant hints at a deeper truth: in the big data era, storage really is becoming a problem. As companies adopt big data analytics, they’re struggling to deal with the mountains of data they’re producing. Compression isn’t making them small enough, and compressed files aren’t searchable, which makes accessing them to take advantage of “big data” a big pain in the ass.

Pied Piper, the fictional startup headed by a tech genius who created a revolutionary algorithm from Silicon Valley, doesn’t exist in real life. But Terark does. It’s nowhere near Silicon Valley – the startup is based in Beijing – but fans of the show will find a lot familiar about the Chinese compression startup. Built on the back of a revolutionary algorithm created by a brilliant but (sorry Lei Peng!) somewhat socially awkward engineer, Terark is taking on what Gavin Belsen called “data-geddon” – and aiming to beat some global tech powerhouses in the process.

Now, Terark is a database compression company, and thanks to Lei’s algorithm, it’s blowing global competitors like Facebook and Google out of the water – at least in terms of performance.

 

This is an excerpt from Tech in Asia. You can read the full article here