A secret about analytics

Clean and structured data is a big challenge in the telecom space

Today's big buzzword is big data. Its companion, buzzword analytics, is also a big favourite. There are good reasons why this is so. The last five years, after all, have seen a massive upsurge of data; we have generated 90 per cent of the entire world's data in the last two years.

This change has been dramatic. Twenty years ago, we had all the data analysis tools in place but data was scarce. Today there is a frightening abundance of data, but nobody quite knows what to do with it.

What do we do with all this data? What can we do with all this data?

Think, for example, of telecom data. While an infinite amount of data is being generated, till recently, only a small fraction was actually used and that too for billing. The rest of the data simply flowed away, because it wasn't profitable to either store or exploit the data. If it cost you $100 to store and analyse data, and you could at best earn $50 from it, why would anyone take the plunge?

But now things are changing. Data storage is getting dramatically cheaper, and data retrieval is getting unbelievably faster. So there's now a good chance that your $100 investment will return you $120 instead. No wonder then that everyone is jumping onto the bandwagon, and making wild promises. "We'll make your data talk and sing," one analytics enthusiast recently told me. I am predicting that there will be no easy song and dance. Analytics, after all, isn't a magic wand. There is a lot of hard work ahead before the party can start.

Let us start with the basic nature of this data. You can't read data as you might read the next novel from Jeffrey Archer. Telecom data is an endless stream of zeroes, ones and mumbo-jumbo. Hidden among this mad rubble is a great story but you need immense skill to ferret out noteworthy gems out of the impossibly hard looking data pile.

First of all, the data must be collected with care. Simply put, pile together all the useful pieces from the rubble and discard everything that's unnecessary or irrelevant. Next, it must be 'mediated' in that the nonsensical looking data has to be transformed into something understandable and usable. So you should be able to precisely tell just how much time the subscriber with number 123456, for instance, has spent on local calls, long distance calls, sending text messages or even on data downloads like Flappy Bird (a popular mobile game).

Then it is important to understand just who this subscriber - with the number 123456 - is anyway? Where does she stay? How old is she? In fact, are you even sure of the gender in the first place? Clearly a telecom switchboard isn't going to give you that information but there must surely be a database somewhere that contains the coordinates and personal attributes of this unknown person. We, therefore, enrich subscriber No. 123456's telecom data by joining it to her personal database.

The exercise is anything but simple and it can get even harder along the way. What do you do if there's a 15-minute data blackout during which some interesting event may have occurred? How can we deal with such missing data? Can we model and hypothesise what this data could be? Could we estimate the probability that our estimated data is right?

Collection, mediation, enrichment, costing, modelling, pricing - all these are challenges that telecom analysts must grapple with. All too often we are led to think that analytics simply involves pouring data into a calculating furnace and coming out with exciting inferences that will save millions of dollars. This is simply not true.

While the drama, romance and mystery seems to be about models, algorithms, visualisations, animations and inferences, that's merely the glossy end of the picture. The real success is assured only if the data preparation, aggregation, mediation and enrichment happen correctly. This backroom pain and labour, in fact, is responsible for almost 90 per cent of the eventual success.

In the final count, telecom analytics is about telecom awareness, telecom domain knowledge and hard-earned telecom experience. It is only after you create this painstaking edifice that the song and dance can start.

Let's look at how some telecom software companies have successfully created this riveting platform for song and dance.

Somewhere in the middle of nowhere in America
Every US citizen has the right to connectivity "even if he stays in the middle of nowhere". For telecom carriers providing this statutory long distance connectivity for the 'last mile' can be quite a nuisance. So long distance telecom carriers usually prefer to sign up with local carriers to assure this last mile connectivity.

The local carriers charge a very stiff rate for this last mile connectivity (five cents per minute, instead of the normal 0.5 cents per minute, for example) but that is justified because traffic is expected to be very low and they need to be profitable.

But sometimes the local carrier can get greedy. Wouldn't it be great if he can somehow contrive to 'pump up' big traffic along this normally deserted last mile? What if he tied up with someone offering an adult chat or conferencing service? That could send traffic soaring.

In a celebrated legal battle some years ago, analytics successfully came to the rescue of a US long-distance carrier suffering losses adding up to many millions of dollars. Of course, it required considerable telecom acumen to prepare the evidence needed to satisfy the court of law.

From the switch to the bill
How do telecom carriers determine what rate to charge their customers? In most cases - and this is a surprise - this rate is based on what the competitor charges. But is this rate right? What is the telecom carrier's true cost?

Answers to such questions aren't easy because they involve a detailed breakdown of all the individual costs: the cost of connection, termination, circuits etc. And making such breakdowns again requires the intelligent use of the different elements of telecom analytics like mediation, costing, enrichment, and billing.

Often such analysis results in unexpected findings; that big revenue earning truck route, for example, could actually be a money loser.

The anatomy of a telecom traffic jam
This example may still belong to the realm of fantasy but it is something we are sure to see in the future.

Telecom carriers provide all their services via a very complex network of devices. By positioning sensors at various locations in the network it is possible to 'visualise' exactly how the network looks like (the comparison is tenuous, but think of a helicopter hovering over a city and taking pictures of traffic movement).

Some more sophisticated analytics will help us classify telecom traffic patterns and for every worrisome pattern we could provide a network 're-routing' antidote to ease the congestion. Indeed it could even be possible to configure a 'learning' network that is continually readapting itself so that it always stays efficient and error-free.

These examples points to the tremendous possibilities that big data offer in telecom sector. But we must never forget the underlying secret: only blue-blooded telecom experts can usher in this exciting telecom future.

Srinivas Bhogle
Director & India Country Manager, TEOCO Software