Meerkat: Smart Anomaly Detection for Swisscom Network
Wouldn't it be great to have someone checking if all is normal or if danger is approaching? Meerkats use sentinels to check the surrounding, Swisscom now does the same.
Wouldn’t it be great to have someone checking if all is normal or if danger is approaching? Meerkats use sentinels to check the surrounding, Swisscom now does the same.
Meerkats are not only adorably cute, they are in fact great sentinels. While the group searches for food, one sentry is on guard and watches out for predators. As soon as a snake or a bird of prey is approaching, it whistles alarm. Swisscom develops a system that – just like the real-world meerkats – constantly observes what is happening, and generates alerts when it spots something out-of-the-ordinary. In other words, the system – unsurprisingly dubbed Meerkat – performs smart real-time anomaly detection in temporal data coming from its network.
Meerkat uses big data cluster-computing technologies such as Apache Spark to aggregate and ingest millions of data points coming from the network every few minutes. This data can typically be the numbers of occurrences of some events that have happened in the network. For instance, how many phone calls have succeeded and how many have failed for every time intervals.
A meerkat – talking about the animal right now – is learning from experience what a normal situation looks like and what needs to be judged as danger. So does the Swisscom system. By constantly ingesting new data points as they arrive, Meerkat can learn what the data is supposed to look like under normal circumstances.
Let’s look at a simple example: Meerkat can learn what the typical number of phone calls should be on an average Monday morning from 10:00 to 10:15. This learning phase consists of Meerkat building statistical models that summarize the observed data. This data is further used to predict future data points. That is how Meerkat can grip the normal situation of a Monday morning.
With every fresh data point observed, Meerkat checks it against both the typically expected value, as well as against its own predictions. If the data is sufficiently different from the prediction, it is classified as an anomaly. So if instead 1000 calls only 50 are registered, Meerkat hits the alarm.
What’s the good in counting phone calls on a Monday morning? Well, knowing these details certainly has its perks. An anomaly in the system is quickly detected by Meerkat – long before displeased clients grab the phone to give us a talking-to. Therefore, we are quicker in restoring the usual situation. The customer never even knew what happened.
Meerkat has already generated a handful of alerts that have generated some positive early feedbacks from our network engineers. However, it is still under development. Our goal is to make Meerkat an easy-to-use and seamless anomaly detection service for temporal data that monitors all different kinds of systems within the company. This will be a handful in itself. But at least we don’t need to worry about birds of prey and snakes.
Image Source: http://7-themes.com/6793191-free-meerkat-wallpaper.html