We are Bastien, Guillaume and Alex, founders of Lazy Lantern (https://www.lazylantern.com). We work on detecting what really matters as it happens in your website or app.
As software engineers in various companies, we repeatedly got overwhelmed by the amount of product analytics we had to keep track of. What specific metrics are you supposed to monitor when you have dozens or hundreds of them, each metric having contextual information about the user, device type, location, language, etc.? This can represent thousands to millions of useful sub-metrics. Despite spending significant time monitoring analytics dashboards on Google Analytics, Mixpanel, Amplitude, Grafana and more, we had to keep track of so many metrics and user segments that impactful events regularly went unnoticed. We often missed technical incidents, but also business opportunities such as not knowing that a feature really moved the needle or that there was sudden adoption for a specific user group.
We started Lazy Lantern to build an automated way of analyzing any number of metrics in real-time. The goal is to provide a good picture of impactful events as they happen, both in the case of negative anomalies (outages, bugs, crashes) and positive anomalies (virality, marketing, growth). In practice, we automatically detect abnormal patterns for each metric, in particular temporary spikes/drops, level changes, trend changes and seasonality changes. In case of anomaly, we surface the user segments that are most affected and we group correlated anomalies together to give you a better picture of what parts of the product are impacted.
On the implementation side, there were a couple of requirements for an effective anomaly detection algorithm. It has to be:
- Autonomous: avoiding manual configuration to be able to scale to arbitrarily high numbers of metrics
- Unsupervised: being able to detect anomalies for all types of businesses without knowing beforehand what a typical anomaly for each business looks like
- Dynamic: accommodating all kinds of seasonalities and trends, which excludes using static thresholds
- Fast: deciding whether a data point is indicative of an incident in minimal time
To fulfill these requirements, we first tried the Holt-Winters seasonal models, but finally got the best results with a procedure based upon Facebook’s Prophet forecasting model. To provide a better sense of each anomaly’s severity as well as what areas of the product are affected, we integrated two additional functionalities:
- Anomaly severity scoring based on the number of impacted users, deviation from prediction and anomaly duration
- Anomaly grouping using a reproduction of VARCLUS, which groups metrics by clusters based on their partial correlations
For this initial launch, we are targeting Segment customers, which makes enabling our product a breeze. If people find it useful, we will provide wider support. Pricing is based on the number of metrics you want to track. If you email us at [email protected] mentioning this post, we’ll extend the free trial to 3 months. If you are interested, sign up in one minute on our website at www.lazylantern.com.
We’d love to know if you think this product might be useful to you or if there is a better way to approach the problem. Thank you!
by Andrey
Azimov