Before Orbiter, we were product managers and data scientists at Tesla, DoorDash, and Facebook. It often felt impossible trying to keep up with the different dashboards and metrics while also actually doing work and building things. Even with tools like Amplitude, Tableau, and Google Data Studio, we would still catch real issues late by days or weeks. This led to lost revenue and bad customer experiences (i.e. angry customers who tweet Elon Musk). We couldn't stare at dashboards all day, and we needed to quickly understand which fluctuating metrics were concerning. We also saw that our engineering counterparts had plenty of tools for passive monitoring and alerting—PagerDuty, Sentry, DataDog, etc.—but the business and product side didn’t have many. We built Orbiter to solve these problems.
Here’s an example: at a previous company, a number of backend endpoints were migrated which unknowingly caused a connected product feature in the Android shopping flow to disappear. Typically, users in that part of the shopping flow progress to the next page at a 70% rate but because of the missing feature, this rate dropped by 5% absolute. This was a serious issue but was hard to catch by looking at dashboards alone because: 1) this was just one number changing out of hundreds of metrics that change every hour, 2) this number naturally fluctuates daily and weekly, especially as the business grows, 3) it would have taken hours of historical data analysis to ascertain that a 5% drop was highly abnormal for that day. It wasn’t until this metric stayed depressed for many days that someone found it suspicious enough to investigate. All in, including the time to implement and deploy the fix, conversion was depressed for seven days costing more than $50K in reduced sales.
It can be especially challenging for the human eye to judge the severity of a changing metric; seasonality, macro trends, and sensitivity all play a role in equivocating conclusions. To solve this, we build machine learning models for your metrics that capture the normal/abnormal patterns in the data. We use a supervised learning approach for our alerting algorithm to identify real abnormalities. Then, we forecast the expected “normal” metric value and also classify whether an abnormality should be labeled as an alert. Specifically, forecasting models identify macro-trends and seasonality patterns (e.g. this particular metric is over-indexed on Mondays and Tuesdays relative to other days of the week). Classifier models determine the likelihood of true positives based on historical patterns. Each metric has an individual sensitivity threshold that we tune with our customers so the alerting conditions catch real issues without being overly noisy. Models are re-trained weekly and we take user feedback on alerts to update the model and improve accuracy over time.
Some of our customers are startups with sparse data. In these cases, it can be challenging to build a high-confidence model. What we do instead is work with our customers to define manual settings for “guardrails” that trigger alerts. For example, “Alert me if this metric falls below 70%!” or “Alert me if this metric drops more than 5% week over week”. As our customers grow and their datasets grow, we can apply greater intelligence to their monitoring by moving over to the automated modeling approach.
We made Orbiter so that it's easy for non-technical teams to set-up and use. It’s a web app, requires no eng development, and connects to existing analytics databases the same way that existing dashboard tools like Looker or a SQL editor just plug in. Teams connect their Slack to Orbiter so they get immediate notifications when a metric changes abnormally.
We anticipate that the HN community has members, teammates, or friends who are product managers, businesspeople, or data scientists that might have the problems we experienced. We’d love for you and them to give Orbiter a spin. Most importantly, we’d love to hear your feedback! Please let us know in the thread, and/or feel free to send us a note at [email protected]. Thank you!
by Andrey
Azimov