We talked to dozens of software engineers and kept hearing about three problems caused by poor cloud cost observability:
(1) Cost anomalies are slow to detect and hard to diagnose. For example, a computer vision company noticed their AWS costs spiking one month. Costs accrued until they identified the culprit: one of their customers had put up a life-size cutout of John Wayne, and they were running non-stop facial recognition on it.
(2) No cost accountability in big orgs. For example, a public tech company’s top priority last year was to increase gross margin. But they had no way to identify the highest cost managers/products or measure improvement despite tagging efforts.
(3) Uncertain and variable per-customer gross margins. For example, a SaaS startup had one customer generating >50% of its revenue. That customer’s usage of certain features had recently 1,000x’ed, and they weren’t sure the contract was still profitable.
(If you’ve had an experience like this, we’d love to hear about it in the comments.)
We built Dashdive because none of the existing cloud cost dashboard products solves all three of these problems, which often requires sub-resource cost attribution.
Existing tools combine AWS, GCP, Datadog, Snowflake, etc. cost data in a single dashboard with additional features like alerting and cost cutting recommendations. This is sufficient in many cases, but it falls short when a company (a) wants per-customer, per-team or per-feature cost visibility and (b) has a multitenant architecture.
By contrast, Dashdive uses observability tools to collect granular cloud usage data at the level of individual user actions (e.g. each API call or database transaction). We attribute this activity to the corresponding feature, the responsible customer and team and estimate its cost based on the applicable rate. The result is more detailed cost and usage data than can be obtained with tagging. This information can be used to detect anomalies in real-time and identify costly teams, features and customers. One of our customers is even using Dashdive to charge customers for their cloud usage.
We use Kafka to ingest large volumes (>100m/day) of product usage events, and our web dashboard supports real-time querying thanks to ClickHouse. This makes it fast and easy to answer questions like: “Over the past 14 days, how much vCPU time did customer X use on Kubernetes cluster A, and how much did that cost me?” You can answer such questions even when the same container or pod is shared by multiple customers, features and/or teams.
You can test drive the product with example data here: https://demo.dashdive.com/. Given the high per-customer cost of our infrastructure and the manual steps required for setup on our part, we don’t offer self-serve onboarding or a public “free tier” to monitor your own cloud usage, but this demo gives a basic view of our product.
Right now, Dashdive supports S3 and S3-compatible object storage providers. We’re working to add support for other providers and services, particularly compute services (EC2, GCP VMs, ECS, EKS, GKE, etc.).
If there’s any service in particular you want to see supported, please tell us in the comments. We’re eager to see your comments, questions, concerns, etc.
by Andrey
Azimov