Developers often launch a feature without understanding the impact it has on their users and business. This is a big deal, because only 1/3 of product launches actually improve the desired metrics. Of the rest, 1/3 have no effect, and the last 1/3 actually hurt [1]. The best way to measure this is to use feature flags and controlled experiments (A/B tests).
Jeremy and I worked together for 10 years at an ed-tech startup as CTO and software architect. We spent far too long just building and launching features without really knowing how they impacted our users and if they were adding value to the company. We had product analytics, but there was too much noise in the data to draw real conclusions. We knew the “right” way to do this was to build feature flags and run controlled experiments, but that was daunting for our small team.
We looked into 3rd party tools, but it bothered us that they didn't use our existing data warehouse and metric definitions, and we really didn't like the idea of adding an API call in the critical rendering path of our application. We also didn’t want to send our data to 3rd parties, didn’t feel good about vendor lock-in, plus the vendors were expensive. So, we did what any engineers would do—build it ourselves. After all, how hard could it be?
After a couple painful years, we hacked something together that (mostly) worked and used it to help grow revenue 10x. We started talking to other teams and realized just how many larger companies spend years building these feature flagging and experimentation platforms in-house because, like us, they couldn’t find any tools that met their needs. So we took everything we learned and built the tool we wish had existed back when we started.
GrowthBook is an open source platform for feature flagging and A/B experimentation. Our SDKs are built to be fast and cache-friendly. We take data privacy seriously and don’t use cookies or deal with PII. We sit on top of your company’s existing data warehouse and metrics so you can maintain a single source of truth. We’re open source (MIT), so you can either self-host the platform (with Docker containers), or use our managed cloud offering.
In GrowthBook, feature flags are added and controlled within the UI. Engineers or PMs can add targeting rules (e.g.”beta users get feature X, everyone else does not”), do gradual rollouts, and run A/B tests on the features. The current state of features are stored in a JSON file and exposed via an API or kept in-sync with a cache/database using webhooks. Engineers install our SDK and pass in the JSON file. Then they can do feature checks throughout their code (e.g. `if feature.on { ... } else { ... }`).
For A/B test analysis, a data analyst or engineer connects GrowthBook to their data warehouse, then they write a few SQL queries that tell GrowthBook how to query their experiment assignment and metric conversion data. After that initial setup, GrowthBook is able to pull experiment results, run it through our Bayesian stats engine, and display results. Users can slice and dice the data by custom dimensions and/or export results to a Jupyter notebook for further analysis.
We’re used by over 60 companies in production. We have self-hosted and cloud versions (see our pricing here: https://www.growthbook.io/pricing), and both are self-serve and simple to set up. We currently have SDKs for Javascript/Typescript, React, PHP, Ruby, Python, Go, and Android with more in the works (C#, Java, Swift, and Elixir). We support all the major SQL data warehouses as well as Mixpanel and Google Analytics.
You can give it a spin at https://github.com/growthbook/growthbook. Let us know what you think! We would especially love feedback from anyone who has built platforms like this in the past.
[1] https://exp-platform.com/Documents/ExPThinkWeek2009Public.pd...
by Andrey
Azimov