The idea grew out of Erik’s research when he was a PhD student at MIT’s Center for Collective Intelligence. In short, he found that by aggregating the opinions of multiple people--even including some people with little or no medical expertise--they could reliably distinguish cancerous moles from benign ones better than individual dermatologists.
The three of us have been friends since we were undergrads. When we would chat about Erik’s research, it seemed like a no-brainer that there’d be demand for more accurate diagnoses. We all had our frustrations that as patients, you usually have to trust one doctor’s opinion.
So we built a mobile app called DiagnosUs where users around the world analyze medical images and videos. Many are doctors who simply enjoy looking at cases or want to improve their skills. Other users like competing with their peers, seeing themselves on our leader boards, and winning cash prizes in our competitions.
Different people (and algorithms) have different skills. Using data on how our users perform on cases with “gold standard” answers, we train a machine-learning model to identify how differently-skilled people complement each other and cover each other’s blind spots. The more we learn about our users’ skills and expertise, the better we get at aggregating their opinions. It is a bit like putting together the optimal trivia team: you don’t just need the five best people, you need someone who is good at pop culture, someone who knows sports, etc. Experts trained in the same way often have the same blind spots, so outcomes improve when you include a range of opinions.
We initially thought we’d go straight to providing opinions on demand for consumers like ourselves. There aren’t nearly enough doctors to meet the demand around the world to have everyone’s medical images analyzed. But it didn’t take long to realize that our fledgling startup wasn’t yet prepared to deal with the regulatory issues that would entail.
Meanwhile, we’d been hearing for years that AI was on the verge of replacing radiology, but it seemed like the hype didn’t match the reality. Many companies trying to develop medical AI are impeded by bad data. They try to hire doctors to go through thousands or millions of images and re-label them, but this has proven hard for them to manage and scale.
Our customers have giant medical datasets and want to use them to train AI. But the quality of the data holds them back, and they can’t find nearly enough doctors to label the data accurately. Our platform provides a high volume of labels quickly, and our performance analytics enables us to get highly accurate labels from groups of people with a range of skills.
We’d love to hear from anyone working on medical AI who’s faced the challenge of dealing with flawed datasets. If you’re interested in trying our app, you can download DiagnosUs for iOS in the App Store. Thanks for reading!
by Andrey
Azimov