I’m Ulrik from Cord (https://cord.tech) in the current YC W21 batch [1] - we are building software that allows people to label their data intelligently using a toolbox of various ‘labeling algorithms’. Labeling algorithms are any units of intelligence (e.g. a pre-trained model, or an interpolation algorithm) that help automate the annotation process. This enables data science and machine learning teams to rapidly iterate on their ML models without having to farm out labeling tasks to an external workforce.
Today we’re launching the first part of our product, our Web App, which serves our initial set of automation features through a GUI. It also allows you to classify images and draw vector labels, visualize data, and perform collaborative QA.
Computer vision ML algorithms are widely used for tasks like detecting everyday objects such as cars and pedestrians. However, they are yet to see widespread adoption for things like detecting cancerous polyps during an endoscopic procedure or blood clots in MRI scans. The lack of massive-scale labeled training datasets that fuel contemporary approaches is often the blocking element in building ML applications that solve these more specialised tasks. We also believe that the core part of the IP of an ML application stems from the labeled data used to train it.
Creating these datasets is challenging for several reasons. Labeling the data requires expensive domain-expert annotators, and privacy might prevent the data from being sent to an external workforce. Ultimately most labeling work tends to be done using open-source tools that were not created for speed and purpose-built to handle massive-scale datasets[2]. These tools also tend to provide a poor experience for the end consumer of the training data (e.g., data scientists, ML engineers) because they lack intelligence and require high manual input.
The initial seed of the idea came while I was working on a CS master’s project of visualizing massive-scale medical image datasets. I saw saw how much time and effort was being spent by doctors on labeling data. I met my co-founder Eric, who had worked as a quant researcher in finance, and after meeting him we realized we could take an algorithmic approach to tackling the labeling problem. Instead of writing trading algorithms, we turned our focus to writing labeling algorithms.
For example, for a food calorie estimation project we translated image level classifications of food items to individualized bounding box labels using a labeling algorithm we wrote with our SDK, requiring only one manual label per food item. Although it was an image dataset, our algorithm approximated noisy bounding box labels by using a CSRT object tracker across images. It then trained a shallow Faster RCNN ‘micro-model’ on the noisy labels, ran inference on the data, and suppressed earlier noisy labels. We then quickly visually reviewed and adjusted the results on our Web App[3]. We have applied a similar approach in areas such as gastroenterology[4] and pathology.
The days of relying on an army of human annotators and waiting to start the model building process are hopefully (soon) over. We are incredibly excited to be driving for that change - and are delighted to be sharing Cord with the HN community! We would love to hear your feedback. How are you going about creating and managing training data today? What are your key constraints? If you have used a creative method to label your data before, please share. Thank you so much in advance!
[1] What I Learned From My First Month at Y Combinator - https://medium.com/swlh/what-i-learned-from-my-first-month-a...
[2] Why You Should Ditch Your In-House Training Data Tools (And Avoid Building Your Own) - https://medium.com/p/ef78915ee84f
[3] Label a Dataset with a Few Lines of Code - https://eric-landau.medium.com/label-a-dataset-with-a-few-li...
[4] Pain Relief for Doctors Labelling Data - https://eric-landau.medium.com/pain-relief-for-doctors-label...
by Andrey
Azimov