We’re Ranko and Shaoyi, co-founders of Numericcal (https://numericcal.com). We’re building developer tools to simplify building, optimizing and maintaining Machine Learning models for embedded, mobile and IoT applications.
This platform is the result of our own struggles while helping on two ML powered projects; one using Computer Vision and one Voice Recognition. In both cases running models in the cloud was too expensive and slow for a good user experience. Moreover, it was not clear whether the training data our Data Science team had, at the beginning of the project, was representative of the data to be encountered in production.
The solution for cost and speed was to run inference on some end-device. However, we did not know which device would be feasible, nor did we know what model the Data Science team would end up with.
We initially considered two options:
* Postpone working on ML related software tasks until the Data Science team figured out what they wanted to do. Unfortunately, this meant that we could not parallelize the development. Moreover, the Data Science team would not get any feedback on the model runtime performance until we integrated them towards the end.
* Make a bunch of assumptions about models that the Data Science team would use and implement something. This would allow us to work in parallel, but we would be running the risk of having integration issues if those assumptions turned out to be wrong.
We looked for tools online to solve this but we couldn’t find anything comprehensive. So we decided to build our own! We settled on a design that packages ML models into (very primitive) serializable objects, encapsulating all the information and supporting files to use the models. For example, an object detection model would package the DNN graph, shape of the input and output tensor, bounding box anchors, etc.
This gave us what we wanted. On the software side we could write against the “interface” of the package. During initial development, we simply packaged dummy DNN models that always returned the same value. Later, the Data Science team would simply drop in a new package and everything just worked. One more victory for abstraction!
Since then, we’ve added a number of features around the packaging system. We can now add models during compilation or remotely through a cloud service and a user-friendly GUI. Models are versioned and report basic performance metrics to the cloud so Data Science teams can get feedback and guide model exploration. Remote delivery also allows sending different models to devices with different compute capabilities. We also added the possibility of running models on different runtimes (TensorFlow Mobile, TensorFlow Lite, our own library, Caffe2 is coming soon). Finally, we wrapped the app side code into ReactiveX API to make working with multiple models asynchronously easier.
Today we’re releasing the Numericcal Cloud Tools and Android SDKs in Beta. You can read about the system in more detail at (overview.workspace.numericcal.com) and check out demos (https://github.com/numericcal/BetaDemoApp).
Machine Learning on mobile and IoT devices is yet to gain wide adoption. Making this transition will bring personalization of mobile apps and automation of business and industrial processes to a whole new level. We’re currently working on projects for automatic damage assessment on vehicles and preventative maintenance of mobile assets.
We hope these tools will add to the momentum by speeding up development iterations for other teams, as they did for us. In the long term, we plan to open source edge integration and packaging SDKs. We will also open up more of our cloud-hosted functionalities, such as automatic model benchmarking, deployment, model compression and runtime optimization as services.
We’re eager to hear comments, ideas and your experiences in building Edge applications with ML. Thanks for reading!
by Andrey
Azimov