We are launching an app (Mirror) that uses a single selfie to create ~250 emoji/stickers that look like you and your friends.
We previously started a large online travel company in Russia (Ostrovok, >$500m in sales this year) and while there got very interested in AI as a tool to solve business objectives like predicting cancelations or conversion rates to inform marketing spend. And when we started something new, we wanted a product that had extremely high-frequency usage to build feedback loops into + something that a lot of humans really want to do, but that AI can do better.
Emoji keyboards sound like trivial toys, but this is actually a path to teaching AI one of the most complex human social behaviors. Offline, our faces communicate the majority of information that passes between us; far more than words do. In fact there is a whole piece of our brain (fusiform gyrus) dedicated to facial analysis and if lesioned humans lose ability to read faces while losing nothing else. Basically humans pay closer attention to faces than to any other object in the universe – from the day we are born.
AI needs data that informs it whether its decisions are right. And emoji are by far the largest structured source of facial data + usage in the world – >15 billion sent a day. Getting that ocean of data will help train AI that can solve all kinds of future problems – from VR characters that look like you in the gameworld, to personalized advertising that has your face in it the way you like, and a lot of other things. Everyone asks us why after starting a big “serious" company, we decided to work with emoji, and this is the reason.
On to the actual product. We think custom emoji apps like Bitmoji are a good step towards bringing our real, unique faces into messaging. We take this further and let you:
- Get your emoji in 3 seconds with one tap.
- Not spend mental energy wondering what your head shape/nose shape etc. are
- Add your friends/family (soon your pets and more) to your own emoji in seconds. Manually constructing 10+ friends is just too hard.
- Use all text-containing emoji in 8 languages (will expand to >25 shortly). We really have no idea why Bitmoji and other such apps tend to be English-only.
- Get emoji that become more and more recognizable/liked with time by learning from data (explicit choices + implicit metrics). For example we see that people want larger-than proportional eyes and darker skin. Except in Japan where it seems users want lighter skin. Really excited to find more like this with usage data.
Google Allo (and some others) tried doing this but got very little done. The reason is that while the idea is extremely obvious, the tech is quite hard. Here's why:
- You have to get a lot of facial features right. Some of these (e.g. hair, facial hair) are very diverse across age/sex/ethnicity. Some (e.g. head shape) are 3d but need to be inferred from shadows in a 2d image. Some (e.g. lower eyelids or wrinkles) can be detected but have a tradeoff of authenticity vs beauty. Some (colors of skin, hair, lips, eyes, eyebrows) are highly dependent on lighting conditions. All are must-haves. And this is still not enough – you have to detect and place wrinkles, birthmarks, sunspots etc. too
- You have to compensate for a lot of variability in selfie-taking – lighting/shadows, turn/pitch/tilt, camera distance, device camera type, background, whether the user is smiling or not etc.
- You have to use one photo to infer and reconstruct how that person looks with different emotions.
- There’s a whole dedicated part of the brain for this task, with a lot of neural machinery (fusiform gyrus). Humans can’t formally explain how they recognize other humans, but are very demanding to result quality. Humans take our exceptional skill at face recognition for granted, although for us a face is THE most well-perceived object in the universe. Think about it: can you instantly recognize a piece of a random wall that looks similar to every other wall, and that you stared at for an hour 10 years ago? You can do this with a face.
All the above makes training neural nets that solve recreating (rather than simply recognizing) faces much harder.
It took us close to a year and a large engineering team + multiple people with many years of experience with faces to get this to beta.
We know there is still a long way to go to make the product great. Right now we are working on several things to make it better:
- improving the tech from just “personalized” to “recognizable by friends” – both by utilizing feedback loops and by looking at new developments in AI. Particularly excited about GANs although we don't see a pure-GAN approach as viable (longer discussion).
- adding new features – e.g. auto-detection of groups of people, pets, clothes, backgrounds. doing a lot of this automatically based on your Photo Stream content and on-device ML technologies.
- expanding our content library to ~10,000 emoji in 25 languages.
You can get the app here: https://www.mirror-ai.com/
Let us know what you think – we really appreciate all your advice!
by Andrey
Azimov