Facebook Computer Vision AI Can Now Creepily Search Your Photos And Understand What’s In Them
Little did we know it, but Facebook's engineers have been experimenting with a computer vision platform that is able to sort through photographs and search for what's contained in them, even when they're not tagged or captioned. For example, the platform can tell if the photo has a tiger in it or if someone is wearing a black shirt. Facebook initially worked on this to boost the experience for the visually impaired but has now been incorporated into the social network's search function for all users.
This started out as a small research project. Engineers needed a general-purpose platform that they leverage at scale, so they built one and called it FBLearner Flow. In the beginning, there was no need to worry about provisioning or dealing with scaling for real-time traffic. Over time, however, it became more widely used. Facebook continued to build on top of it, including the addition of tools to automate the process of machine learning.
FBLearner Flow and its team of engineers eventually transitioned to Facebook's Applied Machine Learning team when the platform reached production scale. It is now the engine for the current computer vision team at Facebook with a platform called Lumos built on top of it. Facebook built Lumos to understand photos and videos, and it continually gets better over time through newly labeled data that is fed into it, and through annotated data from the applications that Facebook builds.
"Advances in deep learning have allowed us to make big improvements in image classification—questions like 'What is in the image?' and 'Where are the objects?' are being answered by systems more accurately than ever. We've advanced this research by designing techniques that detect and segment the objects in a given image," Facebook explains.
By leveraging Lumos, Facebook is able to offer users visual search. You might have liked someone's photo recently but don't remember exactly when it was posted or by who. That makes it hard to find, though if you can recall other details, such as the color of shirt that someone was wearing, Facebook can use that info to dig it up.
It sounds like a simple thing but there is some advanced computing taking place here. To make sure that search results are relevant, Facebook has be able to understand the the contents of a photo. For that happen, Facebook's engineers used cutting-edge deep learning techniques to process billions of photos and understand their semantic meaning.
Cool stuff, if not just a little bit creepy too.
This started out as a small research project. Engineers needed a general-purpose platform that they leverage at scale, so they built one and called it FBLearner Flow. In the beginning, there was no need to worry about provisioning or dealing with scaling for real-time traffic. Over time, however, it became more widely used. Facebook continued to build on top of it, including the addition of tools to automate the process of machine learning.
FBLearner Flow and its team of engineers eventually transitioned to Facebook's Applied Machine Learning team when the platform reached production scale. It is now the engine for the current computer vision team at Facebook with a platform called Lumos built on top of it. Facebook built Lumos to understand photos and videos, and it continually gets better over time through newly labeled data that is fed into it, and through annotated data from the applications that Facebook builds.
"Advances in deep learning have allowed us to make big improvements in image classification—questions like 'What is in the image?' and 'Where are the objects?' are being answered by systems more accurately than ever. We've advanced this research by designing techniques that detect and segment the objects in a given image," Facebook explains.
By leveraging Lumos, Facebook is able to offer users visual search. You might have liked someone's photo recently but don't remember exactly when it was posted or by who. That makes it hard to find, though if you can recall other details, such as the color of shirt that someone was wearing, Facebook can use that info to dig it up.
It sounds like a simple thing but there is some advanced computing taking place here. To make sure that search results are relevant, Facebook has be able to understand the the contents of a photo. For that happen, Facebook's engineers used cutting-edge deep learning techniques to process billions of photos and understand their semantic meaning.
Cool stuff, if not just a little bit creepy too.
