Building a fashion-forward AI system
Computer systems have gotten very good at recognizing whether a picture shows a dog or a cat, but it is much more challenging to teach them to spot when a sweater clashes with a pair of pants. For Fashion++ to work, it must be able to take a photo and then differentiate the individual items of clothing and accessories, as well as how they’re being worn. (For example, are the sleeves rolled up or down?) It also has to pick up on the essential aesthetic shown in the sample images it is trained on. Do they show a preference for minimalist, understated, office-ready outfits? Or something a little more funky?
“Automatically suggesting minimal edits is challenging because the difference between outfits is subtle — often just a few pixels, which is difficult to capture with traditional computer vision models,” says Grauman. “Furthermore, ideal training data would consist of curated pairs of better and worse versions of the same outfit, but that would be much too time-intensive to gather manually. Our approach takes steps to address both issues.”
To round out Fashion++’s sense of style, the system also creates its own examples of “unfashionable” outfits. Fashion++ takes the on-trend images it has been given as samples and alters them by swapping out garments, changing patterns, and adjusting the fit. It then uses these artificially generated looks to help teach itself what qualifies as a fashion “don’t.”
After analyzing the examples, Fashion++ is ready to help dress to impress. Give it an image of what someone is wearing and it can respond with a modified version that more closely resembles the examples it was trained on.
To gauge whether these changes were successful, the research team asked human evaluators to rate Fashion++’s advice. Overall, participants not only preferred the AI's suggestions, but also judged those upgraded looks as being similar to their examples used to train it.
“A desirable feature of Fashion++ is how it incrementally improves an outfit's fashionability, thereby providing a spectrum of edits,” says Kimberly Hsiao, an AI researcher who worked on the project. “Users can choose a preferred endpoint, starting from the least changed and moving towards the most fashionable.”
Because Fashion++ learns from whatever examples its given, it might be possible to have it account for local or regional style trends. If you trained it only with photos of, say, Harajuku styles, then Fashion++ would try to help you adjust your look accordingly. And if you decided that Fashion++ was falling behind the times, you could feed it new images of the latest spring runway looks.