Face Masks Causing Problems for Facial Recognition, NIST Says
COVID-19 has apparently completed what AI ethicists started: Eliminating the public use of facial recognition technology. At least, that’s one possible conclusion that can be drawn from a new study from the National Institute of Standards and Technology, which found that even the best facial recognition algorithms struggled to identify people wearing masks.
The NIST study found that masks degraded the performance of facial recognition algorithms across the board. It found that the highest performing facial recognition algorithms, which had error rates of 0.3% on images of unmasked people, struggled when a mask was digitally applied to the image, raising failure rates to about 5%.
But even otherwise competent algorithms that work well with unmasked faces failed to correctly identify a masked image between 20% and 50% of the time, NIST found.
“We can draw a few broad conclusions from the results, but there are caveats,” said Mei Ngan, a NIST computer scientist and an author of the report, in a story on the NIST website. “None of these algorithms were designed to handle face masks, and the masks we used are digital creations, not the real thing.”
The researchers measured the accuracy of 89 different facial recognition algorithms from a variety of companies and universities against two government collections of images–one from government passport applications, and the other from images taken at border crossings. The images were previously used in the NIST’s Face Recognition Vendor Test (FRVT), and totaled about 6 million.
The researchers ran the algorithms through “one-to-one” matching routines, where the lower quality border crossing images are compared to the higher quality passport photo of the same person, which created a baseline for accuracy for the specific algorithm. The researchers then digitally applied nine mask variants , which differed in terms of shape, color, and nose coverage, and measured the performance of the algorithms again.
The results were about what you would expect. According to the NIST report, the more that a mask covers the face, the more the algorithm struggles to correctly identify the person behind the mask. Algorithm error rates were generally lower with round masks, the researchers found. The more of the nose that is left showing, the better the algorithms generally did in correctly identifying the wearer of the mask, they wrote.
Researchers also found that black masks degraded algorithm performance more than surgical blue masks, although they were unable to fully test the effect of color.
None of the facial recognition algorithms in the test were developed to identify masked individuals. Later this summer, NIST plans to run a similar study that measures the effectiveness of facial algorithms specifically designed to identify people wearing masks.
The study was conducted in collaboration with the Department of Homeland Security’s Science & Technology Directorate (S&T), Office of Biometric Identity Management (OBIM), and Customs and Border Protection (CBP).