Actionable Auditing

Although algorithmic auditing has emerged as a key strategy to expose systematic biases embedded in soft- ware platforms, we struggle to understand the real- world impact of these audits, as scholarship on the impact of algorithmic audits on increasing algorith- mic fairness and transparency in commercial systems is nascent. To analyze the impact of publicly naming and disclosing performance results of biased AI sys- tems, we investigate the commercial impact of Gender Shades, the first algorithmic audit of gender and skin type performance disparities in commercial facial anal- ysis models. This paper 1) outlines the audit design and structured disclosure procedure used in the Gen- der Shades study, 2) presents new performance metrics from targeted companies IBM, Microsoft and Megvii (Face++) on the Pilot Parliaments Benchmark (PPB) as of August 2018, 3) provides performance results on PPB by non-target companies Amazon and Kairos and,

Authors: Inioluwa Deborah Raji, Joy Buolamwini

Published in: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society

Publication date: 2019-01-27

Read the paper: https://doi.org/10.1145/3306618.3314244

The authors and publisher do not sponsor or endorse this recording.