Reuben Binns, our Research Fellow in Artificial Intelligence (AI), and Valeria Gallo, Technology Policy Adviser, discuss some of the key safeguards organisations should implement when using solely automated AI systems to make decisions with significant impacts on data subjects.

This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.

The General Data Protection Regulation (GDPR) requires organisations to implement suitable safeguards when processing personal data to make solely automated decisions that have a legal or similarly significant impact on individuals. These safeguards include the right for data subjects:

• to obtain human intervention;
• to express their point of view; and
• to contest the decision made about them.

See:  How Data-driven Strategies Can Improve Impact Investing Outcomes

These safeguards cannot be token gestures. Guidance published by the European Data Protection Board (EDPB) states that human intervention involve

a review of the decision, which “must be carried out by someone who has the appropriate authority and capability to change the decision”.  The review should include a “thorough assessment of all the relevant data, including any additional information provided by the data subject.”

In this respect, the conditions under which human intervention will qualify as meaningful are similar to those that apply to human oversight in ‘non-solely automated’ systems. However, a key difference is that in solely automated contexts, human intervention is only required on a case-by-case basis to safeguard the data subject’s rights.

Why is this a particular issue for AI systems?

The type and complexity of the systems involved in making solely automated decisions will affect the nature and severity of the risk to people’s data protection rights and will raise different considerations, as well as compliance and risk management challenges.

Basic systems, which automate a relatively small number of explicitly written rules (eg a set of clearly expressed ‘if-then’ rules to determine a customer’s eligibility for a product) are unlikely to be considered AI. It should also be relatively easy for a human reviewer to identify and rectify any mistake, if a decision is challenged by a data subject because of system’s high interpretability.

However other systems, such as those based on machine learning (ML), may be more complex and present more challenges for meaningful human review. ML systems make predictions or classifications about people based on data patterns. Even when they are highly accurate, they will occasionally reach the wrong decision in an individual case. Errors may not be easy for a human reviewer to identify, understand or fix.

While not every challenge on the part of data subject will be valid, organisations should expect that many could be. There are two particular reasons why this may be the case in ML systems:

• The individual is an ‘outlier’, ie their circumstances are substantially different from those considered in the training data used to build the AI system. Because the ML model has not been trained on enough data about similar individuals, it can make incorrect predictions or classifications.
• Assumptions in the AI design can be challenged, for example a continuous variable such as age, might have been broken up (‘binned’) into discrete age ranges, eg 20-39, as part of the modelling process. Finer-grained ‘bins’ may result in a different model with substantially different predictions for people of different ages. The validity of this data pre-processing and other design choices may only come into question as a result of an individual’s challenge.

See:  Innovative new law opens Guernsey up to Artificial Intelligence

What should organisations do?

Many of the controls required to ensure compliance with the GDPR’s provisions on solely automated systems are very similar to those necessary to ensure the meaningfulness of human reviews in non-solely automated AI systems.

Organisations should:

• consider the system requirements necessary to support a meaningful human review from the design phase. Particularly, the interpretability requirements and effective user-interface design to support human reviews and interventions;
• design and deliver appropriate training and support for human reviewers; and
• give staff the appropriate authority, incentives and support to address or escalate data subjects’ concerns and, if necessary, override the AI system’s decision.

Continue to the full article --> here