UK Police Use Zipcode Profiles, Garden Size And First Names For AI-Based Custody Decision System

As you have doubtless noticed, Cambridge Analytica has been much in the headlines of late. There is still plenty of disagreement about the extent to which the company's profiling tools provide the kind of fine-grained categorization of people that it claims, and whether it played a significant -- or indeed any -- role in deciding key elections, both in the US and elsewhere. What is not disputed is that such profiling is widely used throughout the online world, mostly to sell ads, and that it is likely to become more accurate as further data is gathered, and analytical techniques are honed. The continuing flow of reports about Cambridge Analytica and related companies has therefore at least served the purpose of alerting people to the important issues raised by this approach. Against that background, news that UK police in the north of England are applying similar techniques is troubling:

Durham Police has paid global data broker Experian for UK postcode [zipcode] stereotypes built on 850 million pieces of information to feed into an artificial intelligence (AI) tool used in custody decisions, a Big Brother Watch investigation has revealed.

Durham Police is feeding Experian's 'Mosaic' data, which profiles all 50 million adults in the UK to classify UK postcodes, households and even individuals into stereotypes, into its AI 'Harm Assessment Risk Tool' (HART). The 66 'Mosaic' categories include 'Disconnected Youth', 'Asian Heritage' and 'Dependent Greys'.

In order to decide whether someone should be charged with an offense, the HART system aims to help the police evaluate whether they are likely to re-offend. "High-risk" offenders are charged. Those with a "moderate" risk of re-offending are offered the option of joining a rehabilitation program; if they complete it successfully, they do not receive a criminal conviction. To build the specialized AI system, the local UK police force has been working with a team of researchers at the University of Cambridge:

Called the Harm Assessment Risk Tool (HART), the AI-based technology uses 104,000 histories of people previously arrested and processed in Durham custody suites over the course of five years, with a two-year follow-up for each custody decision. Using a method called "random forests", the model looks at vast numbers of combinations of 'predictor values', the majority of which focus on the suspect's offending history, as well as age, gender and geographical area.

The basic HART system has been in use since 2016. But Big Brother Watch has discovered that HART has been extended in a significant way through the use of the profiling information acquired from Experian. This Dublin-based company -- not to be confused with Equifax, which works in the same field -- has amassed personal information on hundreds of millions of people around the world. Where things become more problematic is how the profiles that Experian has passed to the Durham police force for its HART system are compiled. As well as using basic zipcodes, a wide range of sensitive "predictor values" are gathered, aggregated and analyzed, such as:

Family composition, including children,
Family/personal names linked to ethnicity,
Online data, including data scraped from the pregnancy advice website 'Emma's Diary', and Rightmove [UK real estate site],
Occupation,
Child [support] benefits, tax credits, and income support,
Health data,
[Children's exam] results,
Ratio of gardens to buildings,
Census data,
Gas and electricity consumption.

The use of first names to help assign people to categories is a striking feature of the approach:

Experian's 'Mosaic' links names to stereotypes: for example, people called 'Stacey' are likely to fall under 'Families with Needs' who receive 'a range of [government] benefits'; 'Abdi' and 'Asha' are 'Crowded Kaleidoscope' described as 'multi-cultural' families likely to live in 'cramped' and 'overcrowded flats'; whilst 'Terrence' and 'Denise' are 'Low Income Workers' who have 'few qualifications' and are 'heavy TV viewers'

By stereotyping people on the basis of where and how they live, there is an evident risk that people will find it harder to escape from more challenging life situations, since those with less favorable stereotypes are more likely to be prosecuted than those with more favorable profiles, thus reducing social mobility.

An additional issue is that the black box nature of the HART system, coupled with the complexity of the 850 million data points it draws on, will inevitably make it very hard for police officers to challenge its outputs. They might disagree with its decisions, but in the face of this leading-edge AI-based approach, it would take a very self-assured and experienced officer to ignore a HART recommendation to prosecute, particularly with the risk that the person might re-offend. It is much more likely that officers will take the safe option and accept the HART system's recommendations, whatever they think. As a result, an essentially inscrutable black box will be making critical decisions about a person's life, based in part on where they live, how big their garden is, and whether they are called "Stacey" or "Terrence".

Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+