After the ‘A’ Level exams fiasco, Zeeshan Ali reveals how a reliance on technology and AI is reinforcing prejudice on the streets
Recent weeks have witnessed a growing outcry against the state’s use of algorithms to predict the exam results of students across the country at the detriment of those predominantly from disadvantaged backgrounds.
Teenagers who were predicted A* and A grades found themselves receiving Bs and Cs without any evidence justifying the downgrades and their teachers being left just as perplexed. Patterns, however, quickly began to emerge.
The keen-eyed noted that students from private schools who lived in affluent areas were predicted better grades by the algorithm compared to counterparts from disadvantaged backgrounds. It soon became apparent that, within the operation of the algorithm, bias had inadvertently seeped in.
Following widespread anger, the Government sought to abandon the use of the particular algorithm (though not entirely), and instead rely on the judgement of teachers and schools to award grades.
Whilst it is tempting to dismiss this scandal as a one-off, the unethical use of automated processes by the state is quickly becoming the new norm.
Sectors such as healthcare and hospitality are undergoing a dramatic transition to reap the benefits of algorithms and artificial intelligence (AI), with efficiency rates improving in orders of magnitudes. Other sectors, however, are adopting algorithms at the detriment of the public, and in particular minority communities – such as police forces’ implementation of AI-based systems without proper consideration of the outcomes.
Gangs Violence Matrix
One example of the police’s questionable use of AI-based systems is the “gangs violence matrix” (GVM), which was introduced soon after the London Riots in 2011. The Metropolitan Police describes the GVM as an “intelligence tool to identify and risk-assess gang members across London who are involved in gang violence”.
The AI-system works by maintaining a database of names of people who have previously come into contact with police and partner agencies, and determining the level of ‘threat’ they pose by their network of friends and acquaintances. The flawed nature of GVM is the consequence of the ‘guilt-by-association’ nature of the system, and a biased data-set being introduced by an institution shown to be systemically racist.
In one telling incident, Bill, a young black man, was repeatedly stopped by the police without ever having committed any crime or done anything to arouse reasonable suspicion. Bill’s initial interaction with the police was at the age of 11 when he was stopped and searched. When he asked why he was being searched, one of the police officers replied: “Because I want to”. The police’s interest in him only grew over the years, as the GVM highlighted that Bill’s network was worthy of such interest. By the time he was 14, Bill was being arrested sometimes more than once a week – again without ever being charged.
Automating state processes in contexts demonstrated to be structurally racist risks further cementing structural bias.
The systemic racism of the Metropolitan Police resulted in Bill being cast into the eyes of the police, but it was the algorithm that reinforced the idea that he was a threat. In essence, the data-set provided to the matrix disproportionately represents individuals from vulnerable communities and the network-based threat modelling further justifies the increased securitisation of particular communities.
Bill’s sole ‘crime’ was to be living in a poor area, knowing people who were engaged in criminal behaviour, and being a victim of an unethical stop and search.
In other incidents, people who had shared videos of grime or drill music were deemed to be expressing ‘gang-affiliation’ thus warranting their addition to the GVM and being considered a ‘threat’.
Problematic Surveillance
The Metropolitan Police is also increasingly normalising the use of facial recognition technology which aims to be “intelligence-led” and deployed to “specific locations”. The danger for minority communities is stark.
The force states that it is “using this technology to prevent and detect crime by helping officers find wanted criminals”. It also states that the system is robust and that errors occur once in every one thousand cases. However, an independent review (commissioned, and later dismissed, by Scotland Yard) stated that the rate of false positives was likely to be four in every five cases.
The review also added that, whilst the Metropolitan Police states that the technology would only be used to identify individuals that were “wanted” (in itself an ambiguous term), the data that was being utilised by the technology was at times significantly out of date. As such, individuals that had already been dealt with by the courts, and who were not wanted for offences outlined in the data-set, were being s