Data aren’t impartial
Algorithms seem objective. People have their biases, but technology doesn’t – or at least, that’s what we believe. Unfortunately, it’s not that straightforward. Software depends on the data that people feed into it. And those data aren’t impartial by any means.
A study carried out by two researchers in the US, Kristian Lum and William Isaac, is a case in point. They applied a predictive policing algorithm used in US police force software to the Oakland police department’s drug crime databases. And what did they find? The software predicted that future drug crimes would occur in areas where police officers had already encountered many drug crimes. The researchers then added public health data on drug use. Based on the new data, the software predicted that drug crimes would also occur in many other parts of the city.
Fairness at risk
The police databases turned out to have a blind spot, in other words, and one that would have caused the Oakland police to overlook crimes in certain neighbourhoods. Self-learning software only makes it more likely that they would have continued overlooking these crimes in the future. Going by the software’s advice, police officers would have only patrolled neighbourhoods with which they were already familiar. They would have recorded the crimes that occurred there in their database. The software would have then used the database to make subsequent predictions and would have overlooked any crimes unknown to the police in neighbourhoods with only minimum policing. Not only would this have put the effectiveness of policing at risk, but it was also unfair: the police would be addressing crime in one neighbourhood but not in another.
Attempts to circumvent this problem produce a human rights paradox: government either has databases with blind spots, or it links up many different databases, which is undesirable for privacy reasons.
Black box algorithm
One of Amnesty’s other major concerns is the lack of transparency regarding what computer systems do with their data input. It is often impossible to see how the system reaches a certain outcome, making it difficult to test its accuracy. Users also don’t notice if the system has made a mistake.
The lack of transparency has negative implications for the rule of law. Once an algorithm has identified you as a potential risk, how can you prove you are not if you have no insight into the rationale behind that assumption?
Who is responsible?
That brings us to the third important risk: who do we hold responsible for decisions, especially wrong decisions? If we allow algorithms to take decisions for us, either directly or because we base our own decisions on their advice, then who is ultimately responsible for that decision, the software developer or the person who uses the software? And who monitors whether the advice issued by the algorithm is actually correct? Where do victims obtain justice when no one can tell them who is responsible? These are questions that we need to discuss.