Subscribe to the Free Future Newsletter
Or get immediate notifications of new posts via Substack
Free Future home

In 2024 the ACLU released a white paper explaining why we didn’t think police departments should be permitted to use AI large language models (LLMs) like ChatGPT to help officers write police reports. These products, most prominently Axon’s Draft One, use AI to process the audio from police body camera recordings and write up a first draft report. We pointed out that there are numerous problems with this concept that could distort officers’ tellings of an incident or interaction.

There is another argument against this technology, however, that I have not seen articulated anywhere: the companies that sell this form of AI assistance to police are likely to have a strong interest in the actual content of police reports — specifically, in ensuring that certain details about a police technology sold by the same vendor do not appear or feature prominently in those reports, that they only appear in a positive light, or other distortions of what really happened.

Several years ago, ALPR company Vigilant Solutions, which has a contract to share license scans with ICE, was embroiled in controversy in California, which has a ban on the sharing of resident location information with ICE. Acting years before the current firestorm over Flock, my colleagues at the ACLU of Northern California warned about the use of the technology by ICE and through a FOIA lawsuit obtained over 1,800 pages of documents shedding light on the technology and how it was being shared and misused.

Amid the controversy (in which Vigilant repeatedly denied and evaded charges that it was sharing data with ICE), the company apparently decided that it wanted to downplay the role its ALPR technology was playing. Dave Maass of the Electronic Frontier Foundation found a statement buried in the ACLU’s document trove — a part of a Vigilant training slideshow — in which the company explicitly recommended that police leave mentions of license plate scanners out of their reports whenever possible. The slideshow tells officers [emphasis added]:

Reporting the use of LPR rests with each agency that utilizes the technology. While we never try to hide anything from the court, law enforcement is allowed to keep processes out of a report if reporting them will teach offenders how to defeat future investigations. If the use of the LPR equipment can be left out of the report, that is what is recommended…

In short, a top ALPR company appears to have had an interest in keeping ALPR out of police reports, and encouraged police to do so. We have seen other companies and their government clients trying to hide the use of controversial technologies, such as cell-phone trackers, including through the notorious law enforcemement strategem of “parallel construction.”

In 2019, Vigilant was acquired by the police technology giant Motorola Solutions. Motorola’s products include not only those provided by Vigilant and their commercial ALPR aggregator Digital Recognition Network (DRN), but also police body and dashboard cameras, 911 and dispatch services, surveillance video cameras (through its acquisition of camera giant Avigilon), face recognition products, and real-time crime center systems.

And it also sells an AI police report product.

What that means is that Motorola may not be limited to merely encouraging police officers in training slideshows to omit mention of license plate scanners in their police reports; they could also nudge their AI to do that work for them. Or to ensure that various other technologies that might surface in police reports are spun in directions helpful to the company even where unjust for the accused.

Axon has incentives at least as strong as Motorola’s to use their AI in this way. Like Motorola, Axon sells license plate readers, surveillance cameras, 911 services, body cameras, and “real time crime center” software; Axon also sells drones, and the highly fraught Taser devices, which are part of numerous controversial police interactions. And of course, Axon sells what appears to be the dominant AI police report-drafting software.

This problem is all the worse because the AI used in Axon and many other companies’ products is so opaque. When it comes to a product like Axon’s Draft One, we know little about the (most likely shifting) models that it uses for transcription and report drafting, any fine-tuning or other post-training that Axon does to make the foundation model more specialized in writing police reports, or the prompt that instructs the LLM model to generate such reports. And as EFF has convincingly explained, Axon has intentionally designed Draft One to avoid providing transparency about the role of AI in the generation of police reports.

Is this scenario far-fetched? I doubt that any of these AI-report companies are doing this yet. But let's remember the diesel emissions scandal in which Volkswagen and other car companies used their control over computer code to secretly advance their own self-interest (in that case by detecting when a car was undergoing an emissions test in order to cheat on those tests). The fact is, AI report drafting companies are not necessarily disinterested parties in what gets included in a police report, what gets left out, and how things are described. And because AI is so opaque, companies have plenty of leeway within which to spin their AI to secretly serve their own interests. In short, they have both the motive and the means.

In the end, the very existence of this possibility is a reminder of the opacity of AI combined with the high degree of control that it gives the company that controls it. And how inappropriate that is in a criminal justice context.