An audio-based software system that helps police departments detect outdoor gunshots remains a mystery in many ways. Called ShotSpotter, this technology was developed by the Silicon Valley-based company SST Inc. and has been implemented for years by police departments across the United States. But the scores of cities that use this system rarely give the public solid answers to a few basic questions: How often does it lead to arrests or successful prosecutions? Where in the areas it covers does it identify the most gunshots? What other, more subtle or long-term, effects might it have on crime.

ShotSpotter’s effectiveness has been difficult to assess, in part due to a complex array of factors that influence crime, but also because SST does not publicly release detailed data its system generates, but instead plans to sell it to researchers.

Data made available by Forbes in November 2016 sheds some light on ShotSpotter’s impact in Milwaukee, which started using the system in 2010 and expanded it to more neighborhoods in 2014. The subsequent two years saw an increase in fatal shootings and other homicides in the city, including ongoing instances of stray bullets killing children and an apparently racially motivated murder of a Hmong couple and Puerto Rican man.

Forbes staff reporter Matt Drange investigated whether or not ShotSpotter helps reduce gun crime and if SST CEO Ralph Clark’s intention to sell its data is a sound revenue strategy. Forbes also examined ShotSpotter and police dispatch data from police departments in seven cities — Brockton, Mass.; East Palo Alto, Calif.; Kansas City, Mo.; Milwaukee; Omaha, Neb.; San Francisco; and Wilmington, N.C.— and made this information publicly available for analysis.

SST insists that data its ShotSpotter software generates is not subject to open records laws. The company’s contracts with police departments often reserve its ownership of that data, even though taxpayers foot the bill for these systems. However, a police department’s records of dispatching officers to follow up on ShotSpotter alerts are a matter of public record.

The Milwaukee Police Department provided Forbes a list of 10,285 incidents from between Jan. 1, 2013 and Sept. 29, 2015 in which officers were dispatched to follow up on gunfire alerts generated by ShotSpotter. The records include the times and dates of the dispatches, as well as codes that indicate the results officers reported. They do not include location information, though, as the department redacted these records’ addresses. SST and police departments have often been guarded about geographic information related to ShotSpotter, saying that they want to protect the locations of the system’s microphones, which are often placed on rooftops with the cooperation of private property owners.

Limited as it is, the data Milwaukee released to Forbes does offer a useful point of comparison. In August 2014, Milwaukee expanded the geographic footprint ShotSpotter covered from around 3 square miles up to about 11, with the system in place on the city’s near south side and central north side. The Milwaukee Journal Sentinel reported this expansion in Oct. 2014, but Milwaukee police spokesperson Sgt. Tim Gauerke confirmed that expanded sensors were up and running as of August of that year.

After its August 2014 expansion, Milwaukee’s ShotSpotter system picked up far more gunshots than previously, but data do not show a corresponding increase in the number of arrests and citations resulting from those alerts.

During the 33-month period the records cover, the vast majority of these cases — 7,027, to be exact — ended with officers reporting a code C15, “Unable to Locate.” In 172 cases, they reported making an arrest, though it is not clear how many of those arrested were actually charged with firearms-related offenses. In 10 cases, an officer issued a municipal citation. And three ShotSpotter alerts led to police handing a domestic-violence situation.

“Unable to Locate” results dominated the ShotSpotter-related dispatches both before and after expansion. However, post-expansion, reports with a C8 notation did increase. That code simply means “filed,” which Gauerke said can indicate an officer was able to conduct some follow-up activities, like gathering casings or talking with residents near the gunshot location.

Arrests and citations increased after the expansion as well, but not very much, especially given that the ShotSpotter system had nearly quadrupled its coverage area. From Jan. 2013 through July 2014, ShotSpotter alerts led to an average of 5.2 arrests per month. From Aug. 2014 through Sept. 2015, that average increased to 5.5 arrests per month.

The role of arrests as a metric

SST has said that its outdoor microphones and software have become more accurate at distinguishing gunshots from other loud, abrupt noises like fireworks or backfiring cars. Milwaukee’s data backs this up in at least one sense — it shows massive spikes from 2013, 2014, and 2015 on New Year’s Day, when celebratory gunfire is a known problem, but not around the Fourth of July.

The company contends that asking how many people arrested by police following up on ShotSpotter alerts is not the best question. CEO Ralph Clark told WisContext in May 2016 that the system should also be assessed on subtler, longer-term effects it may encourage, like nudging more officers to interact with residents in areas where shootings are common — especially as these tend to be neighborhoods where many residents don’t necessarily trust cops. Additionally, the system may aid inter-agency information sharing efforts.

Officers following up on a ShotSpotter alert necessarily aren’t likely to catch a suspect — people who commit gun crimes aren’t known for lingering around at the scene — but they might be able to find some shell casings and enter them into the National Integrated Ballistic Information Network, which tracks the distinctive markings individual guns leave on the casings when they fire. That database is operated by the U.S. Bureau of Alcohol, Tobacco, Firearms and Explosives, and law enforcement agencies across the country use it in an effort to identify shooters.

ShotSpotter has been used in prosecuting gun crimes in Milwaukee, said Milwaukee County Assistant District Attorney Kent Lovern. The system’s ability to pinpoint the geographic locations of gunshots, and occasionally recorded audio of the shots themselves, can help prosecutors illustrate the crime for a jury.

“We consider it to be a valuable investigative tool because it gives us a real-time location for where shots are fired,” he said. Courts in some states have argued over whether ShotSpotter information should be admissible, but Lovern hasn’t found that to be an issue in Milwaukee.

“I don’t think we’ve run into any admissibility problems with it,” he said.

Ashlee Sherrill, a spokesperson for the ATF regional office in St. Paul, said the agency could not comment on how much ShotSpotter contributes to the gathering of evidence through the NIBIN database.

“Those who subscribe to the service are responsible for how that data is entered into NIBIN and should be able to talk about how those entries help their investigations,” Sherrill said.

Ralph Clark wants ShotSpotter to be evaluated not on just immediate and quantifiable results like arrests, but also by its longer-term contributions to evidence, prosecutions and police-community relations. In a December 2016 interview, he said it’s hard to tell how much ShotSpotter has helped lead officers to forensic evidence that ends up in NIBIN.

“That’s mixed because not every department has the process or maybe even resources to effectively give ShotSpotter credit for its NIBIN entries,” Clark said.

SST also prepares forensic reports for police and prosecutors (the company provided a generic example), and it staff have served as expert witnesses at trials. The company does have clearer data on how often it provides those resources, but only divulged some of this information.

SST spokesperson Liz Einbender said that as of Dec. 19, the company had prepared 432 forensic reports in 2016. She said police usually request these reports from ShotSpotter directly as they’re preparing a case.

“We do get requests from the DAs directly in far less frequency,” Einbender explained.

Each report details the time and location of an individual shooting picked up by ShotSpotter’s microphones, as well as details like how many shots were fired in a specific incident.

Clark said he hopes to staff up on the forensic side of SST’s operations.

“We are at capacity,” he said. “We’re actually looking at doubling the capacity of our forensic resources.” This suggests there is demand for ShotSpotter’s information in the courtroom.

The actual audio ShotSpotter records can also help prosecutors create the narratives they need to sway a jury. Clark offered this hypothetical: A defendant admits that he shot a victim, but pleads self-defense, and hopes to be charged with manslaughter rather than face a more severe penalty for premeditated murder.

“They’re not debating the facts about John killing Joe, but John is saying, ‘Hey, it might have been self-defense, he shot at me first,'” Clark said.

The number and locations of shots ShotSpotter identifies could prove or disprove the hypothetical defendant’s story. Additionally, if the ShotSpotter record shows an initial volley of shots, then a pause followed by another shot, that could suggest John not only shot Joe, but deliberately fired a final shot to finish Joe off.

The actual audio ShotSpotter records can be used in the courtroom as well, Clark noted.

“I think when jurors can actually hear real gunfire, it’s theatrical and it can really give people a visceral feel for what might have taken place,” he said.

ShotSpotter’s subtler impacts

Daniel Lawrence, a research criminologist at Research Triangle Institute who is working on a long-term study of ShotSpotter’s use in multiple cities, has found what might be evidence of a persuasive effect on police officers. An analysis from several cities (including Milwaukee) found that officers are more likely to take a ShotSpotter alert seriously, and respond quickly, than when a citizen calls in a gunfire complaint.

“In gunfire cases where an officer responded, SST cases had an 83 percent response rate, while shooting/shots [reported via 911 calls] had a 70 percent response rate,” Lawrence said. “In terms of response time, SST cases have a median response time of 4 minutes and 36 seconds, while shootings/shots [reported via 911 calls] have a median response time of 8 minutes and 59 seconds.”

So far, Lawrence hesitates to conclude that ShotSpotter is having a big influence on police officers’ behavior. But he can see why they might consider the system’s alerts more reliable than a call.

“SST alerts provide an exact location where the gunfire was detected, while a [call from a resident] is an approximate area from where the caller is located,” he explained. “And muzzle blast noise can travel long distances. I think that these factors likely influence an officer because they may feel more likely to obtain evidence from a SST call as opposed to a 911 call.”

Not many people have conducted independent, empirical research about ShotSpotter or using ShotSpotter data — not even the group Milwaukee created to study its homicide and gun-violence problem.

Mallory O’Brien, director of the Milwaukee Homicide Review Commission, said she has not yet incorporated ShotSpotter into efforts to analyze and prevent crime. However, she added that the HRC is hoping to find an innovative use for ShotSpotter data in the coming months.

O’Brien said the HRC and Milwaukee police department are testing a way to combine ShotSpotter data with research and treatment efforts aimed at helping people traumatized by gun violence. With a more detailed picture of where the most persistent gun violence is happening, the city might be able to target mental-health resources more effectively.

“We all talk about PTSD, but I think that’s the way we can really document the kind of consistent trauma that someone’s experiencing in their neighborhood,” O’Brien said.

Missing measurements

Despite ShotSpotter’s potentially innovative uses and the anecdotal evidence of its benefits, there won’t be a full accounting of its effectiveness unless SST and its client departments offer greater transparency, said Jennifer Doleac, assistant professor of public policy and economics at the University of Virginia’s Batten School of Leadership and Public Policy. The biggest barrier to independent research about ShotSpotter is that SST owns the data it generates and wants to sell it rather than sharing this information with the public.

“I think there’s a lot of interest in this and a lot of frustration that the data aren’t available,” Doleac said.

Additionally, police departments — and other clients, like a growing market of schools and colleges worried about on-campus shootings — feel under pressure to address public concerns about gun violence. But they “don’t have the luxury of sitting around waiting for evidence,” Doleac said. If those customers aren’t asking for detailed proof that ShotSpotter helps before buying in, there’s no real pressure on SST to share it.

“If police departments aren’t demanding evidence that this thing works, there’s no real upside to them having a study done,” she added.

When asked what Milwaukee police were doing to evaluate ShotSpotter’s effectiveness, Gauerke pointed out that the department is participating in Lawrence’s study, but did not mention any other evaluation efforts. As for how often ShotSpotter yields evidence for a particular case or for the NIBIN database, Gauerke said the department isn’t currently tracking that.

“Current record management systems and district attorney systems do not track outcomes nor do they have the ability to integrate,” Gauerke said. “Results would have to be generated by hand, such as reading ShotSpotter related incident reports and tracking suspects’ progress in the criminal justice system.