Researchers are Bringing Something Surprisingly Novel to Lawmaking: Rigorous Evaluation

How did you first get interested in data-driven approaches to the law?

I originally got excited about the possibility of using quantitative methods to inform legal questions while I was a practicing lawyer. Among other things, I was doing some cases that involved heavy numbers, such as lawsuits over the 2000 Census. I started examining statistics and really got bitten by the bug.

Fast-forward a couple years, and I’ve gotten through a Ph.D. in statistics. I thought that lots of lawyers would surely want to do RCTs to find out what worked in law, the same as we do in medicine. I thought it was just a matter of bringing it to their attention that there were people like me with quantitative skills. Instead, I encountered disdain, dismissiveness, and skepticism about why anyone would need quantitative techniques to find out what worked.

That is the attitude of too many of the legal profession — experience and what legal elites say are enough to figure out what works. I didn’t believe that. History is littered with consensus professional beliefs that RCTs have gone on to disprove.

Why are RCTs so important? Why isn’t it sufficient to, say, do an empirical study of the recidivism rate of people who are released before trial?

If you want to determine if some practice causes the result you want it to cause, then the RCT is pretty much the only way to answer that question. Because if you don’t do an intervention randomly, then the people who end up in the intervention look nothing like the people who end up in the alternative, and any difference in the outcomes for the two groups can’t be ascribed to the intervention — it could be some other factor.

Randomization is the only way we know to be as sure as we can be that the two groups are comparable in all ways except that intervention.

The comparison to medicine is fascinating. As you’ve pointed out, medicine wasn’t always considered a science.

That’s right. And in law we haven’t even gotten to the point where we think we need the empirical data. We have a long way to go.

Some of the ethical issues involved in performing RCTs on prisoners are analogous to the ethical issues facing physicians, right?

We run into this a lot, even among law professors. How can you possibly randomize legal interventions ethically when the stakes are so high? That’s a coherent objection, but what it means is that you should not be taking any form of modern medicine.

Modern medicine was developed by conducting RCTs on life-saving interventions before we knew they were lifesaving. I would say that it’s because the stakes are so high that we need to randomize. We should be demanding rigorous standards.

Policymakers have no problem rolling out massive interventions without any empirical evidence. But somehow they’re troubled by randomized trials.

This is particularly important in the criminal justice system. We tend to think that people should have a choice about whether to participate in an experiment. But people have no choice when policymakers decide to make a massive policy change, except at the ballot box. In the criminal justice system, interactions between the state and the individual are, by design, incredibly coercive.

When the state arrests somebody, that person has remarkably little freedom over their diet, choice of clothes, communication, ability to express themselves. So, randomization in that environment is absolutely essential to figure out how to do the least harm to further society’s goals.

On the other hand, asking for consent in that setting is problematic, because the person has very few choices. This is a tension we’re still working out.

How has the A2J Lab been able to conduct research given all the resistance to RCTs in the legal community?

It’s basically been an exercise in coalition-building. In our field research projects, we meet all the stakeholders around a particular issue, explain what the knowledge gap is, and how the acquisition of knowledge could be useful for them. We tell them that we’re there to help them solve this problem. Then we run the study. The studies are always slow, just like medical trials. So we have to maintain the coalition over a sustained period of time while we wait for the results.

How many studies are you currently running?

We’ve probably got between one and two dozen that are running in the field, and then around the same number in development. You need to develop a lot of them, because some fail for one reason or another.

Can you talk about some of the studies of pretrial justice interventions currently under way?

We have studies on the effectiveness of everything from risk assessment scores, to the use of social workers as part of a defense team, to the use of text message reminders to help people get to their public defender meetings and court proceedings. We have studies in development that deal with diversion programs and charitable bail funds. Within a couple of months we should probably have a couple of interim reports that we can release.

The resistance of lawyers is one thing, but then you have the resistance of policymakers to evidence-based approaches. Even if you had gold-standard research findings about what works, that might not be enough to persuade some legislators.

That’s a very difficult problem to tackle. The sluggishness of policymakers to adopt policies that have been proven to work is itself an object of study. It’s not that they don’t move; the issue is how to get them to move faster. It takes a while for best practices to be adopted, whether that’s in medicine, law, or governance.

Do you have any studies related to COVID-19 in the works?

We have many exciting things we’re trying to get in the field, on both the criminal and civil side. We’re really interested in studying the effect of holding legal proceedings online — not just jury selection but whole trials. You can even do online proceedings asynchronously, meaning that all the participants in a trial don’t have to be involved at the same time — jurors might be able to watch a trial that has been previously recorded.

The online environment is more manipulable, so you might be able to mask certain things, like the race or class of a defendant. We’re definitely interested in studying that. We’ve also noticed from our existing studies that both crime rates and arrest rates are down across the country during the pandemic.

Why do you think the legal community is uniquely resistant to this approach, given that it’s been adopted in so many other fields?

I’ve spent a lot of time thinking about this, and I don’t have great answers. The idea of professional judgement may have something to do with it. But the same thing is true of medicine, where professional judgement is also valued. The medical profession, nonetheless, understands that RCTs are compatible with professional judgement. Another crass thought is that people who can’t do math in college go to law school.

Or maybe it’s the way we teach law — we teach appellate case law, which encourages people to think about individual cases rather than what’s best across the entire portfolio. But it’s still hard to figure out why lawyers are so disdainful of rigorous empiricism.