Estimating Discrimination in Sentencing:
Distinguishing between Good and Bad Controls

Jose Pina-Sánchez, Melissa Hamilton & Peter Tennant

Thanks for this invitation. It is my first time in Prague, my father brought me here when I was a kid but that was before of the fall of the Soviet Union and I do not remember anything. It is a real honour, you have something truly special going on in this department of Law and Economics (an absolute power house in empirical legal research. Not many other centres out there at your level in Europe.), and it I feel extremely privileged and grateful that you are interested in listening what I have to say on sentencing disparities. This paper I am presenting today is about how to choose good controls, that is, how to create comparable groups of offenders, a rather specific question, but that sits within a broader question, which is quite important, namely how to estimate discrimination in sentencing. In particular, how to do that using real sentences. What is known as observational data. I distinguish that from what is known as experimental data as in my view that is a dead end. We cannot intervene to change the race or gender or nationality of the offender, as it is common for example in studies exploring discrimination in the labour market. Nor do I think that vignette studies are that useful. Their ecological validity is highly questionable.

Can we estimate sentencing discrimination?

In a given jurisdiction, can we take a representative sample of sentences and estimate the level of discrimination? What do you think? Rise your hand if you think we can. Now those who thing we cant. Why not? In this talk I will argue that we can’t establish the exact level, that question is unidentifiable, but we can establish where is the most likely level of discrimination, and in certain contexts, establish with enough confidence whether discrimination is present or not.

You did not control for […]

• If you believe in the principle of individualisation, the list of controls is infinite

The perennial problem: always pointed out to studies exploring disparities using observational data. You missed some factor influencing sentence severity, therefore you cannot claim whether the differences observed when comparing men to women, blacks to whites, or nationals to foreigners, are not due to an imbalance in the offences or offence characteristics defining those two groups. As a side-note, in this paper we employ DAGs, how many of you are familiar with this technique? It is nothing more than representing visually the causal relationships that you expect to be present or govern a particular data generating mechanism, in this case the sentencing process. The advantage of these techniques is transparency but also facilitate the identification of quantitative bias. This is a technique that is now mainstream in Epidemiology and environmental sciences, computer science and other fields, but for some reason is one that Econometricians do not seem to like, and no one has used in sentencing beyond seeing it as a a form of mind map, a rather elementary use. If use DAGs to represent the decision-making process and the bias that stems from the traditional view on the need to control for case characteristics we can do it like this: Case characteristics refer to the legal factors that are expected to affect sentence severity. Offender’s socio-demographic trait refer to the specific personal trait under investigation (race, gender, social class, etc.) Demonstration of how the bias can creep in using the DAG. The corollary is that reviewers, critiques, will often point at factors that affect sentence severity that you forgot to include. Most likely there will always be legal factors that we fail to observe. In particular those that are difficult to measure, such as offender’s dangerousness, or potential for rehabilitation. Some have even said that the principle of individualisation makes this list potentially infinite.

Missing legal factors can be ok

• Diminishing returns of explainability for every additional factor

• Confounding bias requires association with the focal variable

• Sometimes the observed disparities are just too big

Using the top-20 rather than the top-10 legal factors does not affect predictive accuracy in sentencing. This is from a study exploring shoplifting offences in the magistrates’ court in England and Wales. When we use the top-10 factors influencing the probability of receiving a prison sentence we can classify correctly 80.1% of cases. Can you guess how many more cases could we predict correctly after doubling the number of legal factors to introduce the top-20 case characteristics? 81.2%. Even if there was an important factor affecting sentence severity that we failed to control for, that is just a necessary, not a sufficient condition. Confounding bias requires association with the focal variable abd for that association to point in a given direction E.g. In England white offenders have longer previous convictions than anyone else. The needed strength of the missing factor with both the cause (race, gender, etc.) and the effect (sentence severity) is inversely proportional to the size of the observed disparities. We can determine the necessary dimension of the missing factor to render them non-significant We know smoking causes cancer even though we have no experimental evidence. The corollary from this first part of my talk is that confounding bias might always be present, making the identification of the precise degree of discrimination unidentifiable, however, there are instances where, under certain conditions, we can claim confidently enough that discrimination is present.

What if we were ‘overcontrolling’?

I am going to bring up a 2nd problem, which is the focus of my paper. Should we be controlling for all legal factors?

A more realistic representation

• Target estimand: the effect of judicial prejudice on sentencing

• Proxy estimand: the effect of offender’s socio-demographic trait on sentencing, not mediated through warranted differences

A more realistic representation of sentencing. We have added two nodes that are latent (unobserved), judicial prejudice and deliberation. Deliberation captured the moment where all the information is put into the judge’s head. And judicial prejudice refers to the hypothetical prejudice that might arise when the judge finds out the offender’s race or gender or social class (or whatever you are interested on). We do not see the deliberation process or any hypothetical prejudice, which is why they are represented as ovals, and why even though our target estimand is the effect of judicial prejudice on sentence severity we need to use a proxy estimand. This is why we changed the colour of case characteristics to red, a confounder, if we look at judicial prejudice, but a mediator if use the offender socio-demographic trait as a proxy.

Judicially-defined case characteristics

• Pick your poison

Not all case characteristics are exogenous to the judge. Think of mitigating factors like showing remorse or the offender having good character. Same graph, only now I have split case characteristics in two. Explain the problem of post-treatment bias. Explain the problem of confounding bias that lingers if we do not control for judicially-defined case characteristics. Criminologists tend to control for these factors, econometricians have started to choose not to control for them. Pick your poison kind of situation. What would you do? Our view is to do both, run models with and without this kind of endogenous factors and take them both as potentially valid. Corollary, even if we observe all legal factors, the effect of judicial prejudice will always be unidentifiable.

Other offender characteristics

• Estimating discrimination requires accuracy

Judge characteristics

• Useful to explore the origin of disparities, but that is a different question

Not so helpful when estimating the presence of discrimination.

Court/area characteristics

• Another set of endogenous factors

Another chance to pick your poison. We face another dilemma. This is not one that has yet been openly discussed, as far as I am aware. Econometricians like to use court fixed effects, criminologists like to throw in court or area characteristics.

A new modelling framework

In conclusion, when trying to estimate discrimination in sentencing, there are multiple forms of bias that can arise as a result of controlling or failing to control for key legal and non-legal factors. And these go far beyond the simplistic notion of controlling for all legal factors to avoid confounding bias. And often there is no answer, the effect of judicial prejudice cannot be identified. So what to do? Beyond listing the types of bias present in the literature, in this paper we also try to be constructive. We try to facilitate a way to reflect the model uncertainty stemming from the choice of controls in models estimating discrimination in sentencing.

Step-1: Define the target estimand

• State the specific effect you seek to estimate

  – if interested in direct discrimination then say so

• Clearly state the population of interest

  – decision-makers and decision stages under analysis

The concept of unwarranted disparities is really vague. If we care about discrimination then we say so. And if we mean direct discrimination then we say so too. If you prefer the potential outcomes framework, another way to put this is as the average difference in sentence severity were we to modify offender’s race. Crucially, we need to identify the decisions to be studied to avoid selection bias, and the decision-makers under analysis as that will have implications regarding what we consider to be endogenous case characteristics. On decisions, the issue of jurisdictions where both decisions of conviction and sentence severity are imposed by the same body of judges. There, if we are exploring discrimination in judicial decision-making by focusing just on the sentences imposed ignoring those who were found not guilty, then we are most likely going to have a problem of selection bias. On decision-makers, in the US is quite normal to sentence through guilty negotiations between prosecutors and offenders. If we want to explore those sentences too, then we need to expand our list of judicially-defined case characteristics. It is not anymore case characteristics which are influenced by a potential judicial prejudice, but now also those influenced by prejudice from prosecutors.

Step-2: Operationalise sentence severity

• Do you consider adjudication?

• Do you consider non-custodial sentences?

• Is sentencing a one- or two-stage process?

Step-3: List and classify the necessary controls

• List all the relevant legal factors

• Classify them as judicially-defined or not

• List non-legal factors influencing sentence severity

• Note which ones are missing

Relevant legal factors or case characteristics are those listed in the criminal code. A lot easier to do if you consider offence-specific models, rathen than general samples of offences.

Step-4: Estimate model uncertainty

• Regarding endogenous factors

  – compare models with and without them

• Regarding unobserved legal/non-legal factors

  – E-value (VanderWeele & Ding, 2017)

  – Robustness value (Cinelli & Hazlett, 2020)

There is not perfect combination of controls, therefore there is not a perfect answer, but we can still estimate the model uncertainty that arises when we control or fail to control for case characteristics that are subjectively defined by the judge. For example, we can run all possible combinations and uncover how much uncertainty there is as a result of that problem. We should also aim to quantify the uncertainty stemming from having missed relevant legal or non-legal factors. This is not the key contribution of our paper. There is a growing number of options to do this. Here we have recommended two simple options that anyone could use. Rather than estimating the uncertainty around our estimates, these two techniques indiacte how “strong” should the missing factor be to render our observed estimates non-significant. How much it should affect sentence severity and how much more common should be in the group that we hypothesise is discriminated against.

US federal courts: controls

US federal courts: findings

5 endogenous factors, leading to 31 possible combinations (without replacement)

Discrimination against black offenders?

• 6.4% to 15.7% longer sentences than white offenders

• Missing key legal factors

  – drugs quantity; serious bodily injury

• Robustness value: 2.3% to 4.1%

• There is discrimination in sentencing against black offenders, conditional on …

  – missing aggravating factors not being more prevalent in black than white offenders

At this point it is entirely a subjective decision how to interpret that. In our original version of the manuscript we indicated that in our view that was not a strong enough condition to claim discrimination in sentencing. However, one of our reviewers strongly objected to that and indicated: “how we address that comment will be key in their decision of publication”, so we changed and it now we say that could be seen as discrimination. Beyond this particular application, the opint I want to make is that now we have presented transparently our assumptions and tried to quantify them, or to provide an estimation of the uncertainty surrounding our findings, so readers can decide for themselves.

E&W magistrates: controls

E&W magistrates: findings

20 endogenous factors, leading to over a million possible combinations, so we just took a random sample of 100 of those combinations.

• Male offenders much more likley to receive a custodial sentence

• Missing non-legal factors: race, nationality, employment, …

• Yet, the E-value is too high: 1.84

• We conclude there is discrimination against male offenders

Male offenders 26% to 37% more likley to receive a custodial sentence The number of previous convictions reduces the estimated gender disparity by 15.2 percentage points when included in the model, while the presence of injury to the victim increases the estimated disparity by 3.9 percentage points. We do not observed dangerousness or cooperation with the prosecution, but none of those is relevant to shoplifting.

Conclusion

• The precise level of discrimination in sentencing is unkowable

  – yet, in certain contexts, we can establish its presence/absence

• Real-world implications

  – unequivocally demonstrating discrimination can lead to action

  – perceptions of discrimination - real or not - lead to alienation

• We need to embrace uncertainty

  – the most effective way to push against scientific nihilism

Academic research, the research that we all do, often has real-world implications. Wrong conclusions risk alienating groups of the population who will be told they are discriminated against. Failing to detect real discrimination leaves unchecked a terrible injustice. We need to step up our game. Avoid weasel strategies of adding caveats at the end of the study, using terms like unwarranted disparities, and so on. We need to be much more direct at what our research aim is, and equally much more diligent at quantifying the uncertainty around our findings. We should distrust studies presenting a point estimate.

Next steps

• Open research

  – reduce researchers’ degrees of freedom through pre-registrations

  – eliminate publication bias through registered reports

• Sensitivity analysis for unobserved mediators

  – combine different types of model uncertainty

• A framework for the estimation of discrimination

  – beyond sentencing, applicable to any decision-making process

Open science is key to enhance both the reliability and validity of this type of research. Pre-registrations would help a lot in particular when it comes to deciding which factors are considered judicially-defined. None of this matters in the current publishing system. There is publication bias in the literature. Who is familiar with the concept of registered reports? It would be good to make this possible at the EJELS. Whoever does this provides a comprehensive framework for the estimation of discrimination If you know any stats genius who can do this let me know

If we try hard enough, and do not fool ourselves along the way, we can answer pretty difficult questions

I want to finish this talk with an optimist message. Research questions are often harder than we think, but that is no reason to simply give up, embrace scientific nihilism or social constructivism and say there is no truth, or it is unknowable. If we invest in research methods, and keep at it, we can get to crack really hard questions. And maybe contribute to make the world a better a place.