Estimating Discrimination in Sentencing:
Distinguishing between Good and Bad Controls

Jose Pina-Sánchez

Can we estimate sentencing discrimination?

You did not control for […]

If you believe on the principle of individualisation, the list of controls is infinite

Missing legal factors can be ok

Diminishing returns of explainability for every additional factor
Confounding bias requires association with the focal variable
Sometimes the observed disparities are just too big

Using the top-20 rather than the top-10 legal factors does not affect predictive accuracy in sentencing. This is from a study exploring shoplifting offences in the magistrates’ court in England and Wales. When we use the top-10 factors influencing the probability of receiving a prison sentence we can classify correctly 80.1% of cases. Can you guess how many more cases could we predict correctly after doubling the number of legal factors to introduce the top-20 case characteristics? 81.2%. Even if there was an important factor affecting sentence severity that we failed to control for, that is just a necessary, not a sufficient condition. Confounding bias requires association with the focal variable abd for that association to point in a given direction E.g. In England white offenders have longer previous convictions than anyone else The needed strength of the missing factor with both the cause (race, gender, etc.) and the effect (sentence severity) is inversely proportional to the size of the observed disparities. We can determine the necessary dimension of the missing factor to render them non-significant We know smoking causes cancer even though we have no experimental evidence. The corollary from this first part of my talk is that confounding bias might always be present, making the identification of the precise degree of discrimination unidentifiable, however, there are instances where, under certain conditions, we can claim confidently enough that discrimination is present.

What if we were ‘overcontrolling’?

A more realistic representation

Target estimand: the effect of judicial prejudice on sentencing
Proxy estimand: the effect of offender’s socio-demographic trait on sentencing, not mediated through warranted differences

Judicially-defined case characteristics

Pick your poison

Other offender characteristics

Estimating discrimination requires accuracy

Judge characteristics

Useful to explore the origin of disparities, but that is a different question

Court/area characteristics

Another set of endogenous factors

A new modelling framework

Step-1: Define the target estimand

State the specific effect you seek to estimate

– if interested in direct discrimination then say so
Clearly state the population of interest

– decision-makers and decision stages under analysis

The concept of unwarranted disparities is really vague. If we care about discrimination then we say so. And if we mean direct discrimination then we say so too. If you prefer the potential outcomes framework, another way to put this is as the average difference in sentence severity were we to modify offender’s race. Crucially, we need to identify the decisions to be studied to avoid selection bias, and the decision-makers under analysis as that will have implications regarding what we consider to be endogenous case characteristics. On decisions, the issue of jurisdictions where both decisions of conviction and sentence severity are imposed by the same body of judges. There, if we are exploring discrimination in judicial decision-making by focusing just on the sentences imposed ignoring those who were found not guilty, then we are most likely going to have a problem of selection bias. On decision-makers, in the US is quite normal to sentence through guilty negotiations between prosecutors and offenders. If we want to explore those sentences too, then we need to expand our list of judicially-defined case characteristics. It is not anymore case characteristics which are influenced by a potential judicial prejudice, but now also those influenced by prejudice from prosecutors.

Step-2: Operationalise sentence severity

Do you consider adjudication?
Do you consider non-custodial sentences?
Is sentencing a one- or two-stage process?

Step-3: List and classify the necessary controls

Identify all the relevant legal factors
Classify them as judicially-defined or not
Identify non-legal factors influencing sentence severity
Note which ones are missing

Step-4: Estimate model uncertainty

Regarding endogenous factors

– compare models with and without them
Regarding unobserved legal/non-legal factors

– E-value (VanderWeele & Ding, 2017)

– Robustness value (Cinelli & Hazlett, 2020)

US federal courts: controls

US federal courts: findings

Conclusion

The precise level of discrimination in sentencing is unkowable
Yet, in certain contexts, we can establish its presence/absence
We need to embrace uncertainty
The best way to fight scientific nihilism

Next Steps

Open research

– reduce researchers’ degrees of freedom through pre-registrations

– eliminate publication bias through registered reports
Sensitivity analysis for unobserved mediators

– combine different types of model uncertainty
A comprehensive framework for the estimation of discrimination

– beyond sentencing, applicable to any decision-making process