Measurement: Crime Data
During my thesis I explored the prevalence of measurement error in retrospective survey data, and the effectiveness of different adjustment methods in adjusting for it. As a member of the Recounting Crime project I was able to apply this measurement error framework to study the ‘dark figure’ of crime. Recently, I’ve been involved in a study led by Dr. David Buil-Gil exploring the reliability of cross-national data on homicides.
Estimating the Reliability of Crime Data in Geographic Areas
Crime data are problematic: Crimes that are never reported undermine its validity and differences in police recording practices affect its reliability. However, the true extent of these problems is not well known, with existing studies suffering from a number of methodological limitations. We examine the quality of police recorded crime data and survey-based crime estimates recorded in England and Wales using a robust latent trait model that effectively represents the competing sources of error. We find that whilst crime rates derived from police data systematically underestimate the true extent of crime, they are substantially more reliable than estimates from survey data. Reliability is lower for violence and criminal damage and is getting worse over time.
Brunton-Smith, I., Cernat, A., Pina-Sánchez, J., and Buil-Gil, D. (2024). Estimating the reliability of crime data in geographic areas. The British Journal of Criminology, 64(6):1347–1361. https://doi.org/10.1093/bjc/azae018
Exploring the Impact of Measurement Error in Police Recorded Crime Rates Through Sensitivity Analysis
It is well known that police recorded crime data is susceptible to substantial measurement error. However, despite its limitations, police data is widely used in regression models exploring the causes and effects of crime, which can lead to different types of bias. Here, we introduce a new R package (‘rcme’: Recounting Crime with Measurement Error) that can be used to facilitate sensitivity assessments of the impact of measurement error in analyses using police recorded crime rates across a wide range of settings. To demonstrate the potential of such sensitivity analysis, we explore the robustness of the effect of collective efficacy on criminal damage across Greater London’s neighbourhoods. We show how the crime reduction effect attributed to collective efficacy appears robust, even when most criminal damage incidents are not recorded by the police, and if we accept that under-recording rates are moderately affected by collective efficacy.
Pina-Sánchez, J., Brunton-Smith, I., Cernat, A., and Buil-Gil, D. (2023). Exploring the impact of measurement error in police recorded crime rates through sensitivity analysis. Crime Science, 12(14):1–8. https://doi.org/10.1186/s40163-023-00192-5
The Impact of Measurement Error in Models Using Police Recorded Crime Rates
Objectives: Assess the extent to which measurement error in police recorded crime rates impact the estimates of regression models exploring the causes and consequences of crime.
Methods: We focus on linear models where crime rates are included either as the response or as an explanatory variable, in their original scale or log-transformed. Two measurement error mechanisms are considered, systematic errors in the form of under-recorded crime, and random errors in the form of recording inconsistencies across areas. The extent to which such measurement error mechanisms impact model parameters is demonstrated algebraically using formal notation, and graphically using simulations.
Results: The impact of measurement error is highly variable across different settings. Depending on the crime type, the spatial resolution, but also where and how police recorded crime rates are introduced in the model, the measurement error induced biases could range from negligible to severe, affecting even estimates from explanatory variables free of measurement error. We also demonstrate how in models where crime rates are introduced as the response variable, the impact of measurement error could be eliminated using log-transformations.
Conclusions: The validity of a large share of the evidence base exploring the effects and consequences of crime is put into question. In interpreting findings from the literature relying on regression models and police recorded crime rates, we urge researchers to consider the biasing effects shown here. Future studies should also anticipate the impact in their findings and employ sensitivity analysis if the expected measurement error induced bias is non-negligible.
Pina-Sánchez, J., Buil-Gil, D., Brunton-Smith, I., and Cernat, A. (2022). The impact of measurement error in models using police recorded crime rates. Journal of Quantitative Criminology, 39(4):975–1002. https://doi.org/10.1007/s10940-022-09557-6
Estimating Crime in Place
Moving Beyond Residence Location
We assess if asking victims about the places where crimes happen leads to estimates of “crime in place” with better measurement properties. We analyze data from the Barcelona Victimization Survey (2015–2020) aggregated in 73 neighborhoods using longitudinal quasi-simplex models and criterion validity to estimate the quality of four types of survey-based measures of crime. The distribution of survey-based offense location estimates, as opposed to victim residence estimates, is highly similar to police-recorded crime statistics, and there is little trade off in terms of the reliability and validity of offense location and victim residence measures. Estimates of crimes reported to the police show a better validity, but their reliability is lower and capture fewer crimes.
Cernat, A., Buil-Gil, D., Brunton-Smith, I., Pina-Sánchez, J., and Murrià-Sangenís, M. (2022). Estimating crime in place: Moving beyond residence location. Crime & Delinquency, 68(11):2061–2091. https://doi.org/10.1177/00111287211064779
Comparing Measurements of Violent Crime in Local Communities
A Case Study in Islington, London
Police-recorded crime data are prone to measurement error, affecting our understanding of the nature of crime. Research has responded to this problem using data from surveys and emergency services. These data sources are not error-free, and data from different sources are not always easily comparable. This study compares violent crime data recorded by police, ambulance services, two surveys and computer simulations in Islington, London. Different data sources show remarkably different results. However, crime estimates become more similar, but still show different distributions, when crime rates are calculated using workday population as the denominator and log-transformed. Normalising crime rates by workday population controls for the fact that some data sources reflect offences’ location while others refer to victims’ residence, and log-transforming rates mitigates the biasing effect associated with some multiplicative forms of measurement error. Comparing multiple data sources allows for more accurate descriptions of the prevalence and distribution of crime.
Buil-Gil, D., Brunton-Smith, I., Pina-Sánchez, J., and Cernat, A. (2022). Comparing measurements of crime in local communities: A case study in Islington, London. Police Practice and Research, 23(4):489–506. https://doi.org/10.1080/15614263.2022.2047047
Adjusting for Measurement Error in Retrospectively Reported Work Histories
An Analysis Using Swedish Register Data
We use work histories retrospectively reported and matched to register data from the Swedish unemployment office to assess: 1) the prevalence of measurement error in reported spells of unemployment; 2) the impact of using such spells as the response variable of an exponential model; and 3) strategies for the adjustment of the measurement error. Due to the omission or misclassification of spells in work histories we cannot carry out typical adjustments for memory failures based on multiplicative models. Instead we suggest an adjustment method based on a mixture Bayesian model capable of differentiating between misdated spells and those for which the observed and true durations are unrelated. This adjustment is applied in two manners, one assuming access to a validation subsample and another relying on a strong prior for the mixture mechanism. Both solutions demonstrate a substantial reduction in the vast biases observed in the regression coefficients of the exponential model when survey data is used.
Pina-Sánchez, J., Koskinen, J., and Plewis, I. (2019). Adjusting for measurement error in retrospectively reported work histories: An analysis using Swedish register data. Journal of Official Statistics, 35(1):203–229. https://doi.org/10.2478/jos-2019-0010
Adjustment of Recall Errors in Duration Data Using SIMEX
It is widely accepted that due to memory failures retrospective survey questions tend to be prone to measurement error. However, the proportion of studies using such data that attempt to adjust for the measurement problem is shockingly low. Arguably, to a great extent this is due to both the complexity of the methods available and the need to access a subsample containing either a gold standard or replicated values. Here I suggest the implementation of a version of SIMEX capable of adjusting for the types of multiplicative measurement errors associated with memory failures in the retrospective report of durations of life-course events. SIMEX is a method relatively simple to implement and it does not require the use of replicated or validation data so long as the error process can be adequately specified. To assess the effectiveness of the method I use simulated data. I create twelve scenarios based on the combinations of three outcome models (linear, logit and Poisson) and four types of multiplicative errors (non-systematic, systematic negative, systematic positive and heteroscedastic) affecting one of the explanatory variables. I show that SIMEX can be satisfactorily implemented in each of these scenarios. Furthermore, the method can also achieve partial adjustments even in scenarios where the actual distribution and prevalence of the measurement error differs substantially from what is assumed in the adjustment, which makes it an interesting sensitivity tool in those cases where all that is known about the error process is reduced to an educated guess.
Pina-Sánchez, J. (2016). Adjustment of recall errors in duration data using SIMEX. Advances in Methodology and Statistics, 13(1):27–58. https://doi.org/10.51936/cspz2183
Measurement Error in Retrospective Work Histories
Measurement error in retrospective reports of work status has been difficult to quantify in the past. Issues of confidentiality have made access to datasets linking survey responses to a valid administrative source very problematic. This study uses a Swedish register of unemployment as a benchmark against which responses from two survey questions are compared and hence the presence of measurement error elucidated. We carry out separate analyses for the different forms that measurement error in retrospective reports of unemployment can take: miscounting of the number of spells of unemployment, mismeasuring duration in unemployment, and misdating starts of spells and misclassification of status. The prevalence of measurement error for different social categories and interview formats is also examined, leading to a better understanding of the error-generating mechanisms that interact when interviewees are asked to produce retrospective reports of past work status.
Pina-Sánchez, J., Koskinen, J., and Plewis, I. (2014). Measurement error in retrospective work histories. Survey Research Methods, 8(1):43–55. https://doi.org/10.18148/srm/2014.v8i1.5144
Decentralization as a Multifaceted Concept
A More Encompassing Index Using Bayesian Statistics
Most measures of political decentralization seem to capture only specific facets of the concept. In particular, the excessive dependence on fiscal indicators has often been criticized since they seem unable to assess the degree of autonomy exerted by subnational governments. On the other hand, efforts directed at developing more encompassing indexes have had to rely on the aggregation of items developed by experts, a process that is prone to idiosyncratic errors. In this paper I propose the development of a measurement framework using a Bayesian factor analysis model for mixed ordinal and continuous outcomes. This model can efficiently combine multiple measures of decentralization regardless of their level of measurement, and in this way make use of both the rigour of fiscal indicators and the wider coverage of qualitative indi-cators. Applying this model to a set of 14 indicators I elaborate a more encompassing index of decen-tralization for 33 OECD countries. In order to illustrate the importance of using non-partial measures of decentralization, I use this index to replicate parts of De Mello and Barenstein (2001) exploratory analysis regarding the relationship between decentralization and corruption, showing that such relationship is practically non-existent.
Pina-Sánchez, J. (2014). Decentralization as a multifaceted concept: A more encompassing index using Bayesian statistics. Revista Española de Ciencia Política, 34:9–34
Implications of Retrospective Measurement Error in Event History Analysis
It is commonly accepted that the use of retrospective questions in surveys makes interviewees face harder cognitive challenges and therefore leads to less precise measures than questions asking about current states. In this paper we evaluate the effect of using data derived from retrospective questions as the response variable in different event history analysis models: an accelerated life Weibull, an accelerated life exponential, a proportional hazards Cox, and a proportional odds logit. The impact of measurement error is assessed by a comparison of the estimates obtained when the models are specified using durations of unemployment derived from a retrospective question against those obtained using validation data derived from a register of unemployment. Results show large attenuation effects in all the regression coefficients. Furthermore, these effects are relatively similar across models
Pina-Sánchez, J., Koskinen, J., and Plewis, I. (2013). Implications of retrospective measurement error in event history analysis. Metodología de Encuestas, 15:5–25