National statistics are like public toilets: you may choose to look closely, but you might not like what you see.
Statisticians go to immense lengths to make sure the data they produce offers, in aggregate, broadly accurate information about countries, economies and population. Modern standards for statistical reproduction are high: processes at bodies like the UK’s Office for National Statistics or the EU’s Eurostat undergo constant, diligent review, both internally and externally.
Sometimes, the best way to do this is by fudging: data has to be smoothed and gaps must be filled. But the smaller your data set, the harder this process is — a pinprick does more damage to a butterfly than to a blue whale.
And data sets are getting smaller. A couple of recent notes have looked at the effect this is having on US labour market surveys. A JPMorgan Asset Management note from March highlighted declining rates of response to the Bureau of Labor Statistics’ current employment survey, and Job Openings and Labor Turnover Survey (JOLTS). Here’s how response rates have shifted for each:
Researcher Stephanie Aliaga wrote (our emphasis):
Along with concerns about whether seasonal adjustments are keeping up with evolving seasonal patterns, the decline in the survey’s response rate raises further questions about the reliability of employment data. Indeed, the Bureau of Labor Statistics (BLS) has undergone increased challenges in conducting its surveys with response rates declining broadly over the last decade, now at just 45% and 31% for the current employment survey (CES) and JOLTS, respectively…
Another issue is the widening difference in responses between the preliminary and final releases, which can lead to large backward revisions. In the establishment survey, response rates for the preliminary release have been declining, but by the time the BLS releases its final estimates, the rate among the active sample improves to an annual average of 94%1. The JOLTS survey is more worrisome, as the response rate for the active sample only improves to 64% from 58% between releases, and both rates have trended downwards significantly. This lower rate combined with a much smaller sample size (21,000 vs. 651,000 establishments in the CES) increases the risk of bias in the JOLTS survey.
Goldman Sachs took its own swing at the topic this week, noting the JOLTS response rate has “plummeted” by nearly a third since the pandemic started, “raising questions about the survey’s reliability at a time when the focus on its monthly prints has never been greater”.
Ronnie Walker et al. write (our emphasis):
— We see two main channels through which a lower response rate can impact the JOLTS job openings statistics. First, a lower response rate reduces the sample size and the accuracy of the resulting statistics. Indeed, the Bureau of Labor Statistics (BLS) reports that the median monthly standard error—a measure of the expected discrepancy between a sample estimate and the true value of a population—had already begun increasing before the pandemic. We estimate that the monthly standard error is now at least 90% higher than in 2002-2013. This implies a 90% confidence interval of roughly 700k for the latest readings.
— More importantly, a declining response rate can systematically bias the job vacancy estimates in one direction or the other if the probability of nonresponse is correlated with changes in job openings. And indeed, the BLS cites the increase in remote work as one of the key drivers of nonresponse.
As response rates both dwindle and bifurcate, it’s companies that have adopted flexible working arrangements — which tend to align with lower job openings — that are dropping out of the response cohort and therefore skewing the figures. In this instance, Goldman thinks this is biasing JOLTS higher (that is to say, indicating more openings than there actually are). They write:
It is difficult to accurately estimate the magnitude of the bias without knowing the difference in response rates between companies that are back in the office and those with flexible working arrangements, but our best guess is that JOLTS would have declined an additional 200-300k over the last year absent the bias.
It’s a transatlantic phenomenon. In the UK, the Labour Force Survey (which involves contacting households instead of businesses) has seen a long-running decline in response rates accelerate since the pandemic. Here’s the latest figures from the Office for National Statistics:
Thee longer trend is remarkable: the vast majority of those contacted by the ONS used to respond, now few do (this chart runs up to 2016, so overlap it with the above one in your head):
These issues got beyond the headlines: for instance, response rates in inner London during the latest wave were just 12.8 per cent, versus 20.2 per cent across England.
The Bank of England’s Swati Dhingra flagged the potential problems at a Resolution Foundation event earlier this year:
There is an element of judgement doing in here, there’s an element also of data fog… of, for example, response rates in the Labour Force Survey have fallen to the levels that they have, how reliable is that and how much we can be certain about what we’re picking up from there is, in fact, really data?
She referred to “years gone by when data was better, it was better quality”.
Eurostat’s figures aren’t as easy to uncover as the ONS’s, and its press officers said they don’t track aggregate non-response data as a time series. But its latest quality report for the EU LFS (covering 2020) at least points to some of the issues that can occur.
For instance, every year a handful of countries don’t get responses to certain questions:
In aggregate, Eurostat identified issues of under-coverage — when parts of the target population surveyors had hoped to reach — in almost half the countries it looked at. Non-respondence (when the would-be respondent doesn’t give an answer to any or all of the question) is an issue, per the report, but for varying reasons:
In most countries, non-response was due to the impossibility to contact the sampling unit or the respondent. By contrast, in Czechia, Estonia, Croatia, the Netherlands, Poland, Slovakia and the North Macedonia, most of the non-respondents refused to participate in the EU-LFS. In Belgium and Austria, non-response was mainly due to other reasons, as lost interviews due to technical issues, expired time for completing an interrupted interview, etc.
Eurostat isn’t even sure how much is being fudged. The report says twenty countries “adopted editing and correction procedures to EU-LFS data” in 2020… but adds “for most countries, the overall editing rate remains unclear”, rending it unable to estimate the degree of overall processing errors.
As ever, the annexes are full of thought-provoking delights. For instance, here’s Ireland’s notes on a compulsory question about how why respondents have a temporary job:
Not stated answers arise from respondents. Non-response is rising fast. Any other reason behind it?
Or Croatia, on income deciles:
People do not want to give an answer on this sensitive issue especially in the case of proxy interview.
But without broad data on respondence overall, it’s hard to know whether Europe’s problems are on the same scale as the US and UK’s. The UN’s International Labour Organization, while designs labour force surveys, told us it doesn’t track national nonresponse levels.
What does it all subtract down to? On the US’s response woes, Aliaga writes:
In general, these concerns remind us that the government’s monthly jobs report should be seen as just one piece of the broader labor market mosaic.
Might a similar downgrade to also-ran status be awaiting some other statistical heavyweights?
Further reading— ONS oh noes— Eurostat: Das Numberwang!
