Read on Twitter
Following up on a conversation with @cblatts and @namalhotra on the literature claiming non-response rates don't correlate with non-response bias.
I've now skimmed a bit of the literature. (I'm out of my depth, so read on at your own risk.)
TL;DR; I'm not super convinced.
1/
I've now skimmed a bit of the literature. (I'm out of my depth, so read on at your own risk.)
TL;DR; I'm not super convinced.
1/
The most relevant study seems to be Groves (2006) which is a meta-analysis of 30 studies attempting to assess non-response bias in household surveys.
2/
2/
Groves notes that there are many ways to try to measure non-response bias.
Ethan notes that none of them are great.
They include things like:
3/
Ethan notes that none of them are great.
They include things like:
3/
1. Comparing response rates across subgroups
2. Assessing covariate imbalance
3. Comparing estimates across different studies
4. Follow up w/ non-respondents and then compare
5. Comparing estimate stability under different covariate adjustments/weighting schemes
4/
2. Assessing covariate imbalance
3. Comparing estimates across different studies
4. Follow up w/ non-respondents and then compare
5. Comparing estimate stability under different covariate adjustments/weighting schemes
4/
You'll notice, while each of these is a thing that seems worth doing, none is a particularly rock-solid measure of bias. And some are way less convincing than others.
5/
5/
The meta-analysis appears to pool all these different approaches to assessing bias (w/o, in my quick skim, saying anything about how many of each approach there are or how to compare them).
It is also focused exclusively on household surveys, not political opinion polls.
6/
It is also focused exclusively on household surveys, not political opinion polls.
6/
It the asks whether there is a correlation between the assessed non-response bias (on any of these indirect measures of bias) and the non-response rate of the study the assessment came from.
7/
7/
There are 30 studies and 235 bias estimates. My quick skim did not pick up a discussion of the implications of multiple measures per study for standard errors (it could be there though).
Average non-response rate is 30. Support of the data is ~15-70.
8/
Average non-response rate is 30. Support of the data is ~15-70.
8/
Despite all this, the correlation between non-response rate and "absolute relative bias" (a very hard to interpret variable, given all the many types of tests that are being pooled) is in fact positive (.33).
9/
9/
Here's the key graph.
Groves says the correlation is "modest" and that most of the variation is within study not across.
It is unclear to me how to think about magnitudes, given the nature of the dependent variable.
10/
Groves says the correlation is "modest" and that most of the variation is within study not across.
It is unclear to me how to think about magnitudes, given the nature of the dependent variable.
10/
Overall, to me this just isn't very convincing evidence on which to hang our hats that we shouldn't be worrying about non-response bias from very low response rates to political polls.
11/
11/
1. It isn't about political polls.
2. The measures of bias are weak.
3. The correlation is nonetheless positive.
4. The support of the data don't include contemporary non-response rates.
So I think I'm going to keep worrying about non-response rates and model error.
12/12
2. The measures of bias are weak.
3. The correlation is nonetheless positive.
4. The support of the data don't include contemporary non-response rates.
So I think I'm going to keep worrying about non-response rates and model error.
12/12
