I received a mailer ad from an insurer that reported “an average of $507 in savings.” That headline grabbed my attention.
However, I read the footnotes, which tells us that the statistic comes from those,
…who reported savings by switching their auto insurance…
Subtly, multiple selection effects are present:
- only those who experienced savings
- only those who reported their savings
Those who did not experience savings may have experienced the opposite, but were excluded from the sample. Of those who reported savings, we can expect that the sample will bias toward those who have experienced the greatest benefit, and considered their savings worthy of reporting.
Thus, it seems that this ad campaign benefits from the statistical illiteracy of its audience to overestimate their expected savings.
The full footnote says,
Average annual per household savings based on a 2018 national survey by State Farm of new policyholders who reported savings by switching to State Farm.
https://www.facebook.com/statefarm/videos/391907058398668/
https://www.statefarm.com/insurance/multiline
In the below screenshot, see the footnote in a variation of this ad:
This version, which I pulled from their site, adds in a third qualifier to the effect size, “up to” (!) By doing this, they can present an even higher dollar amount – even further from the expected savings the average consumer can experience.
Honestly, I’m not even sure how to interpret this myself: what does “up to an average value of $X” mean in this context?