Experiments: Incentive experiment

Five experimental conditions were tested, comprising four versions of an 'early bird' incentive and one universal £10 incentive. 

The findings indicated that all five conditions improved response rates and reduced carbon footprint. However, only one version of the early bird incentive appeared likely to reduce overall survey costs by offsetting increased incentive expenditure with savings in printing and postage costs.

Due to the relatively small sample sizes used in the initial experiment, it was not possible to robustly assess the impact on respondent profiles or key survey estimates. 

A larger follow-up experiment was therefore conducted to validate the response rate findings and to assess the implications for survey estimates and demographic profile, and to reassess likely impacts on cost and carbon footprint.

The experiment was conducted across Waves 119 and 120 of the survey (starting in September and October 2025 respectively). 

The experiment was run on a separate boost sample, with the main sample serving as the control group. This design eliminated the risk of the experiment contaminating the results of the main survey.

Respondents in the experimental group received a modified Mailing 1 letter stating that they would receive a £10 incentive if they completed the survey before a specified cut-off date. 

The letter also explained that the incentive would reduce to £5 for responses received after the cut-off date. The cut-off date was selected to fall immediately before Mailing 2 was sent to print.

Respondents in the control group received the standard survey materials and were informed that they would receive a £5 incentive. Apart from the wording and incentive offer contained in Mailing 1, all subsequent mailings were identical between the two groups.

The experiment used a total sample size of 6,250 issued addresses across the two waves. This sample size provided 80% statistical power to detect a difference of three percentage points in the headline Active Lives estimate.

The experimental group achieved a higher response rate than the control group, replicating the findings from the 2024 experiment. 

The household response rate increased from 18.2% in the main sample to 19.9% in the experimental group. After weighting, the number of returns per 1,000 issued addresses increased from 235 to 281.

The impact on key survey measures was generally small. The headline activity estimate showed a slight increase in the proportion classified as active (63.4% to 64.8%) and a corresponding reduction in the proportion classified as fairly active (11.0% to 9.5%), although confidence intervals indicated that these differences were not statistically significant. 

Similar mixed patterns were observed across other key measures, with some activities appearing slightly more prevalent and others slightly less prevalent within the experimental group.

The experiment continued to demonstrate a positive effect on the survey’s carbon footprint. Modelling suggested that the early bird incentive would reduce carbon emissions by approximately 17%, primarily because higher response rates reduce the number of reminders and issued cases required to achieve survey targets.

However, unlike the findings from the 2024 pilot, the incentive model was projected to increase overall survey costs by around 5%. 

This was largely driven by a higher proportion of respondents qualifying for the £10 incentive than in the previous experiment (44.2% compared with 34.1%), reducing the extent to which increased incentive costs could be offset through savings in printing and postage.

The impact on respondent profile was generally small. Differences were observed across a number of demographic characteristics, but the largest effects were seen for ethnicity and deprivation, where the experimental group moved slightly further away from the population profile than the control sample. 

Analysis of respondents completing before and after the cut-off date suggested that the higher incentive encouraged participation among some harder-to-reach groups. 

However, this effect appeared to be offset by lower participation rates among those groups once the incentive reverted to £5, resulting in little overall improvement in representativeness.

Overall, the experiment confirmed that the early bird incentive increases response rates and reduces carbon footprint. 

However, it also indicated that the approach is currently unlikely to achieve cost neutrality and may introduce small changes to respondent profile and survey estimates. 

Consequently, the evidence did not support implementation of the early bird incentive within the main survey design at this time.