Food and You 2: Technical Report: Questionnaire development and cognitive testing

Questionnaire design 

Food and You 2 uses a sequential mixed-mode approach involving an initial online stage, with non-respondents then followed up using a postal questionnaire. Therefore, the questionnaire is designed in such a way that it can be presented online and on paper. Like many other push-to-web surveys, the online version of the questionnaire is too long and complex to translate into an equivalent self-completion questionnaire suitable for postal administration. This means there are some differences between the online and postal questionnaires. To help address this limitation, two versions of the postal questionnaire are developed, thereby enabling more questions to be asked across the sample as a whole. However, even with two versions of the postal questionnaire, there is insufficient space to include some of the online questions. 

Given the wide range of topic areas that the FSA and external stakeholders are interested in investigating, the issue of questionnaire length is considered throughout the questionnaire development period. Ipsos recommends that, in the interest of reducing drop-out rates, the online questionnaire should not take longer than 30 minutes for the average participant to complete and the postal questionnaires should not be more than 20 pages in length. This time limit for the online survey and page limit for the postal survey were recommended to minimise the risk of participants not completing the survey, and to minimise the risk of straight-lining (i.e. selecting the same answer consistently) when going through the survey.

A modular approach is required for Food and You 2 to keep the length of the survey to a maximum of 30 minutes, and to minimise the likelihood of participants starting but not completing the survey. It also maximises coverage of topics and allows for new modules or questions to be added on emerging topic areas. When developing the Food and You 2 Wave 1 questionnaire, the topic areas the FSA were interested in were grouped into broad ‘modules’ (such as food shopping, food concerns or food we can trust). These modules were then assessed for frequency of fielding (6 months, 12 months or 24 months). For instance, attitudinal questions that are used to measure the FSA’s performance (e.g. trust in the FSA) or where fluctuations over time are more likely (e.g. concerns with food) were considered to be ‘core’ and therefore collected every 6 months. Whereas behavioural questions (e.g. on food practices in the home) that were relatively stable over time in previous studies were deemed to be appropriate for fielding less frequently.

Questionnaire development draws upon the work done for previous waves. The development for Wave 1 involved questionnaire development workshops, cognitive testing, usability testing and a pilot (covered in more detail in the Wave 1 Technical Report). The questionnaire development for Wave 2 onwards was shorter as core questions and materials had been developed in Wave 1. When newly developed questions are added to the survey, a phase of cognitive testing is held to test consumer understanding. For waves with few new questions, cognitive testing is not conducted.

Design of questions

The content and nature of the questions is informed by previous research conducted by the FSA, the FSA and stakeholders’ research priorities, and by Ipsos’ prior experience in survey research.

In Wave 1 a prolonged period of questionnaire development took place which involved an extensive review of questions from previous FSA surveys (Food and You and Public Attitudes Tracker). After all relevant questions were compiled a workshop with the Food and You 2 advisory group was held to discuss key priorities for the questionnaire. This was followed by a second workshop with key internal stakeholders to discuss their priorities for the questionnaire and provide Ipsos with direction regarding questionnaire content.

Following this, draft questionnaire modules were compiled based on questions from previous FSA surveys. Numerous alterations to the wording, ordering, format and content of the questions were made in the process based on survey design best practice, with additional questions designed based on stakeholder needs.

To determine content for the questionnaire for each wave, meetings were held between Ipsos, the FSA and key stakeholders to discuss research priorities and to decide which questions from the online questionnaire should be included in the postal questionnaires.

To enable comparability of the data between waves, questions carried over from earlier waves are largely kept consistent in wording and format. For the exceptions, see the ‘Differences between Waves’ sheets in the Tables User Guide for each wave. 

Cognitive testing

In social and market research, cognitive testing refers to a form of qualitative data collection in which participants are asked by an interviewer to examine a set of materials and explain their understanding of them. In questionnaire development, cognitive testing interviews are used to evaluate how participants approach a questionnaire so that any issues regarding participant comprehension may be highlighted. 

Following the completion of the first questionnaire draft, a series of cognitive testing interviews are arranged to test a sub-set of questions from the questionnaire, specifically those new or modified for that wave. Around 20 interviews are conducted with members of the public, including some conducted in the Welsh language. During recruitment participants are screened on age, gender, ethnicity, geographical region, employment status, income and, depending on the focus of the topics, whether or not they have a food allergy, intolerance or Coeliac disease. This ensures people with relevant food behaviours and habits are spoken to, which is important for assessing the questions.

Key aims of the cognitive testing are: 

• to gauge the simplicity of questions and participant comprehension of key terms;
• to note any ambiguity in the interpretation of the questions; and
• to identify any questions that may not produce meaningful data.

The Welsh language interviews also help to evaluate the accuracy and clarity of the translations. 

Each cognitive interview is undertaken with a single participant, lasts approximately one hour, and is conducted by a moderator using online video conferencing software. During each interview, the moderator records the participant’s answers and notes further observations regarding how the participant interprets the questionnaire, with attention paid to any problems encountered. The English language interviews are conducted by moderators from Ipsos, while the Welsh language interviews are conducted by a trusted external qualitative researcher. Some of the interviews are conducted in the (virtual) presence of an observer from the FSA.

Following completion of the interviews, Ipsos submits a written report to the FSA detailing the findings with recommendations. An extended meeting is subsequently held to discuss the findings and agree on further edits to the questionnaire.

Postal questionnaire design and modular approach 

The postal questionnaires consist of a selection of questions from the online survey. The full questionnaire is not included in the postal versions due to concerns regarding questionnaire length. 

Questions are selected for inclusion in the postal questionnaires based on a number of factors. For instance, questions that are a key strategic measure for the FSA (e.g. trust in the FSA) are included to provide the FSA with robust data. Questions are also included to maximise the base sizes for specific groups of interest (e.g. participants with food allergies). Finally, questions where the mode of delivery and sample profile may impact the data collected, for example questions on food security. It is important to include the majority of the demographic questions in the postal survey to enable subgroup analysis.

As with the online questionnaire, there are differences between England, Wales and Northern Ireland in the wording of a small number of questions. Participants in Wales are sent copies of the questionnaires in English and in Welsh. 
As noted, the survey is conducted using a modular approach. Certain ‘core’ modules are included in each biannual survey wave, while others are rotated every 12 or 24 months. The content of the survey for each wave is detailed in the section below. 

Overview of survey content

Introductory questions (core module)

In the online survey, this module begins with a question asking for confirmation of age (as those under 16 years are not eligible to participate). This is followed by a small number of questions asking participants for some basic information about themselves and their household, such as their gender identity, and the number and age of any other household members. The module also asks participants whether they have a food allergy, food intolerance or Coeliac disease so that the questionnaire can be tailored to individuals. This module was included in Wave 1 and is kept unchanged between waves to enable comparability of subgroup trend data. 

Food Concerns (Core module)

In this core module, participants are asked whether they have any concerns about the food they eat, followed by a spontaneous question asking them to give details on these. This is followed by questions which list specific food concerns, prompting participants on the food concerns they may have.

Food You Can Trust (Core module)

This core module gauges participant confidence in the food supply chain (including in farmers, food manufacturers, and shops) and asked participants questions relating to the FSA, and trust in its ability to fulfil its key responsibilities. 

Household Food Security (Core module)

This module incorporates the USDA 10-item US Adult Food Security module, a standardised measure that uses indicator questions to assess different levels of food security experienced by participants and their households. It asks a series of questions regarding participants’ ability to afford food over the previous 12 months. It also asks about changes participants have made to their eating habits in the last 12 months, and the reason for these changes (e.g. financial reasons, health reasons). The USDA has published the most up to date guidance, including how to calculate food security scores. For more detailed information please visit the guidebook. Questions on food bank use, use of food clubs and social supermarkets are asked within the household food security module. 

Due to the sensitive nature of the topic area, all questions in this section are optional and include a ‘Prefer not to say’ option, in addition to ‘Don’t know’ or ‘Not stated’ options. Any questions that have any of these three responses, or that were left blank, are treated as ‘missing’, with no data imputed. 

About You and Your Household (Core module) 

This module asks participants various questions about their personal circumstances and those of their household, including age, marital status and working status. The inclusion of these questions is primarily intended to enable demographic subgroup analysis of the data.

Food Shopping (Wave 1 and repeated annually)

In this module, participants are asked about their food shopping activities, including: where and how often they shopped for food, whether they check food labels when shopping, and whether they check for food and allergy alerts considering dietary requirements when shopping and confidence in allergy labelling. 

Online Food Platforms (Wave 3 and repeated annually)

This module includes questions about how often participants order food online, what type of products they order and the platforms they use to do this. It then asks whether participants have encountered any problems when using such platforms.

Eating at Home (core questions and full module, repeated from Wave 4)

The core module includes a sub-set of questions asked in the ‘full’ Eating at Home module. It is intended to gauge participant knowledge of and adherence to the key FSA food safety and hygiene guidelines. Participants are asked about the ways in which they store, prepare, and consume food in the home. 

Eating out and the Food Hygiene Rating Scheme (Wave 2 and repeated annually)

In this module, participants are asked how often they eat out or buy food to take away and the factors they consider when choosing where to eat. Participants are also asked about their awareness and use of the Food Hygiene Rating Scheme (FHRS).

Defra Questions (Waves 1 and repeated annually) 

This module includes questions requested by the Department for Environment, Food and Rural Affairs (Defra) (who make a small financial contribution to the study). 

This module asks questions about participants’ food-buying activities. This included questions relating to environmental concerns, provenance and what influences purchasing choices. 

Emerging issues (Wave 4 and repeated every 2 years)

A new module was introduced in Wave 4 focusing on emerging issues of interest to the FSA and its stakeholders. More specifically, the questions focus on sustainable shopping and dietary behaviours, meat alternatives and awareness of Genetically Modified (GM) and Genetically Edited (GE) food.