Evaluation of the Food Standards pilot
Evaluation of Food Standards pilot: Appendices
England and Northern Ireland specific
Appendices for the Evaluation of Food Standards pilot report, this includes a glossary, intervention logic and quantitative evaluation.
Annex 1: Glossary
Food business - As defined by Article 3(2) of Regulation (EC) No 178/2002 it means any undertaking, whether for profit or not and whether public or private, carrying out any of the activities related to any stage of production, processing, and distribution of food.
Food hygiene – The measures and conditions necessary to control hazards and to ensure fitness for human consumption of a foodstuff, considering its intended use as set out in Regulation (EC) No 852/2004.
Inspection - The examination of any aspect of feed, food, animal health and animal welfare to verify that such aspect(s) comply with the legal requirements of feed and food law and animal health and welfare rules.
Intelligence - Information that has been subject to a defined evaluation and risk assessment process to assist with regulatory decision making.
Intelligence led - Intelligence-led working is defined by National Trading Standards (NTS) as ‘a business process for systematically collecting, organising, analysing, and utilising intelligence and information to guide operational and tactical decisions. Intelligence led aids in identifying, assessing, and managing targets, threats and problems at the local, regional and national level.’
Intervention - Regulatory actions taken to affect or interfere with decisions made by individuals, groups, or organizations regarding social and economic matters. Interventions include official food controls and other interventions such as education, advice and coaching, information and intelligence gathering (including sampling where the analysis is not to be carried out by an Official Control Laboratory).
Proactive intervention – A proactive intervention refers to any intervention conducted as part of the LAs planned intervention programme.
Public Analyst – Scientists that ensure the safety and correct description of food by testing for compliance with legislation as specified in Section 27 of the Food Safety Act 1990 and Regulation 4 of The Food Safety (Sampling and Qualifications) (England) Regulations 2013.
Reactive intervention – A reactive intervention refers to any LA intervention completed as a response to a complaint or to any other intelligence received, such as a sampling result or referrals. It is not planned.
Risk - The chance or probability that a person will be harmed or experience an adverse health effect if exposed to a hazard.
Risk analysis - A process consisting of three interconnected components: risk assessment, risk management and risk communication.
Sampling - Taking feed or food or any other substance (including from the environment) relevant to the production, processing and distribution of feed or food or to the health of animals, to verify through analysis or examination compliance with feed or food law or animal health rules.
Standards – Rules or principles defined in food safety law that are used as the basis for judgement against.
The National Food Crime Unit (NFCU) - The NFCU provides a nationwide focus on enforcement against serious fraud and related criminality in food and feed supply chains.
Annex 2: Food Standards Intervention Logic
This Annex introduces the intervention logic for the food standards project and the work completed so far under the pilot. Figure 9 below sets out the intervention logic model as described in the inception report.
Figure 9 Intervention logic model for the Food Standards Project (accessible version)
Objective: to pilot a new operating model for food standards which seeks to establish consistent risk rating approach that is dynamic and integrates intelligence so that LA's can provide a more targeted and efficient response.
Problems
- risk rating approaches not consistent and not fit for purpose (LAs don't have enough resources; and approaches do not reflect all FBO types)
- no consistent use of intelligence and data sharing by LAs.
- sampling use could be more efficient and its use is declining.
A. Inputs (and activities)
- A.1 Prepare LAs to pilot the new model through tailored training and discussions
- A.2 LAs use a uniform risk rating scheme to produce an indicative risk profile
- A.3 Intelligence systematically gathered and shared between FSA's Food Standards Intelligence function and LAs including through national sampling programme
- A.4 FSA and LA resources and skills to implement the pilot
- A.5 Evaluation plans implemented.
B. Outputs
- B.1 Number and types of LA visits/training received
- B.2 Number of intervention type and frequency based on FBOs risk rating
- B.3 Number and quality of intelligence exchanged between FSA and LAs including sampling activity data
- B.4 Evaluation reports presented (FSA and ICF)
C. Short-term outcomes (6 months)
- C.1 LAs understand how to operate under the new approach and engage with the pilot
- C.2 Consistent and proportionate approach to regulation food standards across pilot LAs
- C.3 Increase in the proportion of non-compliances identified and resolved
- C.4 Increase in the proportion of LA resources directed by intelligence rather than being drive by an establishment-based risks assessment scheme.
- C.5 FSA develops knowledge about the new model and adapts / modifies integrating these lessons
D. Medium-term outcomes (1 year)
- D.2 Intelligence driven work nominated within LAs
- D.3 FSA has a better oversight of the national picture of food standards and better able to identify trends and emerging issues relating to food standards compliance
- D.4 Lessons learnt are used to take informed decisions about new operating model for food standards
Assumptions:
- There is buy-in from LAs to engage in this new way of working
- The new approach works across LAs and FBO types with LAs having access to the resources (staff time, infrastructure, knowledge and data) to correctly develop the indicative risk profiles
- FSA and the LAs will have the resources, permissions, infrastructure and skills in place to gather, share, process, and interpret intelligence data and the required communication between each other
- Training and engagement activities are of sufficient quality for LAs to ahve the skills and knowledge to adopt this new way of working
- The new approach is capable of generating accurate scores that leads to proporiationate minimum intervention frequency and type requirements
Figure 9 Intervention logic model for the Food Standards Project
Annex 3: Food Standards Pilot - quantitative evaluation
1. Introduction
FSA’s Analytics Unit designed excel spreadsheets and/or online forms to collect the necessary data from all participating LAs (control and pilot groups) in a standardised format to support the evaluation of the pilot. Data was collected monthly.
Below is a short summary of the forms’ content:
- the risk scheme scores from every inspection.
- a detailed questionnaire for every: reactive (intelligence led) intervention LAs undertook during a given month and proactive (programmed) intervention that LAs undertook where the premises was found to be non-compliant.
- through the questionnaire we collected data on the number of reactive versus proactive interventions, amount and effectiveness of intelligence and risk assessment model, samples outcome, time resources involved, etc.
- for businesses found to be broadly compliant, we asked for the detailed questionnaire to be completed for a small sample of 10 proactive interventions and TRIs.
The quantitative evaluation is based on data received from all participating LAs for the period January 2021 to March 2022. Due to lack of inspections from January 2021 to April 2021, we extended the pilot period to 15 months instead of the initial 12 months.
With the proposed FSDM, we expected to find more non-compliances due to inspection plans being focused on higher risk businesses. We also expected to see more intelligence led interventions because of increased sharing of intelligence reports. In addition, we expected that individual interventions may take longer as they were more likely to find non-compliance and require follow up corrective action.
In this annex we have provide data referred to in the main body of the report plus a second which allows us to provide some sensitivity analysis. This second set has two main differences. Firstly 9% of interventions in the pilot group and 11% of interventions in the control group were remote interventions due to COVID-19, although these were counted within the total interventions carried out during this period. They were not counted within the detailed analysis of outcomes, as these do not generally fall within the criteria of TRIs within the proposed model. These are included in the sensitivity analysis
Secondly, as mentioned in section 1.4.3, results for one of the control LAs skewed the overall analysis results. The LA in question dealt with a higher than usual number of food incidents during the pilot. Multiple samples were taken in relation to the incidents which is normal practice in this scenario. As would be expected, the analysis for many of these samples were unsatisfactory due to the targeted nature of this work and the time taken on interventions due to these incidents was significant.
2. Performance quantitative measures
We measured the following outcomes to analyse the effectiveness of the pilot. To do so, we compared the results from the existing FLCoP model (control group) with the results from the proposed operating model (pilot group).
2.1 The effectiveness of intelligence
By embedding intelligence into the delivery of food standards official controls we will have a regulatory model that is more dynamic by nature, enabling us to identify non-compliances more effectively and earlier, and to disseminate intelligence across stakeholders more often.
The proportion of reactive (intelligence led) interventions
On the figure below, we measured the proportion of interventions which were led by any form of intelligence (reactive interventions) in comparison with the proportion of interventions that were scheduled according to the outputs from the risk schemes (proactive interventions) within pilot vs control.
As can be observed from Figure 10, the number of reactive interventions is higher in the pilot group, accounting for 20% of the total number of interventions, compared to 15% in the control group (1% if we exclude the anomalous LA). This means that the proposed FSDM is more dynamic, allowing LAs to use intelligence more frequently and to carry out specific directed actions to prevent or address food standards non-compliances in a more timely and more targeted manner.
Figure 10 Proportion of proactive (scheduled) vs reactive (intelligence led) interventions in pilot LAs vs control LAs
Figure 11 shows the proportion of intelligence-led inspections identifying non-compliance is higher in the control group at 22% compared to 15% in the pilot. However, it should be noted that it is likely this figure may not be typical as it includes the control LA that dealt with a high number of food incidents that skewed the overall figures.
Figure 11 Proportion of reactive (intelligence led) interventions finding non-compliance
Figure 12 shows the dissemination of intelligence reports. All but one of the reports in the control group came from the anomalous LA.
Figure 12 Intelligence reports LAs disseminated to other stakeholders, pilot vs control
| Group | Other LA | Environmental Health | FSA | PA | Industry | NFCU | Total |
|---|---|---|---|---|---|---|---|
| Control (as in main report) | 0 | 0 | 1 | 0 | 0 | 0 | 1 |
| Control (including anomalous LA) | 12 | 10 | 14 | 9 | 0 | 0 | 45 |
| Pilot | 44 | 20 | 16 | 12 | 5 | 1 | 98 |
Figure 13 shows that 86% of the total number of intelligence reports within the pilot group led to follow-up corrective action, compared to 82% on the control group (although all of these actions were within the single LA discussed above that dealt with a high number of incidents during the pilot).
Figure 13 Proportion of reactive (intelligence led) interventions leading to a follow-up corrective action by LA
Overall, these results demonstrate that the process of intelligence sharing has increased the level of understanding of potential risks, driving up the quality of intelligence reports shared across all stakeholders. This also suggests that the process of communication has improved within the pilot, with LAs being able to co-ordinate and collaborate better with their neighbouring LAs, FSA, Primary Authorities, etc. This statement is also verified and supported by qualitative assessment.
2.2 The effectiveness of the proposed risk scheme
We expect that the proposed risk scheme and decision matrix will enable LAs to target their resources on those businesses deemed to be highest risk e.g. high levels of non-compliance with food standards law.
As can be observed from Figure 14, the proactive interventions led to the identification of a higher number of non-compliant food businesses within the pilot group at 16%, compared to 4% within the control group. This demonstrates that the new risk scheme is more effective at identifying non-compliances and directing LA resources to the higher-risk establishments to resolve issues.
Figure 14 Proportion of proactive (programmed) interventions finding not broadly compliant businesses.
Similarly, the proportion of proactive (programmed) interventions that led to a follow up corrective action by LAs to address issues of non-compliance was higher for the pilot group than the control group, at 57% and 45% respectively (37% excluding the anomalous LA) (see Figure 15).
Figure 15 Proportion of proactive (programmed) interventions resulting in follow up corrective actions
The table below (Table 4.1), shows the breakdown of intervention types, pilot vs control.
Table 4.1 Interventions by type. Base size of 576 (469) for control, and 1,258 for the pilot
| Group | On-site intervention | TRI | Business Registration | Remote intervention due to COVID-19 |
|---|---|---|---|---|
| Control (as in main report) | 90% | - | 10% | - |
| Control (including anomalous LA) | 81% | - | 8% | 11% |
| Pilot | 72% | 14% | 4% | 9% |
2.3 The effectiveness of sampling
We can observe from Figure 16 that the pilot group took significantly more proactive samples (as a proportion of total sampling) than the control group because of focusing more on the assessment of intelligence. It can be seen in Figure 17 that the proportion of reactive samples that are unsatisfactory within the pilot group is larger than the proportion of unsatisfactory proactive samples (when the anomalous control LA is removed from the analysis), which demonstrates a more effective use of intelligence in targeting sampling activity to areas where there are likely to be issues.
Figure 16 Proportion of proactive vs reactive samples, pilot vs control (excluding FSA directed sampling)
Figure 17 Proportion of proactive and reactive sample outcomes, pilot vs control (excluding FSA directed sampling)
In addition to regular locally coordinated sampling work, FSA directed sampling accounts for 22% of the total number of samples taken within the pilot group and returned 50% unsatisfactory results. It is expected that the new FSA intelligence function will contribute to the identification of more unknown or emerging risks, increasing the likelihood of unsatisfactory sampling results as we become better at targeting sampling activity.