Evaluation of the Food Standards pilot

This report presents the findings from the evaluation of the pilot programme that the Food Standards Agency (FSA) has completed to test a proposed Food Standards Delivery Model (FSDM) as part of the Achieving Business Compliance (ABC) programme. The proposed model aims to support local authorities (LAs) in England, Northern Ireland, and Wales in targeting resources more effectively, and provide better assurance and more flexibility to LAs. The pilot programme was carried out with seven LAs in England and Northern Ireland ^(footnote). The pilot started in January 2021 and finished in March 2022.  

LAs in Wales were unable to pilot the proposed model at the same time, as the Food Law Code of Practice (Wales) (the Code) does not contain the same provision to allow LAs to deviate from the Code for pathfinders and pilots. There were also concerns about the proposal on how very low risk businesses would be regulated. These concerns have been addressed as part of the post-pilot model review and a pilot of the model is expected to be carried out in Wales in 2023, subject to Ministerial agreement. 

The evaluation methodology focused on capturing LAs’ experiences working to the new model through the entire pilot programme, to adequately capture nuances in its use and factors impacting its performance. It comprised of a scoping phase; data collection, including three rounds of interviews with LAs, interviews with FSA staff, quantitative data collected by FSA’s analytical unit; and an analysis and triangulation phase, integrating the qualitative and quantitative findings. 

The findings show that the three components introduced by the proposed FSDM (a risk scheme, a decision matrix and the use of intelligence) worked well. The new risk scheme was seen as more effective in identifying food businesses that present the greatest risk. Key benefits included greater flexibility in the model, the ability to re-score premises and the ability to use intelligence to select the most appropriate intervention. LAs highlighted the new risk scheme was straightforward to use and, by the end of the pilot, they had already embedded the new way of working into their work practices. 

The process of implementing the pilot was successful overall, with LAs identifying the support received from FSA as a key enabling factor. The support received helped LAs to change working practices to facilitate the proposed model and to strengthen communication channels, which promoted greater intelligence sharing. 

LAs identified challenges with the model. Some of these were resolved within the pilot, while others will require further work between both LAs and FSA. One challenge which was mitigated during the pilot centred on ensuring consistency of officer interpretation. For example, it was identified that ‘single local supplier’ was interpreted differently by LA officers. During the pilot, FSA and LAs collaborated to identify terms which were being interpreted inconsistently and provide clear definitions. The second challenge was related to the perceived effectiveness of Targeted Remote Interventions (TRIs), as some LAs perceived TRIs as being only useful for specific types of business. This led to LAs using TRIs in the context/s they deemed most appropriate and choosing to do a physical inspection in others. 

One outstanding challenge LAs identified during the pilot process relates to data inconsistencies about the registered Food Business Operators (FBOs) in the LAs’ respective Management Information Systems (MIS). The LAs were not always able to integrate the proposed model into their MIS software for the purposes of the pilot. While LAs were aware of this in relation to the pilot, it still led to some initial difficulties, such as duplicating the recording of work for some officers. Looking forward, FSA will recommend that LAs databases are up to date before starting to work with the proposed model. The use of intelligence (gathering and sharing it) was also identified as a challenge for some LAs, and this is an area which FSA and LAs continue to work on. 

Overall, the communication and ongoing support between FSA and LAs has been a success of the programme. The pilot model has been shown to work well and deliver the intended short-term outcomes. Most of the initial challenges identified which were not related to the practicalities of the pilot were addressed and the pilot has been a vital stage for FSA to inform the launch of the proposed model at a national scale. 

Introduction

The Food Standards Agency (FSA) commissioned ICF to carry out an evaluation of phase 1 of the ongoing Achieving Business Compliance (ABC) programme. The ABC programme aims to modernise the way that food businesses are regulated in England, Northern Ireland, and Wales by FSA and LAs.

The ABC programme aims to:

• make it easier for businesses to provide safe and trusted food for consumers
• target regulatory resources at the areas which pose the greatest risk
• improve compliance across the system by working with and through others, including regulatory  partners and influential businesses.

For more information on the ABC programme please follow this link: Achieving Business Compliance (ABC) programme | Food Standards Agency.

This report presents the findings from the evaluation of one of the projects of this programme: the pilot of a proposed FSDM in England and Northern Ireland. The proposed model introduces a modernised risk assessment approach, including a new risk scheme, a decision matrix and the development of an intelligence-led approach to regulatory activity. 

This study aimed to answer the following questions: 

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What has been the effect on resources for each of the stakeholders because of the proposed model?
4. What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences? 
5. What lessons were learnt from the proposed model?

Methodology

The evaluation started with a scoping phase, which included a review of existing ABC documentation, interviews with FSA food standards team, a virtual workshop to develop an initial theory of change for the ABC programme and the intervention logic specific to this proposed model, and the definition of the evaluation framework. 

The next phase was the data collection phase. It included three rounds of interviews with all the LAs using the proposed model during the pilot, plus two LAs from the control group for the pilot. The study also included two interviews with FSA staff, and a series of meetings with the FSA throughout the project. In addition, FSA’s Analytics Unit gathered quantitative data throughout the project that has been integrated into the findings of this study. 

Summary of findings

Findings have been structured around the research questions listed above.

How did the proposed model perform compared to the current framework? What worked well and less well? 

The LAs using the proposed model during the pilot consistently identified the following aspects of the proposed model that worked well: 

• risk scheme and rating:  The proposed risk scheme and model introduced a single, uniform risk rating approach to assess the risk profile of FBOs more accurately. It was dynamic and integrated intelligence for a more targeted and efficient response, enabling LAs to target their resources to food businesses presenting the greatest risk. They were found to be suitable for identifying risk and prioritising resources within LAs. 
• integration/‘fit’ with existing working practices: The proposed model was easy to use, which led to the successful integration of the model into LAs’ existing working practices.
• sampling approach: LAs viewed the directed sampling approach as being effective at identifying non-compliance and gathering intelligence. 

On the other hand, LAs using the proposed model perceived the intelligence element was still a work in progress:

• the intelligence element of the model was not fully developed by the end of the pilot, with participating LAs at different stages of maturity in their use of intelligence as a driver of regulatory activity. 

At the end of the pilot, there was no consistent approach to how FSA and LAs understand what information is considered intelligence; the type of information which needs to be shared between LAs and FSA; the frequency of sharing information between the LAs and FSA; and the most suitable and accessible mechanism to share intelligence. 

LAs found it challenging to carry out all the directed sampling activities as piloted under the proposed model, as it was not always possible to provide sufficient notice of proposed sampling activity given that priorities were identified through analysis of current intelligence. 

In terms of the implementation of the pilot: 

• the support provided by FSA throughout the pilot was well-received and perceived as crucial to the success of the pilot. This level of support was specific to the pilot project.  
• the main barriers to the implementation of the pilot were the lack of clarity with some of the risk scheme terminology (resolved during the pilot), lack of consistency of food business data held by LAs (addressed during the pilot), and a lack of compatibility between LAs MIS and the proposed delivery model during the pilot period. 

What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?

The experience of each stakeholder (LAs and FSA) was overall very positive. 

• LAs decided to join the pilot because they wanted to co-create the proposed model with FSA, influence the change, and to be able to adapt early to it. Their experience during the pilot met these expectations. 
• LAs identified that a main challenge during the pilot was related to staff capacity to be able to test the proposed model, particularly at the beginning (as the pilot start date coincided with the introduction of national COVID-19 lockdown measures) and initial challenges linked to LA MIS. LAs mentioned there was a learning curve when using the proposed model, but that once their staff were aware, the resources were more effectively allocated (as discussed below)

FSA mentioned that regular communication with LAs had been the key to the success of the project. They perceived LAs as being very open and honest, which allowed FSA to adapt and develop the tools and guidance in a way that could best serve LAs and the objectives of the proposed model. As part of the proposed model, the main challenge raised was linked to the intelligence function (as discussed above).

What has been the effect on resources for each of the stakeholders because of the proposed model?

In terms of changes in resources, LAs mentioned they required time to adapt to the proposed model. However, once the LA staff adapted to the implementation of the proposed model LAs perceived that the time required was the same compared to the current framework. 

In terms of effectiveness and efficiency of LA resources, the participating LAs identified that the proposed model allowed them to target resources more effectively. 

What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences? 

The pilot achieved its objectives and had a positive effect on: 

• identification of FBOs that present the greatest risk
• risk-based targeting of interventions 
• proportion of LA resources driven by intelligence

Evaluation of the proposed model did not identify any unintended consequences because of the proposed model.

What lessons were learnt from the proposed model?

The lessons learnt were structured in three areas, as shown in Table 1.1 below: 

Table 1.1 Lesson learned from the pilot

Area	Lessons learnt
Implementation of the pilot (section 2.3)	The support provided by FSA throughout the pilot was well received and crucial to the success of the pilot.  LAs created a peer support network that was very useful, where they shared lessons learned and challenges to support each other.  Not all LAs started from the same baseline, particularly in terms of experience of using intelligence. This meant the level of support required varied across the pilot group.
The proposed new model (sections 2.3, 2.5, and 2.6)	Need to communicate proposed directed sampling activities early, if possible, to LAs and Public Analyst (PA) laboratories. If this is not possible, engage with PAs to harmonise work schedules. Communicate the new intelligence function across other agencies or organisations, to promote collaboration and cross-sector learnings.
National roll-out (section 3)	Need to ensure that there is consistency in the way LAs implement the proposed model. When disseminating the tools to use the proposed model (e.g. proposed risk scheme and decision matrix), ensure that consistent and clear terminology and definitions are used. If possible, facilitate consistency exercises between LAs to create a shared understanding. If not, develop guidelines for LAs, including a summary of relevant frequently asked questions (FAQ) LAs had during the pilot.  Develop case studies that can be disseminated to LAs and wider organisations to showcase the benefits of the proposed model and best practice ways of working, for example, conducting internal consistency exercises.   Some LAs completed internal authority consistency exercises (additional to FSA ones), as they found them very useful to ensure all of their officers applied the model in the same manner Ask pilot LAs to share their experience and examples of best practice with others. Allow sufficient time for LAs to adjust to working with the proposed new model.  Define mechanisms to share intelligence between LAs and with FSA or other agencies, as this was identified as a challenge. Before fully rolling out the intelligence-led approach at national level, it would be beneficial to identify better ways of sharing data across organisations.

The Food Standard Agency (FSA) commissioned ICF to complete an evaluation of phase 1 of the ongoing Achieving Business Compliance (ABC) programme, which aims to modernise the way that food businesses are regulated in England, Wales and Northern Ireland by the FSA and local authorities (LAs). 

This is the final report of the evaluation of the food standards pilot project under the ABC programme. The pilot tested a proposed FSDM that aims to support LAs to target resources more effectively; provide better assurance and more flexibility to LAs; and to help LAs meet their statutory obligations. The proposed model introduced a modernised risk assessment approach, including a new risk assessment scheme, a decision matrix and the development of an intelligence-led approach to LA regulatory activity. The pilot programme was carried out with a number of LAs in England and Northern Ireland , starting in January 2021 and finishing in March 2022.  

The report is organised as follows: 

• the rest of this section introduces the background context for this pilot project. It discusses the ABC programme, the food standards project and summarises the methodology followed by the evaluation study.
• Section 2 summarises the findings of the evaluation, organised by research question.
• Section 3 includes a series of considerations and lessons learned.
• Section 4 closes the report with the conclusions. 
• a Glossary for key terms used in this report is included in Annex 1. 

1.1 The ABC Programme

The ABC programme was initiated in 2020 to build on lessons from the Regulating Our Future (ROF) Programme, which ran from 2016-2019. The programme has undergone a complete restructure establishing a clear vision, objectives and delivery structure for the future, whilst overseeing the delivery of some legacy ROF projects, and beginning the delivery of new projects to support its objectives. 

In a rapidly evolving food sector, there is a need to regulate in a smarter way to make sure that food is safe and is what it says it is. The ABC programme aims to ensure consumers continue to have food they can trust through the development of smarter regulatory approaches which: 

• make it easier for businesses to provide safe and trusted food for consumers
• target regulatory resources at the areas which pose the greatest risk 
• improve compliance across the system by working with and through others, including regulatory partners and influential businesses.

The ABC programme is currently organised into 3 workstreams:

1. Targeted and proportionate regulation - Designing a more targeted and proportionate approach to LA regulation of all food businesses within their remit. Introducing a proposed FSDM, which incorporates a revised risk assessment scheme and more emphasis on intelligence as a driver for LA regulatory activity. Modernising the food hygiene delivery model.
2. Enterprise level approaches - Designing new regulatory models for a set of large businesses that are compliant with regulation and influential in the food chain. Working with LAs, primary authorities and businesses. Developing approaches to assure compliance at a business level rather than an individual premises level. 
3. Assurance of online food sales - Using data and research to build up an evidence picture of the new online food sales landscape. Assessing the potential risks to consumers from buying food online and the regulatory levers available to address these, including working with other regulators, international food safety regulators, government bodies and influential businesses to improve compliance. Identifying and implementing a series of interventions to improve consumer safety when buying food online. 

The intention in all three workstreams is to develop regulatory approaches which are deliverable within existing food law, but the programme will also identify any areas in which legislative change could support the objectives and may make recommendations for change.

For the development of the proposed FSDM, FSA consulted with food businesses and other local and national government agencies. In July 2017, FSA published its plan to reform food regulation, entitled “Why food regulation needs to change and how we are going to do it”. It then launched the reform programme in England, Wales and Northern Ireland. 

1.2 The proposed Food Standards Delivery Model

1.2.1 Background

Food standards regulation aims to ensure food on the UK market meets applicable legal requirements, notably in terms of labelling (including allergens), presentation, composition, permitted/authorised ingredients and levels of additives, contaminants, and residues. 

The Food Law Code of Practice (FLCoP) establishes a framework for the delivery of food standards official controls by LAs. The FLCoP determines the appropriate intervention frequency for food businesses based on the associated risk profiles for different establishments and businesses. LAs must have regard to the FLCoP in the discharge of their duties in relation to food. There are three risk assessment schemes currently in use in relation to food standards, namely the National Trading Standards Risk Assessment Scheme (NTS), the LACORS Trading Standards Risk Assessment Scheme (LACORS) and the Food Standards Intervention Rating Scheme within the FLCoP itself. At the time of writing this report, the most common risk assessment scheme used by LAs is the FLCoP intervention rating scheme.  

The FLCoP intervention rating scheme groups food establishments into Category A (high risk, requires an intervention every 12 months), Category B (medium risk, requires an intervention every 24 months), and category C (low risk, should be subject to an intervention at least once every five years). This allows LAs to prioritise their interventions. Under the current intervention rating scheme, some establishments, due to the nature of their activities, are identified as Category A (high risk) regardless of their level of compliance. Interventions by LAs can take the form of an inspection, partial inspection or audit. 

A series of reports highlighted the need for the current food standards operating model to be reviewed, with FSA Board approving a root and branch review of food standards delivery in December 2018. 

The need for a review of the system was first identified in a report commissioned by FSA that highlighted a series of long standing issues within the food standards intervention scheme (see Table 1.1). These challenges, combined with a general decline in LA resources and a growth in different types of food establishments, led to stakeholders noting that the current FSDM was not fit for purpose. 

Additional reports underpinned the need for changes in the regulatory regime, including a 2018 report by FSA on LA views and experiences of the current framework for regulating food standards, and the 2019 report from the National Audit Office, which examined the effectiveness of the current regulatory arrangements .

The findings from the 2018 report mentioned above were presented to the FSA Board in December 2018, which sanctioned a fundamental review of the current delivery model as it was deemed not fit for purpose and had not kept pace with changes in the way LAs work, such as the use of intelligence. Following these insights, FSA worked in collaboration with LAs and other key stakeholder representatives to develop a new FSDM. 

FSA then started a participatory and interactive process with a number of voluntary LAs to review together the way food standards official controls were delivered by LAs and develop the proposed food standards delivery model described below and tested in the pilot.

1.2.2 The proposed Food Standards Delivery Model

The proposed FSDM incorporates three elements: 

• a single, modernised risk assessment scheme that aims to unify the way that LAs risk assess establishments (a new risk scheme).
• using the risk assessment scheme to identify the appropriate frequency for official control activity based on levels of inherent risk and compliance (decision matrix).
• greater integration of intelligence as a driver of local authority regulatory activity and to inform our national understanding of food standards risk.

The proposed model attempted to address the challenges linked to the existing model as identified in the aforementioned reports. The challenges have been summarised in Table 1.1, together with an indication as to how the proposed model sought to address them. 

Table 1.1 How the proposed models seeks to address previously identified challenges

Challenges identified in current food standards regulatory regime	How the proposed model aims to address these challenges
Inconsistent approaches to regulating food standards across LAs. Different risk rating schemes in use that don’t always reflect the overall level of food business risk.	introduction of a single, uniform risk rating scheme supporting a consistent risk rating approach to food standards by LAs. a risk rating scheme that produces a more accurate assessment of the food business risk profile.
Current model follows an establishment risk-based approach, which is perceived as not the most effective in identifying non-compliances compared to an intelligence-driven approach.	the introduction of intelligence-led action can enable LAs to target their resources effectively and efficiently to emerging issues as required.  LAs can direct resources to food businesses presenting the greatest risk. it develops a proactive approach by using intelligence to identify issues before/as they emerge.  seeks to embed and normalise intelligence as an intrinsic element of regulation by recognising its role in the FLCoP and through the creation of a LA intelligence team within FSA. introduction of a nationally coordinated and directed, intelligence-led sampling programme to further improve market surveillance and our understanding of risk.
Need to target resources more efficiently at the areas of greatest risk.	it better reflects the current context in which LAs operate. It aims to be more dynamic and responsive to emerging risks.  it aims to ensure that LA resources are targeted at the highest risk food establishments.  it provides LAs with greater flexibility in their approach to official controls.   it aims to improve communication, co-ordination, and collaboration between FSA and LAs.

1.2.2.1 The proposed risk scheme

The proposed risk scheme was designed to address known issues with the existing risk schemes, and to modernise the regulatory approach so that it better reflects new food business models and provides a more dynamic and accurate assessment of food business risk. 

For example, the existing FLCoP risk assessment scheme was felt to give too much emphasis to the inherent risk of a food business, failing to adequately recognise the business’s level of compliance. This often resulted in highly compliant businesses being inspected by LAs at a higher frequency than was deemed necessary under the current framework. 

There are two elements to the proposed risk scheme – the Inherent Risk Profile and the Compliance Assessment. Under each of the risk elements sit several risk factors, listed below. Following an official control, food businesses are given a score of 1 (high risk/low compliance) to 5 (low risk/high compliance) for each of the Inherent Risk and Compliance Assessment risk factors. These risk factor scores are then used to inform the frequency of official controls at the food business using the decision matrix.

For the Inherent Risk Profile element, the risk factors (sub-categories) considered were: 

• scale of supply and distribution
• ease of compliance
• complexity of supply chain
• responsibility for information
• potential for product harm

For the Compliance Assessment element, the risk factors (sub-categories) considered were: 

• Confidence in Management (CIM)
• Current compliance level
• Management systems and procedures 

1.2.2.2 The decision matrix

The risk factor scores for each of the two risk elements above are averaged, rounded to the nearest whole number, and then applied to the decision matrix (Figure 1). The decision matrix determines the minimum frequency and the regulatory approach for the LA to apply to businesses. 

For the pilot, the following categories of regulatory approach were introduced:

• Intervention: Regulatory activity including official food controls and other interventions such as education, advice and coaching, information and intelligence gathering. 
• Priority intervention: Where a food business has serious non-compliances they will be subject to more frequent and intense interventions until a satisfactory level of compliance is achieved. 
• Targeted Remote Interventions (TRIs): Where businesses are identified as posing a lower risk, the proposed model recommended that they were subject to remote monitoring (for example, labelling, website or documentary checks) to assess their levels of compliance in place of a physical intervention.
• No Actionable Risk (NAR): Under the piloted risk scheme, food establishments deemed to be low risk and with high levels of sustained compliance would not require a programmed intervention. They would instead be subject to ongoing monitoring by the LA using intelligence and other indicators to determine whether intervention was necessary.

The matrix also included a timeline (number of months) to indicate the minimum intervention frequency expected for each of the categories. Figure 4 shows the decision matrix and the intervention action and frequency that is recommended. With regards to the scores on the matrix, 1 is for the highest risk and 5 the lowest.

Figure 1 Food Standards decision matrix

1.2.2.3 Intelligence-led approach

Intelligence is fundamental to the proposed FSDM. Intelligence-led working is defined by National Trading Standards (NTS) as ‘a business process for systematically collecting, organising, analysing and utilising intelligence and information to guide operational and tactical decisions. Intelligence led aids in identifying, assessing, and managing targets, threats and problems at the local, regional and national level’^(footnote) .

In the context of this report, ‘intelligence’ refers to information that has been subject to a defined evaluation and risk assessment process to assist with regulatory decision making. Where the report references the ‘use of intelligence’ in the context of this evaluation, it describes the way information gathered through intelligence-led working is applied. 

The intelligence function aims to generate a feedback loop whereby strategic food standards priorities are established and an action plan is set (via focused intervention and/or targeted sampling, for example). LAs then deliver the action plan and gather intelligence, with the intelligence data being fed back to FSA to inform the next set of strategic priorities.  

1.2.2.4 Directed sampling

LAs implement a sampling policy and programme as part of their food standards work in line with the FLCoP. Research conducted by FSA in 2020 on food standards sampling showed that this activity had decreased in recent years, due to reduced budgets and resource pressures, as well as differences in the structure and approach to sampling taken by LAs, where individual authorities decide how to distribute their sampling budget. 

As part of the pilot, FSA introduced a directed sampling programme to test a coordinated approach to LA sampling that could be responsive to national intelligence, supported through central LA funding. The programme was intended to supplement, rather than replace, any locally implemented sampling activity undertaken by LAs.

To determine food standards sampling priorities, FSA consulted with wider FSA teams, and used available data (e.g. strategic surveillance data, National Food Crime Unit (NFCU) intelligence, incidents data) to support the need for surveillance or enforcement and/or the identification of intelligence gaps where further knowledge of risk was needed. FSA also considered feedback from local and regional sampling activity to help identify wider food issues.
 
FSA issued sampling criteria to pilot LAs for each phase of the directed sampling programme, which included the products, premises type and analyses to be conducted. The aim of the programme was to utilise intelligence to inform a coordinated and effective sampling strategy. Hypothetical risks and confirmed risks were tested in the directed sampling programme. It was expected that the approach to directed sampling within the pilot would lead to the identification of high proportions of non-compliance, as the sampling had been supported through an assessment of intelligence to identify sampling priorities. This is considered in the following section.

Seven pilot LAs participated in the directed sampling programme and followed the below process:

1. FSA issued LAs with sampling criteria based on intelligence, which outlined the products, premises type/s and specified analyses identified, targeting products with high potential for non-compliance.
2. LAs procured samples and submitted them to their nominated Public Analyst (PA) for analysis. 
3. PAs conducted analysis and issued certificates of analysis to LAs.
4. LAs completed a results spreadsheet for FSA, which was sent with certificates of analysis to FSA.
5. FSA interrogated and analysed the results and outcomes and progressed centralised work to address nationwide issues as appropriate. 

Table 1.2 provides an overview of directed sampling activities and the number of participating LAs. Some LAs were unable to participate in all phases of sampling due to COVID-19 officer redeployment or related restrictions, organisational restructuring, or lack of local availability of specified food samples. LAs were reimbursed for the cost of samples procured and the cost of laboratory analysis of the samples.

Table 1.2 Overview of LA directed sampling participation 

LA directed sampling participation">

Directed sampling phase	Number of participating LAs	Total number of product samples taken	Number of unsatisfactory samples	Sampling priorities
Phase 1  (March to July 21)	4	49	10	Online food businesses, newly registered, unregistered or diversified food businesses
Phase 2  (August to October 21)	6	155	113	Imported food products and unregistered food businesses
Phase 3  (November 21 to February 22)	5	113	43	Vegan products, pork products, weight loss products

1.3 The pilot project

The pilot project started in January 2021 and ended in March 2022, testing the proposed FSDM with 11 LAs (seven of them as part of the pilot and four of them as part of a control group). The timeline of the pilot can be seen in Figure 2 below.

Figure 2 Pilot timeline:

Following a request for expressions of interest to participate in the pilot, LAs were selected by FSA based on their willingness to participate and in consideration of a series of characteristics to ensure a representative sample. The selection criteria were: 

• LAs from England and Northern Ireland^(footnote). 
• LAs using/not using the Register a Food Business (RAFB) service.
• LAs using different risk schemes (FLCoP, NTS, or LACORS).
• LAs delivering food hygiene and standards, and LAs delivering food standards only.
• LAs delivering food standards controls through environmental health/trading standards practitioners. 
• LAs of different authority types for example, county council, unitary authority.

The above criteria ensured that a diverse range of LAs participated in the pilot to avoid bias. While LAs were self-selecting, the criteria above ensured a wide representation of LAs within the pilot. The LAs were also selected on the basis of their ability to conduct inspections and having a variety of types of food businesses registered. The diversity of LAs is described further in the sections below. 

1.3.1.1 Background information on LAs

There are 333 local authorities in England, and 11 in Northern Ireland. In England, there are five categories of LAs^{LAs as of 2022/23</p> " href="#">(footnote)}:

• county councils (24 in total)
• district councils (181 in total)
• unitary authorities (59 in total)
• metropolitan districts (36 in total)
• London boroughs (33 in total)

County and district councils are two tiered and have their responsibility for council services split between them, with food hygiene delivered by district councils and food standards by county councils. Unitary and metropolitan authorities and London boroughs operate under a single tier structure, with those LAs responsible for all services in their area, including food hygiene and standards. There are 163 LAs with responsibility for food standards based on these figures.

1.3.2.1 Selection of participating LAs 

Table 1.3 includes the final selection of 11 LAs^{LAs to participate, but some of these had to drop out before starting the pilot for different reasons such as the impact of COVID-19 on their services.</p> " href="#">(footnote)} that participated in the pilot and their main characteristics, of which 7 piloted the proposed model and 4 were used as the control group. The evaluation team interviewed all 7 LAs testing the model, and 2 of the control LAs. 

Table 1.3 Overview of participants interviewed

Pilot or control	Type of authority	Risk Scheme	Hygiene or standards	RAFB	Participated in the interview (Y/N)
Pilot	Unitary Authority	FLCoP	Both	Yes	Y
Pilot	County Council	Hybrid	Food standards only	No	Y
Pilot	Unitary Authority	NTSB	Both	No	Y
Pilot	Unitary Authority	FLOP	Both	Yes	Y
Pilot	County Council	FLCoP	Food standards only	No	Y
Pilot	Unitary Authority	FLCoP	Both	Yes	Y
Pilot	County Council	NTSB	Food standards only	No	Y
Control	Unitary Authority	FLCoP	Both	Yes	Y
Control	County Council	NTSB	Food standards only	Joining soon	Y
Control	Unitary Authority	FLCoP	Both	N/A	N
Control	Unitary Authority	FLCOP	Both	N/A	N

1.4 Evaluation approach

The evaluation used a mixed methods approach, focusing on capturing LAs’ experience of the proposed model through the entire pilot, to adequately capture nuances in its use and factors impacting its performance. In addition, LAs reported monthly progress and activity to FSA. FSA used this data to analyse, quantitatively, the progress of the pilot and the impact of the new model. This report seeks to answer the following evaluation questions, as agreed with FSA: 

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What was the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What was the effect on resources for each of the stakeholders because of the proposed model?
4. What was the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?
5. What lessons were learnt from the pilot?

The ICF study team developed an evaluation matrix to answer the questions, shown in Table 1.4 below. 

Table 1.4 Analytical approach to answering each evaluation question

Evaluation question	Analytical Approach	Data collection method	Examples of data collected/ indicators
1.    How did the proposed model perform compared to the current framework? What worked well and less well?	Assessment of data to look at whether there was:  Increase in the proportion of non-compliances: -    identified  -    resolved Changes in how LA resources (staff time) are used	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Data generated by FSA evaluation plan	LA and FSA perspective on ease of use of the new approach Enablers / barriers to using the new proposed model (IT, skills, resources, location, type of FBO) LA and FSA perspective on the comprehension of the proposed model
2.    What was the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?	Analysis of the responses to the interviews and survey.  Analysis of the outcomes of the online workshop.	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff	Perceptions by LAs on quality of training received to prepare for proposed model  Opinion of users of the proposed model on ease of communication, frequency, and quality of data FSA perceptions on the proposed model
3.    What was the effect on resources for each of the stakeholders because of the proposed model? Analysis of answers to questions on adequacy and use of resources	Analysis of answers to questions on adequacy and use of resources	Baseline, and two waves of interviews with LAs An online meeting with FSA staff	Opinion on the adequacy of resources for implementing the proposed model Changes to how LA use resources  Changes made by LAs and FSA to adapt to proposed model (costs, staff, IT systems, skills)
4.    What was the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?	Analysis based on outcome indicators from the intervention logic	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Data generated by FSA evaluation plan	LA perceptions of impact on the identification of non-compliance and actions to resolve  Frequency and type of intelligence shared Perceptions of consistency under the proposed model LA views on potential unintended consequences
5.    What lessons were learnt from the pilot?	Analysis of how lessons learned have been captured, and used in the context of the ABC programme	Data generated by evaluation plan Interviews with FSA staff  A series of meetings with FSA staff	Adaptation of the pilot using data generated and analysed if needed Use of lessons learned from the pilot to inform the proposed model

The evaluation has been delivered through the following key tasks over three phases of work - the scoping phase, the data collection phase, and the analysis phase.

1.4.1 Phase 1: Evaluation scoping

The scoping phase included: 

• review of existing documentation, outlining the food standards objectives, activities, and outcomes to date. 
• interviews with FSA staff to understand their objectives, activities and progress to date, and to inform definition of the evaluation methodology.  
• a virtual workshop with FSA to develop an initial Theory of Change (ToC) for the ABC programme. This ToC included the intervention logic for the food standards project. The workshop enabled the development of a shared understanding of the ABC programme between FSA and ICF, a reflection on how the food standards project will be delivered, the outcomes it aims to achieve, and the assumptions underpinning it. This provided an opportunity to think how the overarching workstreams operate together and what contextual factors may influence their success. The intervention logic for this pilot project is included in Annex 2 
• definition of the evaluation framework, including developing an intervention logic model, identifying, and defining the questions to be addressed by the evaluation and the tools to be used. 

1.4.2 Phase 2: Data collection 

The data collection phase included: 

• three rounds of interviews with LAs to understand their expectations, their experience with the process including any challenges, as well as to document the changes implemented due to the proposed model. The first round was completed at the start of the programme in January 2021, the second round took place during April and May 2021, and the last round was completed a month after the pilot project had ended in April 2022. Interviews were held with all LAs piloting the proposed model and included two of the control LAs. The interviews followed a semi-structured format and were recorded. 
• Interviews with FSA staff at the end of the pilot, to capture their experience with the process. 
• Quantitative data collected by FSA (as discussed in Annex 3): FSA’s Analytics Unit collected monthly data from all participating LAs (control and pilot groups) in a standardised format. LAs self-reported this data during the pilot period. Data collected included risk scores from every inspection, data on reactive (intelligence led) vs proactive (programmed) interventions, data on the effectiveness of risk assessment model, the amount of intelligence received, sample results and time/resources involved.

1.4.3 Phase 3: Data analysis 

The data analysis phase was continuous through the life of the pilot project. The data analysis phase included:  

• a series of meetings with FSA staff to share the findings collected, ensure a common understanding of the main challenges and enablers, and refine the data collection tools. It also included meetings with FSA analytics team to understand their data collection process to integrate the data into this report. 
• analysis of the evidence collected such as interview responses, workshop outputs, review of documents provided by FSA. The ICF team collated and assessed the evidence based on the themes in this report. 
• analysing and integrating the quantitative data collected by FSA Analytics team.

The analysis of the quantitative data showed that one of the control LAs distorted the overall analysis results. This was because the LA dealt with a higher than usual number of food incidents during the pilot. This resulted in multiple samples being taken by the LA in relation to these incidents, which is a normal practice in this scenario (but it does not occur often), which led to a large number of unsatisfactory results due to the targeted nature of this work (common when there is an incident). For this reason, the team decided to remove this LA from the analysis and the figures in the main body of the report. The data including this LA have been included in Annex 3 for transparency.

It provides an overview of the baseline findings, focusing on the key factors that influenced the progress of the pilot project, and answers the evaluation questions through analysis of the data and evidence gathered from interviews.

2.1 Baseline

A baseline round of LA interviews was conducted in early 2021 after the Pilot had begun. The purpose of baseline interviews was to compare the changes between the baseline and the end of the pilot, as well as to observe whether the control LAs evolved in a different manner to the pilot LAs. 

Normally, baseline interviews would be conducted before the pilot project starts. However, the study team was not able to interview the LAs prior to the start date due to COVID-19 restrictions and busy LA workloads prior to the Christmas period. The evaluation team, together with FSA, decided to carry out the baseline interviews two weeks after the pilot project had started. It was understood that, at that point in time, officers would not have received any training and would just be familiarising themselves with the supporting documentation provided by FSA. During the interviews, the evaluation team made clear the purpose of the interview and explained what it aimed to capture. 

Each LA was different in terms of levels of available resource, the systems and processes used to deliver food standards controls, their geographical location (e.g. rural or urban) and the overall business profile registered (e.g. manufacturing companies, restaurants or supermarkets). The baseline highlighted how these different attributes may impact the LA’s implementation of the proposed model. 

2.1.1 LA resources

Participating LAs varied in relation to their available staffing resource. Key variations were identified regarding the following:

• number of authorised food standards officers employed;
• experience of officers within teams;
• team capacity;
• officer absence, either planned (maternity, retirement) or unplanned.

LAs stated that staff were typically experienced and enthusiastic about their job. However, staff resource was a commonly reported issue as well as the lack of new professionals entering the workspace. 

LAs mentioned that the higher the number of dedicated staff, the easier it would be for them to carry out their food standards work. In addition, some LAs reported experiencing a high turnover of staff. Prior to the start of the pilot, it was expected that those LAs with more experienced team members and greater available resources would be able to adapt and use the proposed model more successfully.

2.1.2 Geographical location 

The location of a LA can lead to operational differences. LAs highlighted the following considerations: 

• the size of the LA area can affect the amount of travel time it takes for officers to conduct physical inspections. For example, officers in rural areas may be required to spend more time travelling to deliver interventions compared to urban areas. 
• LAs with a particular local interest in the Geographical Indication  scheme or products claiming to be ‘locally produced’ can experience a higher risk of food fraud compared to other areas.
• the type/s of business registered with the LA can pose different risks, with some being more complex than others. For example, some areas have a greater proportion of large manufacturers registered than others, which can have an impact on regulatory delivery. 

The implications of this in relation to the proposed model could include a greater need for resource in some LAs compared to others, and the need to ensure the new risk scheme can identify all types of risk (including those specific to local areas). These variations could also mean that the use of Targeted Remote Interventions (TRIs) gives greater benefit to LAs covering large geographical areas, compared to LAs covering smaller, densely populated areas. 

2.1.3 Type of local authority

As highlighted in section 1.3 above, the pilot included LAs of different types to ensure suitable representation.

In the pilot there were: 

• 4 Unitary Authorities implementing the pilot and 3 as control;
• 3 County Councils implementing the pilot and 1 as control.

The variation in LA type can affect different elements of the proposed model. For example, some of the LAs in the pilot reported that they did not have access to the Trading Standards Intelligence Database (IDB). This could have an impact on the LAs ability to share intelligence with FSA. IDB is a database where LA officers can upload intelligence and access intelligence submitted by others. FSA has access to the database, providing oversight of intelligence recorded in the system at a national level.

The type of authority will also have geographical implications, as discussed above.

2.1.4 Management Information system (MIS)

Participating LAs used different Management Information Systems (MIS) to record their activity, and capture and record data in different ways. Some LAs reported concerns around data quality and accuracy in relation to registered food business data held on their MIS. For example, some food establishments had duplicate records, or showed as active despite having ceased to trade. This posed further challenges during COVID-19, as LAs experienced a significant increase in the number of food businesses registering. 

2.1.5 Use of intelligence

LAs were already using intelligence to guide their work to different extents. In terms of intelligence gathering and sharing, some LAs reported strong relationships with regional liaison groups, FSA, and strong relationships across environmental health and trading standards teams, whilst others did not have such relationships in place. As mentioned above, some LAs had access to IDB while others did not. Prior to the pilot, participating LAs had different ways of recording and sharing intelligence, information, and data. 

Existing practices for sharing intelligence by LAs

LAs used a range of networks to share or receive intelligence, including other LAs, regional/in-house intelligence officers, internal colleagues, FSA, other agencies, sampling data, regional and national food standard groups, HMRC, the police, public analysts, and the Knowledge Hub(footnote).

Food standards intelligence has been inconsistently uploaded on the National Trading Standards IDB database. LAs had different perceptions on uploading information to IDB, with some LAs not uploading information at all, and others uploading frequently. 

The type of information that was uploaded to IDB also differed among LAs, with some only uploading verified information that they perceived to be a national consideration. Other LAs uploaded information that they perceived could contribute to developing a national picture of an emerging risk, and recorded intelligence more frequently. 

Due to the differing levels of maturity in terms of operating an intelligence-led approach at a local level within LA regulatory services, during the pilot some LAs were better able to adapt and use the intelligence function of the proposed model than others.

2.1.6 Ways of working

Most of the LAs interviewed were not following the Food Law Code of Practice (FLCoP) in a consistent manner to guide their work. The LAs explained that they were already working in similar ways to the changes suggested by the proposed model and had already adapted their approach to delivery to be able to target their resources more effectively. For example, some LAs stated that they would prioritise interventions at all high-risk establishments identified under the FLCoP intervention rating scheme (or equivalent), and target remaining resources based on a consideration of national, regional and/or local intelligence. 

As discussed in section 1.2.2.5, LAs were already taking different approaches to food standards sampling activity. Some LAs were participating in sampling programmes set by regional groups, or used intelligence to guide their local sampling, while others undertook sampling on a reactive basis as a result of information received, such as consumer complaints. 

2.2 External factors

Several external factors impacted the pilot, with COVID-19 being the most significant. LA resources were affected differently by the impacts of COVID-19, however most LAs experienced staff reallocation due to the need to carry out COVID-19 related tasks. 

Some LAs reported a large increase in new food businesses registering during COVID-19, with some of these businesses being based in domestic settings. LAs struggled to keep their databases up to date during this time and struggled to assess the risk profile of some of these new businesses in a timely manner.  

The various impacts of COVID-19 hampered the ability of pilot LAs to fully embrace the proposed model and had an adverse effect on FSA’s ability to collect evaluation data due to the enforced changes in LA workload during this time. To mitigate these impacts, FSA extended the pilot for 3 months to enable additional evaluation data to be collected, with the pilot ending in March 2022 rather than the original date of December 2021.  

In terms of the directed sampling programme, the pilot allowed LAs to choose whether to get involved in the three phases. Some LAs struggled to participate, particularly during the first phase, for several reasons. These included the impacts of COVID-19 (the redeployment of LA food officers to public health work) or, in some cases, due to officers not being able to visit their offices, meaning they were not able to access internal systems necessary to support the sampling (for example, to make online purchases or to take receipt of samples). 

2.3 How did the proposed model perform compared to the current framework – what worked well and less well?

Overall, the model operated well, with evaluation of the pilot indicating that: 

• the proposed model was appropriate and fit for purpose. The proposed risk scheme was suitable for assessing and identifying risk and prioritising LA resources. 
• the proposed model was easy to use and could be easily integrated into existing working practices.
• the proposed model introduced a single, uniform risk rating scheme to assess the risk profile of FBOs more accurately. It was dynamic and integrated intelligence for a more targeted and efficient response, enabling LAs to focus their resources on food businesses presenting the greatest risk. 
• LAs viewed the directed sampling approach as being effective at identifying non-compliance and providing useful intelligence.

On the other hand, the intelligence element of the proposed model requires further consideration:                         

The intelligence element was not fully developed by the end of the pilot. 

• at the end of the pilot, there was no consistent approach to what information FSA and LAs considered to be intelligence; the type of information which should be shared between LAs and FSA; the frequency of intelligence sharing and the most suitable mechanism to share intelligence. 
• LAs found it challenging to carry out all the directed sampling activities as they felt these were not planned far enough in advance. 

In terms of the implementation of the pilot: 

• the support provided by FSA throughout the pilot was well received and crucial to the success of the pilot. the main challenges to the first steps during the implementation of the pilot were the lack of clarity with some terms used in the risk scheme, the lack of consistency of data held by LAs (both issues addressed during the pilot), and the lack of compatibility of the LAs MIS for the purposes of the pilot. It should be noted that LAs’ MIS integration was not part of the pilot to using the proposed model.     

This section presents the findings relevant to how well the proposed operating model worked. It assesses whether the model was fit for purpose, its ease of use and any challenges identified in using the proposed model. This section also analyses the context in which the pilot was delivered, assessing the enabling factors and barriers faced during the implementation of the pilot project itself. The effects of the proposed model, namely whether it achieved the outputs and outcomes as set out by the intervention logic (see Annex 2) are discussed in section 2.5 of this report. 

2.3.1 Appropriateness of the proposed model (fit for purpose)

The proposed model tested during the pilot sought to address the shortcomings identified with the current FSDM.

LAs participating in the pilot agreed that the new risk scheme was fit for purpose and more appropriate at identifying risk than the previous approach they were using. LAs felt the three elements of the proposed model worked well, providing a more realistic and up to date assessment of the risk posed by businesses and the associated frequency of official controls. This provided greater flexibility, allowing LAs to further target their resources more effectively. As such the proposed model addressed the challenges identified in Table 1.1 Each of the three elements of the proposed model (see section 1.2.2) are assessed below:

2.3.1.1 The proposed risk scheme

The proposed risk scheme was very well received by LAs overall. A positive change when compared to the existing model was the fact that LAs had experienced difficulties in effectively assessing all business types under the current framework. For example, LAs explained that there had been an increase in home catering businesses, but that these did not easily fit in with the existing scoring system, whereas the proposed risk scheme allowed LAs to score them more accurately. 

Also, previously large manufacturers were identified as high risk, regardless of levels of previous compliance due to the categorisations in the FLCoP. However, these types of businesses (where premises are often part of a larger company who conduct internal audits and have specific processes to follow) were identified as being less of a concern for officers.  Officers highlighted the benefit of using their knowledge of a manufacturers’ previous performance and management to assess the risk posed by these businesses, and for this to inform the intervention frequency. As such, under the proposed model this type of business would now be more likely to be considered lower risk if it can demonstrate high levels of compliance with suitable management systems in place. LAs appreciated being able to use their judgement to reflect the risks and prioritise resources, as this allowed them to target their resources towards businesses presenting a higher risk, rather than focusing on the same businesses for annual interventions.

LAs reported that the proposed model allows for more flexibility than the existing one, enabling them to effectively target resources to higher risk establishments. This is because the proposed model combines the assessment of risk and compliance of FBOs, giving increased emphasis to the level of compliance within a business rather than focusing on inherent risk, allowing for more granularity in the risk assessment process. 

In terms of implementation of the pilot, LAs identified initial issues relating to the conversion of their food business risk data to the new risk scheme. This was because not all LA databases were up to date, and the resulting lack of data quality negatively impacted the conversion process in some cases. These issues were addressed together with FSA during the pilot.

LAs mentioned that the identification of non-compliances linked to allergens had been difficult for them using the proposed model. By the end of the pilot, LA officers became more familiar with scoring allergens under the proposed risk scheme. 

“I think part of the issue [with allergens] was that we were not scoring them quite correctly. So like allergens for us, because it’s a safety issue, we’d always revisit and I think some officers were sort of scoring that as a three, and then when we went through it with FSA it was actually a one because its safety. So I think that was maybe a bit of an issue with officers not scoring them harshly enough.” – Unitary Authority

LAs believed businesses with allergen risks should be targeted with a physical intervention, as these are more effective at identifying non-compliance, particularly in takeaways or restaurants. These types of businesses were perceived as being less suitable for a TRI. One LA suggested that the new risk scheme could be improved in relation to allergens by developing the descriptors included the risk rating scheme. Another LA suggested that the proposed risk scheme identified establishments that are more likely to have issues across their business, rather than identifying and giving a suitable level of emphasis to specific allergen issues.

2.3.1.2 The decision matrix

LAs highlighted that the decision matrix allowed risks to be assessed and balanced across different types of businesses. Further, it included multiple criteria (the inherent risk profile and the compliance assessment risk sub-categories), allowing for a more accurate assessment of the risk posed by an FBO and a more appropriate frequency of official controls based on the associated level of risk. 

LAs agreed that the decision matrix was easy to understand. They liked the ability to rescore premises after a re-visit, allowing a change in the risk rating. For example, a LA mentioned that previously, if a non-compliance was identified at a Category A establishment, continued re-visits were required to the premises until compliance was achieved, but the officers were not able to change the risk score to reflect this. The proposed model enables rescoring to take place in such instances. If the officer is satisfied that the non-compliance/s have been resolved and the officer has confidence in the FBO, premises can more easily switch risk ratings, thereby impacting the date of next intervention.  

As previously mentioned, several LAs highlighted that a key benefit of the proposed risk scheme and decision matrix was that manufacturers were no longer considered high risk by default. This was perceived as a benefit by LAs who felt that TRIs could be an appropriate approach for some manufacturers, as they typically have strict systems and procedures in place and they could usually supply copies of relevant documentation, internal procedures, and product labelling etc. to enable their levels of risk and compliance to be assessed remotely. 

2.3.1.3 Intelligence-led approach and directed sampling

The proposed model encourages LAs to collect, share and action intelligence to inform the delivery of official controls. It defines intelligence as an all-encompassing term and aims to encourage intelligence-led working. LAs recognise that intelligence is important and has the potential to help identify risks and target work. This flexibility in working and conducting interventions is a key feature of the proposed model, and LAs reported that officers positively perceived using intelligence to guide their work as the pilot continued. At the end of the pilot, LAs reported becoming more familiar with using intelligence in accordance with the proposed model, and as a result were uploading information to IDB more frequently. 

Prior to the pilot, most LAs reported using intelligence to guide their work to some extent but noted that intelligence was infrequently and inconsistently received. Challenges relating to the use and receipt of intelligence persisted throughout the pilot, but a general positive trend was reported through increased recognition of the role of intelligence. 

LAs noted that, during the pilot, they had received more intelligence from FSA, and they recognised the potential benefits of working in a more intelligence-led way, rather than organising their work based on a list of programmed interventions identified by the FLCoP. 

LAs reported that officers were increasingly considering intelligence received to determine the relevant priority of their workload. One LA suggested that intelligence recorded by officers and shared with FSA should be prioritised and assessed against multiple sources of data to ensure that officer resource is targeted at areas that present the greatest risk. They also noted that acting on early intelligence received by FSA could limit the risk posed to consumers. LAs mentioned further that, if no risk was found after an initial investigation, LAs could communicate their findings to FSA to gather insight from other authorities. 

However, as discussed later in (see section 2.3.3), LAs still had an inconsistent approach to using intelligence by the end of the pilot. Challenges identified during the pilot, regarding inputting information onto IDB and strengthening relationships with external agencies, persisted throughout the pilot, and were still present at the end of the pilot. This was explained partially by the fact that some LA officers were more familiar with the type of information to share, and the mechanisms to do so than others. 

The pilot addressed these challenges to some extent, and there was an improvement in the way LAs using the proposed model were using intelligence and facilitating communication channels from LAs to FSA particularly. Once the pilot concluded, FSA recognised the challenges associated with embedding this way of working across all LAs and established a team to lead these changes and address the challenges.

Part of the pilot was to test the proposed directed sampling programme, to ensure that sampling was co-ordinated and effective, and resulted in measurable corrective action by LAs where appropriate. This intelligence-led programme introduced a more targeted approach to sampling, supported by the identification of risk-based national food standards sampling priorities based on an assessment of available intelligence and other data.

Overall, LAs viewed sampling as being effective at identifying non-compliance and gathering intelligence. One LA highlighted that they had a large sampling programme and that participating in FSA’s directed sampling did not cause them significant issues.

“FSA directed some of the samples that we were routinely going to take anyway so we took less of our own samples and substituted those with samples that FSA would take” – Unitary Authority

LAs did not identify testing the directed sampling approach as challenging. However, they reported practical challenges related to planning the implementation of the sampling (for example, locating all necessary food products) and the availability and capacity of the Public Analysts (as discussed in the intelligence section).

2.3.2 Ease of use of the proposed model

LAs participating in the pilot perceived the new risk scheme to be easy to understand and felt that it allowed them to prioritise interventions in a more effective way than before. LAs reported that the flexibility of the new model enabled them to effectively target resources to higher risk FBOs. However, there were mixed perspectives regarding the role of allergens and the use of TRIs as part of the new model. 

Prior to the pilot, LAs had used the FLCoP to prioritise their resources on an annual basis. In some cases, LAs used local, regional and/or national intelligence to help target their resource. However, in baseline interviews, some LAs reported that they were not able to complete all scheduled interventions required under the FLCoP due to a lack of capacity. These LAs noted that before the pilot, they would prioritise completing high risk (Category A) and medium risk (Category B) interventions. 

The flexibility in the proposed risk scheme allowed officers to make an informed decision on the appropriate intervention frequency for an establishment based on their informed assessment of risk, which considered the internal processes and management of the business. This was identified as being a positive feature because under the current FLCoP, officers highlighted that if they acted on intelligence, it would not be recognised as part of their annual report on official control delivery.

Overall, the proposed model was viewed as being compatible with the way that LAs perceived food standards should operate, as it identified the major risks, and for some the proposed model fitted with how they were already working. The ability for LAs to act on intelligence received was viewed as a key strength of the proposed model, because it enabled them to justify the redirection of resources as necessary, rather than strictly following the FLCoP. 

2.3.2.1 Targeted Remote Interventions

LAs had mixed perceptions on the benefit of TRIs. Typically, TRIs were viewed as being appropriate for certain business types, such as manufacturers or microbreweries, where it was typically felt that the FBO would be able to provide the required information to an officer quickly and easily. Some LAs highlighted that smaller businesses may not always have the information to hand, and that it can take a lot of time for officers to receive and assess the information. 

For establishments that were viewed as presenting a higher risk for allergens, such as takeaways and restaurants, a TRI was typically not viewed as suitable. LAs identified that a physical inspection is more appropriate when conducting a proper allergen assessment because officers can verify information that is provided by the FBO. The verification of information was identified as a weakness of TRIs, and some LAs highlighted that FBOs could provide information that would likely negate the need for a physical visit to intentionally mislead officers. Officers also typically provide advice and assistance to FBOs when they conduct a physical intervention, which is viewed as being a valuable means to increase compliance.

“[The FBO] is going to send you data or information or specifications but then they might have something to hide. They might still hide it when you’re there for an intervention but at least you can open cupboards and have a good look round and see their menus.” – Unitary Authority

“[TRIs] work well for manufacturers, especially manufacturers you’ve been to before where you don’t necessarily need to do a site visit. So, if you can look at product specs and labelling and stuff, the TRIs work well for that because you can get them to email it to you and have a chat… it doesn’t work for pubs, shops, that sort of thing because you need to go and see them.” – County Authority 

Some LAs also reported a difference in officers’ commitment to TRIs. Some officers were engaged with the TRI approach and reported positive experiences in conducting them, whereas other officers reportedly were not as engaged. 

One LA highlighted they were not considering TRIs because they had a significant number of establishments that are categorised as needing an annual intervention. This LA also highlighted that, for weights and measure inspections, a TRI would be unsuitable, so they would carry out a physical intervention whilst doing the weights and measures inspection. 

One LA highlighted that they had historically sent out questionnaires for their low-risk establishments and conducted a physical intervention on a 10% sample of these responses to identify how truthful FBOs were in their answers. This LA highlighted that they would decide to conduct a physical intervention at an establishment that was identified as being suitable for a TRI if they received intelligence. 

2.3.3 Use of intelligence

The main challenge identified with the proposed model related to the intelligence element. While improvements were made through the duration of the pilot, there remain considerable challenges. 

The proposed intelligence-led approach features a feedback loop whereby LAs share data or information with FSA, who then collect and analyse the information before disseminating relevant intelligence back to LAs so it can drive their work (see section 1.2.2.3). The proposed model aims to produce strategic assessments informed by intelligence, information and data shared by LAs and other sources. This will allow FSA to set priorities and intelligence requirements for LAs to assist in directing their interventions and intelligence gathering activities. 

Prior to the pilot, engagement indicated that most LAs were already using intelligence to inform their delivery of food standards official controls. It was anticipated that the implementation of the intelligence-led approach to food standards delivery would be less complex than it became.

During the pilot, FSA team identified that each pilot LA understood and worked with intelligence very differently, meaning the assumed baseline understanding was not consistent. The team also recognised a need for FSA to review and adapt its internal processes regarding sharing and receiving intelligence to properly support the new model. These were unexpected barriers that led to a delay in integrating an intelligence-led approach to the delivery of food standards. This work is still ongoing and essential to supporting LAs with the integration of intelligence-led working. Further work is required to continue to understand the range of processes and knowledge of intelligence across England, Wales, and Northern Ireland. 

2.3.3.1 Challenges on what type of intelligence is shared

LAs interpreted intelligence as being verified information of local, regional or national relevance that has an evidence base and includes specific details. In terms of definitions of intelligence, some LAs mentioned specific features of what could be classed as intelligence including: it should be corroborated somehow, it should lead them to carry out some sort of action or to target resources better, it relates to something that is potentially illegal or fraudulent. Some LAs perceived a difference between "intelligence" and "intelligence that would be put on IDB" and would only share information to IDB if they perceived it to be relevant beyond their local area and serious enough to warrant notification, such as cases involving fraud or risk of harm. 

LAs reported being familiar with receiving intelligence and working with other agencies, and they welcomed the intelligence-led approach under the proposed model. Typically, LAs use formal and informal sources of data to guide their work, though ‘formal’ intelligence received through IDB or trading standards groups was reported as being more reliable than complaints (considered ‘informal’). LAs noted that, before and during the pilot, they did not receive intelligence frequently from external agencies. Most of their daily work was guided based on local intelligence, such as officer knowledge or consumer complaints, but not on national or regional intelligence. This was also reflected by a control LA, who reported that intelligence received from other enforcement agencies can be vague, which is unhelpful in guiding their work. 

The ground level intelligence gathered by officers and received through public complaints was identified as being essential in identifying local issues. Some LAs suggested that it would be detrimental to uncovering food standards issues if there was an overreliance on intelligence that is communicated through formal agencies. 
LAs also mentioned they were familiar with working with regional groups, FSA (including the NFCU), and wider agencies such as Port Health Authorities and the Police. LAs often only share data they consider ‘exceptional’ information. One pilot LA highlighted that they would only share ‘important’ data with other bodies such as the police and HMRC, and that most of the intelligence that guided their work would be considered ‘mundane’ by other agencies. Usually, routine intelligence provided in a complaint would not be communicated to other agencies or uploaded to the IDB database. The type of information that LAs would consider intelligence includes regional or national issues, such as counterfeit spirits or product labelling issues. LAs highlighted that if they identified a high-risk issue that could harm consumers, they would communicate this information straight to FSA. 

This way of working appeared to be a relatively standard practice, however LAs were not sure whether, due to the proposed model, they should now report more intelligence through IDB, or whether local intelligence should be shared with FSA (via IDB or some other channel). 

Further, LAs suggested that the information received via other mechanisms (not collected by the LA itself) often was not enough to guide their food standards work. This could be because other agencies (e.g. NFCU) collect very specific types of intelligence, which LAs don’t always find useful. During the pilot, LAs reported sharing intelligence more frequently and reliably than before the pilot, particularly with FSA. However, LAs mentioned that consumer complaints remained the largest source of intelligence they use, and that they did not really receive much information from external agencies. There seemed to be disconnect between LAs and FSA regarding the interpretation of intelligence, which persisted in all rounds of interviews. 

At the end of the pilot, one LA reported that their officers were considering information that they would have previously regarded as being a local isolated issue as potentially providing intelligence to more LAs and informing FSA. However, the findings suggest that LAs could benefit from intelligence shared by other LAs or agencies. There was not a consistent approach agreed by the end of the pilot on the type of data that needed to be captured and shared. 

2.3.3.2 Challenges on the frequency that intelligence is shared

The other difficulty was the lack of agreement on how often data should be shared. 

LAs record complaints and share data in different ways, with some of them recording every type of information as soon as they receive it and sharing it with others, while other LAs do not always consider the information they gather as intelligence, leading to data only being shared with FSA sporadically. One LA highlighted they would not record every complaint received on IDB but they would only do so if multiple complaints were received. One LA highlighted that there was no available resource to dedicate officer time to uploading information on to IDB so they would not share data, with two other LAs reporting that officers inconsistently upload information. Across LAs, not every complaint is actioned, and officers use their professional judgement to decide whether further investigation is needed. Allergens were a priority for all LAs and non-compliance would always lead to further investigation. 

Some LAs agreed that receiving intelligence about an issue at an early stage could lead to risks being identified and addressed earlier on a national scale. LAs also suggested that intelligence sharing would be more effective if there were routine collaborative efforts made to coordinate the intelligence collected by LAs and other partners. 

2.3.3.3 Challenges with the mechanisms to share intelligence 

There was no standard method for gathering and sharing intelligence among the LAs in the pilot project. At the same time, FSA was yet to develop an effective mechanism to receive intelligence from LAs in a way that would facilitate rapid processing and analysis to provide timely feedback to the LAs. 

Most LAs viewed regional food liaison groups as being a useful forum to share intelligence, and it was noted that Trading Standards are a well-connected professional community. Groups that include wider agencies were identified as being a less useful source of intelligence. LAs perceived that some agencies were reluctant to share information with them. LAs mentioned using regional groups to share data. This could mean that LAs and FSA could strengthen intelligence networks to enable and facilitate communication between partners.

In addition, some LAs use databases to share intelligence. One control LA reported that all complaints and intelligence are recorded on their MIS, whereas a LA testing the proposed model reported scoring intelligence, received in the form of complaints, based on their own internal intelligence matrix, which provided criteria that could trigger an intervention if needed.

All LAs with Trading Standards functions and with access to IDB shared and recorded intelligence using the database. However, not all LAs use it in the same manner, leading to potential inconsistencies in the way that LAs share intelligence with other agencies. During the pilot, LAs were encouraged to use IDB more frequently for food intelligence and to input more information than they did prior to the pilot. Consistent challenges were reported throughout the pilot related to IDB. Two LAs identified that IDB does not provide any feedback to officers or provide any information as to how their intelligence is used. These LAs suggested that if this additional feature was included, officers may be more proactive in uploading information to IDB.

Challenges with using IDB and inputting information remained at the end of the pilot. Officers have a limited amount of time available to input information and there were different perspectives regarding the usefulness of IDB, with LAs typically reporting that it is less appropriate for recording food standards information. However, throughout the pilot, in general LAs have increased how frequently they upload information to IDB. It was suggested that if LAs received recognition for contributing intelligence to the database, it may encourage officers to upload information more consistently. 

“[IDB] is based on the police model, so it’s not really ideal for food, which is the first problem with it. Its more based on people whereas food is more based on products, so the system is about identifying people and activity and linking people together, which is great if you’re looking at car crime but with food it’s much more about the product, so it doesn’t lend itself to the other issues, so we don’t tend to put food information on because its data entry twice” – County Council

“we’ve increased our usage [of IDB] a bit but I think officers still struggle with the idea that IDB is an appropriate place for food” – Unitary Authority

Finally, some LAs mentioned using the online platform Knowledge Hub and local complaints as informal channels of communication to guide their work. LAs also reported having communication with the police, other local authority departments and the public on an informal frequent basis. 

2.3.3.4 Challenges of communicating outside FSA

Food is not one of the National Trading Standards (NTS) priorities. The NTS priorities include fair trading, illegal tobacco, intellectual property, and marketing fraud. As food issues are not considered in the decision making that informs these priorities, food standards teams reported feeling overlooked in this process. This can be demotivating for officers who upload food intelligence frequently and consistently to IDB because they do not receive recognition from other agencies for their contributions. LAs suggested that promoting their food work and improving relationships with other agencies, could lead to a mutually beneficial relationship, where food standards concerns would be more respected. 

The challenges related to the sharing and use of intelligence across organisations are beyond the scope of the proposed model but are key considerations in the practicalities of the effectiveness of intelligence. 

2.3.4 Implementation of the pilot

2.3.4.1 Enabling elements

FSA support was the key enabling element in the success of the pilot. The support can be divided into informal and formal support, as discussed below:  

The support received from FSA before and during the pilot was reportedly very useful to LAs. The pilot opened a line of informal communication between FSA and the LAs, where they could collaborate to solve issues together. 

FSA were identified as being responsive to communication. LAs felt they had a ‘point of contact’ where their queries would be addressed or signposted within FSA to the relevant department if the query was not related to the pilot. Most LAs highlighted that they always had a positive relationship with FSA but FSA’s receptiveness during the pilot had helped to strengthen relationships. 

‘I’ve certainly got to know a few more officers over there [at FSA] and I feel like they’ve always been very helpful when I’ve had a query and very approachable and so hopefully that will continue following the pilot’ – Unitary Authority 

FSA helped build confidence across LAs and helped embed the proposed model into the LAs’ working practices, which is a positive indication on the success of the pilot. Overall, LAs perceived FSA as being proactive and receptive to their ideas throughout the pilot, and they hoped that the communication and engagement would continue. 

In addition, the collaborative process adopted to develop the risk assessment scheme and the decision matrix was appreciated by LAs and FSA. 

LAs also reported sharing learning with their teams. In particular some of the LAs participating in the pilot highlighted they were sharing learning as part of the regional groups they participate in. Some LAs reported conducting their own in-house consistency exercises with officers to help provide further reassurance. 

A key enabling element for the success of the pilot was effective regular communication between LAs and FSA, as there needed to be a common interpretation of the risk scheme and decision matrix. Prior to the pilot, FSA carried out a series of training activities to introduce the aspects of the proposed model. FSA provided specific training on intelligence and on the proposed risk assessment model. LAs appreciated the training provided and found it valuable in preparing them for the pilot. 

Some LAs which were already familiar with using intelligence reported that the training on the use of the risk assessment scheme and decision matrix was more useful than intelligence training, as intelligence was already well understood by them. 

In addition, FSA’s National Food Crime Unit (NFCU) delivered intelligence training to the pilot LAs as part of a wider training programme to aid consistency. Feedback from this training indicated that officers would appreciate a greater focus on applying intelligence in a food-related context, with specific examples related to their areas of work, as this would help officers apply the training in their day-to-day activities. 

FSA delivered two consistency exercises in relation to the new risk assessment scheme. These helped reassure officers that they were scoring correctly by comparing food businesses that presented different risks and explaining the scoring criteria. The first session was delivered in the early stages of the pilot (January 2021) to refine some of the scoring criteria and provide officers with a greater understanding of each element of the risk factor scoring criteria. This clarity was very useful to officers. LAs also reported carrying out this type of consistency exercise with officers in their internal team meetings, where they would discuss scoring and share ideas with each other, helping to build on the consistency exercises hosted by FSA. LAs perceived that these consistency exercises would be useful in the early stages of a potential national roll out of the programme. 

“Each officer who has gone through that consistency exercise comes out with roughly the same scoring and so I think the information that’s supplied or been amended throughout the pilot has helped enormously with the consistency of the scheme and the scoring system” – County Council

Pilot LAs mentioned, however, that the second session (‘Risk Assessment Scheme Consistency exercise’ February 2022) was delivered slightly late in the pilot, and that it would have been more beneficial to have had this session earlier in the pilot. 

2.3.4.2 Barriers to implementation 

Most of the challenges identified during the interviews were related to practical issues experienced by LAs in integrating the proposed way of working with their current practices for the purposes of the pilot. The main challenges were: 

Clarity on terminology

LAs identified that the description of some terms in the proposed model were not always clear. Two LAs identified that the risk scheme could be improved by increasing the clarity of definitions. A specific example was provided in relation to the definition of a ‘single local supplier’ within the ‘Complexity of Supply Chain’ risk factor, and where they procure their stock. The LA stated that the risk scheme considers where an FBO purchases their food. Whilst many

FBOs will buy from one supplier local to their business, that supplier could be a multinational business that imports foods globally from a number of distributors. However, suppliers that import food do not have a consistent supply chain because they will buy products from the cheapest source. As such, one officer could identify an FBO as having a single local supplier and being low risk, whereas another could view the FBO as being more complex because they were sourcing products from a larger, multinational business. Another LA highlighted that the proposed model could be improved by including the descriptors from the FLCoP as this would provide clarity to officers, helping to ensure consistency, particularly for definitions of single local supplier.

''The only hiccup was really regarding some of the definitions within the risk assessment and officers perhaps having different views but I mean we're only talking one or two, on the whole it [the pilot] worked fine for us'' - Country Authority

Data consistency

Not all LAs had their databases up to date. As such, there were difficulties when incorporating the food businesses into the proposed model. For example, there was out of date information, duplicate entries, and uncertainty over whether businesses were still in operation. One LA experienced this more acutely than others. Nevertheless, LAs noted that FSA officers were very helpful in providing support to ensure data consistency. These data issues led to challenges when the LA food business risk data was converted to the proposed risk scheme. 

Data issues were reportedly compounded by COVID-19, as LAs experienced a significant increase in the number of FBOs registering at a time when footfall in businesses was being discouraged due to the pandemic. It was reported that LAs were not aware of FBOs that may have registered and then not opened. 

“During the pandemic, people were looking to keep themselves busy… we’ve had a national explosion in the amount of food registrations that are received for new food businesses… most are home factories” – Unitary LA

“We had about 1,000 new food registrations in about 3 or 4 months” – County Council

They highlighted that this could be a wider issue, and it will likely be experienced by other LAs during wider implementation. Ensuring consistency in LA data prior to national roll out will be important.

Compatibility of MIS 

LAs use MIS to record and store information across their authority’s service delivery. MIS databases are used by authorities to plan work and record premises-level activity and enforcement data and other information about the businesses they regulate in several different service areas. LAs usually have longstanding contractual commitments to their MIS software provider and the databases are often used by different teams and services within an authority to store a range of information. 

MIS are used by food standards teams to generate inspection dates, plan interventions, record their activities and to maintain premises records. It is difficult for LAs to change software provider and the function of the MIS database is established centrally, by the software provider. The proposed model was not integrated into the LAs MIS for the purposes of the pilot as this was not practicable. This presented challenges related to the recording of premises information for most LAs for the duration of the pilot. 

The most significant challenge identified related to the operational delivery of the proposed model and its compatibility with LA MIS. At the time of the pilot, across LAs there were several MIS providers, including Civica, IDOX and Tascomi. LAs highlighted that it will be important for FSA to ensure the proposed new risk model is compatible with all MIS providers prior to rollout. 

While the pilot did not expect LAs to reconcile MIS databases, the issue caused operational difficulty for pilot LAs and led to initial differences between the progress made by LAs participating in the pilot. Some LAs were not able to use their MIS during the pilot and had to input data manually. They identified this as a major challenge of pilot participation and reported that this affected their resources, as officers were required to duplicate information, recording it on both the excel spreadsheet provided by FSA for evaluation and on their MIS. 

One LA did manage to configure their MIS to be compatible with the new risk scheme and was the only LA that did not report challenges related to the MIS integration with the proposed model. The ability of this LA to resolve this compatibility issue is an important consideration for FSA, as it suggests that the issue can be resolved. 

Given the lack of consistency in MIS across LAs, it was recommended that FSA communicates with all providers to ensure that systems would be compatible with the proposed model.

Use of intelligence

As indicated above, Pilot LAs were at different stages of understanding and use of intelligence at the start of the Pilot. This led to intelligence-led activities, such as directed sampling, being adapted, or delayed.  

FSA, with input from the NFCU, delivered intelligence training for the pilot LAs. This included what it is, how it can be used and the appropriate mechanisms for gathering and sharing it. Intelligence training, guidance and support is an ongoing piece of work that FSA is seeking to develop and continue. It is recognised that there is no consistent approach to how LAs understand and use intelligence and something that will incrementally develop.  

Directed sampling 

Practical challenges were reported by LAs related to the directed sampling programme. These included locating the specified food samples and the timeframes for receiving results from the public analyst (PA) laboratories. 

Typically, LAs plan their sampling programme for the year ahead and inform their PA, which allows the PA to plan their resources accordingly. The intelligence-led approach to directed sampling meant that the timeframes for notification of sampling criteria and the submission of samples to the PA were shorter than usual. The proposed approach requires LAs to be more dynamic and responsive in the management of resources to address emerging risks, but this was identified as a difficulty by some LAs. 

“The sampling caused us problems we really need to plan at the beginning of the year… having to get samples back in the next 4 weeks did not work for us or the analyst… we still haven’t got the samples back that we took in February because the analyst needs to plan throughout the year.” – Unitary Authority

“It’s managing the [sampling] process both here and at the analyst and there’s no guarantee what samples the analyst is going to get next year so it puts an enormous amount of pressure on a system which is already straining.” – Unitary Authority

The lack of planning in advance also affected LA officer resources and work scheduling. It was identified that the directed sampling programme did not always correlate to officer availability and one LA experienced difficulties in locating the samples required, which was identified as a further strain on resources. 

​There were three distinct phases within the directed sampling programme. In phases 1 and 2, FSA issued sampling criteria to LAs in a bundle, with a long period of time assigned to officers to procure samples. In the third phase, following feedback from pilot LAs, sampling criteria were issued to LAs as multiple, staggered priorities with shorter sample procurement windows. Feedback from pilot LAs on the different approaches was then considered by FSA during the evaluation period. 

2.4 What has been the experience of each of the stakeholders with respect to the specific elements of the proposed model and the proposed model changes as a whole?

 The experience of each stakeholder (LAs and FSA) was overall very positive. 

• LAs decided to join the pilot to co-create the program with FSA, influence the change, and to be able to adapt early to it.  
• LAs identified that the main challenge was related to staff capacity and some technological challenges. 
• FSA mentioned that regular communication with LAs had been the key to the success of the project. They perceived LAs as being very open and honest, which allowed FSA to adapt the tools and guidance developed in a way that could best serve LAs and the objectives of the proposed new model. 
• a significant challenge raised was linked to the intelligence function

This section analyses the experience of the different stakeholders engaged in the pilot and their perspective on the new proposed model. The evaluation team was not able to interview any FBOs during the evaluation period, as such, this section focuses on the experience of LAs and FSA. The section addresses the reasons for LAs to join the project, considers LAs’ attitudes towards the pilot and finally summarises FSA experience of the pilot. 

2.4.1 Reasons for LAs to join the pilot programme

LAs decided to join the pilot programme for a number of reasons. All LAs noted that the previous model was not working well for them and that there was a need to update it, with many having already departed from the approach to some extent. Several LAs had been part of the existing food standards working group and were engaged in the topic already, contributing to the development of the proposed model. 

All LAs stated that the decisive factors for joining were their willingness to influence the model to ensure it fits their needs, and the capacity to anticipate and adapt to the changes as soon as possible. LAs appreciated having the opportunity to start early and co-create the new model with FSA. This shows that all LAs participating in the pilot either testing the proposed model or in the control group, had already identified flaws with the prior model, were willing to try the proposed one, and to collaborate with FSA to ensure a smooth transition. They were all self-selected and knowledgeable. 

It may be that the wider population of LAs – those who did not offer to participate in the pilot – are materially different from those who did in terms of their willingness to embrace change or their ability to do so (e.g., in terms of their resources or knowledge base). This could mean that during the planned national roll-out challenges not encountered in the context of the pilot may be experienced. Lessons learned (discussed in Section 3) highlight how showcasing the pilot LAs experience with the proposed model could be a good way of engaging other LAs. 

2.4.2 LA experience - implementing the pilot and working with the proposed model

Section 2.3.4 discusses the barriers and enablers to the success of the pilot project. When discussing LAs’ experience, they highlighted again the communications both with FSA and with the other LAs, where they were able to share progress, challenges experienced and ways forward. 

FSA provided inspection programmes based on the new model (conversion spreadsheets and reporting templates) to support implementation of the new model within the pilot, and these were well received. LAs mentioned that increased pilot paperwork constrained their resources, however, officers were aware that it was specifically due to the pilot and would not be replicated as part of the proposed model itself. 

“In officers’ minds they are looking at it and going ‘this is really good actually because the work that we are doing is being recognised and also what we think is important is being deemed by the risk scheme as being important” – County Council

“For the officers it was just getting used to the new system and once they got used to it, they carried on and I don’t think there’ll be problems because compliance and risk should always be looked at together and that is a positive of the pilot” – Unitary Authority

“The model formalised what we were already doing” – Unitary Authority

LAs were working in similar ways prior to the pilot to deliver food standards controls, however these practices were not harmonised across LAs. LAs perceived that the guidance provided by FSA to deliver the proposed model was what was needed to address the flaws identified with the existing approach. A control LA also reported being guided by intelligence in their work, which could suggest that if the model was rolled out nationally, some LAs could adapt to the proposed model faster than others. 

Some of the LAs participating in the pilot had been engaging with FSA to develop some aspects of the proposed model, so they were not surprised by the changes implemented. Six out of the seven LAs testing the proposed model had embedded the new practices by the end of the pilot and were committed to continue working in line with the proposed approach. The area that was more challenging was related to using intelligence to drive the risk assessment process. 

2.4.3 FSA experience - implementing the pilot and working with the proposed model

FSA reflections on the implementation of the pilot were very positive. The staff involved in the implementation and working regularly with the LAs shared that the regular communication with LAs had been the key to the success of the project. As discussed, FSA also perceived open communication with LAs as a key enabler for the implementation of the pilot. 

The main challenge raised was the effect that COVID-19 had on LAs, and how the first phases of the project were slower than anticipated. The second challenge was linked to the intelligence function. The intelligence team was set up to deliver the new function, which meant they worked in parallel with the pilot. While progress was made, they are aware that there is still work to be done before intelligence is embedded into the new proposed model. 

2.5 What has been the effect on resources for each of the stakeholders because of the proposed model?

LAs discussed the changes in resources in terms of:

• staff requirements to deliver the pilot, adapt to the proposed model, and continue the implementation of the proposed model. Overall, they perceive the number of staff required to be the same. 
• regarding technical skills and tools required to implement the proposed model, some LAs perceived a system adapted to the proposed model would make the work more effective.

In terms of effectiveness and efficiency of LA resources, it was identified that the proposed model allowed LAs to target resources more effectively. 

The answer to this question focuses on the changes made by the stakeholders to adapt the proposed model. This includes changes linked to the implementation of the pilot, and changes to resources linked to the delivery of the proposed model. It then assesses the efficiency of the resources used during the implementation of the pilot, using the quantitative data gathered by FSA analytics team.  

2.5.1 How LAs adapted to the proposed model 

In terms of resources, LAs mentioned two main types of change. They required additional staff resource to deliver the pilot (supported by FSA), and to adapt to the proposed model. However, once adapted to, they perceived the number of staff required was the same compared to the current framework. LAs also adapted some of their skills and tools to implement the new proposed model in a more effective manner.

Staff redeployments due to COVID-19 impacted on the capacity of LAs to adapt to and implement the proposed model in a timely manner. FSA mitigated this by extending the pilot by three months so that additional data could be gathered. FSA explained to LAs the requirements and anticipated burdens before starting the pilot. Some LAs mentioned that a lot of staff time was required for the monthly data reporting requirements, ongoing meetings, and new proposed sampling approaches, however, these aspects were only linked the pilot of the model. 

Finally, in terms of the resources required for the implementation after the pilot project, LAs highlighted that the proposed model allowed them to target resources at areas that pose the greatest risk, making them more efficient.

One of the LAs from the control group stated that they would welcome the introduction of a risk model that prioritises risk and targets inspections in a more efficient way. LAs also perceived that TRIs, where feasible, would use resources more efficiently. 

When discussing the resources required, LAs explained that they valued better targeting of their resources thanks to the proposed model. However, they perceived that the same number of staff would be required to deliver the new model. The main change would be that the proposed model would enable LAs to be more effective at prioritising food standards work and identifying and resolving non-compliances with food law. 

The second area discussed by LAs was related to the skills and tools required to follow the proposed model.

Typically, LAs had different experiences of resource needs throughout the pilot, due to their organisational differences and their perceptions of the usefulness of TRIs. These differences are important to consider for national roll out and TRIs could utilise resources more effectively, however mixed perspectives were identified throughout all three rounds of interviews.

LAs perceived that they required improved mechanisms to gather and share intelligence data. Some LAs already had an integrated intelligence function as part of their work planning and used multiple channels of communication whereas others were aware of intelligence but were not guided by it. LAs that were driven by intelligence mentioned that they had adapted their MIS, so they are fit-for purpose and up to date. They also mentioned they had trained their officers so they would be familiar with the systems and the intelligence function. FSA did not require LAs to do this, however, those that adapted their systems found it easier to work with the new proposed model. 

2.6 What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?

 The pilot had a positive effect on: 

• improved identification of food businesses that present the greatest risk. The LAs using the proposed model identified a higher number food businesses that were not broadly compliant with food law (16%) when compared to the control group (4%) during the pilot period.
• targeting interventions better. The proportion of proactive (programmed) interventions that led to a follow up corrective action to prevent or address an issue was higher for the pilot group (57%) than the control group (37%). 
• increase in the proportion of LA resources driven by intelligence. The number of reactive interventions (intelligence led) is much higher in the pilot group, accounting for 20% of the total number of interventions, compared to 1% in the control group. 

The evaluation team did not identify any unintended consequences.

This section summarises the effects of the proposed model, namely whether it achieved the outputs and outcomes as set out by the intervention logic (see Annex 2): 

• whether the proposed model is effective at targeting higher risk FBOs. 
• whether LAs changed the way they were working. 
• whether there has been an increase in the proportion of LA resources driven by intelligence rather than establishment-based risk assessment. 

The evaluation team also assessed whether there had been any unintended consequences. The potential for improvement is discussed later, in section 3.) 

This section draws on data supplied to the FSA by LAs engaged in the pilot. As mentioned in section 1.4.3, the quantitative data for one of the control LAs distorted the overall analysis results and has been excluded from this chapter. An analysis of the full data set is included in Annex 3, which also includes a note on data quality. 

2.6.1 Identifying FBOs that present the greatest risk

The new risk model was identified as offering a more accurate approach to identifying risk, as it considers the inherent risk of a business and their previous compliance record. This was identified as a positive element because businesses are not rigidly limited to always being in the same risk category.

LAs highlighted that the new risk model also allows for officers to be more reactive to emerging risks and current trends. For example, LA officers identified that during the COVID-19 lockdowns, there was an increase in new businesses opening from domestic settings (e.g., people baking at home and selling their produce). Under the current FLCoP model these businesses would not fit into a category that would be deemed as high risk, however, LA officers considered them high risk due to levels of awareness of food risk and applicable legislative requirements, in addition to the risks introduced when producing food in a domestic setting. This makes the proposed model more effective at identifying higher risk establishments.

2.6.1.1 New types of interventions: Targeted Remote Intervention (TRI) and No Actionable Risk (NAR)

The proposed decision matrix introduced two new types of interventions, TRIs and NARs. 

LAs reported having different experiences with TRIs. Some LAs, including a control LA, had carried out some type of remote interventions outside of the pilot due to COVID-19 restrictions. Some LAs highlighted that TRIs were useful because they can provide an intervention for FBOs who would not have previously had any contact with the LA. One LA reported that TRIs did not work for their geographical area partly because there were a lot of small businesses who were not equipped for remote inspections. Typically, TRIs were viewed as being appropriate for some types of businesses, where the information exchange would be easier to facilitate than for other businesses. For example, manufacturers or microbreweries would often be able to send the required information to an officer faster than other types of businesses, allowing LAs to complete the TRI in a more efficient manner. Smaller business did not always have the information to hand, leading to long waiting times for officers to receive the required information, delaying the outcome of the intervention, resulting in a less efficient TRI. 

There were mixed responses regarding the effectiveness of the NAR category. The NAR category was added as part of the proposed food standards risk model, and applied to compliant low risk premises who do not require a regular intervention but would instead be subject to ongoing monitoring by the LA through other means. One LA highlighted that some premises that were identified as NAR perhaps presented a higher risk, because the business activities had changed. An example was provided for an FBO that was previously NAR but had started to sell sandwiches with labels on. The pre-packed for direct sale (PPDS) legislation change had meant that this FBO was no longer suitable for the NAR category. Therefore, it was suggested that the NAR category may require a verification of FBO activity to ensure they are consistent with the NAR classification.  On the other hand, other LAs suggested that the NAR category did not have a significant impact on them because they did not have many FBOs in the category.

2.6.1.2 Overall effectiveness

To assess the effectiveness of the proposed model, the FSA analysed the types of intervention comparing those using (pilot) and not using (control) the proposed model. They also assessed the number of non-compliances identified, whether a business was broadly compliant with food law, the number of follow-up interventions scheduled, as well as the proportion of proactive (programmed) and reactive (intelligence led)  interventions by type of intervention completed. 

Following an intervention, the food business is risk assessed. Using the outcome of the risk assessment, a food business can be classed as broadly or not broadly compliant with relevant food law. A typical food business that is considered to be broadly compliant will have no identified non-compliances or only minor ones that pose a minimal risk to consumers (e.g. equivalent to a Food Hygiene Rating Scheme rating of 3 or above). A business that is considered to be not broadly compliant will have serious non-compliances that need to be rectified by the FBO – LAs will focus on these businesses to bring them back into compliance. 

Figure 3 shows that the scheduled interventions for LAs testing the proposed model led to the identification of a higher number of ‘not broadly compliant’ food businesses when compared to the control group (16% compared to 4% respectively). Moreover, the difference between the control and pilot groups is statistically significant. This demonstrates that the new risk scheme is more effective at identifying non-compliant businesses and directing LA resources to the higher-risk establishments to resolve non-compliance. 

Figure 3 Proportion of proactive (programmed) interventions that found not broadly compliant food businesses, control vs. pilot
 

Similarly, the proportion of proactive (programmed) interventions that led to a follow up corrective action by the LA to prevent or address an issue was higher for the pilot group than the control group, at 57% and 37% respectively, as shown in Figure 4.

Figure 4 Proportion of proactive (programmed) interventions that resulted in follow up corrective action by LAs

A chart bar graph with control and pilot in the axis. It shows that control LAs, with a base size of 477 interventions, had a total of 37% of proactive (programmed) interventions that resulted in follow up corrective action by LAs. It also shows that pilot LAs, with a base size of 1,399 interventions, had a total of 57% of proactive (programmed) interventions that resulted in follow up corrective action by LAs.

Figure 5 shows the proportion of interventions by type within the pilot vs control groups. As can be seen, 14% of the pilot group interventions were TRIs instead of on-site interventions. 

Figure 5 Proportion of intervention types, control vs pilot.^(footnote) 

Group	On-site intervention	TRI	New food business	Remote intervention due to COVID-19	Total
Control	90%	-	104%	-	100%
Pilot	73%	14%	-	9%	100%

2.6.2 Better targeting of interventions 

The proposed model tested in the pilot project led LAs to identify and resolve more non-compliances thanks to the better profiling of FBO risk because of the changes to the risk assessment scheme. This enabled LAs to more accurately identify those businesses deemed to be the highest risk. 

The proposed risk scheme allowed officers to assess the risk of food businesses more accurately and determine their intervention frequency based on the officers’ professional assessment of risk. This would not have been possible when operating under the previous model. Under the proposed risk scheme officers can use their knowledge of an FBOs’ previous performance and management to assess the risk posed by these businesses and determine the appropriate intervention frequency.

2.6.3 Proportion of LA resources driven by intelligence

Another aspect of the model was that it led some LAs to use intelligence to drive their work more. There was a suggestion by one LA that intelligence received under the pilot was viewed as an improvement compared to before the pilot. This was also identified by another LA who highlighted that officers’ perception of intelligence had become broader, and they were recognising when problems may not be local and sharing more information with FSA.

“Our intel both ways, to be fair, from FSA was always massively lacking and under the pilot it’s been much better and I think making the team think about intelligence has definitely come out from the pilot rather than [officers] saying ‘we’re going to do our 400 premises’ and [now they’re] thinking about where do we want to go and what intelligence have we got coming through” – Unitary Authority

This indicative finding could be viewed as positively suggesting that the pilot is contributing to improvements in the use of intelligence and facilitating collaborative working.

The data gathered by FSA analytics team showed that by embedding intelligence into the delivery of food standards official controls, the regulatory model became more dynamic, enabling the system to identify non-compliances more effectively and earlier, and to disseminate intelligence across stakeholders more often. However, the intelligence function requires further development (as discussed in section 2.3.3). 

The proposal to embed intelligence into the delivery of food standards official controls will result in a regulatory model that is more dynamic by nature, enabling FSA and LAs to identify non-compliances more effectively and earlier, and to disseminate intelligence across stakeholders more often.

2.6.3.1 The proportion of reactive (intelligence led) interventions

FSA measured the proportion of interventions led by any form of intelligence (reactive interventions) in comparison with the proportion of interventions that are scheduled according to the indicative risk profile (proactive interventions) within the pilot group versus the control group. The sample size for the pilot group was 1,305 interventions, where for the control group was 473. 

As can be observed from Figure 6, the number of reactive interventions is much higher in the pilot group, accounting for 20% of the total number of interventions, compared to 1% in the control group. This means that the proposed FSDM is more dynamic, allowing LAs to use intelligence more frequently and carry out specific directed corrective actions to prevent or address food standards non-compliances in a more timely and more targeted way. 

Figure 6 Proportion of proactive (scheduled) vs reactive (intelligence led) interventions in control LAs vs pilot LAs 

A chart bar graph with control/pilot, and proactive / reactive in the axis. It shows that control LAs, with a base size of 473 interventions, had a total of 99% of proactive interventions vs .1% of reactive interventions. It also shows that pilot LAs, with a base size of 1,305 interventions, had a total of 80% of proactive interventions and 20% of reactive interventions

As can be observed from Figure 7, each LA within the pilot group was disseminating intelligence to other stakeholders, compared to a single intelligence report from an LA within the control group. Overall, this demonstrates that the process of intelligence sharing has increased the level of understanding of potential risks, driving up the quality of intelligence reports shared across all stakeholders. This also suggests that the process of communication has improved within the pilot, LAs being able to coordinate and collaborate better with their neighbouring LAs, FSA, Primary Authorities, etc. This statement is also verified and supported by the qualitative assessment.

Figure 7 Intelligence reports LAs disseminated to other stakeholders, control vs pilot

Other	Other LA	Environmental health	FSA	PA	Industry	NFCU	Total
Control	0	0	1	0	0	0	1
Pilot	44	20	16	12	5	1	98

2.6.3.2 Directed sampling 

The directed sampling programme has enhanced the national intelligence picture on food standards and facilitated national action to address areas of concern. Overall, the directed sampling programme supported FSA’s priorities to ensure food is safe and is what it says it is. The directed sampling programme was evidence-led, based on intelligence, and helped to develop a coordinated approach to sampling. During interviews, LAs supported the way that intelligence influenced the directed sampling programme to target emerging risk areas. 
Directed sampling during the pilot identified that half of the 317 products sampled were unsatisfactory to some degree. The highest rate of unsatisfactory results (i.e. not compliant with all applicable legal requirements considered as part of the analysis) was identified in phase II of the directed sampling programme, which focused on products where existing non-compliance data (IDB and FSA Incidents and Resilience data) indicated products were likely to be unsatisfactory.

2.6.3.3 Other sampling during the pilot

Figure 8 below shows that the pilot group took significantly more proactive samples (as a proportion of total sampling) than the control group because of focusing more on the use of intelligence. The numbers do not include the directed sampling figures. It can be noted that the proportion of reactive samples that were unsatisfactory within the pilot was larger than the proportion of unsatisfactory (non-compliant) proactive samples, which demonstrates a more efficient use of intelligence. Also, as shown in the base sizes, control LAs completed a lower number of samples than the pilot LAs (control reactive completed 3 interventions, control proactive 16; pilot reactive completed 36 interventions, and pilot proactive 769). Based on the small size, the control LAs found no unsatisfactory samples.  

Figure 8 Proportion of proactive and reactive sample outcomes, pilot vs control (excluding FSA directed sampling) 

A chart bar graph with control reactive, control proactive, pilot reactive and pilot proactive; and satisfactory / unsatisfactory. It shows that control reactive LAs, with a base size of 3 interventions, had a total of 100% satisfactory results, control proactive LAs, with a base size of 16 interventions, had a total of 100% satisfactory results. Pilot reactive LAs, with a base size of 36 interventions, had a total of 50% satisfactory results and 50% unsatisfactory, and pilot proactive LAs, with a base size of 769 interventions, had a total of 64% satisfactory results, and 36% unsatisfactory

2.6.4 Unintended consequences

The evaluation team did not identify any unintended consequences because of piloting the proposed model, except the challenges discussed in section 2.3.4.2. Overall, participating in the pilot was perceived favourably by LAs.

3.1 Implementation of the pilot

The implementation of the pilot (see section 2.3.4) highlighted that: 

The support provided by FSA throughout the pilot was well received and crucial to the success of the pilot. This level of support was specific to the pilot and may not be provided to the same extent during any national roll-out. However, some of the support provided can indicate areas to consider in the event of roll-out happening (see section 3.2). Support was important to: 

• ensure consistent understanding and interpretation of the terms used to define the proposed risk scheme and decision matrix
• quickly resolve any queries or react to any feedback LAs had during the testing of the proposed model
• build confidence across LAs and help embed the pilot and model into the LA’s work practices
• create a LA peer to peer network where they were able to share examples of best practice amongst themselves
• while the intelligence element is still under development, LAs mentioned that meetings with FSA intelligence team helped clarify a common understanding of what intelligence is and what type of intelligence should be shared. Through the pilot, LAs learned how to gather and share intelligence, and after the pilot LA officers were more proactive in gathering and sharing intelligence. 

The pilot also highlighted some challenges that may be present with other LAs as well. These were: 

• LA MIS will need updating to operate the new model
• LAs databases may be out of date
• unequal experience across LAs of using intelligence, and different understandings of what intelligence is
• practical challenges linked to the dynamic, intelligence-led sampling programme. As the programme had not been scheduled earlier in the year, it was difficult to find the resources to carry it out to completion. 

3.2 Lessons about the proposed model

The evaluation has also illustrated some lessons about how the proposed model operates (see section 2.3 and 2.6): 

• LAs shared learning points about the model within their teams. They also shared them with LAs as part of the regional groups they participate in. To provide further reassurance, some LAs reported conducting their own in-house consistency exercises with officers and developing their own guidelines. 
• LAs differ on their opinion regarding the effectiveness of TRIs. Further work has been carried out on the use of remote assessments by LAs, which can help draft guidance for LAs. 
• while progress has been made, the intelligence component of the proposed model continues to develop in respect of building or otherwise consolidating intelligence capacity and capability both in the FSA and within LAs, and developing appropriate and lawful sharing mechanisms between LAs and FSA. 
• internally, FSA can do more work regarding how they plan to analyse the data and what feedback they will provide LAs with.

The study team would recommend the following during the national roll-out: 

• ensure that LA MIS are compatible with the proposed model
• support LAs to update their food businesses databases so they are ready to switch to the new model. Some LAs noted that this could take a significant amount of resource, as it was difficult to know whether premises that had registered had opened or closed. Trying to map data using hygiene scores was viewed as useful, but not a perfect solution
• facilitate a shared understanding of the terminology used in the proposed risk scheme and decision matrix, together with practical guidance illustrated with examples from the LAs in the pilot. LAs mentioned that some of the descriptions initially used for the proposed risk scheme were not very clear. LAs suggested they could benefit from re-wording or some examples. An improvement in the definitions and descriptors would help with consistency in interpretation between LAs or officers during a national roll-out
• explain how allergens are assessed and integrated in the proposed model more clearly, by improving the descriptors in the risk rating
• FSA could consider monitoring LA performance in relation to interventions, against the risk profile of food businesses in their area. One LA noted that there was a need to understand the context within which each LA operates to accurately assess their performance, against their local food business profile. For example, one LA could complete 100% of their high-risk interventions, whereas another may only complete 20%, but they have tackled and improved the levels of non-compliance
• ensure the risk scheme recognises Geographical Indications/locally made food as an enforcement priority for some LAs. For example, some areas have premium products attached to their location, which could increase the likelihood of food fraud related to specific products in their area
• consider that some areas have fewer high-risk premises, which can affect the risk profile of FBOs and affect the types of interventions that are performed. A LA that had many small businesses in their area reported having less confidence in the efficacy of TRIs compared to LAs that have a wider mix of premises
• engage LAs that participated in the pilot to share their experiences and lessons learned with other LAs. They all mentioned they would be happy to share their experiences with other LAs. Discussion groups, regional groups and the facilitation of an online discussion thread could be considered as useful channels to help LAs share learning and to embed the proposed model nationally. 

The pilot project to test the proposed delivery model has been an important step towards the national roll-out of the proposed FSDM. It has also contributed to the ABC Programme workstream ‘targeted and proportionate regulation’, that aims to design a more targeted and proportionate approach to LA regulation of all food businesses within their remit. 

The testing of the proposed FSDM, which incorporates a revised risk assessment scheme and more emphasis on intelligence as a driver for LA regulatory activity, has been a success. Despite some challenges, the pilot has shown that the proposed model works better that the existing one. It is more effective at targeting high risk businesses, at identifying non-compliances and resolving them, and at giving LAs flexibility on how to target their resources. The intelligence function is still being developed, and, while not fully operational yet, the concept was very well received by LAs, and there are already early signs that indicate increased intelligence sharing is happening. 

Annex 1: Glossary 

Food business - As defined by Article 3(2) of Regulation (EC) No 178/2002 it means any undertaking, whether for profit or not and whether public or private, carrying out any of the activities related to any stage of production, processing, and distribution of food.

Food hygiene – The measures and conditions necessary to control hazards and to ensure fitness for human consumption of a foodstuff, considering its intended use as set out in Regulation (EC) No 852/2004.

Inspection - The examination of any aspect of feed, food, animal health and animal welfare to verify that such aspect(s) comply with the legal requirements of feed and food law and animal health and welfare rules.

Intelligence - Information that has been subject to a defined evaluation and risk assessment process to assist with regulatory decision making.

Intelligence led - Intelligence-led working is defined by National Trading Standards (NTS) as ‘a business process for systematically collecting, organising, analysing, and utilising intelligence and information to guide operational and tactical decisions. Intelligence led aids in identifying, assessing, and managing targets, threats and problems at the local, regional and national level.’

Intervention - Regulatory actions taken to affect or interfere with decisions made by individuals, groups, or organizations regarding social and economic matters. Interventions include official food controls and other interventions such as education, advice and coaching, information and intelligence gathering (including sampling where the analysis is not to be carried out by an Official Control Laboratory). 

Proactive intervention – A proactive intervention refers to any intervention conducted as part of the LAs planned intervention programme. 

Public Analyst – Scientists that ensure the safety and correct description of food by testing for compliance with legislation as specified in Section 27 of the Food Safety Act 1990 and Regulation 4 of The Food Safety (Sampling and Qualifications) (England) Regulations 2013.

Reactive intervention – A reactive intervention refers to any LA intervention completed as a response to a complaint or to any other intelligence received, such as a sampling result or referrals. It is not planned. 

Risk - The chance or probability that a person will be harmed or experience an adverse health effect if exposed to a hazard.

Risk analysis - A process consisting of three interconnected components: risk assessment, risk management and risk communication. 

Sampling - Taking feed or food or any other substance (including from the environment) relevant to the production, processing and distribution of feed or food or to the health of animals, to verify through analysis or examination compliance with feed or food law or animal health rules.

Standards – Rules or principles defined in food safety law that are used as the basis for judgement against.

The National Food Crime Unit (NFCU) - The NFCU provides a nationwide focus on enforcement against serious fraud and related criminality in food and feed supply chains. 

Annex 2: Food Standards Intervention Logic

This Annex introduces the intervention logic for the food standards project and the work completed so far under the pilot. Figure 9 below sets out the intervention logic model as described in the inception report. 

Figure 9 Intervention logic model for the Food Standards Project (accessible version)

Objective: to pilot a new operating model for food standards which seeks to establish consistent risk rating approach that is dynamic and integrates intelligence so that LA's can provide a more targeted and efficient response. 

Problems

• risk rating approaches not consistent and not fit for purpose (LAs don't have enough resources; and approaches do not reflect all FBO types)
• no consistent use of intelligence and data sharing by LAs.
• sampling use could be more efficient and its use is declining.

A. Inputs (and activities)

• A.1 Prepare LAs to pilot the new model through tailored training and discussions
• A.2 LAs use a uniform risk rating scheme to produce an indicative risk profile
• A.3 Intelligence systematically gathered and shared between FSA's Food Standards Intelligence function and LAs including through national sampling programme
• A.4 FSA and LA resources and skills to implement the pilot
• A.5 Evaluation plans implemented.

B. Outputs

• B.1 Number and types of LA visits/training received
• B.2 Number of intervention type and frequency based on FBOs risk rating
• B.3 Number and quality of intelligence exchanged between FSA and LAs including sampling activity data
• B.4 Evaluation reports presented (FSA and ICF)

C. Short-term outcomes (6 months)

• C.1 LAs understand how to operate under the new approach and engage with the pilot
• C.2 Consistent and proportionate approach to regulation food standards across pilot LAs
• C.3 Increase in the proportion of non-compliances identified and resolved
• C.4 Increase in the proportion of LA resources directed by intelligence rather than being drive by an establishment-based risks assessment scheme. 
• C.5 FSA develops knowledge about the new model and adapts / modifies integrating these lessons

D. Medium-term outcomes (1 year)

• D.2 Intelligence driven work nominated within LAs
• D.3 FSA has a better oversight of the national picture of food standards and better able to identify trends and emerging issues relating to food standards compliance
• D.4 Lessons learnt are used to take informed decisions about new operating model for food standards

Assumptions:

1. There is buy-in from LAs to engage in this new way of working
2. The new approach works across LAs and FBO types with LAs having access to the resources (staff time, infrastructure, knowledge and data) to correctly develop the indicative risk profiles
3. FSA and the LAs will have the resources, permissions, infrastructure and skills in place to gather, share, process, and interpret intelligence data and the required communication between each other
4. Training and engagement activities are of sufficient quality for LAs to ahve the skills and knowledge to adopt this new way of working
5. The new approach is capable of generating accurate scores that leads to proporiationate minimum intervention frequency and type requirements

Figure 9 Intervention logic model for the Food Standards Project

Annex 3: Food Standards Pilot - quantitative evaluation

1. Introduction

FSA’s Analytics Unit designed excel spreadsheets and/or online forms to collect the necessary data from all participating LAs (control and pilot groups) in a standardised format to support the evaluation of the pilot. Data was collected monthly. 

Below is a short summary of the forms’ content:

• the risk scheme scores from every inspection. 
• a detailed questionnaire for every: reactive (intelligence led) intervention LAs undertook during a given month and proactive (programmed) intervention that LAs undertook where the premises was found to be non-compliant. 
• through the questionnaire we collected data on the number of reactive versus proactive interventions, amount and effectiveness of intelligence and risk assessment model, samples outcome, time resources involved, etc.
• for businesses found to be broadly compliant, we asked for the detailed questionnaire to be completed for a small sample of 10 proactive interventions and TRIs.

The quantitative evaluation is based on data received from all participating LAs for the period January 2021 to March 2022. Due to lack of inspections from January 2021 to April 2021, we extended the pilot period to 15 months instead of the initial 12 months.

With the proposed FSDM, we expected to find more non-compliances due to inspection plans being focused on higher risk businesses. We also expected to see more intelligence led interventions because of increased sharing of intelligence reports. In addition, we expected that individual interventions may take longer as they were more likely to find non-compliance and require follow up corrective action.

In this annex we have provide data referred to in the main body of the report plus a second which allows us to provide some sensitivity analysis. This second set has two main differences. Firstly 9% of interventions in the pilot group and 11% of interventions in the control group were remote interventions due to COVID-19, although these were counted within the total interventions carried out during this period. They were not counted within the detailed analysis of outcomes, as these do not generally fall within the criteria of TRIs within the proposed model. These are included in the sensitivity analysis

Secondly, as mentioned in section 1.4.3, results for one of the control LAs skewed the overall analysis results. The LA in question dealt with a higher than usual number of food incidents during the pilot. Multiple samples were taken in relation to the incidents which is normal practice in this scenario. As would be expected, the analysis for many of these samples were unsatisfactory due to the targeted nature of this work and the time taken on interventions due to these incidents was significant. 

2. Performance quantitative measures

We measured the following outcomes to analyse the effectiveness of the pilot. To do so, we compared the results from the existing FLCoP model (control group) with the results from the proposed operating model (pilot group). 

2.1 The effectiveness of intelligence

By embedding intelligence into the delivery of food standards official controls we will have a regulatory model that is more dynamic by nature, enabling us to identify non-compliances more effectively and earlier, and to disseminate intelligence across stakeholders more often.

The proportion of reactive (intelligence led) interventions
On the figure below, we measured the proportion of interventions which were led by any form of intelligence (reactive interventions) in comparison with the proportion of interventions that were scheduled according to the outputs from the risk schemes (proactive interventions) within pilot vs control. 

As can be observed from Figure 10, the number of reactive interventions is higher in the pilot group, accounting for 20% of the total number of interventions, compared to 15% in the control group (1% if we exclude the anomalous LA). This means that the proposed FSDM is more dynamic, allowing LAs to use intelligence more frequently and to carry out specific directed actions to prevent or address food standards non-compliances in a more timely and more targeted manner.

Figure 10 Proportion of proactive (scheduled) vs reactive (intelligence led) interventions in pilot LAs vs control LAs

Figure 11 shows the proportion of intelligence-led inspections identifying non-compliance is higher in the control group at 22% compared to 15% in the pilot. However, it should be noted that it is likely this figure may not be typical as it includes the control LA that dealt with a high number of food incidents that skewed the overall figures. 

Figure 11 Proportion of reactive (intelligence led) interventions finding non-compliance

Figure 12 shows the dissemination of intelligence reports. All but one of the reports in the control group came from the anomalous LA.

Figure 12 Intelligence reports LAs disseminated to other stakeholders, pilot vs control 

LAs disseminated to other stakeholders, pilot vs control ">

Group	Other LA	Environmental Health	FSA	PA	Industry	NFCU	Total
Control (as in main report)	0	0	1	0	0	0	1
Control (including anomalous LA)	12	10	14	9	0	0	45
Pilot	44	20	16	12	5	1	98

Figure 13 shows that 86% of the total number of intelligence reports within the pilot group led to follow-up corrective action, compared to 82% on the control group (although all of these actions were within the single LA discussed above that dealt with a high number of incidents during the pilot). 

Figure 13 Proportion of reactive (intelligence led) interventions leading to a follow-up corrective action by LA

Overall, these results demonstrate that the process of intelligence sharing has increased the level of understanding of potential risks, driving up the quality of intelligence reports shared across all stakeholders. This also suggests that the process of communication has improved within the pilot, with LAs being able to co-ordinate and collaborate better with their neighbouring LAs, FSA, Primary Authorities, etc. This statement is also verified and supported by qualitative assessment.

2.2 The effectiveness of the proposed risk scheme

We expect that the proposed risk scheme and decision matrix will enable LAs to target their resources on those businesses deemed to be highest risk e.g. high levels of non-compliance with food standards law. 

As can be observed from Figure 14, the proactive interventions led to the identification of a higher number of non-compliant food businesses within the pilot group at 16%, compared to 4% within the control group. This demonstrates that the new risk scheme is more effective at identifying non-compliances and directing LA resources to the higher-risk establishments to resolve issues. 

Figure 14 Proportion of proactive (programmed) interventions finding not broadly compliant businesses.

Similarly, the proportion of proactive (programmed) interventions that led to a follow up corrective action by LAs to address issues of non-compliance was higher for the pilot group than the control group, at 57% and 45% respectively (37% excluding the anomalous LA) (see Figure 15).

Figure 15 Proportion of proactive (programmed) interventions resulting in follow up corrective actions

The table below (Table 4.1), shows the breakdown of intervention types, pilot vs control. 

Table 4.1 Interventions by type. Base size of 576 (469) for control, and 1,258 for the pilot

Group	On-site intervention	TRI	Business Registration	Remote intervention due to COVID-19
Control (as in main report)	90%	-	10%	-
Control (including anomalous LA)	81%	-	8%	11%
Pilot	72%	14%	4%	9%

2.3 The effectiveness of sampling

We can observe from Figure 16 that the pilot group took significantly more proactive samples (as a proportion of total sampling) than the control group because of focusing more on the assessment of intelligence. It can be seen in Figure 17 that the proportion of reactive samples that are unsatisfactory within the pilot group is larger than the proportion of unsatisfactory proactive samples (when the anomalous control LA is removed from the analysis), which demonstrates a more effective use of intelligence in targeting sampling activity to areas where there are likely to be issues.

Figure 16 Proportion of proactive vs reactive samples, pilot vs control (excluding FSA directed sampling)

Figure 17 Proportion of proactive and reactive sample outcomes, pilot vs control (excluding FSA directed sampling)

In addition to regular locally coordinated sampling work, FSA directed sampling accounts for 22% of the total number of samples taken within the pilot group and returned 50% unsatisfactory results. It is expected that the new FSA intelligence function will contribute to the identification of more unknown or emerging risks, increasing the likelihood of unsatisfactory sampling results as we become better at targeting sampling activity.