Evaluation of the PATH-SAFE programme

The report is intended for HM Treasury as the programme sponsor and delivery partners of the programme, primarily Food Standards Agency (FSA), Food Standards Scotland (FSS), Department for Environment and Rural Affairs (DEFRA), UK Health Security Agency (UKHSA), Department for Health and Social Care (DHSC), and the Environment Agency (EA). The findings of the evaluation will help the delivery partners manage PATH-SAFE adaptively and assess the impact created through the programme on foodborne pathogen and antimicrobial resistance surveillance. The report is structured as follows: 

• The remainder of this introduction describes the context for the PATH-SAFE programme, the goals and structure of the programme, the aims of the evaluation and its limitations, and our approach to developing this evaluation framework report.
• Chapter 2 presents the analytical framing for the evaluation comprising the PATH-SAFE theory of change (ToC).
• Chapter 3 describes our overarching evaluation approach through which we will collect evidence to assess the PATH-SAFE programme. The evaluation approach will comprise three types of evaluation: a process evaluation; an outcome evaluation; and an impact feasibility assessment.
• Chapter 4 presents the process and outcome evaluation frameworks through which the evaluation of the PATH-SAFE programme will be operationalised. Presented in tabular form, each framework comprises key evaluation questions (EQs), derived from the PATH-SAFE ToC, which our evaluation will seek to answer, alongside indicators and proposed data sources. The chapter builds on the frameworks and provides further detail on the methodology that will be undertaken to conduct the evaluation.
• Chapter 5 shows the evaluation timelines and key deliverables. 
• Chapter 6 outlines the risks to the evaluation and our mitigations in place. 
• The Annexes to the report contain additional information on the four workstreams (WSs) and sub streams of the PATH-SAFE programmes (Annex A), which is useful context for the evaluation frameworks, and more detailed information on the activities that led to the development of the evaluation framework report (Annex B) and the process and outcomes evaluation frameworks (Annex C). 

1.1.    Background

Foodborne diseases pose a major public health risk to the UK population, and creates a significant burden on our health services and economy. The majority of human disease is caused by a handful of pathogens that, in most cases, enter the food chain from farmed animals or the environment. In addition, these foodborne pathogens can also develop antimicrobial resistance (AMR) due to the overuse of antimicrobials in food production systems. Since they are transmissible to humans via the food chain, causing illness and disease, antimicrobial-resistant foodborne pathogens are a global health issue affecting countries of all economic levels. The combined threat of foodborne and AMR pathogens creates a crucial risk for the food chain as well as the environment and need to be investigated and monitored holistically.

For AMR specifically, as its disease and economic burden rises globally, there is significant demand for new and emerging diagnostic and identification technologies to reduce its spread together with a coordinated and collaborative regulatory approach. According to the World Health Organisation (WHO), antimicrobial resistant infections are estimated to cause around 700,000 deaths each year, a figure which is expected to rise to 10 million deaths by 2050. They also pose a significant economic threat that could cost a cumulative 100 trillion dollars of economic output within the same timeframe. In January 2019, the UK government published a 5-year action plan to tackle AMR, developed as a cross-governmental effort in collaboration with a range of stakeholders across academia, industry, and professional bodies. The action plan included measures to improve the development and access to diagnostics, and adopted a ‘One Health’ approach, setting out commitments that cut across human and animal health, as well as food production and the environment. A coordinated, multi-sectoral ‘One Health’ response to surveillance of foodborne pathogens and AMR is possible with the research and development of new and safe antimicrobials, diagnostics, vaccines, waste management tools, as well as the availability of affordable and quality versions for both animals and humans. The ambitions of the UK government-funded PATH-SAFE programme integrate cohesively into this approach.

1.2.    The PATH-SAFE Programme

PATH-SAFE is a pilot programme that aims to develop comprehensive surveillance of foodborne pathogens (FBPs) and AMR in all four nations of the UK. It proposes to create a national genomic surveillance infrastructure, through improved and novel uses of data and technology creating a blueprint for national surveillance. Through a range of parallel proof of concept pilot workstreams, it will demonstrate how this infrastructure will support a national roadmap for One Health surveillance across the UK.

The PATH-SAFE programme is funded by HM Treasury through the Shared Outcomes Fund.  This Fund was established to pilot projects to test innovative ways of working across the public sector. It has had two rounds of funding, in 2019 and 2020, with a £200 million allocation in each round. PATH-SAFE was funded in round 2 with a funding allocation of £19.2 million which will last until 2024. It is a cross-governmental collaboration across the FSA, FSS, DHSC, DEFRA, UKHSA, and the EA. 

The PATH-SAFE programme consists of four workstreams (WSs,), which are summarised below, and are being delivered through multiple partners from the government departments and their agencies, centres and directorates mentioned above as well as devolved administrations in Wales and Northern Ireland and a variety of academic institutes. Further details on the WSs and their sub streams can be found in the Annex.

WS1: Establish a curated, national foodborne disease genomic data platform
This WS’s key ambition is to work with academic colleagues and major ‘big data’ stakeholders to create a ‘user-friendly’ platform for the rapid interrogation and of genomic data.  They will build easy to use reporting capabilities to create powerful, but easily understood, interfaces that can be used by decision makers (for example, epidemiologists or other public health professionals).  A key element of the data platform development will be allowing the integration of sample data with other existing data sources to create new knowledge.  

A distinct project of the WS is WS1b which is focussed on understanding source attribution, infection threat and level of AMR of E. coli. in Scotland using whole genome sequencing, with samples isolated from a range of reservoirs across Scotland.  

WS1 is led by FSA and delivered by a consortium of government and academic partners (Project 1a) and FSS (Project 1b) and is due to finish in March 2024. 

WS2: Develop a pilot infrastructure for regular, multi-location sampling 
The WS will develop a pilot infrastructure to provide high granularity whole genome sequencing (WGS) data from regular, multi-location sampling of wastewater and food products to capture AMR and FBP data. This is being done across multiple projects and multiple settings (i.e. milk laboratory, sheep abattoirs etc.). 

WS2 is led by Defra and delivered by the Centre for Environment, Fisheries and Aquaculture Science (CEFAS) (Project 2a); Animal and Plant Health Authority (APHA)/Veterinary Medicines Directorate (VMD) (Project 2b); Public Health Agency Northern Ireland (PHA NI) (Project 2c) and FSA (Project 2d) and is due to finish in March 2024. 

WS3: Understand the feasibility of using portable diagnostics as inspection tools
The WS will investigate the technology readiness levels of existing and new portable diagnostics. The results of these studies will inform options for in-field testing and/or development. The co-design of applications with end-users (for example, policy teams/inspectorates, operational staff) will be critical to ensure real-world applicability. The WS will also undertake a pilot study investigating the feasibility of using wastewater approaches developed in response to the Covid-19 pandemic with complimentary diagnostic technology (for example, Loop-mediated isothermal amplification or LAMP) to understand Norovirus outbreaks in a contained setting. WS3 is led by FSA and delivery by Fera Science Ltd. (Project 3a) and UKHSA (Project 3b) and is due to finish in March 2024. 

WS4: Develop a pilot environmental AMR surveillance system
The overall aim in WS4 is to create an evidence-based understanding of the nature and extent of AMR in the environment and the drivers that influence this. This pilot will deliver an agreed and tested methodology for environmental AMR surveillance, as well as an environmental information technology (IT) platform that aims to enable a scaled-up surveillance programme to be undertaken. This IT platform will be designed and developed so that it will have the capability to integrate AMR surveillance data collected from animals so that the ambition of having a UK ‘One Health’ surveillance system for AMR can be realised.  WS4 is led by DEFRA (with EA and VMD) and UKHSA and is due to finish in June 2023. 
 
Figure 1. PATHSAFE WSs

1.3.    Evaluation of PATH-SAFE 

In November 2022, FSA commissioned RAND Europe to undertake the evaluation of the PATH-SAFE programme. There are three main objectives of the evaluation:

• Design and framing: to design a programme ToC for the programme, articulating the change that is intended to be achieved by PATH-SAFE. This includes outlining assumptions, external factors and managing risks associated with the programme.
• Formative process evaluation:  to utilise bespoke evaluation methods for assessing the processes underpinning the programme and delivering the outputs of the four WSs to identify areas for learning and improvement.
• Understanding impact: to identify and prioritise outcome indicators and data sources to validate the outcomes of the ToC and develop an assessment of PATH-SAFE’s contribution. 

The main evaluation questions to be addressed in the process and outcome evaluations are listed below and can also be found in Table 1 and Table 2:

• How appropriately resourced has PATH-SAFE been throughout the stages of inception, design and implementation?
• How effective and appropriate is the governance in place to support delivery of PATH-SAFE?
• How is cross-government interaction being enabled/conducted?
• How is PATH-SAFE linked to existing/developing surveillance programmes?
• To what extent have relevant end users been engaged and how has have their needs been incorporated into the design of the database?
• How has data interconnectivity and interoperability been considered in designing the platform?
• What existing and novel analysis technologies are being utilised?
• What is the extent of data collection and curation?
• How (if at all) are new capabilities being generated to improve surveillance?
• How is data being accessed/ shared across relevant stakeholders and departments?
• To what extent is the technology readiness level (TRL) assessment approach valuable for identification of relevant technology?
• How is LAMP assessment feeding into TRL mechanisms for FBP diagnostics?
• What is being learnt and incorporated from existing AMR surveillance systems and tools?
• How is connectivity between the WS4 AMR environment platform and WS1a being considered?
• How is evidence being aggregated across the multiple departments involved in WS4 delivery?
• How has PATH-SAFE (if at all) enabled a community of practice and decision makers to come together to inform and act on surveillance of FBPs and AMR?
• How and to what extent has PATH-SAFE evidence (if at all) contributed to national policies and frameworks for improved public health?
• Has data access and use for FBP and AMR been enabled and improved across government departments?
• To what extent has the platform supported use of relevant metadata and historic isolates for comparative assessments and risk profiles of FBP?
• How has the collective source detection efforts and use of novel technology translated to (if at all) improved surveillance of FBP and AMR?
• To what extent have the pilot efforts been able to exemplify practice and enhance national surveillance capability?
• What kind of strategies and operations have been enhanced, enabled and influenced enabled (if at all) through the surveillance activities?
• Have the tools identified been useful for end users? Can they be utilised?
• To what extent have gaps been identified to further development of onsite rapid FBP detection?

The PATH-SAFE evaluation will look to understand existing surveillance mechanisms that precede the programme, which will be useful context for assessing the additionality of PATH-SAFE. The evaluation project will run from November 2022 to June 2024 

1.4.    Evaluation framework report

This document presents the evaluation framework report for PATH-SAFE. The purpose of the report is to detail our evaluation approach, including evaluation questions as well as data collection and analysis methods to guide the process and outcome evaluations of PATH-SAFE. We outline the key activities undertaken by the evaluation team to inform the development of this report in Annex B.  

1.5.    Developing the Process and Outcomes evaluation frameworks

The evaluation frameworks were developed systematically by reviewing and refining the EQs initially provided by FSA (developed with the programme partners) and developing new additional ones. The EQs were mapped to all key outputs and outcomes of the ToC to ensure coherence. Against each EQ, process and outcome indicators were developed which were then mapped against relevant data sources and the data collection methodology. We list the key activities undertaken by the evaluation team to inform the development of these frameworks in Annex C.

We have chosen a theory-based approach for this evaluation. Whilst experimental evaluation approaches usually measure the effect of an intervention in comparison to a counterfactual group, hence assessing the causal relationship between an intervention and its effects, they do not uncover why the intervention worked or not and how, if it did. A theory-based evaluation addresses these questions and considers the complexity within which an intervention is being delivered. The combination of complexity of the external environment, large and disparate areas of focus for the WSs, and lack of a counterfactual makes a theory-based approach the most useful and feasible for the PATH-SAFE evaluation. Given the use of a theory-based approach, the evaluation of PATH-SAFE is underpinned by a ToC described below, which is the foundational structure used to develop the evaluation framework presented in Chapter 4. 

2.1.    PATH-SAFE ToC

A ToC, read from left to right, is a programme theory that hypothesises the intended change an intervention is likely to bring about. It assumes a causal relationship between the intervention activities and its outputs and outcomes. An integral part of conducting a robust evaluation, a ToC helps articulate how various programme inputs and activities are expected to work, as well as identify the strength of the evidence that underpins them. However, the further one moves to the right-hand side of the ToC and the longer term the outcomes and impacts become, the effect or the contribution of the intervention becomes diluted and direct causality is less attributable. To account for this dynamic programme environment and complexity, an iterative and participatory approach to refining the ToC was followed, involving key stakeholders and triangulation with desk research. This approach is also in alignment with the Magenta book guidelines on handling complexity in policy evaluation while developing the logic model for a ToC.

A ToC serves two broad purposes: 

• It clarifies for stakeholders the role they can play in accomplishing the goals of the intervention through articulating a shared understanding of the aims of the intervention in question and how these will be achieved.
• It also functions as a tool, a base framework upon which to map evaluation questions, indicators, and data sources by displaying the logic through which the performance of the intervention can be assessed (see Chapter 4. Evaluation framework).

FSA, in conjunction with programme partners, developed a ToC for the PATH-SAFE programme that was shared with RAND Europe. This ToC adopted the standard logic model approach of modularly stating inputs, activities, outputs, outcomes and impacts of the programme. After documentary review, consultation with subject-matter experts, and a validation workshop with the central team, we revised the original ToC by providing more specificity in the outputs and outcomes of the programme and a clear linkage between these and the four WSs. 

As an example, the original output listed for WS1, “Pilot FBP/AMR genomic data system using exemplar species”, was divided into two separate outputs that differentiated between the delivery of the database itself and its ability to integrate with other data systems (see ‘Outputs’ column in Figure 2.) Similarly, the original list of five outcomes were expanded and further nuanced to take into account the anticipated changes realised through the four WSs as well as changes achieved by the programme holistically building on the individual outcomes of the WSs. For instance, WS2 has been expanded to contribute to distinct outcomes. This includes the original outcome focused on understanding source attribution of FBP and AMR, with an added focus on infection threat as well as an explicit mention of international entry points in the newer version. 

Additional outcomes focused on bringing together key stakeholders and decision makers to engage with key evidence, and contribution to ‘One Health’ goals and ambitions for public health have been added to signify the holistic change anticipated at the programme-level. The revised ToC is presented in Figure 2, and will be edited following the impact feasibility assessment (see Section 3.3). The activities and outputs are intended to be carried out and delivered by March 2024 when the current phase of funding completes. The outcomes and impacts are anticipated to be realised over the medium (2-5 years) to long-term (5-10 years). 

Figure 2 PATH-SAFE programme ToC

2.2.    Factors influencing the ToC 

This section outlines the main assumptions that need to be fulfilled for the ToC to be realised and the external factors to be aware of in the evaluation. These are key factors that could impact the programme’s delivery, so the evaluation needs to refer to these in its methodological approach. The original assumptions underpinning the ToC, provided by the central programme management team, were modified based on the documentary review and desk research conducted, when revising the ToC. Furthermore, a list of external factors was developed, supported by desk research, that could affect the delivery of PATH-SAFE and hence impact the ToC and the ensuing evaluation. The list of assumptions and external factors presented is not exhaustive and, where appropriate, will be revisited at the conclusion of the evaluation. 

2.2.1.    ToC assumptions 

The ToC is underpinned by a range of assumptions about the expected behaviour of key entities across the PATH-SAFE programme, which in turn affect the realisation of the intended outputs and outcomes of the intervention. These assumptions cover the actions of end users, stakeholders, and the programme itself. We have identified the following as relevant for the PATH-SAFE programme: 

• end users know about and engage with programme outputs facilitated through a strong engagement strategy. 
• key collaborations, with stakeholders needed for programme delivery, are established and maintained at the programme and project level.
• datasets, surveillance systems, and innovations are fit for purpose and functional to track AMR and FBP.
• further funding covers running costs of legacy products and financial input continues until projects draw to a close with a plan for infrastructure maintenance.
• there is use of systems and frameworks produced from the programme across the agrifood landscape in the UK.
• programme activities are commissioned and awarded on time and as intended.
• programme funds activities that align with the aims of PATH-SAFE.

2.2.2.    ToC external factors

The implementation of the PATH-SAFE programme is taking place in an environment that includes exogenous shocks such as the UK’s exit from the European Union (EU) and Covid-19, and other relevant initiatives in the broader ecosystem of pathogen surveillance and detection. Exogenous shocks can result in significant changes to the economic and societal landscape within which PATH-SAFE operates. EU exit, for example, could impact which pathogens are selected for surveillance due to divergent policies on AMR and crop technologies between the UK and the EU. Covid-19 and the ongoing war in Ukraine, other shock variables, have already compressed resource availability and the ability to deliver projects as intended. Beyond shocks, we also include relevant initiatives that could impact PATH-SAFE delivery or affect its intended outcomes and impacts by either accelerating or hindering them (for example, the UK national action plan for AMR). These activities can help us assess where the contributions of PATH-SAFE are unique and where they form part of a larger effort across the UK and international agri-tech sector. They will also show us whether PATH-SAFE is compatible with other interventions that predate it and are in development (see Section 3.2).

The external factors identified so far are as follows: 

Shocks

• external events (for example, Covid-19, war in Ukraine) may impact resource availability across the program, impacting ability to deliver as intended.
• EU exit’s effect on UK and EU divergence on AMR policies and crop technologies could impact how, when, and on what pathogens the surveillance is conducted and also contribute to PATH-SAFE impacts. 

Relevant initiatives that may impact PATH-SAFE outputs and outcomes

• UKHSA investment into another surveillance platform, developing data linkage pipelines with NHS hospital episode statistics, could fortify or detract from PATH-SAFE impacts.  
• UK National Action Plan for AMR entailing reduction in use of antibiotics in livestock will impact AMR surveillance datasets and mapping. 
• investment of $1 billion by industry to set up the AMR Action fund to bring 4 new antibiotics to market by 2030 will potentially have an impact on AMR reduction but this is outside the timelines of PATH-SAFE. The lead candidates are BV100, BV 200, BV300, and BVL-GSK098. 
• work carried out by Centre for Genomic Pathogen surveillance, enabling genomic data for surveillance of AMR tracking in the UK and globally could be complementary to or overlapping with PATH-SAFE. 
• WHO’s Global Genomic Surveillance Strategy 2022-2032 for pathogens with pandemic and epidemic potential, which aims to facilitate connectivity between different disease control programs and surveillance networks, has the potential to improve the PATH-SAFE outputs. 
• Centre for Pandemic Preparedness developing a global early warning system to detect new infectious disease threats by bolstering surveillance and sequencing capacity could provide an opportunity for learning or to contribute to a global agenda thus realising PATH-SAFE outcomes and impacts. 
• EU Farm to Fork Strategy (2020): Its objective is the reduction by 50% of the overall EU sales of antimicrobials for farmed animals and in aquaculture by 2030. It will impact AMR surveillance and mapping and will need to have coherence against PATH-SAFE systems. 
• climate change sector policy impact on use of genetically engineered/genetically modified editing in the agrifood sector could impact how, when, and on what pathogens the surveillance is conducted and also contribute to PATH-SAFE impacts. 
• a new National Biosurveillance Network (NBN) is about to enter a discovery phase and will form part of Pillar 3 “Detect” of the new Biological Security Strategy. The aim of the discovery phase is to understand biosurveillance capabilities and then develop a biosurveillance ‘pilot’ business case, covering all biological threats. The NBN could offer legacy opportunities for PATH-SAFE if FBP and associated AMR are deemed to be within scope.

We propose to conduct three types of assessments: a process evaluation; an outcome evaluation based on contribution analysis methodology; and an impact feasibility assessment using an adapted context mechanisms and outcomes framework. The sections below provide more information on the aims of each assessment. 
 
Figure 4. Evaluation approach

3.1.    Process evaluation

The process evaluation establishes how the programme is working, whether it is progressing as intended, and identifies any lessons learned that can be applied to programmes that are still ongoing as well as their future iterations. It will use the Organisation for Economic Cooperation and Development (OECD) evaluation criteria of relevance and coherence, assessing if the intervention is doing what it should (i.e. incorporating needs of stakeholders and considering the context), and whether it is compatible with other interventions (carried out by programme partners or other actors) that predate it or are in development within the same field. 
Our process evaluation is based on the ToC, focussing on the inputs, activities, and the resulting outputs of the PATH-SAFE programme and its WSs. The process evaluation will consider the mechanisms and structures in place leading to the delivery of outputs, which are primarily governance arrangements, cross-government collaboration, delivery barriers and enablers, links with existing surveillance and monitoring approaches, and end user engagement. Given the programme is a pilot and looking to build on existing capabilities as well as generate new ones, the process indicators will be focussed on considering ‘the extent’ to which activities have created step change and resulted in the anticipated outputs rather than looking to quantify processes.

3.2.    Outcome evaluation

The outcome evaluation will be focussed on whether the programme and its WSs have realised the changes expected at a given point in time and determine how  the changes may or may not have occurred. The goal of the outcome evaluation is not to attribute outcomes exclusively to PATH-SAFE but rather to provide evidence-based explanations of whether and how the programme contributed to the outcomes of interest alongside other external factors through undertaking contribution analysis (CA) (see Section 3.4.1 Contribution analysis). Given the start of the programme in early 2022, most outcomes will likely not have emerged at the time when the evaluation is being conducted and concluded.  Therefore, the contribution claims assessment will look to focus on iterative trends and leading indicators of progress. The outcome evaluation will use the lens of the OECD evaluation criteria of effectiveness in assessing if PATH-SAFE is on the path towards accomplishing its objectives. 

3.3.    Impact feasibility assessment

The impact feasibility assessment is an exercise to determine how to best evaluate the longer-term impact of PATH-SAFE. The assessment of outcomes based on the CA methodology will provide us with a useful baseline of impact and whether the contribution claims being tested are realistic or feasible. It will clarify which impacts remain relevant for the programme and what methodologies and indicators may be useful to consider. We will adapt and use the context, mechanisms and outcomes (CMO) framework, usually used in a realist evaluation approach (see section 3.4.2 Context, Mechanism, Outcomes Framework), to develop projections of impact. The purpose of utilising a CMO-style framework is not to actually conduct an impact evaluation using the CMO which would duplicate the work of the CA analysis, but rather to use the CMO in a novel way to create hypothesis of what the future outcomes/impacts might be, what the potential mechanisms of action and the context for it might be. This will be entirely based on the knowledge amassed from the process and the outcome evaluation, which will culminate in the CA. The outputs of the CA will inform the CMO style projections/hypotheses. The study team will reflect on the PATH-SAFE context (i.e. the external environment) to assess its potential effect on outcomes yet to be realised, and also consider the mechanisms in place in the PATH-SAFE programme (uncovered during the process and outcome evaluation) that are contributing and could continue to contribute to realising the anticipated outcomes and impacts. We will not be undertaking a CMO evaluation but rather utilising the framework for considering appropriateness of PATH-SAFE future outcomes and impacts and their potential realisation pathways which can inform a future measurement approach. 

3.4.    Methodological frameworks

The theory-based approach being utilised is underpinned by the programme ToC discussed in Section 2.1. Further to that, the outcome evaluation will be analysed through the framework of contribution analysis to assess PATH-SAFE’s contribution to outcomes and impacts, while the impact feasibility assessment will be undertaken using an adapted CMO framework. 

3.4.1.    Contribution analysis

To help attribute causality in a programme of this size and complexity, this theory-based evaluation will use the CA methodology on the data collected. CA is a method for assessing causal claims that examines the contribution of an intervention to observed results. It provides a framework for capturing progress towards aims at a relatively early stage through testing working hypotheses and establishing a case to explain the contribution made by PATH-SAFE and its projects over alternative hypotheses. Determining contributions requires qualitative methodologies (for example, deciding whether the relevant evidence has been identified, or if it is sufficient to discard alternative hypotheses), but is informed by both quantitative and qualitative evidence from all the methods undertaken throughout the evaluation. We will place greater weight on findings stemming from multiple data sources to assess the added value and true contribution of PATH-SAFE to the outcomes anticipated and realised. See Chapter 4 for further details on how this will be done. 

3.4.2.    Context, Mechanism, Outcomes Framework

The impact feasibility assessment will be conducted through utilising the CMO framework. The CMO framework will be used to create a projection of how the outcomes and impacts of PATH-SAFE may arise, as anticipated, based on the ToC. This projection will rely on abductive reasoning and the evidence gathered during the evaluation on identifying contextual factors and trends, as well as identified mechanisms of actions within PATH-SAFE. This assessment will allow us to iterate on the ToC and develop a realistic measurement approach for a longer-term and/or follow-up evaluation of PATH-SAFE. As mentioned above, the data gathered during the evaluation culminating in a CA will inform the basis of the CMO projection exercise. Although we are not undertaking a CMO based realist evaluation, the use of this framework provides a useful and structured template for impact feasibility assessment.

An illustrative example of a projection for PATH-SAFE utilising CMO is depicted in Figure 4 below. When assessing one of the anticipated impacts of PATH-SAFE, preventing the increase in foodborne illness, the evaluation of PATH-SAFE could help identify the mechanism through which this could occur. In this instance, work of WS3 could result in identification and development of onsite diagnostics for FBP and AMR which, if adopted, could help decrease the incidence of foodborne illness. This change might be possible if the technology in question is scalable and can be commercialised. This is an entirely hypothetical projection and will need to draw on the PATH-SAFE evaluation for validity. The next step after creating the projections would be to develop recommendations to modify outcomes and impacts in the original ToC (if required) and to propose methods for conducting future-focussed evaluations. 
 
Figure 5. Illustrative example of CMO analysis

3.5.    Limitations of the evaluation approach

The approach and methodologies outlined in the chapters above will provide a wide-ranging set of data and evidence around the ambitions of the PATH-SAFE programme, and whether these ambitions have been achieved. However, our approach to the evaluation is also subject to a number of important limitations. 

Firstly, whilst our approach aims to be comprehensive and cover different impacts of the PATH-SAFE programme, the lack of counterfactuals to compare the programme against poses a significant limitation. The programme being a pilot means that there are new outputs being developed such as the creation of a new genomic database and a pilot surveillance infrastructure. However, a potential mitigation of these limitations is to understand what was already in place preceding PATH-SAFE and to position the outputs of PATH-SAFE as building on existing capabilities.  

Second, our evaluation approach focuses mainly on the PATH-SAFE programme with a limited role for analysing the interactivity with external programmes of work in this space such as the AMR national action plan or the EU Farm to Fork strategy. On a similar note, the lack of international programme assessments means that it is more difficult to position the programme in a broader/international context. Lastly, developments in industry are not factored into the programme itself, so the evaluation has also not included them. This is a blind spot in understanding the state of play in terms of surveillance. 

Third, given that much of the anticipated impact of the PATH-SAFE programme will only emerge over a lengthy time horizon, the evaluation will not be able to capture its outcomes nor its long-term impacts in full. Ideally, the evaluation would involve a long-term follow up and assessment of PATH-SAFE to track these impacts of the programme. What we are proposing is a step in this direction, setting out a range of indicators that can be used to assess whether the programme is on track to achieve longer-term desired outcomes and impacts. Additionally, the evaluation will provide recommendations on a future-focussed evaluation approach to further the assessment of longer-term outcomes and impacts. 

Finally, as the programme is at pilot stage, and our data sources are limited and reliant upon the programme data availability itself, the possibility of low availability of baseline data due to project delays could be a challenge, limiting the range of data available across our evaluation timeline. This lack of data will need mitigation and caveats as the evaluation progresses.

The chapter comprises two distinct but interrelated frameworks: a process evaluation framework and an outcome evaluation framework. Presented in tabular form, each framework outlines evaluation questions (EQs), looking to assess progress made towards the PATH-SAFE ToC components. For each EQ, the frameworks contain indicators and proposed data sources that will be used to collect evidence to enable us to answer the EQ. Alongside process and outcome evaluation frameworks, the chapter also provides further detail on the data collection methods and analytical approaches that will be used for each type of evaluation. Impact feasibility is not included in this chapter as Section 3.3 describes the approach that will be undertaken to conduct the feasibility assessment. We have not ascribed indicators and data sources for the ToC impacts at this point as a result. 

4.1.    Process evaluation framework 

As highlighted in Chapter 3, the process evaluation will seek to understand the extent to which PATH-SAFE’s programme governance and resourcing has been fit for purpose and assess the mechanisms of actions across the four WSs. This assessment will be done through the lens of the principles of relevance and coherence based on the OECD criteria. Table 1 below sets out the process evaluation framework. Each WS’s activities (A) and outputs (O) are assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ.  Sections 4.1.1 and 4.1.2 describe in more detail the data collection tools and analytical approaches that will be used to undertake the process evaluation. 

Table 1. Process evaluation framework

4.1.1.    Process evaluation data collection methods

As shown in Table 1, the process evaluation will rely on three main methods of data collection: document review, desk research, and key informant interviews. These data collection methods are described in more detail below.  

Document review

We will conduct a review of PATH-SAFE management information such as business case bids, initial design documentation, and governance and monitoring requirements/criteria to further develop our understanding of PATH-SAFE programme processes. Documents to be reviewed will also include programme WS specific documentation such as WS project briefs (noting any changes in scope and delivery), latest highlight reports, and latest documentation for a given month/quarter from the Data Advisory Group (DAG), Shared Outcomes Fund, Scientific Advisory Group (SAG) and the Strategic Board. We will also review WS activity/technical reports where appropriate and available. This will be undertaken at both the interim process evaluation and the final process evaluation stages to assess the extent to which the intended outputs have been delivered. 

Desk research

We will review the AMR national action plan and the NBN documents to assess alignment with PATH-SAFE in more detail as helpful context of the process evaluation. We will also undertake a high-level grey literature search to map out key pathogen surveillance initiatives across Europe and the devolved nations in the UK to create a robust assessment of surveillance mechanisms and infrastructure already in place in the agriculture/environment sectors. 

Key informant interviews

Alongside document review, data on how the programme has been received by key delivery partners, government stakeholders and any other end users, as well as experience of engagement and incorporation of views into WSs, will be collected primarily through key informant interviews. To inform the process evaluation at the interim stage, we will conduct interviews with: 

• Up to four central operational staff at FSA 
• Up to 10 delivery partners including academics across WSs 1-4
• Up to 15 end users/key government stakeholders across DEFRA, UKHSA, FSS, EA, DHSC and Public Health Wales and NI, etc.

To inform the final process evaluation (and the outcome evaluation) we will conduct interviews with:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

Interviewees will be selected based on the PATH-SAFE stakeholder database, using a purposive sampling approach to ensure representation across WSs, government departments and types of end users. This will be done in consultation with the PATH-SAFE central team at FSA. Interview topic guides and analysis coding will be guided by the evaluation questions as specified in Table 1. We will also complement interviews through engagement with PATH-SAFE central and delivery teams at bi-weekly meetings and attendance at monthly Delivery Board meetings. 

4.1.2.    Process evaluation analysis

Data collected through the methods above will be brought together and triangulated against our process evaluation framework to create an understanding of how processes supported and/or created barriers in delivery of PATH-SAFE. In addition, to create an exemplified picture of effectiveness of PATH-SAFE processes, we propose to develop two case studies based on existing data collection methods highlighted with a potential for deeper dives into the proposed topics via interviews and documentary reviews. 

Case studies

Case studies will be selected purposively and will be used to tease out instances in which processes have worked exceptionally well, or to highlight examples where things haven’t gone as expected, highlighting opportunities for improvement. This will be determined through consultation with the programme team and considering the data emerging during the interim process evaluation. We propose to develop two process case studies. 

Given the central importance of cross-government engagement, we suggest focusing one case study on exemplifying good practice of an instance where cross-government collaboration has worked particularly well (this could be at central programme or at WS level). The case study will not only centre on what worked well but also look to identify enabling factors and levers for change that could be applied across the rest of the programme. 

We propose to focus the second case study on data sharing enablement, given its importance across multiple WSs and the programme as a whole. We will again look to exemplify good practice of where data sharing has been enabled or an agreement put in place and go further to identify what catalysed the process and what barriers remain to be addressed. 

4.2.    Outcome evaluation framework

As described in Chapter 3 Evaluation approach, the outcome evaluation will provide an assessment of the extent to which the outcomes outlined in the ToC have been realised. This will be a theory-led approach and will utilise CA to validate central claims made about the programme’s success, utilising the evidence collected against key outcomes and the key EQs (see section 3.4.1 and 4.2.2 for more info). Within the outcome evaluation framework, most outcomes listed are broadly mapped to the key WSs that are likely to contribute towards them, but some are at a programme-level, to which all WSs are anticipated to contribute. All outcomes have been assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ. Section 4.2 describes in more detail the data collection tools and analytical approaches that will be used to undertake the outcome evaluation.
 
Table 2. Outcome evaluation framework

4.2.1.    Outcome evaluation data collection methods

As indicated in Table 2, the outcome evaluation will draw on a wide range of sources underpinned by four main methodologies: documentary review, desk research, key informant interviews, and a workshop.  

Documentary review

Analysis of key activity reports and papers from meetings of the SAG, DAG, shared outcomes fund, and the strategic board will be analysed to assess the extent to which outcomes have been realised. More focus will be placed on direct WS reports to provide a sense of progress towards intended outcomes at the WS level.

Desk research

Desk research will be conducted on Google Scholar to assess grey literature outputs that can be attributed to PATH-SAFE. We will do this for the first 100 hits through a targeted search. In addition, an assessment of publications of academic papers, strategy and policy documents will be conducted through a search on bibliographic data platforms to assess what publications PATH-SAFE has enabled, if any, which will provide an understanding of PATH-SAFE knowledge generation and wider influence. 

Key informant interviews

To further strengthen our understanding of the extent and mechanism of outcome realisation, we will conduct key informant interviews. Please note that these will be the same set of interviews that are proposed for the final process evaluation stage to reduce burden on interview respondents. We foresee speaking to the same set of stakeholders and given the parallel timelines of the final process and the outcome evaluation, these set of interviews will look to assess both process and outcome EQs. As mentioned in section 4.1.1, we will conduct the following interviews for the final process and outcome evaluations:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

As with the process evaluation interviews, topic guides will be developed based on the key EQs in Table 1 and Table 2, and all interviews will follow a semi-structured format. 

Workshop

Assessment of step changes or any improvements made on high level outcomes of ‘improvements in surveillance capabilities and mechanisms’ and ‘awareness and engagement across government departments and key decision/policy makers’ will be more appropriately gleaned through a large workshop/group exercise (with up to 15 participants) undertaken with the relevant stakeholders. The central programme team will be key in determining the most appropriate mix of stakeholders to engage in this exercise.  

4.2.2.    Outcome evaluation analysis

The evidence from the methodologies outlined above will be triangulated to develop a holistic understanding of the difference PATH-SAFE has made. This will be crucially underpinned by undertaking a contribution analysis exercise (detailed below) and development of two case studies exemplifying a select component of a given outcome. 

Case studies

Case studies will be selected purposively and will be used to tease out instances where tangible examples of progress can be seen towards outcome realisation and/or to highlight examples where things haven’t gone as expected, or where outcomes have been significantly delayed, highlighting key barriers. This will be determined through consultation with FSA and considering the data emerging during the early phase of the outcome evaluation. We propose to develop two outcome case studies. 
We propose to focus one case study on showcasing an example (if available) of PATH-SAFE influencing a nationally linked operation/strategy (for example, the NBN), and focus on the enablers of influence and the nature of the influence to understand its importance. We propose to focus the second case study on an example of a novel tool or framework for testing/surveillance that has been developed and assess its value to improvement of surveillance. 

Contribution analysis

As mentioned before, CA is a method for assessing causal claims that provides a framework for capturing progress towards aims through testing working hypotheses and establishing a case to explain the contribution made by PATH-SAFE and its projects over alternative hypotheses. The six steps involved in CA are as follows:

1. Set out the cause-effect issue to be addressed.
2. Develop the postulated ToC and risks to it, including other influencing factors.
3. Gather the existing evidence on the ToC.
4. Assemble and assess the contribution claim, and challenges to it. 
5. Gather new evidence from the implementation of the intervention.
6. Revise and strengthen the contribution story. 

At this stage of the evaluation, Steps 1, 2 and 3 have been completed. Based on our understanding of what PATH-SAFE is aiming to achieve and the ToC underpinning the evaluation, we propose three main contribution claims for the programme as an output of Step 4. These are hypotheses that are central to the programme and can be interpreted as high-level and holistic outcomes of the programme. 

• The processes established in PATH-SAFE programme lead to cross-government collaboration on FBP and AMR surveillance because of increased transparency and engagement across departments through the work on interrelated WSs.
• The development of the data platform in PATH-SAFE leads to easier data sharing across government departments because of data sharing agreements put in place and extent of user engagement carried out.
• The collective outputs of the WSs in PATH-SAFE leads to establishment of a nationally connected and improved FBP and AMR surveillance approach because of multilocation sampling, novel testing tools and an interconnected data platform.

We plan to utilise the process and outcome evaluation evidence holistically (i.e., evidence from interviews, workshops and case studies) to address Step 4 in assessing the body of evidence to validate the contribution claim. We will then create an overarching narrative (i.e., the contribution story) relative to the strength of the evidence that makes a qualitative judgement on whether the contribution claims stand or whether an alternative hypothesis exists for what caused the change to occur. The alternative hypothesis in particular will be tested through interviews and a workshop, and will be derived from the external initiatives listed in Section 2.2.2. The contribution story will identify any gaps in the evidence or weak links, where we will look to alternative sources of data and revise the contribution narrative accordingly. The contribution narrative will ultimately rest on the collective evidence surfaced through the process and outcome evaluations.

5.1.    Evaluation plan

A Gantt chart visualising the planned timeframe for the implementation of evaluation activities is presented in Figure 5. The deliverables along with their deadlines are highlighted in Table 3.  The final report will be prepared by June 28, 2024. 

Figure 6. PATH-SAFE evaluation Gantt

5.2.    Evaluation deliverables and deadlines

Table 3 below shows the main deliverables and their associated deadlines for the project.

Table 3. Main deliverables and deadlines

Table 4. Evaluation risks and mitigations

Table 5 Overview of PATH-SAFE WS activity

B.1. Developing a programme theory of change

PATH-SAFE programme partners developed a first draft ToC ahead of RAND being commissioned.  As a key underpinning tool for the overall evaluation, we co-designed a refined ToC for the PATH-SAFE programme with the central programme team, outlining the pathways of change for each WS along with their interconnections and dependencies. We utilised the following activities to refine the ToC.

B.2. Documentary review and desk research

To revise the ToC, we conducted a review of key PATH-SAFE documents including relevant business cases and WS documentation outlining aims and ambitions. We also referred to external publications for developing appropriate external factors impacting the ToC and the evaluation such as the UK 5-year action plan for antimicrobial resistance 2019 to 2024, the UK Government Food Strategy, the Scottish Government Strategy for Environment, Natural Resources and Agriculture Research 2022-2027, and A Green Future: Our 25 Year Plan to Improve the Environment.

8.3. Engagement with expert advisors

We consulted with our expert advisors on the project, Dr. Arnoud van Vliet and Dr. Jennifer Ritchie, both from the University of Surrey. Both our advisors have deep expertise in AMR, with Dr. van Vliet providing specialised expertise on foodborne bacterial pathogens and microbial genomics, and Dr. Ritchie on host-pathogen interactions and transmission. Feedback from them helped finetune the ToC and its underpinning assumptions and external factors.

B.4. Central programme team engagement

This included engaging with the central programme team and WS managers through bi-weekly meetings and email exchange. Feedback received from this stakeholder engagement enabled us to further develop the ToC and refine it to reflect the scope and anticipated impact of PATH-SAFE. This initial development was then built upon by conducting a refinement and prioritisation workshop to further refine the ToC. Workshop attendees included stakeholder representatives from the central programme team. Moving forward, it is intended that this report (and thus the ToC and evaluation frameworks) will be shared with other cross-department stakeholders to ensure the validity and utility of the evaluation approach outlined. 

C.1 Document review

This included review of WS specifications, delivery plans and contractual reports available to flesh out the ToC components pertaining to each WS. 

C.2 Central programme team engagement

This engagement included meetings with the central programme team, feedback from WS managers, and a refinement and prioritisation workshop. The workshop helped us build on the document review and plug gaps in our knowledge to improve our understanding of the WSs and their interconnectivity and dependencies as well as finetune and prioritise the key EQs. This will be followed up through engagement with other government stakeholders, via FSA, to ensure the validity and utility of the evaluation approach outlined.  

C.3 Engagement with expert advisors

We also liaised with our expert advisors, Dr. Arnoud van Vliet and Dr. Jennifer Ritchie, who provided the team with more nuanced information on the important data sources to refer to for key indicators, along with vital feedback on further refining the EQs.