Evaluation of the PATH-SAFE programme - Evaluation framework

The chapter comprises two distinct but interrelated frameworks: a process evaluation framework and an outcome evaluation framework. Presented in tabular form, each framework outlines evaluation questions (EQs), looking to assess progress made towards the PATH-SAFE ToC components. For each EQ, the frameworks contain indicators and proposed data sources that will be used to collect evidence to enable us to answer the EQ. Alongside process and outcome evaluation frameworks, the chapter also provides further detail on the data collection methods and analytical approaches that will be used for each type of evaluation. Impact feasibility is not included in this chapter as Section 3.3 describes the approach that will be undertaken to conduct the feasibility assessment. We have not ascribed indicators and data sources for the ToC impacts at this point as a result. 

4.1.    Process evaluation framework 

As highlighted in Chapter 3, the process evaluation will seek to understand the extent to which PATH-SAFE’s programme governance and resourcing has been fit for purpose and assess the mechanisms of actions across the four WSs. This assessment will be done through the lens of the principles of relevance and coherence based on the OECD criteria. Table 1 below sets out the process evaluation framework. Each WS’s activities (A) and outputs (O) are assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ.  Sections 4.1.1 and 4.1.2 describe in more detail the data collection tools and analytical approaches that will be used to undertake the process evaluation. 

Table 1. Process evaluation framework

WS	Category	ToC	Key evaluation questions	Indicators	Data sources	Methods
Programme-level	Programme-level	Programme-level	How appropriately resourced has PATH-SAFE been throughout the stages of inception, design and implementation? How effective and appropriate is the governance in place to support delivery of PATH-SAFE?	Feedback from inter/cross-govt stakeholders on strength of relationships established and any perceptions of barriers   Interviews with FSA PATH-SAFE programme team and governance documentation	Interviews with relevant PATH-SAFE partners and FSA PATH-SAFE programme team Interviews with inter/cross-govt PATH-SAFE stakeholders/partners	Interviews and documentary review
Programme-level	Programme-level	Programme-level	How is cross-government interaction being enabled/conducted?	Number and nature of opportunities and communication platforms set up to facilitate cross-govt interaction Feedback from inter/cross-govt stakeholders on strength of relationships established and any perceptions of barriers	Interviews with relevant PATH-SAFE partners and FSA PATH-SAFE programme team Interviews with inter/cross-govt PATH-SAFE stakeholders/partners	Interviews
Programme-level	Programme-level	Programme-level	How is PATH-SAFE linked to existing/developing surveillance programmes?	Level of alignment and linkages between PATH-SAFE and other relevant surveillance programmes mapped and outlined using conceptualisation documents	Management information (project business case and bids and approval outputs); interviews with FSA programme management team; desk research on key surveillance mechanisms across Europe and US (for example, GenomeTrackr) and devolved nations	Interviews, documentary review and desk research
WS1	Activity and Outputs	A: Establish a curated and national FBP (and their AMR) genomic data platform with Salmonella as exemplar pathogen O: Functional and scalable data platform that houses sequences and facilitates analysis of exemplar pathogens (for example, Salmonella and their AMR genes  O: Data platform is interoperable and can interact with other systems like Enterobase and provide an interrogatable user interface	To what extent have relevant end users been engaged and how have their needs been incorporated into the design of the database? How has data interconnectivity and interoperability been considered in designing the platform?	Breadth of end users engaged  Satisfaction of end users Types of databases and datasets consulted for interoperability (for example, NCBI, Enterobase, etc.) Interoperability assessments undertaken and recommendations Data access and sharing arrangements in place	Interviews with intended end users and delivery partners; review of updates/notes from delivery board meetings, discovery project outputs and end user reports Interviews with intended end users Review of highlight reports and DAG and SAG reports Interviews with delivery partners Interviews with delivery partners and FSA management, review of DES/highlight/DAG/SAG reports	Interviews and documentary review Interviews Documentary review Interviews Interviews and documentary review
WS2 and WS1b	Activity and Output	A: Pilot new FBP and AMR surveillance approaches based on regular, multi-location sampling in a range of settings, combined with novel technologies (for example, WGS) O: AMR and FBP and AMR curated sample data captured from multiple sources, and tested using novel analysis techniques                 O: Evidence on the utility and suitability of the piloted FBP and AMR surveillance and modelling approaches	What existing and novel analysis technologies are being utilised? What is the extent of data collection and curation? How (if at all) are new capabilities being generated to improve surveillance How is data being accessed/ shared across relevant stakeholders and departments?	Number and type of analysis technologies being utilised; assessment of existing capability utilisation Number of samples taken; number of sampling sites accessed; number of genome sequences generated Consolidation of sampling and data curation outputs; number of new tools and models developed;  Data access and sharing arrangements in place	Interviews with sponsors and delivery partners; review of highlight/activity reports Delivery partners reports/Delivery board updates Interview with sponsors and delivery partners; review of highlight and activity reports Interviews with delivery partners and FSA management; review of highlight/DAG/SAG reports	Interviews and documentary review Documentary review Interview and documentary review Interviews and documentary review
WS3	Activity and Output	A: Map and test new and repurposed technologies for rapid onsite FBP testing in collaboration with end users                   O: TRL assessment of rapid onsite FBP testing tools with end users     O: Evidence on utilising COVID-19 testing technology (LAMP) for FBP detection in wastewater	To what extent is the TRL assessment approach valuable for identification of relevant technology? To what extent has the work divulged utility of LAMP as a feasible method? How is LAMP assessment feeding into TRL mechanisms for FBP diagnostics?	Type of technologies being assessed; review of process of assessment; end users views on TRL assessments and other outputs being fit for purpose Assessment of utilisation of WS3b outputs into 3a	Interviews with delivery partners and end users; review of activity reports and TRL assessment outputs Interview with delivery partners; review of activity reports and TRL assessment outputs	Interviews and documentary review Interviews and documentary review
WS4	Activity and Output	A: Develop a pilot AMR surveillance system based on mechanisms of AMR spread in the environment O: AMR surveillance framework and suite of diagnostics enabling monitoring of AMR across the environment within a catchment area	What is being learnt and incorporated from existing AMR surveillance systems and tools? How is connectivity between the WS4 AMR environment platform and WS1a being considered? How is evidence being aggregated across the multiple departments involved in WS4 delivery	Breadth of mapping and engagement with existing AMR surveillance systems and tools Engagement between WS4 and WS1a; understanding of interoperability between platforms Assessment of WS4 delivery partner engagement mechanisms and frequency	Interviews with delivery partners and review of activity reports Interviews with delivery partners; review of shared terms/project outputs/highlight reports Review of WS4 governance and reporting mechanisms	Interviews and documentary review Interviews and documentary review Documentary review

4.1.1.    Process evaluation data collection methods

As shown in Table 1, the process evaluation will rely on three main methods of data collection: document review, desk research, and key informant interviews. These data collection methods are described in more detail below.  

Document review

We will conduct a review of PATH-SAFE management information such as business case bids, initial design documentation, and governance and monitoring requirements/criteria to further develop our understanding of PATH-SAFE programme processes. Documents to be reviewed will also include programme WS specific documentation such as WS project briefs (noting any changes in scope and delivery), latest highlight reports, and latest documentation for a given month/quarter from the Data Advisory Group (DAG), Shared Outcomes Fund, Scientific Advisory Group (SAG) and the Strategic Board. We will also review WS activity/technical reports where appropriate and available. This will be undertaken at both the interim process evaluation and the final process evaluation stages to assess the extent to which the intended outputs have been delivered. 

Desk research

We will review the AMR national action plan and the NBN documents to assess alignment with PATH-SAFE in more detail as helpful context of the process evaluation. We will also undertake a high-level grey literature search to map out key pathogen surveillance initiatives across Europe and the devolved nations in the UK to create a robust assessment of surveillance mechanisms and infrastructure already in place in the agriculture/environment sectors. 

Key informant interviews

Alongside document review, data on how the programme has been received by key delivery partners, government stakeholders and any other end users, as well as experience of engagement and incorporation of views into WSs, will be collected primarily through key informant interviews. To inform the process evaluation at the interim stage, we will conduct interviews with: 

• Up to four central operational staff at FSA 
• Up to 10 delivery partners including academics across WSs 1-4
• Up to 15 end users/key government stakeholders across DEFRA, UKHSA, FSS, EA, DHSC and Public Health Wales and NI, etc.

To inform the final process evaluation (and the outcome evaluation) we will conduct interviews with:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

Interviewees will be selected based on the PATH-SAFE stakeholder database, using a purposive sampling approach to ensure representation across WSs, government departments and types of end users. This will be done in consultation with the PATH-SAFE central team at FSA. Interview topic guides and analysis coding will be guided by the evaluation questions as specified in Table 1. We will also complement interviews through engagement with PATH-SAFE central and delivery teams at bi-weekly meetings and attendance at monthly Delivery Board meetings. 

4.1.2.    Process evaluation analysis

Data collected through the methods above will be brought together and triangulated against our process evaluation framework to create an understanding of how processes supported and/or created barriers in delivery of PATH-SAFE. In addition, to create an exemplified picture of effectiveness of PATH-SAFE processes, we propose to develop two case studies based on existing data collection methods highlighted with a potential for deeper dives into the proposed topics via interviews and documentary reviews. 

Case studies

Case studies will be selected purposively and will be used to tease out instances in which processes have worked exceptionally well, or to highlight examples where things haven’t gone as expected, highlighting opportunities for improvement. This will be determined through consultation with the programme team and considering the data emerging during the interim process evaluation. We propose to develop two process case studies. 

Given the central importance of cross-government engagement, we suggest focusing one case study on exemplifying good practice of an instance where cross-government collaboration has worked particularly well (this could be at central programme or at WS level). The case study will not only centre on what worked well but also look to identify enabling factors and levers for change that could be applied across the rest of the programme. 

We propose to focus the second case study on data sharing enablement, given its importance across multiple WSs and the programme as a whole. We will again look to exemplify good practice of where data sharing has been enabled or an agreement put in place and go further to identify what catalysed the process and what barriers remain to be addressed. 

4.2.    Outcome evaluation framework

As described in Chapter 3 Evaluation approach, the outcome evaluation will provide an assessment of the extent to which the outcomes outlined in the ToC have been realised. This will be a theory-led approach and will utilise CA to validate central claims made about the programme’s success, utilising the evidence collected against key outcomes and the key EQs (see section 3.4.1 and 4.2.2 for more info). Within the outcome evaluation framework, most outcomes listed are broadly mapped to the key WSs that are likely to contribute towards them, but some are at a programme-level, to which all WSs are anticipated to contribute. All outcomes have been assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ. Section 4.2 describes in more detail the data collection tools and analytical approaches that will be used to undertake the outcome evaluation.
 
Table 2. Outcome evaluation framework

Workstream	Category	TOC	Key evaluation question (s)	Indicator	Data source	Method
WS1	Outcomes	Key stakeholders can more easily share and access data across organisations for rapid identification and tracking of foodborne pathogens and AMR, bringing together multiple data sources	Has data access, sharing, and use for FBP and AMR been enabled and improved across government departments?
WS1	Outcomes	Predictive assessment of risk and threat is enabled when assessing a new isolate through access to a comparative repository of pathogen sequences and metadata	To what extent has the platform supported use of relevant metadata and historic isolates for comparative assessments and risk profiles of FBP?
WS2 and 4	Outcomes	Improved understanding of source attribution and infection threat of FBP and AMR through various environments and international entry points. Additional knowledge of how to expand existing surveillance mechanisms to support a robust national surveillance infrastructure and improved monitoring Informed consideration, based on evidence surfaced, on how proactive, rapid and efficient management can be used to reduce the risk of FBP and AMR introduction into the wider environment and food systems.	How has the collective source detection efforts and use of novel technology translated to (if at all) improved surveillance of FBP and AMR? To what extent have the pilot efforts been able to exemplify practice and enhance national surveillance capability? What kind of strategies and operations have been enhanced, enabled and influenced (if at all) through the surveillance activities?	Speed of FBP/AMR detection in number of days looking at end to end process Comprehensiveness of coverage for example, density of testing, number of sampling sites covered, and sequences curated and comparative strain assessment Feedback from end users and relevant PATH-SAFE partners/govt stakeholders on improvements made in surveillance Feedback from end users and relevant PATH-SAFE partners on national surveillance capability improvements Types of strategies and operations that have been enabled; other national strategies and action plans enhanced or influenced (for example, NBN, AMR NAP, etc.); knowledge generated	Review of project activity reports/highlight reports Review of project activity reports/highlight reports Workshop with PATH-SAFE delivery partners and key government stakeholders  Workshop with relevant PATH-SAFE stakeholders (include representatives of UK and devolved governments and their agencies (for example, FSA, DEFRA, Welsh Government), health agencies and health boards (for example,  Public Health Wales, UKHSA) Review of final reports, board reports, publications/grey lit citations; and interviews with FSA programme management	Documentary review Documentary review Workshop Workshop Interviews and documentary review
WS3	Outcomes	Guide the use of novel and existing/repurposed rapid onsite FBP testing technology with improved knowledge of where further development is needed	Have the tools identified been useful for end users? Can they be utilised? To what extent have gaps been identified to further development of onsite rapid FBP detection?	Types (and number) of technologies and tools identified; feedback from end users on relevance and utility; evidence of gaps identified to proceed further on tech development	Review of project activity reports/highlight reports; end user interviews	Interviews and documentary review
Programme Level	Outcomes	Key stakeholders and decision makers are brought together to engage with evidence and take forward policy recommendations. Contributing to the ‘One Health’ ambitions of reducing threats to public health and the ecosystem.	How has PATH-SAFE (if at all) enabled a community of practice and decision makers to come together to inform and act on surveillance of FBPs and AMR? How and to what extent has PATH-SAFE evidence (if at all) contributed to national policies and frameworks for improved public health	Feedback from end users and policymakers on awareness of and engagement with PATH-SAFE  Knowledge generated (publications/grey lit citations); Feedback from end users and policymakers on use of PATH-SAFE evidence into policy and strategies for public health, agriculture and environment interventions	Workshop with relevant PATH-SAFE stakeholders (including representatives of UK and devolved governments and their agencies (for example, FSA, DEFRA, Welsh Government), health agencies and health boards (for example,  Public Health Wales, UKHSA) Desk research and use of bibliographic databases Interviews with key government decision makers	Workshop Desk research Interviews

4.2.1.    Outcome evaluation data collection methods

As indicated in Table 2, the outcome evaluation will draw on a wide range of sources underpinned by four main methodologies: documentary review, desk research, key informant interviews, and a workshop.  

Documentary review

Analysis of key activity reports and papers from meetings of the SAG, DAG, shared outcomes fund, and the strategic board will be analysed to assess the extent to which outcomes have been realised. More focus will be placed on direct WS reports to provide a sense of progress towards intended outcomes at the WS level.

Desk research

Desk research will be conducted on Google Scholar to assess grey literature outputs that can be attributed to PATH-SAFE. We will do this for the first 100 hits through a targeted search. In addition, an assessment of publications of academic papers, strategy and policy documents will be conducted through a search on bibliographic data platforms to assess what publications PATH-SAFE has enabled, if any, which will provide an understanding of PATH-SAFE knowledge generation and wider influence. 

Key informant interviews

To further strengthen our understanding of the extent and mechanism of outcome realisation, we will conduct key informant interviews. Please note that these will be the same set of interviews that are proposed for the final process evaluation stage to reduce burden on interview respondents. We foresee speaking to the same set of stakeholders and given the parallel timelines of the final process and the outcome evaluation, these set of interviews will look to assess both process and outcome EQs. As mentioned in section 4.1.1, we will conduct the following interviews for the final process and outcome evaluations:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

As with the process evaluation interviews, topic guides will be developed based on the key EQs in Table 1 and Table 2, and all interviews will follow a semi-structured format. 

Workshop

Assessment of step changes or any improvements made on high level outcomes of ‘improvements in surveillance capabilities and mechanisms’ and ‘awareness and engagement across government departments and key decision/policy makers’ will be more appropriately gleaned through a large workshop/group exercise (with up to 15 participants) undertaken with the relevant stakeholders. The central programme team will be key in determining the most appropriate mix of stakeholders to engage in this exercise.  

4.2.2.    Outcome evaluation analysis

The evidence from the methodologies outlined above will be triangulated to develop a holistic understanding of the difference PATH-SAFE has made. This will be crucially underpinned by undertaking a contribution analysis exercise (detailed below) and development of two case studies exemplifying a select component of a given outcome. 

Case studies

Case studies will be selected purposively and will be used to tease out instances where tangible examples of progress can be seen towards outcome realisation and/or to highlight examples where things haven’t gone as expected, or where outcomes have been significantly delayed, highlighting key barriers. This will be determined through consultation with FSA and considering the data emerging during the early phase of the outcome evaluation. We propose to develop two outcome case studies. 
We propose to focus one case study on showcasing an example (if available) of PATH-SAFE influencing a nationally linked operation/strategy (for example, the NBN), and focus on the enablers of influence and the nature of the influence to understand its importance. We propose to focus the second case study on an example of a novel tool or framework for testing/surveillance that has been developed and assess its value to improvement of surveillance. 

Contribution analysis

As mentioned before, CA is a method for assessing causal claims that provides a framework for capturing progress towards aims through testing working hypotheses and establishing a case to explain the contribution made by PATH-SAFE and its projects over alternative hypotheses. The six steps involved in CA are as follows:

1. Set out the cause-effect issue to be addressed.
2. Develop the postulated ToC and risks to it, including other influencing factors.
3. Gather the existing evidence on the ToC.
4. Assemble and assess the contribution claim, and challenges to it. 
5. Gather new evidence from the implementation of the intervention.
6. Revise and strengthen the contribution story. 

At this stage of the evaluation, Steps 1, 2 and 3 have been completed. Based on our understanding of what PATH-SAFE is aiming to achieve and the ToC underpinning the evaluation, we propose three main contribution claims for the programme as an output of Step 4. These are hypotheses that are central to the programme and can be interpreted as high-level and holistic outcomes of the programme. 

• The processes established in PATH-SAFE programme lead to cross-government collaboration on FBP and AMR surveillance because of increased transparency and engagement across departments through the work on interrelated WSs.
• The development of the data platform in PATH-SAFE leads to easier data sharing across government departments because of data sharing agreements put in place and extent of user engagement carried out.
• The collective outputs of the WSs in PATH-SAFE leads to establishment of a nationally connected and improved FBP and AMR surveillance approach because of multilocation sampling, novel testing tools and an interconnected data platform.

We plan to utilise the process and outcome evaluation evidence holistically (i.e., evidence from interviews, workshops and case studies) to address Step 4 in assessing the body of evidence to validate the contribution claim. We will then create an overarching narrative (i.e., the contribution story) relative to the strength of the evidence that makes a qualitative judgement on whether the contribution claims stand or whether an alternative hypothesis exists for what caused the change to occur. The alternative hypothesis in particular will be tested through interviews and a workshop, and will be derived from the external initiatives listed in Section 2.2.2. The contribution story will identify any gaps in the evidence or weak links, where we will look to alternative sources of data and revise the contribution narrative accordingly. The contribution narrative will ultimately rest on the collective evidence surfaced through the process and outcome evaluations.