Evaluation of the PATH-SAFE programme
Evaluation of the PATH-SAFE programme - Evaluation risks
This chapter considers potential risks to the evaluation of the PATH-SAFE programme and outlines mitigation approaches.
Table 4. Evaluation risks and mitigations
| Identified risk | Likelihood of risk (high, medium, low) | Impact of Risk (high, medium, low) | Risk management strategy |
|---|---|---|---|
| Complex policy environment makes it difficult to attribute contributions of PATH-SAFE to outcomes | High | Medium | The programme has been delivered in an evolving and fluid policy environment, which makes attributing impacts directly to PATH-SAFE challenging. Adopting contribution analysis will help clarify what PATH-SAFE has delivered and consider any alternative hypotheses. |
| Low engagement of stakeholders in interviews |
Medium | High | Since there is a risk of a low response rate if PATH-SAFE stakeholders have not been properly introduced to the evaluation and its importance, we will look to draw support from the central programme team in engaging with stakeholders to contribute to the evaluation. |
| Outcomes have not fully emerged during the timeframe of the evaluation | High | Medium | PATH-SAFE is a large, complex programme whose outcomes may take several years to fully emerge. Given that the programme began in early 2022 and the evaluation will conclude in 2024, many of these outcomes will not be captured. However, by rigorously assessing outputs and outcomes which are apparent, we can determine whether PATH-SAFE is on track to accomplish its intended outcomes and what further actions are required to maximise them. |
| Lack of counterfactual to assess additionality of the programme | High | Low | As PATH-SAFE is a pilot programme, and is multifaceted and multisectoral, it would be unlikely to find a counterfactual suited for this evaluation to determine additionality of PATH-SAFE. However, given the fragmented surveillance ecosystem and the need for a ‘One Health’ approach, assessing the value add of PATH-SAFE will be measured against what predated the programme in surveillance capabilities and way of working. |
| Low quality/availability of data | Medium | High | Where there are considerable gaps or certain documents are unavailable, we will discuss with the central programme team and our experts to identify the best way forward. The overall evaluation will draw on additional insights from interviews, focus, case studies and wider secondary data to ensure multiple data avenues are available. |
| Risks related to reliance on secondary data | Medium | Medium | We will use mixed methods to mitigate limitations of individual datasets. While we plan to rely heavily on secondary data, this will be complemented by primary data collection through interviews and focus groups, allowing triangulation of sources to ensure an evaluation that is as robust as is feasible. |
| Direction and coverage of the evaluation is not as expected by the programme | Low | Medium | RAND will maintain regular dialogue and engagement with central programme team and use this report as an opportunity for feedback on the approach to ensure alignment. |
| Poor quality of outputs | Low | High | All RAND reports go through quality assurance (rigorous peer-review by two independent reviewers) ensuring their quality. |
| Poor communication with the central programme team | Low | Low | Frequent communication with the FSA is included in the project plan through bi-weekly meetings and emails. We will also be attending monthly Delivery Board meetings. |
| Scope creep and moving goal posts | Low | Medium | The scope and objectives of the study will be confirmed and finalised through the approval of this evaluation framework report; should any changes be necessary over the course of the project, they will be agreed in writing between the study team and FSA; regular communication with the central programme team maintained through regular calls and/or e-mail; project manager acts as the main points of contact for the central programme team if any issues come up. |
| Overrun of timescales | Low | Medium | Our strong project management and experienced team should ensure that the project runs to schedule and that the FSA is kept regularly informed of developments. RAND Europe’s management information systems provide detailed weekly information on the status of each project and each team member, allowing project managers to respond rapidly to any issues arising. |
| Data and security breaches | Low | High | All data collection and processing will be in line with GDPR requirements as RAND Europe is ISO27001 certified, and we have in-house GDPR support. |
| Delays to WS outputs and overrun of the programme | Medium | High | Overrun of programme/WS outputs will entail adaptation of our questions and indicators to assess progress towards the anticipated objectives. |