Gaia, together with the Finland Futures Research Centre (FFRC) of the University of Turku, recently completed the evaluation of Finnish Innovation Fund Sitra’s Sustainability Solutions theme, focusing on the 2019-2022 period. The theme’s impact goal has been to ensure the adaptation to the earth’s carrying capacity by ecological reconstruction of society and everyday life.
The evaluation was implemented with an innovative mixed-method approach, combining desktop research, an online questionnaire, interviews, participatory workshops, and two adapted foresight methods; horizon scanning and delphi panel. While the primary objective of the evaluation was to evaluate the outcomes of the interventions, the evaluation process also included a foresight phase which aimed to identify and evaluate the most significant futures phenomena around ecological sustainability, which could be used to validate the relevance and timeliness of Sitra’s strategic choices during the evaluated period, and for strengthen Sitra’s future-oriented approach around the thematic work.
The planning work for the foresight process was conducted in early 2023 together with Sitra’s strategy and foresight specialists at the same time of finalizing the work plan for the evaluation. The foresight phase was based on a long list of phenomena that was first generated based on the recent foresight exercises done at Sitra, Gaia and FFRC. This long list was further processed with Sitra in a sense-making session and then presented to international experts to assess in a two round Delphi-process.These external contributors with futures expertise and environmental science background helped us to anticipate and assess most relevant futures phenomena in Finland and at EU level. Details of the methods and execution of the foresight phase can be found from the report.
Based on the experience and lessons generated during the evaluation process, here are our four key takeaways from the work for evaluation practitioners planning to integrate foresight methods in the evaluation framework.
Key takeaway #1: Combining foresight methods in the evaluation frame makes the evaluation more transformational
The world is in a poly-crisis and changing at a fast pace. All organizations must go through sustainable transformation – susformation – in order to survive and thrive in the future. Therefore, it is vital that when evaluating an organisation’s sustainability operations, impact, decisions or focus topics, the process is forward looking taking into account possible future trajectories, trends and phenomena. This is where foresight methods can make the evaluation more transformational: future-proofing evaluation recommendations or way-forward and truly oriented towards sustainable organizational development.
Key takeaway #2. Diversifying methods – including those of foresight – improves the robustness of the evaluation design
One of the key elements in evaluations is triangulation of information – applying several methods and datasets in order to overcome possible biases or other weaknesses. At its best, a foresight process is also a combination of methods complementing other evaluation methods. This allows the process to try out what works best – and to change course in case the method used does not deliver anticipated outcomes.
Our project combined traditional evaluation methods such as desktop research, horizon-scanning, co-creation and sense-making workshops together with Sitra experts with a Delphi-method, which in itself had both surveys and interviews. This allowed us to gain versatile viewpoints and to be able to triangulate relevant data from various data points. Even if some of the phases had not gone as planned, we would still have had insights through other methods used.
Key takeaway #3: Using a participatory, co-creation approach is critical in an innovative evaluation design.
Using a multi-stakeholder, participatory approach makes the process and its results more robust. While Sitra (the commissioner) had developed the initial evaluation design, the evaluation team (Gaia and Futures Research Center) further enriched and grounded the evaluation methodology and brought the external, independent views to the process. The Futures Reseach Center and Gaia could also draw from the good practices they had from previous Delphi-processes. The participatory process meant that Sitra’s experts from evaluation, programme, and foresight teams participated in the key evaluation stages together with the evaluation team. This co-creation approach helped the evaluation scope and outputs to stay truly relevant to the evaluated organisation, and above all, was a great learning journey to a wider range of stakeholders within the organisation.
One of the most positive outcomes of the evaluation process was related to the buy-in from the programme staff, whose work was under evaluation. While traditional backward-looking evaluation doesn’t always get programme staff excited- as many of the evaluation findings are already well-known to them at the time of the evaluation- a forward-looking, foresight-infused, participatory evaluation process brings something new to the table and actually helps them future-proof their programming and plan ahead.
Key takeaway #4 Paying extra attention to the scoping of foresight work.
It is important to assess and make sure that the foresight process truly supports and brings value for the evaluation. Particularly in the light of previous key takeaways calling for multiple methods and participatory approaches. Thus, the scope of the foresight work and selection of the used methods must be relevant both for the evaluation and the stakeholders (e.g. organization). In our project, the key scoping decision was to limit the foresight work on a list of phenomena related to Sitra’s work and thus directly link the foresight with the evaluation. The list of phenomena was co-created with Sitra staff. This was seen as extremely useful, as it ensured that the selection and de-selection of phenomena served the ultimate goal of the evaluation. Scoping of the work needs also to take into account the available resources and required competencies. Foresight brings in added value to the evaluation but does increase the resources needed for evaluation. While at least in Finland there is a strong cadre of foresight and evaluation professionals, there are less evaluators who have foresight competencies and less foresight practitioners with evaluation expertise. This, however, is hopefully changing as the value of futures- focused evaluation work gains momentum!
Curious to read more about our actual evaluation and foresight results? The evaluation report of the Sustainability Solutions theme (in Finnish, with English summary) has been published on Sitra’s website, and the foresight results were also published as a separate report (in English).
Gaia is an expert in future-proof, development-oriented evaluations with a track record of evaluations of over 20 years. Sitra is an independent future fund, engaging in foresight activities to understand the phenomena that influence the future and working together with partners to conduct experiments and pilot projects that strengthen the reform and resilience of society. The authors are evaluation and foresight experts.
Authors:
Matleena Moisio, Senior Consultant, Gaia Consulting
Jasmin Järvinen, Consultant, Gaia Consulting
Mari Hjelt, Director, Gaia Consulting
Rose Thompson Coon, Senior Lead, Impact & Evaluation, Sitra
Ville Lauttamäki, Post Doctoral Researcher, Finland Futures Research Centre
