The GSIP Project started in November 2018 and ended on 31 July 2022. All Work Packages focused on one of the key challenges for the development of a sound local labour market, namely low-skilled workplaces and an under-educated workforce. This challenge causes major risks for the Vantaa-based SMEs’ competitiveness in the current era of digitalization and automation.
Project Partners have adopted a comprehensive monitoring and evaluation mechanism (M&E) comprising of three complementary components:
- Self-evaluation and analysis conducted by the Project Management Team
- Impact assessment carried out by the two independent research partners of the GSIP Project, the Research Institute of the Finnish Economy “ETLA” and the Labour Institute for Economic Research PT
- An external ex-post evaluation process, with the aim to analyse the implementation of the Project using four pre-defined criteria (relevance, effectiveness, efficiency and impacts) as well as to support recommendations for continuity, replicability and the scalability of the Project’s results and findings.
PCM helped Project partners to ensure that all activities were relevant to the agreed strategy (as laid down in the Project application form) and to the problems of selected target groups. It supported to a great extent their adaptation to the COVID-19 challenges, given that the achievement of project objectives is always subject to influences beyond project manager’s direct control (assumptions and risks). It is therefore important to monitor this ‘external’ environment to identify whether or not the assumptions that have already been made are likely to hold true, what new risks may be emerging, and to take action to manage or mitigate these risks where possible.
(a) The Project was implemented according to relevant UIA terms and conditions, taking advantage of an internal monitoring system to oversee and follow implementation and to support project management. It has used the Project Cycle Management (PCM) approach, which represents the whole of management activities and decision-making procedures used during its life cycle. A comprehensive M&E plan was designed and put into practice during the first implementation phase of the Project (https://uia-initiative.eu/en/news/gsip-expert-journal-1-get-know-project-and-what-happened-first-6-months), given that the initial Project Plan did not include a comprehensive M&E plan that would allow also systematic collection and analysis of qualitative data.
(b) The City of Vantaa has contracted the independent agency MDI (Etusivu - MDI) to perform an external evaluation process of the GSIP Project. The key objective of the external evaluation was to produce an independent analysis of the Project using four pre-defined assessment criteria: relevance, effectiveness, efficiency and impacts. In addition, it addressed recommendations for continuity, replicability and the scalability of the Project’s results and findings.
The evaluation was conducted as a qualitative evaluation and a variety of qualitative methodologies, such as interviews and document analysis, were used. Information was collected from GSIP Project managers, partner and pilot companies, the UIA Expert for the GSIP Project, the UIA secretariat and experts on lifelong learning (e.g., Sitra, OECD). The evaluation has also utilized several participative methods. The collection of the evaluation materials and the findings were based on material produced by and the views and experiences of different stakeholders.
The main conclusions of the external evaluation process are classified as follows:
CONCLUSION 1. The project’s demand-driven operating model and tailoring of the services provided to participating companies supported the Project’s ability to meet the needs of its target groups. Adopting this demand-driven approach in planning and executing services for companies creates prerequisites for better effectiveness in the long run.
CONCLUSION 2. The Project supported the project partners ability to manage, carry out and participate in challenging experimental and co-creation processes in cooperation with a diverse range of stakeholders. In addition, the project improved the understanding of the target group among the project partners.
CONCLUSION 3. The Project activities have created a large amount of experiential learnings and observations that can be further utilised when creating and producing models and services for competence development.
CONCLUSION 4. The innovativeness and novelty value of the Project are linked to the adaptation of new operating models and approaches.
CONCLUSION 5. The Project partners have seized on the opportunity to change plans when a planned activity has not worked out. In this sense, project implementation has been agile, flexible and adaptable and thus highly suitable for innovative-type projects.
(c) Etla and Labore adopted and applied the counterfactual (outputs and outcomes in the absence of the intervention) impact evaluation method (CIE) during the project implementation period to provide solid information about the impacts produced by the intervention by comparing GSIP participants (Vantaa based companies employing 10-200 people, particularly companies involved in human intensive and routinely operated industrial sectors and IT-companies which have workforce of outdated skills caused by rapid changes in technologies and future business) to a randomly assigned control group. In practice, companies were randomly divided into two groups: the treatment and control groups. Only the companies in the treatment group were offered the service and incentive models. Due to the randomization, treatment and control groups were on average similar, and thus, comparing the changes in the mean outcomes of the groups, provided reliable information on how well the developed models work. Both extensive administrative data and multi-round survey data were used in the analysis.
The main conclusions of the impact evaluation process are classified as follows:
CONCLUSION 1. The quantitative targets set for participation of the key beneficiaries (local companies) were clearly achieved. 65 companies took advantage of the Project services, which surpass the goal set in the Project Plan by 8 percent. A large share of the participating companies was involved in more than one Growth Pact (https://uia-initiative.eu/en/news/gsip-project-story-during-2021-journal-3), indicating high level of satisfaction among the treated companies. Growing firms and firms that were making also other investments simultaneously were more likely to participate in the Project services.
CONCLUSION 2. The quantitative targets set for each Growth Pact were hardly achieved due to a variety of external factors strongly connected to the COVID-19 pandemic and its impact on the Vantaa labour market. In addition, the impact evaluation confirmed that companies still follow long-term strategies in terms of productivity and use of human resources, which are not eager to change drastically. Achieving changes in these fundamentals is a challenging and a long process.
CONCLUSION 3. Due to these challenges, it is believed that the businesses and the Vantaa area may have experienced improvements, not firmly validated by the impact evaluation process. A major lesson for any future local jobs and skills initiative must be to take into account also the requirements of an impact evaluation when planning the project.