Impact evaluation in the Urban Growth Vantaa project

A central idea of the project is to create and test innovative models that provide incentives and tools for both firms and labour force to adapt to technological change. The focus of the project is on firms with higher shares of routine jobs and on low-skilled individuals. The Urban Growth Vantaa includes three distinct stages, Growth pacts 1-3, which each provide service and incentive models that focus on different targets: 1) recruiting and educating new employees, 2) updating skills and competences of employees, and 3) digitalization and adoption of new technologies. Each growth pact stage begins with developing and testing new service and incentive models. Next, these models are offered to companies in the treatment group. Finally, the data and experiences are evaluated and used to improve the developed models as well as in designing services and incentives for the next growth pact.

The impact evaluation

The Urban Growth project is making a substantial effort to measure and evaluate the impact of the project. Firstly, all partners take part in collecting and assessing the data on the quality and success of the service and incentive models. Moreover, all stages and activities of the project are analysed critically by an outside consultant. Finally, two project partners, Etla and Labore, evaluate the effects of the models using experimental design that provides credible causal evidence.

This article focuses solely on the impact evaluation carried out by Etla and Labore. This impact evaluation differs from the other evaluation activities of the project in that it utilizes a randomized control design to overcome potential issues present if the participants were chosen merely based on their willingness to participate.

A common challenge of any impact evaluation is, how to design a project that enables learning from it.  How to make sure that treatment is not credited for a normal regional development or results are not inflated by selection? For example, it is very common that the most motivated and better performing firms and individuals participate in available services. Hence, better results of the treated agents may be merely due to this positive selection. In this project, we attempt to overcome these challenges by comparing program participants to a randomly assigned control group.

In practice, companies are randomly divided into two groups: the treatment and control groups. Only the companies in the treatment group are offered the service and incentive models. Due to the randomization, treatment and control groups are on average similar, and thus, comparing the changes in the main outcomes of the groups, provides reliable information on how well the developed models work. Both extensive administrative data and multi-round survey data and are used in the analysis.

The findings

The numerical targets set for participation were clearly achieved. A total of 65 companies took part in the project services, which surpasses the goal set in the project plan by 8 percent. A large share of the participating companies was involved in more than one growth pact, indicating high level of satisfaction among the treated companies. In line with prior literature, growing firms and firms that were making also other investments simultaneously were more likely to participate in the project services.

We do not find any systematic evidence that the Urban Growth Vantaa succeeded in its core targets and improved the growth potential of the firms or the local economy in general. There are some positive signals in particularly with respect to the targets of Growth pact 2. However, there are also many indicators where the development in the treatment firms is worse than in the control firms. The evidence is too ambiguous to draw any definite conclusions. The results are much more cohesive regarding the targets of Growth pact 1. For most of the indicators, the firms in the treatment group are outperformed by the firms in the control group. Based on the impact evaluation, the project did not increase new recruitments. Finally, the evidence regarding Growth pact 3 is mixed.

There are several disadvantages and limitations to our analysis that may prevent the impact evaluation capturing any potential effects. In particular, due to the small number of target firms and treated firms the actual effect of the service and incentive models would have needed to be almost unrealistically large to be visible in the average statistics. The problem is further magnified when using the survey data. Although the survey attracted a decent response rate, unfortunately, it was still too low for reliable inference.

On the other hand, the most reliable and extensive data, namely administrative registers, can take several years to be available for researchers, and thus, we were not able to observe all outcomes for the full project period. This is a substantial drawback. The innovative and developmental nature of the project meant that there was a great deal of learning that took place during the project, and thus, it is very likely that the quality and effectiveness of the project services improved towards the end of the project. Hence, missing this information for the latter stages of the project may lead to significantly underestimating the effects of the project.

Another crucial challenge for the impact evaluation is that companies have long-term strategies for their employment and production choices, and they do not alter these decisions lightly. Achieving changes in these fundamentals is a challenging and a long process. We have attempted to capture early signs of any improvements by including outcome indicators that measure the attitudes of the companies towards increasing the skills and competences of their employees or adopting new technologies. However, here again the data suffers more from the lack of statistical power to draw reliable conclusions.

Due to these challenges, the businesses and the Vantaa region may have experienced improvements, even when this analysis fails to detect them. A major lesson for any future development project must be to take into account also the requirements of an impact evaluation when planning the project. This is the only way to increase our understanding of what works.