Rotterdam BRIDGE

The project was implemented by the City of Rotterdam as the Main Urban Authority, supported by the Metropolitan Region Rotterdam - The Hague, Stichting Economisch Onderzoek Rotterdam (SEOR), the Rotterdam University of Applied Science and the RebelGroup Executives BV. SEOR, an independent research organisation linked to the Erasmus School of Economics, was the main partner responsible for monitoring and evaluation. Some decisions regarding the monitoring and evaluation design were made by the project Steering Committee, composed of two directors from the city of Rotterdam and one representative from each of the project partners.

At the design stage of the project, SEOR was not envisaged as an evaluation partner. The original idea for the BRIDGE project was to gather an evidence base for long-term continuation, with public or private funding – a sort of social impact bond scheme. Within this approach, a consultancy experienced with working on social bond projects was considered as an evaluation partner. However, the idea of a social bond scheme was abandoned at the writing stage of the project, due to the lack of opportunity to gather in-depth data (better understanding of the contributing factors in the south of the city of Rotterdam).

Once the idea of the project changed, the partnership changed as well, and an evaluating partner with a more scientific background was sought. The evaluation was assigned to SEOR and planned as a distinct project component. This was managed by SEOR together with the City of Rotterdam. Data collected through this work package was intended to be enriched by qualitative data from the NPRZ programme, collected in a different project component, dealing with the implementation of the Career Start Guarantee.

The change of the evaluation framework put extra pressure on the evaluation partner, who had to be present at every project meeting and explain to the rest of the partnership what the effects of the interventions were in terms of the broader picture. The evaluators became more central to the project than it was intended at the proposal stage, and their presence in project meetings also facilitated communication between partners; it gave SEOR the opportunity to discuss information gaps and engage the other partners in collecting information and reaching out to relevant stakeholders, such as schools.

The project adopted a participatory approach to evaluation where the partners were involved in the design of some parts of the evaluation and the Steering Committee decided on the level of application for the interventions. The involvement of both teachers and parents proved to be a valuable choice, as they are important for the career guidance of the pupils. However, it could be argued that an increased role in the design of the evaluation itself would benefit the evaluation team, providing valuable insights.

The evaluation team had to face several challenges, leading to the need to adapt to several deviations from the final intervention plan due to practical reasons. In one case, the team had to renounce the idea of using personal portfolios for the pupils involved (longitudinal intervention) because this would have meant obtaining permission from each participating pupil, which (due to the size of the project) was problematic. This would have been a complex situation to handle, since there is no one system used in every school and, as a result, the format of the data would have varied. Using portfolios would be an option better suited to countries where the education system is more centralised and the systems align. The next-best solution was to implement surveys among pupils.

The methodology for the evaluation exercise was an intensely discussed part of the project and underwent several changes over time, due to political, ethical and practical considerations, as well as data gaps. Importantly, however, the model that was eventually developed satisfied the partners.

The initial intention was to use an experimental design, with a treatment and a control group – some schools would participate in the project and receive the interventions, while others would not. This idea was abandoned within a few weeks of the start of the implementation, due to ethical and political reasons. The Steering Committee considered it was politically unacceptable to have the control and treatment groups within the same part of the city. Instead, it was decided that the interventions would be applied to all participating schools. 

Having to rely on a different approach, the evaluation team designed an evaluation framework, which was based on data gathered during the implementation of the project, but also on previously existing data. This approach was chosen as a way to mitigate the limitations and technical challenges that the team encountered in assessing the project’s impact. BRIDGE is a good example of a non-experimental approach in which microdata is used to build a baseline for the pupils receiving the treatment.

The main research questions that were addressed in this evaluation exercise were:

• If BRIDGE would increase pupils’ participation in interventions
• To what extent this participation would lead the pupils to choose a high-demand specialisation in the areas of technology, harbours and healthcare;
• If this choice of high-demand specialisation led to better prospects on the labour market.

In order to find an answer for these questions, the evaluation team employed three methods, each with its own strengths and weaknesses, in an effort to mitigate information gaps. The impact of the project was to be evaluated by assessing the direction in which the results of the three methods pointed. The team opted for a mixed-methods approach, in which they were using both qualitative and quantitative data collection methods.

1. First method: Comparing the education and labour market outcomes of pupils who received the treatment and pupils who did not. The sources for comparing participants and non-participants:

• Having an aggregated look at the development of educational choices in Rotterdam South versus other regions (data from the Dutch Central Bureau of Statistics – CBS);
• Looking specifically at the development of intake in training with Career Start Guarantee in comparison with other regions;
• Surveys among pupils (some pupils were participants in the interventions, others not);
• Information about participation in the interventions at the school level (per grade), linked to the pupils of the schools that were included in the microdata (more of a pilot analysis due to data limitations).

2. Second method: Comparing educational choices from Rotterdam South with those in Rotterdam North and other major cities in the Netherlands, in order to assess if these develop into choosing a specialisation high in demand in the technology, port-related and care sectors in secondary and secondary vocational education.

3. Third method: Assessing perceptions of the different target groups involved, such as pupils, teachers, parents, employers/companies and agencies responsible for the implementation of the intervention. This data was collected through surveys, interviews and group discussions. For this part of the evaluation, the approach was inspired by theory of change (ToC). The goal of using ToC was not only to assess if an intervention has the desired effect, but also to identify the mechanism that made reaching this effect possible, by asking how the policy intervention is supposed to work.

What is the policy problem? Which mechanisms are responsible for it, which interventions have been chosen to encourage transformation mechanisms for a more desirable behaviour, and what are the limiting and facilitation factors for the desired outcomes?

The most significant horizontal challenge encountered by the project was political and was related to the fact that the counterfactual approach was considered unacceptable. This forced the project partners and evaluator to be flexible and resourceful in their evaluation approach, finding alternative options which, between them, would manage to capture the impact of the project as close as it can to a counterfactual evaluation.  

However, further issues soon arose with the new methods employed; more specifically, with access to the pupils’ career guidance portfolios. At the start of the project, the evaluators wanted to obtain pupils’ career guidance portfolios from schools, which contained rich information on the beneficiaries. This was not possible due to privacy issues. As an alternative, the evaluators implemented anonymised surveys among pupils and students from different education levels, from primary school to upper secondary vocational education.

The sustainability of the project was an important part in the development of the intervention. The evaluation exercise managed to prove that BRIDGE has an impact on policy and that it has a potential for a high social value return for its cost. The policy impact is highlighted by the increase in the range of interventions in recent years, including the introduction of new ones. The conducted cost-benefit analysis showed that even a low number of pupils (approximately 20 pupils a year) changing their career orientation in the expected direction is enough to compensate for the cost of BRIDGE.

There was a cost-benefit analysis – it started as a special section in our final report; ‘What would the effects be if [the young people in vocational schools] made a change and started their specialisation as intended by the BRIDGE project?’ – using microdata on what affects choices on certain specialisations [and] later market careers, in terms of chances on the job, wage levels, etc. We did it both from an individual perspective and societal perspective, so we are using gross wages […] It shows that only a small change would be enough to compensate the costs of BRIDGE; that was the outcome.

Source: Rotterdam project hearing

However, due to the complexity of the intervention, methodological limitations and the limited timeframe for implementation, it was difficult to discern its effects. It also cannot be ruled out that the effects are actually small. In addition, the surveys and interviews showed little confirmation of an effect on choice of educational direction towards the referred sectors (the interviews with pupils and teachers did confirm the perception that the interventions facilitate the selection process, but do not directly contribute to the choice for education in the preferred sectors).

The approach of BRIDGE represents a comprehensive attempt to evaluate the project’s impact and processes, including carrying out a cost-benefit analysis. The partners were purposeful in selecting their approach.

The evaluation design has potential to be transferred to other projects in the field of social innovation, specifically projects with a large target group which would like to adopt a counterfactual evaluation but are prohibited for various reasons. A baseline can be created through the use of microdata, however this would require the availability of a database with microdata at city-level.

Throughout the implementation of the project, several shortcomings were identified by the evaluation team, which in the end represented lessons learnt. Some valuable learning points and specific recommendations are presented below:

1. A counterfactual approach can be sensitive from ethical and political points of view. If possible, it is worth checking the position of all relevant stakeholders at the writing stage of the project, including the local decision-makers.
2. Openness to amending the evaluation framework may be necessary and beneficial. This might trigger negative results (in this case, losing the control group meant lower insight into the impact of the project), but also positive results (such as surprising insights you have due to the utilisation of other research methods). In the case of BRIDGE, partners were satisfied with their final approach.
3. To the extent possible, data availability and data structure should be verified beforehand. If faced with an unexpected situation where data is not available due to data gaps or privacy issues, be flexible and adopt alternative measures. Related to this specific project, the following recommendations were made by the project evaluators:

• The quality of data at school-level needs to be improved. In the best-case scenario, individual data on participation in interventions could be linked to the Central Bureau of Statistics’ (CBS) microdata. This would provide opportunities for testing the relationship between participation in interventions, school career (dropout rate, progression of education direction and level) and position on the labour market. A less precise variant would be if policy at the school level (preferably per grade) is linked to the individual pupils of the school, in order to determine whether this policy (use of interventions) has an effect on their school careers. However, this is less precise because, for a range of interventions, some pupils will participate and others will not.
• The quality of data at the individual level needs to be improved. This could be achieved through the systematic monitoring of pupils’ participation in interventions and the choices they make at important educational moments. The availability of this data at the individual level would allow for more accurate measurement of effect, which in turn would determine how effective the instruments are.
• The evaluation results confirmed the importance of having teachers and parents involved as target groups, due to their support in career guidance. The evaluation results also promote supporting teachers in this role and even higher involvement of parents.

• Benefits of longitudinal data: The project would have benefited from the existence of a ‘panel’ of young people, which would be followed through repeating surveys, in order to improve the information and opportunities for data analysis. This panel would be an alternative to the option of linking the participation in interventions at an individual level with CBS microdata.