Edit 04 March 2021
Monitoring and evaluation practices: UIA lessons learnt
Report homepage

Vienna CORE


Integration of migrants and refugees
EUR 4,786,272
01/11/2016 – 31/10/2019
Learn more about this project

About CORE

kids raising hands

Centre of Refugee Empowerment (CORE) was a project implemented between 2016 and 2019, which set out to develop new approaches and solutions to address the integration challenges for asylum seekers and refugees. The project aimed to facilitate the integration of beneficiaries in Vienna by providing integrated support in three main aspects:

  • A participatory approach: Pooled together the resources and know-how of public institutions,  NGOs, civil society initiatives/volunteers, worked with refugees and beneficiaries themselves, planning and implementing the project alongside them. The project team aimed to develop a bottom-up approach, where refugees are seen as equal partners instead of passive beneficiaries, and which (together with the rest of the project team) could initiate a smart transformation process for the whole integration system.

  • A ‘hub’: Putting the CORE centre at the disposal of the beneficiaries and project partners, a place which acted as a community and service space (including the space where activities were implemented).

  • Set up a think tank to monitor and analyse activities, as well as test new solutions.

The need for this project arose in 2015, when Austria received almost 90,000 applications from asylum seekers, one of the highest numbers in Europe correlated with the country’s population size. The vast majority of those who received refugee status settled in Vienna; in 2015, the population of the city grew by 43,000, 23,000 more than the average increase of 20,000 that the city previously experienced. This sudden increase put considerable stress on the social welfare and social housing systems, as well as on the public authorities in general, who could not by themselves ensure the integration of such a high number of refugees.

Within this context, the CORE project was born as a new model of cooperation between the public and private sector. Aiming for ‘integration from day one’, the project was developed as a participative, needs-based initiative, which aimed to reach and empower a high number of beneficiaries, whilst keeping the project response flexible at the same time.

CORE offered close to 20 different activities. These included information modules offered from the first days of arrival, a competencies assessment, support for doctors to allow their faster access to the labour market and practice in Austria, entrepreneurship training, qualifications training, a certification course for teachers and peer mentoring (refugees4refugees).

At the time of writing of the evaluation report (2019), 15 of the implemented activities were set to continue after the end of the project.

Evaluation governance

The project was implemented by the City of Vienna as lead partner, together with the support of four other partners: the Vienna Social Fund, the Vienna Employment Promotion Fund, the Vienna Business Agency and the Board of Education for Vienna – EuropeanOffice.

During the implementation of the project, three levels of management were foreseen:

  • Operational support with the day-to-day management and technical coordination of the project;
  • Coordination and joint decision making: A steering group, made up of representatives from the urban authority and all delivery partners, which met on a regular basis;
  • Central coordination and consultation: A body was set up in order to address all strategic issues. The strategic consultation body was made up of refugees, representatives from refugee initiatives, associated stakeholders, NGOs and civil society initiatives, and was meeting regularly throughout implementation of the project.

An interesting aspect of the evaluation exercise was the fluidity of the roles of the stakeholders. The asylum seekers and refugees were subjects of the research exercise, but at the same time co-designers of the methodology. On the other hand, project partners were implementing partners, but research subjects at the same time as well. Such fluidity of roles was made functional through strong collaboration and coordination ties between project partners, the external consultant and the refugees, as well as due to the existence of a strong Steering Committee.

Due to the participatory design of the monitoring and evaluation, a learning culture (and thus learning loops) was embedded in the monitoring and evaluation activity. The monitoring and evaluation methodology allowed the project partners to quickly adapt and change the interventions or adapt their format, depending on the results of the evaluation. Initially the info modules were given in the format of lectures, for example, in order to reach multiple beneficiaries at a time. When the monitoring and evaluation results showed that, after a time, the difficulties faced by beneficiaries started to become more personalised, the format was changed. When the analysis showed that women were unable to attend language training due to childcare responsibilities, the project started to offer childcare at the centre, in order to allow them to participate. The topics of the entrepreneurship classes were defined depending on the interests of the beneficiaries, while also taking into account labour market demand. This reflexivity and adaptability had its limitations, however. Offering support with finding a job was one of these limitations, as it went beyond the scope of the project.

Evaluation process

The partners intensely reflected upon the evaluation approach. A theory-driven evaluation was considered, but the idea was abandoned, as it was believed that it would impose limitations. A longitudinal study was considered as well, but this approach was not adopted either, since the project team preferred to focus on the activities instead.

After careful consideration, it was decided to design and implement a participatory research methodology. This choice was a good approach for the project, as it allowed for the flexibility needed to feed the results of the evaluation into the activities and constantly adapt them in order to best respond to the needs of the beneficiaries. 

A think tank was set up, including a dedicated work package, with the role of monitoring and analysing the project activities. The monitoring and evaluation activities were managed by an external contractor, Urban Innovation Vienna (UIV), which was supported by the project management and partners. The monitoring and evaluation design was proposed by UIV after consultations with the project partners, and was based on the special nature of the activities. It was proposed as a realistic design to implement, taking into consideration the high number of beneficiaries and wide array of activities. It was also meant to be constructed in such a way as to look more at the process, rather than the outputs.

The core team of the think tank was composed of external consultants, representatives of the partners, and also of refugees who had settled in Vienna for years and already spoke German. They participated in the co-creation of the evaluation design. The role of the refugees was to transpose the participatory and bottom-up approach of the project to the evaluation itself. In this way, they brought the perspective of a beneficiary to the design and provided advice regarding what questions should not be asked to avoid harming the asylum seekers, as well as how some questions should be phrased (language issues, connotations, etc.). The selection of the refugees was done through an assessment centre and, besides their qualifications and experience, the gender criterion was also taken into consideration.

The think tank was – among other things – responsible for the preparation of 3 SWOT reports and the evaluation report. A SWOT analysis and workshop was done for every single activity; the results were used to change the focus of the activities or make minor adjustments in order to best address the changing needs of the beneficiaries. The data collection (through SWOT workshops and interviews) was done by UIV, with the support of the project partners and the refugees, who prepared the list of interviewees and the list of questions. Besides beneficiaries, project partners and practitioners (social workers, activities’ providers) were interviewed as well. The external contractor also conducted observations of different activities and had its own viewpoint of what was happening.

This data was complemented by data collected by the social workers and those responsible for implementing activities (trainers, workshop facilitators, etc.), which was collected continuously throughout implementation of the project.

Since the project was needs-based, the research questions of the evaluation exercise were very much centred around them:

  • What are the needs of the asylum seekers/refugees?
  • Do we meet their needs?
  • Can we do better?
Theory-based approach

Theory-based evaluation has at its core two vital components. Conceptually, theory-based evaluations articulate a policy, programme or project theory, i.e. how activities are supposed to lead to results and impact, given specific assumptions and risks. Empirically, they seek to test this theory, to investigate whether, why or how interventions cause intended or observed results. Testing the theories can be done on the basis of existing or new data, both quantitative (experimental and non-experimental) and qualitative.

Source: European Commission, Evalsed Sourcebook – Method and techniques, 2013. For more information, visit e.g. Better Evaluation website or the website of the Treasury Board of Canada Secretariat.

Participatory approach

Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study.

Source: Better evaluation

“Participatory research is giving power from the researcher to research participants. The research participants and researchers together control the research agenda, process and actions.”

Source: Dawance, T., Smetcoren, A., Ryckewaert, M.,  Aernouts, N.,  De Donder, L.,  Care and Living in Community, CALICO. Groundwork for evaluation and state-of-play, 2019.

In order to collect the necessary data, the project team used a mixed-methods approach, including qualitative (i.e. interviews, feedback groups, journaling, observations, etc.) and quantitative (i.e. survey) methods. This broad array of methods was used in order to address the needs of every beneficiary sub-group. Written feedback worked for the doctors that they supported, but they needed a different method for the women who could not read or write (written and oral methods, structured and unstructured depending on the activity).

The interviews were conducted at the CORE centre, to create a safety net for the beneficiaries and to provide support through interpreters and social workers, if needed.

When you’re working with this target group [refugees], even questions [that] you would never think that they could trigger emotion, might cause an emotional reaction. You might ask a question [that] you think that it’s totally not dangerous at all and at some point it triggers something in the interviewee. So that’s why [we were doing] the interviews at the CORE centre, where we had social workers around, where we had native speakers around […] Because with this target group you never know what happens.

Source: CORE project hearing

While interviews, observations and surveys are methods commonly encountered in other projects financed by the UIA initiative, journaling is a less frequently used method. In the case of this project, it was used by social workers in order to make sure that the information gathered during daily social situations was used as well. This was possible thanks to the structure of the CORE centre, which, due to its size, would host the relevant stakeholders and activities (including interviews), thus increasing the interaction between the beneficiaries themselves, as well as between the beneficiaries and the stakeholders. Journaling was not only used by the social workers, but also by the activity providers (such as workshop facilitators) who had to learn how to recognise valuable data for the evaluation. Initially, they started sharing the relevant observations in a more informal way, but with time it became a more formalised practice during the project meetings.

The ad-hoc groups of beneficiaries proved to be an unexpected source of information. These were groups that gathered in the CORE centre rooms in order to practice a common activity they were interested in (woodwork, knitting etc.). The informal talks between the participants (who usually had the same language background) touched upon sensitive issues that would not be captured through other methods of data collection. This new information could then be brought immediately to the attention of the social worker or relevant stakeholder, as they were all hosted in the same premises, and used to make minor modifications to the intervention, so that it better benefited the target group (e.g. introducing a new workshop on a specific topic of interest).

Some of the most important data that we collected during the CORE project was data that was not meant to be collected at the beginning. So, right at the beginning we realised that there is a lot of valuable data that we haven’t thought of. Like all [the] chit-chatting that was going on between the participants of a sewing group […] We saw the possibilities that lie in this data […] The social workers decided to take down notes after the activities. […] We took the input we got from those notes to do some modifications in the activities.

Source: CORE project hearing

The micro-perspective of the monitoring and evaluation activities was complemented by the macro-perspective related to international best practices. Besides the interviews and workshops, the think tank broadened the project’s perspective through international benchmarking. The project partners identified 11 benchmark cities that had experience in the field of integrating refugees and whose integration policy was relevant for Vienna. Interviews were also conducted with national and international experts, with a focus on volunteering, as well as mentoring and the buddy system.

Mixed-methods approach

Mixed-methods approach can be defined as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry”.

Source: Tashakkori, A., Creswell, J.W., “Editorial: the new era of mixed methods”, J Mixed Methods Res 1: 3–7, 2007.

Horizontal issues

Over the three years of implementation, the project faced several challenges. Some were due to the changing political and public support for the project, others to ethical or everyday implementation considerations.

When the government changed in 2017, moving from a central-left to an almost far right coalition, the project partners had issues with implementation of the apprenticeship intervention because the legal opportunity for asylum seekers to undertake an apprenticeship was abolished. After consultations between the partners, it was decided that this intervention would be replaced. The initial favourable public opinion towards asylum seekers/refugees also changed.

When it comes to ethical considerations, the project adopted a ‘privacy by design’ approach. If personal data was not needed (name, age, gender) then it was not required (or it was presented in a coded format). If sensitive questions were intended to be asked just because it would be good to have the information, but not necessary, then it was preferred to drop such questions.

 What was really important for us working with such a vulnerable group was that we only collect data that was needed as part of the evaluation process. So, we decided that with a target group that always fears that all kind of information given could be used against them [in the asylum process], we would focus only on the information needed.

Source: CORE project hearing

In order to ensure the privacy of the attendees and encourage their participation, the indicator for information modules for refugees (or other low-threshold activities) consisted of the number of participations, not the number of participants (no data collected on the participants).

The researchers also faced initial mistrust on the behalf of the beneficiaries with regards to sharing their honest opinion during the interviews, surveys and feedback opportunities, as they were afraid that this would harm their asylum procedure.

Another possible challenge was the fact that centralised data on refugees is not available in Austria. There is data on asylum seekers, but once they are granted asylum or subsidiary protection they are on the same footing as everyone else residing in Vienna. Without this data, hard-to-reach groups, such as women with a low education and no knowledge of German, are difficult to reach. However, through the ad-hoc groups hosted at the CORE centre, the project team has managed to reach out to many of them.

Lessons learnt

The CORE project is a good example of a needs-based participatory approach, both at the level of activities and at the level of evaluation. Its approach seems particularly transferable to bottom-up projects, where one of the goals is the empowerment of the vulnerable group it is trying to reach.

At governance level, the good results of such an approach depend on:

  • The existence of a strong steering group, where all partners participate at the same level, irrespective of their share of the activities/budget;
  • Building links with the local government through ongoing exchange and updates;
  • A strong network of partners for the areas that they manage (entrepreneurship, education system, labour market etc.).

Some other valuable lessons learnt are presented below:

  • The evaluation methodology should be aligned with project goals: At the design stage of the project, it is important to have a clear objective and choose an evaluation methodology that is beneficial to the project. If the goal is to have a needs-based project, like in the case of CORE, then a participatory research method might be a good choice.
  • There should be space for flexibility in data collection: At the design stage of the project, researchers may not be fully aware of all important sources of information that might appear during the implementation of the project. Once new sources of data are identified (such as the informal conversations between the beneficiaries during ad-hoc workshops) it is important to address them and decide which is the best method to collect and use the new information, as well as how to integrate it into the evaluation design.
  • Involvement of the beneficiaries in the design of the evaluation can bring positive results, in addition to adding to their empowerment: When working with vulnerable groups, involving representatives of the beneficiaries in the evaluation can ensure an extra layer of protection for the wellbeing of the group, by indicating limits to avoid (re)traumatisation.
  • It may be beneficial to consider creating a physical safe space for the target group in which to conduct monitoring and evaluation activities, and where representatives of the target group can be supported by social workers and interpreters at a moment’s notice, if necessary.
  • At the structural level, having the project team and activities hosted in the same ‘hub’, the  Centre, has created the opportunity for increased social interaction and support, as well as better data collection (e.g. journaling).
  • Since staff turned out to be such a valuable source of information through their observations, it might be considered beneficial to train and prepare them in how to recognise valuable information and how to react when specific issues arise.
  • The challenge of the evaluation exercise was to isolate impact. According to one of the team members, a possible way to mitigate this might be to break it down into single activities and assess the development of the beneficiaries from before they joined the activity to after (i.e. if before joining the project the beneficiary did not know how the school system worked in Vienna, afterwards, he/she had an understanding).


Integration of migrants and refugees
EUR 4,786,272
01/11/2016 – 31/10/2019
Learn more about this project