Horizontal issues

As mentioned in other sections of this report (Evaluation approaches and Data collection), strong inception and planning of the evaluation prior to starting project activities helps to build the foundations for a successful process. Such planning should also include careful analysis and consideration of any potential vulnerabilities of the communities and individuals designated to provide inputs for data collection.

While it is impossible to have a complete picture of vulnerabilities prior to engaging with groups, exhaustive analysis of potential weaknesses based on the general characteristics is possible during the inception phase and provides a basis for adequate selection and formulation of data collection tools, engagement with people and analysis. Being aware of potential vulnerabilities (in terms of traumatic past experiences and/or current instability and insecurity; language limitations, disabilities or cultural or societal limitations) will help with planning data collection (choosing methods adequate to specific groups, frequency, formulating questions in a sensitive manner, selecting adequately prepared research personnel).

Additionally, such considerations might affect the settings and environment for data collection to ensure subjects feel safe. Vienna CoRE is an example of a project that paid significant attention to adjusting its evaluation approach to the possible vulnerabilities of its target group. Aware that newly arrived refugees and migrants carry significant luggage of difficult experiences, strengthened by the challenges of building lives in culturally and societally distant Austria, Vienna CoRE evaluation team decided to include refugees in the project’s evaluation think tank. Having these refugees help to formulate evaluation questions ensured that questions were culturally sensitive and carefully worded. Additionally, during data collection, the project made sure that social workers and native speakers were available to provide additional support to the refugees if needed. Data collection took place at Vienna CoRE premises that refugees were familiar with.

While potential vulnerabilities of target groups have been rather evident and accounted for in projects targeting refugees and migrants (such as CoRE, Curing the Limbo, CURANT, U-RLP), it is important to consider the potential weak spots of groups that are not so evidently exposed. Since UIA-financed projects aim to solve urgent urban problems, in many instances these will involve individuals and communities struggling with many issues. Some characteristics might clearly indicate possible vulnerabilities and the need for additional considerations during evaluation (such as mental and physical disabilities), whiles others require a more sensitive analysis. To this end, projects aimed at increasing employment (such as Steam City) will inevitably target individuals struggling with unemployment, which might result in an increased sense of insecurity and vulnerability. Here, the formulation of data collection questions may benefit from particular sensitivity so that the failure of some of the project components does not increase the sense of failure of the beneficiaries. Other less obvious sets of vulnerabilities might come from the age group of the respondents (such as working with children and youth). To this end, Paris OASIS also targeted pre-school children with learning disabilities. Including these children in the evaluation posed specific ethical considerations and the need to adjust data collection instruments when necessary.

Many vulnerabilities and sensitivities can be foreseen during the evaluation inception phase and adequate instruments and mitigation measures can be designed. Still, the most experienced evaluators and the most comprehensive designs will inevitably overlook some sensitive aspects concerning evaluation. It is important that once such vulnerabilities are detected, the evaluation team is ready to adjust the course of assessment.

Antwerp CURANT opted to adjust the questions having taken into consideration refugees’ vulnerabilities. The first version of the baseline questionnaire covered topics relating to their mental wellbeing and their resilience, including examining if they were getting panic attacks, feeling lonely or stealing things. As it became clear that youngsters found it difficult to answer questions about these trauma symptoms and family-related questions, the questions were deleted from the baseline survey. It was also feared that the questions may trigger the youngsters to re-live their trauma. In the final questionnaire, the evaluators focused on whether Antwerp CURANT improved – from the perspectives and experiences of the youngsters themselves – their wellbeing, social skills, language skills and knowledge of the Belgian society, so the focus was on the intervention.

Identifying the potential vulnerabilities of various stakeholders at the early stages of designing evaluation can be taken a step further and transformed into an evaluation’s ethical protocol or guidelines. This type of document might limit itself to outlining possible risks and mitigation actions for engagement with vulnerable stakeholders (including recommendations for action during data collection, needs for additional training of data collectors) or it might address a broad range of ethical issues. Not exhaustive, such a list would include possible political and societal implications of the evaluation (for example the ways in which the findings might be interpreted, misinterpreted or politically used-issue – this is especially important when UIA projects tackle politically contentious issues such as the integration of refugees), questions of evaluators’ possible bias and prejudice, or the soundness of collected data for making strong policy claims.

There are multiple possible ethical implications of projects and they differ based on the character of the project activities (whether they target vulnerable groups or include extensive, and possibly intrusive, monitoring instruments such as wearable sensors). Some ethical issues identified in the UIA M&E case studies included a moral basis of not providing services to some beneficiaries (such as youth in Rotterdam BRIDGE) so that counterfactual evaluation could be carried out, or the risk of beneficiaries providing what they consider ‘desirable’ answers in the case of refugees receiving services in Antwerp CURANT. Here, there was a fear that receiving services prevented the youngsters from fully sharing their negative experiences about the project. Researchers noticed a big difference in the responses of youngsters who felt obliged to evaluate the project. When they were asked “How was that for you?”, they always simply replied: “good”. Nevertheless, other youngsters were critical about certain aspects of the project, and the researcher was trained to foster critical thinking in the youngsters and to go deeper into specific topics.

One of the key ethical challenges of evaluations carried out for projects specifically targeting refugees has been the potential for obtaining information to possibly negatively affect individuals’ asylum procedures. Identifying a good range of ethical concerns, even if some of them cannot be easily followed by tangible steps to be taken, can significantly prepare the evaluation team and inform the evaluation design to better address ethical issues as they arise. While such considerations should be part of the evaluation’s inception phase, they will hardly be exhaustive at the early stages. If ethical guidelines or protocol are created, they need to be flexible enough to accommodate any newly discovered challenges, or updated as the project progresses.

On the one hand, the active and meaningful participation of the beneficiaries in the evaluation is essential to its success (understood as elaboration of meaningful data and conclusions). On the other hand, evaluators should be careful not to overburden respondents with too many and overly demanding data collection activities. This may require limitation of the number of questions in the interviews or surveys or frequency of engagements.

From the very beginning, we had this concern about how to approach families during evaluation. It was challenging because the families were from the most vulnerable groups, this also include issues regarding language. There were people who had not been involved in any participatory processes or in the community before. We had this concern how to approach them and not look like they were in a laboratory and we were just watching them. It was equally challenging to design something that would not require too much commitment from the beneficiaries.

Source: B-MINCOME representative

For Utrecht U-RLP the challenge was double as the project targeted asylum seekers – a population exposed to many interviews in the course of the administrative process of asylum, which may make them feel tested as a result.

Ethical considerations do not end with data collection and direct engagement with the stakeholders. Consulted projects revealed that how data is presented through the project’s publicity or in the final report could be equally challenging. In its work, Vienna CoRE managed to actively include a wide range of activist groups engaged in the provision of assistance to newly arrived refugees. The project’s ability to build a constructive relationship between informal volunteer groups and the city’s government is an indisputable achievement. At the same time, since the activist groups have been active members of the project, they too were consulted in the course of the evaluation, not seldom presenting very critical and politically incorrect opinions. The challenge evaluators were facing was that of adequately and genuinely representing these critical voices and doing so in acceptable, constructive manner, not jeopardising the governance structures involved in the project. Utrecht U-RLP was faced with the challenge of considering beneficiaries’ interests and their families’ safety when striving to publicise the project activities. When one of the beneficiaries appeared on Dutch public television, his family back in his country of origin faced repercussions, leading to other beneficiaries becoming more cautious about sharing their stories with the evaluators.

Similar challenges were faced by the evaluators of Brussels CALICO at the reporting stage, when they had to, on the one hand, deliver a balanced representation of all of the project components while, on the other hand, they felt that some aspects of the project produced more important or interesting findings.

The issue of representation has implications in project reporting – when qualitative data methods are used (including in-depth interviews and photographs), a clear policy for protecting the subjects’ privacy needs to be applied. This will safeguard the rights and interests of the stakeholders in the most vulnerable situations (for instance asylum seekers) but also ensure that the dignity of the project participants remains at the core of the evaluation. Evaluators of Vienna CoRE ensured that only data specifically necessary for project evaluation was collected from the beneficiaries and no other information was obtained. In case of Rotterdam BRIDGE, a decision was made not to make use of available data due to privacy-related reasons. While a number of schools work with career guidance portfolios that would contain valuable information for the project implementers, this data was not accessed and anonymised surveys were used as an alternative.

Ensuring privacy can take the form of a written policy regulating who of the data collection and evaluation team members has access to the collected data (preferably limited to the needs-basis and in line with the EU’s GDPR regulation) and how data collection subjects are informed of the evaluation’s purpose and the ways in which their responses will be used. The latter, a matter of informed participation consent (or assent in the case of children), is of particular importance. Ethical evaluations will pay attention to making sure that all the informants are well informed about the purpose of their participation in the evaluation and have a good understanding of the process in which their inputs will be used. Respondents need to know that they may decline participation and this should not impact their ability to access services provided by the project. This is especially important in case of projects offering essential services to the most vulnerable groups (housing or educational services) – beneficiaries need to participate in the evaluation voluntarily and consciously for the findings to have evidence value.

This issue is generally straightforward in classical data collection methods (such as surveys or interviews) but becomes more complicated in evaluations based on an action research approach where any implemented project activity can effectively be considered a source of information for evaluation. As such beneficiaries are observed by evaluators or data collectors (in the role of ethnographers observing activities) or engage in casual conversations which in turn inform the evaluation. Since evaluations based on action research begin with the first project activities and blur the lines between the project and the evaluation, it is worthwhile to present and explain the evaluation component of the project as an integral part so that participants are aware of it.

A review of the evaluation approaches in UIA M&E case studies revealed that a ‘participatory approach’ means different things to different people. On a very general level, it can be understood as participation and involvement of different project partners in the development of the evaluation approach and design. Extensive dialogue and consultation on how the evaluation should be carried out  are key no matter what evaluation governance model is applied. It is equally important when one entity is responsible for the entire evaluation (such as in CURANT where evaluation was managed by CeMIS or in Aveiro STEAM City where it was led by CEDES) and where all the project implementing partners play a significant role in the task (as was the case in Curing the Limbo for instance). On this level, a participatory approach means that different partners have a say in how the evaluation is conceptualised and carried out, sometimes playing a role in data collection and analysis.

In the beginning partners were a bit frustrated when we asked them to collaborate in the evaluation process. Each partner had their own indicators and they thought that all we had to do was to add all these indicators together. Changing this into a shared culture takes time.

Source: Curing the Limbo representative

This type of participatory approach also enhances the coherence of the evaluation and consistency of the approach.

The participation of beneficiaries in the evaluation is another level of inclusive practice which, when done meaningfully, enhances the accuracy and relevance of the final findings. Beneficiaries can be incorporated at different levels of the process, from the design (in CoRE refugees were invited to co-create the instruments), through to data collection (in Helsinki HOPE they conducted data collection with the use of wearable sensors) or contributing to learning loops (in projects based on action research such as Curing the Limbo where beneficiary inputs guided changes in project activities as it evolved).

Supporting a participatory approach to evaluation has a number of advantages, such as increasing ownership of the evaluation results and team building, empowering beneficiaries and improving the relevance of the data collection instruments. The latter is particularly important as involving project beneficiaries in the design or finetuning of data collection instruments enhances the chance that they will truly capture issues and aspects of real life and maintain a high level of sensitivity.

At the same time, an extensive participatory approach is very time and resource-consuming, it may require skilful facilitation of confronting visions and approaches, and generally evolves better under clear leadership. What is important that beneficiaries are not only included in the data collection, but also have a say in how their voices are interpreted and represented.

Lastly, a participatory approach should always be applied with a clear purpose in mind, rather than as a self-serving exercise. The evaluation teams consulted showed great appreciation of a participatory approach but also awareness of the limitations of its applicability within the framework of UIA projects.

Good rapport that establishes open and transparent communication with the respondents, based on mutual respect and trust, is both highly desirable and difficult. This includes ensuring that beneficiaries are familiar with the evaluators who are present during at least part of the project’s activities (as seen in CALICO) or having empathy towards respondents’ personal circumstances (as seen in Vienna CoRE where childcare was provided for mothers willing to participate in the project and data collection).

The authors of the concept for the CoRE project emphasised the project’s ‘needs-based’ character in the proposal multiple times because it was their priority. Consequently, the evaluation also focused on the project’s relevance, i.e. whether the activities were in fact meeting the needs of the target group, especially women. From the beginning, the results showed that the project struggled to attract female asylum seekers due to shortages of childcare. So the project staff made use of the M&E results and implemented a childcare facility at the CoRE centre.

UIA projects are embedded in the political and administrative milieus of the city authorities, often tackling politically and socially contentious topics (such as migration or exclusion) and targeting sensitive groups (such as children). These two elements might have implications for the evaluation set up.

Firstly, being embedded in governance and administrative units (such a city halls) might result in a need to adjust M&E practice to fit the requirements of dominant public policy evidence based approaches. These, generally speaking, are seldom characterised by flexibility and understanding of the ways in which innovation works. For instance, while Athens Curing the Limbo evaluation approach was characterised by a strong understanding of the limitations of measuring impact in the context of an innovative social project (and focus was placed on qualitative data), the evaluation team also made sure to keep the city administration updated on the project milestones and to provide regular information on progress.

Consortium partners, like the city, they sometimes think that all the data that is available should be used. But privacy issues for instance will prohibit us from using all the data. One of the key factors of success is accepting this. We as a city will not get a perfect evaluation, the evaluator will not get a perfect scientific data.

Vienna CoRE reflects that staying faithful to what it thought was useful to the project in terms of evaluation effort rather than trying to satisfy external stakeholders proved beneficial in the long term. Now that the project is over, it is left with a wealth of useful policy analysis to shape future interventions. Antwerp CURANT experiences indicate a challenge of managing the expectations of political stakeholders as to what data can actually be gathered throughout the evaluation and what analysis can be proposed on that basis.

UIA project evaluations also had to account for practical constraints to access to data and stakeholders prompted by the broader political and governance structures. In Rotterdam BRIDGE, the original idea was to use an experimental design whereby some schools from Rotterdam South would participate in the project and others would not. This was a solid way to measure effects, but it would also mean denying valuable support to some students, creating a politically and ethically challenging solution. Therefore, it was decided that all students in all schools had access to the interventions of the project and a non-experimental design was applied instead. It included combining extensive work with available statistical data comparing trends for Rotterdam South with other regions in the Netherlands, and surveys among students and companies, both participating and not participating in interventions. The evaluators in BRIDGE also had to tackle the fact that politicians opposed the idea of interviewing pupils in primary schools as part of data collection. The project had to defend their plans, proving it had an adequate methodology for engaging with the children.

In Utrecht U-RLP, the application of an experimental design was also hindered by the decision of the Dutch asylum authority (COA) to deny the evaluation team access to asylum seekers residing in the centre, unless they participated in the Utrecht U-RLP project. This meant that the evaluation team did not have access to the asylum seekers from the Overvecht centre that decided not to enter the project. Consequently, the sample of asylum seekers was biased towards those receiving support and the opportunities for comparison with those who did not receive it were limited.