As mentioned in other sections of this report (Evaluation approaches and Data collection), strong inception and planning of the evaluation prior to starting project activities helps to build the foundations for a successful process. Such planning should also include careful analysis and consideration of any potential vulnerabilities of the communities and individuals designated to provide inputs for data collection.
While it is impossible to have a complete picture of vulnerabilities prior to engaging with groups, exhaustive analysis of potential weaknesses based on the general characteristics is possible during the inception phase and provides a basis for adequate selection and formulation of data collection tools, engagement with people and analysis. Being aware of potential vulnerabilities (in terms of traumatic past experiences and/or current instability and insecurity; language limitations, disabilities or cultural or societal limitations) will help with planning data collection (choosing methods adequate to specific groups, frequency, formulating questions in a sensitive manner, selecting adequately prepared research personnel).
Additionally, such considerations might affect the settings and environment for data collection to ensure subjects feel safe. Vienna CoRE is an example of a project that paid significant attention to adjusting its evaluation approach to the possible vulnerabilities of its target group. Aware that newly arrived refugees and migrants carry significant luggage of difficult experiences, strengthened by the challenges of building lives in culturally and societally distant Austria, Vienna CoRE evaluation team decided to include refugees in the project’s evaluation think tank. Having these refugees help to formulate evaluation questions ensured that questions were culturally sensitive and carefully worded. Additionally, during data collection, the project made sure that social workers and native speakers were available to provide additional support to the refugees if needed. Data collection took place at Vienna CoRE premises that refugees were familiar with.
When you’re working with refugees, even questions you would never think could trigger emotion, might cause an emotional reaction. You might ask a question you think is not dangerous at all, and at some point, it triggers something in the interviewee. That’s why we were doing the interviews at the CoRE centre where we had social workers [and] native speakers around.
Source: CoRE representative
While potential vulnerabilities of target groups have been rather evident and accounted for in projects targeting refugees and migrants (such as CoRE, Curing the Limbo, CURANT, U-RLP), it is important to consider the potential weak spots of groups that are not so evidently exposed. Since UIA-financed projects aim to solve urgent urban problems, in many instances these will involve individuals and communities struggling with many issues. Some characteristics might clearly indicate possible vulnerabilities and the need for additional considerations during evaluation (such as mental and physical disabilities), whiles others require a more sensitive analysis. To this end, projects aimed at increasing employment (such as Steam City) will inevitably target individuals struggling with unemployment, which might result in an increased sense of insecurity and vulnerability. Here, the formulation of data collection questions may benefit from particular sensitivity so that the failure of some of the project components does not increase the sense of failure of the beneficiaries. Other less obvious sets of vulnerabilities might come from the age group of the respondents (such as working with children and youth). To this end, Paris OASIS also targeted pre-school children with learning disabilities. Including these children in the evaluation posed specific ethical considerations and the need to adjust data collection instruments when necessary.
Some teachers raised the issue that a child with a disability could experience cognitive or language difficulties during data collection. In such cases, the interviewer negotiated with the teacher either a possibility not to invite a child to the interview or to have a simplified interview adapted to the child’s capacities, to make sure the child wouldn’t feel excluded.
Source: OASIS representative