Athens Curing the Limbo

The evaluation of Curing the Limbo was carried out by the five implementing partners: the Municipality of Athens, Athens Development and Destination Management Agency S.A. (ADDMA), the National and Kapodistrian University of Athens (UoA), the Catholic Relief Services (CRS) and the International Rescue Committee (IRC Hellas).

Each entity was responsible for collecting and analysing data pertaining to the activities they implemented. Consequently, for some time, implementing partners were carrying monitoring and evaluation of their respective components and services, each in accordance with their own beneficiary data base. Such approach did not allow partners to capture the impact of the project on the integration of the refugees. The latter required integration of the evaluation processes across the different project components and different project partners.

While initially there was no clear leadership of the evaluation among the partners, this role was eventually taken up by the quality assurance team. Originally tasked with ensuring the quality of project deliverables, activities and risk management, the team took it upon themselves to integrate and standardise the evaluation approaches employed by the five cooperating partners. This change was motivated by several factors, as follows.

Firstly, it was established that the project lacked a clearly developed, overarching evaluation approach. Rather, each implementing partner was focused on collecting and analysing data specific only to their own line of activities, in response to indicators relevant to their segment of the project. As such, the project lacked an integrative evaluation framework where all data collected by various partners could be assimilated and analysed, giving insight into the project’s impact in its totality. The quality assurance team took the lead in developing such a framework through guiding a process of collaboration and consultation.

Secondly, all five of the partners in the project brought significant expertise in evaluation, each having their own experiences, methodological approaches and understanding of the process. While such a rich combined expertise was definitely an advantage, it also created divergences in the interpretation of certain methods or theoretical approaches among evaluation actors. Here, the quality assurance team played an important role in arriving at shared definitions and harmonised approaches to data collection and analysis.

The process of developing a harmonised, integrated approach to evaluation was conducted in a consultative manner, building on extensive discussions among the partners. This strong collaborative process was both an asset and a great challenge. While the quality assurance team supported integration, arriving at a consensus was time and energy consuming. On the one hand, such discussions were highly stimulating intellectually, enhancing the quality of the final results and leading to the development of a strong learning culture among the implementing partners. On the other hand, at times, partners felt tired by the process; when delays hindered the collection of adequate baseline data, for instance. When reflecting on the process, the evaluation team wished they had clearly defined an evaluation leader from the project’s outset.

The fact that the evaluation was carried out directly by the project implementing partners and in a close, frequent dialogue allowed for efficient and effective integration of learning loops into project activities. The baseline survey, for instance, revealed that the initially planned six months of housing support for refugees should be extended to one year, and this was consequently applied. Similarly, close monitoring of community activities showed that, in order for this element to be successful, more empowering work needed to be done in the first instance. As a result, the project component was prolonged to ensure meaningful implementation. Such adjustments were possible both due to the governance model and the choice of action research as the evaluation’s governing principle. At the late stage of the project (November 2020), an external collaborator National Centre of Social Research (EKKE) was contracted to provide expertise and contribute to drafting the final evaluation report. EKKE will extensively utilise the collected data and will further conduct semi-structured interviews and focus group discussions.

 The project’s overall approach to evaluation was based on the action research model for refugee social inclusion. The approach was very experimental in nature and embedded in project implementation, meaning that activities were introduced to the project, reviewed and adjusted based on the data collected. In essence, the focus was placed on a continuous development loop and real time change. This approach offered a strong participatory aspect, whereby the feedback from beneficiaries and a broader group of stakeholders could be accounted for. This, in turn, helped to adapt and adjust implementation during its course, resulting in a circular design which continuously assessed if new services were effective and useful for beneficiaries. Provision of safe housing became the first step towards further support, for instance, as it was established that safety of abode was the precondition for any integration and utilisation of other services. At every stage, attention was paid to feedback from beneficiaries, but also from the broader spectrum of project stakeholders, such as landlords and teachers. Using action research meant that beneficiaries had a meaningful say in how services were shaped, in terms of topics and scope. 

The action research paradigm proved adequate in Curing the Limbo’s evaluation for two key reasons. Firstly, it promoted the participation and meaningful engagement of beneficiaries, which was in line with the project’s overall aim of empowerment. Secondly, the project targeted a dynamic, heterogenous group and evolved in an unstable socio-political environment. Action research supported the real time adjustment of activities and immediate capturing of the partial effects of actions.

The partners started to measure progress towards common goals well into the project’s duration. The project proposal did not elaborate on the common evaluation approach, albeit foreseeing the development of an ‘Evaluation and Monitoring Handbook’. The project did not benefit from a partner tasked exclusively with evaluation nor a blueprint on how to evaluate, but such a role was taken on by the quality assurance team. When the latter occurred, the evaluation approach was developed alongside the activities. It was first harmonised in March 2019 in the draft ‘Evaluation and Monitoring Handbook’, which was further updated. The document explained the overall evaluation approach, but also included indicators collectively developed through two co-creation workshops, which involved all the evaluation partners and was led by the quality assurance team. The indicators, outputs and results measured were aligned with the project’s overall objectives. All partners agreed on a pattern of reporting regarding project evaluation. A questionnaire regarding beneficiaries was developed, and a pilot study informed the final version. The harmonised and standardised approach greatly facilitated further data collection and analysis, creating a basis for tackling the project’s key research questions.

Curing the Limbo’s evaluation approach displayed a nuanced and critical examination of what the key research questions and lines of inquiry should be. The project was built around a notion of ‘exchange’ where beneficiaries were given support by the city and, in turn, expected to give back. This definition, and consequently the ability to evaluate the project’s impact, was verified during implementation, however. This was closely linked to the notion of empowerment and integration and the bigger evaluative question of how it can be measured.

We had this lofty definition that this is going to be an exchange programme, where the refugees will give back to the city. When we initially described this, it looked like a very good idea, but when we had to really pin down what will they give back, what are we giving them and what are we expecting from them, this became a big question that we grappled with along the implementation.

Source: Curing the Limbo project hearing

The project’s evaluation approach additionally offers a sophisticated understanding of power relations in communities and breaks down the methodological question of what empowerment means and how to evaluate it. This was specifically analysed in the case of the joint projects developed by collectives made up of Athenians and refugees. The evaluation looked not only at the products of such initiatives, but also at the genuine relations within the groups producing them, analysing who actually has a say in how they are shaped and how democratic and inclusive the processes are.

The project team constantly faced the question of how much can be assessed within such a short time. To this end, the development of the evaluation framework alongside the implementation of activities produced a nuanced, realistic and meaningful way to account for and analyse the project’s impact. It supported reflection on how the intervention’s shared objectives and desired impact were crystalising beyond the notions established at its conception. The methodologies, best practices, results, lessons learnt and all collected data will be put together in a relevant manual at the end of the third year of implementation.

The data collection is based on a mixed-method approach, including both qualitative and quantitative methods. Particular emphasis was placed on the application of qualitative evaluative methods, however. This was due to the complexity of the project, its innovative design and the high mobility and dropout rates of refugees. The approach was chosen to build an in-depth understanding of multi-layered processes and the different ways in which the project created impact. Quantitative data was meant to reveal certain trends and patterns, providing a general overview of progress. Qualitative data collection, in turn, facilitated in-depth understanding of why and how effects came to be.

In terms of quantitative data collection, a baseline survey was designed and implemented from July 2019. It consisted of descriptive statistics (e.g. age, sex, country of birth, socioeconomic status) and specific questions relevant to understanding the respondents’ needs and capabilities in terms of the project objectives. The study regarding beneficiaries’ evaluation was adapted to capture how the project facilitated their integration process, since it was not possible to capture Time 1 (a baseline) during the first semester of their involvement in the program.

Curing the Limbo offers a set of interesting lessons, particularly in relation to evaluation governance and the applicability of action research as an approach to projects aimed at empowering beneficiaries:

• It is highly desirable to appoint a person or organisation tasked with developing and overseeing evaluation from the project outset. Such a person is especially important when evaluation is distributed across a number of project partners, in which case they are required to lead the integrative process. Curing the Limbo benefited greatly from the fact that the quality assurance team assumed a leadership role. Had this taken place earlier, the evaluation could have been made easier.
• Time invested early on in reflecting on a shared understanding of the project’s common goal, as well as the creation of common definitions, will pay off during the evaluation stage. This aspect is also related to the issue of governance, particularly when evaluation partners bring their own significant expertise in evaluation and the project has various components. Creating an integrative evaluation framework early on will help to align the data collected by various partners and highlight ways in which it can be combined into a project, rather than component-level research. It is essential to look at relationships between project segments and services rather than their cumulative results.
• The action research paradigm helps to create learning loops built on participatory principles. The project shows that meaningful accounting for inputs and feedback offered by the beneficiaries can improve the overall design and implementation, and that beneficiaries can effectively express their needs and preferences through the co-designing of services and activities.
• Projects tackling the issues of integration and empowerment, particularly in relation to vulnerable beneficiaries, should set realistic goals for what can be established during the evaluation. Since social processes take time and project implementation periods (and consequently the data collection needed for evaluation) are limited, it is essential to define realistic objectives and measurement instruments that can, at best, capture elements of integration or empowerment rather than provide definite answers. Instead of trying to measure integration, empowerment and exchange, the project set out to understand how it has impacted its beneficiaries and supported them in their individual journeys towards the end goal (integration into Athenian society), which has previously been inevitably beyond its scope.