Barcelona B-MINCOME

The project was implemented by the Barcelona City Council together with five delivery partners. Because of the project’s nature as both a public policy intervention and evaluation, almost all partners were involved in evaluation to some extent:

• The Catalan Institute of Public Policy Evaluation (Ivalua) conducted:

1. the quantitative impact evaluation;
2. the quantitative economic evaluation.

• The Young Foundation carried out the ethnographic study.
• The Institute of Governance and Public Policies of the Autonomous University of Barcelona (IGOP) conducted:

1. the analysis of the deployment and effects of the community participation policy;
2. the study on the implementation process and governance of the project.

• The Institute of Environmental Science and Technology of the Autonomous University of Barcelona (ICTA) was involved in evaluating the determinants of subjective wellbeing among participants.
• The Institut Internacional per l'Acció Noviolenta (Novact) was in charge of carrying out the evaluation of the implementation of the Real Economy Currency (REC).

The project partners represented different professions and areas of expertise, which provided access to diverse knowledge and enabled synergies.

In particular, B-MINCOME benefited from strong expertise in research methodologies. The project’s access to extensive quantitative expertise allowed it to function both as a public policy intervention and an experiment, enabling strong causal inferences in relation to the intervention’s effectiveness and impact. The B-MINCOME evaluation also represented a complex effort to involve actors beyond project partners (see more under ‘Horizontal issues’).

Co-creation and close cooperation between all project partners were key for the evaluation. The dual nature of the project made it particularly important to find a common design and mutual understanding from the start. Since both public policies and experiments face important limitations, the partners needed to be aware of and account for them during design and implementation. As the evaluators noticed, the partners had to compromise on various ideas (such as the need for eligibility criteria, which were mandatory in the implementation of public policy and inevitably influenced the shape of the respondent sample for the experiment) to get to their end goal and to be at least comfortable with what they were trying to achieve.

B-MINCOME secured substantial resources within the project budget for evaluation, although evaluators noted during the hearing that resources still posed some challenges. The counterfactual experimental design chosen as one of the approaches involved a large research effort. Over 1,400 households were surveyed repeatedly (approximately 1.000 treatment households and over 400 control households, in three waves). While the counterfactual evaluation is often more costly than other approaches, in B-MINCOME it was further supplemented with an ethnographic study and other qualitative evaluations pertaining to different elements of the intervention. Additional funds were also mobilised for integrating all research results. These came from the municipal budget independently of UIA financing.

As a social experiment, B-MINCOME represents a comprehensive effort to evaluate the intervention as a whole and some of its specific parts (e.g. community participation policies, governance or the REC). Major emphasis was placed on evaluating the impact of the intervention at three levels – individual, community and institutional. Components were also devoted to evaluating efficiency (i.e. economic evaluation) and processes (i.e. governance and implementation evaluation). The table below gathers all the elements of the B-MINCOME evaluation.

Source: IGOP, excerpt from report on the final results (2017–2019), ‘Integration of evaluation results’, shared in October 2020.

While parts of the design were included in the project application presented to UIA, the development of the final approach took a substantial amount of time. The intervention therefore significantly benefited from the addition of a preparatory phase to all UIA projects starting from Call 2. 

Specifically for impact and economic evaluation, Ivàlua employed the counterfactual approach (i.e. measurement relative to what would have happened had the project never taken place). Towards this end, Ivàlua divided households eligible for participation in the project into two groups – those who received the intervention in various forms (the treatment group) and those who did not (the control group).

The partners chose this approach as it corresponded with the circumstances of the project. The demand for services exceeded the funds available for GMI schemes.

The counterfactual method used in B-MINCOME was based on an experimental design which meant that eligible households were randomly assigned – by way of a lottery – either to the treatment or control group. The randomisation was stratified, with eligible participants divided into specific categories called strata and, within those categories, randomly assigned to the treatment or control group. The blocking variables for the randomization were a) eligibility for the rent room promotion policy, b) employability of at least one household member (yes / no) and c) the expected amount of the monthly GMI benefit for the household (high, medium, low).

For economic evaluation, Ivàlua employed among others the cost-effectiveness analyses, focusing on the evaluation of the impact of the program on selected results (general satisfaction, social support, health status, employment situation and education) and the use of resources.

For the ICTA study evaluating the determinants of subjective wellbeing among the participants, cross-sectional regressions were conducted. ICTA also carried out a study of panel data which explored individual and collective changes in life satisfaction over time and sought to explain the variations in the levels of life satisfaction of each individual.

As part of the qualitative research effort, the Young Foundation conducted an ethnographic study, which also helped to determine what should be measured in quantitative surveys. 

The ethnographic study consisted of three components:

• A study of the experiences of a select number of households over 12 months when beneficiaries received GMI. The report from the study was published in February 2020.
• A community-level study exploring the dynamics of poverty which resulted in a separate report.
• A study on the perceptions of individuals experiencing poverty. The result of this was 8 videos  presenting beneficiaries’ stories of change, struggle and resilience, and how B-MINCOME had affected their lives. A film screening was hosted in the neighbourhood of Bon Pastor in November 2019 and the films are also available online.

The B-MINCOME evaluation employed a mixed-methods approach to data collection and analysis. This offered a combination of strengths from different methods and allowed for mitigation of their shortcomings.

The evaluation of B-MINCOME involved extensive data collection. From the start, the partners wanted to ensure the availability of appropriate data. The table below summarises all the data collection activities undertaken.

Data collection methods in B-MINCOME.
Source: IGOP, excerpt from the report on the final results (2017–2019), “Integration of evaluation results”, shared in October 2020.

The project was a public policy intervention and a social experiment at the same time. This meant that the households selected to participate in the project would also be participants in the three waves of the survey which fed information into the impact and economic evaluation, as well as the evaluation of the determinants of subjective wellbeing. In other words, the sampling of beneficiaries was also the sampling for the survey. The control group was surveyed as well in all waves. The potential beneficiary sample was determined based on social assistance administrative records. One of the limitations of administrative data is that it lags behind reality and may, consequently, contain outdated entries. There was, therefore, a need for further verification of beneficiaries.

The City Council identified all the families that were active users of social services from 10 selected neighbourhoods. The initial list included approximately 4,305 households. These people were contacted via letters. Subsequently, up to 400 informative sessions were held to explain the project. Some 2,525 households eventually applied.

After an in-depth audit, as many as 1,527 households were found to be eligible to participate in B-MINCOME. For this group, a lottery was held which assigned the participants to treatment, control and reserve groups. The reserve group was created to make sure that the project could react quickly to non-participation or resignation of beneficiaries from the treatment group.  

For the baseline measurement, a computer assisted telephone survey was conducted. It had an 87% response rate. The first follow-up survey had a similar structure to the baseline, with selected questions modified or added. Some surveys were conducted in person as a solution to language difficulties experienced by respondents. Additionally, some families that could not be contacted by phone were interviewed by social services in person when they arrived. The response rate in the follow-up survey was 79.49%. The second follow-up survey, with a response rate of 75.72%, was again similar in structure to the first follow-up survey, with some questions added or eliminated. Surveys were filled out by the person receiving the GMI and active policies. This person would respond both to questions about their specific circumstances, but also about their household situation.

Additional surveys were conducted by IGOP for the purpose of evaluating the community participation policies. In implementing the surveys, researchers encountered challenges related to the respondents’ levels of education and, as with the impact evaluation surveys, language. They were, however, able to adapt to the situation. 

As is visible in the table, this substantial quantitative effort was complemented by qualitative research involving many data collection methods, such as in-depth individual interviews and focus groups. The Young Foundation employed ethnography, adding household observations to in-depth interviews. Importantly, the research had a longitudinal nature, with some households being visited more than once.

With the evaluation involving so many components and conducted throughout the entire project, integration of results became crucial. Even more so, if shortcomings of the quantitative research were indeed to be mitigated by results from qualitative data collection and analysis, and vice versa. It is, therefore, a positive sign that the integration effort was carried out despite the project’s termination by IGOP in 2020.

B-MINCOME evaluation involved various stakeholders. Most project partners were included as evaluators in the evaluation design and implementation. Other partners and individuals implementing the project were consulted as sources of information. IGOP organised focus groups with professional representatives of the four active policies and social workers from B-MINCOME, for example. Through in-depth individual interviews, the evaluation sought to incorporate the perspective of other stakeholders, including administration professionals linked to the B-MINCOME project and representatives from the social communities of the Besòs Axis neighbourhoods.

The evaluation also involved the project’s target group – mostly as a source of information, however. Beneficiaries’ participation was crucial for the experiment to produce results, so they were widely consulted through quantitative surveys and qualitative interviews. Ensuring wider beneficiary response to the survey required strong communication, encouragement and support from researchers. Apart from engaging beneficiaries as sources of information, the ethnographic study allowed them to play a more active role in analysis and in assessment of their own experiences. This was also possible through the co-creation of videos summarising beneficiaries’ experiences of poverty and the significant changes they experienced as a result of the B-MINCOME project.

The beneficiaries were not involved in the co-creation of the evaluation due to a number of challenges. Their vulnerabilities, limited experience of participation in research, as well as language barriers coupled with a short project timeframe – in the partners’ view – reduced avenues for co-creation of the evaluation.

One of the issues which emerged in B-MINCOME was how to avoid placing too much of a research burden on project beneficiaries. During the evaluation, beneficiaries were asked to fill out repeated surveys for the sake of the research, participate in interviews and put up with observations. This could have created an imbalance in relations between them and the researchers, creating a feeling that the beneficiaries were indeed just objects for various actions from project partners.

The researchers were aware of this danger and introduced mitigating strategies which involved increased researcher participation in activities with beneficiaries in the initial months of project implementation, coupled with extensive communication to justify the research activities.  

The project clearly represents a very strong evaluation design which aims to provide a comprehensive picture of the intervention, both in terms of its impact – at the individual, community and institutional levels – and processes. It goes far beyond monitoring. The evaluation made use of the counterfactual approach, and combined it with qualitative studies, including ethnography. A plethora of stakeholders were consulted with the application of various data collection methods. Different types of analysis were applied to the obtained data. Many partners collaborated on the evaluation not only to determine whether the action worked, but also to answer questions about how and why.

Some observations can be made in relation to transferability of the evaluation design. The availability of adequate research expertise would be an important consideration in determining whether the approach can be followed elsewhere. For ethnographic research, one of the main considerations would be the language of communication, especially for interventions involving migrants and vulnerable populations with lower education levels. Different linguistic competencies may therefore be needed in the research team. With proper qualitative expertise, however, ethnographic research as implemented in the project can be transferred onto many other types of interventions.

The transferability of the experimental design seems to be a more complicated question, however. The feasibility of an experiment relies on the availability of a control group that does not receive the intervention’s support. The creation of a control group may be controversial from an ethical or political standpoint, but it may just as well – in some circumstances – be impossible. Further still, the counterfactual approach requires repeated data collection – mostly quantitative and usually wider than in qualitative studies – which creates the need for higher budgets. In this sense, the full design may not be transferable in the case of all interventions.

The research approaches and methodologies that partners used drew from the well-known research toolbox. The innovation that the project offers, therefore, lies in combining and linking all these different elements and applying this integrated result onto an intervention, such as B-MINCOME.

A number of specific lessons can be drawn from B-MINCOME for future projects:

à The counterfactual approach can work in projects tackling sensitive social issues, if it corresponds with the conditions that the project creates, is well-integrated into an intervention from the start and is well-communicated. Interventions that concern sensitive social issues, especially those involving vulnerable populations, often cannot be evaluated through a counterfactual. However, as the B-MINCOME example shows, the approach can be used – for example, when the project meets a situation of excessive demand which would have created a sort of control group anyway. It also helps when the partners select the approach early on and can take several months to fully develop the evaluation before implementation starts. Such a preparatory phase allows the design to come to fruition through a cooperative process and helps to align the intervention with research requirements (and vice versa).

• Tensions may occur between policy implementation and research, but these should be worked out rather than feared. The project’s dual nature created some challenges (related to the official eligibility criteria influencing the sample and the need to ensure consistent participation and high response rates, for example). In both cases, internal and external communication and cooperation proved to be effective in leading partners to working solutions. The preparatory phase allowed the partners to work out the most important project details from the start.

• The capacity of the target group (i.e. survey and interview respondents) should be analysed and solutions should be developed to widen participation. In interventions which tackle sensitive social issues, such as B-MINCOME, the evaluators should have enough understanding of the target group to be able to set up an evaluation design that also allows vulnerable individuals to participate. The challenges could include low literacy levels among the target group or language barriers. In these cases, more resources can be allocated (for the facilitation, interpretation, and translation of surveys, for example). Different modes of survey implementation can also be employed. 
• Big research efforts run the risk of overburdening respondents with demands. Given the intensity of research, evaluations such as B-MINCOME should have mechanisms which force evaluators to examine their own demands towards research subjects. There should be moments for reflection on whether the respondents are asked for too much, and whether they are treated with respect and not just as sources of data. The researchers should build trust through their own participation in activities, putting themselves forward and providing the rationale for their own actions.
• There is great value in involving specialised research institutions with both qualitative and quantitative expertise. Thanks to the advanced expertise on board with the project, B-MINCOME was able to carry out extensive data collection and advanced analyses within the challenging context of a big and innovative project.