Expert article
Edit 15 April 2021
by Nils Scheffler

The challenge of evaluating an innovative housing project

Measuring changes
Measuring changes
The city of Mataró has developed an innovative housing project to provide affordable housing in the private rental market. In order to learn from the experience of the pilot phase for an adaptation and upscaling of the project, the city has developed an evaluation scheme. How such an innovative project that evolves and changes over time can be evaluated and which challenges arise in the implementation of the evaluation are described in the following article.
Share

When you walk through Mataró, a city of 127,000 inhabitants 34 km north-east of Barcelona at the Costa del Maresme, you might not realize it at first sight, but the city has more than 2.800 private empty flats. At the same time an increasing number of the inhabitants are overburdened by the rental housing cost or they have little access to the private rental market, in particular younger people. Since 2015, rental prices in Mataró have been growing by 27 %. The expenses for rent can be significantly over a 1/3 of the monthly income. Moreover, there is a dramatic shortage of affordable housing in the city. The city’s social housing sector is totally incommensurate to demand. It comprises just 265 units while 1400 households are currently on the waiting list.

The ‘Yes, we rent!’ project of the city of Mataró aims to increase the number of affordable, privately rented flats in the city and improve their accessibility to households with difficulties in the private rental housing market. Key to this is a cooperative-based housing approach, which incentivises owners of empty flats to rehabilitate their flats and rent it below market price through the cooperative.

General objective to evaluate/monitor the ‘Yes, we rent!’ project / your motivation to evaluate the project.[Darstellung des Ziel des Monitorings und was damit erreicht werden soll/ warum das Projekt evaluiert wird]

In order to evaluate the successes and impact of the project the city of Mataró developed with the support of TecnoCampus Mataró-Maresme and the Institute on Government and Public Policy (IGOP) of the Autonomous University of Barcelona an evaluation scheme, for which implementation the two partners are in charge. The monitoring is done with a view to evaluate:

  • Housing market changes: the changes in the provision and accessibility as well as the quality of affordable rental flats in Mataró.
  • Institutional changes: the institutional and governance changes in the city administration for the provision of affordable rental housing.
  • Organisational changes: the economical and institutional capability of a cooperative to take on the function of an organiser and mediator between owners of vacant flats and household looking for affordable rental accommodation.
  • Personal changes: the effects on the lifes of the cooperative members who can enjoy affordable housing and the willingness of owners to let their vacant flats below market price.

Based on the evaluation results the city of Mataró intends to take decisions on the continuity of the project and its possible scalability. The evaluation aims also at the replication of the project in other cities in Catalonia.

To be able to evaluate the above mentioned “changes”, TecnoCampus and the Institute on Government and Public Policy have developed for each evaluation area research questions and their corresponding hypotheses as well as qualitative and quantitative indicators to be able to analyse the effect and the variables.

 

The “changes” are primarily measured:

§  “Personal change” through owner and tenant surveys as well as secondary data analysis from tax and census/ population register and from local health authorities;

§  “Institutional change” through questionnaires, focus groups with tenants, municipal staff and owners and interviews i.e. with the technical office of the ‘Yes, we rent! Project, other involved technicians from the city administration and the city council;

§  “Organisational change” through questionnaires, focus groups with tenants, municipal staff and owners, interviews with members of the driving group of the cooperative and observations taking part in meetings of the cooperative and working sessions of the cooperative with the municipality and the technical support team;

§  “Housing market change” through secondary data analysis from the tax and census/ population register as well as from two major real state agencies.

 

For a detailed overview about the evaluation scheme (in Spanish) download this document.

While TecnoCampus and IGOP implemented the evaluation scheme different challenges arose.

GDPR restrictions

The legal requirements of the General Data Protection Regulation (GDPR) make it difficult to TecnoCampus and IGOP to receive access to the city administration data (and other sources) to be able to monitor the project as planned in the evaluation scheme. This required the two partners to clarify all binding steps of the data protection regulation and, based on the results, to work out together with the city council a legal framework and a procedure to obtain legal and periodically the data of the different city administration units and other sources. Also with other institutions legal documents had to be prepared to get data access for the monitoring of the project.

This circumstance required time and familiarisation with a new matter on all sides. To facilitate this, the partners hired external support to prepare the legal frameworks. As the project continued, relevant data could be retrieved first at a later stage of the project.

Vague or changing project objectives

Innovative projects cannot be planned in all details from the beginning and evolve. This is also the case with the 'Yes, we rent! project. As the project has evolved and gained focus over the time, also objectives, outcomes and potential impacts became clearer and even changed. This demanded the adaptation of the evaluation scheme and data frame during the ongoing monitoring process. Thus, the project could not be monitored the same way from the beginning.

COVID 19

Covid-19  – as everywhere – had its impact on the monitoring of the project. Due to the COVID-19 restrictions interviews and direct observations could not be carried out as planned. Focus group discussions had to be postponed. As solution interviews were shifted online and the observations concentrated on online-meetings of the project partners and the ‘driving group’ of the cooperative. This shift from qualitative evaluation fieldwork to “online” fieldwork demanded its time.

Lack of input or collection of data by project participants

For the monitoring, data had to be collected and provided by project participants during the implementation of the project. But some project participants did not comply with this to the extent required filling in monitoring documents or collecting relevant data as requested in the evaluation scheme. The reasons may have been that the importance for the data collection was not seen. To counteract this TecnoCampus and IGOP had to encourage partners to fill in questionnaires and monitoring forms, explaining the necessity and meaningfulness of the evaluation of the project. In support they set up a data register and follow-up system.

Also some administrative units, not directly involved in the project, did not comply with the data provision as expected. To gain their commitment, extra meetings were organised to explain the importance of the required administrative data for the evaluation of the project. Also examples of legal frameworks from other projects were presented to gain their support.

Based on the experience gained so far, the following recommendations for monitoring similar projects are made:

§  Develop and agree on legal framework and general procedure on data transfer before the project starts. For this, get together with the necessary institutions early on and get professional support if needed.

§  When implementing new approaches be open and flexible to adapt your monitoring and evaluation scheme, even if it demands extra workload. So check your monitoring scheme once a while whether it still meets the adapted objectives and can measure relevant impacts of the project.

§  In the beginning explain well to institutions from which you need data why certain data is important and needs to be collected and provided by them. Consider how these institutions can also benefit from the evaluation.

§  When being responsible for the evaluation, engage in non-participatory observation so as to maintain one’s status as neutral observer and to gain the trust of all participants. Do not take sites in discussions or get directly involved to enable frank dialogue during interviews and observations.

§  Measuring impacts is often only possible after a certain period of time has elapsed since the project was implemented. Thus, consider monitoring and evaluation activities even after a longer period of time after the pilot project has been completed.

If you would like to know more about the evaluation scheme, get in touch with Catalina Llaneza Hesse (cllaneza@tecnocampus.cat) or Lorenzo Vidal-Folch Duch (Lorenzo.VidalFolch@uab.cat).

For first evaluation results, take a look at the report about institutional and organisational changes (in Spanish).

Other news from this project

Logo Bloc Cooperatiu

A day to remember: 18.2.21, Mataro has a new offspring – the tenant housing cooperative BlocCooperatiu!

On 18th of February 2021, the notary LOPEZ & ALBIOL formally approved Mataro’s new tenants housing cooperative ‘BlocCooperatiu’. Dedicated residents o...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Affordable housing
Co-housing
Housing
Housing cooperative
‘Yes, We Rent!’ Journal 2: Reactivating the private empty flats has started!

‘Yes, We Rent!’ Journal 2: Reactivating the private empty flats has started!

UIA Expert Nils Scheffler updates us how “the acquisition of empty flats as well as the training of troubled adolescents for the renovation of the emp...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

video conference about housing cooperative

A look ahead - What does 2021 bring for the 'Yes, we rent!’ project

COVID-19 and its effects continue to be a reality in Mataró, but also for the project 'Yes, we rent!'. ...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Affordable housing
Co-housing
Education
Energy retrofitting
Housing
Housing cooperative
Vulnerable groups
Yes, We Rent! – Training vulnerable young adults

Mataro Zoom-in 1: Vacant flats for a better future of vulnerable young adults - The ‘Yes, We Rent!’ training programme

Yes, We Rent! – Training vulnerable young adults...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

First owner signed contract to bring his flat to the ‘Yes, we rent!’ housing scheme

First owner signed contract to bring his flat to the ‘Yes, we rent!’ housing scheme

On 28th of September the time had come. The first owner, Antonio Martínez, signed the contract to make his empty flat available to the 'Yes, we rent!'...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Adolescent renovating flat

Do not bury your head in the sand - especially despite COVID 19!

Interview with Laura Font Mela, director of Salesians Sant Jordi-PES Mataró, how they have successfully shifted their physical training program for tr...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Affordable housing
Education
Housing
Jobs and skills in the local economy
housing is a human right

Cities engaging in the right to housing

Urban Innovative Actions (UIA) and URBACT are joining forces to push the right to housing...
UIA Article
Affordable housing
Journal 1 for ‘Yes, We Rent!’: Mataro’s first steps

Journal 1 for ‘Yes, We Rent!’: Mataro’s first steps

UIA Expert Nils Scheffler explains that “so far, no serious challenges for the project have arisen. There is a quite strong political and administrati...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Picture of the "Yes We Rent!" Website

Owners, make your empty apartments available!

In order to acquire apartments for the 'Yes, we Rent!' housing scheme, the first information campaign measures were started and first apartments visit...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Affordable housing
Circular economy
Co-housing
Energy retrofitting
Housing
Housing cooperative
housing is a human right

Cities engaging in the right to housing

Urban Innovative Actions (UIA) and URBACT are joining forces to push the right to housing...
UIA Article
Affordable housing
"Yes, we rent" project picture

Mataró has kicked off its project for a larger audience: The Learning event!

Mataró faces the problem of “people without flats and flats without people”....

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain

Affordable housing
Circular economy
Co-housing
Co2 reduction
Energy retrofitting
Housing
Housing cooperative
Rehabilitation works within the Yes We Rent! project

Salesians Sant Jordi starts training 12 youngsters in rehabilitation works within the Yes We Rent! project

Yes We Rent! is a project coordinated by Mataró City Council (SPAIN), to try to bring solutions to the dramatic lack of affordable rental housing in t...

Yes We Rent! - Leveraging vacant private property to build up a cooperative affordable housing scheme

Housing

Mataró - Spain