Paris OASIS

Evaluation has been strongly embedded in the implementation of the OASIS project from the start and this has been considered an advantage. Being ‘in the middle of things’ allowed evaluators to feel at ease with those designing and implementing activities.

The evaluation didn’t only take place at the end, but throughout the process, which means it was an interactive evaluation that could lead to the adjustments.

Source: OASIS project hearing

Being part of the project team and the process provided evaluators with a good sense of what is feasible, in terms of suggesting changes and improvements, and helped with the design of activities with impact indicators in mind.

Evaluation has been managed by two entities; the Laboratory for Interdisciplinary Evaluation of Public Policies of the Sciences Po (LIEPP) for the behavioural aspects, and the Interdisciplinary Laboratory of Energies of Tomorrow (LIED), associated with ESIEE Paris, for the climate impact. The evaluation has also been strongly supported by Météo-France, the country’s key weather monitoring and forecasting entity. Météo-France was in charge of climate modelling and defining the climatic and micro-climatic evaluation protocol for quantifying the cooling effects created by the OASIS playgrounds at neighbourhood level.

The fact that the entities brought significant expertise in evaluating urban policies was considered important for the quality of the service. Importantly, LIED’s researchers have been involved in similar data collection activities at other sites in Paris, and this expertise facilitated the evaluation of OASIS. Evaluation governance has been characterised by clear definition of the responsibilities assigned to the involved entities and the scope of their engagement, as well as effective communication. This approach was helpful, for instance, when LIEPP worked on adjusting and fine-tuning its data collection instruments (questionaries for children). While LIEPP strived for simplicity of questions, its researchers consulted LIED and Météo-France to make sure that the instruments remained true to the scientific nature of the project.

Cooperation between the evaluation team and the project implementation stakeholders has been very active and enhanced learning loops for the project. The project implementing team was already able to efficiently use suggestions from the evaluators and quickly adjust activities as they evolved. For instance, thanks to monitoring inputs, the protocol initially developed for ensuring active participation of local community members was adjusted to better reflect the character of the groups.

OASIS sets a blueprint for the evaluation of projects that tackle urban climate change by incorporating data collection on both environmental and human impact (social and climate impact assessment). It is precisely this relationship, looking at how the environmental and the social are interwoven in the urban area and how they depend on one other, that is worth exploring. From a broader policy perspective, the purpose of the evaluation is to develop a replicable OASIS framework, which could be further scaled up across the city.

The project represents a comprehensive evaluation approach built on two pillars: assessing the cooling effects of the newly designed schoolyards and evaluating impact the project has on children (their climate change awareness and social behaviour) and local communities (their active participatory approach to new spaces and levels of ownership). As such, this framework looks closely not only at how the physical environment changes due to the project, but also how these changes affect people. Furthermore, analysing how the educational activities increase children’s awareness about climate change can, in turn, give some indication for how they can engage with the physical environment in the future. The evaluation set out to explore how improved outdoor areas, offering a better temperature and quality of air as well as new sensory experiences (smells, textures, visuals), can impact children’s behaviour. Does it prompt cooperation instead of conflict? Does it encourage more collaborative and integrative patterns of engagement?

The project opted for a mixed-methods approach, combining quantitative and qualitative data collection methods in the form of ex-ante and ex-post evaluations. Evaluation of the project’s impact on the people involved consists of analysing attitudes, interaction patterns and levels of ownership. Importantly, societal impact is assessed not only through methods based on declarations (in the case of this project, interviews and surveys), but also behaviour-based, ethnographic techniques (observation). If successful, such methodological triangulation should allow for accurate capturing of project results. Since one of the project’s key contributions was reclaiming schoolyards as communal spaces (contrary to the common practice of these premises being exclusively used by the school pupils and physically separated from the localities), the evaluation ecosystem included both school pupils and community members.

Data collection followed the evaluation’s general two-pillar approach and, as such, LEID collected information about microclimatic and thermal changes through deployment of computerised weather sensors (Internet of Things). LIEEP was responsible for data collection on the potential impacts the project had on the people involved (children and community members) and key information was sourced through questionnaires and ethnographic observations, supported by qualitative interviews.

The first component, monitoring temperature and microclimate change, rested on a set of fixed and a set of mobile measurements. The climate diagnosis was designed to evaluate the reduction of the UHI effect and the increase in the levels of comfort experienced by citizens in the summer months by analysing and comparing measurements before/after the project transformation work on schoolyards took place. Instalment of mobile weather stations allowed for the development of a relative spatial layout of the microclimate. Complementarity of these two measurements allows for the better capturing of varying degrees of change in the environment.  Each of the ten project sites was equipped with two fixed weather measuring stations, one located in the playground and one in a nearby reference area (control stations) that would not experience any significant change during the project in terms of temperature, humidity and wind speed. Specifically, the improvement of summer comfort in the playgrounds is measured by the heat stress index, and in the buildings using temperature sensors integrated into the premises and modelling of the sites under heatwave conditions. Rainwater infiltration capacity is measured in the laboratory on different surfaces and will be characterised at playground level thanks to modelling of runoff and drainage phenomena. As a result, data collected through sensors and in the laboratories should allow for the creation of models and 3D simulations of the environmental impacts in the schoolyard premises, compared to areas not impacted by the project.

Human-centred evaluation progressed based on a protocol established during the project’s first year and is set out in the document as three main axes: the use of schoolyards, the project’s co-design approach, and pupils’ knowledge and attitudes on climate change.

Ten schools were selected for in-depth data collection on children’s behaviour and attitudes, although schools were covered by data collection to various degrees. Data collection on the human impact consisted of two core elements: surveys among the beneficiaries (pupils) in five schools and quantitative ethnographic observations of how pupils interacted in the schoolyards in six schools. Both sets of data collection instruments (questionnaires and ethnographic observation grids) were tested and fine-tuned before being rolled out in the selected schools.

The purpose of the questionnaire was to grasp changes in attitude among the children that might have occurred as a result of the educational and awareness raising activities conducted in the framework of the project. The multiple-choice ex-ante questionnaire was tailored to the contents of the workshops (e.g. awareness of environmental change, mastery of scientific notions, e.g. temperature). It was administered in a total of five classes, one per school.

While administration of the survey amongst older children was relatively straightforward, carrying out the same activity with the pre-school pupils (unable to read or write) required the development of adequate data collection techniques. The project evaluation team decided to employ two puppets and apply a simple puppet show scenario where the interviewer would make a claim about the environment and the two puppets would agree or disagree with what was being said. Children then had to indicate which of the two puppets they agreed with. Such adjustment allowed for interaction with the children, but also required careful preparation. It built on a lot of pre-testing (as well as previous studies), which included checking whether the colour of the puppets could influence the answers of children, and how to conduct the whole exercise without fostering any bias among the youngsters.

The second aspect of the evaluation, accounting for possible behavioural changes, was carried out through ethnographic observations in accordance with the observation grid. Data was collected on the use of yards and green spaces, cooperative/conflicting interactions between pupils in courtyards, gender segregation and patterns of socialising, and coded in accordance with the operational definition codebook which was developed for the evaluation. The data collection lasted one month and included a large sample of children in three age groups: three, five and six years old; between six and eleven years old; and teenagers between eleven and fifteen years old. Between June and September 2020, a total of 18 days of observation were completed in five schools, amounting to 90 hours of observation in total, and three days spent in each school (observations during morning breaks, lunch breaks and afternoon breaks). Data collectors then quantified pupils’ behaviour, categorising it as attendance, acts of violence or cooperation, and noting how space the was used and how the children interacted with the premises. The idea of the evaluation was to compare the quantitative data on the children’s patterns of behaviour from the ex-ante data collection (when pupils were interacting in the regular schoolyards) with data collected after the children had been exposed to the transformed, cooler and greener spaces. Naturally, final analysis will have to take changes in the children/young people’s behaviour arising simply from the fact that they are growing and maturing emotionally into account. Similarly, teens were characterised by significant non-participation levels in the survey data collection.

Since the project’s co-design aspect could not be evaluated using the ex-ante and ex-post approach, a more flexible observation grid was developed for data collection on the activities. 21 observations were conducted in three (primary and middle) schools during the co-design workshops, of which six were directed at adults and the rest directed at pupils.

Both project implementation and monitoring have been strongly impacted by widespread strikes occurring in Paris, and further by the COVID-19 pandemic. These events led to the temporary closure of schools and limited access to the schoolyards. As such, the community and cohesion building activities (citizen assemblies) had to be abandoned due to limitations on communal meetings. The evaluation team has been in the process of adjusting to these new conditions where new data collection methods are explored.

OASIS’s clear advantage comes from the fact that it is strongly embraced and incorporated into the Paris metropolitan policy, with significant chances for upscaling. The project is not the first instance where schoolyards have been transformed in the city, as over 30 schools were the sites for pilot schemes in 2018 and 2019, while fifteen more were set to be transformed in parallel to OASIS. The importance of OASIS in comparison to the other transformations comes precisely from its strong focus on learning and evaluation of impact. One of the main objectives for upscaling the UIA OASIS project is to develop a standardised cross-departmental process, from design to implementation, together with a set of guidelines and specifications for securing the high quality of the project.

Given the project’s location within schools and the fact that it largely addresses minors as the key target group, a set of ethical issues was identified with regards to evaluation. Firstly, the question of informed consent to participation in data collection is problematic when it comes to children. The project evaluation team made all possible efforts to clearly convey the purpose and scope of the evaluation to school gatekeepers (principals), whose decision it then was to allow or prohibit in-depth evaluation of their school. Consequently, it was the schools’ principals who granted the evaluation team permission to question and observe the pupils. While the pupils were being observed during breaks and the time they spent in the school yards, they were never explicitly asked for consent. The pupils had the option to decline responding to the questionnaires (resulting in non-participation), but did not have a say in being observed by the data collectors during the ethnographic portion of the research. Whenever approached by children in the school yards, however, researchers gave an explanation of their presence on the school premises. Given the OASIS’s strong emphasis on participation, collective decision making and co-creation, it seems the evaluation framework could further benefit from addressing the issue of pupils as subjects rather than objects of data collection.

The evaluation team was also faced with ethical challenges in relation to involving pupils with learning disabilities in the data collection process. Here, decisions were made on individual case basis and in strong cooperation with the teachers supervising the pupils. This was a balancing act where priority was given to children’s wellbeing, whilst also attempting to ensure that they were allowed meaningful participation through the adjustment (simplification) of data collection instruments, when possible. Simultaneously, decisions were also made to exclude children from data collection if it was judged to be too cognitively challenging for them.

OASIS offers several good practices in terms of evaluation governance and approaches and data collection. Firstly, it is a case for the strong integration of evaluation experts into the project implementation team from the outset. Secondly, the project is a valuable example of how intervention with strong environmental and infrastructural components can effectively integrate and evaluate both physical (environmental) changes and the social aspects of a project. Lastly, careful preparation and implementation of data collection in the school environment, particularly among younger children, offers interesting approaches to data collection that can be replicated by other cities.

As such, the following lessons were identified by the project team:

• Incorporation of the evaluators in the project design and implementation from the very start allows for shaping of the project cycle and activities with the end evaluation in mind. In the case of OASIS, such integration and early dialogue allowed for the building of rapport and trust between the evaluators and the partners directly implementing the activities. This further facilitated giving and receiving feedback. Additionally, being part of the project design phase enabled evaluators to enhance the shaping of activities with the data collection in mind.
• Combining data collection on both the environmental and social impacts of the project ensures that the evaluation fully captures the possible effects of an intervention. In this regard, OASIS stands out as a project that has paid equal attention to monitoring of the microclimate changes produced by activities as it has to devoting resources for comprehensive quantitative and qualitative data collection on how the changed environment affects people. Only such a combination of data can reveal a full range of effects the project was able to achieve. This lesson is especially valuable for projects with strong environmental or material components, where the social impact can be easily overlooked.
• Data collection in a school environment requires careful preparation, particularly if an intervention involves small children. OASIS offers a wealth of learning for other projects interested in data collection among young children, as it has developed and tested innovative survey techniques with the use of puppets. It proves that even children who cannot yet read or write can be meaningfully engaged in data collection, and their voices included and analysed. Young children are often overlooked in the evaluation of initiatives and OASIS proves that this does not have to be the case.

Review of the OASIS evaluation experience suggests that projects implemented within institutions (such as schools, in the case of OASIS) could benefit from approaching the establishments as systems, built out of various elements and groups. In this regard, OASIS evaluation focused first and foremost on how the intervention affected the pupils during the scope of their breaks between classes, specifically. A system approach to school could reveal interesting findings about how other school members (such as teachers and other staff) can also be impacted by transformed and friendlier school yards. Equally interesting could be the exploration of possible changes in pupils’ behaviour outside of the time spent directly in the school yards (during classes, for instance). Generally speaking, projects similar to OASIS in terms of their location within institutions should consider various elements of the systems in which they operate.