Aveiro STEAM City

The partner responsible for monitoring and evaluation was CEDES – the Association for a Centre of Studies in Sustainable Development. It is an organisation with experience in various activities related to urban challenges, sustainability, technological development, research, education and innovation. In terms of project management, monitoring and evaluation was structured as a separate component, along with the design and testing of a business model.

Since CEDES was a project partner and not an external evaluating institution, it participated in all activities and communications, having ample opportunity to observe and understand project progress. All project partners cooperated in monitoring and evaluation through joint and one-on-one meetings, which also provided the opportunity for regular feedback and learning loops.

Monitoring and evaluation in Aveiro STEAM City was achieved through a participatory approach, which aimed to engage project partners, stakeholders and target groups. The approach could also be considered to be based on a theory of change, since significant attention was given to defining a set of indicators and outcomes linked to activities conducted in the project. Nevertheless, this was not explicitly expressed.

In fact, defining this set of indicators proved to be somewhat challenging due to the relative complexity of the project. The team aimed to design a simple monitoring and evaluation model that not only integrated the impact of the main project components, but also aggregated data and paired feedback from various sources into the same indicators. This way, as one indicator would represent information from various activities and target groups, the project’s broader impact could be grasped.

The resulting set consisted of several types of indicators:

• Impact indicators – measured through national statistics data.
• Result indicators – captured primarily through surveys and focus groups with users and stakeholders.
• Output indicators – measured through project monitoring data and accounting for what is created or conducted during implementation.

Importantly, the indicators developed were treated in a dynamic and flexible way, with partners ready to change them should such a need arise based on insights from project implementation.

As the project evaluation team stressed while developing the indicator set, it is necessary to cooperate with partners who implement the activities. This may be tricky, as partners focus on implementing the activities they are responsible for, but it is the only way to develop a useful and relevant indicator framework.

At the time of the case study, Aveiro STEAM City was being implemented, as noted above. Because of this, information on data collection was limited. The project team planned to use a mixed methods approach, including both quantitative and qualitative research methods. Combining methods can improve the validity of results and balance out the weaknesses and advantages of single methods. Since Aveiro STEAM City is largely based around deploying and using ICT tools, it also aimed to employ them in data collection.

At a general level, the project presents commendable openness to modifications in its approach, should challenges to data collection occur. Furthermore, although data collection methods are primarily quantitative, the project team foresees the need to conduct qualitative interviews and focus groups with users and stakeholders at the end of implementation to understand the process and its effects. This is crucial for learning and continuing activities after UIA funding ends (designing improved projects, for example).

Several lessons can be drawn from Aveiro STEAM City monitoring and evaluation approach:

• The monitoring and evaluation process needs to be easy, affordable to implement and effective, both for research participants and the project team.
• All project partners should be involved in designing and conducting monitoring and evaluation because they have the best knowledge on the limitations, constraints and potential impacts of activities. Such knowledge is more difficult to access if monitoring and evaluation are conducted from an external perspective.
• Research methods and tools used need to consider the type of target group that is being asked for feedback – different questions need to be directed at different groups (e.g. primary school students and active professionals).