A framework to assess the impact of the project outputs was developed, including planning and monitoring phases as follows:
- Baseline Setting and Monitoring and Availability Framework: Conceptual analysis of the mobility ecosystem of the city including its entities, the city itself and the society. Definition of the baseline to compare the project outputs.
- Monitoring and Availability: Plan to monitor and assess the impact of the project outputs.
- Display: Visualization of the project results to increase transparency of public services.
Baseline Setting and Monitoring and Availability Framework
First an extensive literature review was done to identify benchmark mechanisms for monitoring and assessing project implementations, concluding in the following:
- Two assessment levels: one level for the impact assessment (technical, social and economic in the short, medium and long term) and a second level for assessing the process itself and understand the reasons for its success or failure.
- Principles to consider: equilibrium (positive and negative impact to be assessed); relevancy and non-redundancy (select the relevant indicators only and avoid redundancies); exhaustive (all project aspects should be measured); availability and measurability (the indicators should be available and measurable by the system); evolutive (indicators should be measurable in time).
- A multi-layer framework of metrics is adequate (2-3 levels), including various dimensions (transport, society, quality of life, economy, energy and the environment).
- Aggregated index facilitates to monitor the project implementation. The definition of the weights to calculate this metric are key.
- Normalize metrics facilitate comparability of indicators (within the project or between projects).
Second, the project partners were interviewed, and workshops were held to identify ambitions and expectations. In these, the partners highlighted the capability or VoxPop to develop new decision support tools using quantitative models and including social impacts. Additionally, a pool of external experts has validated the framework.
Finally, the VoxPop Monitoring and Availability Framework developed included two areas: reflexive monitoring to analyse the evolution of the urban mobility context during the project implementation, assessing its alignment with the project long-term ambition; and impact assessment to quantify the impacts of the project at different levels. In both cases the project defined indicators, dimensions (accessibility, inclusivity, pollution… for reflexive monitoring and entities, value proposition, business models… for the impact assessment), domains (society, economy and environment for the reflexive monitoring and institutions, citizens, market… for the impact assessment) and ending in the aggregated index, calculated as shown in the below tables.
|
Reflexive monitoring |
||||||||||
|
Domain |
Weight |
Dimension |
Weight |
Indicators |
Weight |
|||||
|
Society |
55% |
Accessibility |
30% |
Cost of mobility services |
50% |
|||||
|
Geographic coverage |
50% |
|||||||||
|
Attarctivity |
20% |
Public transport attractiveness |
100% |
|||||||
|
Inclusiveness |
35% |
Awareness about VRUs |
100% |
|||||||
|
Well-being and comfort |
15% |
Management of public space |
20% |
|||||||
|
Quality of public space |
33% |
|||||||||
|
Level of service |
47% |
|||||||||
|
Economy |
20% |
Revenues |
20% |
Revenues |
100% |
|||||
|
Costs |
50% |
Costs |
100% |
|||||||
|
Job creation |
30% |
Job creation |
100% |
|||||||
|
Environment |
25% |
Air pollution |
100% |
GHG emissions |
60% |
|||||
|
Air quality |
40% |
|||||||||
|
|
||||||||||
|
Domain |
W |
Dimension |
W |
Indicators |
W |
|||||
|
Institutions |
30% |
Entities |
20% |
Companies willing to share datasets |
100% |
|||||
|
Capacity Building |
50% |
Capacity Building for innovation and analytics |
100% |
|||||||
|
Partnerships |
30% |
Companies sharing datasets |
35% |
|||||||
|
Volume of shared dadasets |
35% |
|||||||||
|
Interoperability of systems |
20% |
|||||||||
|
Interoperabilty of European systems |
10% |
|||||||||
|
Citizens' participation |
35% |
Co-creation |
100% |
Citizens engaged in co-creation activities |
100% |
|||||
|
Decision making |
20% |
Challenges and opportunities |
35% |
Challenges and opportunities identified |
100% |
|||||
|
Policy design |
30% |
Design of public policies |
100% |
|||||||
|
Value proposition |
20% |
Value proposition for added value services |
100% |
|||||||
|
Business model |
15% |
Business models developed |
100% |
|||||||
|
Market introduction |
15% |
New services |
40% |
New services |
40% |
|||||
|
New services for niche markets |
60% |
|||||||||
|
Results dissemination |
10% |
Communication and dissemination |
70% |
|||||||
|
Scientific and technical production |
30% |
|||||||||
|
Penetration of services |
50% |
Use of new services |
40% |
|||||||
|
Use of new services for VRUs |
60% |
|||||||||
Monitoring and Display
A plan based on the above framework was defined to monitor the project execution and impact assessment, which included indicators and metrics description, data sources and stakeholders involvement. The data was collected in three phases: ex-ante (to define the baseline), ex-durante (to do an intermediate evaluation) and ex-post (to do the final evaluation).
The plan includes the methodology for data analysis in terms of quantitative and qualitative indicators computation and analysis, monitoring report to and by project partners and revision of project objectives to decide if corrective measures are needed.
’
Digital transition
