Evaluate & Iterate
Step 5: Are We Going Right?
With a roadmap from Step 4, this step evaluates whether the journey, its length, and the places it passes are feasible and conform to the goals set in Step 1. It also determines whether another cycle is needed or whether the process is ready to move toward implementation.
Evaluation Questions
Goal alignment: Do proposed solutions address system-level goals, or have we drifted toward object-level fixes?
Indicator performance: Do KPIs show improvement across the full ELSI spectrum, or have we created new externalizations?
Feasibility: Are interventions realistic given the project's boundary conditions (budget, time, resources)?
Systemic effects: Have we accounted for rebound effects, lock-in risks, and unintended consequences?
Stakeholder alignment: Do all key stakeholders support the proposed direction?
Evaluation Methods
Evaluation matrix: List all goals and boundary conditions, and score each solution from low to high for each aspect. This is especially helpful when comparing competing solutions or when the team cannot reach consensus through discussion alone.
External jury: Form a panel of experts outside the project team. External reviewers look at solutions with fresh eyes and will likely have insights not yet considered by the core team. This is especially valuable for projects with high stakes or public impact.
Mid-session harvests: During longer sessions, gather the group every half day to recap activities, re-list goals, and surface the best ideas. This prevents drift and ensures that emerging insights are captured before they fade.
The Iterative Cycle
If everything is satisfactory, stop cycling and begin implementation. More typically, identify what to keep, what to improve, and what needs another pass. Each cycle produces more refined understanding and more robust solutions.
We recommend at least three method cycles, regardless of whether the session spans one day or five. Stop when solutions are robust enough, stakeholders are aligned, indicators show consistent improvement, and additional cycles would yield diminishing returns. Recognizing that point of diminishing returns is itself a system behavior worth understanding.
The Desert Flower Lesson
The town of Merredin in Western Australia, a declining desert community, illustrates why evaluation must include stakeholder alignment. Through ELSI resource mapping, the SiD team discovered that Spirulina algae was a perfect match for the town's saline groundwater, abundant sunlight, and cheap land, creating a multi-million dollar business case. But the first attempt failed because stakeholders were not involved in the process. Without a collaborative process that involves the community, one is just building castles in the sky. This lesson made stakeholder engagement central to SiD evaluation.