Step 5: Evaluate and Iterate

Where This Fits

This is the fifth and final step of the SiD five-step method (Goals and Indicators, System Mapping, System Understanding, Solutioning and Roadmapping, Evaluate and Iterate). You have set goals, mapped the system, developed understanding, and designed solutions. Now you check: did we actually get where we wanted to go?

This step is both a checkpoint and a launchpad. It closes one cycle and opens the next.

---

What Evaluation Does

The evaluation step reflects on the activities of the cycle, compares solutions to goals, and determines whether they were successful. It also answers the crucial question: is another method cycle necessary?

This step is just as much a combination of creativity and analysis as the others. It is not a bureaucratic formality. For larger projects, it can be valuable to set up an evaluation system that tracks progress and provides running feedback on performance.

The evaluation stage may reveal that goals or boundaries were set incorrectly, resulting in their adjustment before executing another cycle. It is also the point where new research questions may be formulated for separate trajectories or future use.

---

Evaluation Changes with Each Cycle

Evaluation has a different nature at each stage of development.

In early cycles, evaluation may only estimate whether the process is heading "in the right direction." In later stages, it may involve extensive checking of details against predetermined edge conditions, indicators, and goals.

Setting up a good evaluation framework in advance helps manage the quality of the process. A shortcut: involve a person who is critical and experienced in the matters at hand, and have them serve as an outside critic of the solutions and decisions made.

---

Methods of Evaluation

There are many evaluation methods, and most disciplines have their own preferred tools. Covering all of them goes beyond the scope of this book. Here are several that we have found helpful in most circumstances.

For all of them, the main question is:

Did we achieve our goals with the chosen solutions, within the chosen boundaries?

A second question, no less important:

What are the potential unintended consequences of the solutions on the system and all stakeholders, and what is their impact?

Evaluation Matrix

Make a list of all goals and edge conditions. Score the found solutions from low to high for each aspect. This is especially helpful for comparing competing solutions and identifying the strengths and weaknesses of each.

The matrix may reveal that the best path forward is a hybrid: combining parts of competing solutions into something new. Creating the evaluation matrix for the first time may also surface a new set of edge conditions and desires that become part of the goals in a later stage.

System Impact Check

What impact will the chosen solutions have on the system as a whole? Work with the entire team to formulate pitfalls, possible externalizations, or system dynamics that may influence the solution positively or negatively. See if you can make the solutions more resilient. It can help to draw additional system maps that trace the impact of a solution through the system.

External Evaluation

For larger projects, form a panel of experts to maintain quality at key points. External evaluators bring fresh eyes and almost certainly have insights not yet considered. This does not replace the necessity of critical self-reflection at each stage.

---

Iterate or Move On?

The evaluation step is the moment when you determine whether to run another cycle or proceed to the next phase of the process.

In our experience, during early cycles of the SiD method, teams often realize that goals should be redefined, that additional maps are needed to answer questions, or that they still feel "lost" in the project's complexity. If so, continue cycling. If you are confident (and enthusiastic) about your solutions, move on to the next SiD process step, covered in the next chapter.

---

Mid-Session Harvests

An evaluation stage in the middle of SiD sessions helps check direction, cull excess ideas, and refocus. This should be done several times during sessions to effectively cycle through generating ideas and selecting among them.

During facilitated SiD sessions, it helps to gather the group every half day to:

• Recap the activities
• Re-list the goals
• Collect the best ideas that have emerged so far

Write these down on a sheet to return to later, as a collection of "best ideas at the time." This tracks how the group's perspective shifts as the session progresses.

It also helps to evaluate the process itself at this point. Review the goal, the mapping, whether the mapping is understood, what solution ideas have come forward, and whether the roadmap is complete or missing something.

A rule of thumb: Perform the method cycle at least three times during a session, whether it is one day or five days. The evaluation is also performed three times. A scoring matrix or similar tool helps evaluate both process and content with each cycle.

Another meaningful evaluation: at the end of each day and each session, ask every participant to personally evaluate the process. This strengthens understanding of each group member's perspective and provides space for focused dialogue.

---

Case Study: The Desert Flower

Adapted from a project executed in 2000-2001 in Western Australia.

The question: What to do with a remote desert town in decline?

The city council of Merredin, a small desert town in Western Australia, faced the renovation of a historical pumping station on the outskirts of town. Government funds covered restoration, but not enough to do something useful with it. The town had no funds of its own. The question: what to do?

Mapping the System

A small university team visited and investigated beyond the pump station itself. The town had bigger problems: economic and social decline, faltering infrastructure, young people leaving for better prospects elsewhere.

Using a quick-scan investigation organized by what we later named the ELSI-8 categories (Energy, Land use, Materials, Ecosystems, Species, Culture, Economy, Health and Happiness), the team examined each category over about three days, held interviews with local residents, and formed a coherent picture of the town's issues and assets.

Understanding the Hidden Resources

Historically, Merredin existed as a through-stop on the way to inland mines. It served as a truck-stop and supply station with restaurants and motels, hosted a small train station, and maintained a water pumping station bringing fresh water to the goldfields. Over time, agriculture developed (wheat and grains under irrigation), and the town branded itself around healthy food.

But trucks got bigger and faster, reducing their numbers, while the town kept growing. A system growing on a limited resource eventually runs into trouble. The largest factory closed. Youth left. The historic theater shut its doors.

An additional problem compounded things: an unusually high groundwater table, strange for a desert town. Merredin sits in an invisible underground valley. Rain falling for miles around flows underground toward the town, picking up salt from the desert. The high saline groundwater seeps into roads and foundations, where it evaporates under the sun, leaving salt behind. The salt cracks foundations and roads, causing a constant maintenance nightmare and heavy costs.

The Asset Matrix

With rough analysis complete, the asset matrix could be filled: abundant sunlight, available labor, a "health" brand image, access to strong road and rail logistics. Other assets: saline groundwater and cheap land. The groundwater's regeneration capacity offered some sustainable use potential. Culturally, the town had interesting historic buildings. Despite a lack of financial resources, there were plenty of eager people wanting to start new enterprises, but few ideas for a viable economic model.

From Understanding to Solution

Lining up problems and resources made something clear: what was a problem in one light could be seen as a resource in another. The town needed a new economic engine. Depending on passing trucks for sustenance was a weak proposition, leaving the town vulnerable to mine management decisions, with a very low resilience score. To become healthy and flourishing, the town needed autonomy in its own resources, becoming a destination rather than a through-stop.

The search for a solution started by "cross-breeding" every asset with every other. Could abundant sunlight desalinate the saline groundwater? Too expensive and slow. A health spa using salt water as a selling point? Not a solid business case.

Then came a shift in strategy. The saline groundwater, land, sunlight, and old pumping station were all assets. What catalyst element, added between them, would achieve the goal?

Progressing through the ELSI-8 categories, the search focused on biological resources. A thriving biological system generates value without human intervention when conditions are right. If something could grow using abundant sunlight, semi-arid land, and saline groundwater, the pieces would fit.

The gem turned out to be Spirulina algae. Spirulina thrives in saline water and sunlight, grows rapidly in open-air shallow basins that require exactly what Merredin had in abundance: surface area and land.

The Concept

Spirulina serves multiple markets: edible blue coloring agent for cosmetics, source of vitamin B12, and health-food product. It is sold freeze-dried, which stores well. The pumping station would become headquarters. The old water agitator could serve as a demonstration tank. Empty lands around it would host algae growing pools. The algae would be pumped in, freeze-dried, packaged, and shipped directly to the highway.

The business case showed a multi-million dollar annual enterprise, employing dozens of people, with a return on investment under five years.

Evaluation

The Spirulina plant addressed every goal. It provided economic foundation, made the town visible along the highway, created employment, resolved the saline groundwater issue, and used a completely renewable ecological resource that aligned with the town's health brand. The plan was presented to the city council.

The Lesson

Two lessons came from this project.

First: the method works. This was one of the earliest projects using an integrated approach. It worked beyond expectation and provided the conviction to develop what became SiD.

Second: stakeholder involvement is not optional. The town officials hardly took the plan seriously. In 2001, "sustainability" was a word rarely used by general audiences. Algae seemed outlandish. Nobody checked the numbers. The critical mistake: developing the project without involving stakeholders in the process. When the plan was presented, the officials felt uninvested. To them, it was a crazy idea that dropped out of the sky. They rejected it, justifiably. Without a collaborative process involving the community, one is building castles in the sky.

A consolation: a decade later, the feasibility study was redone. The time was ripe for broader acceptance. This pattern recurs; it usually takes about a decade for innovative solutions to move from concept to implementation. The new study confirmed the project is still profitable and feasible, and the town is now looking for investors.

The early SiD approach worked beautifully, quickly, and efficiently. But failure to involve stakeholders meant the project was never executed. Since then, stakeholder involvement and communication strategy have been central elements of the SiD system.

---

Takeaway

Evaluation is not the end. It is the hinge between cycles, the moment where honesty about what worked and what did not makes everything that follows better. Do not skip it. Do not rush it. Use it to sharpen goals, cull weak solutions, and build confidence. And never forget that the best technical solution in the world is worthless if the people it affects were not part of creating it.

Next: The SiD Process chapter covers how to structure the full project around the method, including team formation, stakeholder engagement, and communication strategy.