evaluation-tools

These analyses were developed especially for SOD by the SOD team.

To help make decisions for the systems interventions, the value of I-points, their implementation and effect, the below evaluation analyses is a good starting point. Evaluate and range all the I-points according to the following criteria:

1.  Systemic impact (leverage)

1.1 Radius of ripple effects
1.2 Short term
1.3 Long term
1.4 Platform effect (to what degree the intervention is creating conditions for further interventions)

2.  Thresholds

2.1 Economic
2.2 Technological
2.3 Cultural
2.4 Organisational
2.5 PESTEL (political, economic, social, technological, environmental, legal)

3. Synergies

3.1 Synergies between intervention and the existing system
3.2 Synergies between interventions
3.3 Orchestration effects (combined effects, high-level synergies)
3.4 Orchestration thresholds (how easy is it to orchestrate the implementation?)

4. Counter (-inntuitive) effects (unwanted and counter intuitive effects)

4.1 Short term
4.2 Long term
4.3 Counter effects between the intervention and existing system
4.4 Counter effects between interventions

5. Resilience

5.1 Resilience toward micro-fluctuations
5.2 Resilience toward macro-fluctuations
5.3 Resilience toward extreme scenarios
5.4 Black swans

Create the normal risk diagram according to two axes: probability and consequences. Select the most important threats from your worst-case scenarios or through group discussions, individual considerations, SWOT analyses, or similar.

Worst-case scenarios

Create two extreme scenarios that are at the edge of what is probable. Develop the scenario so that it shows how your system will survive or adapt to a new situation. For example, imagine that a ship’s bridge is burned out. A small emergency bridge at a different place on the ship allows for emergency operation. The system will survive, though in a reduced state.

This is about the resilience or the robustness of your intervention.

Black swans

A scenario of something unexpected happening to your design. You might already have developed this idea of the black swan in the development of the extreme scenario.

Back-checking

The design process is not linear; it contains periods of incubation and moments of illumination (Hadarmad). Therefore, mapping by itself does not always lead to ideas and innovations. Also, furthering the development of ideas into concepts and designs is a process that is not harnessed by mapping; rather, it is one that most often plays out in conventional design work. It is ok to ‘forget’ the maps and the rich systemic information for times of creative development. To allow this to happen, we have used ‘back-checking’. This means that the concept is back-checked at the checkpoints with the system represented by the gigamaps. This allows for an uninterrupted creative flow.

Back-checking for synergies

Check your idea and intervention regarding the ripple effects on the system. Try to find synergies. Are there new possibilities that are opened by your intervention or idea?

Back-checking to the gigamap

Draw your idea or intervention back into the gigamap, and draw the relations to everything in the map that is influenced, changed or needed for your design to work.

“Drøft” is an old Norwegian word used a lot in essay writing as an approach to shed light on an issue from different perspectives. Normally translated to Discuss, the etymology of Drøft is more complex:  Purify; discuss, address and scrutinize from different sides. Related to drip, that is, Drøft actually means “let something pass or drip through, for example, grain through a seed so that chaff and dust stand out”  (Bokmålsordboka , Katlev). Though the etymology is similar the association is less dualistic than the term discuss.

The result of such a project is to engage in a disinterest or COLD perspective, reinforcing the distance of the Mertonian CUDOS and strengthening the practice of organised scepticism, a central principle in the western philosophy of science.

In practical terms, it helps to improve the design concept substantially, to find the weaknesses, and to let unheard voices emerge since you will actively search for what’s wrong with your approach, your perspectives and your proposals.

In most cases, this process will not stop the proposals but induce changes to them that might reinforce resilience and realism. In some cases this can work as a creative process.

This post, Impact and Threshold Analyses, was originally published in 2013.