| SOD | CORE

evaluation-tools

Gigamaps are rich and dense information representations. To work with such semi-organised complexities and develop the ideas within and to work with the interventions or innovations derived from the ZIP-analysis and the creative process, we must use a series of evaluation tools.
IMG 8623

Impact & Threshold Analyses

These analyses were developed especially for SOD by the SOD team.

To help make decisions for the systems interventions, the value of I-points, their implementation and effect, the below evaluation analyses is a good starting point. Evaluate and range all the I-points according to the following criteria:

1.  Systemic impact (leverage)

1.1 Radius of ripple effects
1.2 Short term
1.3 Long term
1.4 Platform effect (to what degree the intervention is creating conditions for further interventions)

2.  Thresholds

2.1 Economic
2.2 Technological
2.3 Cultural
2.4 Organisational
2.5 PESTEL (political, economic, social, technological, environmental, legal)

3. Synergies

3.1 Synergies between intervention and the existing system
3.2 Synergies between interventions
3.3 Orchestration effects (combined effects, high-level synergies)
3.4 Orchestration thresholds (how easy is it to orchestrate the implementation?)

4. Counter (-inntuitive) effects (unwanted and counter intuitive effects)

4.1 Short term
4.2 Long term
4.3 Counter effects between the intervention and existing system
4.4 Counter effects between interventions

5. Resilience

5.1 Resilience toward micro-fluctuations
5.2 Resilience toward macro-fluctuations
5.3 Resilience toward extreme scenarios
5.4 Black swans

Application

A slider type of graphics is useful for evaluation.

The evaluation can be contextualized on a map, here for evaluating relations.

Oslo School of Architecture and Design (AHO) 2013

Threshold Analysis

A simpler score system but more factors are included.

Håvard Banne et. al.

Norwegian University of Science and Technology (NTNU) 2016

Julie Grytten hials 2 2016

Another more detailed evaluation map

Julie Grytten et.al.

Norwegian University of Science and Technology (NTNU) 2016

Julie Grytten et al hials 2016 2 s

There are many ways to apply the principal.

Julie Grytten et.al. 

Norwegian University of Science and Technology (NTNU) 2016

Salto connect systemic evaluation 1 1200

Risk Analyses

Create the normal risk diagram according to two axes: probability and consequences. Select the most important threats from your worst-case scenarios or through group discussions, individual considerations, SWOT analyses, or similar.

Common Risk Evaluation Diagram

Y axis: Consequence

X-axis: Probability

Diagram conducted for PolicySupport.net project with the Policy Lab.

M. Aguirre, R. Mikalauskaite, L Blaasvær

Oslo School of Architecture and Design (AHO) 2011

 

14 Risk management manuela linda et al small

Elaborate Risk Analyses diagram

The normal risk analysis diagram can be adapted in many ways, illustrated in the Kwant Controls project.

Jan Kristian Strømsnes, 2011

Oslo School of Architecture and Design (AHO)

RISK ANALYSIS Kwant Controls 1200

A simple risk analyses diagram

Markus Gundersen

Norwegian University of Science and Technology (NTNU) 2016

Risk MArkus Gundersen 2016 1200

Elaborate Risk analyses

From Salto Connect, second-year bachelor project.

Marie Frogner, Trym Abrahamsen, Oda Heier, Torgeir Hæreid, Julie Sandvoll

Oslo School of Architecture and Design (AHO) 2017

SaltoConnect riskanalyses 2017 1200

Scenarios & Backchecking

Worst-case scenarios

Create two extreme scenarios that are at the edge of what is probable. Develop the scenario so that it shows how your system will survive or adapt to a new situation. For example, imagine that a ship’s bridge is burned out. A small emergency bridge at a different place on the ship allows for emergency operation. The system will survive, though in a reduced state.

This is about the resilience or the robustness of your intervention.

Black swans

A scenario of something unexpected happening to your design. You might already have developed this idea of the black swan in the development of the extreme scenario.

Back-checking

The design process is not linear; it contains periods of incubation and moments of illumination (Hadarmad). Therefore, mapping by itself does not always lead to ideas and innovations. Also, furthering the development of ideas into concepts and designs is a process that is not harnessed by mapping; rather, it is one that most often plays out in conventional design work. It is ok to ‘forget’ the maps and the rich systemic information for times of creative development. To allow this to happen, we have used ‘back-checking’. This means that the concept is back-checked at the checkpoints with the system represented by the gigamaps. This allows for an uninterrupted creative flow.

Back-checking for synergies

Check your idea and intervention regarding the ripple effects on the system. Try to find synergies. Are there new possibilities that are opened by your intervention or idea?

Back-checking to the gigamap

Draw your idea or intervention back into the gigamap, and draw the relations to everything in the map that is influenced, changed or needed for your design to work.

Simple Backchecking

Backchecking of concepts to the gigamap to trace how the concept connects to the system.

Stig M. Henriksen et. al.

Norwegian University of Science and Technology (NTNU) 2016

Stig M Henriksen backchecking hials 2016 s

Reflect and discuss

In the philosophy of science, the discussion is seen as fundamental. Ronald N. Giere (1991), for example, claims that scientific reasoning is based on judgment and common sense. Yet how do we ensure that our reasoning is critical and well-grounded?

We suggest the Pro-et-Contra analyses as a valuable tool for argumentation in the SOD process. The Pro-et -Contra argumentation is best described as DRØFTING by Tor Eivind Førland. He suggests an argumentation tree with pro and counter arguments supporting or weakening the main proposal. These can then again be supported or weakened with pro and con arguments. This can be repeated as far as it makes sense.

“Drøft” is an old Norwegian word used a lot in essay writing as an approach to shed light on an issue from different perspectives. Normally translated to Discuss, the etymology of Drøft is more complex:  Purify; discuss, address and scrutinize from different sides. Related to drip, that is, Drøft actually means “let something pass or drip through, for example, grain through a seed so that chaff and dust stand out”  (Bokmålsordboka , Katlev). Though the etymology is similar the association is less dualistic than the term discuss.

A pro-et contra argumentation tree

Analyses of Graphic User Interface (GUI)

From “Ship Bridge of the Future”

Stig M. Henriksen 2016

NTNU Ålesund (Former HIALS)

Stig M Henriksen hials 2016

The result of such a project is to engage in a disinterest or COLD perspective, reinforcing the distance of the Mertonian CUDOS and strengthening the practice of organised scepticism, a central principle in the western philosophy of science.

In practical terms, it helps to improve the design concept substantially, to find the weaknesses, and to let unheard voices emerge since you will actively search for what’s wrong with your approach, your perspectives and your proposals.

In most cases, this process will not stop the proposals but induce changes to them that might reinforce resilience and realism. In some cases this can work as a creative process.

Førland, T. E. (1996). Drøft. Oslo: Ad Notam.

  1. Make a list of the ‘pro’ arguments that support your idea.
  2. Make a list of the ‘con’ arguments for your project.
    • Select the most important pros and cons and develop pro and con arguments for each of them.
    • Repeat this to a level where you think you have some well-supported arguments.
    • If these analyses result in heavy counter-arguments to your project, you should tweak, redesign and reform your project to address the issue. If this is not possible and you still want to continue with the project, you need to honestly explain the issue to stakeholders.

Pro et Contra analyses. “Drøft” by Førland

This post, Impact and Threshold Analyses, was originally published in 2013.

more | SOD | CORE

About Systems Oriented Design

About Systems Oriented Design

The main mission of SOD is to help designers to become better at dealing with very complex problems. Complex problems are described as problem fields, networks of problems, wicked problems and problematiques.

SOD is open source

SOD is open source

If your project corresponds to all or most of the core parameters, you are welcome to reference it as a SOD project.

The Mind Shifts of SOD

The Mind Shifts of SOD

We need to make several small shifts in our understanding to truly grasp systems thinking in general and to start to become systemic designers.