Method “O-18”, revision 1.0
Method passport: 18 review procedures
Eighteen lenses are not a shop-window list of techniques but a division of labour: every critic gets their own set, without duplicates, and each procedure is expected to produce something specific. Below is every one of them: where it comes from, what error it looks for, who gets it, what it produces and where it does not work.
That last part is not modesty for show. A method with no stated boundaries cannot be checked: any finding can be credited to a lens, and any missed disaster written off as “not that case”.
Four lenses are mandatory in every matter — steelman, pre-mortem, key assumptions and chain-of-verification. The rest are distributed across the critics by affinity to their role, without duplicates: a panel of 3 does not cover all 18, and we say so plainly on the method page.
Steelman
What it looks for. Your position is first restated more strongly than you put it yourself: attacking a weak version of a decision proves nothing.
Who gets it. every critic. What it produces: a reasoned passage.
Where it comes from. Rapoport's rules as set out by Daniel Dennett (Intuition Pumps, 2013) · source
Where it does not work. A strengthened version can drift from what you actually meant: if the wording changes in substance, the criticism misses.
Pre-mortem (Klein)
What it looks for. The panel assumes the decision has already failed a year on and writes the story of that failure — not a list of what might go wrong.
Who gets it. every critic. What it produces: a reasoned passage.
Where it comes from. Gary Klein, “Performing a Project Premortem”, Harvard Business Review, 2007 · source
Where it does not work. An invented failure story convinces regardless of how likely it is: a convincing story is not a probability.
Key assumptions
What it looks for. Every load-bearing assumption is written out, including the unspoken ones, and for each — what follows if it turns out to be false.
Who gets it. every critic. What it produces: a structural artefact (table, matrix, diagram).
Where it comes from. Structured analytic practice (key assumptions check), fixed in the US intelligence community standard ICD 203 · source
Where it does not work. An unspoken assumption is unspoken precisely because it is invisible: nobody can guarantee this list is complete.
Falsification (Popper)
What it looks for. The panel states the observation that would refute the position, and the cheapest way to obtain it.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Karl Popper, The Logic of Scientific Discovery, 1934 · source
Where it does not work. For decisions about people and negotiations a refuting observation often does not exist at all — then the lens yields an honest “there is nothing to test”.
Toulmin warrant attack
What it looks for. The argument is broken into three parts: data, claim, and the unspoken warrant that ties them. The attack lands on the warrant — decisions usually rest there, not on the numbers.
Who gets it. one of the critics. What it produces: a structural artefact (table, matrix, diagram).
Where it comes from. Stephen Toulmin, The Uses of Argument, Cambridge University Press, 1958 · source
Where it does not work. It works on the text of the argument: if the decision was made on instinct and the argument written afterwards, what breaks is the write-up, not the decision.
Inversion (Munger)
What it looks for. The conclusion is inverted: what must be true for the decision to be right. Every such condition is tested separately — usually one of them does not hold.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Charlie Munger's device (in his telling, from the mathematician Carl Jacobi: “invert, always invert”); we have no primary publication to link
Where it does not work. The list of conditions is drawn up by the same review that tests it: a missed condition will not surface by itself.
Quality of information
What it looks for. Every fact is measured four ways: where it came from, how fresh it is, whether the source has an interest, and whether it can be checked. Unverified is marked as unverified.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Richards Heuer, Psychology of Intelligence Analysis, CIA, 1999 · source
Where it does not work. We assess what we were shown: a forged document looks like a document, and an omission leaves no trace.
What If? (the surprise has already happened)
What it looks for. What you call impossible is taken as already having happened, and the panel explains by what route it came about.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. The “What If?” technique from Heuer's structured analysis and ODNI guidance · source
Where it does not work. Any event can be explained — a convincing route does not make it likely. The lens gives readiness, not a forecast.
High impact, low probability
What it looks for. Two to four events whose probability is low and whose damage is devastating are named outright, with a plain answer for each: are you ready or not.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. ISO/IEC 31010 “Risk management — risk assessment techniques” · source
Where it does not work. Everyone estimates the probability of rare events badly, ourselves included: the numbers here are orders of magnitude, not calculations.
Good faith of the source
What it looks for. If the data came from an interested party, the panel works out what that party gains from you believing it.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Denial & deception — a section of Heuer's structured analysis · source
Where it does not work. A source's interest does not make its words false: the lens raises the bar for trust, it does not pass sentence.
Competing hypotheses (ACH, Heuer)
What it looks for. Alternative explanations are tested on equal terms with the main one: which version survives the evidence rather than which we liked first.
Who gets it. one of the critics. What it produces: a structural artefact (table, matrix, diagram).
Where it comes from. Richards Heuer, Analysis of Competing Hypotheses (ACH), CIA, 1999 · source
Where it does not work. The method compares ONLY the hypotheses that were named: a version nobody thought of is not caught by the matrix.
Four points of view (stakeholders)
What it looks for. For every key participant: their goal, their version of events, and what they will do when your decision affects them.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Standard practice in stakeholder analysis; there is no single primary source
Where it does not work. We model other people's motives from your account of them — that is one side's version, not testimony from the other.
Failure mode map (FMEA)
What it looks for. For a plan or a process, failure modes are walked through: how likely, how damaging, and whether it would be noticed in time.
Who gets it. one of the critics. What it produces: a structural artefact (table, matrix, diagram).
Where it comes from. FMEA — the engineering failure mode and effects analysis, fixed in reliability standards (including IEC 60812) · source
Where it does not work. It analyses the process as described: whatever happens outside the written procedure never reaches the table.
Bow-tie (ISO 31010)
What it looks for. The principal risk is laid out as a chain — causes → event → consequences — with the barriers that actually exist on each side.
Who gets it. one of the critics. What it produces: a structural artefact (table, matrix, diagram).
Where it comes from. ISO/IEC 31010, the bow-tie technique · source
Where it does not work. A barrier on the diagram and a barrier in real life are different things: we take the barrier's effectiveness from your words.
Base rate and the outside view (Tetlock)
What it looks for. Estimates and forecasts are checked against the base rate in a similar class of cases: how it usually ends for others, not how it feels from inside.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. The outside view and base rates — Kahneman and Tversky; forecast calibration — the work of Philip Tetlock · source
Where it does not work. The review picks the comparison class: too narrow gives too few cases, too broad gives someone else's statistics.
Scan for biases in the position
What it looks for. Confirmation search, anchoring, sunk costs, survivorship — named with the exact quote from your own words, not as a diagnosis.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. The catalogue of cognitive biases in analytic practice (Heuer, ICD 203); we apply no diagnostic instruments · source
Where it does not work. This assesses the text, not the person: a sign of bias in the wording does not mean the decision itself was biased.
Chain-of-Verification
What it looks for. The critic asks verification questions about their own key claims and answers them separately — the cheapest known way to cut invented facts.
Who gets it. every critic. What it produces: a reasoned passage.
Where it comes from. Chain-of-Verification — a technique from language-model research (fewer invented facts when a model separately checks its own claims) · source
Where it does not work. The model checks itself: a systematic error shared by question and answer survives this pass — that is what the second-vendor review is for.
Code and engineering
What it looks for. For technical decisions: extreme inputs, race conditions, boundaries, dependency failure, and what happens when load grows tenfold.
Who gets it. one of the critics. What it produces: a reasoned passage.
Where it comes from. Engineering practice in failure analysis; there is no single primary source
Where it does not work. Without access to the system itself the review works from your description: what is not in the account will not be seen.
What is deliberately absent
A single score for a decision — “74 out of 100” — is not part of the method and never will be: it is false precision that hides what exactly is being contested. Instead you get a verdict, a severity for every finding, conditions for revision and the panel's map of disagreement.
Where the method is weaker than we would like it is set out on a separate page. Primary sources with live links are on the method page.