Show your working
Where do you lose the trust of your participants — when the machine shows its working, or when it doesn't? Notes on the design of Strategy Soup's Taiwan Strait report.
I advised on the design of Strategy Soup’s latest visual report, Rock and a Hard Place: Australia in a Taiwan Crisis. It’s worth a look, less for the scenario than for how it’s built.
The design question I kept coming back to was one I’d put in a LinkedIn conversation:
Where do you lose the trust of your participants? When the AI shows its working, or when it doesn’t?
— James Moorhouse, on LinkedIn
My answer: when it doesn’t. A polished number with no trail gets either blind faith or a shrug. Neither is useful.
What “showing the working” looks like
-
Every move has a reason. Each player’s move is logged with why it was made. You can read why, and argue with it.
-
Chances are visible. Chances are printed on the move cards, and the limitations section lists them. If you think a number is wrong, you can see exactly which one.
-
One run is a story, not a finding. Each tested change was replayed 300 times. Endings are reported as “how many of 300”, not as the future.
-
Pick your depth. Easy, Normal and Deep versions of the same report. The minister and the analyst read the same evidence at different resolutions.
What it lets the report say
The design lets the report say something uncomfortable and still be believed. Australia’s own choices barely moved the ending. Other players’ choices did.
If that came out of a black box, people would dismiss it, and fair enough. Because every move, chance and replay is visible, the argument moves to the right place: are the assumptions right?, not can I trust the machine?
Exploratory v analytical wargaming
Strategy Soup’s reports sit at the exploratory end. They’re good at opening up a problem: surfacing possibilities, assumptions and questions you hadn’t thought to ask. Analytical wargaming asks more of a game: findings you can defend against a clear question, with the method to back them.
There wasn’t enough clarity up front on which of those the game was trying to be, and what makes it a good game versus an analytically useful one. Both are valid. Not deciding early means you end up bolting one onto the other.
One of the most useful things I did was simple. I wrote a two-page A3 player aid: a quick-reference sheet and one worked turn, an escort run to Kaohsiung. Walking a single turn by hand surfaced a real issue with one of the engine’s automatic rules. Thousands of replays would have repeated it without anyone noticing.
The player aid is key. Get it right in early design, and again at the end, so players know what they’re getting into.
If a player can’t follow one turn on paper, they won’t trust the results of thousands.
Limits
Showing your working invites people to pick it apart. That’s the point. The report is upfront that every finding rests on chances set by the designers, and that player briefings haven’t yet been checked against their sources. Being upfront about that is what makes the rest credible.
If you can’t explain a machine’s move in the hotwash, it shouldn’t change your findings. If you can, it might change your mind.
So what
AI shouldn’t smooth away the friction. It should surface the human friction as a key finding, and AI can help us do that at volume.
A wargame is worthless if your participants don’t buy in, it isn’t recorded, and it isn’t used to inspire change.
Showing the working is how you get the first of those three.