Is Terraforming Mars Random?

Pairwise Elo from 15 games · 66 player pairs · permutation testing

Data

15 games of Terraforming Mars on Board Game Arena since August 2024 between four players (anonymized as A, J,N, P). Each game has 3 or 4 players and a final score per player.

To turn each multi-player game into something testable, I expand the ranking into all pairwise matchups — a 4-player game gives $\binom{4}{2} = 6$ pairs, a 3-player game gives 3 — for 66 pairs total across 15 games. The outcome for a pair is the score share of player A:

y_{A,B} = \frac{\text{score}_A}{\text{score}_A + \text{score}_B}

So a 90–60 result gives $y = 0.6$ for the winner; a tight 85–84 gives $y \approx 0.503$ . Ties score $y = 0.5$ (no Elo update). Margin-weighted outcomes carry more information for the rating updates than a flat win/loss; the test scoring later is binary, which is a separate choice.

The prediction for each pair, made before the game is added to ratings, is the standard Elo expectation:

\hat{p}_{A,B} = \frac{1}{1 + 10^{(R_B - R_A)/400}}

After each game, each player's rating moves by $K \sum_{j} (y_{P,j} - \hat{p}_{P,j})$ summed over their pairwise matchups in that game, with $K = 24$ and everyone starting at $R_0 = 100$ . Ratings are computed from scratch using only these 15 games, so every prediction is strictly out-of-sample.

Final ratings

101.2

100.7

99.3

98.8

Mean absolute Elo difference at prediction time: 4.98 points. That's small — an Elo gap of 100 implies a ~64% win probability for the favorite; our gap of 4.98 implies only ~51%. So even if Elo correctly ranks the players, the predictive edge it produces over this sample is thin, which limits the power of any test.

Elo over time

Outcome vs. Elo difference

Each dot is one of the 66 pairs, oriented so player A is the higher-Elo player. X-axis is the pre-game Elo gap $R_A - R_B \geq 0$ ; y-axis is the realized outcome $y_{A,B}$ (A's score share). Dots above 0.5 mean Elo was right; below 0.5 is an upset. The dashed curve is the Elo prediction $\hat{p}_{A,B}$ : if Elo is predictive, dots should follow it. If outcomes are random, dots should scatter around 0.5 with no upward trend.

The test

Null hypothesis: Elo has no predictive power on pairwise outcomes. Equivalently, the rating assigned to each player is interchangeable with any other player's rating at the moment of prediction.

I test this by permutation. For each of $N = 10,000$ iterations, I shuffle which pre-game Elo gets assigned to which player within each game, recompute the pairwise predictions, and re-score them. This preserves the game structure (same players, same actual results, same marginal distribution of pre-game ratings) while severing the link between rating and identity. The p-value is the fraction of permuted runs that score at least as well as the observed Elo assignment.

All three metrics compare Elo's probability prediction against the binary outcome (did the higher-Elo player actually win?). Margin still drives Elo updates — only the scoring is binary, matching the question "higher Elo should always win."

Log-likelihood( $\sum y \log \hat{p} + (1-y) \log(1-\hat{p})$ with $y \in \{0, 1\}$ ). Rewards being well-calibrated, not just directionally right. Punishes overconfidence harshly, so with Elo predictions clustered near 0.5 this test has limited resolution.
Accuracy: fraction of pairs where the higher-Elo player won.
Binomial test: among non-tied pairs, is "higher Elo wins" rate significantly above 50%? (One-sided.)

Results

Log-likelihood (permutation)

p = 0.544n.s.

observed = -45.7782, null mean = -45.7598 ± 0.1806

Accuracy (permutation)

p = 0.413n.s.

observed = 50.0%, null mean = 47.7% ± 7.1%

Binomial vs. 50%

p = 0.500n.s.

32/63 = 50.8% [39.8%, 100.0%]

Permutation null distributions

The histograms below show what the test statistics look like under the null hypothesis (Elo shuffled within each game). The vertical line is the observed value with the actual Elo assignment. The p-value is the area to the right of the line.

Log-likelihood null

Accuracy null

Interpretation

All three tests fail to reject H0. The higher-Elo player wins32 of 63 pairs —51%, indistinguishable from a coin flip (binomial p = 0.50). The permutation tests agree: the observed accuracy and log-likelihood sit squarely in the middle of their null distributions.

So: this sample is consistent with Terraforming Mars being randomfor these four players. That doesn't prove it is random — it just means the 15-game history doesn't give Elo enough to work with to detect a skill gap if one exists.

Why so little signal? Final Elos span only ~2.4 points end-to-end (mean pre-game gap across all 66 pairs was 4.98), implying ~51% win probability for the "favorite." Updates from the non-tied games have largely cancelled — everyone has beaten and lost to everyone else in roughly equal measure. The players are statistically indistinguishable given this evidence.

An earlier version of this analysis (with ties counted as full wins for the listed-higher player) found significant results — but those were almost entirely driven by three tied games producing artificial ~24-point Elo swings each. With honest tie handling (y = 0.5, no Elo update) the apparent signal evaporates. A useful lesson in how much a small modeling choice can move a result when the underlying sample is small.

Caveats

Elo computed from scratch. The first few games' predictions are essentially 0.5 because all ratings start equal, which dilutes the test. A proper prior (e.g. each player's BGA Elo at the start of the window) would give the test more power. I don't have that data.
K-factor and initial rating are arbitrary. I picked K = 24and R₀ = 100. A lower K produces an even flatter trajectory; a higher K would inflate the swings from each game. With only 15 games the result is sensitive to these choices.
Margin-vs-binary for updates is a choice. Elo updates use $y = s_A/(s_A + s_B)$ , so a 110–50 game moves ratings more than a 90–85 game. Running Elo with binary updates ( $y \in \{0, 1\}$ ) would dampen swings further and almost certainly push the test toward the same null-consistent conclusion.
The permutation null is conservative. Shuffling within games preserves correlation structure but doesn't account for the fact that ratings evolve. A more principled test might use sequential cross-validation. I think the permutation version is appropriate for the "is there any signal at all?" question.
15 games is not a lot. Even if Elo is "really" right about player ranking, this sample size limits how confidently we can detect it.

All 66 pairs (raw)

Game	Date	A	B	Score	Elo A	Elo B	P(A)	y(A)
1	2024-08-24	A	P	104–84	100	100	0.50	0.55
1	2024-08-24	A	J	104–83	100	100	0.50	0.56
1	2024-08-24	P	J	84–83	100	100	0.50	0.50
2	2024-09-02	A	N	79–74	103	100	0.50	0.52
2	2024-09-02	A	P	79–69	103	99	0.51	0.53
2	2024-09-02	A	J	79–65	103	99	0.51	0.55
2	2024-09-02	N	P	74–69	100	99	0.50	0.52
2	2024-09-02	N	J	74–65	100	99	0.50	0.53
2	2024-09-02	P	J	69–65	99	99	0.50	0.51
3	2024-12-07	A	N	67–67	105	101	0.51	0.50
3	2024-12-07	A	P	67–59	105	98	0.51	0.53
3	2024-12-07	N	P	67–59	101	98	0.50	0.53
4	2024-12-07	P	N	105–88	97	102	0.49	0.54
4	2024-12-07	P	A	105–85	97	105	0.49	0.55
4	2024-12-07	N	A	88–85	102	105	0.50	0.51
5	2025-02-23	P	A	69–65	100	103	0.49	0.51
5	2025-02-23	P	J	69–61	100	96	0.50	0.53
5	2025-02-23	P	N	69–59	100	101	0.50	0.54
5	2025-02-23	A	J	65–61	103	96	0.51	0.52
5	2025-02-23	A	N	65–59	103	101	0.50	0.52
5	2025-02-23	J	N	61–59	96	101	0.49	0.51
6	2025-10-18	A	P	115–107	103	102	0.50	0.52
6	2025-10-18	A	N	115–98	103	99	0.51	0.54
6	2025-10-18	P	N	107–98	102	99	0.50	0.52
7	2025-12-20	N	P	95–93	98	102	0.49	0.51
7	2025-12-20	N	A	95–85	98	105	0.49	0.53
7	2025-12-20	N	J	95–71	98	96	0.50	0.57
7	2025-12-20	P	A	93–85	102	105	0.50	0.52
7	2025-12-20	P	J	93–71	102	96	0.51	0.57
7	2025-12-20	A	J	85–71	105	96	0.51	0.54
8	2026-01-05	P	N	89–83	104	101	0.50	0.52
8	2026-01-05	P	J	89–81	104	92	0.52	0.52
8	2026-01-05	N	J	83–81	101	92	0.51	0.51
9	2026-01-17	P	J	81–79	104	92	0.52	0.51
9	2026-01-17	P	A	81–79	104	104	0.50	0.51
9	2026-01-17	P	N	81–70	104	100	0.51	0.54
9	2026-01-17	J	A	79–79	92	104	0.48	0.50
9	2026-01-17	J	N	79–70	92	100	0.49	0.53
9	2026-01-17	A	N	79–70	104	100	0.51	0.53
10	2026-02-01	P	A	110–85	105	104	0.50	0.56
10	2026-02-01	P	N	110–74	105	98	0.51	0.60
10	2026-02-01	A	N	85–74	104	98	0.51	0.53
11	2026-02-07	J	P	99–93	94	108	0.48	0.52
11	2026-02-07	J	N	99–85	94	95	0.50	0.54
11	2026-02-07	J	A	99–80	94	103	0.49	0.55
11	2026-02-07	P	N	93–85	108	95	0.52	0.52
11	2026-02-07	P	A	93–80	108	103	0.51	0.54
11	2026-02-07	N	A	85–80	95	103	0.49	0.52
12	2026-04-11	A	P	92–85	100	108	0.49	0.52
12	2026-04-11	A	N	92–79	100	95	0.51	0.54
12	2026-04-11	P	N	85–79	108	95	0.52	0.52
13	2026-04-26	J	N	77–77	97	94	0.50	0.50
13	2026-04-26	J	A	77–68	97	101	0.49	0.53
13	2026-04-26	J	P	77–65	97	107	0.49	0.54
13	2026-04-26	N	A	77–68	94	101	0.49	0.53
13	2026-04-26	N	P	77–65	94	107	0.48	0.54
13	2026-04-26	A	P	68–65	101	107	0.49	0.51
14	2026-05-16	J	A	94–86	99	100	0.50	0.52
14	2026-05-16	J	P	94–67	99	104	0.49	0.58
14	2026-05-16	A	P	86–67	100	104	0.49	0.56
15	2026-05-16	N	P	109–97	97	100	0.49	0.53
15	2026-05-16	N	A	109–85	97	101	0.49	0.56
15	2026-05-16	N	J	109–84	97	102	0.49	0.56
15	2026-05-16	P	A	97–85	100	101	0.50	0.53
15	2026-05-16	P	J	97–84	100	102	0.50	0.54
15	2026-05-16	A	J	85–84	101	102	0.50	0.50

Generated 2026-09-12T18:17:40Z. K = 24, R₀ = 100,10,000 permutations.