How poker solvers evolved: from Los Alamos to CFR, Libratus and today’s solvers

The ideas behind poker solvers are almost a hundred years old, and two of the people who shaped them worked on the atomic bomb. The popular version of that story gets a few things wrong, and one of the mistakes matters for how you use a solver today: modern solvers do not work the way Ulam’s Monte Carlo method does.

So this is the history with only the parts I can point to a paper or a documented match for. At the end I show what all of it produces in the app: a preflop chart, a postflop solve and a report across 184 flops.

1928 and 1944: game theory, with poker in it

In 1928 John von Neumann proved the minimax theorem. In a two-player zero-sum game, if both players are allowed to mix their strategies, there is a value of the game: a strategy that guarantees you at least that value, whatever the opponent does. That guarantee is the same idea we call GTO today. It says you cannot be beaten, and it says nothing about how to beat any particular player.

In 1944 von Neumann and Oskar Morgenstern published Theory of Games and Economic Behavior. Poker was one of their examples, in a very simplified version: each player gets a random number instead of cards, and there is a single round of betting. The model did not prove that mixed-strategy equilibria exist, because the minimax theorem had already done that. What it did show is that bluffing belongs in optimal play, which every poker player still has to accept.

In 1950 John Nash extended the equilibrium idea to games with any number of players. The guarantees stay strongest for two players though, and poker at a 6-max table has six.

Los Alamos: Ulam, solitaire and Monte Carlo

Stanisław Ulam grew up in Lwów, then in Poland, and spent the war on the Manhattan Project at Los Alamos, where von Neumann also worked. He wrote about the poker games there in his autobiography, Adventures of a Mathematician.

The Monte Carlo method came from a card game too, but not poker. Around 1946, recovering from an illness, Ulam wondered how likely a game of Canfield solitaire was to come out. The combinatorics were hopeless, while dealing it many times and counting was easy. With the first electronic computers that became practical, and together with von Neumann he turned it into a way to study neutron transport for the weapons programme. Nicholas Metropolis suggested the name, after the casino. Metropolis and Ulam published The Monte Carlo Method in 1949.

If you can’t calculate a probability, sample it. The error shrinks with the square root of the number of samples. If the true answer is around 50% and you run 10,000 random trials:

That is how many equity calculators work. They deal thousands of random boards for two hands or two ranges and count who wins. Four times more boards halves the error. That is where Monte Carlo lives in poker software today: in equity estimates, and in sampled versions of solver algorithms. It is not how a solver finds a strategy.

Los Alamos also ran one of the first chess programs, in 1956, on the MANIAC I computer. It played on a 6×6 board without bishops because the machine was too small for the full game. Chess, not poker, was the target for computers for the next forty years.

2007: counterfactual regret minimization

The algorithm behind modern poker solvers is counterfactual regret minimization, CFR. Martin Zinkevich, Michael Johanson, Michael Bowling and Carmelo Piccione from the University of Alberta published it in 2007.

The program plays the game against itself over and over. At every decision point it asks how much better it would have done with each of the other actions, and that difference is the regret. On the next iteration it plays actions in proportion to their positive regret:

Actions that would have won more get played more. Actions with no positive regret are not played, unless no action has any. In a two-player zero-sum game the average strategy over all iterations converges to a Nash equilibrium. That is why a solver shows you mixed strategies. A hand split between two actions is not an undecided solver; it is the average of millions of adjustments settling down.

Then came faster variants. In 2014 Oskari Tammelin introduced CFR+, which converges much faster in practice. Monte Carlo CFR (Marc Lanctot and co-authors, 2009) samples parts of the game tree each iteration instead of walking all of it. So sampling does appear inside solvers, where it speeds CFR up instead of replacing it. Commercial desktop solvers such as PioSOLVER, GTO+ and Simple Postflop appeared in the mid-2010s. Their exact algorithms are not always public, but the family is CFR.

2015 to 2019: poker AI beats humans

• In 2015, the Alberta group published in Science that heads-up limit hold’em is essentially weakly solved. Their program, Cepheus, plays so close to equilibrium that a human could not show it is beatable over a lifetime of play. It used CFR+.
• In 2017, Libratus, written by Noam Brown and Tuomas Sandholm at Carnegie Mellon, played four professionals in heads-up no-limit hold’em: Jason Les, Dong Kim, Daniel McAulay and Jimmy Chou. About 120,000 hands over 20 days in Pittsburgh, and Libratus won clearly. The same year the Alberta program DeepStack also published wins against professionals in heads-up no-limit.
• In 2019, Brown and Sandholm did it again with Pluribus, this time in six-player no-limit hold’em, published in Science. Six players means the equilibrium guarantees from the two-player case no longer apply, which is why that result mattered.

Two corrections to the old version of this article. It said the Libratus match was “tens of thousands of hands”; it was about 120,000. And it named a privately built 2013 program as “the first modern poker solver”. I can’t find a source for that, and CFR itself is from 2007, so it is gone.

What the output looks like today

You don’t need to run CFR yourself to use it. This is what comes out of it in the app, from the simplest spot to a whole report.

A preflop chart. Normal NL200 GG cash, 100bb, 6-max, no ante. Everyone folds to the button, who opens to 2.5bb or folds. The solver raises 46.5% of hands, 617 of the 1,326 combos, and folds 53.5%, 709 combos.

617 + 709 = 1,326, every starting hand. Hands like J3s, Q8o and A2o are split between raising and folding: that is the mixing CFR produces when two actions are worth about the same.

A postflop solve. Same cash game. HJ opens to 2.5bb, CO calls, the blinds fold. The pot on the 8♣︎ 8♥︎ 6♦︎ flop is:

The HJ checks 72.6% of its range, and when it bets it mostly goes big: 23.0% at 150% of pot against 2.1% and 2.3% for the smaller sizes. Look at AA in the matrix, which mixes checking and the big bet. That is the 1944 lesson in modern form, with strong hands and bluffs sharing the same actions so the opponent can’t read the size.

A report across flops. One flop is one example. An aggregated report solves the same preflop spot on a whole set of flops and puts them side by side. This one comes from our library: CE-Symmetric tournament, 50bb, 9-max, 1bb big-blind ante. The small blind raises to 4bb, the big blind calls, a 9bb pot with 46bb behind. Every board in it was solved with the same bet menu, 15%, 30%, 60% and 120% of pot, and that is what makes the columns comparable at all. The chart shows the small blind’s first decision, out of position after its own raise, on all 184 flops, grouped by the highest card.

How often the small blind bets barely moves with the high card: 65.4% of the time on ace-high flops, 62.7% on 7-high. The size moves a lot. On ace-high it takes the cheapest bet, 15% of pot, 30.80% of the time, and the 60% size only 4.55%. On 7-high the small bet almost disappears, 2.85%, and the 60% size takes over at 23.70%. High cards let the raiser bet cheap with its whole range; low connected boards need real money behind the bet. No single flop teaches you that, because the pattern only shows up across many solves. Seventy years ago it would have needed a Los Alamos computer and a Monte Carlo estimate. Now it is a click.

How to use this history

• Monte Carlo answers “how often does this win?” It is behind equity calculators.
• CFR answers “how should I play so I can’t be exploited?” It is behind solvers, and the mixes you see are averages that converged.
• The equilibrium guarantees are clean heads-up. Multiway, treat a solver strategy as a very strong baseline and not a proof.
• GTO is a baseline, the same way it was in 1928: the strategy that can’t be beaten. Which is a different strategy from the one that beats a specific player most.

The longer version, with more of the Los Alamos story, is in our video:

Cheers! 🙂