Wednesday, March 4, 2009

Qualifying Pools: Round Robin vs. Double-Elimination

In a previous post I analyzed the pros and cons of the World Baseball Classic's switch to a 4-team double elimination tournament from a 4-team round robin. Both systems attempt to advance the top two teams into a subsequent round of the tournament. In this post I intend to do a statistical analysis - which system is more efficient at advancing the top teams?

The round-robin style pool is used widely in many international sports, including the World Cup and many Olympic team sports, while the double elimination tournament is relatively little used in major competition. The goal of any tournament is to identify the best team and hence the best tournaments allow the best team to win a high proportion of the time. Is the round-robin pool really the best way to accomplish this?

Simulation

To analyze this, I ran a simulation of both methods - each method playing 1,000,000 tournaments. "True" team winning percentage was determined by a random normal variable with mean .500 and a standard deviation of .100. When two teams played a simulated game against each other, the probability of winning was determined by Bill James' Log 5 Rule.

Simulating the tournament 1,000,000 times, I counted the number of times that the tournament correctly produced the "best" 2 teams (as determined by their "true" winning percentage simulated above). Round-robins sometimes produce ties for the top two spots and in these cases, the ties were broken randomly. The results are below:







As we can see from the chart above, the double elimination tournament correctly advanced the best two teams 32.0% of the time compared to only 30.6% of the time for the round-robin - a small but significant advantage.

Simulation With Correct Seeding

However, that's not all. While the round-robin does not have need for any seeding, the double-elimination tournament is affected by the seeds (which determine the first round matchups). What is the effect of this? To measure the effect I went back to the simulation. I assumed the 4 teams in the pool had true winning percentages of .700, .600, .400, and .300. When the seeds gave first round matchups of .700 vs .300 and .600 vs. 400 in the double elimination tournament, the advantage of the double-elimination tournament increased even more. The results of which are seen below:






As we can see, both styles had a much better chance of advancing the top 2 teams when the teams were pre-selected to have .700, .600, .400, and .300 winning percentages (this versus the much more tightly packed random WPCT assignment in the previous example). However, we also see that the advantage of the double-elimination tournament grows when the teams are properly seeded, making the double-elimination tournament vastly superior to the round-robin tournament when this is the case.

Simulation with Incorrect Seeding

However, when the seeds are poorly selected, we see the reverse effect in action. In this next example we have the same 4 teams, but this time the 1st round matchups are .700 vs. .600 and .400 vs. 300. In this case, the best two teams are matched against each other in the first round. It's not likely that any tournament organizer would purposely seed the teams this way, but of course if the seeds were randomly drawn this combination would be possible. The results of the simulation are below.







Here the effectiveness of the double-elimination tournament is drastically reduced, advancing the top two teams only 45% of the time (as compared to 60% when the seeds are properly ordered). In this case the round-robin tournament is preferable.

This indicates that proper seeding is essential to making a double elimination tournament work at maximum efficiency. This is a drawback compared to the round robin where no seeds are necessary.

Conclusion

So which is better? In the first example, with randomly drawn true winning-percentages, the double-elimination tournament was slightly better than the round-robin and it was much better when the teams were properly seeded. The only case when the round-robin was preferable was when the teams were poorly seeded, with the best two teams facing each other in the first round.

However, in applications where this type of qualifying pool is used, often there is a priori knowledge of the teams' strength and the seeds can be ordered with relative ease. For instance, in the World Baseball Classic there are two clearly "good" teams in each pool and two "poor" teams. The same is often true in Olympic competition and in soccer. This type of seeding is necessary to create the pools themselves so that the pools are evenly divided in skill and all of the best teams are not placed in the same pool.

All in all, the second example shown (with the properly seeded teams) is probably the most realistic approximation to the decisions facing the organizers of these tournaments. Some teams are very good, some are very bad, and generally everybody has an idea which is which. The job of a good tournament is to separate the good from the bad and it's clearly shown in the simulations that the double-elimination tournament does this more efficiently. Not to mention that the double-elimination tournament can do the job in only 5 games (if the meaningless "final" between the two advancing teams is skipped) vs. 6 games for the round-robin.

The hope here is that some people from the IOC and World Cup may read this analysis and reconsider the pros and cons of the double elimination tournament against the round-robin. The double-elimination tournament clearly gives the best teams the best chance to go on - and isn't that what we all want to see?