FIFA rankings and football's interestingness

This week, the group stages of UEFA qualifying for the 2014 football World Cup came to an end. The draws for these groups were based on the FIFA world football rankings we know and love.

I read an article by Gabriele Marcotti, who sometimes writes reasonable things, written before the final round of qualifying matches. I've done a semi-close read of it here, followed by some of my own stuff. I'm focusing on the bit of the article that deals with the UEFA section of World Cup qualifying.

Here we go - Marcotti quotes in italics
"UEFA get 13 spots in the World Cup. Nine qualify directly and another four via two-legged playoff."

Fair enough.

"And of the 13 highest-ranked nations who went into qualifying, just two -- Denmark (eighth at the time) and the Czech Republic (13th) -- are out of the running."

Seems to make sense.

"Three of the top six -- Germany (second), Italy (fifth) and Holland (sixth) -- have already clinched a place. Another two -- Spain (first) and England (third) -- should also win their groups and guarantee passage."

This was written before the final round - Spain and England did indeed win their groups.

"The one team that have significantly underperformed are Portugal -- fourth at the time but who have had a tricky qualifying campaign -- though they are heavily favored [sic] to make the playoffs."

"Fourth at the time but have had a tricky qualifying campaign" doesn't seem to mean anything. Surely fourth because they've had a tricky qualifying campaign.

"Greece (10th at the time), Russia (ninth), Sweden (12th) and France (11th) have already made at least the playoffs, while, in addition to Portugal, Croatia (seventh) are also just about there."

This is where it really starts going downhill, as it exposes some clumsiness. Marcotti is clearly working with the rankings published in August 2011, rather than on 5th September 2011 (first qualifiers took place on 7th September).

"Some will point to Switzerland (14th) and Belgium (29th) and the fact that they've clinched qualifying as evidence of surprises. But really, Switzerland’s track record (and a fairly cream-puff group) and Belgium's deep and talented squad suggest that many saw this coming."

These might be points worth making but they have nothing to do with the rankings, but shoving them into a piece which you’re trying to write to say “the rankings work” (more on that later) doesn’t strike me as being overly scientific.

"Bosnia (19th) is a great tale given the nation's history." Actually 17th if you use the right rankings.

"They're guaranteed the playoffs and could yet win the group, but again, looking at the talent in the squad and the relative weakness of the group, it's not really a shock."

"It's not really a shock" is again something which makes any look at the rankings irrelevant if you can just claim that any time they don’t work then you might as well not look at them at all. It's a bit like selective reading of the bible (which I'm not suggesting Marcotti does).

"Iceland -- 46th in Europe, 130th in the world when qualifying kicked off –" actually 44th and 118th. "have a shot at the playoffs, which would be remarkable for a nation of just 300,000."

Seems to be a fair point – the least populous nations getting to playoffs in recent World Cup or Euro qualifiers would be Montenegro and Estonia, both of whom have populations well above 300,000.

"But given the seeded playoffs and the possibility of facing France or Portugal, the shock would be if their journey did not end there."

The seeded playoffs mean that Iceland can’t face France. And suggesting that the seeded playoffs mean it would be a shock if Iceland didn’t win gives an awful lot of credence to the ranking system which, elsewhere, can be dismissed simply by taking a look at talent in a squad. Or, on the other hand, you're assuming the effectiveness of the ranking system within your argement for its effectiveness.

"What all [Marcotti's all includes similar analyses of the other continental federations] this suggests is that maybe the FIFA rankings aren't as absurd as they sometimes seem. Or, at least, they weren't this time around. The vast majority of the teams that were supposed to make it did make it."

Clearly, Marcotti is confusing the rankings, which are supposed to recognise previous achievement, with some kind of tool for predicting future achievement.

I don’t think the word “supposed” is appropriate in this case, because he’s taken the rankings at a time which is irrelevant for the organisation of the competition, August 2012. (That he meant to take September 2012 doesn't matter in this instance.)

The only time when the rankings actually influence who teams face is at the draw, when the teams are sorted into pots – this is a deliberate attempt to improve the best-ranked teams’ chance of qualification by keeping them apart. If there is any “supposed to” anywhere, then it is here.

Of the nine teams that went into pot 1, five have qualified. This is about 55.56 % - a simple majority but not a vast one.

Of the nine teams that went into pot 2, one has qualified and two have made the playoffs, that’s 33% (a minority) doing anything remotely reasonable.

What if we try to be a bit more scientific?
As a little experiment, I did the following:

Gave all teams a score according to the pot they were in at the draw (pot 1 is 1 point, pot 2 is 2 points and so on).

Looked at the possible total points for all group winners and playoff reachers. Here, I doubled the points for a group win as a mechanism for weighting this more than reaching the playoffs. I think this works.

So the lowest possible total is ((9*1)*2)+(8*2) = 34

The highest possible total is ((8*6)+5)*2+(8*5) = 146

The lowest total would be if all pot 1 teams won their group and all playoff reachers came from pot 2. The highest would be if all pot 6 teams won their groups and all playoff reachers (and the group winner from the 5-team group) came from pot 5.

This gives us a possible range of 146 – 34 = 112 points.

The score of the actual group winners and playoff reachers comes out as 52.

This is 18 above the lowerst possible, in a range of 112, putting it at the:

18/112 = 16.07th percentile.

This gives us something I feel can be a useful figure for measuring how closely the rankings have “predicted” the results of qualification. I’m not suggesting any “supposed to” here, but a higher percentile would indicate that lower-ranked teams did better than the rankings would have “predicted”.

It’s up to the observer whether he wants to question the ranking system or applaud the interestingness of international football for not always having teams perform exactly as well as they have done over the past four year period used for ranking-point calculation.

If we get a bit more detailed and use the UEFA ranking of each team, rather than just a pot (so Spain are 1, Netherlands are 2 and so on), then the result comes out around the 13.49th percentile. And if you take the FIFA ranking, you get the 9.79th percentile.

This would seem to suggest that the more detailed you get (i.e. the more you reflect the relative strengths of the teams in the points you assign them), the better the rankings are at predicting the results and the less interesting the qualifiers become.

It’s only a one-off though, so all we can see at this stage is that in this case the better-ranked teams in each pot did better than the worse-ranked teams from the same pot. Generally speaking.

Anyway, having a nice figure like this also means that we can look, for example, at what the figure is if we take the rankings just before the qualifiers (not Marcotti’s August ones, the September ones).

If we do this and go through all the same calculations (being a little careful because, for example, there isn’t a team in the top 9 (pot 1) in every group), then we get the actual results as being at the 10.48th percentile. This is a marked improvement over the 16.07th percentile we had when we took the rankings at the draw.

Similarly, the percentiles when using the UEFA and FIFA rankings also both improve, from 13.49 to 8.84 and from 9.79 to a very respectable 6.57.

So the main issue this highlights is the fact that the draw is made based on rankings over a year before play begins, by which time the rankings have changed considerably. And, as you would probably expect, the rankings immediately before the games start are a better indicator than those at the time of the draw.

Sweeping statement check
It also lets us look sensibly at some things Marcotti said. He refers to Switzerland’s group as “fairly cream-puff”. Whilst the group as a whole is, by total ranking (at time of first game) of all teams, 6th out of 9 and slightly worse than average, Switzerland were the highest ranked side in it, despite being in pot 3.

This is, for my money, far more relevant than overall group strength because it explains why Switzerland were the ones to emerge from the (slightly) weak group.

Bosnia’s group is also said to display “relative weakness”. According to ranking, it is indeed 8th of 9, but, more importantly, Bosnia are the second best-ranked team in it, so have only actually overcome Greece (difference of 8 ranking points) to win the group.

And Belgium, far from simply having to rely on a “deep and talented squad”, were also 2nd best in their group and have only had to overcome Croatia (albeit making up 15 ranking places).

And if we go back to Iceland, regardless of their population, they have overhauled three better-ranked teams with a combined superiority of 64 ranking points. Next best, but a long way back are Romania, who also overhauled three teams, but only made up 27 ranking points in the process.

Hope you enjoyed all that.

[These italics aren't from Marcotti.] PS: If you're trying to recreate my calculations, it's possible you'll get minimally different results, as there are different ways of treating countries of level ranking. Don't worry about it.