Should You Trust Player Ratings?

(Note: the data used in this piece occurs prior to MD 26.)

The quantifying arts have gained much ground in the deliciously chaotic world of football this past decade. Increasingly, data is the paint by which we color our footballing experiences. I’m no different, because I can always say about something about data in football. As a football writer, the variations thereof are endless and complex. Besides, thanks to tracking tools and software, football data sets sprout prolifically, like Kudzu over the American Deep South. Data kudzu is here, there, everywhere.

For better or (mostly) worse, player ratings have become a data commonplace in football. Pick any random football site and, chances are, the site does player ratings after each Matchday. In the pop culture vein of, say, American Idol or The X Factor, evaluating performers is a popular phenomenon — evaluations get those eyeballs glued and get those pages viewed. So in the same pop evaluation spirit, you’ll find player ratings everywhere in football media.

Everyone does them. And the methods vary greatly. Variation and difference abound. The rating scales are different. The methodologies are different. And, predictably, the results are different. Sometimes very different.

Some outlets create their ratings with just good ol’ eyeball intuition. For example, take kicker.de‘s famous 6-1 rating scale (6 = horrible, 1 = Pele’s ghost); Ousmane Dembélé is currently kicker‘s highest rated player this season with a 2.44 average rating, while Dynamo Dresden’s midfielder, Akaki Gogia, is the 2.Bundesliga’s highest rated player with a 2.57 average rating.

Meanwhile, others create the ratings with statistical formulas, usually with something like a linear weights model. Typically, these models are proprietary “secret sauce” formulas, even if, for example, WhoScored.com offers a ratings explainers, which of course doesn’t divulge the actual secret sauce recipe. In case you’re wondering, WhoScored lists Thiago Alcantara (shout out to Abel Meszaros!) as the Bundesliga’s best player this season with an average rating of 8.2 (on a 10 point scale), while St. Pauli’s defender Lasse Sobiech with average rating of 6.49 is tied with two other players — Heidenheim’s striker Tim Kleindienst and midfieder Marc Schnatterer– for the same honor in the 2.Bundesliga.

Did anyone else catch the discrepancies in results?

Let me spell out the discrepancies more clearly:

  • Ousname Dembélé: ranked 1st in kicker.de; ranked 12th in WhoScored.
  • Thiago Alcantara: ranked 11th in kicker.de; ranked 1st in WhoScored.
  • Akaki Gogia: ranked 1st in kicker.de, ranked 15th in WhoScored.
  • Lasse Sobiech: ranked 15th in kicker.de, ranked t-1st in WhoScored.
  • Tim Kleindienst: ranked 77th(!) in kicker.de, ranked t-1st in WhoScored.
  • Marc Schnatterer: ranked 2nd in kicker.de, ranked t-1st in WhoScored.

However, some discrepancies should be expected, given the different scales and methods being used; moreover, just between the WhoScored.com and kicker.de ratings, there is obviously some overlap. For example, here’s WhoScored.com’s top 10 Bundesliga players:

  1. Thiago Alcantara
  2. Emil Forsberg
  3. Robert Lewandowski
  4. Arjen Robben
  5. Kerem Demirbay
  6. Pierre-Emrick Aubameyang
  7. Naby Keita
  8. Benjamin Hübner
  9. Sead Kolasinac
  10. Hakan Calhanoglu

For comparison’s sake, here’s kicker.de’s top 10 Bundesliga players:

  1. Ousname Dembélé
  2. Arjen Robben
  3. Rune Jarstein
  4. Robert Lewandowski
  5. Philipp Lahm
  6. Roman Bürki
  7. Naby Keita
  8. Oliver Baumann
  9. Sebastian Rudy
  10. Emil Forsberg

For good measure, let’s toss in Squawka.com’s top 10 Bundesliga players this season (here’s Squawka’s ratings “explainer“:

  1. Thiago Alcantara
  2. Robert Lewandowski
  3. Mats Hummels
  4. Pierre-Emrick Aubameyang
  5. Javi Martinez
  6. Niklas Süle
  7. David Alaba
  8. Philipp Lahm
  9. Arjen Robben
  10. Anthony Modeste

The only players to make all three lists? Robert Lewandowski and Arjen Robben. Of course, a core of other plays made two of the three lists — and Thiago topped two lists, yet failed to appear in another.

By season’s end, however, despite the various methodologies and different results, these lists and average ratings seem to slide toward the same mean and the discrepancies work themselves slightly. Yet the ratings discrepancies that occur during a single MD are pretty silly and, in turn, reveal why index ratings are so damn tricky in football.

Case Study: Dennis Diekmeier, Matchday 25

Take the case of HSV’s right fullback, Dennis Diekmeier, from Matchday 25. The match itself was an ugly 0-0 draw at Eintracht with both sides creating only 12 shots total, as well as both mustering pass completion rates under 60% and committing 42 fouls.

My sources will remain nameless, but one source rated Diekmeier’s performance above average for all Bundesliga players on the matchday, another source rated him waaaay below average, while a third source rated him just below average.

For the record, in terms of stats that are provided, Diekmeier complete 4 (!) of 18 passes, completed 3 of 3 tackle attempts, 4 of 4 aerials, and 2 fouls committed.

What do you make of these numbers? I dunno. Obviously, the passing work sucks, and it’s not as if Diekmeier was instructed to cross the ball into the box all day:

Dennis Diekmeier’s 18 passes against Eintracht Frankfurt on MD 25. Source: Squawka.com.

So was Diekmeier above average on the Matchday? Just below average? Magnitudes below average? From what I saw of this match, my “eyeball test” judgement is below average. But this judgement is meaningless.

For starters, I have no idea how Diekmeier was instructed (if at all!) to play the match, or what larger plan he was part of. I also have no idea that he did off the ball most of the time, because of where the cameras go. I also have no idea, phenomenologically, what it was like to be Diekmeier mostly not completing those 18 passes.

Do you?

Defeated by Dynamism

Here’s where it gets tricky.

Player rating stats have a bias toward on-ball pitch events, that is, a “measurable” action that occurs involving  player directly touching the ball or ball handler. These on-ball events are then collated as discrete bits of information and slapped with a number (e.g. pass complete, pass incomplete, interception, header, etc.). Events.

On-ball events are as discrete as you can get in football, which, remember is a sport that is mostly spent by a player being off-ball or even the ball itself not being touched by a player, as it’s lobbed around the pitch.

Plus, the transition between on-ball and off-ball in football is metaphysically maddening, given the fluid and dynamic flow of a football match. All events are contingent on other events, which are contingent on other events, which all flow into future events, which … you get the point. For example, a player might have a snappy 88% pass completion rate, but what if many of his passes are to teammates who immediately get mugged off the ball, resulting in a lost possession?

Context really is everything in football, by which I mean that any piece of data hangs upon a Gordian knot of factors: the match game plan, specific tactics, specific assignments, the play(s) preceding, the play(s) proceeding, weather/pitch conditions, referring, angle, speed, teammates’ movements and positioning, game states within a match …

But I’m hoping that you already know this.

By comparison, baseball is a sport — perhaps the sport — consisting of discrete events. In baseball, events are plays built around pitches. Instead of a clock, baseball itself is confined by and defined by a quantity of events: innings, outs, balls, strikes, etc.  These events are all discrete units that are tallied statistically with relative ease. It’s little wonder that baseball’s statistical tradition is almost as old as the sport itself. It’s also little wonder that baseball probably has the richest statistical trove in all of sports, just explore the “Glossary” on Fangraphs, for example.

Even American football is by magnitudes a sport with more discrete actions than our football. American football is built around specific plays within a lined-off gridiron. The players themselves have high defined roles (e.g. center vs. guard vs. tackle in the offensive line, or the cornerback vs. slotback vs. safety in defense) within highly structures and plays, all of which are quite discrete in nature. Indeed, the likes of Football Outsiders and their innovative DVOA method have successfully quantified the sport.

By comparison, our football is discrete actions poor. In this light, defining football by on-ball events seems like defining a David Lynch film by the plot synopsis offered on Wikipedia (try doing with the likes of Inland Empire or Eraserhead, good luck). Like these synopses, on-ball events aren’t exactly wrong, but perhaps can at best only sketch a match’s lines of motion.

But you don’t have to believe me. Instead, you can believe a former pro footballer, like Jeb Brovsky, who recently wrote  about this very issue for Howler magazine online. As a former player, Brovsky contributes an invaluable and truly “insider” perspective to this issue. Brovsky’s stories about seeing his rating for a given match are funny, but also sobering, especially for us spectators. I mean, how much do we really know about football? How much can we ever know by just watching it on our cable networks?

At the very least, Brovsky’s piece singles out player ratings as an absurdly crude tool. As a sort of corollary, it singles out the overdetermined weight we all attribute to on-ball statistics.

Football is too dynamic, fluid, and lovely for crude tools defining players.

However, I’m not disappointed by the limits a single crude tool. Rather, I’m excited by the rich quantitative land of untapped riches that football always already is. Thinking about football quantitatively only continues to deepen my wonder at the stunningly complex contigency that governs football at the highest professional levels.

In the meantime, beware those player ratings.

The following two tabs change content below.
Travis serves as an editor and regular columnist here. Born and groomed in Santa Fe, New Mexico, Travis is a college English instructor in Pittsburgh. Coffee, books, and sports are his passions. His writing has also appeared in Howler magazine, 11Freunde, America Magazine, The Short Pass, Bloomberg Sports, the Good Man Project, his former blog, Sportisourstory.tumblr.com, and elsewhere. He tweets at @tptimmons. Heja BVB!

Be the first to comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.