Let's wander through the world of xG, variance and hypothesis tests. How good is Vardy? How good was Pukki early on? Is 12 games enough to judge? What about 6? Nothing advanced (or new) here, just adds a bit of context to xG and a correction for cherry-picking.

Let's start out with Bernoulli random variables (biased coin toss). Suppose we compute the xG (prob. a goal is scored) for a given shot, let's call it p1. The outcome of this shot (goal/no goal) is a Bernoulli RV with success probabilty p1. Its variance is p1*(1-p1).

So for a sample size of 1, the variance depends on the xG (the quality of the shot). Really poor quality chances like 30 yard pot shots are low-variance. Really good chances like penalties are low variance. 50/50 chances like a header 6 yards out from a cross are high variance.

What does this mean? Essentially, outcomes are harder to predict for middling xG values -- that's nothing other than what you'd expect. You've got a better chance of predicting the outcome of a coin toss with a 10% chance of heads (it's gonna be a tail) than one with a 50% chance

What happens when we look at multiple shots? Let's do a comparison in an idealised situation first. Suppose player 1 takes 10 shots worth 0.2 xG each. Player 2 takes 10 shots worth 0.5 xG each. Player 3 takes 10 shots worth 0.8 xG each. What are the variances of numbers of goals?

Clearly the total xGs for players 1, 2, 3 are 2, 5, 8 respectively. Handily, the variances are just the sums of the variances associated with each shot. For players 1 and 3, their shots each have a variance of 0.16, so their total numbers of goals have a variance of 1.6.

For player 2, that variance jumps up to 2.5. This should be telling you that the sample size isn't the only thing that matters when we're thinking about over-/under-performance of xG. It's 'easier' to over-/under-perform if you're taking shots worth around 0.5xG.

Here are the (binomial) distributions over goals scored by each of the 3 players. It's difficult for players 1 and 3 to substantially under- or over-perform respectively. Player 2 can easily under- *or* over-perform.

But you don't ever see a player take 10 shots all having the same xG value! Well, quite. But the sum-of-variances principle still applies. In other words, when you're evaluating under-/over-performance of xG, you need to look at the xGs of the individual shots.

Let's do a different little experiment. Players 1 and 2 have total xGs of 20 from 40 shots. Player 1's total xG is made up of 40 0.5 xG shots. Player 2's total xG is made up of 20 0.1 xG shots and 20 0.9 xG shots. What are the variances in total number of goals?

Player 1's variance is 10. Player 2's variance is 3.6. That's quite a big difference. It's more impressive if player 2 substantially over-performs. Essentially, having more 50/50 chances makes it more likely you'll outperform xG by a decent margin.

This means that in order to compute the distribution over number of goals scored, we need more than just the total xG. We need the xGs associated with each individual shot. Once we have them (sorry, understat), we can plug them in to a Poisson-Binomial distribution.

The Poisson-Binomial distribution gives us a distribution over the total number of successes from n independent Bernoulli trials with (possibly) different success probabilities/expectations. I used the poibin python library to compute the pmfs.

That brings us to Jamie Vardy, who's having a stonking season having scored 11 goals from an xG of 5.19. But what are the xG values of his shots and does that make his performance less impressive? Here are histograms of Vardy's xG values and those of Barnes & Aguero.

All of them have this kind of bimodal flavour. Strikers tend to have quite a few really poor chances, and then a bunch of decent chances around 0.5xG. We've already seen that decent chances contribute disproportionate amounts of variance, but does that make Vardy less amazing?

Here's the Poisson-Binomial distribution over the number of goals from the chances he's had.

For an average player, the probability that they score 11 or more goals from the chances Vardy's had is 0.03%. That's really small. Is it small enough to claim that Vardy is above average? There's a multiple hypothesis testing issue here.

We've chosen to look at Vardy *because* he's doing so well. There are quite a few strikers in the league, so there's a fair chance that one of them is going to post crazy numbers just by chance even if their finishing abilities are all bang-average.

Typically the way to deal with that is using some kind of correction for the fact that we're testing multiple players. The simplest (not the best, but easy) is the Bonferroni correction. There are about 50 PL strikers who've scored a goal this season.

The Bonferroni correction basically says that we should multiply our 0.03% by 50 to make it comparable to the significance level for a single hypothesis test. That gives us a new p-value of 1.5%, quite a bit bigger but still small. Someone was gonna do well, but not this well!

Depending on the arbitrary significance cut-off you want to use, Vardy may or may not be significantly better than average using just 12 games of data. I'd say there's enough to be fairly confident he is. Note that this doesn't mean 12 games is enough for any old situation.

It just means that if Vardy was an average finisher, it's quite unlikely that he'd have been able to have a 12 game run like this given the chances he's had. He might *look* completely average over the next 12 games, despite being better than that.

For interest, here are the distributions over numbers of goals for Barnes & Aguero. Much more sensible. Neither of them has done anywhere near enough over the few games this season to declare that they're above average.

And remember Teemu Pukki? Finnish bloke (he's not finished etc.). Would we have said he's above average after his remarkable 6 goals from 2.69xG earlier in the season? Answer: not at all. The p-value for him was like 25%. Somebody was gonna do it, it just happened to be him.