Read on Twitter
1/?) Since a lot of you liked my book suggesting, I decided to make a short thread about the difference between frequentism and Bayesianism. Enjoy!
2/?) In frequntist statistics, observations y_1, ..., y_N are assumed to be generated by a probabilistic model p(y|x) (likelihood) parameterized by a fixed but unknown parameter true_x.
The statistician estimates the true value of x using an estimator:
est_x = T(y_1,...,y_N)
The statistician estimates the true value of x using an estimator:
est_x = T(y_1,...,y_N)
3/?) The estimator is a function of the observations. Its most important frequentist properties are the bias and variance:
b = true_x - E[T(y_1,...,y_N)]
var_T = E[(E[T(y_1,...,y_N)] - T(y_1,...,y_N))^2]
where the expectation is taken with respect to p(y|true_x).
b = true_x - E[T(y_1,...,y_N)]
var_T = E[(E[T(y_1,...,y_N)] - T(y_1,...,y_N))^2]
where the expectation is taken with respect to p(y|true_x).
5/?) Good estimators have small (ideally zero) bias and low variance.
Frequentism does not offer a unique recipe for estimating an arbitrary parameter x. The choice of the estimator T is a degree of freedom in a frequentist analysis.
Frequentism does not offer a unique recipe for estimating an arbitrary parameter x. The choice of the estimator T is a degree of freedom in a frequentist analysis.
6/?) Frequentist statistics consider the parameter true_x as fixed and the observations y_i as random. In other words, frequentist statistics look at the idealized distribution of all possible repetition of the same experiment with a fixed true value of the parameter x.
7/?) The exact opposite is true in Bayesian statistics! The observations y_1, ..., y_N are now fixed. Bayesian statisticians do not need the idealized distribution of repeated experiments. Randomness in Bayesian statistics come from the uncertainty over the parameter x.
8/?) Bayesian statisticians quantify the uncertainty over x with a distribution p(x). After observing the data, this distribution is updated using the rules of conditional probability p(x|y) = p(y|x)p(x)/p(y) which in this context is referred to as Bayes theorem.
9/?) p(x|y) is the so called posterior distribution and summarize all the information provided by both the observations and the prior distribution p(x). While the Bayesian math is rock solid, frequenstists criticize the arbitrariness of the prior p(x).
10/?) To summarize, frequentist statistics treats as random the y axis of p(y|x) while Bayesians see x as random. The randomness of y is interpreted by frequenstists in terms of the idealized distribution of repeated experiments.Bayesians see randomness as subjective uncertainty
