Biased Estimators vs Unbiased Ones

Let us consider a numerical difference between a biased estimator and unbiased one. I will take a simple case of standard deviation of a sample mean. Say we have some measurements \(\{ x_1, x_2, \ldots, x_N\}\) of a particular outcome. We can compute its average, \({\bar x}\), and we want to estimate its deviation as well. The formula for unbiased one is the following: \[ s=\sqrt{\frac{\sum_{i=1}^{N}({\bar x}-x_i)^2}{N-1}} \] In the same time if we knew what is a true mean \(\mu\) of the outcome, we can compute the deviation the following way: \[ s=\sqrt{\frac{\sum_{i=1}^{N}(\mu-x_i)^2}{N}} \] So it looks natural to write the deviation using \({\bar x}\) instead of \(\mu\) as \[ \sqrt{\frac{\sum_{i=1}^{N}({\bar x}-x_i)^2}{N}} \] But we know from statistics that the last formula is biased.

Let us consider how different are these values numerically. I will denote the sum of squares on the top by \(S=\sum_{i=1}^{N}({\bar x}-x_i)^2\), to simplify the look of my calculations. Thus \[ s=\sqrt{\frac{\sum_{i=1}^{N}({\bar x}-x_i)^2}{N-1}}= \sqrt{\frac{S}{N-1}}, \] and I want to compute \[ \sqrt{\frac{S}{N-1}}-\sqrt{\frac{S}{N}}= \] We can take out \(\sqrt{S}\) as a common factor: \[ \sqrt{S}\left(\frac{1}{\sqrt{N-1}}-\frac{1}{\sqrt{N}}\right)= \] and then bring fractions to common denominator and combine them. \[ \sqrt{S}\left(\frac{\sqrt{N}}{\sqrt{N(N-1)}}- \frac{\sqrt{N-1}}{\sqrt{N(N-1)}}\right)= \sqrt{S}\left(\frac{\sqrt{N}-\sqrt{N-1}}{\sqrt{N(N-1)}}\right)= \] For difference of square roots there is a special trick in math, based on formula \[ (a-b)(a+b)=a^2-b^2 \] As you see, if \(a\) and \(b\) are square roots, then by multiplying by their sum we can get rid of them. Usually we cannot multiply expressions on will, but with a fraction we are allowed to multiply top and bottom by the same expression. \[ \sqrt{S}\left( \frac{\left(\sqrt{N}-\sqrt{N-1}\right)\left(\sqrt{N}+\sqrt{N-1}\right)} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)} \right)= \] Using the above mentioned trick simplifies our top (but not the bottom): \[ \sqrt{S}\left( \frac{N-(N-1)} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)} \right)= \] \[ \sqrt{S}\left( \frac{1} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)} \right)= \] \[ \frac{\sqrt{S}} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)} \] This is not very convenient. Let us consider it in relation to sample deviation. \[ \frac{\mbox{The difference between unbiased and biased values}} {\mbox{sample deviation}}= \] \[ \frac{\frac{\sqrt{S}} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)}}{\sqrt{\frac{S}{N-1}}}= \] Here we can cancel \(\sqrt{S}\), flip fractions and then cancel \(\sqrt{N-1}\). I hope you’re still with me. \[ \frac{\frac{1} {\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)}} {\frac{1}{\sqrt{N-1}}}= \frac{\sqrt{N-1}}{\sqrt{N(N-1)}\left(\sqrt{N}+\sqrt{N-1}\right)}= \] \[ \frac{1}{\sqrt{N}\left(\sqrt{N}+\sqrt{N-1}\right)} \] The last expression is better, but still can use some doctoring. I can replace all \(\sqrt{N}\) by \(\sqrt{N-1}\), thus reducing the fraction denominator and getting an upper bound for the whole expression: \[ \frac{1}{\sqrt{N}\left(\sqrt{N}+\sqrt{N-1}\right)}< \frac{1}{\sqrt{N-1}\left(\sqrt{N-1}+\sqrt{N-1}\right)}= \frac{1}{2(N-1)} \] Finally we got a handy formula. If \(N=101\), then our accuracy for sample deviation computed with a biased formula is \(\frac{1}{2(100)}\cdot 100\%=0.5\%\). When \(N=10,001\), then it will be \(0.005\%\). When \(N=1,000,001\), which is more close to what happens in Big Data, then the accuracy will be \(0.5\cdot 10^{-6}\%\).

Conclusion.

In Big Data a difference between biased and unbiased estimators of the same value becomes very small and could be even negligible.