notes on regression - lecture 1

notes on regression - lecture 1

· 12 min read

thumbnail

Expectation Rules

Let’s first review rules for expectation, variance, and covariance so I won’t have to go through it throughout my notes:

First, we define the term “expectation.” Expectation is basically the average value of a function, which we can calculate both continuously and discretely:

E[X]=∫xp(x)dx,E[X]=∑xxp(x)\mathbb{E}[X] = \int x p(x) dx, \mathbb{E}[X] = \sum_x x p(x)

,where p(x)p(x) is the probability density function, or the probability that the xx value occurs.

We can also define variance and covariance.

Variance averages the squared distance between the true XX value and the mean.

Var[X]=E[(X−E[X])2]\text{Var}[X] = \mathbb{E}[(X-\mathbb{E}[X])^2]

Covariance measures how XX and YY move together: a larger covariance value indicates a stronger relationshp between XX and YY. It’s important to note that covariance cannot be compared across mutliple different situations, and it scales differently!

Cov[X,Y]=E[(X−E[X])(Y−E[X])]\text{Cov} [X,Y] = \mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[X])]

Here are some important algebra rules. Although not too relevant in this lecture, it’ll definetely come up more later:

Linearity of expectations: E[aX+bY]=aE[X]+bE[Y]\mathbb{E}[aX + bY] = a \mathbb{E}[X] + b \mathbb{E} [Y]

Variance identity: Var[X]=E[X2]−(E[X])2\text{Var}[X] = \mathbb{E} [X^2] - (\mathbb{E} [X])^2

Covariance identity: Cov[X,Y]=E[XY]−E[X]E[Y]\text{Cov} [X,Y] = \mathbb{E} [XY] - \mathbb{E} [X] \mathbb{E} [Y]

Covariance symmetry: Cov[X,Y]=Cov[Y,X]\text{Cov} [X,Y] = \text{Cov} [Y,X]

Variance is covariance with itself: Cov[X,X]=Var[X]\text{Cov} [X,X] = \text{Var} [X]

Variance is not linear: Var[aX+b]=a2Var[X]\text{Var} [aX+b] = a^2 \text{Var} [X]

Covariance is not linear: Cov[aX+β1Y]=aCov[X,Y]\text{Cov} [aX + \beta_1 Y] = a \text{Cov} [X,Y]

Variance of a sum: Var[X+Y]=Var[X]+Var[Y]+2Cov[X,Y]\text{Var} [X+Y] = \text{Var} [X] + \text{Var} [Y] + 2 \text{Cov} [X,Y]

Variance of a big sum:

Var[∑i=1nXi]=∑i=1n∑j=1nCov[Xy,Xj]=∑i=1nVar[Xi]+2∑i=1n−1∑j>iCov[Xi,Xj]\text{Var} [\sum^n_{i=1} X_i] = \sum^n_{i=1} \sum^n_{j=1} \text{Cov} [X_y, X_j] = \sum^n_{i=1} \text{Var} [X_i] + 2 \sum^{n-1}_{i=1} \sum_{j>i} \text{Cov} [X_i, X_j]

Law of total expectation (Note: This looks fancy but it’s actually really intuitive; if you average out every XX value on each event YY, you just get the average of XX): E[X]=E[E[X∣Y]]\mathbb{E} [X] = \mathbb{E} [\mathbb{E} [X|Y]]

Independence implies zero covariance: If XX and YY are independent, then Cov[X,Y]=0\text{Cov} [X,Y] = 0, but the reverse is not true!

Law of large numbers (if you repeat something multiple times, the average result gets closer to the mean, and more formally): With random variables X1,X2,X3,…,XnX_1, X_2, X_3, …, X_n with expected value E[X]\mathbb{E}[X], then:

1n∑i=1nXi→E[X]\frac{1}{n} \sum^n_{i=1}X_i \rightarrow \mathbb{E} [X]

Central limit theorem (when you average many independent variables, the distribution of the average is around normal, like a bell curve): Let X1,X2,…,XnX _1, X_2, …, X_n be independent, identically distributed random variables with mean γ\gamma and variance σ2\sigma^2 as n→∞n \rightarrow \infty:

nXˉn−E[X]Var[X]≈N[0,1]\sqrt{n} \frac{\bar{X}_n - \mathbb{E} [X]}{\text{Var} [X] } \approx N[0,1]

Statistic: A statistic is a function of the data and the data alone.

Estimator: Statistic that guesses at the parameter or a function of it, written as θ^\hat{\theta}

Best prediction given random Y

We can write the error for a given value as (Y−m)2(Y-m)^2, where YY is a random, real value and mm is our prediction. The MSE, or the expected value of this is: E[(Y−m)2]\mathbb{E}[(Y-m)^2]. We can define the MSE as:

We can also introduce the following formulas for any values ZZ, YY and mm:

Var(Z)=E[(Z−E[Z])2]\text{Var}(Z) = \mathbb{E}[(Z - \mathbb{E}[Z])^2]

and:

E[Z2]=(E[Z])2+Var(Z)\mathbb{E}[Z^2] = (\mathbb{E}[Z])^2 + \text{Var}(Z)

Combining these two formulas results in a something called bias-variance decomposition, whcih we can apply to our function for MSE:

E[(Y−m)2]=(E[Y−m])2+Var(Y−m)\mathbb{E}[(Y-m)^2] = (\mathbb{E}[Y-m])^2 + \text{Var}(Y-m)

We can also rewrite: E[Y−m]=E[Y]−m \mathbb{E}[Y-m] = \mathbb{E}[Y] - m, because subtracting a constant doesn’t change the expected value!

Also: Var(Y−m)=Var(Y)\text{Var}(Y-m) = \text{Var}(Y), because subtracting a constant does not change variance.

So, we can rewrite the MSE as:

MSE(m)=(E[Y]−m)2+Var(Y)\text{MSE}(m) = (\mathbb{E}[Y]-m)^2 + \text{Var}(Y)

Since our prediction is mm, we’re trying to minimize the (E[Y]−m2)(\mathbb{E}[Y]-m^2) term. We can turn to calulus, take the derivitave, and set it equal to zero to find the “minimum” of our function.

We can use the chain rule:

ddm(E[Y]−m)2=−2(E[Y]−m)\frac{\text{d}}{\text{d}m} (\mathbb{E}[Y]-m)^2 = -2(\mathbb{E}[Y]-m)

Now we set it equal to zero:

−2(E[Y]−m)=0  ⟹  E[Y]=m-2 (\mathbb{E}[Y]-m) = 0 \implies \mathbb{E}[Y] = m

This means that the best single number prediction of a random variable under squared error loss is just the mean! (This is decently intuitive, but it’s cool to go out and derive it yourself).

Prediction one random variable from another

Let’s say we observe XX and want to predict YY. If X=xX=x, we predict m(x)m(x).

The law of total expectation states that:

E[Y−m(X)2]=E[E[(Y−m(C))2∣X]]\mathbb{E}[Y-m(X)^2] = \mathbb{E}[\mathbb{E}[(Y-m(C))^2 | X]]

Now we restrict m(X)=β0+β1Xm(X) = \beta_0 + \beta_1 X so that we can find the optimial linear predictor (just for now)! So:

MSE(β0,β1)=E[(Y−(β0+β1X))2]\text{MSE}(\beta_0, \beta_1) = \mathbb{E}[(Y-(\beta_0 + \beta_1X))^2]

We can multiply this out and distribute the expectation:

E[Y2−2Y(β0+β1X)+(β0+β1X)2]\mathbb{E}[Y^2 - 2Y(\beta_0 + \beta_1X) + (\beta_0 + \beta_1X)^2] =E[Y2]−2β0E[Y]−2β1E[XY]+E[(β0+β1X)2]= \mathbb{E}[Y^2] - 2\beta_0\mathbb{E}[Y] - 2\beta_1 \mathbb{E}[XY] + \mathbb{E}[(\beta_0 + \beta_1X)^2]

We can deal with the last term:

E[(β0+β1X)2]=E[β02+2β0β1X+β12X2]\mathbb{E}[(\beta_0 + \beta_1X)^2] = \mathbb{E}[\beta^2_0 + 2\beta_0\beta_1X + \beta^2_1X^2]

We then distribute the expectation, noting that E[β02]=β02\mathbb{E}[\beta^2_0] = \beta^2_0 , E[2β0β1X]=2β0β1E[X]\mathbb{E}[2\beta_0\beta_1X] = 2\beta_0\beta_1\mathbb{E}[X], E[β12X2]=β12E[X2]\mathbb{E}[\beta_1^2X^2] = \beta_1^2 \mathbb{E} [X^2]

Plugging that ito our original function gives us:

MSE(β0,β1)=E[Y2]−2β0E[Y]−2β1E[XY]+β02+2β0β1E[X]+b12E[X2]\text{MSE}(\beta_0, \beta_1) = \mathbb{E}[Y^2] - 2\beta_0\mathbb{E}[Y] - 2\beta_1 \mathbb{E}[XY] + \beta^2_0 + 2\beta_0\beta_1 \mathbb{E}[X] + b^2_1 \mathbb{E}[X^2]

Let’s find values of β0\beta_0 and β1\beta_1 that minimizes the function. We can use our same derivitave trick we did earlier to find a minimum, starting with β0\beta_0. We need a partial derivitave this time since we’re dealing with multiple varibles β0\beta_0 and β1\beta_1, starting with β0\beta_0. We’re eliminating all constants without β0\beta_0

∂MSE∂β0=∂∂β0(β02−2β0E[Y]+2β0β1E[X])\frac{\partial \text{MSE}}{\partial \beta_0} = \frac{\partial}{\partial \beta_0} (\beta^2_0 - 2\beta_0 \mathbb{E}[Y] + 2\beta_0\beta_1 \mathbb{E}[X])

I’m going to skip over some trivial calculus steps, so if you can’t follow, then you should learn calculus before reading this article. This is equal to:

2β0−2E[Y]+2β1E[X]2\beta_0 - 2\mathbb{E}[Y] + 2\beta_1 \mathbb{E}[X]

Setting equal to 00, we can then solve and get β0=E[Y]−E[Y]\beta_0 = \mathbb{E}[Y] - \mathbb{E}[Y]

We can do the same thing for β1\beta_1:

∂MSE∂β1=∂∂β1(−2β1E[XY]+2β0β1E[X]+b12E[X2])\frac{\partial \text{MSE}}{\partial \beta_1} = \frac{\partial}{\partial \beta_1}(-2\beta_1 \mathbb{E}[XY] + 2\beta_0 \beta_1 \mathbb{E}[X] + b^2_1 \mathbb{E} [X^2])

Which is equal to:

−2E[XY]+2β0E[X]+2β1E[X2]-2 \mathbb{E}[XY] + 2 \beta_0 \mathbb{E}[X] + 2\beta_1 \mathbb{E}[X^2]

Now we set it equal to 0, and with some careful algebra algebra we get E[XY]=β0E[X]+β1E[X]2+β1E[X2]\mathbb{E}[XY] = \beta_0 \mathbb{E}[X] + \beta_1 \mathbb{E}[X]^2 + \beta_1 \mathbb{E} [X^2]

add stuff here askdfjhfks

Here are some important notes about the equation that was mentioned:

  1. The optimal intercept β0\beta_0 makes the line go through the mean Y value of the mean X value. This means that β0+β1E[X]=E[Y]\beta_0 + \beta_1 \mathbb{E} [X]= \mathbb{E} [Y] and the line passes through E[X],E[Y]\mathbb{E}[X], \mathbb{E}[Y]. We can easily prove this by substituting E[X]\mathbb{E}[X] for all XX in our equations.
  2. We should sanity check our full equation by making sure that the units “balance” on both sides of the equation.
  3. Only the variance and the covariance impact the β1\beta_1 term, meaning the actual expectation doesn’t matter! β1\beta_1 will stay the same, regardless if you shift the function.
  4. We must also keep in mind some important morals (although a bit self-explanatory): 1. This approximation might not even be good! 2. We make no assumptions about the noise or fluctuations of the function.
← Back