Question

OLS estimator derivation with constant

Answer 1

\[y_{i}=2+bx_{i}+\epsilon_{i}\]

Answer 2

Following normal means of derivation I get to point where I need to differentiate with respect to two........

Answer 3

\[\sum_{}^{}\epsilon_{i}^{2}=\sum_{}^{}(y_{i}-2-b x_{i})^{2}\]

Answer 4

if you know the number given for \(b_0\) (or maybe you call it \(a\), then I believe you'd only need to differentiate with respect to \(b\) since you only need to estimate b. and you actually know the constant value.

Answer 5

Ah, nice! I'm also told there will be something special about \[\sum_{i}^{} (\hat \epsilon_{i}) \] Are you aware of any differences having the constant involved creates on the error terms?

Answer 6

Thanks a lot by the way, had been going around in circles looking for a way out of the differential issue!

Answer 7

Hm I am not 100% sure. I have to admit I haven't really encountered a situation where the constant was given before. Since OLS aims to minimize the sum of the errors, but you were given some "arbitrary" constant (maybe it's given for some reason), then my guess is that the sum of the errors won't actually be minimized without using the "optimal" estimate for the constant

Answer 8

In my head, I imagine they give you the constant, and from that you are trying to fit the best line using the slope only... so imagine fixing the line at the intercept and you are "swinging" the line up and down to minimize the error terms , so you cannot guarantee the minimization of the error terms, unless 2 is actually the best estimate

Answer 9

That is a really nice visualisation with the swinging - thanks a lot for that!

Answer 10

you should take the derivative with respect to "b"
You are trying to find the slope that will minimize the error

Answer 11

Thanks @phi. I was initially trying to construct the intercept estimate, but obviously a little silly since as @kirbykirby pointed out, what's the use in an estimate when one has the actual value.. Thanks for confirmation!

Answer 12

what do you get for the derivative ?

Answer 13

\[-2\sum_{}^{}(y_{i}-2-bx_{i})x_{i}\]

Answer 14

I would distribute the x_i and write out each term as a separate sum
set that = 0 and solve for b

Answer 15

What does distribute the x_i mean? Surely there could be any number of x's?

Answer 16

you have N terms of the form (yi -2 -b xi)* xi
or N terms of the form
xi*yi - 2 xi - b (xi)^2
put a summation sign in front of each term to "collect" like terms

Answer 17

in other words, each (xi, yi) pair creates an expression xi*yi - 2 xi - b (xi)^2
and you add them up

Answer 18

Oh this is interesting; by 'distributing' the terms if I've interpreted correctly, I'm finding the construction of this b term is the same method as a constant term

Answer 19

b is the variable that is allowed to vary to minimize the total squared error, so I would not call it a constant. But it does have the same value for each of the N expressions.

Answer 20

you have
\[ error= f(b)\]
you find the derivative with respect to b and set it equal to 0
\[ \frac{df}{db} = 0 \]
solve for the value of b. This will be the value that makes the "slope" zero, indicating we found a min or max of the f(b). In this case, it will be the minimum value of f(b)
i.e. the minimum squared error

notice all the "data" (the x and y's ) are treated as constants

Answer 21

after taking the derivative with respect to b you have
\[ -2\sum_{}^{}(y_{i}-2-bx_{i})x_{i} = 0 \\
\sum y_{i} x_i - 2 x_i -b (x_i)^2 = 0 \\
\sum y_{i} x_i - 2 \sum x_i -b \sum (x_i)^2 = 0
\]

Answer 22

Amazing - thank you for such detailed insight! I did manage to get to that position but without considering moving b outside the summation I was a little stuck. I guess now it's a case of moving the -b(summation)(x_i)^2 over and dividing by the (summation)(x_i)^2?

Answer 23

if you divide every term by N (number of terms in the summation)
you can say, for example,
\[ \frac{\sum x_i}{N} = \bar{x} \]
similarly for the other terms

Answer 24

\[b=\frac{ \sum_{}^{}y_{i}x_{i} -2\sum_{}^{}x_{i} }{ \sum_{}^{}(x_{i})^2 }\]

Answer 25

yes. or in terms of averages
\[ b = \frac{ \overline{xy} - 2 \bar{x}}{\overline{x^2}} \]

Answer 26

Thank you so much for your help. Really appreciated!