in lecture 16 at 14'06": I don't understand why the errors are measured vertically to the line, instead of perpendicular to the line itself? What happened to the orthogonal projection onto the colum space of A?

Question

in lecture 16 at 14'06":   I don't understand why the errors are measured vertically to the line, instead of perpendicular to the line itself?  What happened to the orthogonal projection onto the colum space of A?

anonymous · Answer

One thing to note is that, it is the error vector e, that is orthogonal to the projection of b, onto the column space of A, whereas the vertical lines represent the individual errors bi-pi corresponding to the line that minimizes the overall sum of errors squared E = e1^2+..+en^2. I'll refer to figure 4.6 of section 4.3 in Strang's Intro to Linear Algebra Ed.4 because that speaks directly to your question. Note that once we have the 'best fit line', then we look at a given pair of points (xi,bi) in comparison to (xi,pi) where pi is the point on the line evaluated at xi. If the error lines in this figure were drawn perpendicular to the 'best fit line', we would be evaluating a point on the line corresponding to a different xj not equal to xi which appeared in (xi,bi). I hope this helps. If I can dissect this further than I update the post as well.

anonymous · Answer

Thanks very much for the reply.
It took a while for me to grasp it.
I wrongly thought that the resulting line was the column space of A.
Now I see that in this example the column space is something completely different.
C(A) = the span of [the intercept vector with all ones] with [the points on the x axis] and the vector b are [the targetted points on the y axis]. 
The solution x gives the exact linear combination of those 2 column vectors in A to get to the projection of b on to C(A), which is as close you can get to the real b.
So now I see (or at least think) that it's impossible to represent the perpendicular projection of b onto C(A) in the left hand side of figure 4.6, after all... the column space itself is not "visible" there.
What helped me is thinking of the optimal solution, without the intercept term. Also by viewing another example where the goal was to find the intersection of 3 lines.
Hope I came to the correct conclusion...