Linear Regression
The Linear Regression is a statistical approach to find the relationship within the data where the data consist of dependent variables $y_i$ and independent variables $\{ x_{i,k} \}_{k=1}^{L}$ with the dimension $L$. The most famous approach to fit the data set is called Least Square Method which finds the best fitted line to the data set.
Least Square Method
The Least Square method is a way of finding the line that best fits the given data set. By saying best fitted line, we mean that the line that minimizes the distance(Error) between the line and each data point.
Let the dimension be $L=1$, and assume that we have a data set ${ y_i, x_i}$ for $i \in \{1, 2, \ldots , D\}$. With the equation of a line $y = mx + b$ on hands, we may define the error function as follows:
$$\begin{equation*} E(m, b) = \sum_{i = 1}^{D} \left[ y_i - (mx_i + b) \right]^2 \end{equation*}$$In order to find the best fitted line with this error function, the focus comes down to find the parameters, the slope $m$ and the y-intercept $b$. In calculus, finding the critical points of a function can give us the points that minimizes the function. So setting the partial derivatives of the error functions equal to 0, we have:
$$\begin{align*}
\frac{\partial E(m,b)}{\partial m} &= 2 \cdot \sum_{i=1}^{D} \left[ y_i - (mx_i + b) \right] \cdot (-x_i) \\
&= 0 \\
\frac{\partial E(m,b)}{\partial b} &= 2 \cdot \sum_{i=1}^{D} \left[ y_i - (mx_i + b) \right] \cdot (1) \\
&=0
\end{align*}$$
Each of the partial derivative can be expanded and re-rewritten as follow :
$$\begin{align*} & 2 \cdot \sum_{i=1}^{D} \left[ y_i x_i - m x_i^2 - b x_i \right] = 0 \\ \implies & \sum_{i=1}^{D} m x_i^2 + \sum_{i=1}^{D} b x_i = \sum_{i=1}^{D} y_i x_i \\ \end{align*}$$and
$$\begin{align*}
& 2 \cdot \sum_{i = 1}^{D} \left[ y_i - m x_i - b \right] = 0 \\
\implies & \sum_{i=1}^{D} m x_i + \sum_{i=1}^{D} b = \sum_{i=1}^{D} y_i
\end{align*}$$
In matrix form, this can be re-rewritten as
$$\begin{bmatrix}
\sum_{i=1}^{D} x_i^2 & \sum_{i=1}^{D} x_i \\
\sum_{i=1}^{D} x_i & \sum_{i=1}^{D} 1
\end{bmatrix}
\cdot
\begin{bmatrix}
m \\
b
\end{bmatrix}
=
\begin{bmatrix}
\sum_{i=1}^{D} y_i x_i \\
\sum_{i=1}^{D} y_i
\end{bmatrix}$$
Since the goal is to find the parameters $m$ and $b$, the above matrix equation can be solved by finding the inverse matrix as follow :
$$\begin{bmatrix}
m \\
b
\end{bmatrix}
=
\begin{bmatrix}
\sum_{i=1}^{D} x_i^2 & \sum_{i=1}^{D} x_i \\
\sum_{i=1}^{D} x_i & \sum_{i=1}^{D} 1
\end{bmatrix}^{-1}
\cdot
\begin{bmatrix}
\sum_{i=1}^{D} y_i x_i \\
\sum_{i=1}^{D} y_i
\end{bmatrix}$$