Linear Regression

What is it?

Linear regression is a widely used statistical model used to determine the Correlation between a dependent variable, that you’re trying to predict, and a independent variable, which you’re basing the prediction on.

In Statistics and Machine Learning, a linear regression model tries to capture the linear relationship between two or more variables, in a way that it produces an equation that approximates a prediction for any point in the known data range.

---

The statistical model

The statistical model for linear regression is a linear function, given by a pair of data $x$ and $y$. The dependent variable is given by $y$, and the independent variables is $x$.

Given $\beta$ as the angular coefficient, $\alpha$ as the linear coefficient, and $ϵ$ as the random error, one can write a simple linear regression model as:

$$y_i=\alpha +\beta \ast x_i+ϵ$$

---

The assumptions of linear regression

However, there are four main assumptions related with a linear regression model:

• Linearity

  The relationship between $x$ and $y$ must be linear.

• Homoscedasticity

  The Variance of the residual error must be the same for any value of $x_i$.

• Independence and Normality

  For any value of $x,y$, the data must be i.i.d., independent and identically distributed.

Making data linear

While it may be not the best choice, it’s possible to linearize data so that it fits in the assumptions made. The process is called Data Linearization.

---

Estimating the model parameters

To determine the best fitting line that captures the data, one can use several metrics. The most used method is the Least Squares method, also called OLS, standing for Ordinary Least Squares.

The idea is to estimate parameters that minimize the error $ϵ$, meaning the fitted line is close to the observed values. The sum of squared errors is given by:

$$SSE=\sum _{i=1}^n{ϵ}_i^2\text{, which can be rewritten as}SSE=\sum _{i=1}^n[y_i-(\alpha -\beta \ast x_i){]}^2$$

To find the values of $\alpha$ and $\beta$ that minimizes $SSE$, we can use Derivatives with optimization techniques to find the critical points, with:

$$f^{\prime }(\alpha )SQE=0\text{and}{f}^{\prime }(\beta )SQE=0$$

Which can be applied the Chain Rule from Differentiation:

$$\{\begin{array}{c}{\sum }_{i=1}^n{y}_i=n\alpha +\beta .{\sum }_{i=1}^n{x}_i\\ {\sum }_{i=1}^n{x}_i{y}_i=\alpha \cdot {\sum }_{i=1}^n{x}_i+\beta .{\sum }_{i=1}^n{x}_i^2\end{array}$$

Simplifying to:

$$\beta =\frac{n\ast {\sum }_{i=1}^n{x}_i{y}_i-({\sum }_{i=1}^n{x}_i\ast {\sum }_{i=1}^n{y}_i)}{n\ast {\sum }_{i=1}^n{x}_i^2-({\sum }_{i=1}^n{x}_i{)}^2}$$

The linear coefficient $\alpha$ has an easier calculation with the linear regression model equation itself:

$$\alpha =\overline{y}-\beta \ast \overline{x}$$

---

Evaluating a linear regression model

To evaluate a linear regression model, the most used metric is the Coefficient of Determination, referred as $R^2$, which captures the explained Variance in the model.

One can also use other metrics like Mean Squared Error, or Mean Absolute Error to determine the goodness of fit.

---

Other reference

• Sobre a Regressão Linear - Own Medium post.

• Regressão Linear - Matheus Facure