```{r include = FALSE}
# OPTIONAL: Set a more color blind friendly palette 
palette("Okabe-Ito")
scale_colour_discrete <- function(...) scale_colour_manual(values = palette())
scale_fill_discrete   <- function(...) scale_fill_manual(values = palette())

knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env='figure',
  fig.pos = 'h',
  fig.align = 'center')
```



# Learning Goals {-}

- Explain why training/in-sample model evaluation metrics can provide a misleading view of true test/out-of-sample performance
- Implement testing and training sets in R using the `tidymodels` package


# Notes: Overfitting {-}

**Overfitting**    

When we add more and more predictors into a model, it can become *overfit* to the noise in our sample data:

- our model loses the broader trend / big picture
- thus does not generalize to *new* data
- thus results in bad predictions and a bad understanding of the relationship among the new data points




**Preventing overfitting: training and testing**

- **In-sample** metrics, i.e. measures of how well the model performs on the same sample data that we used to build it, tend to be overly optimistic and lead to overfitting. 
- Instead, we should *build* and *evaluate*, or **train** and **test**, our model using different data.




# Notes: R Code {-}

This section is for **future** reference. It is a summary of code you'll learn below for creating and applying training and testing data. Throughout, suppose we wish to build and evaluate a linear regression model of `y` vs `x1` and `x2` using our `sample_data`. 

```{r eval = FALSE}
# Load packages
library(tidyverse)
library(tidymodels)
```

\

**Split the sample data into training and test sets**    

```{r eval = FALSE}
# Set the random number seed
set.seed(___)

# Split the sample_data
# "prop" is the proportion of data assigned to the training set
# it must be some number between 0 and 1
data_split <- initial_split(sample_data, strata = y, prop = ___)

# Get the training data from the split
data_train <- data_split %>% 
  training()

# Get the testing data from the split
data_test <- data_split %>% 
  testing()
```


**Build a training model**    

```{r eval = FALSE}
# STEP 1: model specification
lm_spec <- linear_reg() %>% 
  set_mode("regression") %>% 
  set_engine("lm")

# STEP 2: model estimation using the training data
model_train <- lm_spec %>% 
  fit(y ~ x1 + x2, data = data_train)
```


**Use the training model to make predictions for the test data**        

```{r eval = FALSE}
# Make predictions
model_train %>% 
  augment(new_data = data_test)
```



**Evaluate the training model using the test data**

```{r eval = FALSE}
# Calculate the test MAE
model_train %>% 
  augment(new_data = data_test) %>% 
  mae(truth = y, estimate = .pred)
```



\
\
\
\


# Exercises {-}

The following exercises are inspired by Chapter 5.3.1 of ISLR.

```{r}
# Load packages & data
# NOTE: You might first need to install the ISLR package
library(tidyverse)
library(tidymodels)
library(ISLR)
data(Auto)
cars <- Auto %>% 
  select(mpg, horsepower, year)
```


Let's use the `cars` data to compare three **linear regression models** of fuel efficiency in miles per gallon (`mpg`) by engine power (`horsepower`):

```{r}
# Raw data
cars_plot <- ggplot(cars, aes(x = horsepower, y = mpg)) + 
  geom_point()
cars_plot
```

```{r}
# model 1: 1 predictor (y = b0 + b1 x)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE)
```

```{r}
# model 2: 2 predictors (y = b0 + b1 x + b2 x^2)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 2))
```

```{r}
# model 3: 19 predictors (y = b0 + b1 x + b2 x^2 + ... + b19 x^19)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 19))
```


\
\



**Goal**    

Let's evaluate and compare these models by **training** and **testing** them using different data. 


\
\


**Directions**

- Continue to work together as a group.
- We will not discuss these exercises as a class. Be sure to ask questions as I walk around the room.


\
\


## Exercise 1: 155 review -- set.seed() {-}

Run the two chunks below *multiple* times each. Afterward, summarize what `set.seed()` does and why it's important to being able to *reproduce* a random sample.

```{r}
sample_n(Auto, 2)
```

```{r}
set.seed(253)
sample_n(Auto, 2)
```

\
\

    

## Exercise 2: Training and test sets {-}
    
Let's *randomly* split our original 392 sample cars into two separate pieces: select 80% of the cars to **train (build)** the model and the other 20% to **test (evaluate)** the model.
    
```{r}
# Set the random number seed
set.seed(8)

# Split the cars data into 80% / 20%
# Ensure that the sub-samples are similar with respect to mpg
cars_split <- initial_split(cars, strata = mpg, prop = 0.8)
```

```{r}
# Check it out
# What do these numbers mean?
cars_split
```

```{r}
# Get the training data from the split
cars_train <- cars_split %>% 
  training()

# Get the testing data from the split
cars_test <- cars_split %>% 
  testing()
```

```{r}
# The original data has 392 cars
nrow(cars)

# How many cars are in cars_train?

# How many cars are in cars_test?

```
    


\
\


## Exercise 3: Reflect on the above code {-}

a. Why do we want the training and testing data to be similar with respect to `mpg` (`strata = mpg`)? What if they *weren't*?



b. Why did we need all this new code instead of just using the *first* 80% of cars in the sample for training and the *last* 20% for testing?
    




\
\




## Exercise 4: Build the training model {-}

```{r eval = FALSE}
# STEP 1: model specification
lm_spec <- linear_reg() %>% 
  set_mode("regression") %>% 
  set_engine("lm")

# STEP 2: model estimation using the training data
# Construct the 19th order polynomial model using the TRAINING data
model_19_train <- ___ %>% 
  ___(mpg ~ poly(horsepower, 19), data = ___)
```

```{r}

```


\
\


## Exercise 5: Evaluate the training model {-}

```{r eval = FALSE}
# How well does the TRAINING model predict the TRAINING data?
# Calculate the training (in-sample) MAE
model_19_train %>% 
  augment(new_data = ___) %>% 
  mae(truth = mpg, estimate = .pred)
```

```{r}

```

```{r eval = FALSE}
# How well does the TRAINING model predict the TEST data?
# Calculate the test MAE
model_19_train %>% 
  augment(new_data = ___) %>% 
  mae(truth = mpg, estimate = .pred)
```

```{r}

```






\
\


## Exercise 6: Punchline {-}
    
The table below summarizes your results for `train_model_19` as well as the other two models of interest. (You should confirm the other two model results outside of class!) 

Model                         Training MAE   Testing MAE
---------------------------- ------------- -------------
`mpg ~ horsepower`                    3.78          4.00
`mpg ~ poly(horsepower, 2)`           3.20          3.49
`mpg ~ poly(horsepower, 19)`          2.99          6.59



Answer the following and reflect on why each answer *makes sense*:

a. *Within* each model, how do the training errors compare to the testing errors? (This isn't *always* the case, but is common.)   


b. What about the training and test errors for the third model suggest that it is *overfit* to our sample data?


c. Which model seems the best with respect to the *training* errors?  


d. Which model is the best with respect to the *testing* errors?   


e. Which model would you choose?



\
\



## Exercise 7: Final reflection {-}

a. The training / testing procedure provided a more honest evaluation and comparison of our model predictions. How might we *improve* upon this procedure? What problems can you anticipate in splitting our data into 80% / 20%?


b. Summarize the key themes from today in your own words.




\
\



## Exercise 8: STAT 155 REVIEW -- data drill {-}

a. Construct and interpret a plot of `mpg` vs `horsepower` and `year`.

```{r}

```

b. Calculate the average `mpg`.

```{r}

```

c. Calculate the average `mpg` for each `year`. HINT: `group_by()`

```{r}

```

d. Plot the *average* `mpg` by `year`.

```{r}

```


\
\



## Exercise 9: Digging deeper (optional) {-}

Check out the online solutions for exercise 6. Instead of calculating MAE from scratch for 3 different models, I wrote a function `calculate_MAE()` to automate the process. After picking through this code, adapt the function so that it also returns the $R^2$ value of each model.    





\
\



## Done! {-}

- Knit your notes.
- Check the solutions in the course website.
- If you finish all that during class, start your homework!








