Feature Engineering

class: title-slide, center

# Feature Engineering

## Tidy Data Science with the Tidyverse and Tidymodels

### W. Jake Thompson

#### [https://tidyds-2021.wjakethompson.com](https://tidyds-2021.wjakethompson.com) &#183; [https://bit.ly/tidyds-2021](https://bit.ly/tidyds-2021)

.footer-license[*Tidy Data Science with the Tidyverse and Tidymodels* is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).]

<div style = "position:fixed; visibility: hidden">
  `$$\require{color}\definecolor{blue}{rgb}{0, 0.623529411764706, 0.717647058823529}$$`
  `$$\require{color}\definecolor{light_blue}{rgb}{0.0392156862745098, 0.870588235294118, 1}$$`
  `$$\require{color}\definecolor{yellow}{rgb}{0.996078431372549, 0.843137254901961, 0.4}$$`
  `$$\require{color}\definecolor{dark_yellow}{rgb}{0.635294117647059, 0.47843137254902, 0.00392156862745098}$$`
  `$$\require{color}\definecolor{pink}{rgb}{0.796078431372549, 0.16078431372549, 0.482352941176471}$$`
  `$$\require{color}\definecolor{light_pink}{rgb}{1, 0.552941176470588, 0.776470588235294}$$`
  `$$\require{color}\definecolor{grey}{rgb}{0.411764705882353, 0.403921568627451, 0.450980392156863}$$`
</div>
  
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    TeX: {
      Macros: {
        blue: ["{\\color{blue}{#1}}", 1],
        light_blue: ["{\\color{light_blue}{#1}}", 1],
        yellow: ["{\\color{yellow}{#1}}", 1],
        dark_yellow: ["{\\color{dark_yellow}{#1}}", 1],
        pink: ["{\\color{pink}{#1}}", 1],
        light_pink: ["{\\color{light_pink}{#1}}", 1],
        grey: ["{\\color{grey}{#1}}", 1]
      },
      loader: {load: ['[tex]/color']},
      tex: {packages: {'[+]': ['color']}}
    }
  });
</script>

---
class: your-turn

# Your Turn 0

.big[
* Open the R Notebook **materials/exercises/13-recipes.Rmd**
* Run the setup chunk
]

---
<div class="hex-book">
  <a href="https://recipes.tidymodels.org">
    <img class="hex" src="images/hex/recipes.png">
  </a>
  <a href="https://www.tmwr.org/recipes.html">
    <img class="book" src="images/books/tmwr-recipes.png">
  </a>
</div>

---
background-image: url(images/recipes/wf-12.png)
background-position: center 85%
background-size: 70%

# Machine Learning

---
class: center
background-image: url(images/recipes/tidymodels/tm-01.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: center
background-image: url(images/recipes/tidymodels/tm-02.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: center
background-image: url(images/recipes/tidymodels/tm-03.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: center
background-image: url(images/recipes/tidymodels/tm-04.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: center
background-image: url(images/recipes/tidymodels/tm-05.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: center
background-image: url(images/recipes/tidymodels/tm-06.png)
background-position: center 70%
background-size: 90%

# tidymodels

---
class: pop-quiz

# Pop quiz!

What is multicollinearity?

When multiple predictors are strongly correlated. It can impair linear models.

---
# Principle Components Analysis

Transforms variables into the orthogonal "components" that most concisely capture all of the variation.

---
class: middle, center, frame

# Goal

To fit a linear model to the main Principal Components of the `ames` data.

---
# To build a recipe

1\. Start the `recipe()`

2\. Define the .display[variables] involved

3\. Describe **prep**rocessing .display[step-by-step]

---
# `recipe()`

Creates a recipe for a set of variables.

```r
recipe(Sale_Price ~ ., data = ames)
```

---
# `step_*()`

Adds a single transformation to a recipe. Transformations are replayed in order when the recipe is run on data.

```r
rec %>%
  step_novel(all_nominal()) %>%
  step_zv(all_predictors())
```

---
# `step_*()`

Complete list at: https://recipes.tidymodels.org/reference/index.html

---
# Selectors

Helper functions fro selecting sets of variables.

<code class ='r hljs remark-code'>rec %>%<br>&nbsp;&nbsp;step_novel(<span style="background-color:#FED766;color:#009FB7">all_nominal()</span>) %>%<br>&nbsp;&nbsp;step_zv(<span style="background-color:#FED766;color:#009FB7">all_predictors()</span>)</code>

---
class: middle

#phgzwvmjhj .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #F0F0F0;
  width: 10%;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 1px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#phgzwvmjhj .gt_heading {
  background-color: #F0F0F0;
  text-align: left;
  border-bottom-color: #F0F0F0;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#phgzwvmjhj .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #F0F0F0;
  border-bottom-width: 0;
}

#phgzwvmjhj .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #F0F0F0;
  border-top-width: 0;
}

#phgzwvmjhj .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#phgzwvmjhj .gt_col_headings {
  border-top-style: solid;
  border-top-width: 3px;
  border-top-color: #F0F0F0;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#phgzwvmjhj .gt_col_heading {
  color: #333333;
  background-color: #F0F0F0;
  font-size: 80%;
  font-weight: bolder;
  text-transform: uppercase;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#phgzwvmjhj .gt_column_spanner_outer {
  color: #333333;
  background-color: #F0F0F0;
  font-size: 80%;
  font-weight: bolder;
  text-transform: uppercase;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#phgzwvmjhj .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#phgzwvmjhj .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#phgzwvmjhj .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#phgzwvmjhj .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #F0F0F0;
  font-size: 80%;
  font-weight: bolder;
  text-transform: uppercase;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#phgzwvmjhj .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #F0F0F0;
  font-size: 80%;
  font-weight: bolder;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#phgzwvmjhj .gt_from_md > :first-child {
  margin-top: 0;
}

#phgzwvmjhj .gt_from_md > :last-child {
  margin-bottom: 0;
}

#phgzwvmjhj .gt_row {
  padding-top: 3px;
  padding-bottom: 3px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#phgzwvmjhj .gt_stub {
  color: #333333;
  background-color: #F0F0F0;
  font-size: 80%;
  font-weight: bolder;
  text-transform: uppercase;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#phgzwvmjhj .gt_summary_row {
  color: #333333;
  background-color: #F0F0F0;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#phgzwvmjhj .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#phgzwvmjhj .gt_grand_summary_row {
  color: #333333;
  background-color: #F0F0F0;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#phgzwvmjhj .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#phgzwvmjhj .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#phgzwvmjhj .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#phgzwvmjhj .gt_footnotes {
  color: #333333;
  background-color: #F0F0F0;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#phgzwvmjhj .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#phgzwvmjhj .gt_sourcenotes {
  color: #333333;
  background-color: #F0F0F0;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 0px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 0px;
  border-right-color: #D3D3D3;
}

#phgzwvmjhj .gt_sourcenote {
  font-size: 12px;
  padding: 10px;
}

#phgzwvmjhj .gt_left {
  text-align: left;
}

#phgzwvmjhj .gt_center {
  text-align: center;
}

#phgzwvmjhj .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#phgzwvmjhj .gt_font_normal {
  font-weight: normal;
}

#phgzwvmjhj .gt_font_bold {
  font-weight: bold;
}

#phgzwvmjhj .gt_font_italic {
  font-style: italic;
}

#phgzwvmjhj .gt_super {
  font-size: 65%;
}

#phgzwvmjhj .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="phgzwvmjhj" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table" style="table-layout: fixed;; width: 10%">
  <colgroup>
    <col style="width:400px;"/>
    <col style="width:500px;"/>
  </colgroup>
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="text-align: center; vertical-align: middle; font-weight: bold; border-bottom-width: 3px; border-bottom-style: solid; border-bottom-color: black;">selector</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="text-align: center; vertical-align: middle; font-weight: bold; border-bottom-width: 3px; border-bottom-style: solid; border-bottom-color: black;">description</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p><code>all_predictors()</code></p>
</div></td>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p>Each x variable  (right side of ~)</p>
</div></td>
    </tr>
    <tr>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p><code>all_outcomes()</code></p>
</div></td>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p>Each y variable  (left side of ~)</p>
</div></td>
    </tr>
    <tr>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p><code>all_numeric()</code></p>
</div></td>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p>Each numeric variable</p>
</div></td>
    </tr>
    <tr>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p><code>all_nominal()</code></p>
</div></td>
      <td class="gt_row gt_left" style="background-color: #F0F0F0;"><div class='gt_from_md'><p>Each categorical variable (e.g. factor, string)</p>
</div></td>
    </tr>
    <tr>
      <td class="gt_row gt_left" style="border-bottom-width: 2px; border-bottom-style: solid; border-bottom-color: transparent; background-color: #F0F0F0;"><div class='gt_from_md'><p><code>dplyr::select()</code> helpers</p>
</div></td>
      <td class="gt_row gt_left" style="border-bottom-width: 2px; border-bottom-style: solid; border-bottom-color: transparent; background-color: #F0F0F0;"><div class='gt_from_md'><p><code>starts_with('Lot_')</code>, etc.</p>
</div></td>
    </tr>
  </tbody>
  
  
</table></div>

---
# Combining selectors

Use commas to separate

```r
rec %>%
* step_novel(all_nominal(), -all_outcomes()) %>%
  step_zv(all_predictors())
```

---
class: pop-quiz

# Pop quiz!

How does recipes know what is a **predictor** and what is an **outcome**?

```r
rec <- 
* recipe(Sale_Price ~ .,
         data = ames)
```

The .display[formula] ➡️  *indicates outcomes vs predictors*

---
class: pop-quiz

# Pop quiz!

How does recieps know what is **numeric** and what is **nominal**?

```r
rec <- 
  recipe(Sale_Price ~ .,
*        data = ames)
```

The .display[data] ➡️  *is only used to catalog the names and types of each variable*

---
class: pop-quiz

# Pop quiz!

PCA requires variables to be **centered** and **scaled**. What does that mean?

Standardize or *z*-score

---
# `step_center()`

Centers numeric variables by subtracting the mean

```r
rec <-
  recipe(Sale_Price ~ .,
         data = ames) %>%
* step_center(all_numeric())
```

---
# `step_scale()`

Scales numeric variables by dividing by the standard deviation

```r
rec <-
  recipe(Sale_Price ~ .,
         data = ames) %>%
  step_center(all_numeric()) %>%
* step_scale(all_numeric())
```

---
class: pop-quiz

# Pop quiz!

Why do you need to "train" a recipe?

Imagine "scaling" a new data point. What do you subtract from it? What do you divide it by?

---
background-image: url(images/recipes/prep-recipe.png)
background-position: center middle
background-size: 90%

---
background-image: url(images/recipes/bake-recipe.png)
background-position: center middle
background-size: 90%

---
# `prep()` and `bake()`

"trains" a recipe and then transforms data with the prepped recipe

```r
rec %>%
  prep(training = ames_train) %>%
  bake(new_data = ames_test) # or ames train
```

.footnote[.display[You don't need to do this! The fit functions do it for you]]

---
background-image: url(images/recipes/recipes.png)
background-size: cover

.footnote[Artwork by [@allison_horst](https://twitter.com/allison_horst)]

---

```r
rec %>% 
  prep(ames_train) %>%
  bake(ames_test) 
#> # A tibble: 732 x 81
#>    MS_SubClass      MS_Zoning     Lot_Frontage Lot_Area Street Alley  Lot_Shape 
#>    <fct>            <fct>                <dbl>    <dbl> <fct>  <fct>  <fct>     
#>  1 One_Story_1946_… Residential_…      2.48     3.00    Pave   No_Al… Slightly_…
#>  2 One_Story_1946_… Residential_…      0.663    0.217   Pave   No_Al… Regular   
#>  3 One_Story_1946_… Residential_…      1.05     0.153   Pave   No_Al… Regular   
#>  4 Two_Story_1946_… Residential_…      0.514   -0.00732 Pave   No_Al… Slightly_…
#>  5 Two_Story_1946_… Residential_…     -0.320    6.00    Pave   No_Al… Moderatel…
#>  6 One_Story_1946_… Residential_…      0.901    0.185   Pave   No_Al… Regular   
#>  7 One_Story_1946_… Residential_…      0.216   -0.221   Pave   No_Al… Regular   
#>  8 Two_Story_PUD_1… Residential_…     -1.09    -1.16    Pave   No_Al… Regular   
#>  9 Two_Story_1946_… Residential_…     -1.72    -0.304   Pave   No_Al… Regular   
#> 10 Two_Story_1946_… Residential_…      0.00782  0.927   Pave   No_Al… Moderatel…
#> # … with 722 more rows, and 74 more variables: Land_Contour <fct>,
#> #   Utilities <fct>, Lot_Config <fct>, Land_Slope <fct>, Neighborhood <fct>,
#> #   Condition_1 <fct>, Condition_2 <fct>, Bldg_Type <fct>, House_Style <fct>,
#> #   Overall_Qual <fct>, Overall_Cond <fct>, Year_Built <dbl>,
#> #   Year_Remod_Add <dbl>, Roof_Style <fct>, Roof_Matl <fct>,
#> #   Exterior_1st <fct>, Exterior_2nd <fct>, Mas_Vnr_Type <fct>,
#> #   Mas_Vnr_Area <dbl>, Exter_Qual <fct>, Exter_Cond <fct>, Foundation <fct>,
#> #   Bsmt_Qual <fct>, Bsmt_Cond <fct>, Bsmt_Exposure <fct>,
#> #   BsmtFin_Type_1 <fct>, BsmtFin_SF_1 <dbl>, BsmtFin_Type_2 <fct>,
#> #   BsmtFin_SF_2 <dbl>, Bsmt_Unf_SF <dbl>, Total_Bsmt_SF <dbl>, Heating <fct>,
#> #   Heating_QC <fct>, Central_Air <fct>, Electrical <fct>, First_Flr_SF <dbl>,
#> #   Second_Flr_SF <dbl>, Low_Qual_Fin_SF <dbl>, Gr_Liv_Area <dbl>,
#> #   Bsmt_Full_Bath <dbl>, Bsmt_Half_Bath <dbl>, Full_Bath <dbl>,
#> #   Half_Bath <dbl>, Bedroom_AbvGr <dbl>, Kitchen_AbvGr <dbl>,
#> #   Kitchen_Qual <fct>, TotRms_AbvGrd <dbl>, Functional <fct>,
#> #   Fireplaces <dbl>, Fireplace_Qu <fct>, Garage_Type <fct>,
#> #   Garage_Finish <fct>, Garage_Cars <dbl>, Garage_Area <dbl>,
#> #   Garage_Qual <fct>, Garage_Cond <fct>, Paved_Drive <fct>,
#> #   Wood_Deck_SF <dbl>, Open_Porch_SF <dbl>, Enclosed_Porch <dbl>,
#> #   Three_season_porch <dbl>, Screen_Porch <dbl>, Pool_Area <dbl>,
#> #   Pool_QC <fct>, Fence <fct>, Misc_Feature <fct>, Misc_Val <dbl>,
#> #   Mo_Sold <dbl>, Year_Sold <dbl>, Sale_Type <fct>, Sale_Condition <fct>,
#> #   Longitude <dbl>, Latitude <dbl>, Sale_Price <dbl>
```

---
class: pop-quiz

# Pop quiz!

.left-column[

```
# A tibble: 6 x 1
  Roof_Style
  <fct>     
1 Hip       
2 Gable     
3 Mansard   
4 Gambrel   
5 Shed      
6 Flat      
```
]

.right-column[

```
# A tibble: 2,930 x 6
     Hip Gable Mansard Gambrel  Shed  Flat
   <dbl> <dbl>   <dbl>   <dbl> <dbl> <dbl>
 1     1     0       0       0     0     0
 2     0     1       0       0     0     0
 3     1     0       0       0     0     0
 4     1     0       0       0     0     0
 5     0     1       0       0     0     0
 6     0     1       0       0     0     0
 7     0     1       0       0     0     0
 8     0     1       0       0     0     0
 9     0     1       0       0     0     0
10     0     1       0       0     0     0
# … with 2,920 more rows
```
]

---
# Dummy Variables

```r
lm(Sale_Price ~ Roof_Style, data = ames)
```

```
#> # A tibble: 6 x 5
#>   term              estimate std.error statistic  p.value
#>   <chr>                <dbl>     <dbl>     <dbl>    <dbl>
#> 1 (Intercept)        184799.    17167.    10.8   1.57e-26
#> 2 Roof_StyleGable    -14487.    17240.    -0.840 4.01e- 1
#> 3 Roof_StyleGambrel  -46514.    23719.    -1.96  5.00e- 2
#> 4 Roof_StyleHip       41891.    17475.     2.40  1.66e- 2
#> 5 Roof_StyleMansard  -18573.    28818.    -0.644 5.19e- 1
#> 6 Roof_StyleShed       8401.    38386.     0.219 8.27e- 1
```

---
# `step_dummy()`

Converts nominal data into dummy variables which are numeric and suitable for linear algebra.

```r
rec %>%
* step_dummy(all_nominal())
```

.footnote[You don't need this for decision trees or ensembles of trees]

---
class: pop-quiz

# Consider

Let's think about the modeling.

What if there were no homes with shed roofs in the training data?

Will the model have a coefficient for shed roof?

.display[No]

What will happen if the test data has a home with a shed roof?

.display[Error!]

---
# `step_novel()`

Adds a catch-all level to a factor for any new values, which lets R intelligently predict new levels in the test set.

```r
rec %>%
* step_novel(all_nominal()) %>%
  step_dummy(all_nominal())
```

.footnote[Use *before* `step_dummy()` so new level is dummified]

---
class: pop-quiz

# Consider

What would happen if you try to scale a variable that doesn't vary?

.display[Error!] You'd be dividing by zero!

---
# `step_zv()`

Intelligently handles zero variance variables (variables that contain only a single value)

```r
rec %>%
  step_novel(all_nominal()) %>%
  step_dummy(all_nominal()) %>%
* step_zv(all_predictors())
```

---
class: pop-quiz

# Pop quiz!

What step function would do PCA?

---
# `step_pca()`

Replaces variables with components

```r
rec %>%
  step_pca(all_numeric(),
*          num_comp = 5)
```

---
class: your-turn

# Your Turn 1

Write a recipe for the `Sale_Price ~ .` variables that:

1. Adds a novel level to all factors
1. Converts all factors to dummy variables
1. Catches any zero variance variables
1. Centers all of the predictors
1. Scales all of the predictors
1. Computes the first 5 principal components

Save the result as `pca_rec`

---
class: your-turn

.panelset[
.panel[.panel-name[Code]

```r
pca_rec <- 
  recipe(Sale_Price ~ ., data = ames) %>%
  step_novel(all_nominal()) %>%
  step_dummy(all_nominal()) %>%
  step_zv(all_predictors()) %>%
  step_center(all_predictors()) %>%
  step_scale(all_predictors()) %>%
  step_pca(all_predictors(), num_comp = 5)
```
]

.panel[.panel-name[Recipe]

```r
pca_rec
#> Data Recipe
#> 
#> Inputs:
#> 
#>       role #variables
#>    outcome          1
#>  predictor         80
#> 
#> Operations:
#> 
#> Novel factor level assignment for all_nominal()
#> Dummy variables from all_nominal()
#> Zero variance filter on all_predictors()
#> Centering for all_predictors()
#> Scaling for all_predictors()
#> No PCA components were extracted.
```
]
]

---
# Roles

You can also give variables a "role" within a recipe and then select by roles.

```r
has_role(match = "privacy")
add_role(rec, Fence, new_role = "privacy")
update_role(rec, Fence, new_role = "privacy", old_role = "yard")
remove_role(rec, Fence, old_role = "yard")
```

---
class: pop-quiz

# Pop quiz!

If we use `add_model()` to add a model to a workflow, what would we use to add a recipe?

Let's see!

---
class: your-turn

# Your Turn 2

Make a workflow that combines `pca_rec` and with `lm_spec`.

---
class: your-turn

.panelset[
.panel[.panel-name[Code]

```r
pca_wf <-
  workflow() %>%
  add_recipe(pca_rec) %>%
  add_model(lm_spec)
```
]

.panel[.panel-name[Workflow]

```r
pca_wf
#> ══ Workflow ════════════════════════════════════════════════════════════════════
#> Preprocessor: Recipe
#> Model: linear_reg()
#> 
#> ── Preprocessor ────────────────────────────────────────────────────────────────
#> 6 Recipe Steps
#> 
#> • step_novel()
#> • step_dummy()
#> • step_zv()
#> • step_center()
#> • step_scale()
#> • step_pca()
#> 
#> ── Model ───────────────────────────────────────────────────────────────────────
#> Linear Regression Model Specification (regression)
#> 
#> Computational engine: lm
```
]
]

---
# `add_recipe()`

Adds a recipe to a workflow.

```r
pca_wf <-
  workflow() %>%
* add_recipe(pca_rec) %>%
  add_model(lm_spec)
```

---
class: pop-quiz

# Pop quiz!

Do you need to add a formula if you have a recipe?

Nope!

```r
rec <-
* recipe(Sale_Price ~ .,
         data = ames)
```

---
class: your-turn

# Your Turn 3

Try our PCA workflow on `ames_folds`. What is the estimated RMSE?

---
class: your-turn

```r
pca_wf %>%
  fit_resamples(resamples = ames_folds) %>%
  collect_metrics()
#> # A tibble: 2 x 6
#>   .metric .estimator      mean     n   std_err .config             
#>   <chr>   <chr>          <dbl> <int>     <dbl> <chr>               
#> 1 rmse    standard   38937.       10 1349.     Preprocessor1_Model1
#> 2 rsq     standard       0.761    10    0.0158 Preprocessor1_Model1
```

---
# `update_recipe()`

Replace the recipe in a workflow.

```r
pca_wf %>%
* update_recipe(bc_rec)
```

---
class: your-turn

# Your Turn 4

Modify the code to build a new PCA recipe that uses a BoxCox transformation instead of centering and scaling the data.

Then update `pca_wf` to use the new recipe.

*Hint: Guess. Use tab completion. Or visit https://recipes.tidymodels.org/reference/index.html.*

---
class: your-turn

```r
bc_rec <-
  recipe(Sale_Price ~ ., data = ames) %>%
  step_novel(all_nominal()) %>%
  step_dummy(all_nominal()) %>%
  step_zv(all_predictors()) %>%
* step_BoxCox(all_predictors()) %>%
  step_pca(all_predictors(), num_comp = 5)

bc_wf <- 
  pca_wf %>%
  update_recipe(bc_rec)
```

---

```r
bc_wf %>%
  fit_resamples(resamples = ames_folds) %>%
  collect_metrics()
#> # A tibble: 2 x 6
#>   .metric .estimator      mean     n   std_err .config             
#>   <chr>   <chr>          <dbl> <int>     <dbl> <chr>               
#> 1 rmse    standard   44203.       10 2286.     Preprocessor1_Model1
#> 2 rsq     standard       0.699    10    0.0287 Preprocessor1_Model1
```

---
class: middle, center

# Feature Engineering

.pull-left[
Before

![](https://media.giphy.com/media/Wn74RUT0vjnoU98Hnt/giphy.gif)
]

.pull-right[
After

![](https://media.giphy.com/media/26tPgV8ceZTSxH9zG/giphy.gif)
]

---
# StackOverflow Data

```r
library(modeldata)
data(stackoverflow)
```

```r
glimpse(stackoverflow)
#> Rows: 5,594
#> Columns: 21
#> $ Country                              <fct> United Kingdom, United States, Un…
#> $ Salary                               <dbl> 100000.000, 130000.000, 175000.00…
#> $ YearsCodedJob                        <int> 20, 20, 16, 4, 1, 1, 13, 4, 7, 17…
#> $ OpenSource                           <dbl> 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, …
#> $ Hobby                                <dbl> 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, …
#> $ CompanySizeNumber                    <dbl> 5000, 1000, 10000, 1000, 5000, 20…
#> $ Remote                               <fct> Remote, Remote, Not remote, Not r…
#> $ CareerSatisfaction                   <int> 8, 9, 7, 9, 5, 8, 7, 7, 8, 9, 10,…
#> $ Data_scientist                       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Database_administrator               <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Desktop_applications_developer       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, …
#> $ Developer_with_stats_math_background <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, …
#> $ DevOps                               <dbl> 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Embedded_developer                   <dbl> 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Graphic_designer                     <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Graphics_programming                 <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Machine_learning_specialist          <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Mobile_developer                     <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, …
#> $ Quality_assurance_engineer           <dbl> 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Systems_administrator                <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ Web_developer                        <dbl> 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, …
```

???

Bigger version of what we used earlier.

---
class: pop-quiz

# Pop quiz!

Name that package!

```r
set.seed(100) # Important!
so_split <- initial_split(stackoverflow, strata = Remote)
so_train <- training(so_split)
so_test  <- testing(so_split)
```

---
class: pop-quiz

# Pop quiz!

Name that package!

```r
tree_spec <- 
  decision_tree() %>%         
  set_engine("rpart") %>%      
  set_mode("classification")
```

---
class: pop-quiz

# Pop quiz!

Name that package!

```r
so_rec <- recipe(Remote ~ ., data = so_train) %>% 
  step_dummy(all_nominal(), -all_outcomes()) %>% 
  step_lincomb(all_predictors())
```

---
class: pop-quiz

# Pop quiz!

Name that package!

```r
so_wf <- workflow() %>% 
  add_model(tree_spec) %>% 
  add_recipe(so_rec)
```

---
# `fit()`

```r
set.seed(1980)

so_fit <- so_wf %>%
  fit(data = so_train)

so_preds <- bind_cols(
  predict(so_fit, new_data = so_test, type = "class"),
  predict(so_fit, new_data = so_test, type = "prob")
) %>%
  mutate(truth = so_test$Remote)

so_metric_set <- metric_set(accuracy, roc_auc)
so_metric_set(so_preds, truth = truth, .pred_Remote, estimate = .pred_class)
#> # A tibble: 2 x 3
#>   .metric  .estimator .estimate
#>   <chr>    <chr>          <dbl>
#> 1 accuracy binary         0.898
#> 2 roc_auc  binary         0.5
```

---

```r
so_metric_set <- metric_set(accuracy, roc_auc, sens, spec)
so_metric_set(so_preds, truth = truth, .pred_Remote, estimate = .pred_class)
#> # A tibble: 4 x 3
#>   .metric  .estimator .estimate
#>   <chr>    <chr>          <dbl>
#> 1 accuracy binary         0.898
#> 2 sens     binary         0    
#> 3 spec     binary         1    
#> 4 roc_auc  binary         0.5
```

---
class: pop-quiz

# Pop quiz!

Can you guess what the confusion matrix looks like?

.left-column[
## uh oh
]

.right-column[

```r
conf_mat(so_preds, truth = truth, estimate = .pred_class)
#>             Truth
#> Prediction   Remote Not remote
#>   Remote          0          0
#>   Not remote    143       1254
```
]

---
class: pop-quiz

# Pop quiz!

Can you guess what the confusion matrix looks like?

.left-column[
## uh oh
]

.right-column[
<img src="images/recipes/plots/uh-oh-roc-1.png" width="50%" style="display: block; margin: auto;" />
]

---

```r
so_train %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       432
#> 2 Not remote  3765

so_test %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       143
#> 2 Not remote  1254
```

---
class: center middle inverse

# How can we get better at identifying the less frequent class?

Sub-class sampling

---
# Downsampling

.pull-left[
<img src="images/recipes/plots/uni-biscatter-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="images/recipes/plots/uni-downsamp-1.png" width="100%" style="display: block; margin: auto;" />
]

---
# Upsampling

.pull-left[
<img src="images/recipes/plots/uni-biscatter2-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="images/recipes/plots/uni-upsamp-1.png" width="100%" style="display: block; margin: auto;" />
]

---
class: your-turn

# Your Turn 5

Add a recipe step to downsample the remote variable majority class in the training set prior to model training. Edit your workflow, then re-fit the model and examine the metrics. Is the ROC AUC better than chance (.5)?

---
class: your-turn

.panelset[
.panel[.panel-name[Fit Workflow]

```r
so_down <- recipe(Remote ~ ., data = so_train) %>%
  step_dummy(all_nominal(), -all_outcomes()) %>%
  step_lincomb(all_predictors()) %>%
* step_downsample(all_outcomes())

so_downwf <- so_wf %>%
* update_recipe(so_down)

set.seed(1980)
so_downfit <- so_downwf %>%
  fit(data = so_train)

so_downpreds <- bind_cols(
  predict(so_downfit, new_data = so_test, type = "class"),
  predict(so_downfit, new_data = so_test, type = "prob")
) %>%
  mutate(truth = so_test$Remote)
```
]

.panel[.panel-name[Workflow Metrics]

```r
so_metric_set(so_downpreds, truth = truth, .pred_Remote, estimate = .pred_class)
#> # A tibble: 4 x 3
#>   .metric  .estimator .estimate
#>   <chr>    <chr>          <dbl>
#> 1 accuracy binary         0.658
#> 2 sens     binary         0.552
#> 3 spec     binary         0.670
#> 4 roc_auc  binary         0.630
```
]
]

---
.left-column[
## Ahh!
]

.right-column[
<img src="images/recipes/plots/ahh-roc-1.png" width="70%" style="display: block; margin: auto;" />
]

---
# `juice()`

Get the preprocessed training data back from a prepped recipe. Returns a tibble.

```r
so_down %>%
  prep(training = so_train) %>%
  juice()
```

---
# Downsampling

.pull-left[

```r
so_train %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       432
#> 2 Not remote  3765
```
]

.pull-right[

```r
so_down %>%
  prep(training = so_train) %>%
* juice() %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       432
#> 2 Not remote   432
```
]

---
# `step_downsample()`

Down-sampling is performed on the training set *only*. Default is `skip = TRUE`.

.pull-left[

```r
so_test %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       143
#> 2 Not remote  1254
```
]

.pull-right[

```r
so_down %>%
  prep(training = so_train) %>%
  bake(new_data = so_test) %>%
  count(Remote)
#> # A tibble: 2 x 2
#>   Remote         n
#>   <fct>      <int>
#> 1 Remote       143
#> 2 Not remote  1254
```
]

---
class: title-slide, center

# Feature Engineering

## Tidy Data Science with the Tidyverse and Tidymodels

### W. Jake Thompson

#### [https://tidyds-2021.wjakethompson.com](https://tidyds-2021.wjakethompson.com) &#183; [https://bit.ly/tidyds-2021](https://bit.ly/tidyds-2021)

.footer-license[*Tidy Data Science with the Tidyverse and Tidymodels* is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).]