At the start of a new modeling project, there is usually an initial finite pool of data available for all the tasks, which we can think of as an available data budget.
When a project has a limited amount of appropriate data, wise spending of a project’s data is necessary. How should the data then be applied to different steps or tasks? The idea of data spending is an important first consideration when modeling, especially as it relates to empirical validation.
The primary approach for empirical model validation is to split the existing pool of data into two distinct sets:
The training set. It is usually the majority of the data. These data are a sandbox for model building where different models can be fit, feature engineering strategies are investigated, and so on. As modeling practitioners, we spend the vast majority of the modeling process using the training set as the substrate to develop the model.
The test set. This is held in reserve until one or two models are chosen as the methods most likely to succeed. The test set is then used as the final arbiter to determine the efficacy of the model. More specifically, it is critical to:
Note that the proportion of data that should be allocated for splitting is highly dependent on the context of the problem at hand. Too little data in the training set hampers the model’s ability to find appropriate parameter estimates. Conversely, too little data in the test set lowers the quality of the performance estimates. Bear in mind that keeping the training data in a separate data frame from the test set is one small check to make sure that information leakage does not occur by accident.
Suppose we allocate 80% of the data to the training set and the remaining 20% for testing. The most common method is to use simple random sampling. The rsample package has tools for making data splits such as this; the function initial_split() was created for this purpose. It takes the data frame as an argument as well as the proportion to be placed into training.
Example:
library(tidymodels)
tidymodels_prefer()
# Set the random number stream so that the results can be
# reproduced later
set.seed(501)
# Save the split information for an 80/20 split of the data
ames_split <- initial_split(ames, prop = 0.80)
ames_split<Training/Testing/Total>
<2344/586/2930>The printed information denotes the amount of data in the training set (n=2,344), the amount in the test set (n=586), and the size of the original pool of samples (n=2,930).
The object ames_split is an rsplit object and contains only the partitioning information; to get the resulting data sets, we apply two more functions:
ames_train <- training(ames_split)
ames_test <- testing(ames_split)
dim(ames_train)[1] 2344 74These objects are data frames with the same columns as the original data but only the appropriate rows for each set.
Simple random sampling is appropriate in many cases but there are exceptions. When there is a dramatic class imbalance in classification problems, one class occurs much less frequently than another. Using a simple random sample may haphazardly allocate these infrequent samples disproportionately into the training or test set. To avoid this, stratified sampling can be used. The training/test split is conducted separately within each class and then these subsamples are combined into the overall training and test set.
As an example, a stratified random sample would conduct the 80/20 split within each of these data subsets and then pool the results. In rsample, this is achieved using the strata argument:
Example:
set.seed(502)
# Only a single column can be used for stratification (e.g. Sale_Price)
ames_split <- initial_split(ames, prop = 0.80, strata = Sale_Price)
ames_train <- training(ames_split)
ames_test <- testing(ames_split)
dim(ames_train)[1] 2342 74There is one situation when random sampling is not the best choice: when the data have a significant time component, such as time series data. Here, it is more common to use the most recent data as the test set.
The rsample package contains a function called initial_time_split() that is very similar to initial_split(). Instead of using random sampling, the prop argument denotes what proportion of the first part of the data should be used as the training set; the function assumes that the data have been pre-sorted in an appropriate order.
In the neural network and deep learning literature it is common to hear about validation sets as an answer to the question about how we can tell what is the best test set that should be used to properly evaluate a model performance on the final model(s), if we don’t measure performance until the test set?
During the early days of neural networks, researchers realized that measuring performance by re-predicting the training set samples led to results that were overly optimistic (significantly, unrealistically so). This led to models that overfit, meaning that they performed very well on the training set but poorly on the test set.12 To combat this issue, a small validation set of data were held back and used to measure performance as the network was trained. Once the validation set error rate began to rise, the training would be halted. In other words, the validation set was a means to get a rough sense of how well the model performed prior to the test set.
Validation sets are a special case of resampling methods that are used on the training set.
Example:
set.seed(52)
# To put 60% into training, 20% in validation, and 20% in testing:
ames_val_split <- initial_validation_split(ames, prop = c(0.6, 0.2))
ames_val_split<Training/Validation/Testing/Total>
<1758/586/586/2930>To get the training, validation, and testing data, the same syntax is used:
ames_train <- training(ames_val_split)
ames_test <- testing(ames_val_split)
ames_val <- validation(ames_val_split)