27  Modeling Workflow

A model workflow is important in two ways. First, using a workflow encourages good methodology since it is a single point of entry to the estimation components of a data analysis. Second, it enables the user to better organize projects

The typical path to determining an appropriate model is an iterative process. The general phases are:

• Exploratory data analysis (EDA): Initially there is a back and forth between numerical analysis and data visualization where different discoveries lead to more questions and data analysis side-quests to gain more understanding.

• Feature engineering: The understanding gained from EDA results in the creation of specific model terms that make it easier to accurately model the observed data. This can include complex methodologies (e.g., PCA) or simpler features (using the ratio of two predictors).

• Model tuning and selection: A variety of models are generated and their performance is compared. Some models require parameter tuning in which some structural parameters must be specified or optimized. This is the stage where repeated data splitting is used for resampling.

• Model evaluation: During this phase of model development, we assess the model’s performance metrics, examine residual plots, and conduct other EDA-like analyses to understand how well the models work. In some cases, formal between-model comparisons help you understand whether any differences in models are within the experimental noise.

After an initial sequence of these tasks, more understanding is gained regarding which models are superior as well as which data subpopulations are not being effectively estimated. This leads to additional EDA and feature engineering, another round of modeling, and so on. Once the data analysis goals are achieved, typically the last steps are to finalize, document, and communicate the model. For predictive models, it is common at the end to validate the model on an additional set of data reserved for this specific purpose.