9.6 Validation, Deployment, and Monitoring
9.6.1 Model Validation
Validation assesses whether a model will perform well on data it has not seen. Two approaches are commonly used:
A practical demonstration of k-fold cross-validation:
# 5-fold cross-validation of a linear model
set.seed(42)
folds <- cut(sample(1:nrow(mtcars)), breaks = 5, labels = FALSE)
cv_rmse <- numeric(5)
for (k in 1:5) {
test_idx <- which(folds == k)
cv_train <- mtcars[-test_idx, ]
cv_test <- mtcars[test_idx, ]
fit <- lm(mpg ~ wt + cyl + hp, data = cv_train)
preds <- predict(fit, cv_test)
cv_rmse[k] <- sqrt(mean((cv_test$mpg - preds)^2))
}
data.frame(
Fold = 1:5,
RMSE = round(cv_rmse, 2)
) |> rbind(data.frame(Fold = NA, RMSE = round(mean(cv_rmse), 2))) |>
mutate(Fold = ifelse(is.na(Fold), "Mean", as.character(Fold)))| Fold | RMSE |
|---|---|
| 1 | 1.92 |
| 2 | 3.16 |
| 3 | 2.64 |
| 4 | 3.39 |
| 5 | 1.94 |
| Mean | 2.61 |
Each fold uses a different 20% of the data as the test set, giving five independent estimates of prediction error. The mean RMSE across folds is a more robust estimate than any single train/test split.
In-sample metrics evaluate how well the model fits its training data:
- R² (R-squared): The proportion of variance explained by the model. Higher is better, but a high R² on training data alone can indicate overfitting.
- Adjusted R²: Penalizes R² for the number of predictors, favoring simpler models.
- AIC and BIC: Information criteria that balance fit against complexity — lower values indicate a better trade-off.
Out-of-sample metrics evaluate how well the model predicts new data:
- Holdout method: Split data into training and test sets (as demonstrated above).
- k-fold cross-validation: Divide data into k subsets, train on k-1, test on the remaining one, and rotate. This gives a more robust estimate of performance than a single train/test split.
- RMSE (Root Mean Squared Error): For regression models — the average magnitude of prediction errors, in the same units as the outcome variable.
- Accuracy, Precision, Recall: For classification models — how often the model correctly identifies each class.
Out-of-sample performance is always more informative than in-sample performance. A model that looks great on training data but fails on test data is not ready for deployment.
9.6.2 Deployment
Deployment integrates the model into business processes where it begins informing real decisions. The deployment strategy depends on the model’s purpose: a fraud detection model must operate in real time within the transaction processing workflow, while a demand forecasting model might run weekly within a BI dashboard.
Key considerations include compatibility with existing IT systems, scalability, security, and stakeholder buy-in. Deployment often requires collaboration across data science, IT, and business teams.
9.6.3 Monitoring and Maintenance
Models degrade over time as the world changes — customer behaviors shift, markets evolve, and new competitors emerge. Continuous monitoring tracks performance metrics to detect when a model’s accuracy begins to decline.
When performance drops, the model may need retraining with fresh data, hyperparameter adjustment, or in some cases a complete rebuild. This maintenance cycle also provides an opportunity to revisit ethical and regulatory compliance — especially in sectors like finance and healthcare, where models directly affect people’s lives (Barocas et al. 2023).
The modeling workflow is not a one-time process but an ongoing cycle: build, validate, deploy, monitor, and refine.