Skip to main content

14.5 Validating Against Spec

The acceptance criteria in Section 2 only become useful if they can be checked — preferably without manual effort, and ideally before the project is shipped. Validation-first analytics, introduced in Chapter 13, is the discipline of turning each spec criterion into an executable test that runs alongside the analysis itself, so failures surface immediately rather than during a stakeholder demo.

We follow the pattern from Chapter 13’s validate_absenteeism() function. Each acceptance criterion becomes a check that returns TRUE (passed) or FALSE (failed), with an informative message either way. Wrapped together, the checks form the project’s validation suite.

validate_data_quality <- function(data) {
  list(
    completeness_age = list(
      passed  = all(!is.na(data$Age)),
      message = "All employees have a non-missing Age."
    ),
    completeness_service = list(
      passed  = all(!is.na(data$`Service time`)),
      message = "All employees have a non-missing Service time."
    ),
    accuracy_age_range = list(
      passed  = all(data$Age >= 27 & data$Age <= 58),
      message = "All Age values fall in the documented 27-58 range."
    ),
    accuracy_bmi_range = list(
      passed  = all(data$`Body mass index` >= 19 & data$`Body mass index` <= 40),
      message = "All BMI values fall within plausible bounds (19-40)."
    ),
    consistency_education = list(
      passed  = all(levels(data$Education) %in%
        c("high school", "graduate", "postgraduate", "master and doctor")),
      message = "Education levels match the documented factor labels."
    )
  )
}

validate_deliverables <- function(model, accuracy, threshold = 0.70) {
  predictors <- attr(terms(model), "term.labels")
  forbidden  <- c("Gender", "Ethnicity", "Religion")
  list(
    no_protected_predictors = list(
      passed  = !any(forbidden %in% predictors),
      message = "Model formula excludes protected characteristics."
    ),
    accuracy_meets_kpi = list(
      passed  = accuracy >= threshold,
      message = sprintf(
        "Held-out accuracy %.1f%% meets the %.0f%% KPI threshold.",
        accuracy * 100, threshold * 100
      )
    )
  )
}

The two functions split validation along the dimensions the BRS itself uses: data must meet quality criteria before the analysis runs, and deliverables must meet KPIs before the project ships. Wrapping each check as a list of passed and message makes the output easy to inspect and easy to render in a report.

checks <- c(
  validate_data_quality(employee_data),
  validate_deliverables(model, accuracy)
)

results <- tibble::tibble(
  check   = names(checks),
  passed  = vapply(checks, \(x) x$passed,  logical(1)),
  message = vapply(checks, \(x) x$message, character(1))
)

Table 14.9: Table 14.10: Spec validation results.

check	passed	message
completeness_age	PASS	All employees have a non-missing Age.
completeness_service	PASS	All employees have a non-missing Service time.
accuracy_age_range	PASS	All Age values fall in the documented 27-58 range.
accuracy_bmi_range	PASS	All BMI values fall within plausible bounds (19-40).
consistency_education	PASS	Education levels match the documented factor labels.
no_protected_predictors	PASS	Model formula excludes protected characteristics.
accuracy_meets_kpi	PASS	Held-out accuracy 81.8% meets the 70% KPI threshold.

When every row reads PASS, the project meets the spec it was built against. When any row reads FAIL, the message tells the analyst exactly what is wrong — the missing variable, the failed range, the protected predictor that slipped into the formula. This is the closing of the spec-driven loop: the spec drove the work, and the work is now mechanically checked against the spec before it is shipped.

The validation suite earns its keep on iteration. When HR returns next quarter and asks for an updated model, the analyst re-runs the analysis, re-renders the report, and re-runs the validation. If anything has drifted — a new data source has different ranges, a stakeholder added a forbidden predictor, accuracy dipped below the KPI — the failing check surfaces it before anyone reads the report. Specs that exist on paper alone tend to rot. Specs that exist as executable tests tend to stay current.