Beyond churn: applying modelimpact to other use cases

The same engine, many problems

modelimpact was first written to reason about the business value of a customer churn model, but nothing in the maths is churn-specific. Every function does the same thing: it takes a column of predicted probabilities for an event of interest and a column with the actual outcome, then translates the model’s behaviour into money using a handful of cost and value assumptions.

That makes the package directly usable for any binary classification problem where acting on a prediction has a cost and being right (or wrong) has a value, for example:

• Fraud detection — value of blocking a fraudulent transaction vs. the cost of reviewing a flagged one and the loss from fraud that slips through.
• Credit default — expected loss avoided vs. the cost of intervention.
• Lead scoring and marketing response — value of a converted lead vs. the cost of contacting it.
• Upsell and cross-sell — margin on an accepted offer vs. the cost of making it (use prob_accept to model take-up).
• Predictive maintenance — cost of an averted failure vs. the cost of an inspection.

The key is the positive argument, which every function accepts: it names the value in your outcome column that identifies the event of interest. It defaults to "Yes", but it can be anything ("fraud", "default", "converted", …).

A worked example: fraud detection

Suppose we have scored a batch of transactions with a fraud model. Each row is a transaction with the model’s predicted probability of fraud (score) and whether it later turned out to be fraudulent (fraud). Fraud is rare, so the classes are highly imbalanced.

library(modelimpact)
library(magrittr)  # for the %>% pipe

set.seed(42)
n <- 5000

# ~3% of transactions are actually fraudulent
is_fraud <- rbinom(n, size = 1, prob = 0.03)

# the model scores fraud higher than legitimate transactions, with overlap
score <- plogis(rnorm(n, mean = ifelse(is_fraud == 1, 1.5, -2.5), sd = 1.2))

transactions <- data.frame(
  score = score,
  fraud = ifelse(is_fraud == 1, "fraud", "legit")
)

head(transactions)
#>        score fraud
#> 1 0.14689107 legit
#> 2 0.07547704 legit
#> 3 0.06852903 legit
#> 4 0.11711395 legit
#> 5 0.14266641 legit
#> 6 0.07445047 legit

review_cost  <- 8      # cost to manually review a flagged transaction
fraud_value  <- 400    # average value recovered when fraud is caught
missed_fraud <- -400   # average loss when fraud is not caught

library(ggplot2)

transactions %>%
  cost_revenue(
    var_cost  = review_cost,
    tp_val    = fraud_value,
    prob_col  = score,
    truth_col = fraud,
    positive  = "fraud"
  ) %>%
  autoplot()

transactions %>%
  profit(
    var_cost  = review_cost,
    tp_val    = fraud_value,
    prob_col  = score,
    truth_col = fraud,
    positive  = "fraud"
  ) %>%
  autoplot()

impact_summary(
  transactions,
  var_cost  = review_cost,
  tp_val    = fraud_value,
  prob_col  = score,
  truth_col = fraud,
  positive  = "fraud"
)
#> # A tibble: 1 × 6
#>   optimal_pct max_profit roi_at_optimum capture_at_optimum breakeven_pct
#>         <dbl>      <dbl>          <dbl>              <dbl>         <dbl>
#> 1        12.5      53384           10.6              0.993           100
#> # ℹ 1 more variable: profit_target_all <dbl>

transactions %>%
  profit_thresholds(
    var_cost  = review_cost,
    tp_val    = fraud_value,
    fp_val    = 0,
    tn_val    = 0,
    fn_val    = missed_fraud,
    prob_col  = score,
    truth_col = fraud,
    positive  = "fraud"
  ) %>%
  autoplot()

Adapting to your own use case

To reuse any of these functions for a different problem, you only need to:

1. Point prob_col at your model’s predicted-probability column and truth_col at the actual-outcome column.
2. Set positive to the label of the event you care about.
3. Translate your domain into the cost and value arguments (var_cost, fixed_cost, tp_val, and — for profit_thresholds() — fp_val, tn_val, fn_val), optionally using prob_accept when an action only incurs its cost if accepted.

The interpretation of every plot and summary then carries over unchanged.

Regression models: targeting by predicted value

Not every model predicts a class. Sometimes the model predicts a continuous value — expected customer lifetime value (CLV), next-year spend, claim size, or expected loss — and we want to target the cases with the highest predicted value. value_gains() and value_profit() are the regression counterparts of cumulative_gains() and profit(): instead of a positive class they take the model’s prediction (pred_col) and the realised value (value_col).

Here we simulate customers whose predicted CLV is correlated with, but not identical to, their realised CLV.

set.seed(99)
n <- 2000

realised_clv  <- rgamma(n, shape = 2, scale = 300)          # actual value
predicted_clv <- realised_clv * runif(n, 0.4, 1.6)          # noisy model score

customers <- data.frame(
  predicted_clv = predicted_clv,
  realised_clv  = realised_clv
)

head(customers)
#>   predicted_clv realised_clv
#> 1      633.7986     532.0483
#> 2      452.0508     643.4930
#> 3      638.7189     482.8489
#> 4      404.2273     627.8598
#> 5      244.6784     326.5587
#> 6      579.0380     455.0351

vg <- value_gains(customers, pred_col = predicted_clv, value_col = realised_clv)
autoplot(vg)

# the coefficient is also available programmatically
attr(vg, "gini")
#> [1] 0.8814741

customers %>%
  value_profit(
    var_cost  = 50,
    pred_col  = predicted_clv,
    value_col = realised_clv
  ) %>%
  autoplot()