power: Power of statistical tests for predictive ability testing

In PDtoolkit: Collection of Tools for PD Rating Model Development and Validation

Power of statistical tests for predictive ability testing

Description

power performs Monte Carlo simulation of power of statistical test used for testing the predictive ability of the PD rating model. It covers fours tests: the binomial, Jeffreys, z-score and Hosmer-Lemeshow test. This procedure is applied under assumption that the observed default rate is the true one and it is used to check if calibrated PDs are underestimated for the binomial, Jeffreys, and z-score. Therefore, for the cases where observed default rate is lower than the calibrated PD, the power calculation is not performed and will report the comment. For the Hosmer-Lemeshow test is used to test if the calibrated PD is the true one regardless the difference between the observed and calibrated portfolio default rate.

Usage

power(rating.label, pdc, no, nb, alpha = 0.05, sim.num = 1000, seed = 2211)

Arguments

rating.label	Vector of rating labels.
pdc	Vector of calibrated probabilities of default (PD).
no	Number of observations per rating grade.
nb	Number of defaults (bad cases) per rating grade.
alpha	Significance level of p-value for implemented tests. Default is 0.05.
sim.num	Number of Monte Carlo simulations. Default is 1000.
seed	Random seed needed for ensuring the result reproducibility. Default is 2211.

Details

Due to the fact that test of predictive power is usually implemented on the application portfolio, certain prerequisites are needed to be fulfilled. In the first place model should be developed and rating scale should be formed. In order to reflect appropriate role and right moment of tests application, presented simplified example covers all steps before test implementation.

Value

The command power returns a list with two objects. Both are the data frames and while the first one presents power calculation of the tests applied usually on the rating level (binomial, Jeffreys and z-score test), the second one presents results of the Hosmer-Lemeshow test which is applied on the complete rating scale. For both level of the implementation (rating or complete scale) if the observed default rate is less than calibrated PD, function will return the comment and power simulation will not be performed.

Examples

suppressMessages(library(PDtoolkit))
data(loans)
#estimate some dummy model
mod.frm <- `Creditability` ~ `Account Balance` + `Duration of Credit (month)` +
				`Age (years)`
lr.mod <- glm(mod.frm, family = "binomial", data = loans)
summary(lr.mod)$coefficients
#model predictions
loans$pred <-  unname(predict(lr.mod, type = "response", newdata = loans))
#scale probabilities
loans$score <- scaled.score(probs = loans$pred, score = 600, odd = 50/1, pdo = 20)
#group scores into rating
loans$rating <- sts.bin(x = round(loans$score), y = loans$Creditability, y.type = "bina")[[2]]
#create rating scale
rs <- loans %>%
group_by(rating) %>%
summarise(no = n(),
	    nb = sum(Creditability),
	    ng = sum(1 - Creditability)) %>%
mutate(dr = nb / no)
rs
#calcualte portfolio default rate
sum(rs$dr * rs$no / sum(rs$no))
#calibrate rating scale to central tendency of 27% with minimum PD of 5%
ct <- 0.27
min.pd <- 0.05
rs$pd <- rs.calibration(rs = rs, 
			dr = "dr", 
			w = "no", 
			ct = ct, 
			min.pd = min.pd,
			method = "log.odds.ab")[[1]]
#check
rs
sum(rs$pd * rs$no / sum(rs$no))
#simulate some dummy application portfolio
set.seed(22)
app.port <- loans[sample(1:nrow(loans), 400), ]
#summarise application portfolio on rating level
ap.summary <- app.port %>%
	  group_by(rating) %>%
	  summarise(no = n(),
			nb = sum(Creditability),
			ng = sum(1 - Creditability)) %>%
	  mutate(dr = nb / no)
#bring calibrated pd as a based for predictive power testing
ap.summary <- merge(rs[, c("rating", "pd")], ap.summary, by = "rating", all.x = TRUE)
ap.summary
#perform predictive power testing
pp.res <- pp.testing(rating.label = ap.summary$rating,
		     pdc = ap.summary$pd,
		     no = ap.summary$no,
		     nb = ap.summary$nb, 
		     alpha = 0.05)
pp.res
power(rating.label = ap.summary$rating,
  pdc = ap.summary$pd,
  no = ap.summary$no,
  nb = ap.summary$nb, 
  alpha = 0.05,
  sim.num = 1000,
  seed = 2211)

PDtoolkit documentation built on Sept. 20, 2023, 9:06 a.m.