In the lending industry, investors provide loans to borrowers in exchange for the promise of repayment with interest.
If the borrower repays the loan, then the lender profits from the interest. However, if the borrower is unable to repay the loan,
then the lender loses money. Therefore, lenders face the problem of predicting the risk of a borrower being unable to repay a loan.
To address this problem, we will use publicly available data loans.csv
from LendingClub.com, a website that connects borrowers and investors over the Internet.
This dataset represents 9,578 3-year loans that were funded through the LendingClub.com platform between May 2007 and February 2010.
The binary dependent variable not_fully_paid indicates that the loan was not paid back in full (the borrower either defaulted or the loan was "charged off,"
meaning the borrower was deemed unlikely to ever pay it back).
To predict this dependent variable, we will use the following independent variables available to the investor when deciding
whether to fund a loan:
- credit.policy: 1 if the customer meets the credit underwriting criteria of LendingClub.com, and 0 otherwise.
- purpose: The purpose of the loan (takes values "credit_card", "debt_consolidation", "educational", "major_purchase", "small_business", and "all_other").
- int.rate: The interest rate of the loan, as a proportion (a rate of 11% would be stored as 0.11). Borrowers judged by LendingClub.com to be more risky are assigned higher interest rates.
- installment: The monthly installments ($) owed by the borrower if the loan is funded.
- log.annual.inc: The natural log of the self-reported annual income of the borrower.
- dti: The debt-to-income ratio of the borrower (amount of debt divided by annual income).
- fico: The FICO credit score of the borrower.
- days.with.cr.line: The number of days the borrower has had a credit line.
- revol.bal: The borrower's revolving balance (amount unpaid at the end of the credit card billing cycle).
- revol.util: The borrower's revolving line utilization rate (the amount of the credit line used relative to total credit available).
- inq.last.6mths: The borrower's number of inquiries by creditors in the last 6 months.
- delinq.2yrs: The number of times the borrower had been 30+ days past due on a payment in the past 2 years.
-
pub.rec: The borrower's number of derogatory public records (bankruptcy filings, tax liens, or judgments).
loans = read.csv("loans.csv")
table(loans$not.fully.paid)
We see that 1533 loans were not paid, and 8045 were fully paid. Therefore, the proportion of loans not paid is 1533/(1533+8045)=0.1601.
From summary(loans), we can read that log.annual.inc, days.with.cr.line, revol.util, inq.last.6mths, delinq.2yrs and pub.rec are missing values.
missing = subset(loans, is.na(log.annual.inc) | is.na(days.with.cr.line) | is.na(revol.util) | is.na(inq.last.6mths) | is.na(delinq.2yrs) | is.na(pub.rec))
From nrow(missing), we see that only 62 of 9578 loans have missing data; removing this small number of observations would not lead to overfitting.
From table(missing$not.fully.paid), we see that 12 of 62 loans with missing data were not fully paid, or 19.35%. This rate is similar to the 16.01%
across all loans, so the form of biasing described is not an issue. However, to predict risk for loans with missing data we need to fill in the missing
values instead of removing the observations.
To fill the missing values:
library(mice)
set.seed(144)
vars.for.imputation = setdiff(names(loans), "not.fully.paid")
imputed = complete(mice(loans[vars.for.imputation]))
loans[vars.for.imputation] = imputed
Note that to do this imputation, we set vars.for.imputation to all variables in the data frame except for not.fully.paid, to impute the values using all of the other independent variables.
Imputation predicts missing variable values for a given observation using the variable values that are reported. We called the imputation on a data frame with the dependent
variable not.fully.paid removed, so we predicted the missing values using only other independent variables.
Prediction Models
Now that we have prepared the dataset, we need to split it into a training and testing set. To ensure everybody obtains the same split,
set the random seed to 144 (even though you already did so earlier in the problem) and use the sample.split function to select the 70% of
observations for the training set (the dependent variable for sample.split is not.fully.paid). Name the data frames train and test.
Now, we use logistic regression trained on the training set to predict the dependent variable not.fully.paid using all the independent variables.
library(caTools)
set.seed(144)
spl = sample.split(loans$not.fully.paid, 0.7)
train = subset(loans, spl == TRUE)
test = subset(loans, spl == FALSE)
The model can be trained and summarized with the following commands:
mod = glm(not.fully.paid~., data=train, family="binomial")
summary(mod)
Variables that are significant have at least one star in the coefficients table of the summary output.
Note that some have a positive coefficient (meaning that higher values of the variable lead to an increased risk of defaulting)
and some have a negative coefficient (meaning that higher values of the variable lead to a decreased risk of defaulting).
Consider two loan applications, which are identical other than the fact that the borrower in Application A has FICO credit score 700 while
the borrower in Application B has FICO credit score 710.
Let Logit(A) be the log odds of loan A not being paid back in full, according to our logistic regression model, and define Logit(B) similarly for loan B.
Because Application A is identical to Application B other than having a FICO score 10 lower,
its predicted log odds differ by -0.009317 * -10 = 0.09317 from the predicted log odds of Application B.
Now, let O(A) be the odds of loan A not being paid back in full, according to our logistic regression model, and define O(B)
similarly for loan B. In order to calculate the value of O(A)/O(B):
The predicted odds of loan A not being paid back in full are exp(0.09317) = 1.0976 times larger than the predicted odds for loan B.
Intuitively, it makes sense that loan A should have higher odds of non-payment than loan B, since the borrower has a worse credit score.
The confusion matrix can be computed with the following commands:
test$predicted.risk = predict(mod, newdata=test, type="response")
table(test$not.fully.paid, test$predicted.risk > 0.5)
2403 predictions are correct (accuracy 2403/2873=0.8364), while 2413 predictions would be correct in the baseline model of guessing every loan
would be paid back in full (accuracy 2413/2873=0.8399).
Value of the AUC is 0.672. The test set AUC can be computed with the following commands:
library(ROCR)
pred = prediction(test$predicted.risk, test$not.fully.paid)
as.numeric(performance(pred, "auc")@y.values)
The model has poor accuracy at the threshold 0.5. But despite the poor accuracy, we will see later how an investor can still leverage this logistic regression
model to make profitable investments.
A "Smart Baseline"
In the previous problem, we built a logistic regression model that has an AUC significantly higher than the AUC of 0.5 that would be obtained by randomly
ordering observations.
However, LendingClub.com assigns the interest rate to a loan based on their estimate of that loan's risk. This variable, int.rate, is an independent variable in
our dataset. In this part, we will investigate using the loan's interest rate as a "smart baseline" to order the loans according to risk.
Using the training set, we can build a bivariate logistic regression model (aka a logistic regression model with a single independent variable) that predicts
the dependent variable not.fully.paid using only the variable int.rate.
bivariate = glm(not.fully.paid~int.rate, data=train, family="binomial")
summary(bivariate)
Decreased significance between a bivariate and multivariate model is typically due to correlation. From cor(train$int.rate, train$fico),
we can see that the interest rate is moderately well correlated with a borrower's credit score.
The variable int.rate is highly significant in the bivariate model, but it is not significant at the 0.05 level in the model trained with all the
independent variables.
int.rate is correlated with other risk-related variables, and therefore does not incrementally improve the model when those other variables are included.
pred.bivariate = predict(bivariate, newdata=test, type="response")
summary(pred.bivariate)
According to the summary function, the maximum predicted
probability of the loan not being paid back is 0.4266, which means no loans would be flagged at a logistic regression cutoff of 0.5.
The AUC is 0.624 and can be computed with:
prediction.bivariate = prediction(pred.bivariate, test$not.fully.paid)
as.numeric(performance(prediction.bivariate, "auc")@y.values)
Computing the Profitability of an Investment
While thus far we have predicted if a loan will be paid back or not, an investor needs to identify loans that are expected to be profitable.
If the loan is paid back in full, then the investor makes interest on the loan. However, if the loan is not paid back, the investor loses the
money invested. Therefore, the investor should seek loans that best balance this risk and reward.
To compute interest revenue, consider a $c investment in a loan that has an annual interest rate r over a period of t years. Using continuous compounding
of interest, this investment pays back c * exp(rt) dollars by the end of the t years, where exp(rt) is e raised to the r*t power.
So, How much does a $10 investment with an annual interest rate of 6% pay back after 3 years, using continuous compounding of interest?
As we have c=10, r=0.06, and t=3. Using the formula above, the final value is 10exp(0.063) = 11.97.
While the investment has value c * exp(rt) dollars after collecting interest, the investor had to pay $c for the investment.
The profit to the investor if the investment is paid back in full can be calculated as the investor gets c * exp(rt) but paid c, yielding a profit of c * exp(rt) - c.
Now, consider the case where the investor made a $c investment, but it was not paid back in full.
Assume, conservatively, that no money was received from the borrower (often a lender will receive some but not all of the value of the loan,
making this a pessimistic assumption of how much is received). the profit in this case would be -c$
A Simple Investment Strategy
In the previous subproblem, we concluded that an investor who invested c dollars in a loan with interest rate r for t years makes c * (exp(rt) - 1)
dollars of profit if the loan is paid back in full and -c dollars of profit if the loan is not paid back in full (pessimistically).
In order to evaluate the quality of an investment strategy, we need to compute this profit for each loan in the test set.
For this variable, we will assume a $1 investment (aka c=1). To create the variable, we first assign to the profit for a fully paid loan, exp(rt)-1,
to every observation, and we then replace this value with -1 in the cases where the loan was not paid in full. All the loans in our dataset are 3-year loans,
meaning t=3 in our calculations. Enter the following commands in your R console to create this new variable:
test$profit = exp(test$int.rate*3) - 1
test$profit[test$not.fully.paid == 1] = -1
Let's calculate the maximum profit of a $10 investment in any loan in the testing set.
We see the maximum profit for a $1 investment in any loan is $0.8895. Therefore, the maximum profit of a $10 investment is 10 times as large, or $8.895.