Can automated scoring do everything? Head of Risk (CRO) TAMGA Dmitry Melnik talks about the features of different scoring models and the nuances of their application.
The application of scoring in lending allows the automation of the decision-making process and increases the efficiency of the loan portfolio, minimizing the risks of non-repayment of loans, as well as faster processing of applications. The essence of scoring is to create a mathematical model that estimates the probability that the borrower will repay the loan on time. In practice, this consists in achieving a monotonic dependence of the scoring grade on the probability of finding a certain desirable or undesirable characteristic of a borrower.
The widespread use of scoring models when dealing with large portfolios of homogeneous loans is due to the fact that the analysis of each individual loan takes limited time, while information about the borrower is often limited and well-structured.
Scoring model development
Most often, when developing scoring models, one is faced with a situation where it is necessary to determine whether a borrower has exhibited a required characteristic or not. First of all, it is necessary to be able to distinguish between "bad" and "good" borrowers in the scoring model being developed, namely to classify them by the degree of presence of the required characteristic.
One of the first models for solving this problem is regression, which allows predicting the probability of occurrence of an event or estimating the impact of various attributes on a target variable, such as the probability of loan repayment. As a result of analyzing borrowers, the pool of borrowers is divided into groups in such a way that the scoring points within each group are the same, and each group corresponds to its degree (probability) of the presence of the desired characteristic.
Regression is not the only approach capable of solving the problem of classifying borrowers by the degree of presence of a certain characteristic. Similar results can be achieved using decision trees and neural networks. Both of these models are nonlinear and can account for a wider range of patterns than models based on linear functions such as regression.
The result of applying the decision tree method is to rank borrowers according to the probability of manifestation of the desired characteristic. But unlike regression, where a scoring grade is used, each leaf in the tree is characterized by a probability derived from the observed distribution of the target characteristic. This probability can be converted into a scoring grade using a logistic formula. Neural networks, in turn, combine the ability to account for the non-linear nature of decision trees with the ability to compute the scoring grade explicitly.
Features of different scoring methods application
If we consider the advantages and disadvantages of classification models in general, we can distinguish two opposing approaches - regression and neural networks. In most cases, it is sufficient for regression to correctly classify borrowers as "good" or "bad" based on their scoring grade, as an accurate probability prediction using this method is overly complex. Thus, regressions may display relatively simple relationships, but they have more stability in performance.
Neural networks are able to take into account dependencies of any complexity, but they may suddenly lose their predictive ability. This instability is due to their high sensitivity to the smallest changes in the training sample. To ensure the stability of the model, it is common to use an approach that divides the training sample into learning and validation parts, as well as a separate sample that belongs to a different time period for the final validation.
With all this in mind, the choice of a particular method depends on the data, the task, and the requirements of the scoring model.
Scoring in consumer lending
Two types of rating are commonly used in consumer lending: questionnaire scoring and behavioral scoring.
Questionnaire scoring is the most popular method of assessing new borrowers. It is used at the stage when the loan has not yet been disbursed and only data from the borrower's questionnaire is available. The target property that this scoring should predict is usually whether the borrower reaches default or a certain delinquency status within a certain period after the loan is granted.
Over time, as a loan's payment history accumulates, the importance of the questionnaire data for predicting future defaults decreases and the importance of the credit history increases. After a certain period of time in the portfolio, behavioral scoring can be constructed for a loan.
Behavioral scoring provides more sophisticated and flexible approaches to assessing the creditworthiness of borrowers, taking into account the influence of their behavior and delinquency status on credit decisions. In behavioral scoring, delinquency status can have a significant impact on the importance of other variables. For example, a very different set of variables may become important for a loan with no current delinquency compared to a loan that is already in delinquency.
Attention to detail: distortions in the scoring model
It is important to avoid possible offsets in the scoring model. Different circumstances can be sources of model skewing: from uneven distribution over time of "good" and "bad" borrowers to the use of incomplete information.
For example, when building statistical scoring, we have information only about those to whom loans were granted. In such cases, the samples of approved and all borrowers who applied for a loan differ significantly. This difference can be demonstrated by the questionnaire variable "Age" and data on approved loans and their subsequent behavior in different age groups:
The table shows that we have almost no information on "bad" borrowers for the 21-25 age group, as only 30% of the best applicants are included in the portfolio.
It is also important to exclude known cases of fraud from the training data. Fraudsters can fake details to fit the profile of a "good" borrower, and such fraud cases can distort the image of a "good" borrower and significantly reduce the predictive power of the model.
Therefore, combating fraud must be separate from questionnaire scoring.
Natural limitations of scoring capabilities
Regardless of the model used, scoring should not be expected to last indefinitely. Even scoring based on regression methods should be recalculated periodically, usually at least once every six months.
Neural networks, on the other hand, require daily monitoring because of their tendency to lose predictive power quickly with small changes in the input data.
Preliminary analysis of the data is very important, especially taking into account correlations between it. Such analysis will help to avoid possible problems in the scoring model.