2.1. Loan Evaluation in P2P lending

P2P lending has emerged as a new e-commerce platform in the financial marketplace. As with many crowd-sourced services, P2P lending is bringing new economic efficiencies to financing [1]. Currently, scholars are undertaking three streams of research into this new business model: the reasons behind the development of P2P lending (which is outside the scope of this paper); the factors and methods that affect the likelihood of defaults or funding success; and the performance of a range of P2P platforms and credit risk prediction tools for evaluating loans [3].

Some of the studies that have examined the factors affecting funding success and the risk of defaults find that strong social networking relationships, personal characteristics and variables such as the amount of the loan, the interest rate, and the term of the loan are an important factors in determining a borrower’s credit risk and influence funding success [3].

However, as a new trend in finance, few studies have explored credit risk prediction in P2P lending, although this number is growing. Emekter, Tu, Jirasakuldech and Lu [3] applied logistic regression to investigate the probability of defaults and find some associations between loan defaults and credit scores, debt-to-income ratios, FICO scores, and revolving credit lines. Malekipirbazari and Aksakalli [2] used a range of machine learning methods to classify good and bad loans, such as random forest, logistic regression, k-nearest neighbor, and support vector machines. They find that using a machine learning approach is much more effective than relying on the existing financial metrics, like FICO and LC grades, which the Lending Club provides to help lenders to make loan investment decisions.

Several decision support systems based on credit scoring models have been developed to help banks decide whether or not to extend credit to loan applicants. There are two types of predictive models in credit scoring: statistical approaches and artificial intelligence methods. Statistical methods include linear discriminant analysis and logistic regression. The artificial intelligence methods include decision trees, random forest, support vector machine, neural networks, and naïve Bayes. Despite the development of more advanced AI methods for evaluating the credit risk of borrowers, simple statistical approaches such as linear discriminate analysis and logistic regression still remain popular because of their high accuracy and ease of implementation [6–9].

2.2. Class Imbalance Problem

The class imbalance problems are common in classification problems. Class imbalance occurs when the number of instances in one class is vastly different than the instances in another. In such cases, classifiers tend to be biased towards the majority class, while the minority class is ignored [10]. Many algorithms designed to address imbalanced data classification problems have emerged over the past decade, and resampling is one of the most important strategies for solving this issue [11]. Resampling generates a balance training dataset prior to building the classification model.

The three types of resampling techniques are over-sampling, under-sampling, and a hybrid of the two [11].

Over-sampling creates new samples in the minority class. Three prominent oversampling methods are random over-sampling, synthetic minority over-sampling technique (SMOTE), and adaptive synthetic sampling. Random over-sampling randomly duplicates the minority samples to balance the distribution of data [12]. SMOTE uses k-nearest neighbors to produce new instances based on the distance between the minority data and some randomly selected nearest neighbors [13]. Adaptive synthetic sampling uses the density distribution to generate synthetic samples for each minority instance [14].

Under-sampling discards samples from majority class [15]. Random under-sampling and instance hardness threshold are the state-of-the-art under-sampling techniques. Random under-sampling randomly eliminates examples from the majority class to balance the class distribution [12]. The instance hardness threshold method measures the probability that an instance will be misclassified and uses a constant threshold to filter instances in all iterations [16].

The third resampling approach is a hybrid method that combines over- and under-sampling. SMOTE + Tomek links (SMOTE-TOMEK) and SMOTE+ edited nearest neighbor (SMOTE-ENN) are the two hybrid techniques compared in this study. SMOTE-TOMEK corrects SMOTE data by finding pairs of minimally distanced nearest neighbors of opposite classes. It then identifies and removes Tomek links to produce a balanced dataset with well-defined classes. SMOTE-ENN follows the same procedure as SMOTE-Tomek but uses edited nearest neighbors to balance the dataset.

Class imbalance is a common problem in loan default prediction. Abeysinghe, Li and He [17], Brown and Mues [18], and Sanz, Bernardo, Herrera, Bustince and Hagras [19] have all provided solutions to class imbalance problems for credit risk prediction. However, to the best of our knowledge, only a few studies have addressed the class imbalance problem in detail in the social lending context.

3.1. Feature Engineering

The first component in the model is a feature engineering module. Its main purpose is to enhance data reliability by cleaning data and selecting the subset of data features with the most discriminatory power. In credit risk prediction, ignoring irrelevant features can increase classification accuracy [20] and decrease the computational costs associated with running several machine learning models [21]. Feature selection also reduces the dimensionality of the data, which helps mitigate the risk of overfitting. Our model comprises four important steps in feature engineering: data cleaning, leaky data removal, data transformation and correlation analysis, and deriving new attributes.

The data is cleaned by first removing missing and null values from dataset [22], then all outliers are removed according to the acceptable range defined in [23]. Eq. (1) shows the upper bound and lower bounds of this range.

Once the data has been cleaned, the features that may cause data leakage in the model are identified. These features typically do not have available values at the time the prediction model is used. So, training model with these features may make unrealistically good predictions. Surprisingly, the Lending Club dataset contains leakage data, but few studies relying on this dataset have taken that into consideration.

Data transformation including; converting categorical features to numeric, standardization and log transformation. Some classification algorithms, such as logistic regression, cannot handle categorical features, so they are transformed into new forms of data. For standardization, the min-max normalization in Eq. (2) ensures that all parameters use the same scale.

A log transformation reduces skewness in the data distribution (which cannot be applied to zero or negative values).

Correlations for numeric and binarized nominal attributes are then computed with respect to the loan status to provide a better understanding of the data and its attributes. Finally, according to Malekipirbazari and Aksakalli [2], we define variables that are simple ratio of other features. Moreover, this study introduces the new non-standard financial feature. These ratios help to capture certain borrower characteristics to make the most of the available data.

3.2. Imbalanced Learning Approaches

This study employs resampling approach to deal with the imbalance problem. Figure 2 demonstrates three categories of resampling approach including under-sampling, over-sampling, and hybrid methods. We address the state of the art algorithms in each of these categories. The under-sampling approach includes random under-sampling (RUS), and instance hardness threshold(IHT) algorithms. For over-sampling approach, random over-sampling (ROS), synthetic minority over-sampling technique (SMOTE), and adaptive synthetic sampling (ADASYN) are studied. Finally, SMOTE + Tomek links (SMOTE-TOMEK) and SMOTE+ edited nearest neighbor (SMOTE-ENN) are two prominent hybrid approaches that considered by this research. Section 2.2. presents complete literature on these algorithms.

Fig. 2.

Imbalanced learning approach

3.3. Classification Models

To the best of our knowledge, logistic regression, linear discriminate analysis, and random forest have demonstrated the best performance in the area of classification. Therefore, these three algorithms were selected for loans evaluation in this research.

3.4. Validation

The dataset was divided into a train set and a test set at a ratio of 70:30. Only the training set was balanced, through resampling, then validated with the still imbalanced test set.

4.1. Dataset Description

Advances in P2P lending markets have generated large amounts of data on real-world P2P lending transactions. This analysis is based on 2016 and 2017 data from the publicly available datasets released by the Lending Club, a well-known P2P lending platform (lendingclub.com).

The dataset contains approximately 636K borrower records and 145 features in total. Redundant information, noise, and leakage features were removed from the dataset. The features detected as leaks were: LC grade, interest rate, issue date, outstanding principal, total payment, total received principle, total received interest, total late fees received, recoveries, post charge off collection fee, last payment date, last payment amount, and fund amount.

The variables with skewed distributions were annual income, income to payment ratio, and revolving to income ratio. Log transformations were applied to create normal distributions for these variables.

Four categorical attributes were selected as nominal attributes and transformed into binarized data: term (2 categories), home ownership (3 categories), verification status (3 categories), and purpose (12 categories). Ultimately, 2+3+3+12 = 20 numerical attributes replaced these four categorical attributes.

The new attributes, defined based on monthly income, installment amount, and revolving amount features were Income-to payment ratio and revolving-to-payment ratio [2]. Further, we defined a new ratio that, to the best of our knowledge, has not been previously used in any study called new debt to income (New DTI). New DTI considers the impact of the loan repayments, should it be granted, on the borrower’s solvency. It is a ratio of the repayment amount to the borrower’s monthly income if the loan is approved and is defined as:

The features used in the credit risk prediction process are presented in Table 1, grouped into three categories: loan characteristics, creditworthiness, and solvency [4].

Table 1.

Lending Club attributes

For a better understanding of the variables, we also performed a correlation analysis of all the variables with respect to loan status. Table 2 shows the correlations for the top-20 attributes.

Table 2.

Correlation with respect to loan status

LC grade and interest rate sit at the top of the list with the highest correlations to loan status. The Lending Club assigns an LC grade of between A and G, where A represents safer loans and G represents riskier loans. These grades were transformed into numbers between 1 and 7 (1=A and 7=G). They also assign an interest rate to each loan. G-grade loans are given the highest interest rate; A-grade loans have the lowest rate. However, LC grades and interest rates are leaky data, so despite their high correlation with loan status, they should not be included in the modeling process or overly optimistic predictions may result.

The new DTI ratio sits in third place with a correlation of 0.171, which shows that it can have a positive effect on the classification results.

A brief description of the final dataset appears in Table 3.

Table 3.

Final dataset description

“N” represents the number of instances in the dataset, “features” is the number of variables we used in the prediction analysis. “% default”, “% fully paid” and “imbalance ratio” are reflective of “loan status” in the Lending Club dataset. Loans with a status of “current” have not yet been issued or have not reached maturity and, therefore, do not contain information a borrower’s creditworthiness. These records were removed. “% default” reflects the proportion once the current loans had been removed with a status of “defaulted”; likewise, “% fully paid” reflects the proportion of loans that have reached maturity.

4.2. Performance Measurement

Accuracy is traditionally the most popular performance metric in a binary classification problem. However, when assessing imbalanced datasets, accuracy tends to emphasize the majority class, making it difficult for the classifier to perform well on the minority class. Moreover, in credit risk prediction, measuring accuracy does not consider that false positives are worse than false negatives and, therefore, accuracy can be a misleading criterion that causes erroneous results [28]. As such, other performance measures are more appropriate when working in domains with class imbalance issues [29]. Receiver operating characteristics (ROC), G-mean, and F-measure (FM) are preferred as the likelihood these measures will be affected by imbalanced class distributions is low [11].

In a binary classification problem with good and bad class labels, the classifier’s result is considered successful if both the false positive rate and false negative rate are small [15]. Sensitivity measures the accuracy of positive samples and specificity measures negative samples. Additionally, an effective performance measure in imbalanced settings will indicate the balance between classification performance in both the minority and majority class [30]. The G-mean measure considers both sensitivity and specificity for both classes in calculating its scores and is therefore an effective measure for imbalanced datasets [15]. The G-mean equation is shown in Eq. (5).

The area under curve (AUC) measure determines the area under receiver operating characteristic (ROC) curve. This measure is another effective metric for measuring classification performance with imbalanced datasets [15].

4.3. Classification Results

In this empirical evaluation, we evaluated the performance of selected classifiers in combination with various resampling methods. The combinations are shown in Table 4.

Table 4.

Classifier / resampling method combinations

The credit risk prediction results of the three selected classifiers – random forest, logistic regression, and linear discriminate analysis – were tested with each type of resampling method in groups. The best pair from each group was then compared with each other and with a non-sampling approach, i.e., with an imbalanced dataset that had not been resampled.

As shown in Table 5, the effectiveness of the different under-sampling methods depends on the measure used to gauge performance. As discussed in Section 4.2, G-means is the most effective measure for assessing the classification results when class imbalance problems exist in the dataset. From a G-means perspective, all classifiers in combination with RUS significantly outperformed the combinations with the instance hardness threshold method. The RF-RUS method had the highest G-mean (0.65). Therefore, RFRUS is the best classification approach in the under-sampling group.

Table 5.

Classification Results (Under-sampling approach)

The performance of the over-sampling methods with different classifiers appears in Table 6. In terms of accuracy, adaptive synthetic sampling behaved very differently when combined with a random forest than the other two classifiers. RF-ADASYN achieved an overall accuracy of 0.8 compared to 0.64 for LRADASYN and 0.61 for LDA-ADASYN. In addition, RF-ADASYN achieved the highest sensitivity (0.94); however, it fell to the bottom of the list for specificity. This demonstrates RF-ADASYN’s poor performance in accurately predicting defaulters, and its bias toward the majority class despite an over-sampling approach designed to solve class imbalance issues. Random forest showed the lowest specificity and G-mean, but had the highest false positive rate of all the over-sampling techniques. These results indicate that random forest is an inappropriate classifier to hybridize with an over-sampling approach. When logistic regression and linear discriminate analysis were hybridized with over-sampling techniques, their AUC sat at around 0.7, whereas random forest fell to the bottom of the list in terms of AUC at between 0.65 and 0.68. Among all the over-sampling methods, LDA-SMOTE emerged as the best classification method according to the G-mean measure.

Table 6.

Classification Result (Over-sampling approach)

Given the performance measures in Table.7, the hybrid re-sampling methods did not produce good results. All methods had low accuracy. The models using SMOTE-ENN resulted in lower false positive rates and higher specificity, but their sensitivity scores were incredibly low. This is an indication that the model identifies most customers as defaulters based on significantly high performance in the minority class (the defaulting customers) while showing low performance on the other classes. For example, RF-SMOTE-ENN only correctly predicted 33.7% of the customers as good with a class of 1. Despite the poor performance of models presented in Table 7, LR-SMOTE-Tomek, with a G-means of 0.64, represents the best classification of the hybrid re-sampling methods.

Table 7.

Classification Result (Hybrid approach)

Next, we compared the best combinations from each group and list the results in Table 8. Table 8 also includes the results of the two non-sampling strategies, logistic regression and random forest, to illustrate the significant difference between prediction results on an imbalanced dataset versus a balanced one.

Table 8.

Classification Results (Final Comparison)

The non-sampling strategies were the most accurate; however, they performed worst in terms of G-mean, specificity, and false positive rates. This indicates that these classification models, by considering all the samples in the majority class (i.e., the good customers), are biased towards the majority class, but tend to ignore the minority.

In terms of G-mean, random forest had the lowest rating (0.12) on an imbalanced dataset. While, RF-RUS (i.e., random forest on a balanced dataset) had the highest G of 0.65. However, random forest and logistic regression returned the highest false positive rates on their own at 0.98 and 0.95, respectively, and 0.48 and 0.35, respectively, when combined with a resampling approach. This notable difference between non-sampling and resampling proves the effectiveness of resampling techniques on the performance of prediction modeling.

RF-RUS emerged as the best method for predicting a borrower’s status in a social lending marketplace. Our computational results indicate that considering a random under-sampling technique for class imbalance issues, then use a random forest classifier on the resulting balanced training set outperforms the other methods examined in the experiments.