ABBREVIATIONS

IHDC—Intracranial Hemorrhage Detection Challenge

RSNA—Radiological Society of North America

ICH—Intracranial hemorrhage

CT—Computed tomography

ML—Machine learning

SVM—Support vector machine

DL—Deep learning

CNNs—Convolutional neural networks

3D—Three-dimensional

LSTM—Long short-term memory

CAM—Class activation mapping

HU—Hounsfield units

MBConv—Mobile inverted bottleneck convolution

SE—Squeeze and excitation

GAP—Global average pooling

FC—Fully connected

W—Window width

L—Window level

ROC—Receiver operating characteristic

AUC—The area under the receiver operating characteristic

1. INTRODUCTION

Intracranial hemorrhage (ICH), or bleeding within the skull, is a life-threatening emergency with high rates of mortality and morbidity [1]. Without rapid and intensive medical treatment, it may cause increases in intracranial pressure, resulting in permanent brain tissue damage or brain herniation [2]. ICH is classified as either intra-axial hemorrhage (bleeding within the brain itself) or extra-axial hemorrhage (bleeding within the skull but outside the brain tissue). More specially, intra-axial hemorrhage has two subtypes: intraparenchymal (within the brain tissue) and intraventricular (within brain's ventricles). Extra-axial hemorrhage has three subtypes: epidural hemorrhage caused by trauma, subdural hemorrhage resulting from tearing of the bridging veins in the subdural space, and subarachnoid hemorrhage resulting from trauma or ruptures of aneurysms or arteriovenous malformations. Effective clinical intervention requires an urgent and accurate diagnosis for this life-threatening condition.

Computed tomography (CT) imaging of the brain, usually performed in the emergency department, is the initial diagnostic test for acute ICH. In CT images, the anatomic location and the pattern of bleeding indicate the potential cause of the ICH. Accurate interpretation is vital as a misdiagnosis (incorrect classification or missed ICH) may have clinical consequences [3–6]. In addition, analyzing CT images for certain types and locations of ICH by radiologists can be complicated and time consuming. Delays in diagnosis directly impacts time from onset to treatment for patients with ICH which may affect patient outcome [7,8]. An automatic diagnostic tool to assist just-in-time and accurate detection of ICH along with classification of its subtypes is essential to accelerate the decision-making process in clinical interventions to improve outcomes.

Machine learning (ML) algorithms have been exploited to detect traumatic brain injury using the support vector machine (SVM) model [9] and to predict cerebral ischemia in subarachnoid hemorrhage using various ML algorithms [10]. However, ML approaches require extensive image preprocessing, manual feature extraction, and selection steps; all of which are complicated and time-consuming. Deep learning (DL) methods, unlike ML algorithms, do not require explicit use of radiologic features, but explore the imaging information directly from the medical images by extracting thousands of image features through the use deep neural networks. Previous studies have reported using DL algorithms for the diagnosis of ICH. DL models have been applied on single CT image slices to improve the detection rate of suspected brain hemorrhage [11]. This study compared three CNNs with and without pretraining and found that pretraining on natural images improves the performance in medical image diagnosis.

Gray scale images such as those produced by CT scanners present a limited information problem to DL model. To solve this problem, multiple window settings (window width and level) have been used to improve the detection of subtle differences in brain tissues [12,13]. Another challenge in the computer assisted diagnosis of ICH is the fact that diagnostic information is not contained in just a single image, but from adjacent images as well. Previous work utilized three-dimensional (3D) convolutional neural networks (CNNs) to extract more anatomic information from the full brain scan volumes [14–16]. Grewal et al. developed a Recurrent Attention DenseNet (RADnet) model that combined the original DenseNet architecture with the attention components of neighboring slices to make ICH predictions on the per-scan-level [14]. Ker et al. generated image intensity histogram and removed overlapping image pixels that contributed to both normal and ICH cases before performing the classification [15]. However, 3D CNNs are computationally expensive, and often fail due to limited computer memory. Considering that ICH regions usually exist in 2–3 slices of the CT scan, it is not computational efficient to use 3D CNNs for the full scan volume. More advanced models that combined CNNs and a long short-term memory (LSTM) network have been proposed for detection and subtype classification of ICH in 3D CT by learning neighboring anatomic information, where CNNs extract discriminative features from each slice and the LSTM links these features across continuous slices [17,18]. Nguyen et al. proposed an end-to-end sequential model that attached LSTM architecture with CNNs to extract the spatial dependencies between slices [18]. These two studies [17,18] achieved good model performance using their local datasets, but showed decreased performance with external datasets, and the models were not able to generate visualization maps of the location and subtypes of ICH.

In this study, we developed a novel ensembled DL model to detect the occurrence of ICHs and simultaneously differentiate their subtypes. The ensembled DL model consisted of two parallel pathways, each using different sets of preprocessed images as the input to the CNN. One preprocessing method adjusted the window level/width to enhance the image contrast of brain tissue, blood, and soft tissue, respectively. The other preprocessing method utilized extended spatial anatomic information from adjacent image slices. The final model output was generated by ensembling the probability outcomes from the two network pathways. In addition, the class activation maps (CAMs) [19] were generated to highlight the regions of detected ICHs and the associated subtypes. Finally, we tested the model generalization by using an independent dataset acquired at a different institution. The hypothesis is that the ensembled DL model will provide a useful computer-aided diagnostic tool to assist in the rapid detection of ICHs with simultaneous subtype classification on CT images.

2. MATERIALS AND METHODS

2.1. DATASET

The Radiological Society of North America (RSNA) launched an Intracranial Hemorrhage Detection Challenge (IHDC) in 2019, which contained a total of 674,258 head CT reconstructed tomographic images acquired from 19,530 patients [20]. All images were labeled by 60 radiologists from different institutions, with a majority vote determining the gold standard for the label of each slice. If one or more hemorrhage spots were identified on a given slice by the radiologist, this slice was labeled as “any” and the associated subtype of each hemorrhage spot was provided as well (Figure 1). A given slice free of hemorrhage was labeled as “normal.” Datasets for training, validation, and testing were divided on a patient basis. CT exams acquired from 90% of all the patients were used for training and the other 10% for internal testing purposes (Table 1). The training subset was further divided with 20% of the subset used for validation of the training. This approach ensured reconstructed images from individual patients were not used in different cohorts. The whole dataset was imbalanced with 86% slices labeled as “normal” and 14% labeled as “any.” Subtype distributions were epidural 0.4%, intraparenchymal 4.8%, intraventricular 3.5%, subdural 6.3%, and subarachnoid 4.8%.

Figure 1

Examples of five subtypes of intracranial hemorrhage: (a) intraparenchymal (b) epidural (c) subarachnoid (d) subdural, and (e) intraventricular. The red arrow points to the location of each hemorrhage.

Data Source	RSNA (674258 slices)			CQ500 (488 scans)
Cohorts	Training	Validation	Internal Testing	External Testing
Label: Normal	415534	103803	57817	286
Label: Epidural	1959	521	281	13
Label: Intraparenchymal	23424	5839	3301	134
Label: Intraventricular	17046	4300	2420	28
Label: Subarachnoid	23227	5793	3102	60
Label: Subdural	30651	7722	4123	53
Label: Any	69997	17498	9609	202

Table 1

Labelled class distributions of the RSNA IHDC dataset with the number of cases used in training, validation, and internal testing process. Also listed the number of cases of each class in the external testing dataset (CQ500). RSNA IHDC dataset labeling was slice-based. CQ500 labeling was scan-based.

In addition, a publicly available independent dataset CQ500 [21], acquired from a different institution in New Delhi, was used for external testing to evaluate the model generalization. This dataset included all five subtypes of ICH (Table 1), but the labels were based on the patient level and not slice level (i.e., each CT scan volume had one label).

2.2. Data Preprocessing and Augmentation

2.2.1. Multi-window preprocessing

Image windowing and level adjustment is a routine step taken by the radiologist to change the image brightness and apparent contrast to enhance the appearance of different types of tissues. When reading brain CT images, the radiologists usually choose to use a “brain” window that emphasizes detailed brain structures and subtle brain lesions [22], a “soft-tissue” window that improves the detection of bleeding locations and focal lesions, and a “subdural” window that highlights small collections of subdural blood, which otherwise can be obscured by the skull [22]. To mimic the way the radiologists read CT studies using different window settings for the pixel values in Hounsfield units (HU), we applied three different window level (L) and width (W) settings to enhance the display of brain tissues (L: 40, W: 80), subdural space (L: 80, W: 200), and soft tissues (L: 40, W: 380). All the windowed images were then normalized to [0, 1]. Each tomographic image thus generated 3 images with different window settings, which were fed to three input channels of the DL neural network (Figure 2[a]).

Figure 2

Two preprocessing methods: (a) multi-window preprocessing and (b) adjacent-slice preprocessing on a given CT slice. In (a), three window settings (brain, subdural, and soft tissue) were used to adjust image contrast. In (b), S is the center slice, and S−1 and S+1 are two adjacent slices. All slices in (b) were adjusted using the brain window setting. In (a) and (b), preprocessed images were concatenated and fed to three input channels (R: red, G: green, B: blue) of the neural network.

2.2.2. Adjacent-slice preprocessing

Radiologists interpret studies through the review and observation of the entire scan volume, comprised of many reconstructed images. Diagnostic information related to a finding may appear on multiple contiguous images in the reconstructed volume. To mimic the radiologist's process, 3 image slices including a center slice with two adjacent slices were fed to the three input channels of the DL neural network (Figure 2[b]). All these image slices used the “brain” window (L: 40, W: 80) and were normalized to [0, 1].

2.2.3. Data augmentation

Several image transformation methods were applied to augment the data, including rotation, horizontal and vertical flipping, scaling, and brightness adjustment. Specifically, images were randomly rotated by degrees from −20° to 20°, scaled by a factor from 0.8 to 1.2, and the brightness was adjusted by −20%–20%. More data augmentations were implemented to the subtypes with smaller datasets to alleviate the data imbalance problem. For example, the total number of image slices of the epidural subtype increased by 10 times after augmentations.

2.3. DL Neural Network Model Architecture

2.3.1. Effeicient-B0 network

Our model was based on the recently proposed EfficientNet that achieved better accuracy and efficiency (6.1 times faster) with a smaller number of model parameters (8.4 times less) than other networks [23]. It has been applied to solve medical problems, such as the diagnosis of COVID-19 [24,25] and the classification of skin diseases [26]. EfficientNet utilizes a new scaling method, so-called compound coefficient, to systematically balance the depth (d: the length of the model), width (w: the number of channels) and resolution (r: the image size) of a given network. All three factors were decided by the compound coefficient ∅=1, such that d=α∅,w=β∅,r=γ∅, where α,β,γ were the scaling coefficient. The optimal values of α=1.2,β=1.1,γ=1.15 were obtained via a grid search using a small subset of training data by varying d, w, and r to by finding the minimal loss of the model under the constraint of α≥1,β≥1,γ≥1,α⋅β2⋅γ2≈2. As shown in Figure 3, EfficientNet-B0 ∅=1 in our study was composed of a single-layer convolutional layer (Conv) followed by a series of mobile inverted bottleneck convolution (MBConv) blocks [27]. The single-layer Conv was used to capture the basic features in the input image. The MBConv is a new type of residual block that concatenates the features from the previous layer to the current layer, which is especially useful for deep neural networks. The residual block applied inverted residuals to increase the number of kernels in order to expand the variety of features and increase the representational power. In addition, linear bottlenecks in the residual block decreases the computational burden of the deep model [27]. In total, 16 MBConv blocks with different kernel sizes (3 × 3 or 5 × 5) were connected to extract features with various receptive fields. MBConv1 and MBConv6 refer to the MBConv block with an expansion factor of 1 and 6 respectively. The expansion factor was used to convert low-dimensional representations of features to high-dimensional features with more channels, the reason for doing this was to increase the variety of features. A squeeze and excitation block [28] was used to further optimize the EfficientNet model by adaptively adjusting the weight of each feature kernel to emphasize more discriminative features. A global average pooling (GAP) layer was added at the end to generate a feature map, followed by a fully connected (FC) layer with 6 classes outputs.

Figure 3

Schematic diagram of the ensembled neural network model (a) and architecture of the EfficientNet-B0 CNN model (b). In (a), two parallel pathways containing different preprocessing methods and its own EfficientNet-B0 CNN model were trained independently using the same input images. The outputs of each pathway (i.e., the probability of each predicted class) were averaged to generate the final results. A probability threshold of 0.5 was used to decide if the label of a specific class is true (probability > 0.5) or false (probability < 0.5). In (b), the EfficientNet-B0 model was composed of a single-layer convolutional layer (Conv) followed by a series of mobile inverted bottleneck convolution (MBConv) blocks with different kernel size, global average pooling (GAP), and fully connected (FC) layer.

In the proposed ensembled model, the EfficientNet-B0 network was used in two parallel pathways (multi-window preprocessing pathway and adjacent-slice preprocessing pathway). Each pathway predicted a probability number for 6 classes (i.e. hemorrhage detection (“any”) and 5 subtype classifications) on each slice. Lastly, the probability numbers generated by the two pathways were averaged to predict the final outcome. A specific class was deemed as true if the averaged probability > 0.5, whereas as false if the averaged probability < 0.5.

2.3.2. Loss function

A normalized weighted binary cross-entropy loss function was used in the training process to further compensate for the data imbalance problem. The overall loss was calculated by averaging six binary cross-entropy of each predicted class, defined as

l=1N⋅c⋅∑k=1cθk  ∑n=1N∑i=1cyi,nlogy^i,n+1−yi,nlog1−y^i,n⋅θi(1)

where c=6 is the number of classes, N=8 is the batch size, y and y^ denotes the ground truth class and the predicted probability of class i, respectively. θk is the weight assigned to each class, which was inversely proportional to the sample size of the class after augmentations. A class k with a smaller training dataset was assigned with a larger θk, and vice versa. In this study, θk was set to 1.0 for “any” class and 0.23, 0.18, 0.26, 0.14, and 0.19 for the subtypes of epidural, intraparenchymal, intraventricular, subdural, and subarachnoid, respectively.

3. RESULTS

The ensembled model performance on the internal testing dataset was comparable to that of the training dataset (Table 2). In the internal testing dataset, the overall accuracy, sensitivity, and F1 score in detecting any ICH was 95.7%, 85.9%, and 86.7%. respectively. The performance for classifying five subtypes of ICH achieved an accuracy of [95.1%–98.6%], sensitivity of [79.2%–84%], and F1 score of [82.3%–85.9%], in which epidural hemorrhage classification showed the highest accuracy (98.6%) but the lowest sensitivity (79.2%) and F1 score (82.3%) likely due to the very small number of true positive cases in this subtype. In the external testing dataset (CQ500), the model achieved an overall high accuracy of 92.4%, sensitivity of 92.6%, and F1 score of 93.4% in detection of any ICH. However, the sensitivity and F1 score in classifying subtypes were relatively low, especially for epidural hemorrhage again likely due to the small sample size, i.e., 13 scans out of total 488 scans. The AUC scores were compared between the two separate network pathways and the ensembled network on internal testing dataset (Figure 4). The ensembled model achieved an AUC score of 94.3% for any ICH detection as well as 91.8%, 90.6%, 92.0%, 91.9%, and 94.2% for the subtypes of intraparenchymal, epidural, subarachnoid, subdural, and intraventricular hemorrhage, respectively. The ensembled network outperformed each separate pathway network in terms of ICH detection and subtype classifications (all p < 0.05).

Figure 4

Examples of color-coded heatmap visualizations that revealed the locations of the detected hemorrhage spots overlaid on the reference image in both multi-window and adjacent-slice pathways. Prob (any) represents the probability of that slice predicted as “any,” indicating existing hemorrhage. High probability numbers in both pathways demonstrated similar hemorrhage locations (a–c), whereas different probability numbers predicted by the two pathways demonstrated distinct hemorrhage locations and patterns (d, e).

CAMs were generated from each pathway network to visualize the discriminative regions in model predictions of ICH location and subtype classification (Figures 5 and 6). The heatmaps demonstrated distinguishable ICH locations and associated confidence level (i.e., probability number) which may be used to guide radiologists to identify the hemorrhages. When the probability numbers given by the two network pathways were both high, the heatmaps generated from both pathways demonstrated similar locations (Figure 5[a–c]). For cases in which probability numbers were low the hemorrhage locations detected by the two pathways deviated from each other (Figure 5[d, e]). CAMs also demonstrated the locations of hemorrhage classified as specific subtypes (Figure 6[b, c]). In Figure 6, the probability numbers for the intraventricular subtype and subarachnoid subtype were both high (>0.98), and four hotspots were identified on the heatmaps with the bottom two predicted as intraventricular (Figure 6[b]) and the upper two predicted as subarachnoid hemorrhage (Figure 6[c]). The overall pattern of these two subtypes were consistent with that shown on the heatmap of any ICH detection (Figure 6[a]).

Figure 5

A reprehensive example of multiple detected hemorrhage spots identified as 2 different subtypes on one CT slice. Both subtypes (intraventricular and subarachnoid) showed high probability numbers in the two preprocessing pathways. Interestingly, the color-coded heatmaps revealed different locations of these two hemorrhage subtypes.

Figure 6

Comparisons of AUC scores for model predictions of hemorrhage detection (“any”) and 5 subtype classification using multi-window, adjacent-slice, and ensembled models on RSNA IHDC testing dataset. P < 0.05 indicates statistically significant.

4. DISCUSSION

In this study, we developed an ensembled DL model that provided simultaneous rapid detection assistance of IC and accurate classification of their subtypes based on noncontrast CT images. The novelty of this model included (1) ensembled two parallel network pathways with different input image preprocessing methods providing more image contrast and volumetric features; (2) the use of the EfficientNet-B0 network improving computational efficiency; (3) heatmaps with discriminative image feature patterns providing guidance for radiologists to quickly identify the locations and subtypes of the detected hemorrhages. More importantly, an external testing dataset that was independent from the training dataset was used to also evaluate the model performance and generalizability.

Previous work has demonstrated that the DL model ensemble approach achieves better performance by combining the strengths of multiple sub-models [32]. Chandra et al. used majority voting based classifier ensemble of five supervised classification models to detect COVID-19 on chest X-rays and found significantly improvements using the ensembled model [33]. Lee et al. combined four deep CNNs into an ensembled model to detect ICH and achieved high performance on a small-sized dataset [34]. Our model was inspired by the ensembled CNN concept. The multi-window pathway represented various image contrast settings manipulated by radiologists to highlight subtle tissue differences, and the adjacent-slice pathway leveraged extended structural information of the neighboring slices adjacent to the target slice. Because image features extracted from these preprocessed input images were different, the predicted outcomes from each network pathway may vary. Eventually, the ensembled model took the average of the probability numbers of two pathways to reduce the bias or uncertainty caused by either one of them. Also, two separate probability numbers provided radiologists more evidences to evaluate the model predictions. In addition, feature heatmaps generated by CAM provided visual guidance to help radiologists identify possible locations of ICH and interpret the model predictions.

Compared with other studies using the RSNA IHDC dataset to solve similar classification tasks, our model achieved a relatively better performance. Danilov et al. [35] trained a ResNexT model with multiple window settings and had the highest AUC score of 80.4% in predicting the intraventricular hemorrhage. Burduja et al. [17] took multiple feature embeddings extracted from the CNNs into another LSTM model and achieved an AUC score of 97% on the overall ICH detection, however, the AUC scores for 4 other subtype classifications (91%, 71%, 89%, and 94% for intraparenchymal, subarachnoid, subdural, and intraventricular hemorrhage) were lower than our corresponding AUC scores of 91.8%, 92.0%, 91.9%, and 94.2%.

A major limitation of the RSNA IHDC dataset used in this study was the extremely imbalanced distribution of ICH of subtypes. To alleviate this issue, data augmentation and a weighted binary cross-entropy loss method were applied to compensate for the subtypes with small datasets. High accuracy but low sensitivity and F1 score in our model performance was due to the small number of true positive cases in certain subtypes. Another limitation of the dataset was that the location of each hemorrhage spot and its subtype was not annotated on the image and thus the model generated visualization heatmaps lacked gold standard to compare with.

There are several other limitations to our study. First, the model performance was not compared with radiologists' diagnostic performance, however a recent study reported that the DL model outperformed radiologists [17]. Second, the three window settings used in image preprocessing were manually set. Recently, Karki et al. [12] proposed a trainable CNN to determine the optimal the window settings automatically, which may help further improve the classification model. Lastly, although the ensembled model combining two network pathways provided better classification outcomes compared with each individual pathway, more pathways are believed to contribute to a stronger ensembled model [36]. Future work includes increasing the number of network pathways, using more window setting combinations, and extracting related spatial information from more adjacent image slices. Local dataset from our own institution with more comprehensive location and subtype annotations will be acquired to further fine-tune and improve the model developed in this study.

5. CONCLUSION

In conclusion, a novel ensembled DL neural network was developed for automatic IC detection, localization, and classification into one of 5 subtypes. By using two network pathways in parallel, the emsembling results were significantly improved in both ICH detection and subtype classification compared with training each of pathway separately. Two visual mappings generated from the different pathways gave radiologists guidance to localize the ICH. Finally, the performance evaluated on an external dataset showed the model's good generalizability. Successful implementation of this tool may facilitate fast diagnosis and decision-making to guide clinical interventions in emergent settings.

CONFLICTS OF INTEREST

The authors declare they have no conflicts of interest.

AUTHORS' CONTRIBUTIONS

Y. W., M. S., and J. D. conceptualized the project, Y. W. performed research and build models, Y. W., and J. D. analyzed data, M. S., and J. D. contributed evaluations. Y. W., M. S., and J. D. wrote the paper.

ACKNOWLEDGMENTS

This research is partially supported by the Swim Across America Grant from Rush Translational Sciences Consortium. The authors would also like to thank Dr. Sharon E. Byrd for her administrative support in establishing the medical imaging artificial intelligence research program.