Proposed Methods

The proposed method intends to classify multiple abnormalities and generate a clinically accurate medical report using the Chest X-Ray image. The proposed solution offers a rule-based labeler that can generate the labels corresponding to each image as there is no mechanism to detect the bounding boxes.

Using these labels along with images a CNN-based multi-label classification model can be trained that can output the labels corresponding to a given image. Once this model is trained, it can use the convolutional feature maps derived from this model as an input to the report generation model and subsequently train it to generate the report. The CNN-based multi-label classification model is the encoder model and the report generation model as the decoder model.
Figure 1 shows the overall flow for the solution mechanism.

Figure 1. Solution Architecture

2.1 Label Extraction

As mentioned previously in section 2, the rule-based systems fail when they encounter a different sentence construction and spelling errors. The proposed solution, on the other hand:

Creates a dictionary of all the radiological findings with the support of radiologists, which is then added to the standard dictionary. The reports in the database are passed through a spell correction tool which corrects all the spelling mistakes based on the above created English dictionary. This is the pre-processing step for the reports.

Second, AI-based labeler is built on top of CheXpert labeler and a multi-label extraction model (BERT [13]) is trained. The model used for this paper was initially trained on the labels extracted using the CheXpert labeler on the in-house dataset. Later, it was fine-tuned on the report which was manually annotated by the radiologists. While this work was under review similar approaches were proposed in [14][15]. The present paper however is based on an independent study on the in-house dataset and report its findings on it.

Figure 2 shows the overall label extraction scheme described above.

Figure 2. Report Labeler Method

2.2 Model Architecture

2.2.1 Overview

The proposed model architecture embraces a linear approach. It involves attention-based CNN for image feature extraction and multi-label classification and multiple visual attention-based LSTMs for generating the report. Figure 3 shows the complete model architecture for the multi-label classification and report generation scheme.

Figure 3. Convolution Attention-based sentence REconstruction Scoring (CARES)

2.2.2 Attention-based CNN Encoder

The team used a multi-task learning framework for multi-label classification, involving a common CNN-based base feature extraction model for all labels. This was followed by an Attention layer, Global Pooling layer, and a Fully Connected layer for generating predictions for each label. Each label has a separate FC layer, which enabled the model to generate predictions for multiple labels simultaneously without one impacting the other.

A separate attention layer was used for each tag. This was done in order to ensure that model can attend to different regions of the image simultaneously and generate feature maps relevant for the given label. This had a two-fold benefit.

First, it improved the classification accuracy as the output of each attention layer will be focused only on the part of the image relevant to that label. Secondly, it provided better convolutional feature maps (encoded image features) to the decoder.

Let’s consider an input chest x-ray image X. We pass this image through the base feature extractor base and obtain the base features B

B = base(X)

Using the base features B, we calculate the separate features for each class (F^c)

using the Attention Layer (Attn^c) F^c = Attn^c(B)

These features can be fed directly to the Visual Attention-based LSTM Decoder. For predicting the label using Fc, we perform Global Pool operation followed by Fully Connected Layer (FC^c) and sigmoid to generate the probability.

Y^c= σ(FC^c(GP(F^c)))

2.2.3 Visual Attention-based LSTM Decoder

Instead of going for a single 2-level hierarchical LSTM as suggested in [7] and also adopted for use in [8], the team used a separate LSTM model for each label. This ensured an ability to generate a separate sentence for each label and that each such sentence is directly linked to the corresponding label.

This further helped ensure that each LSTM is now linked with the radiological findings corresponding to that label (broad category mentioned in section 2.2), besides eliminating the need for semantic attention[6] since all the sentences are being generated separately by each LSTM. This allows for great performance boost in terms of accuracy.

The team also leveraged the Visual Attention mechanism proposed in [5] to further boost the report generation accuracy. Each LSTM model was trained only on the images which have a positive occurrence of the corresponding label in the ground truth dataset. By taking this approach, the team eliminated instances of a negative/uncertain mention of the label in the report.

For a given generated sequence at step t (y0, y1,...., yt−1), the output word yt can be calculated using the embeddings of the previous words Embedt and the attention weighted encoding of the encoder output AW Et.

y_t = Softmax(FC(LSTM(Embed_t,AW E_t)))

where embeddings can be calculated using a typical embedding layer and attention weighted encodings can be calculated using [5].

The < START > token is provided at the start of the sequence and < END > token in the end while training. Once the end token was generated, the sentence was completed and further processing was stopped.

2.3 Clinically Relevant Scoring Mechanism (RFQI)

The paper proposes a novel clinically relevant scoring mechanism RFQI. This metric considers the exact synonym which can be directly fetched from the dictionary of radiological findings created previously (section 3.1). It also factors in the relative region (e.g., left, right, upper, mid, lower). The weighted sum of these metrics are then taken to report the overall accuracy.

The overall formula for RFQI for a given class c can be written as:

RFQI^c =∑^Ni=1(w^csyn ∗T P^ci,syn +w^cloc ∗T P^ci,loc) ∑^Ni=1(w^csyn ∗T P^ci,syn +w^cloc ∗T P^ci,loc)

where N represents the number of positive training examples of a given class, w^csyn and w^cloc represent the weights for synonym and localization matching terms, and T P^ci,syn and T P^ci,loc represent the binary value (1 for matching and 0 for non-matching). The scale of this metric is kept as 0-1 meaning that the sum of these weights for a given class is 1 i.e.

w^csyn +w^cloc = 1

Note that in case there is no mention of the region in the ground truth/report, we consider it as a positive match for region for obvious reasons.