1

Automatic Classification of Pathology Reports

using TF-IDF Features

Shivam Kalra, Larry Li, H.R. Tizhoosh

Kimia Lab, University of Waterloo, Canada, http://[email protected]

Abstract—Pathology report is arguably one of the most impor-

tant documents in medicine containing interpretive information

about the visual findings from the patient’s biopsy sample. Each

pathology report has a retention period of up to 20 years after

the treatment of a patient. Cancer registries process and encode

(based on ICD-O code) from pathology reports. We performed

experiments by extracting the TF-IDF features from the reports

and classifying them using three different methods—SVM, XG-

Boost, and Logistic Regression. We constructed a new dataset

with 1,949 pathology reports arranged into 37 ICD-O categories,

collected from four different primary sites, namely lung, kidney,

thymus, and testis. The reports were manually transcribed into

text format after collecting them as PDF files from NCI Genomic

Data Commons public dataset. We subsequently pre-processed

the reports by removing irrelevant textual artefacts produced by

in demonstrating the potential of machine learning methods for

classification and encoding of pathology reports.

I. INTRODUCTION

Cancer is one of the leading causes of death in the world,

with over 80,000 deaths registered in Canada in 2017 (Cana-

dian Cancer Statistics 2017). A computer-aided system for

cancer diagnosis usually involves a pathologist rendering a de-

scriptive report after examining the tissue glass slides obtained

from the biopsy of a patient. A pathology report contains

specific analysis of cells and tissues, and other histopatholog-

variability in terminology due to lack of standards and may

even include misspellings and missing punctuation, clinical

diagnoses interspersed with complex explanations, different

terminology to label the same malignancy, and information

about multiple carcinoma appearances included in a single

report [1].

In Canada, each Provincial and Territorial Cancer Registry

(PTCR) is responsible for collecting the data about cancer

diseases and reporting them to Statistics Canada (StatCan).

Every year, Canadian Cancer Registry (CCR) uses the in-

formation sources of StatCan to compile an annual report

reports, medical billing summaries). The manual classification

With the continued growth in the number of cancer patients,

and the increase in treatment complexity, cancer registries

face a significant challenge in manually reviewing the large

quantity of reports [1], [2]. In this situation, Natural Language

Processing (NLP) systems can offer a unique opportunity

to automatically encode the unstructured reports into struc-

tured data. Since, the registries already have access to the

supervised machine learning approach of feature extraction

and classification is a compelling direction for making their

workflow more effective and streamlined. If successful, such

a solution would enable processing reports in much lesser

challenging problem.

For cancer registries, an important piece of information in a

the patient’s histological diagnosis, as described by the World

Health Organization’s (WHO) International Classification of

Diseases for Oncology [3]. Prediction of the primary diagnosis

from a pathology report provides a valuable starting point

system would be able to drastically increase the efficiency of

diagnostic processes for the majority of cases where in spite

of obvious primary diagnosis, still time and effort is required

from the pathologists to write a descriptive report.

The primary objective of our study is to analyze the efficacy

of existing machine learning approaches for the automated

2

classification of pathology reports into different diagnosis cat-

egories. We demonstrate that TF-IDF feature vectors combined

with linear SVM or XGBoost classifier can be an effective

method for classification of the reports, achieving up to 83%

accuracy. We also show that TF-IDF features are capable

of identifying important keywords within a pathology report.

extraction from unstructured pathology reports.

NLP approaches for information extraction within the

biomedical research areas range from rule-based systems [4],

to domain-specific systems using feature-based classifica-

tion [2], to the recent deep networks for end-to-end feature

extraction and classification [1]. NLP has had varied degree

of success with free-text pathology reports [5]. Various studies

have acknowledge the success of NLP in interpreting pathol-

ogy reports, especially for classification tasks or extracting a

single attribute from a report [5], [6].

The Cancer Text Information Extraction System

is proposed which uses machine learning models to predict the

degree of association of a given pathology or radiology with

the cancer. The performance ranges from an F-measure of 0.75

for lung cancer to 0.94 for colorectal cancer. However, ARC

uses domain-specific rules which hiders with the generaliza-

tion of the approach to variety of pathology reports.

This research work is inspired by themes emerging in

many of the above studies. Specifically, we are evaluating the

task of predicting the primary diagnosis from the pathology

report. Unlike previous approaches, the system does not rely

on custom rule-based knowledge, domain specific features,

patients. The distribution of reports across different primary

diagnoses and primary sites is reported in Table I. The dataset

was developed in three steps as follows.

Collecting pathology reports: The total of 11,112 pathol-

ogy reports were downloaded from NCI’s Genomic Data

Commons (GDC) dataset in PDF format [10]. Out of all PDF

files, 1,949 reports were selected across multiple patients from

four specific primary sites—thymus, testis, lung, and kidney.

TABLE I: Distribution of pathology reports across (a) Primary

diagnosis, used a the label for the study, and (b) Primary site

associated with a report.

We pre-processed the reports by setting their text content

to lowercase and filtering out any non-alphanumeric charac-

3

ters. We used NLTK library to remove stopping words, e.g.,

‘the’, ‘an’, ‘was’, ‘if’ and so on [11]. We then analyzed the

reports to find common bigrams, such as “lung parenchyma”,

“microscopic examination”, “lymph node” etc. We joined the

biagrams with a hyphen, converting them into a single word.

We further removed the words that occur less than 2% in each

Each model is evaluated using two standard NLP metrics—

micro and macro averaged F-scores, the harmonic mean of

related metrics precision and recall. For each diagnostic cate-

gory C

from a set of 37 different classes C, the number of

, the micro F-score is defined as

whereas macro F-score is given by

tion roughly proportional to their test set representation (same

as accuracy for classification problem with a single label per

data point), whereas macro-averaged metrics are averaged by

class without weighting by class prevalence [13].

In this study, we performed two different series of experi-

ments: i) evaluating the performance of TF-IDF features and

various machine learning classifiers on the task of predicting

primary diagnosis from the text content of a given report,

and ii) using TF-IDF and LDA techniques to highlight the

quantitatively on the test dataset using the evaluation metrics

discussed in the previous section. For the second experiment

series, a random report is selected and its top 50 keywords

are extracted using TF-IDF weights. These 50 keywords are

highlighted using different colors based on their associated

topic, which are extracted through LDA. A non-expert based

qualitative inspection is performed on the extracted keywords

and their corresponding topics.

IV. RESULTS AND DISCUSSION

A classification model is trained to predict the primary

diagnosis given the content of the cancer pathology report.

of 5% for macro F-score. It is interesting to note that SVM

with linear kernels performs much better than SVM with RBF

kernel, scoring 9% on the macro F-score and 12% more on

the micro F-score. It is suspected that since words used in

primary diagnosis itself occur in some reports, thus enabling

the linear models to outperform complex models.

4

TABLE II: Different classifiers used in the study

TABLE III: Final train and test performance of classification

models

1. Epithelial (0.377), 2. Presence (0.269), 3. Thymectomy (0.232)

4. Epithelial cells (0.210), 5. Cells (0.180), 6. Small (0.161), 7. Lobulated (0.161)

8. Lung parenchyma (0.151), 9. Appear (0.150), 10. Examination (0.139)

Topic # Keywords

Topic 1

Topic 2

IDF weights. The keywords are color encoded as per the

abstract “topics” extracted using LDA. Each topic is given

a separate color scheme.

We proposed a simple yet efficient TF-IDF method to ex-

tract and corroborate useful keywords from pathology cancer

reports. Encoding a pathology report for cancer and tumor

surveillance is a laborious task, and sometimes it is subjected

to human errors and variability in the interpretation. One of

the most important aspects of encoding a pathology report

involves extracting the primary diagnosis. This may be very