![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"source": [
"# Medical Abstract Classification using Natural Language Processing"
],
"metadata": {
"id": "Uq_ssycxR8Ar"
}
},
{
"cell_type": "markdown",
"source": [
"### The objective is to build a deep learning model which makes medical research paper abstract easier to read.\n",
" - Dataset used in this project is the `PubMed 200k RCT Dataset for Sequential Sentence Classification in Medical Abstract`: https://arxiv.org/abs/1710.06071\n",
" - The initial deep learning research paper was built with the PubMed 200k RCT.\n",
" - Dataset has about `200,000 labelled Randomized Control Trial abstracts`.\n",
" - The goal of the project was build NLP models with the dataset to classify sentences in sequential order."
],
"metadata": {
"id": "bOGykK3Ad-wp"
}
},
{
"cell_type": "markdown",
"source": [
" - As the RCT research papers with unstructured abstracts slows down researchers navigating the literature. \n",
" - The unstructured abstracts are sometimes hard to read and understand especially when it can disrupt time management and deadlines.\n",
" - This NLP model can classify the abstract sentences into its respective roles:\n",
" - Such as `Objective`, `Methods`, `Results` and `Conclusions`.\n",
" "
],
"metadata": {
"id": "RgYVNMflgo86"
}
},
{
"cell_type": "markdown",
"source": [
"#### The PubMed 200k RCT Dataset - https://github.com/Franck-Dernoncourt/pubmed-rct"
],
"metadata": {
"id": "1Lmr_h4FipFw"
}
},
{
"cell_type": "markdown",
"source": [
"#### Similar projects using the dataset:\n",
" - Claim Extraction for Scientific Publications 2018: https://github.com/titipata/detecting-scientific-claim"
],
"metadata": {
"id": "N2d_bTBIi6of"
}
},
{
"cell_type": "markdown",
"source": [
"**Abstract** \n",
"\n",
"PubMed 200k RCT is new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field."
],
"metadata": {
"id": "WXb2Qlu8k2Fk"
}
},
{
"cell_type": "markdown",
"source": [
"**Data Dictionary**\n",
"\n",
"- `PubMed 20k` is a subset of `PubMed 200k`. I.e., any abstract present in `PubMed 20k` is also present in `PubMed 200k`.\n",
"- `PubMed_200k_RCT` is the same as `PubMed_200k_RCT_numbers_replaced_with_at_sign`, except that in the latter all numbers had been replaced by `@`. (same for `PubMed_20k_RCT` vs. `PubMed_20k_RCT_numbers_replaced_with_at_sign``).\n",
"- Since Github file size limit is 100 MiB, we had to compress `PubMed_200k_RCT\\train.7z` and `PubMed_200k_RCT_numbers_replaced_with_at_sign\\train.zip`. \n",
"- To uncompress `train.7z`, you may use `7-Zip` on Windows, `Keka` on Mac OS X, or `p7zip` on Linux."
],
"metadata": {
"id": "MvAjGdF0lACB"
}
},
{
"cell_type": "markdown",
"source": [
"# Importing data and EDA"
],
"metadata": {
"id": "T1JweqbejVX4"
}
},
{
"cell_type": "code",
"source": [
"!git clone https://github.com/Franck-Dernoncourt/pubmed-rct.git\n",
"!ls pubmed-rct"
],
"metadata": {
"id": "zM12MSlAnKCo",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "bb4adf23-506b-4a92-8e86-010efcf4f61f"
},
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Cloning into 'pubmed-rct'...\n",
"remote: Enumerating objects: 33, done.\u001b[K\n",
"remote: Counting objects: 100% (3/3), done.\u001b[K\n",
"remote: Compressing objects: 100% (3/3), done.\u001b[K\n",
"remote: Total 33 (delta 0), reused 0 (delta 0), pack-reused 30\u001b[K\n",
"Unpacking objects: 100% (33/33), done.\n",
"Checking out files: 100% (13/13), done.\n",
"PubMed_200k_RCT\n",
"PubMed_200k_RCT_numbers_replaced_with_at_sign\n",
"PubMed_20k_RCT\n",
"PubMed_20k_RCT_numbers_replaced_with_at_sign\n",
"README.md\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"### Initial Data exploration and modelling with PubMed_20k dataset"
],
"metadata": {
"id": "FFpPwS_SqS48"
}
},
{
"cell_type": "code",
"source": [
"!ls pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "I3mLk-7tqk3K",
"outputId": "9ceb3826-9977-4664-bf2d-de3927555bdf"
},
"execution_count": 2,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"dev.txt test.txt train.txt\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# imports\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import os\n",
"import random\n",
"import tensorflow as tf"
],
"metadata": {
"id": "TZiWPqoaqzBL"
},
"execution_count": 3,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# functions pre-written for workflow\n",
"!wget https://raw.githubusercontent.com/hecshzye/nlp-medical-abstract-pubmed-rct/main/helper_functions.py"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wRZJgoCeFBsX",
"outputId": "0619c26c-096f-410b-f9c6-aae7ec239b53"
},
"execution_count": 4,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"--2022-01-17 02:17:42-- https://raw.githubusercontent.com/hecshzye/nlp-medical-abstract-pubmed-rct/main/helper_functions.py\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 6442 (6.3K) [text/plain]\n",
"Saving to: ‘helper_functions.py’\n",
"\n",
"helper_functions.py 100%[===================>] 6.29K --.-KB/s in 0s \n",
"\n",
"2022-01-17 02:17:42 (59.4 MB/s) - ‘helper_functions.py’ saved [6442/6442]\n",
"\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"from helper_functions import create_tensorboard_callback, calculate_results, plot_loss_curves"
],
"metadata": {
"id": "i6KrTZnaFKtu"
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Function for reading the document\n",
"def get_doc(filename):\n",
" with open(filename, \"r\") as f:\n",
" return f.readlines()"
],
"metadata": {
"id": "JMifgqkxsC5w"
},
"execution_count": 6,
"outputs": []
},
{
"cell_type": "code",
"source": [
"data_dir = \"pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/\"\n",
"filenames = [data_dir + filename for filename in os.listdir(data_dir)]\n",
"filenames"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "RGZi11CDrVxa",
"outputId": "6a971c96-a1c9-43af-9936-98f0dfc75436"
},
"execution_count": 7,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/dev.txt',\n",
" 'pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/test.txt',\n",
" 'pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/train.txt']"
]
},
"metadata": {},
"execution_count": 7
}
]
},
{
"cell_type": "code",
"source": [
"# Preprocessing \n",
"train_lines = get_doc(data_dir+\"train.txt\")\n",
"train_lines[:30]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3ZBDfw7gsX-8",
"outputId": "ffc2d907-985f-4a5f-9659-f03d3126390d"
},
"execution_count": 8,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['###24293578\\n',\n",
" 'OBJECTIVE\\tTo investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( OA ) .\\n',\n",
" 'METHODS\\tA total of @ patients with primary knee OA were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .\\n',\n",
" 'METHODS\\tOutcome measures included pain reduction and improvement in function scores and systemic inflammation markers .\\n',\n",
" 'METHODS\\tPain was assessed using the visual analog pain scale ( @-@ mm ) .\\n',\n",
" 'METHODS\\tSecondary outcome measures included the Western Ontario and McMaster Universities Osteoarthritis Index scores , patient global assessment ( PGA ) of the severity of knee OA , and @-min walk distance ( @MWD ) .\\n',\n",
" 'METHODS\\tSerum levels of interleukin @ ( IL-@ ) , IL-@ , tumor necrosis factor ( TNF ) - , and high-sensitivity C-reactive protein ( hsCRP ) were measured .\\n',\n",
" 'RESULTS\\tThere was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , PGA , and @MWD at @ weeks .\\n',\n",
" 'RESULTS\\tThe mean difference between treatment arms ( @ % CI ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .\\n',\n",
" 'RESULTS\\tFurther , there was a clinically relevant reduction in the serum levels of IL-@ , IL-@ , TNF - , and hsCRP at @ weeks in the intervention group when compared to the placebo group .\\n',\n",
" 'RESULTS\\tThese differences remained significant at @ weeks .\\n',\n",
" 'RESULTS\\tThe Outcome Measures in Rheumatology Clinical Trials-Osteoarthritis Research Society International responder rate was @ % in the intervention group and @ % in the placebo group ( p < @ ) .\\n',\n",
" 'CONCLUSIONS\\tLow-dose oral prednisolone had both a short-term and a longer sustained effect resulting in less knee pain , better physical function , and attenuation of systemic inflammation in older patients with knee OA ( ClinicalTrials.gov identifier NCT@ ) .\\n',\n",
" '\\n',\n",
" '###24854809\\n',\n",
" 'BACKGROUND\\tEmotional eating is associated with overeating and the development of obesity .\\n',\n",
" 'BACKGROUND\\tYet , empirical evidence for individual ( trait ) differences in emotional eating and cognitive mechanisms that contribute to eating during sad mood remain equivocal .\\n',\n",
" 'OBJECTIVE\\tThe aim of this study was to test if attention bias for food moderates the effect of self-reported emotional eating during sad mood ( vs neutral mood ) on actual food intake .\\n',\n",
" 'OBJECTIVE\\tIt was expected that emotional eating is predictive of elevated attention for food and higher food intake after an experimentally induced sad mood and that attentional maintenance on food predicts food intake during a sad versus a neutral mood .\\n',\n",
" 'METHODS\\tParticipants ( N = @ ) were randomly assigned to one of the two experimental mood induction conditions ( sad/neutral ) .\\n',\n",
" 'METHODS\\tAttentional biases for high caloric foods were measured by eye tracking during a visual probe task with pictorial food and neutral stimuli .\\n',\n",
" 'METHODS\\tSelf-reported emotional eating was assessed with the Dutch Eating Behavior Questionnaire ( DEBQ ) and ad libitum food intake was tested by a disguised food offer .\\n',\n",
" 'RESULTS\\tHierarchical multivariate regression modeling showed that self-reported emotional eating did not account for changes in attention allocation for food or food intake in either condition .\\n',\n",
" 'RESULTS\\tYet , attention maintenance on food cues was significantly related to increased intake specifically in the neutral condition , but not in the sad mood condition .\\n',\n",
" 'CONCLUSIONS\\tThe current findings show that self-reported emotional eating ( based on the DEBQ ) might not validly predict who overeats when sad , at least not in a laboratory setting with healthy women .\\n',\n",
" 'CONCLUSIONS\\tResults further suggest that attention maintenance on food relates to eating motivation when in a neutral affective state , and might therefore be a cognitive mechanism contributing to increased food intake in general , but maybe not during sad mood .\\n',\n",
" '\\n',\n",
" '###25165090\\n',\n",
" 'BACKGROUND\\tAlthough working smoke alarms halve deaths in residential fires , many households do not keep alarms operational .\\n',\n",
" 'BACKGROUND\\tWe tested whether theory-based education increases alarm operability .\\n']"
]
},
"metadata": {},
"execution_count": 8
}
]
},
{
"cell_type": "markdown",
"source": [
"#### **Data dictionary**\n",
"\n",
"`\\t` = tab seperator\n",
"\n",
"`\\n` = new line\n",
"\n",
"`###` = abstract ID\n",
"\n",
"`\"line_number\"` = line position\n",
"\n",
"`\"text\"` = text line\n",
"\n",
"`\"total_lines\"` = total number of lines in one abstract\n",
"\n",
"`\"target\"` = objective of the abstract\n",
"\n"
],
"metadata": {
"id": "6ZOKbxQAsyNT"
}
},
{
"cell_type": "code",
"source": [
"# Function for preprocessing the data\n",
"\n",
"def preprocessing_text_with_line_number(filename):\n",
" input_lines = get_doc(filename)\n",
" abstract_lines = \"\"\n",
" abstract_samples = []\n",
"\n",
" for line in input_lines:\n",
" if line.startswith(\"###\"):\n",
" abstract_id = line\n",
" abstract_lines = \"\"\n",
" elif line.isspace():\n",
" abstract_line_split = abstract_lines.splitlines()\n",
" \n",
" for abstract_line_number, abstract_line in enumerate(abstract_line_split):\n",
" line_data = {}\n",
" target_text_split = abstract_line.split(\"\\t\")\n",
" line_data[\"target\"] = target_text_split[0]\n",
" line_data[\"text\"] = target_text_split[1].lower()\n",
" line_data[\"line_number\"] = abstract_line_number\n",
" line_data[\"total_lines\"] = len(abstract_line_split) - 1\n",
" abstract_samples.append(line_data)\n",
"\n",
" else:\n",
" abstract_lines += line \n",
" return abstract_samples "
],
"metadata": {
"id": "AEl-XZPQttN7"
},
"execution_count": 9,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Extracting data using the function\n",
"train_samples = preprocessing_text_with_line_number(data_dir + \"train.txt\")\n",
"val_samples = preprocessing_text_with_line_number(data_dir + \"dev.txt\")\n",
"test_samples = preprocessing_text_with_line_number(data_dir + \"test.txt\")\n",
"len(train_samples), len(test_samples), len(val_samples)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "SCD9DYIU17GH",
"outputId": "252fdd28-96fa-4cee-d94f-9065a1aef0b3"
},
"execution_count": 10,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(180040, 30135, 30212)"
]
},
"metadata": {},
"execution_count": 10
}
]
},
{
"cell_type": "code",
"source": [
"train_samples[:10]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Hacxnrvq2r7n",
"outputId": "f756d753-5bd9-401f-db37-f4359c104962"
},
"execution_count": 11,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[{'line_number': 0,\n",
" 'target': 'OBJECTIVE',\n",
" 'text': 'to investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( oa ) .',\n",
" 'total_lines': 11},\n",
" {'line_number': 1,\n",
" 'target': 'METHODS',\n",
" 'text': 'a total of @ patients with primary knee oa were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .',\n",
" 'total_lines': 11},\n",
" {'line_number': 2,\n",
" 'target': 'METHODS',\n",
" 'text': 'outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .',\n",
" 'total_lines': 11},\n",
" {'line_number': 3,\n",
" 'target': 'METHODS',\n",
" 'text': 'pain was assessed using the visual analog pain scale ( @-@ mm ) .',\n",
" 'total_lines': 11},\n",
" {'line_number': 4,\n",
" 'target': 'METHODS',\n",
" 'text': 'secondary outcome measures included the western ontario and mcmaster universities osteoarthritis index scores , patient global assessment ( pga ) of the severity of knee oa , and @-min walk distance ( @mwd ) .',\n",
" 'total_lines': 11},\n",
" {'line_number': 5,\n",
" 'target': 'METHODS',\n",
" 'text': 'serum levels of interleukin @ ( il-@ ) , il-@ , tumor necrosis factor ( tnf ) - , and high-sensitivity c-reactive protein ( hscrp ) were measured .',\n",
" 'total_lines': 11},\n",
" {'line_number': 6,\n",
" 'target': 'RESULTS',\n",
" 'text': 'there was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , pga , and @mwd at @ weeks .',\n",
" 'total_lines': 11},\n",
" {'line_number': 7,\n",
" 'target': 'RESULTS',\n",
" 'text': 'the mean difference between treatment arms ( @ % ci ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .',\n",
" 'total_lines': 11},\n",
" {'line_number': 8,\n",
" 'target': 'RESULTS',\n",
" 'text': 'further , there was a clinically relevant reduction in the serum levels of il-@ , il-@ , tnf - , and hscrp at @ weeks in the intervention group when compared to the placebo group .',\n",
" 'total_lines': 11},\n",
" {'line_number': 9,\n",
" 'target': 'RESULTS',\n",
" 'text': 'these differences remained significant at @ weeks .',\n",
" 'total_lines': 11}]"
]
},
"metadata": {},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"source": [
"# Creating a DataFrame\n",
"train_df = pd.DataFrame(train_samples)\n",
"val_df = pd.DataFrame(val_samples)\n",
"test_df = pd.DataFrame(test_samples)\n",
"train_df.head()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 206
},
"id": "gEQRm8jZ40lD",
"outputId": "dd84beae-2b55-49ad-a5a8-9692705471a6"
},
"execution_count": 12,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"\n",
" | \n", " | target | \n", "text | \n", "line_number | \n", "total_lines | \n", "
|---|---|---|---|---|
| 0 | \n", "OBJECTIVE | \n", "to investigate the efficacy of @ weeks of dail... | \n", "0 | \n", "11 | \n", "
| 1 | \n", "METHODS | \n", "a total of @ patients with primary knee oa wer... | \n", "1 | \n", "11 | \n", "
| 2 | \n", "METHODS | \n", "outcome measures included pain reduction and i... | \n", "2 | \n", "11 | \n", "
| 3 | \n", "METHODS | \n", "pain was assessed using the visual analog pain... | \n", "3 | \n", "11 | \n", "
| 4 | \n", "METHODS | \n", "secondary outcome measures included the wester... | \n", "4 | \n", "11 | \n", "
| \n", " | accuracy | \n", "precision | \n", "recall | \n", "f1 | \n", "
|---|---|---|---|---|
| model_1 | \n", "72.183238 | \n", "0.718647 | \n", "0.721832 | \n", "0.698925 | \n", "
| model_2 | \n", "78.677347 | \n", "0.783403 | \n", "0.786773 | \n", "0.784393 | \n", "
| model_3 | \n", "71.435191 | \n", "0.714815 | \n", "0.714352 | \n", "0.711263 | \n", "
| model_4 | \n", "66.142592 | \n", "0.656632 | \n", "0.661426 | \n", "0.652652 | \n", "
| model_5 | \n", "73.421157 | \n", "0.736113 | \n", "0.734212 | \n", "0.731693 | \n", "
| model_6 | \n", "83.364226 | \n", "0.832552 | \n", "0.833642 | \n", "0.832461 | \n", "
| \n", " | target | \n", "text | \n", "line_number | \n", "total_lines | \n", "prediction | \n", "pred_prob | \n", "correct | \n", "
|---|---|---|---|---|---|---|---|
| 8545 | \n", "METHODS | \n", "pretest-posttest . | \n", "1 | \n", "11 | \n", "BACKGROUND | \n", "0.947419 | \n", "False | \n", "
| 13874 | \n", "CONCLUSIONS | \n", "symptom outcomes will be assessed and estimate... | \n", "4 | \n", "6 | \n", "METHODS | \n", "0.943379 | \n", "False | \n", "
| 16633 | \n", "CONCLUSIONS | \n", "clinicaltrials.gov identifier : nct@ . | \n", "19 | \n", "19 | \n", "BACKGROUND | \n", "0.931800 | \n", "False | \n", "
| 16347 | \n", "BACKGROUND | \n", "to evaluate the effects of the lactic acid bac... | \n", "0 | \n", "12 | \n", "OBJECTIVE | \n", "0.929503 | \n", "False | \n", "
| 2388 | \n", "RESULTS | \n", "the primary endpoint is the cumulative three-y... | \n", "4 | \n", "13 | \n", "METHODS | \n", "0.926239 | \n", "False | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 24794 | \n", "RESULTS | \n", "we judged that informed consent would undermin... | \n", "11 | \n", "13 | \n", "CONCLUSIONS | \n", "0.791870 | \n", "False | \n", "
| 13921 | \n", "RESULTS | \n", "primary outcome was in-hospital all-cause mort... | \n", "7 | \n", "16 | \n", "METHODS | \n", "0.791018 | \n", "False | \n", "
| 18003 | \n", "RESULTS | \n", "this formulation produced highly significant a... | \n", "12 | \n", "20 | \n", "CONCLUSIONS | \n", "0.790558 | \n", "False | \n", "
| 29638 | \n", "METHODS | \n", "significance for all tests was set at p < @ . | \n", "8 | \n", "12 | \n", "RESULTS | \n", "0.790246 | \n", "False | \n", "
| 19047 | \n", "RESULTS | \n", "a randomized controlled trial and resting-stat... | \n", "2 | \n", "10 | \n", "METHODS | \n", "0.789534 | \n", "False | \n", "
200 rows × 7 columns
\n", "