113 lines (112 with data), 2.7 kB

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "caa16d01",
   "metadata": {},
   "source": [
    "# Using pre-trained models on text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d9f0014c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec0f3fba",
   "metadata": {},
   "source": [
    "Repositories like Hugging Face provide a huge number of pre-trained models that can tranform text data into highly representative features.\n",
    "\n",
    "In this example we use ClinicalBERT, a language model initialized from the more general language model BERT and then further trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases. This approach was inspired by [this blogpost](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb#scrollTo=4wxY3x-ZZz8h)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01dbc978",
   "metadata": {},
   "source": [
    "First install the libraries for HuggingFace: Transformers and Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a160ce7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q transformers datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6297d597",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModel\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"medicalai/ClinicalBERT\")\n",
    "model = AutoModel.from_pretrained(\"medicalai/ClinicalBERT\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1835f4f3",
   "metadata": {},
   "source": [
    "Load in prior dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a790b4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data as a pandas dataframe\n",
    "df = pd.read_csv('../data/overview-of-recordings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6245713",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}