{

  "cells": [

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "view-in-github",

        "colab_type": "text"

      },

      "source": [

        "<a href=\"https://colab.research.google.com/github/sAndreotti/MedicalMeadow/blob/main/Llama_3_2_1B%2B3B_Conversational_%2B_2x_faster_finetuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "IqM-T1RTzY6C"

      },

      "source": [

        "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",

        "<div class=\"align-center\">\n",

        "  <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",

        "  <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",

        "  <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Join Discord if you need help + ⭐ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐\n",

        "</div>\n",

        "\n",

        "To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth?tab=readme-ov-file#-installation-instructions).\n",

        "\n",

        "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save) (eg for Llama.cpp).\n",

        "\n",

        "**[NEW] Try 2x faster inference in a free Colab for Llama-3.1 8b Instruct [here](https://colab.research.google.com/drive/1T-YBVfnphoVc8E2E854qF3jdia2Ll2W2?usp=sharing)**\n",

        "\n",

        "Features in the notebook:\n",

        "1. Uses Maxime Labonne's [FineTome 100K](https://huggingface.co/datasets/mlabonne/FineTome-100k) dataset.\n",

        "1. Convert ShareGPT to HuggingFace format via `standardize_sharegpt`\n",

        "2. Train on Completions / Assistant only via `train_on_responses_only`\n",

        "3. Unsloth now supports Torch 2.4, all TRL & Xformers versions & Python 3.12!"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 1,

      "metadata": {

        "id": "2eSvM9zX_2d3"

      },

      "outputs": [],

      "source": [

        "%%capture\n",

        "!pip install unsloth\n",

        "# Also get the latest nightly Unsloth!\n",

        "!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git"

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "r2v_X2fA0Df5"

      },

      "source": [

        "* We support Llama, Mistral, Phi-3, Gemma, Yi, DeepSeek, Qwen, TinyLlama, Vicuna, Open Hermes etc\n",

        "* We support 16bit LoRA or 4bit QLoRA. Both 2x faster.\n",

        "* `max_seq_length` can be set to anything, since we do automatic RoPE Scaling via [kaiokendev's](https://kaiokendev.github.io/til) method.\n",

        "* [**NEW**] We make Gemma-2 9b / 27b **2x faster**! See our [Gemma-2 9b notebook](https://colab.research.google.com/drive/1vIrqH5uYDQwsJ4-OO3DErvuv4pBgVwk4?usp=sharing)\n",

        "* [**NEW**] To finetune and auto export to Ollama, try our [Ollama notebook](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing)"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 2,

      "metadata": {

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 324,

          "referenced_widgets": [

            "470008e304ad4102a760e39b93dc7428",

            "13e72e122b75476eb9056c4a75089d71",

            "972895da680d49de8bc58957ce4fe341",

            "90346592f9244def90817ea11422648e",

            "cef09d4426c84eb09a8772ad22b02903",

            "e6c28d862df24f8ebfcd956921fa0c6d",

            "cf45cca2721842f5bc51d8e3db6b1fa6",

            "3af46468e29348adbf3c2749394767bc",

            "e7490d0da8854fc6abca6b9ecfb2fdaf",

            "0c356b068a6f498da7bcc5102d7a3f12",

            "1a5e5be75adb4466b90b29d4aecc9d7b",

            "f4c568c8e686433f8763c34879f46fd3",

            "d37ee73737d144479b19aa9c84bcbda5",

            "94b64d29515c49e788c9b0f02f97e131",

            "e0c4626438754fcab9ba26ec84691906",

            "ea9e76e320e0413083985d598cc62bd5",

            "4a1ea4cd642b437db3abb2ff39c19563",

            "a64e99e0c22c4bbf8e0bc70066475126",

            "e9812c0a8e814202a128fd11860b6b0e",

            "66179fda56844c47bb5a2207744370cb",

            "e9eada4508064f3bb063b0640dc04224",

            "1e1cad4d3d3c4612a158551589b5be1e",

            "1dfa05dc7e56420ab071392e802eb233",

            "bee09a78cbb24fd0b8d5edf7f48d2b1e",

            "0153b850366f4f798ef485268e62c775",

            "1a6a1710ac1e468685ec3ac850fbe225",

            "4e44198b265744ad954ad5c71238284a",

            "f5b0ed4646f84e9293198e8a78823f0d",

            "41196d3b7ebb4b3e81ab190be3efdc0f",

            "f9a52bdb2e6d4da49c4a79982aae354f",

            "1a3bd31d499741e398d964904a9e636d",

            "b47a9aebb5b447b282f0b29607bee5d3",

            "3d87f57ad8bf4d6b8da431d141f31218",

            "b84a3a5d44a9417aa3c2ad089edb962f",

            "1fbe9f9d35d64921b567119369591fa5",

            "a8126f821b164a99a0b5514ab8d8b513",

            "e33feef2505d45f7bbfd8f2a73b07b1a",

            "0a1843983b5c4744a7ef148a72dd06d5",

            "e2235493048f44bbb957c5680a2c48e7",

            "5959983c01fb4cbbb45a76d729beef52",

            "5fa9619b96274aa795dff3a5906992d2",

            "1a6c2123525f4ba9bb9967165c06a4a9",

            "8301a8f379af4ba8bfbe8e4b7a2eb812",

            "0ee49d6c54594cd7a5f4d9bdfe4c8a5c",

            "45d7468b23d14ea99d103689c891749f",

            "432ef064af77465b9da7d059179bd705",

            "ba5349ad5a8344709e8924f9ba3a306b",

            "5acddf8d428f4a3daff2b69a0ae45446",

            "e5d0ac47ca4042349ceb9a23f1399bf9",

            "ff04a0a610144d82aa271f72324a49ea",

            "1984ff4e3cc243e1a791ce094ffa87d3",

            "4726b87618454e858d3e7a2f982558df",

            "33647f79114144feae792e43cf17e4b7",

            "138d704d84ba403695d6d8c9538e29bc",

            "bacd0d7737244fd7905d4e40ebd54f6d"

          ]

        },

        "id": "QmUBVEnvCDJv",

        "outputId": "e96c2216-b3f6-4cc7-d555-afe838eda86e"

      },

      "outputs": [

        {

          "output_type": "stream",

          "name": "stdout",

          "text": [

            "🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n",

            "🦥 Unsloth Zoo will now patch everything to make training faster!\n",

            "==((====))==  Unsloth 2024.12.12: Fast Llama patching. Transformers: 4.47.1.\n",

            "   \\\\   /|    GPU: Tesla T4. Max memory: 14.748 GB. Platform: Linux.\n",

            "O^O/ \\_/ \\    Torch: 2.5.1+cu121. CUDA: 7.5. CUDA Toolkit: 12.1. Triton: 3.1.0\n",

            "\\        /    Bfloat16 = FALSE. FA [Xformers = 0.0.29. FA2 = False]\n",

            " \"-____-\"     Free Apache license: http://github.com/unslothai/unsloth\n",

            "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n"

          ]

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "model.safetensors:   0%|          | 0.00/2.24G [00:00<?, ?B/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "470008e304ad4102a760e39b93dc7428"

            }

          },

          "metadata": {}

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "generation_config.json:   0%|          | 0.00/184 [00:00<?, ?B/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "f4c568c8e686433f8763c34879f46fd3"

            }

          },

          "metadata": {}

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "tokenizer_config.json:   0%|          | 0.00/54.6k [00:00<?, ?B/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "1dfa05dc7e56420ab071392e802eb233"

            }

          },

          "metadata": {}

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "tokenizer.json:   0%|          | 0.00/9.09M [00:00<?, ?B/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "b84a3a5d44a9417aa3c2ad089edb962f"

            }

          },

          "metadata": {}

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "special_tokens_map.json:   0%|          | 0.00/454 [00:00<?, ?B/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "45d7468b23d14ea99d103689c891749f"

            }

          },

          "metadata": {}

        }

      ],

      "source": [

        "from unsloth import FastLanguageModel\n",

        "import torch\n",

        "max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!\n",

        "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",

        "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",

        "\n",

        "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",

        "fourbit_models = [\n",

        "    \"unsloth/Meta-Llama-3.1-8B-bnb-4bit\",      # Llama-3.1 2x faster\n",

        "    \"unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit\",\n",

        "    \"unsloth/Meta-Llama-3.1-70B-bnb-4bit\",\n",

        "    \"unsloth/Meta-Llama-3.1-405B-bnb-4bit\",    # 4bit for 405b!\n",

        "    \"unsloth/Mistral-Small-Instruct-2409\",     # Mistral 22b 2x faster!\n",

        "    \"unsloth/mistral-7b-instruct-v0.3-bnb-4bit\",\n",

        "    \"unsloth/Phi-3.5-mini-instruct\",           # Phi-3.5 2x faster!\n",

        "    \"unsloth/Phi-3-medium-4k-instruct\",\n",

        "    \"unsloth/gemma-2-9b-bnb-4bit\",\n",

        "    \"unsloth/gemma-2-27b-bnb-4bit\",            # Gemma 2x faster!\n",

        "\n",

        "    \"unsloth/Llama-3.2-1B-bnb-4bit\",           # NEW! Llama 3.2 models\n",

        "    \"unsloth/Llama-3.2-1B-Instruct-bnb-4bit\",\n",

        "    \"unsloth/Llama-3.2-3B-bnb-4bit\",\n",

        "    \"unsloth/Llama-3.2-3B-Instruct-bnb-4bit\",\n",

        "\n",

        "    \"unsloth/Llama-3.3-70B-Instruct-bnb-4bit\" # NEW! Llama 3.3 70B!\n",

        "] # More models at https://huggingface.co/unsloth\n",

        "\n",

        "model, tokenizer = FastLanguageModel.from_pretrained(\n",

        "    model_name = \"unsloth/Llama-3.2-3B-Instruct\", # or choose \"unsloth/Llama-3.2-1B-Instruct\"\n",

        "    max_seq_length = max_seq_length,\n",

        "    dtype = dtype,\n",

        "    load_in_4bit = load_in_4bit,\n",

        "    # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n",

        ")"

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "SXd9bTZd1aaL"

      },

      "source": [

        "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 3,

      "metadata": {

        "id": "6bZsfBuZDeCL",

        "colab": {

          "base_uri": "https://localhost:8080/"

        },

        "outputId": "34f0a776-1747-4b3a-989b-aff287497f2e"

      },

      "outputs": [

        {

          "output_type": "stream",

          "name": "stderr",

          "text": [

            "Unsloth 2024.12.12 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.\n"

          ]

        }

      ],

      "source": [

        "model = FastLanguageModel.get_peft_model(\n",

        "    model,\n",

        "    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",

        "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",

        "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",

        "    lora_alpha = 16,\n",

        "    lora_dropout = 0, # Supports any, but = 0 is optimized\n",

        "    bias = \"none\",    # Supports any, but = \"none\" is optimized\n",

        "    # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",

        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",

        "    random_state = 3407,\n",

        "    use_rslora = False,  # We support rank stabilized LoRA\n",

        "    loftq_config = None, # And LoftQ\n",

        ")"

      ]

    },

    {

      "cell_type": "code",

      "source": [

        "from datasets import load_dataset\n",

        "from unsloth.chat_templates import get_chat_template\n",

        "\n",

        "# Load the dataset\n",

        "dataset = load_dataset(\"medalpaca/medical_meadow_medical_flashcards\", split=\"train\")\n",

        "\n",

        "def convert_to_conversations(examples):\n",

        "    \"\"\"Convert the dataset format to conversations\"\"\"\n",

        "    conversations = []\n",

        "    for instruction, input_text, output in zip(\n",

        "        examples['instruction'],\n",

        "        examples['input'],\n",

        "        examples['output']\n",

        "    ):\n",

        "        # Combine instruction and input\n",

        "        user_content = f\"{instruction}: {input_text}\"\n",

        "        conv = [\n",

        "            {\"role\": \"user\", \"content\": user_content},\n",

        "            {\"role\": \"assistant\", \"content\": output}\n",

        "        ]\n",

        "        conversations.append(conv)\n",

        "    return {\"conversations\": conversations}\n",

        "\n",

        "# Convert to conversation format\n",

        "formatted_dataset = dataset.map(\n",

        "    convert_to_conversations,\n",

        "    batched=True,\n",

        "    remove_columns=dataset.column_names\n",

        ")\n",

        "\n",

        "# Set up the tokenizer with Llama-3.1 chat template\n",

        "tokenizer = get_chat_template(\n",

        "    tokenizer,\n",

        "    chat_template=\"llama-3.1\",\n",

        ")\n",

        "\n",

        "def formatting_prompts_func(examples):\n",

        "    \"\"\"Apply the chat template to the conversations\"\"\"\n",

        "    convos = examples[\"conversations\"]\n",

        "    texts = [\n",

        "        tokenizer.apply_chat_template(\n",

        "            convo,\n",

        "            tokenize=False,\n",

        "            add_generation_prompt=False\n",

        "        ) for convo in convos\n",

        "    ]\n",

        "    return {\"text\": texts}\n",

        "\n",

        "# Apply the formatting\n",

        "formatted_dataset = formatted_dataset.map(\n",

        "    formatting_prompts_func,\n",

        "    batched=True\n",

        ")\n",

        "\n",

        "# Save the formatted dataset to make it available in next cells\n",

        "import pickle\n",

        "with open('formatted_dataset.pkl', 'wb') as f:\n",

        "    pickle.dump(formatted_dataset, f)"

      ],

      "metadata": {

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 81,

          "referenced_widgets": [

            "a41034aac0fb4301ad12bbd8e2b0cc40",

            "b635e3775c164f94b6acec4dd20b192e",

            "356cc173337649588e8990e9b948e9cf",

            "17996150ba4946bc8b9531db9f5f31b5",

            "927a0a29edcd47e4a4d3d74df7a9ea2b",

            "5ce59b62adfb416b85bc0effcced25dd",

            "913b0f0ccc8848b0b2c3917950f70e93",

            "5b0cadad0628423ea3b308a648c3c56e",

            "5b0e393c08bc492aa4079297d6602aa7",

            "5c9790a723f54ec3aa26d61e1e6880b5",

            "48ca23b61bac4b9583993577ef4451c4",

            "2959bb04464046fd9c5b78e39e4d31aa",

            "5412eabb86f54fa6b39bda564fb08334",

            "362a07edc2ba457d9939bad3ead4f683",

            "d1937db6c00840ef895654c58c5cf844",

            "9324776524bd44ed83972090e1def47e",

            "b07669693dd840249f6d55d8ecc6f394",

            "7fd6c7363ade4591824599e64f62a8d8",

            "d06aac9749f14f74b1a9b4ea7fd4ab20",

            "badae3344d2448d69f5d33f419344c29",

            "afecd6f978f9451bace318a0ad92a593",

            "982831102f1f45959043b619108119b9"

          ]

        },

        "id": "cDgml9s70ymR",

        "outputId": "1e4276cc-1368-4792-bb37-f05e4c935954"

      },

      "execution_count": 10,

      "outputs": [

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "Map:   0%|          | 0/33955 [00:00<?, ? examples/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "a41034aac0fb4301ad12bbd8e2b0cc40"

            }

          },

          "metadata": {}

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "Map:   0%|          | 0/33955 [00:00<?, ? examples/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "2959bb04464046fd9c5b78e39e4d31aa"

            }

          },

          "metadata": {}

        }

      ]

    },

    {

      "cell_type": "code",

      "source": [

        "# Print an example to verify the format\n",

        "print(\"Example of formatted conversation:\")\n",

        "print(formatted_dataset[0]['text'])\n",

        "\n",

        "# The dataset is now ready for training\n",

        "# You can access it as formatted_dataset['text']\n",

        "\n",

        "# If you need to split it into train/validation sets:\n",

        "train_val = formatted_dataset.train_test_split(test_size=0.1, seed=42)\n",

        "train_data = train_val['train']\n",

        "val_data = train_val['test']\n",

        "\n",

        "print(\"\\nDataset sizes:\")\n",

        "print(f\"Train: {len(train_data)}\")\n",

        "print(f\"Validation: {len(val_data)}\")"

      ],

      "metadata": {

        "colab": {

          "base_uri": "https://localhost:8080/"

        },

        "id": "YPZSwk6319Dw",

        "outputId": "f5deb7bb-1e8a-470e-ca58-49a05cb98ab8"

      },

      "execution_count": 11,

      "outputs": [

        {

          "output_type": "stream",

          "name": "stdout",

          "text": [

            "Example of formatted conversation:\n",

            "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n",

            "\n",

            "Cutting Knowledge Date: December 2023\n",

            "Today Date: 26 July 2024\n",

            "\n",

            "<|eot_id|><|start_header_id|>user<|end_header_id|>\n",

            "\n",

            "Answer this question truthfully: What is the relationship between very low Mg2+ levels, PTH levels, and Ca2+ levels?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n",

            "\n",

            "Very low Mg2+ levels correspond to low PTH levels which in turn results in low Ca2+ levels.<|eot_id|>\n",

            "\n",

            "Dataset sizes:\n",

            "Train: 30559\n",

            "Validation: 3396\n"

          ]

        }

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "idAEIeSQ3xdS"

      },

      "source": [

        "<a name=\"Train\"></a>\n",

        "### Train the model\n",

        "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 12,

      "metadata": {

        "id": "95_Nn-89DhsL",

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 49,

          "referenced_widgets": [

            "11f1c457989b4211a66099e84f6beb73",

            "3aa2d2085fc64bfd89d2889621cfd2ae",

            "6d85c1fb9dc8478899b39a8f49ba6fde",

            "2794d95fbb924c58acd1d806c6f847ee",

            "2a1e24790e2e4df99c29fc9e0fc2da9d",

            "5b8c18ab8f934c33bab55beee17d7ac7",

            "34e7a0da7e4b494ba248654e124638b3",

            "e5c927adf18f44cdb054ae3123f8b526",

            "98d8d0fed266494abd1ec14a7e3c4d05",

            "ed3b09500a97415891a3c1cf1a681d6c",

            "5beefaa4ac9347bf8edf5d9e62e6baf0"

          ]

        },

        "outputId": "85db7ebb-50ef-4b1b-e2d2-314c58a84947"

      },

      "outputs": [

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "Map (num_proc=2):   0%|          | 0/33955 [00:00<?, ? examples/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "11f1c457989b4211a66099e84f6beb73"

            }

          },

          "metadata": {}

        }

      ],

      "source": [

        "from trl import SFTTrainer\n",

        "from transformers import TrainingArguments, DataCollatorForSeq2Seq\n",

        "from unsloth import is_bfloat16_supported\n",

        "\n",

        "trainer = SFTTrainer(\n",

        "    model = model,\n",

        "    tokenizer = tokenizer,\n",

        "    train_dataset = formatted_dataset,\n",

        "    dataset_text_field = \"text\",\n",

        "    max_seq_length = max_seq_length,\n",

        "    data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),\n",

        "    dataset_num_proc = 2,\n",

        "    packing = False, # Can make training 5x faster for short sequences.\n",

        "    args = TrainingArguments(\n",

        "        per_device_train_batch_size = 2,\n",

        "        gradient_accumulation_steps = 4,\n",

        "        warmup_steps = 5,\n",

        "        # num_train_epochs = 1, # Set this for 1 full training run.\n",

        "        max_steps = 60,\n",

        "        learning_rate = 2e-4,\n",

        "        fp16 = not is_bfloat16_supported(),\n",

        "        bf16 = is_bfloat16_supported(),\n",

        "        logging_steps = 1,\n",

        "        optim = \"adamw_8bit\",\n",

        "        weight_decay = 0.01,\n",

        "        lr_scheduler_type = \"linear\",\n",

        "        seed = 3407,\n",

        "        output_dir = \"outputs\",\n",

        "        report_to = \"none\", # Use this for WandB etc\n",

        "    ),\n",

        ")"

      ]

    },

    {

      "cell_type": "markdown",

      "source": [

        "We also use Unsloth's `train_on_completions` method to only train on the assistant outputs and ignore the loss on the user's inputs."

      ],

      "metadata": {

        "id": "C_sGp5XlG6dq"

      }

    },

    {

      "cell_type": "code",

      "source": [

        "from unsloth.chat_templates import train_on_responses_only\n",

        "trainer = train_on_responses_only(\n",

        "    trainer,\n",

        "    instruction_part = \"<|start_header_id|>user<|end_header_id|>\\n\\n\",\n",

        "    response_part = \"<|start_header_id|>assistant<|end_header_id|>\\n\\n\",\n",

        ")"

      ],

      "metadata": {

        "id": "juQiExuBG5Bt",

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 49,

          "referenced_widgets": [

            "3be9f1c4ca424b08b995986b31bc6f73",

            "cbaaf16637334b8c907be3c1eea8e36b",

            "ab35ff19da2b4ab49ea4f6ab3bc32981",

            "d9add78673484904995f4b3181c2d3d6",

            "d44d61caea1f4aa79d3fae256b5e351e",

            "4be9d0609a1d4a28b622ada5052fd653",

            "fbf5b0b982d5483bb160f1c745e5fa1d",

            "55be8a889c8a414f83c2af904513f96d",

            "567af20e2fab400aae2131ade113d39d",

            "a17fe71fa4f24581a5a5758d367888c3",

            "106ddc5392a747e88905c314032671fa"

          ]

        },

        "outputId": "69af3385-fc25-41d3-8207-5fa2e9091867"

      },

      "execution_count": 13,

      "outputs": [

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "Map:   0%|          | 0/33955 [00:00<?, ? examples/s]"

            ],

            "application/vnd.jupyter.widget-view+json": {

              "version_major": 2,

              "version_minor": 0,

              "model_id": "3be9f1c4ca424b08b995986b31bc6f73"

            }

          },

          "metadata": {}

        }

      ]

    },

    {

      "cell_type": "markdown",

      "source": [

        "We verify masking is actually done:"

      ],

      "metadata": {

        "id": "Dv1NBUozV78l"

      }

    },

    {

      "cell_type": "code",

      "source": [

        "tokenizer.decode(trainer.train_dataset[5][\"input_ids\"])"

      ],

      "metadata": {

        "id": "LtsMVtlkUhja",

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 55

        },

        "outputId": "f1c670d5-ed1b-4b00-df24-4ad6fc0909b0"

      },

      "execution_count": 14,

      "outputs": [

        {

          "output_type": "execute_result",

          "data": {

            "text/plain": [

              "'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nCutting Knowledge Date: December 2023\\nToday Date: 26 July 2024\\n\\n<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\nAnswer this question truthfully: What does low Mobility and bulging of TM suggest?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\nLow Mobility and bulging of TM is suggestive of Acute otitis media.<|eot_id|>'"

            ],

            "application/vnd.google.colaboratory.intrinsic+json": {

              "type": "string"

            }

          },

          "metadata": {},

          "execution_count": 14

        }

      ]

    },

    {

      "cell_type": "code",

      "source": [

        "space = tokenizer(\" \", add_special_tokens = False).input_ids[0]\n",

        "tokenizer.decode([space if x == -100 else x for x in trainer.train_dataset[5][\"labels\"]])"

      ],

      "metadata": {

        "id": "_rD6fl8EUxnG",

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 36

        },

        "outputId": "05599f09-3443-4fc1-82c1-4fc35958ed7f"

      },

      "execution_count": 15,

      "outputs": [

        {

          "output_type": "execute_result",

          "data": {

            "text/plain": [

              "'                                                   \\n\\nLow Mobility and bulging of TM is suggestive of Acute otitis media.<|eot_id|>'"

            ],

            "application/vnd.google.colaboratory.intrinsic+json": {

              "type": "string"

            }

          },

          "metadata": {},

          "execution_count": 15

        }

      ]

    },

    {

      "cell_type": "markdown",

      "source": [

        "[link text](https://)We can see the System and Instruction prompts are successfully masked!"

      ],

      "metadata": {

        "id": "3enWUM0jV-jV"

      }

    },

    {

      "cell_type": "code",

      "execution_count": 23,

      "metadata": {

        "id": "2ejIt2xSNKKp",

        "colab": {

          "base_uri": "https://localhost:8080/"

        },

        "outputId": "39fbd3c6-7979-484d-bf73-6bacf4dc8189"

      },

      "outputs": [

        {

          "output_type": "stream",

          "name": "stdout",

          "text": [

            "GPU = Tesla T4. Max memory = 14.748 GB.\n",

            "3.324 GB of memory reserved.\n"

          ]

        }

      ],

      "source": [

        "#@title Show current memory stats\n",

        "gpu_stats = torch.cuda.get_device_properties(0)\n",

        "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",

        "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",

        "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",

        "print(f\"{start_gpu_memory} GB of memory reserved.\")"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 17,

      "metadata": {

        "id": "yqxqAZ7KJ4oL",

        "colab": {

          "base_uri": "https://localhost:8080/",

          "height": 1000

        },

        "outputId": "3790f36d-d523-48c4-d930-718fb512a1a7"

      },

      "outputs": [

        {

          "output_type": "stream",

          "name": "stderr",

          "text": [

            "==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1\n",

            "   \\\\   /|    Num examples = 33,955 | Num Epochs = 1\n",

            "O^O/ \\_/ \\    Batch size per device = 2 | Gradient Accumulation steps = 4\n",

            "\\        /    Total batch size = 8 | Total steps = 60\n",

            " \"-____-\"     Number of trainable parameters = 24,313,856\n"

          ]

        },

        {

          "output_type": "display_data",

          "data": {

            "text/plain": [

              "<IPython.core.display.HTML object>"

            ],

            "text/html": [

              "\n",

              "    <div>\n",

              "      \n",

              "      <progress value='60' max='60' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",

              "      [60/60 02:49, Epoch 0/1]\n",

              "    </div>\n",

              "    <table border=\"1\" class=\"dataframe\">\n",

              "  <thead>\n",

              " <tr style=\"text-align: left;\">\n",

              "      <th>Step</th>\n",

              "      <th>Training Loss</th>\n",

              "    </tr>\n",

              "  </thead>\n",

              "  <tbody>\n",

              "    <tr>\n",

              "      <td>1</td>\n",

              "      <td>1.291800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>2</td>\n",

              "      <td>1.210600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>3</td>\n",

              "      <td>1.020600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>4</td>\n",

              "      <td>0.976700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>5</td>\n",

              "      <td>0.968400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>6</td>\n",

              "      <td>1.065500</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>7</td>\n",

              "      <td>1.091000</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>8</td>\n",

              "      <td>0.828600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>9</td>\n",

              "      <td>0.804600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>10</td>\n",

              "      <td>0.832100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>11</td>\n",

              "      <td>0.695600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>12</td>\n",

              "      <td>0.702200</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>13</td>\n",

              "      <td>0.747300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>14</td>\n",

              "      <td>0.753600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>15</td>\n",

              "      <td>0.803100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>16</td>\n",

              "      <td>0.856100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>17</td>\n",

              "      <td>0.883200</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>18</td>\n",

              "      <td>0.863200</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>19</td>\n",

              "      <td>0.730900</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>20</td>\n",

              "      <td>0.716000</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>21</td>\n",

              "      <td>0.851000</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>22</td>\n",

              "      <td>0.849400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>23</td>\n",

              "      <td>0.736900</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>24</td>\n",

              "      <td>0.807300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>25</td>\n",

              "      <td>0.845800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>26</td>\n",

              "      <td>0.920700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>27</td>\n",

              "      <td>0.739800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>28</td>\n",

              "      <td>0.816900</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>29</td>\n",

              "      <td>0.783700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>30</td>\n",

              "      <td>0.701500</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>31</td>\n",

              "      <td>0.734100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>32</td>\n",

              "      <td>0.804600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>33</td>\n",

              "      <td>0.882300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>34</td>\n",

              "      <td>0.776600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>35</td>\n",

              "      <td>0.731700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>36</td>\n",

              "      <td>1.109600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>37</td>\n",

              "      <td>0.821800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>38</td>\n",

              "      <td>0.616000</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>39</td>\n",

              "      <td>0.712900</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>40</td>\n",

              "      <td>0.735600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>41</td>\n",

              "      <td>0.902300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>42</td>\n",

              "      <td>0.832600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>43</td>\n",

              "      <td>0.791600</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>44</td>\n",

              "      <td>0.813500</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>45</td>\n",

              "      <td>0.876400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>46</td>\n",

              "      <td>0.814500</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>47</td>\n",

              "      <td>0.773300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>48</td>\n",

              "      <td>0.739800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>49</td>\n",

              "      <td>0.830400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>50</td>\n",

              "      <td>0.934300</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>51</td>\n",

              "      <td>0.864700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>52</td>\n",

              "      <td>1.015700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>53</td>\n",

              "      <td>0.777400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>54</td>\n",

              "      <td>0.705100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>55</td>\n",

              "      <td>0.844800</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>56</td>\n",

              "      <td>0.861400</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>57</td>\n",

              "      <td>0.848500</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>58</td>\n",

              "      <td>0.768700</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>59</td>\n",

              "      <td>0.684100</td>\n",

              "    </tr>\n",

              "    <tr>\n",

              "      <td>60</td>\n",

              "      <td>0.956200</td>\n",

              "    </tr>\n",

              "  </tbody>\n",

              "</table><p>"

            ]

          },

          "metadata": {}

        }

      ],

      "source": [

        "trainer_stats = trainer.train()"

      ]

    },

    {

      "cell_type": "code",

      "execution_count": 24,

      "metadata": {

        "id": "pCqnaKmlO1U9",

        "colab": {

          "base_uri": "https://localhost:8080/"

        },

        "outputId": "df7922d2-20c8-4c99-99dd-10eff2caf2e6"

      },

      "outputs": [

        {

          "output_type": "stream",

          "name": "stdout",

          "text": [

            "178.5531 seconds used for training.\n",

            "2.98 minutes used for training.\n",

            "Peak reserved memory = 3.324 GB.\n",

            "Peak reserved memory for training = 0.0 GB.\n",

            "Peak reserved memory % of max memory = 22.539 %.\n",

            "Peak reserved memory for training % of max memory = 0.0 %.\n"

          ]

        }

      ],

      "source": [

        "#@title Show final memory and time stats\n",

        "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",

        "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",

        "used_percentage = round(used_memory         /max_memory*100, 3)\n",

        "lora_percentage = round(used_memory_for_lora/max_memory*100, 3)\n",

        "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",

        "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\")\n",

        "print(f\"Peak reserved memory = {used_memory} GB.\")\n",

        "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",

        "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",

        "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "ekOmTR1hSNcr"

      },

      "source": [

        "<a name=\"Inference\"></a>\n",

        "### Inference\n",

        "Let's run the model! You can change the instruction and input - leave the output blank!\n",

        "\n",

        "**[NEW] Try 2x faster inference in a free Colab for Llama-3.1 8b Instruct [here](https://colab.research.google.com/drive/1T-YBVfnphoVc8E2E854qF3jdia2Ll2W2?usp=sharing)**\n",

        "\n",

        "We use `min_p = 0.1` and `temperature = 1.5`. Read this [Tweet](https://x.com/menhguin/status/1826132708508213629) for more information on why."

      ]

    },

    {

      "cell_type": "markdown",

      "metadata": {

        "id": "CrSvZObor0lY"

      },

      "source": [

        " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"

      ]

    },

    {

      "cell_type": "code",

      "source": [

        "from unsloth.chat_templates import get_chat_template\n",

        "from transformers import TextStreamer\n",

        "\n",

        "# Setup tokenizer with Llama-3.1 template\n",

        "tokenizer = get_chat_template(\n",

        "    tokenizer,\n",

        "    chat_template = \"llama-3.1\",\n",

        ")\n",

        "\n",

        "# Enable faster inference\n",

        "FastLanguageModel.for_inference(model)\n",

        "\n",

        "# Example medical question from our dataset\n",

        "messages = [\n",

        "    {\"role\": \"user\", \"content\": \"Answer this question truthfully: What is the relationship between very low Mg2+ levels, PTH levels, and Ca2+ levels?\"},\n",

        "]\n",

        "\n",

        "# Standard generation\n",

        "inputs = tokenizer.apply_chat_template(\n",

        "    messages,\n",

        "    tokenize = True,\n",

        "    add_generation_prompt = True,\n",

        "    return_tensors = \"pt\",\n",

        ").to(\"cuda\")\n",

        "\n",

        "print(\"Standard Generation:\")\n",

        "outputs = model.generate(\n",

        "    input_ids = inputs,\n",

        "    max_new_tokens = 128,\n",

        "    use_cache = True,\n",

        "    temperature = 0.7,  # Lower temperature for medical answers to be more precise\n",

        "    min_p = 0.1\n",

        ")\n",

        "print(tokenizer.batch_decode(outputs))\n",

        "\n",

        "# Streaming generation\n",

        "print(\"\\nStreaming Generation:\")\n",

        "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",