{

 "cells": [

  {

   "cell_type": "markdown",

   "id": "2e4374d0",

   "metadata": {},

   "source": [

    "# Phase 1\n",

    "## UK Smokers Prediction ML Project\n",

    "### 21 April 2024\n",

    "\n",

    "Wong Yi Wei (Ethan) S3966890"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "29f059e3",

   "metadata": {},

   "source": [

    "## Table of Contents\n",

    "* [1.0 Literature Review](#1)<br>\n",

    "* [2.0 Introduction](#2)<br>\n",

    "* [3.0 Goals and Objectives](#3)<br>\n",

    "* [4.0 Data Preprocessing and Cleaning](#4)<br>\n",

    "    * [4.1 Data Cleaning](#4.1) <br>\n",

    "    * [4.2 Data Preprocessing](#4.2) <br> \n",

    "* [5.0 Data Exploration and Visualization](#5)<br>\n",

    "    * [5.1 Descriptive Statistics](#5.1) <br>\n",

    "    * [5.2 One-Variable](#5.2) <br> \n",

    "    * [5.3 Two-Variable](#5.3) <br>\n",

    "    * [5.4 Three-Variable](#5.4) <br> \n",

    "* [6.0 Summary and Conclusion](#6)<br>\n",

    "* [7.0 References](#7)<br>\n",

    "\n",

    "***"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "8eba3ce1",

   "metadata": {},

   "source": [

    "## Setup"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 2,

   "id": "033ee1c4",

   "metadata": {},

   "outputs": [],

   "source": [

    "import pandas as pd\n",

    "import numpy as np\n",

    "pd.set_option('display.max_columns', None)\n",

    "import seaborn as sns\n",

    "import matplotlib.pyplot as plt"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "27a598c1",

   "metadata": {},

   "source": [

    "***\n",

    "\n",

    "## 1.0 Literature Review <a class=\"anchor\" id=\"1\"></a>\n",

    "\n",

    "It is widely known that smoking is harmful to the human health, and also towards the environment. A study on smoking mortality has revealed that mortality among smokers are 2 to 3 times higher than a person who has never touched a cigarette in their lives (Carter et al. 2015), and findings from this study were that in the United States (US), there are at least 60,000 additional deaths among US men and women yearly due to cigarette smoking. Smoking is also responsible for almost 600,000 deaths resulting from second hand smoking every year (Durmuşoğlu and Çiftçi 2019).  Furthermore, a conference paper has noted that their findings support their hypothesis where men will have a much higher rate in smoking when compared to women (Kor et al. 2019).\n",

    "\n",

    "Due to the differences in human biology, men and women bodies deal with risks caused by smoking differently. In a study conducted for smoking and gender, it was revealed that the impact of smoking may be more detrimental amongst women in postmenopausal due to the lack of oestrogens within the body (Bolego et al. 2002). Although the risk is higher, smoking nevertheless still poses a harmful risk to the human body. People who started smoking have their factors that enabled them to pursue smoking, it could be demographic factors, or even socio-economic factors. In a study done to understand socio demographic factors leading to smoking, it was identified that age is an important factor leading to smoking among the younger population where there was an association between smokers and their parent’s smoking habit (Farzana et al. 2014). This is revealing as people in the medium to high age groups have an important role in impacting the younger generation to start smoking. Moreover, another study has shown that smoking in the younger age leads to the acceleration of smoking related diseases (Redgrave et al. 2010).\n",

    "\n",

    "Other demographic factors such as income level could potentially have an impact on smokers. In an article published on associations between income level and smoking, lower income groups were consistently associated with a higher smoking rate, followed by medium income and higher income groups (Casetta et al. 2017). However, results from the World Health Survey in an article noted that smoking prevalence rate were higher in middle income countries compared to lower income countries, and varied amongst gender (Hosseinpoor 2012). Nevertheless, low- and middle-income individuals were more prone to smoking prevalence. A study conducted in the US which tackles the race and smoking prevalence found that back young adults have a much higher prevalence in smoking cigarettes (Chen-Sankey 2021). Although there may be different race groups comprised in other countries, this study reveals that other ethnicities from white could be prone to smoking prevalence. A study done in the United Kingdom (UK) revealed that non-Hispanic whites were less likely to smoke intermittently when compared to other ethnics (Trinidad et al. 2009).  Furthermore, another study has found that there are significant associations between ancestry and smoking related traits for different ethnicities (Choquet et al. 2021). Moreover, with the addition of low education and low-income level, there are a higher association with smoking frequencies (Chen et al. 2022).\n",

    "\n",

    "In a finding from a published article, it was noted that only 21% of countries around the world are on track to achieve their tobacco prevalence targets by 2025 (Bilano et al. 2015). This indicates that there are many countries around the world struggling to control tobacco prevalence amongst their population, which could lead to more smokers in the future. In a conference paper published, it was found that smoking amongst the younger population in the UK have significantly reduced amongst the younger population before the popularity of pre-disposable vapes (Tattan-Birch 2024). Although nicotine prevalence has increased, this paper indicates that smoking amongst younger population have reduced. In addition, a conference held by the UK Centre for Tobacco Control Studies identified that between 1996 and 2010, smoking prevalence amongst young teenagers have significantly declined in their proportion within the population (Britton et al. 2012). \n",

    "\n",

    "\n",

    "\n",

    "***"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "52da6a10",

   "metadata": {},

   "source": [

    "## 2.0 Introduction <a class=\"anchor\" id=\"2\"></a>\n",

    "\n",

    "Smoking prevalence in the United Kingdom (UK) has statistically significantly decreased over time, from 20.2% of smokers in 2011 to 12.9% of smokers in 2022, which accounts for 7.3% decrease within the population (Revie and Mais 2023).  In this machine learning project, we will analyze and provide prediction on the future population of smokers in the UK. This project will be contained within 2 different phases, Phase 1 and Phase 2. In this report, we will be focusing on Phase 1, where the task in this phase is to perform data pre-processing, data exploration, and data visualization as appropriate. <br>\n",

    "\n",

    "This project uses the UK smoking data retrieved from *Kaggle*, which can be accessed at __[https://www.kaggle.com/datasets/mexwell/uk-smoking-data?resource=download](https://www.kaggle.com/datasets/mexwell/uk-smoking-data?resource=download)__ (MacQuarrie 2024). This dataset consists of the survey data on smoking habits in the UK, and is recommended by the author to be used in analyzing the demographic characteristics of smokers and types of tobacco consumed within the UK population. Within this dataset, there is a total of 1691 rows of observations and 12 columns of variables. We will now identify the dataset features and the target features which will be used future analysis. <br>\n",

    "\n",

    "The dataset features are as follows: <br>\n",

    "\n",

    "| Variable | Data Type | Units | Brief Description |\n",

    "|-----------|-----------|-------|-----------------------------|\n",

    "|gender| nominal categorical | Not Applicable | Gender of the respondent | \n",

    "|age| numeric | Not Applicable | Age of the respondent  |\n",

    "|marital_status| nominal categorical | Not Applicable | Marital status of the respondent (Divorced, Married, Separated, and Widowed) |\n",

    "|highest_qualification| ordinal categorical | Not Applicable | Highest education level attained of the respondent (A Levels, Degree, GCSE/CSE, GCSE/ O Level, Higher/ Sub Degree, No Qualification, ONC/ BTEC) |\n",

    "|nationality| nominal categorical | Not Applicable | Nationality of the respondent (British, English, Irish, Scottish, Welsh, Other, Refused, Unknown) |\n",

    "|ethnicity| nominal categorical | Not Applicable | Ethnicity of the respondent (Asian, Black, Chinese, Mixed, White, Refused, Unknown)|\n",

    "|gross_income| ordinal categorical | Pound Sterling (£) | Gross income of the respondent (Under 2,600, 2,600 to 5,200, 5,200 to 10,400, 10,400 to 15,600, 15,600 to 20,800, 20,800 to 28,600, 28,600 to 36,400, Above 36,400, Unknown) |\n",

    "|region| nominal categorical | Not Applicable | Region where the participant is located (London, Midlands & East Anglia, Scotland, South East, South West, The North, Wales)|\n",

    "|smoke| binary | Not Applicable | Smoking status of the participant (Yes/ No) |\n",

    "|amt_weekends| numeric | Not Applicable | Number of cigarettes smoked per day on weekends by the participant |\n",

    "|amt_weekdays| numeric | Not Applicable | Number of cigarettes smoked per day on weekdays by the participant |\n",

    "|type| nominal categorical | Not Applicable | Type of cigarettes used by the participant | \n",

    "<br>\n",

    "\n",

    "The target features are as follows: <br>\n",

    "\n",

    "|Variable| Feature Type |\n",

    "|----------|----------|\n",

    "|age| numerical|\n",

    "|gender| categorical | \n",

    "|highest_qualification| categorical |\n",

    "|nationality| categorical |\n",

    "|ethnicity| categorical |\n",

    "|gross_income| categorical |\n",

    "|smoke| categorical |\n",

    "|amt_weekends| numerical |\n",

    "|amt_weekdays| numerical |\n",

    "\n",

    "For the target features, we shall use the smoke variable as our response variable, and the latter as our dependent variable. The reason the smoke variable is chosen is because it tells us whether the respondent is a smoker or not. Thus, rendering it to only allow two possible outcomes from the variable which are expressed mathematically as:\n",

    "\n",

    "$$smoke =\\begin{cases}0,& non-smoker\\\\1,& smoker\\end{cases}$$\n",

    " \n",

    "\n",

    "***"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "fcb86b09",

   "metadata": {},

   "source": [

    "## 3.0 Goals & Objectives <a class=\"anchor\" id=\"3\"></a>\n",

    "\n",

    "In this project throughout both phases, the primary goal is to predict smokers in the United Kingdom, based on various demographics within the data, and to look for ways to improve upon the current dataset and models for future analysis.\n",

    "\n",

    "In this Phase 1 report, the goal is to obtain a clean and tidy dataset for further analysis and to get the data to be ready for modeling which will be carried out in Phase 2 of the project. The objectives of this report to achieve the goals for this phase is by data handling, and data exploration and visualization, and cleaning the dataset as appropriate, such as by dealing with missing values, outliers, and incorrect values.\n",

    "\n",

    "To simplify and aid in achieving the objectives, there will be 6 sections in this report which are introduction, goals and objectives, data preprocessing and cleaning, data exploration and visualization, and summary and conclusion, respectively.\n",

    "\n",

    "***\n",

    "\n",

    "## 4.0 Data Cleaning and Preprocessing <a class=\"anchor\" id=\"4\"></a>\n",

    "\n",

    "In this section of the report, we will be performing our data cleaning and preprocessing to prepare the data for further analysis. This section would we divided into two sub-sections, data cleaning and data preprocessing. In the first subsection, data cleaning would focus on any data consistencies, and taking appropriate actions to deal with missing values. The second subsection, data preprocessing would focus on ensuring that the data structure is correct and make any changes as appropriate.\n",

    "Before we perform any of the data cleaning or preprocessing process, The data will first be imported into *Python* from the *Kaggle* resource which were already downloaded under the name ```Phase1_Group85.csv```."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 3,

   "id": "b4cc49ae",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "(1691, 12)"

      ]

     },

     "execution_count": 3,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Import data\n",

    "smoke=pd.read_csv('Phase1.csv')\n",

    "smoke.shape"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 3,

   "id": "cc1e16bb",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "['gender',\n",

       " 'age',\n",

       " 'marital_status',\n",

       " 'highest_qualification',\n",

       " 'nationality',\n",

       " 'ethnicity',\n",

       " 'gross_income',\n",

       " 'region',\n",

       " 'smoke',\n",

       " 'amt_weekends',\n",

       " 'amt_weekdays',\n",

       " 'type']"

      ]

     },

     "execution_count": 3,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Column names\n",

    "smoke.columns.to_list()"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 4,

   "id": "2764cf75",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/html": [

       "<div>\n",

       "<style scoped>\n",

       "    .dataframe tbody tr th:only-of-type {\n",

       "        vertical-align: middle;\n",

       "    }\n",

       "\n",

       "    .dataframe tbody tr th {\n",

       "        vertical-align: top;\n",

       "    }\n",

       "\n",

       "    .dataframe thead th {\n",

       "        text-align: right;\n",

       "    }\n",

       "</style>\n",

       "<table border=\"1\" class=\"dataframe\">\n",

       "  <thead>\n",

       "    <tr style=\"text-align: right;\">\n",

       "      <th></th>\n",

       "      <th>gender</th>\n",

       "      <th>age</th>\n",

       "      <th>marital_status</th>\n",

       "      <th>highest_qualification</th>\n",

       "      <th>nationality</th>\n",

       "      <th>ethnicity</th>\n",

       "      <th>gross_income</th>\n",

       "      <th>region</th>\n",

       "      <th>smoke</th>\n",

       "      <th>amt_weekends</th>\n",

       "      <th>amt_weekdays</th>\n",

       "      <th>type</th>\n",

       "    </tr>\n",

       "  </thead>\n",

       "  <tbody>\n",

       "    <tr>\n",

       "      <th>1540</th>\n",

       "      <td>Male</td>\n",

       "      <td>49</td>\n",

       "      <td>Married</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>English</td>\n",

       "      <td>White</td>\n",

       "      <td>20,800 to 28,600</td>\n",

       "      <td>Wales</td>\n",

       "      <td>Yes</td>\n",

       "      <td>20.0</td>\n",

       "      <td>20.0</td>\n",

       "      <td>Packets</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>549</th>\n",

       "      <td>Female</td>\n",

       "      <td>69</td>\n",

       "      <td>Widowed</td>\n",

       "      <td>GCSE/O Level</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>5,200 to 10,400</td>\n",

       "      <td>Midlands &amp; East Anglia</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1100</th>\n",

       "      <td>Male</td>\n",

       "      <td>86</td>\n",

       "      <td>Widowed</td>\n",

       "      <td>Other/Sub Degree</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>20,800 to 28,600</td>\n",

       "      <td>South East</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>855</th>\n",

       "      <td>Male</td>\n",

       "      <td>72</td>\n",

       "      <td>Divorced</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>English</td>\n",

       "      <td>White</td>\n",

       "      <td>10,400 to 15,600</td>\n",

       "      <td>Midlands &amp; East Anglia</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>966</th>\n",

       "      <td>Female</td>\n",

       "      <td>93</td>\n",

       "      <td>Widowed</td>\n",

       "      <td>Other/Sub Degree</td>\n",

       "      <td>English</td>\n",

       "      <td>White</td>\n",

       "      <td>5,200 to 10,400</td>\n",

       "      <td>London</td>\n",

       "      <td>Yes</td>\n",

       "      <td>0.0</td>\n",

       "      <td>3.0</td>\n",

       "      <td>Packets</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1659</th>\n",

       "      <td>Female</td>\n",

       "      <td>31</td>\n",

       "      <td>Single</td>\n",

       "      <td>Other/Sub Degree</td>\n",

       "      <td>Scottish</td>\n",

       "      <td>White</td>\n",

       "      <td>5,200 to 10,400</td>\n",

       "      <td>Scotland</td>\n",

       "      <td>Yes</td>\n",

       "      <td>60.0</td>\n",

       "      <td>30.0</td>\n",

       "      <td>Packets</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1268</th>\n",

       "      <td>Female</td>\n",

       "      <td>21</td>\n",

       "      <td>Single</td>\n",

       "      <td>GCSE/O Level</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>2,600 to 5,200</td>\n",

       "      <td>South East</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1102</th>\n",

       "      <td>Male</td>\n",

       "      <td>61</td>\n",

       "      <td>Married</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>British</td>\n",

       "      <td>Unknown</td>\n",

       "      <td>5,200 to 10,400</td>\n",

       "      <td>South East</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>968</th>\n",

       "      <td>Female</td>\n",

       "      <td>57</td>\n",

       "      <td>Divorced</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>15,600 to 20,800</td>\n",

       "      <td>London</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1531</th>\n",

       "      <td>Female</td>\n",

       "      <td>89</td>\n",

       "      <td>Single</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>Welsh</td>\n",

       "      <td>White</td>\n",

       "      <td>Refused</td>\n",

       "      <td>Wales</td>\n",

       "      <td>No</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "      <td>NaN</td>\n",

       "    </tr>\n",

       "  </tbody>\n",

       "</table>\n",

       "</div>"

      ],

      "text/plain": [

       "      gender  age marital_status highest_qualification nationality ethnicity  \\\n",

       "1540    Male   49        Married      No Qualification     English     White   \n",

       "549   Female   69        Widowed          GCSE/O Level     British     White   \n",

       "1100    Male   86        Widowed      Other/Sub Degree     British     White   \n",

       "855     Male   72       Divorced      No Qualification     English     White   \n",

       "966   Female   93        Widowed      Other/Sub Degree     English     White   \n",

       "1659  Female   31         Single      Other/Sub Degree    Scottish     White   \n",

       "1268  Female   21         Single          GCSE/O Level     British     White   \n",

       "1102    Male   61        Married      No Qualification     British   Unknown   \n",

       "968   Female   57       Divorced      No Qualification     British     White   \n",

       "1531  Female   89         Single      No Qualification       Welsh     White   \n",

       "\n",

       "          gross_income                  region smoke  amt_weekends  \\\n",

       "1540  20,800 to 28,600                   Wales   Yes          20.0   \n",

       "549    5,200 to 10,400  Midlands & East Anglia    No           NaN   \n",

       "1100  20,800 to 28,600              South East    No           NaN   \n",

       "855   10,400 to 15,600  Midlands & East Anglia    No           NaN   \n",

       "966    5,200 to 10,400                  London   Yes           0.0   \n",

       "1659   5,200 to 10,400                Scotland   Yes          60.0   \n",

       "1268    2,600 to 5,200              South East    No           NaN   \n",

       "1102   5,200 to 10,400              South East    No           NaN   \n",

       "968   15,600 to 20,800                  London    No           NaN   \n",

       "1531           Refused                   Wales    No           NaN   \n",

       "\n",

       "      amt_weekdays     type  \n",

       "1540          20.0  Packets  \n",

       "549            NaN      NaN  \n",

       "1100           NaN      NaN  \n",

       "855            NaN      NaN  \n",

       "966            3.0  Packets  \n",

       "1659          30.0  Packets  \n",

       "1268           NaN      NaN  \n",

       "1102           NaN      NaN  \n",

       "968            NaN      NaN  \n",

       "1531           NaN      NaN  "

      ]

     },

     "execution_count": 4,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#10 Random observations\n",

    "smoke.iloc[:, -12:].sample(n=10,random_state=8)"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "19669cdb",

   "metadata": {},

   "source": [

    "In the above codes, firstly for the shape of the data we can see that there are 1697 observations of data and 12 columns of variables. Next, the column names in the list for the data is correct with the identical column names from the dataset file. Finally, 10 random observations from the dataset is provided with the output above. From the random observations, we can identify that there are many missing values which exists in the ```amt_weekends```, ```amt_weekdays```, and ```type``` columns. These issues would be dealt later in this report.\n",

    "\n",

    "Now that we are satisfied that the dataset have been imported into *Python* with no issues, we will now proceed with the data cleaning and transformation of the data.\n",

    "\n",

    "### 4.1 Data Cleaning <a class=\"anchor\" id=\"4.1\"></a>\n",

    "\n",

    "In the first step of data cleaning, we would first look for the unique values in all the variables from all observations. Data cleaning is a very important as uncleaned data would have a negative or lead to misguided directions during analysis (Samarth 2023). In this step, we would understand the data and look out for are any inconsistencies or missing values."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 5,

   "id": "9d8759bc",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "gender\n",

      "['Male' 'Female']\n",

      "\n",

      "age\n",

      "[38 42 40 39 37 53 44 41 72 49 29 79 25 27 30 47 69 55 34 36 56 71 58 83\n",

      " 73 31 26 57 22 78 74 85 75 80 33 81 76 59 54 28 89 64 61 20 82 23 67 43\n",

      " 18 63 50 66 62 17 68 65 35 52 60 16 24 32 48 91 70 87 21 77 46 51 84 45\n",

      " 19 90 86 88 93 95 97]\n",

      "\n",

      "marital_status\n",

      "['Divorced' 'Single' 'Married' 'Widowed' 'Separated']\n",

      "\n",

      "highest_qualification\n",

      "['No Qualification' 'Degree' 'GCSE/O Level' 'GCSE/CSE' 'Other/Sub Degree'\n",

      " 'Higher/Sub Degree' 'ONC/BTEC' 'A Levels']\n",

      "\n",

      "nationality\n",

      "['British' 'English' 'Scottish' 'Other' 'Welsh' 'Irish' 'Refused'\n",

      " 'Unknown']\n",

      "\n",

      "ethnicity\n",

      "['White' 'Mixed' 'Black' 'Refused' 'Asian' 'Chinese' 'Unknown']\n",

      "\n",

      "gross_income\n",

      "['2,600 to 5,200' 'Under 2,600' '28,600 to 36,400' '10,400 to 15,600'\n",

      " '15,600 to 20,800' 'Above 36,400' '5,200 to 10,400' 'Refused'\n",

      " '20,800 to 28,600' 'Unknown']\n",

      "\n",

      "region\n",

      "['The North' 'Midlands & East Anglia' 'London' 'South East' 'South West'\n",

      " 'Wales' 'Scotland']\n",

      "\n",

      "smoke\n",

      "['No' 'Yes']\n",

      "\n",

      "amt_weekends\n",

      "[nan 12.  6.  8. 15.  5. 20. 25. 40.  4. 30. 10.  7.  9.  2. 50. 16. 35.\n",

      " 18.  1.  0.  3. 60. 24. 45.]\n",

      "\n",

      "amt_weekdays\n",

      "[nan 12.  6.  8.  2. 20. 15. 25.  4. 10.  0. 30.  3.  7. 40.  9.  5. 50.\n",

      " 18. 35.  1. 55. 16. 24. 45.]\n",

      "\n",

      "type\n",

      "[nan 'Packets' 'Hand-Rolled' 'Both/Mainly Packets'\n",

      " 'Both/Mainly Hand-Rolled']\n",

      "\n"

     ]

    }

   ],

   "source": [

    "#Unique values\n",

    "colnames=['gender','age','marital_status','highest_qualification','nationality','ethnicity','gross_income'\n",

    "          ,'region','smoke','amt_weekends','amt_weekdays','type']\n",

    "\n",

    "for i in colnames:\n",

    "    print(i)\n",

    "    print(smoke[i].unique())\n",

    "    print('')"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "6a20f69d",

   "metadata": {},

   "source": [

    "From the output above, we can identify that there are missing values present in variables amt_weekdays, amt_weekends, and type. In variables ```nationality```, ```ethnicity```, and ```gross_income```, we can see that there are 2 unique names \"refused\" and \"unknown\". Since both these names represent an unknown value, we will replace it with a singular value labeled \"unknown\". "

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 6,

   "id": "d3631d52",

   "metadata": {},

   "outputs": [],

   "source": [

    "#Nationality, ethnicity, gross_income replacement\n",

    "smoke['nationality'] = smoke['nationality'].replace('Refused', 'Unknown')\n",

    "smoke['ethnicity'] = smoke['ethnicity'].replace('Refused', 'Unknown')\n",

    "smoke['gross_income'] = smoke['gross_income'].replace('Refused', 'Unknown')"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 7,

   "id": "c5044cf7",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "18"

      ]

     },

     "execution_count": 7,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Nationality \"unknown\" count\n",

    "smoke.query('nationality == \"Unknown\"').nationality.count()"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 8,

   "id": "71cf8847",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "15"

      ]

     },

     "execution_count": 8,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Ethnicity \"unknown\" count\n",

    "smoke.query('ethnicity == \"Unknown\"').ethnicity.count()"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 9,

   "id": "276da618",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "126"

      ]

     },

     "execution_count": 9,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Gross_income \"unknown\" count\n",

    "smoke.query('gross_income == \"Unknown\"').gross_income.count()"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "4f3dcea5",

   "metadata": {},

   "source": [

    "In the above codes, we have converted all \"refused\" values into a singular \"unknown\" value. Furthermore, the amount of occurrences of the \"unknown\" value within the dataset for variables ```nationality```, ```ethnicity```, and ```gross_income```  was also calculated for further data cleaning.\n",

    "\n",

    "We can observe that there are 126, 18, and 15 counts of unknowns for variables ```nationality```, ```ethnicity```, and ```gross_income```. Since the sum of these is 159 and we have a total of 1691 observations, we are comfortable with omitting these observations to ensure a more complete dataset for further analysis. However, since these data could be important for analysis in dealing with the other variable's missing values, we will deal with this when the process is completed.\n",

    "\n",

    "Now, to deal with the missing values within the dataset, we will identify the amount of missing values present within the dataset for each variables."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 10,

   "id": "bd1cf428",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "Missing Values: \n",

      " gender                      0\n",

      "age                         0\n",

      "marital_status              0\n",

      "highest_qualification       0\n",

      "nationality                 0\n",

      "ethnicity                   0\n",

      "gross_income                0\n",

      "region                      0\n",

      "smoke                       0\n",

      "amt_weekends             1270\n",

      "amt_weekdays             1270\n",

      "type                     1270\n",

      "dtype: int64\n"

     ]

    }

   ],

   "source": [

    "#No. of N/A in dataset\n",

    "na_values = smoke.isnull().sum()\n",

    "print(\"Missing Values:\",'\\n', na_values)"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "9b8789b8",

   "metadata": {},

   "source": [

    "From the above output which shows the amount of missing values present in the variables, we can see that there are 1270 missing values for variables ```amt_weekends```,```amt_weekdays```, and ```type```. Since ```type``` is not relevant to our target features, we will then drop the column. Variables ```amt_weekends``` and ```amt_weekdays``` will be dealt with in appropriate data cleaning techniques."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 11,

   "id": "c3a719ed",

   "metadata": {},

   "outputs": [],

   "source": [

    "#Remove type column\n",

    "smoke=smoke.drop('type', axis=1)"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "1c369bf5",

   "metadata": {},

   "source": [

    "To deal with the missing values for ```amt_weekends``` and ```amt_weekdays```, we will first replace the values for the respondents who are non smokers with 0 as they do not smoke any cigarettes. However, the categorical feature for ```smoke``` should be converted into numeric as the values are binary, and this step could also aid in Phase 2 of the project."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 12,

   "id": "295eeee5",

   "metadata": {},

   "outputs": [],

   "source": [

    "#Replace 1/0 for Yes/No\n",

    "smoke['smoke'] = smoke['smoke'].map({'Yes':1, 'No':0})\n",

    "\n",

    "#Replace non-smokers with 0 amount\n",

    "smoke.loc[smoke.smoke == 0, ['amt_weekends', 'amt_weekdays']]=0,0"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 13,

   "id": "3ec97f4b",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "Missing Values: \n",

      " gender                   0\n",

      "age                      0\n",

      "marital_status           0\n",

      "highest_qualification    0\n",

      "nationality              0\n",

      "ethnicity                0\n",

      "gross_income             0\n",

      "region                   0\n",

      "smoke                    0\n",

      "amt_weekends             0\n",

      "amt_weekdays             0\n",

      "dtype: int64\n"

     ]

    }

   ],

   "source": [

    "#No. of N/A in dataset\n",

    "na_values1 = smoke.isnull().sum()\n",

    "print(\"Missing Values:\",'\\n', na_values1)"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 14,

   "id": "dc70190d",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "      age  smoke  amt_weekends  amt_weekdays\n",

      "591    42      1           0.0           2.0\n",

      "812    64      1           0.0           0.0\n",

      "927    22      1           0.0           7.0\n",

      "966    93      1           0.0           3.0\n",

      "1201   37      1           0.0           4.0\n",

      "1505   28      1           0.0           1.0\n"

     ]

    }

   ],

   "source": [

    "#Test weekends\n",

    "rslt_weekends=smoke.loc[(smoke['smoke'] == 1) & (smoke['amt_weekends'] == 0)]\n",

    "print(rslt_weekends[['age','smoke','amt_weekends','amt_weekdays']])"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 15,

   "id": "bfb487ed",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "      age  smoke  amt_weekends  amt_weekdays\n",

      "69     44      1           5.0           0.0\n",

      "85     27      1          10.0           0.0\n",

      "247    31      1           5.0           0.0\n",

      "264    40      1           5.0           0.0\n",

      "786    34      1           3.0           0.0\n",

      "812    64      1           0.0           0.0\n",

      "828    55      1           1.0           0.0\n",

      "891    30      1           1.0           0.0\n",

      "923    29      1           6.0           0.0\n",

      "995    26      1           5.0           0.0\n",

      "1016   40      1           2.0           0.0\n",

      "1303   36      1          10.0           0.0\n",

      "1320   43      1           5.0           0.0\n",

      "1386   42      1          12.0           0.0\n",

      "1613   68      1           1.0           0.0\n",

      "1661   23      1           5.0           0.0\n"

     ]

    }

   ],

   "source": [

    "#Test weekdays\n",

    "rslt_weekdays=smoke.loc[(smoke['smoke'] == 1) & (smoke['amt_weekdays'] == 0)]\n",

    "print(rslt_weekdays[['age','smoke','amt_weekends','amt_weekdays']])"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 16,

   "id": "bbbb2241",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "Empty DataFrame\n",

      "Columns: [age, smoke, amt_weekends, amt_weekdays]\n",

      "Index: []\n"

     ]

    }

   ],

   "source": [

    "#Test non-weekends\n",

    "rslt_nonweekends=smoke.loc[(smoke['smoke'] == 0) & (smoke['amt_weekends'] > 0)]\n",

    "print(rslt_nonweekends[['age','smoke','amt_weekends','amt_weekdays']])"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 17,

   "id": "4f2d3d64",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "Empty DataFrame\n",

      "Columns: [age, smoke, amt_weekends, amt_weekdays]\n",

      "Index: []\n"

     ]

    }

   ],

   "source": [

    "#Test non-weekdays\n",

    "rslt_nonweekdays=smoke.loc[(smoke['smoke'] == 0) & (smoke['amt_weekdays'] > 0)]\n",

    "print(rslt_nonweekdays[['age','smoke','amt_weekends','amt_weekdays']])"

   ]

  },

  {

   "cell_type": "markdown",

   "id": "739e03b4",

   "metadata": {},

   "source": [

    "In the above outputs, we have successfully replaced the missing values in ```amt_weekends``` and ```amt_weekdays```. Furthermore, we have also performed testing for any abnormalities such as 0 value during the weekend and weekdays smoking amount if the respondent is a smoker. Another test was also done to test if any of the non-smokers have any data within the smoking amount during the weekends and weekdays. This is important as it ensures that the data is consistent. \n",

    "\n",

    "Interestingly, from the ```rslt_weekends``` and ```rslt_weekdays``` it was observed that some of the respondents only smoke on the weekends and not on weekdays, and vice versa. \n",

    "\n",

    "Now that these these variables with missing values are dealt with, we will revert to our previous suggestion of removing the observations with unknown values in the ```nationality```, ```ethnicity```, and ```gross_income``` variables. Furthermore, we will also create a new dataset with the updated values, and discretize numeric feature for variable ```age``` to 3 levels (Young, Middle-Aged, and Old). The reason behind this discretizing method is to reduce the number of numerical data within the variable and aids in future analysis."

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 18,

   "id": "bc153d05",

   "metadata": {},

   "outputs": [],

   "source": [

    "#Removing unknowns from nationality, ethnicity, and gross_income\n",

    "smoke.drop(smoke[smoke.nationality == \"Unknown\"].index,inplace=True)\n",

    "smoke.drop(smoke[smoke.ethnicity == \"Unknown\"].index,inplace=True)\n",

    "smoke.drop(smoke[smoke.gross_income == \"Unknown\"].index,inplace=True)"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 19,

   "id": "a9d44445",

   "metadata": {},

   "outputs": [],

   "source": [

    "#Discretize numeric features\n",

    "smoke1=smoke.copy()\n",

    "smoke1['age']=pd.qcut(smoke1['age'],\n",

    "                      q=3,\n",

    "                      labels=['Young','Middle-Aged','Old'])"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 20,

   "id": "067340f7",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "Young          553\n",

       "Middle-Aged    510\n",

       "Old            498\n",

       "Name: age, dtype: int64"

      ]

     },

     "execution_count": 20,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Checking discretized features\n",

    "smoke1['age'].value_counts()"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 21,

   "id": "233af364",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/html": [

       "<div>\n",

       "<style scoped>\n",

       "    .dataframe tbody tr th:only-of-type {\n",

       "        vertical-align: middle;\n",

       "    }\n",

       "\n",

       "    .dataframe tbody tr th {\n",

       "        vertical-align: top;\n",

       "    }\n",

       "\n",

       "    .dataframe thead th {\n",

       "        text-align: right;\n",

       "    }\n",

       "</style>\n",

       "<table border=\"1\" class=\"dataframe\">\n",

       "  <thead>\n",

       "    <tr style=\"text-align: right;\">\n",

       "      <th></th>\n",

       "      <th>gender</th>\n",

       "      <th>age</th>\n",

       "      <th>marital_status</th>\n",

       "      <th>highest_qualification</th>\n",

       "      <th>nationality</th>\n",

       "      <th>ethnicity</th>\n",

       "      <th>gross_income</th>\n",

       "      <th>region</th>\n",

       "      <th>smoke</th>\n",

       "      <th>amt_weekends</th>\n",

       "      <th>amt_weekdays</th>\n",

       "    </tr>\n",

       "  </thead>\n",

       "  <tbody>\n",

       "    <tr>\n",

       "      <th>0</th>\n",

       "      <td>Male</td>\n",

       "      <td>Young</td>\n",

       "      <td>Divorced</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>2,600 to 5,200</td>\n",

       "      <td>The North</td>\n",

       "      <td>0</td>\n",

       "      <td>0.0</td>\n",

       "      <td>0.0</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>1</th>\n",

       "      <td>Female</td>\n",

       "      <td>Middle-Aged</td>\n",

       "      <td>Single</td>\n",

       "      <td>No Qualification</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>Under 2,600</td>\n",

       "      <td>The North</td>\n",

       "      <td>1</td>\n",

       "      <td>12.0</td>\n",

       "      <td>12.0</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>2</th>\n",

       "      <td>Male</td>\n",

       "      <td>Middle-Aged</td>\n",

       "      <td>Married</td>\n",

       "      <td>Degree</td>\n",

       "      <td>English</td>\n",

       "      <td>White</td>\n",

       "      <td>28,600 to 36,400</td>\n",

       "      <td>The North</td>\n",

       "      <td>0</td>\n",

       "      <td>0.0</td>\n",

       "      <td>0.0</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>3</th>\n",

       "      <td>Female</td>\n",

       "      <td>Middle-Aged</td>\n",

       "      <td>Married</td>\n",

       "      <td>Degree</td>\n",

       "      <td>English</td>\n",

       "      <td>White</td>\n",

       "      <td>10,400 to 15,600</td>\n",

       "      <td>The North</td>\n",

       "      <td>0</td>\n",

       "      <td>0.0</td>\n",

       "      <td>0.0</td>\n",

       "    </tr>\n",

       "    <tr>\n",

       "      <th>4</th>\n",

       "      <td>Female</td>\n",

       "      <td>Young</td>\n",

       "      <td>Married</td>\n",

       "      <td>GCSE/O Level</td>\n",

       "      <td>British</td>\n",

       "      <td>White</td>\n",

       "      <td>2,600 to 5,200</td>\n",

       "      <td>The North</td>\n",

       "      <td>0</td>\n",

       "      <td>0.0</td>\n",

       "      <td>0.0</td>\n",

       "    </tr>\n",

       "  </tbody>\n",

       "</table>\n",

       "</div>"

      ],

      "text/plain": [

       "   gender          age marital_status highest_qualification nationality  \\\n",

       "0    Male        Young       Divorced      No Qualification     British   \n",

       "1  Female  Middle-Aged         Single      No Qualification     British   \n",

       "2    Male  Middle-Aged        Married                Degree     English   \n",

       "3  Female  Middle-Aged        Married                Degree     English   \n",

       "4  Female        Young        Married          GCSE/O Level     British   \n",

       "\n",

       "  ethnicity      gross_income     region  smoke  amt_weekends  amt_weekdays  \n",

       "0     White    2,600 to 5,200  The North      0           0.0           0.0  \n",

       "1     White       Under 2,600  The North      1          12.0          12.0  \n",

       "2     White  28,600 to 36,400  The North      0           0.0           0.0  \n",

       "3     White  10,400 to 15,600  The North      0           0.0           0.0  \n",

       "4     White    2,600 to 5,200  The North      0           0.0           0.0  "

      ]

     },

     "execution_count": 21,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#View discretized features\n",

    "smoke1.iloc[0:5]"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 22,

   "id": "7861a72e",

   "metadata": {},

   "outputs": [

    {

     "data": {

      "text/plain": [

       "(1561, 11)"

      ]

     },

     "execution_count": 22,

     "metadata": {},

     "output_type": "execute_result"

    }

   ],

   "source": [

    "#Updated shape\n",

    "smoke1.shape"

   ]

  },

  {

   "cell_type": "code",

   "execution_count": 23,

   "id": "a0e0e7d2",

   "metadata": {},

   "outputs": [

    {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "gender\n",

      "['Male' 'Female']\n",

      "\n",

      "age\n",

      "['Young', 'Middle-Aged', 'Old']\n",

      "Categories (3, object): ['Young' < 'Middle-Aged' < 'Old']\n",

      "\n",

      "marital_status\n",

      "['Divorced' 'Single' 'Married' 'Widowed' 'Separated']\n",

      "\n",

      "highest_qualification\n",

      "['No Qualification' 'Degree' 'GCSE/O Level' 'GCSE/CSE' 'Other/Sub Degree'\n",

      " 'Higher/Sub Degree' 'ONC/BTEC' 'A Levels']\n",

      "\n",

      "nationality\n",

      "['British' 'English' 'Scottish' 'Other' 'Welsh' 'Irish']\n",

      "\n",

      "ethnicity\n",

      "['White' 'Mixed' 'Black' 'Asian' 'Chinese']\n",

      "\n",

      "gross_income\n",

      "['2,600 to 5,200' 'Under 2,600' '28,600 to 36,400' '10,400 to 15,600'\n",

      " '15,600 to 20,800' 'Above 36,400' '5,200 to 10,400' '20,800 to 28,600']\n",

      "\n",

      "region\n",

      "['The North' 'Midlands & East Anglia' 'London' 'South East' 'South West'\n",

      " 'Wales' 'Scotland']\n",

      "\n",

      "smoke\n",

      "[0 1]\n",

      "\n",

      "amt_weekends\n",

      "[ 0. 12.  6.  8. 15.  5. 20. 25.  4. 30. 10. 40.  9.  7.  2. 50. 16. 35.\n",

      " 18.  1.  3. 60. 24. 45.]\n",

      "\n",

      "amt_weekdays\n",

      "[ 0. 12.  6.  8.  2. 20. 15. 25.  4. 10. 30.  3. 40.  9.  5. 50.  7. 18.\n",

      " 35.  1. 55. 16. 24. 45.]\n",