import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.display import Markdown, display
def printTitle(title):
display(Markdown(("### **"+title+"**").upper()))
title = "TABLE 1 : importing our dataset, reading it into our dataframe, viewing the dataset"
printTitle(title)
data = pd.read_csv('word/insurance.csv')
data
4.3.2 ANALYSIS OF DATA ACCORDING TO RESEARCH QUESTIONS¶
Using the model for prediction, the steps to follow includes:
• Data processing
• Model evaluation
• Predicting insurance premium charge with our model1. DATA PROCESSING¶
a. DATA CLEANING¶
We need to represent all data numerically. Some of our predictors (sex and smoker) are categorical data and since they have two categories:
- sex - male and female
- smoker - yes and no
we encode these categories in binary values.
For sex:
- 0 for male
- 1 for female
For smoker:
- 1 for yes
- 0 for no
Also 'region' column has 4 categories:
data['region'].value_counts()
These will be encoded accordingly:
- southeast : 0
- northwest : 1
- southwest : 2
- northeast : 3
data['sex'] = data.sex.map({'male':0,'female':1})
data['smoker'] = data.smoker.map({'yes':1,'no':0})
data['region'] = data.region.map({'southeast' : 0, 'northwest': 1, 'southwest': 2, 'northeast':3})
After converting all categorical data to numerical data we have our dataset:
title = "table 2 : Our Dataset, After converting all categorical data to numerical data"
printTitle(title)
data
b. DATA VISUALIZATION¶
Here we are going to visualize the relationship between the dependent and the independent variable using scatterplots .
The relationship between charges and each independent variables can be seen below:
import seaborn as sns
col = list(data)
col.remove('charges')
title = "Graph 1 : Relationship between charges and each independent variables"
printTitle(title)
for i in col:
sns.pairplot(data, x_vars=[i], y_vars='charges', height=4, aspect=1.5)
c. QUESTIONS ABOUT THE INSURANCE DATASET¶
- Is there a relationship between:
- Sex and charges?
- Age and charges?
- Smoker and charges?
- bmi and charges?
- Children and charges?
- regions and charges?
- If there is, how strong is that relationship?
- What is the effect of each independent variable on the dependent variable?
- With this data set, can premium be estimated for a new life insurance policy holder?
- How accurate is the prediction?
This questions will be answered in the model evaluation stage
2. MODEL EVALUATION¶
a. PREPARING THE DATASET FOR TRAINING THE MODEL¶
At this stage we select independent variables to used in our model by evaluating their p-values.
We select and use independent variables that have strong relationship with the dependent variable and drop the column or not include in our model the independent variables that have weak or no relationship with the dependent variable.
import statsmodels.formula.api as smf
mod_coef = smf.ols(formula='charges ~ age + sex + bmi + children + smoker + region', data=data).fit()
Hypothesis Testing And P-Values¶
print "These are our independent variables and their corresponding p-values"
mod_coef.pvalues
From the output above each feature have significant p-values except sex and region. Thus we reject the null hypothesis for age, bmi, children, smoker and fail to reject the null hypothesis for sex and region. This is one way to determine which columns are insignificant and should be dropped from a dataset.
'smoker' has the strongest relationship with charges, followed by age, body mass index, children. Sex and region both have very weak relationship with charges therefore will be dropped from our data set and not be used in our model.
Here's a view of our dataset after dropping the sex and region column:
data_ii = data.drop(['sex','region'], axis=1)
title = "table 3 : Dataset after dropping the insignificant independent variables (sex and region column)"
printTitle(title)
data_ii
b. ESTIMATING OR 'LEARINING' THE MODEL COEFFICIENT¶
We use the Statsmodels Library to estimate our model coefficients.
import statsmodels.formula.api as smf
print "Independent variables and their Coefficient"
mod_coef = smf.ols(formula='charges ~ age + bmi + children + smoker', data=data_ii).fit()
mod_coef.params
The coefficient of each independent variable to be used in our model can be seen above
c. INTERPRETING THE MODEL COEFFICIENT¶
Generally a unit increase in any of: age, bmi, children, smoker, is associated with a unit increase in premium charges up to the value of the corresponding model coefficient.
More clearly: A Unit increase in body mass index of a policy holder is associated with a 321.851402 unit increase in his/her premium charge. If the model coefficient is negative, then a unit increase will cause a unit decrease in the premium charge.
d. SPLITTING THE DATASET INTO TRAINING AND TESTING SET¶
A higher percentage is used for training the model while a smaller portion of the data is used to test the model. This is implemented below:
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
X = data_ii[['age','bmi','children','smoker']]
y = data_ii.charges
#split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)
e. TRAINING AND TESTING THE MODEL¶
The model is trained to predict the known outputs/premium charges using the training dataset and later tested using the test dataset. The test data is used to test the accuracy of the model.
# instantiate the model
model = LinearRegression()
#fit Model
model.fit(X_train, y_train)
# make prediction based on the test data
predictions = model.predict(X_test)
f. MODEL EVALUATION: PREDICTION ACCURACY OF TEST DATASET AND R-SQUARED¶
Prediction Accuracy Of The Test Set¶
After training the model with our training dataset, we try to predict our already know dependent variable/premium charge in our test dataset then compare this prediction (predicted test charges) against the actual test charges to evaluate the accuracy of the model.
Below is a table showing Actual Test Dataset Premium Charges against their Predicted Test Dataset Premium Charges:
from pandas import Series, DataFrame
data_iii = {'Actual Test Dataset Premium Charges':y_test, 'Predicted Test Dataset Premium Charges':predictions}
frameData_iii = pd.DataFrame(data_iii)
title = "table 4 : Test Dataset Premium Charges: Actual Charges and their corresponding Predicted Charges"
printTitle(title)
frameData_iii
# accuracy of the prediction
model_accuracy = model.score(X_test, y_test)
model_accuracy
The accuracy of the model is : 73.86%¶
Visualizing The Relationship Between The Actual Premium Charges In Test Data Set And Their Corresponding Predicted Value Using Our Model¶
sns.distplot(y_test - predictions, axlabel="Test vs Prediction")
title = 'Graph 2 : Relationship Between Actual Premium Charges In Test Dataset VS Their Corresponding Predicted Value (Using Our Model For Prediction)'
printTitle(title)
R-squared¶
To evaluate the overall fit of a linear model, we analyse the R-squared. The proportion of variance in the observed data that is explained by the model is the R-squared. The R-squared is between 0 and 1 and the higher the R-square the better because it means that more variance is explained by the model.
# R-square
model_rsquare = mod_coef.rsquared
model_rsquare
The R-squared value for our model is 74.97%¶
The R-squared is most useful in comparing different models
g. SUMMARY OF THE FITTED MODEL¶
title = "table 5 : SUMMARY OF THE FITTED MODEL"
printTitle(title)
mod_coef.summary()
3. USING OUR MODEL TO PREDICT INSURANCE PREMIUM CHARGE¶
CASE STUDY:¶
A new policy holder insured his life at Leadway Insurance Company. Among other details provided, the following were relevant to this study:
- Age : 55
- Sex : male
- BMI : 47.843
- Children : 4
- Smoker : no
- Region : Southeast
Determine/predict the premium that should be charged.
IMPLEMENTATION¶
We predict insurance premium charge for a new policy holder following the steps:
- Import the Multiple Linear Regression Model from the Python’s Scikit-Learn Library
- Instantiate or run the model
- Fit the data into our model
- Predict for a new policy holder using model coefficient
Recall that region and sex do not have strong relationship with insurance premium charge. Hence for better prediction, they will be excluded from our model.
#import the linear regression model
from sklearn.linear_model import LinearRegression
# x represent the independent variables while y, the dependent variable
X = data_ii[['age','bmi','children','smoker']]
y = data_ii.charges
#instantiate the model
my_model = LinearRegression()
# fit our dataset into our model
my_model.fit(X,y)
PREDICT PREMIUM CHARGE FOR A NEW POLICY HOLDER¶
my_model.predict([[55,47.843,4,0]])