3. Linear Regression

Exercise 15: Regression models for the Boston Dataset

Preparing the data

import pandas as pd

boston = pd.read_csv('../../datasets/Boston.csv', index_col=0)
boston.head()

boston.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 506 entries, 1 to 506
Data columns (total 14 columns):
crim       506 non-null float64
zn         506 non-null float64
indus      506 non-null float64
chas       506 non-null int64
nox        506 non-null float64
rm         506 non-null float64
age        506 non-null float64
dis        506 non-null float64
rad        506 non-null int64
tax        506 non-null int64
ptratio    506 non-null float64
black      506 non-null float64
lstat      506 non-null float64
medv       506 non-null float64
dtypes: float64(11), int64(3)
memory usage: 59.3 KB

a. Regression models for each predictor

We want to predict the per capita crime rate crim.

import numpy as np
import statsmodels.formula.api as smf

# get predictor names
predictors = np.delete(boston.columns.values, np.where(boston.columns.values==['crim']))

# dictionary for models
models = {}

# fit single predictor models
for predictor in predictors:
    models[predictor] = smf.ols('crim ~ ' + predictor, data=boston).fit()

models_keys_iter = iter(models.values())

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.16e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.49e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

next(models_keys_iter).summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now we check which predictors are statistically significant, using the common rule-of-thumb $p < 0.05$

for model in models:
    if models[model].pvalues[1] < 0.05:
        print("{} is statistically significant with p-value {}". format(model, models[model].pvalues[1]))
    else:
        print("{} is NOT statistically significant with p-value {}". format(model, models[model].pvalues[1]))

zn is statistically significant with p-value 5.506472107679307e-06
indus is statistically significant with p-value 1.4503489330272395e-21
chas is NOT statistically significant with p-value 0.2094345015352004
nox is statistically significant with p-value 3.751739260356923e-23
rm is statistically significant with p-value 6.346702984687839e-07
age is statistically significant with p-value 2.8548693502441573e-16
dis is statistically significant with p-value 8.519948766926326e-19
rad is statistically significant with p-value 2.6938443981864414e-56
tax is statistically significant with p-value 2.357126835257048e-47
ptratio is statistically significant with p-value 2.942922447359816e-11
black is statistically significant with p-value 2.487273973773734e-19
lstat is statistically significant with p-value 2.6542772314731968e-27
medv is statistically significant with p-value 1.1739870821943694e-19
full is statistically significant with p-value 0.01702489149848948

Looks like all predictors were significant

b. Full regression model

# formula for full model
formula = ''
for predictor in predictors:
    formula += predictor + ' + '
formula = formula[:-3]
formula

# add full model
models['full'] = smf.ols('crim ~ ' + formula, data=boston).fit()

models['full'].summary()

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.58e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

Now we see which predictors were significant

TBC…