Step 1 : Import libraries and modules¶
In [1]:
import numpy as np
In [2]:
import pandas as pd
In [3]:
from sklearn.model_selection import train_test_split
In [4]:
# Next, we'll import the entire preprocessing module. This contains utilities for scaling, transforming,and wrangling data.
from sklearn import preprocessing
In [5]:
#Import random forest model
from sklearn.ensemble import RandomForestRegressor
In [6]:
#we'll only focus on training a random forest and tuning its parameters.Let's move on to importing the tools to help us perform cross-validation.
from sklearn.pipeline import make_pipeline
In [7]:
#Next, let's import some metrics we can use to evaluate our model performance later.
#Import evaluation metrics
from sklearn.metrics import mean_squared_error, r2_score
In [8]:
#And finally, we'll import a way to persist our model for future use.
#Import module for saving scikit-learn models
from sklearn.externals import joblib
Note :joblib is an alternative to Python's pickle package, and we'll use it because it's more efficient for storing large numpy arrays.
Step 2: Load red wine data.¶
The Pandas library that we imported is loaded with a whole suite of helpful import/output tools. You can read data from CSV, Excel, SQL, SAS, and many other data formats. Here's a list of all the Pandas IO tools (http://pandas.pydata.org/pandas-docs/stable/io.html). The convenient tool we'll use today is the read_csv() function. Using this function, we can load any CSV file, even from a remote URL!
In [13]:
dataset_url = 'http://mlr.cs.umass.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv'
data = pd.read_csv(dataset_url)
In [15]:
print (data.head())
In [16]:
data = pd.read_csv(dataset_url, sep=';')
print(data.head())
In [17]:
print(data.shape)
In [18]:
print(data.describe())
Here's the list of all the features:
quality (target)
fixed acidity
volatile acidity
citric acid
residual sugar
chlorides
free sulfur dioxide
total sulfur dioxide
density
pH
sulphates
alcohol
All of the features are numeric, which is convenient. However, they have some very different scales,
so let's make a mental note to standardize the data later.
As a reminder, for this tutorial, we're cutting out a lot of exploratory data analysis we'd typically
recommend.
For now, let's move on to splitting the data.
Step 3: Split data into training and test sets¶
Splitting the data into training and test sets at the beginning of your modeling workflow is
crucial for getting a realistic estimate of your model's performance.
First, let's separate our target (y) features from our input (X) features:
In [19]:
#Separate target from training features
y = data.quality
X = data.drop('quality', axis=1)
This allows us to take advantage of Scikit-Learn's useful train_test_split function:
In [20]:
#Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2,
random_state=123,
stratify=y)
As you can see, we'll set aside 20% of the data as a test set for evaluating our model. We also set
an arbitrary "random state" (a.k.a. seed) so that we can reproduce our results.
Finally, it's good practice to stratify your sample by the target variable. This will ensure your
training set looks similar to your test set, making your evaluation metrics more reliable.
Step 4: Data Processing¶
Remember, in Step 2, we made the mental note to standardize our features because they were on different scales.
What is standardization?
Standardization is the process of subtracting the means from each feature and then dividing by the feature standard deviations. Standardization is a common requirement for machine learning tasks. Many algorithms assume that all features are centered around zero and have approximately the same variance.
So instead of directly invoking the scale function, we'll be using a feature in Scikit-Learn called the Transformer API. The Transformer API allows you to "fit" a preprocessing step using the training data the same way you'd fit a model and then use the same transformation on future data sets!
Here's what that process looks like:
What is standardization?
Standardization is the process of subtracting the means from each feature and then dividing by the feature standard deviations. Standardization is a common requirement for machine learning tasks. Many algorithms assume that all features are centered around zero and have approximately the same variance.
So instead of directly invoking the scale function, we'll be using a feature in Scikit-Learn called the Transformer API. The Transformer API allows you to "fit" a preprocessing step using the training data the same way you'd fit a model and then use the same transformation on future data sets!
Here's what that process looks like:
- Fit the transformer on the training set (saving the means and standard deviations)
- Apply the transformer to the training set (scaling the training data)
- Apply the transformer to the test set (using the same means and standard deviations) This makes your final estimate of model performance more realistic, and it allows to insert your preprocessing steps into a cross-validation pipeline (more on this in Step 6).
In [21]:
#Fitting the Transformer API
scaler = preprocessing.StandardScaler().fit(X_train)
Now, the scaler object has the saved means and standard deviations for each feature in the training
set.
Let's confirm that worked:
In [22]:
#Applying transformer to training data
X_train_scaled = scaler.transform(X_train)
print (X_train_scaled.mean(axis=0))
In [24]:
print (X_train_scaled.std(axis=0))
In [25]:
#Applying transformer to test data
X_test_scaled = scaler.transform(X_test)
print (X_test_scaled.mean(axis=0))
print (X_test_scaled.std(axis=0))
In practice, when we set up the cross-validation pipeline, we won't even need to manually fit the
Transformer API. Instead, we'll simply declare the class object, like so:
In [26]:
#Pipeline with preprocessing and model
pipeline = make_pipeline(preprocessing.StandardScaler(),
RandomForestRegressor(n_estimators=100))
Step 5 :Hyperparameters to tune.¶
Now it's time to consider the hyperparameters that we'll want to tune for our model.
WTF are hyperparameters?
There are two types of parameters we need to worry about: model parameters and
hyperparameters. Models parameters can be learned directly from the data (i.e. regression
coefficients), while hyperparameters cannot.
Hyperparameters express "higher-level" structural information about the model, and they are typically
set before training the model.
Example: random forest hyperparameters.
As an example, let's take our random forest for regression:
X_train_scaled = scaler.transform(X_train)
print X_train_scaled.mean(axis=0)
However, the algorithm does not know which of the two criteria, MSE or MAE, that it should use. The algorithm also cannot decide how many trees to include in the forest. These are examples of hyperparameters that the user must set. We can list the tunable hyperparameters like so:
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]¶
print X_train_scaled.std(axis=0)[ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]¶
X_test_scaled = scaler.transform(X_test) print X_test_scaled.mean(axis=0)[ 0.02776704 0.02592492 -0.03078587 -0.03137977 -0.00471876 -0.04413827¶
-0.02414174 -0.00293273 -0.00467444 -0.10894663 0.01043391]¶
print X_test_scaled.std(axis=0)[ 1.02160495 1.00135689 0.97456598 0.91099054 0.86716698 0.94193125¶
1.03673213 1.03145119 0.95734849 0.83829505 1.0286218 ]¶
pipeline = make_pipeline(preprocessing.StandardScaler(), RandomForestRegressor(n_estimators=100)) Applying transformer to test data Python Pipeline with preprocessing and model Python Within each decision tree, the computer can empirically decide where to create branches based on either mean-squared-error (MSE) or mean-absolute-error (MAE). Therefore, the actual branch locations are model parameters.However, the algorithm does not know which of the two criteria, MSE or MAE, that it should use. The algorithm also cannot decide how many trees to include in the forest. These are examples of hyperparameters that the user must set. We can list the tunable hyperparameters like so:
In [27]:
print (pipeline.get_params())
Now, let's declare the hyperparameters we want to tune through cross-validation.
In [28]:
#Declare hyperparameters to tune
hyperparameters = { 'randomforestregressor__max_features' : ['auto', 'sqrt', 'log2'],
'randomforestregressor__max_depth': [None, 5, 3, 1]}
As you can see, the format should be a Python dictionary (data structure for key-value pairs) where
keys are the hyperparameter names and values are lists of settings to try. The options for parameter
values can be found on the documentation page.
Step 6: Tune model using a cross-validation pipeline.¶
Now we're almost ready to dive into fitting our models. But first, we need to spend some time talking
about cross-validation.
This is one of the most important skills in all of machine learning because it helps you maximize model performance while reducing the chance of overfitting.
How can you decide?
That's where cross-validation comes in. Using only your training set, you can use CV to evaluate different hyperparameters and estimate their effectiveness. This allows you to keep your test set "untainted" and save it for a true hold-out evaluation when you're finally ready to select a model. For example, you can use CV to tune a random forest model, a linear regression model, and a knearest neighbors model, using only the training set. Then, you still have the untainted test set to make your final selection between the model families!
This is one of the most important skills in all of machine learning because it helps you maximize model performance while reducing the chance of overfitting.
What is cross-validation (CV)?¶
Cross-validation is a process for reliably estimating the performance of a method for building a model by training and evaluating your model multiple times using the same method. Practically, that "method" is simply a set of hyperparameters in this context. These are the steps for CV:- Split your data into k equal parts, or "folds" (typically k=10).
- Train your model on k-1 folds (e.g. the first 9 folds).
- Evaluate it on the remaining "hold-out" fold (e.g. the 10th fold).
- Perform steps (2) and (3) k times, each time holding out a different fold.
- Aggregate the performance across all k folds. This is your performance metric.
Why is cross-validation important in machine learning?¶
Let's say you want to train a random forest regressor. One of the hyperparameters you must tune is the maximum depth allowed for each decision tree in your forest.How can you decide?
That's where cross-validation comes in. Using only your training set, you can use CV to evaluate different hyperparameters and estimate their effectiveness. This allows you to keep your test set "untainted" and save it for a true hold-out evaluation when you're finally ready to select a model. For example, you can use CV to tune a random forest model, a linear regression model, and a knearest neighbors model, using only the training set. Then, you still have the untainted test set to make your final selection between the model families!
So What is a cross-validation "pipeline?"¶
The best practice when performing CV is to include your data preprocessing steps inside the crossvalidation loop. This prevents accidentally tainting your training folds with influential data from your test fold. Here's how the CV pipeline looks after including preprocessing steps:- Split your data into k equal parts, or "folds" (typically k=10).
- Preprocess k-1 training folds.
- Train your model on the same k-1 folds.
- Preprocess the hold-out fold using the same transformations from step (2).
- Evaluate your model on the same hold-out fold.
- Perform steps (2) - (5) k times, each time holding out a different fold.
- Aggregate the performance across all k folds. This is your performance metric.
In [31]:
from sklearn.model_selection import GridSearchCV
clf = GridSearchCV(pipeline, hyperparameters, cv=10)
In [32]:
# Fit and tune model
clf.fit(X_train, y_train)
Out[32]:
Yes, it's really that easy. GridSearchCV essentially performs cross-validation across the entire "grid"
(all possible permutations) of hyperparameters.
It takes in your model (in this case, we're using a model pipeline), the hyperparameters you want to
tune, and the number of folds to create.
Obviously, there's a lot going on under the hood. We've included the pseudo-code above, and we'll
cover writing cross-validation from scratch in a separate guide.
Now, you can see the best set of parameters found using CV:
In [33]:
print (clf.best_params_)
Interestingly, it looks like the default parameters win out for this data set.
*Tip: It turns out that in practice, random forests don't actually require a lot of tuning. They tend to work pretty well out-of-the-box with a reasonable number of trees. Even so, these same steps can be used when building any type of supervised learning model.
*Tip: It turns out that in practice, random forests don't actually require a lot of tuning. They tend to work pretty well out-of-the-box with a reasonable number of trees. Even so, these same steps can be used when building any type of supervised learning model.
Step 7: Refit on the entire training set.¶
After you've tuned your hyperparameters appropriately using cross-validation, you can generally get a small performance improvement by refitting the model on the entire training set.
Conveniently, GridSearchCV from sklearn will automatically refit the model with the best set of hyperparameters using the entire training set.
This functionality is ON by default, but you can confirm it:
Conveniently, GridSearchCV from sklearn will automatically refit the model with the best set of hyperparameters using the entire training set.
This functionality is ON by default, but you can confirm it:
In [34]:
#Confirm model will be retrained
print(clf.refit)
Now, you can simply use the clf object as your model when applying it to other sets of data. That's
what we'll be doing in the next step.
Step 8: Evaluate model pipeline on test data¶
This step is really straightforward once you understand that the clf object you used to tune the
hyperparameters can also be used directly like a model object.
Here's how to predict a new set of data:
In [35]:
#Predict a new set of data
y_pred = clf.predict(X_test)
Now we can use the metrics we imported earlier to evaluate our model performance.
In [36]:
print (r2_score(y_test, y_pred))
In [37]:
print (mean_squared_error(y_test, y_pred))
Great, so now the question is... is this performance good enough?
Well, the rule of thumb is that your very first model probably won't be the best possible model.
However, we recommend a combination of three strategies to decide if you're satisfied with your
model performance.
- Start with the goal of the model. If the model is tied to a business problem, have you successfully solved the problem?
- Look in academic literature to get a sense of the current performance benchmarks for specific types of data.
- Try to find low-hanging fruit in terms of ways to improve your model.
- Try other regression model families (e.g. regularized regression, boosted trees, etc.).
- Collect more data if it's cheap to do so.
- Engineer smarter features after spending more time on exploratory analysis.
- Speak to a domain expert to get more context (...this is a good excuse to go wine tasting!). As a final note, when you try other families of models, we recommend using the same training and test set as you used to fit the random forest model. That's the best way to get a true apples-toapples comparison between your models.
Step 9: Save model for future use¶
Great job completing this tutorial!
You've done the hard part, and deserve another glass of wine. Maybe this time you can use your
shiny new predictive model to select the bottle.
But before you go, let's save your hard work so you can use the model in the future. It's really easy
to do so:
In [38]:
#Save model to a .pkl file
joblib.dump(clf, 'rf_regressor.pkl')
Out[38]:
And that's it. When you want to load the model again, simply use this function:
In [39]:
#Load model from .pkl file
clf2 = joblib.load('rf_regressor.pkl')
In [40]:
# Predict data set using loaded model
clf2.predict(X_test)
Out[40]:
Congratulations, you've reached the end of this tutorial!
We've just completed a whirlwind tour of Scikit-Learn's core functionality, but we've only really
scratched the surface. Hopefully you've gained some guideposts to further explore all that
sklearn has to offer.
0 Comments:
Post a Comment