Mean Prediction — Sam Yarney

Project description

The objective of this project is to use data mining and machine learning methods to predict the mean and variance of 200 realizations of Y for given pairs of (X1, X2). We are given two independent variables, X1 and X2, and response variables for the mean and variance of the 200 realizations, across 10,000 observations.

Exploratory data analysis

The correlation plot below shows no correlation between the two independent variables, X1 and X2. There is a strong positive correlation (0.83) between the two response variables, muhat (mean) and Vhat (variance). There is also a strong positive relationship between X1 and both response variables, and a low-to-medium negative correlation between X2 and both response variables.

Correlation plot between X1, X2, muhat, and Vhat — Correlation between predictors and response variables

The plot below shows the predictors (x-axis) against the response variables (y-axis). Consistent with the correlation plot, X1 has a positive slope against both muhat and Vhat, while X2 has a negative slope against both.

Scatter plots of X1 and X2 against muhat and Vhat showing slope direction — Predictors vs. response variables

Methods

I split the dataset into three parts: 70% training, 15% validation, and 15% testing. All models were trained on the training set for muhat and Vhat, then evaluated on the validation set by mean squared error (MSE). I selected the model with the lowest MSE, and confirmed that choice with 10-fold cross-validation on the training set, again comparing MSE across models. Final model selection was based on the cross-validation results.

I trained five models: linear regression, decision tree, random forest, boosting, and support vector machine (SVM). Boosting and SVM required parameter tuning — for boosting, I ran cross-validation to tune n.trees; for SVM, I fit the model across C values from 0.1 to 10,000 in steps of 10 and selected the value with the lowest MSE.

Results

SVM with a Gaussian kernel (C = 1,000) produced the lowest MSE for muhat (mean), with boosting a close second, followed by random forest and linear regression. Decision tree performed the worst on predicting muhat.

Table comparing model MSE for predicting muhat, with SVM performing best — Model comparison for muhat (mean)

Boosting produced the lowest MSE for Vhat (variance), with random forest second, followed by SVM and decision tree. Linear regression performed the worst on predicting Vhat.

Table comparing model MSE for predicting Vhat, with boosting performing best — Model comparison for Vhat (variance)

Conclusion

SVM with a Gaussian kernel (C = 1,000) performed best at predicting muhat (mean), and boosting performed best at predicting Vhat (variance). Random forest performed well on the training set but did not hold up as well on validation, testing, and cross-validation. The mean squared error for muhat is significantly larger than the mean squared error for Vhat.